Websocket/API advice?

[Jason]

Checking out other posts here has been very helpful to me so I thought I would ask this here in case the discussion is of value to anyone else developing an application using the API.

In my workflow up to this point the software I created has been designed to facilitate the opening/closing of multiple positions with each trade being independently managed by a separate instance of a script which would monitor the price of the symbol and take actions conditionally. None of the code could be multithreaded and the implementation made use of Windows IPC to allow the various scripts running to communicate with each other as needed. This solution is clunky and now, with the new API, avoidable.

I have begun developing a replacement for my old workflow by adapting the APITester program you guys provided to mirror the functionality of that workflow. I can send commands, parse the JSON responses, etc. but I'm not sure of the best path to take in duplicating the ability to manage multiple positions.

My idea is to create a new thread to take care of managing each trade. As I see it though, the problem becomes making sure that each response received is directed to the thread that generated the corresponding request. I could try matching them by the "reqID" through a complex function that processes all of the responses from the ReceiveBuffer but this feels like a very heavy-handed or brute force way of dealing with it.

Is there any way for a request to be sent from a particular thread and have the response automatically directed back to that thread?

Would it be possible/better for each thread to open a separate websocket/API connection?

Just asking for some advice on how I could proceed as I don't have experience with multithreaded programming or with websockets.

Thanks!

[Mike Medved]

Jason, MT's Websocket API is mostly asynchronous.

That means that you send a command and don't wait for immediate response to the command. The response will just come when it comes (usually pretty fast but still). You can match the command to the response by sending a unique reqID in the command and matching it to the reqID in the response.

The part that processes responses and decides where to direct it does not really has to be multithreaded - it should be very fast. But once you receive and parse the response you can direct it to a separate thread or queue if you like. Although I would wonder if that's overkill. Usually such processing is fast enough and the events that happen are (comparatively) rare enough (that is, not 100s a second) that multithreading may not be needed.

[Jason]

Thanks for the feedback.

Would it be feasible though for each thread managing a trade to establish it's own connection through the websocket? The reason I am attracted to this idea is that I believe doing so would eliminate the need for any parsing to forward responses. As I understand it each response would automatically go right to the thread that issued the request through its own connection eliminating the need for any overhead in keeping track of which responses belong to which trades. But I'm not sure if that's possible.

[Mike Medved]

You can definitely just have a separate connection - the internal server can handle thousands of connections.

[Jason]

Okay, very interesting, thanks! Will just have to start researching how to do that with the websocket interface.

[Jerry Medved]

note though that if you have separate connections and getting trading updates on all, you will get all updates on all connections, so each one will need to filter out what it does not need. Would be more efficient to have single thread getting updates. Then if you want, just have a dictionary of symbol/UpdateQue - that would get updated by that thread and set a semaphore.  Your other threads (one per symbol or position) would just wait for the update trigger and then check just their entry in the dictionary and process the entries in that queue

[Jason]

Thanks, just the kind of heads up I was needing. I had anticipated there might be some roadblocks of that sort if I tried this approach especially since I don't exactly know how websockets will behave.

Okay, when you refer to updates do you mean specifically the streaming quotes and trade position updates? Currently, my understanding is that if I create a thread with its own socket connection which will manage a specific trade then any command sent from that thread would receive a specific response sent back to only that thread via that threads connection. If my thinking about this is correct, I'm looking at dedicating a single thread to receive quotes/updates as you suggest and then taking actions (placing trades, maybe requesting historical candle data for the symbol, etc.) from within the threads managing the trades.

The endgame will be to have alerts triggered off of paintbars sent to the corresponding thread managing a trade which will then trigger actions. Which sounds pretty complex considering the framework I'm looking at but I'll cross that bridge when I come to it. Will definitely have to read up on using dictionaries and semaphores etc. All the multi-threading and Windows Forms GUI stuff is the steepest part of the learning curve for me at this point. On the whole though, it's been surprisingly easy adapting to a C# environment.

[Jerry Medved]

any command you send, will get a response back on that thread.  However, there are unsolicited messages sent as well. Any time any transactions, positions, balances change, those values are sent to all threads that subscribe to that data. So even if you send the order on thread 1 (of 10), all 10 will end up getting transaction update message once the order shows up in transactions.  

[Jason]

Got it. Though I'm having quite a bit of difficulty getting a separate connection set up. I can make a connection through a separate function called by a new thread but it's not obvious to me how to separate the responses received from commands sent over the connection by that thread from those intended for the main thread. This use case seems a little outside of what the APITeser program was designed to accommodate and I wasn't sure how to proceed. So without messing with the mulithreading part just yet I tried creating a separate object of type WebSocketConnection and duplicating the relevant functions for sending/receiving data so that they reference that new connection name instead of MainSocket. I assumed there was an easier way but the idea was to create a proof-of-concept that would allow me to play with sending and receiving commands through two separate connections. The result was absolutely no response from the server through the 2nd connection and no errors or crashes to speak of.

Can you point me in a better direction?

[Jerry Medved]

ok lets step back a bit. I don't know what language you are using for your development, but if C#, I can give more specific suggestions

My general suggestion - forget doing multiple connections. It really does not seem to help any in your situation.  Just make it so everything goes through the one connection.

Create some sort of a list of active orders that you wish to track.  Whenever you receive any transaction/position data, look through the list of active orders and update the ones that are relevant.  

[Jason]

Oh yes I'm using C#. Initially I was attracted to using separate connections but I do think that just having a single connection is most likely not going to create any bottlenecks. Eventually will need to be able to monitor as many as 16 trades simultaneously each one monitoring the last price traded for order placement. And it's not like having a single connection would prevent it from being multithreaded anyway.