SmartQuant Discussion

This is a major problem.

This is OQ 2.1.3 running live with paper account at IB.

It appears that a partial fill does not immediately result in the Portolio getting updated with the new position resulting from that partial fill. Instead OQ appears to wait until the order is completed to either the Fill or Cancelled or other "Done" state before updating the Portfolio.

This is not desirable as the portfolio should immediately reflect the results of a partial fill.

It is particularly a problem when compounded by Orders that get stuck in the PendingCancel or other state, that have had partial fills, and thus never get added to the Portfolio.

IB positions and OQ positions then get permanently out-of-sync.

Regards,
DFT

Yes, these problems have been around for quite some time. If IB TWS doesn't confirm an order or order cancellation for some reason, OQ has no way of getting rid of it. So you may have stuck orders piling up, without any method of getting rid of them.

Ideally OQ should take care of all orders confirmed or non confirmed. If this is not possible, then we should have a manual method to deleting unconfirmed orders, but that is not possible with OQ either.

At least my strategy tries all result in IB TWS and OQ being out of sync in orders. For example this evenings 288 orders yielded 8 stuck orders by the end of the trading day! My strategy tries to cancel those stuck orders all the time without success. I find this really frustrating that there is no way fix this problem even thought it has been know for a long time already. For example: http://www.smartquant.com/forums/viewto ... highlight=

I saw your post. Your situation appears to be another way that orders can get stuck. I believe my problems come from using OCA orders.

For any strategy, it is a bad situation if IB and OQ get stuck or out-of-synchronization on either positions or orders.

For my strategy, it makes the product unusable. I hope these issues can get fixed very, very soon - i.e., in the next release.

Regards,
DFT

Hi DFT,

Could you please post a screenshot of the OrderManager showing this kind of stuck order and all of its execution reports?

Regards,
Sergey.

Sergey,

Attached is a screen shot and test code that generated it.

If you run the code, it will generate lots of examples for you to debug of orders stuck in PendingCancel, PendingNew, and PartiallyFilled.

This test code will also result in the OQ Portfolio fairly quickly getting out-of-sync with IB.

Note that due to a bug in TWS paper trading, TWS may have to be reloaded if you observe the bid or ask for a security getting stuck.

In the attached screen shot you can see lots of order stuck in various states, even though they have all been cancelled by IB with OCA. The order.text shows the OCA Group name.

Regarding incorrect reporting, the particular order highlighting shows that the OrderQty for YHOO initated at 2:25:47 is 171. However, the order was placed with 644, and it was Replaced with 171. The report should show the original OrderQty (644), and the new one when the Replace is acknowledged. Here is the log from the code:



Regards,
DFT

The test code downloads the positions from the broker and uses them as the basis of the portfolio. Would similar functionality be possible for open orders? This would be really excellent if it were possible.

Thanks,
Eelofi

I find it necessary to keep track of my positions in my own code. Then, I can use OnOrderDone() to see when an order is partially filled and add the total to my known position. This is awkward, but it works.

DFTrader,

looking at your sample code it seems to fire the position adding procedure on each instrument added to the solution not only once. I think I have a ready fix for this which I'll into tomorrow and post here.

Br,
Eelofi

Eelofi,

That is done on purpose. The code randomly, but simultaneously, adds buy/sell orders on multiple instruments, and has the potential to add them multiple times to the same instruments. This is just a test case to add lots of open orders and randomly hit them with replace and cancels.

However, any simultaneous placement of orders are all part of the same OCA Group, and should all get cancelled when anyone of them is cancelled or completely filled, according to the OCA rules.

The point is that, regardless of what orders the code places, replaces, or cancels:
1. Orders should never get stuck in PendingNew, PendingCancel, PartiallyFilled, etc. when in fact they have been cancelled
2. IB and OQ should never, never get out of sync on positions

Regards,
DFT

Unfortunately this problem appears to be harder to solve than it sounds, probably due to the nature of having to rely on various third party API's to sync positions (On top of that even the most popular API IB-TWS continues to have bugs in this critical area... incredible).

For some time now people have been lining up to agree with you
http://www.quanthouse.com/website/forum ... hp?p=12818
http://www.quanthouse.com/website/forum ... hp?p=13495
http://www.quanthouse.com/website/forum ... php?p=9227

Alex,

I have updated the test case code to clean up a few things: positions getting too large, unqiue order.txt when OCA disabled, and closing out all positions when strategy is stopped. I also don't recommend using GOOG as a test case instrument, as its bid/ask and volume is so different from the others, the code doesn't really exercise it very well. I am now using DELL,YHOO,MSFT as the instruments for my testing.

After further investigation, I have noticed that the problems generated by this test case are related to the use of either Replace or OCA. If I turn off using OCA orders (EnableOCA=false), and also inhibit Replace orders being sent (by if(RandomClass.NextDouble()<0.0)...), I have yet to observe OQ getting orders stuck or out-of-sync with TWS. I have run it for over an hour with no problems.

However, if you have OCA or Replace orders enabled, the simulation will very quickly generate stuck orders, especially with OCA enabled.

It must be related to the use of OCA and/or Replace - and again, there may be problems with messages occasionally not being sent by TWS, but that can't explain the near instantaneous creation of stuck orders and getting out-of-sync when Replace or OCA is used. If messages were not being sent, I would see stuck orders with just orders&cancels, but I do not. So the problem must be related to Order state processing by OQ, when orders are being replaced or cancelled by IB under OCA.

I have attached the updated code.

Regards,
DFT

I observed the same OQ behavior as DFTrader. I also really wanted to deploy my strategies, so I ended up coding my error handling logic to the events I receive reliably, not the ones I had originally planned.

Here is what I did (nothing too clever, but got me going):
- When I cancel order1 in an OCA group, order2 is also cancelled in TWS, but stays in a PendingCancel status in OQ. What I did: when I want to cancel an OCA pair, I call Cancel() on both (yes, unnecessary, but code looks prettier !). I then assume that both orders were really cancelled (or wait to receive confirmation for order1 in OnOrderCancelled, which I always receive on cancellation). I null out my pointers to the orders. I forget about them. order2 stays in the OrderManager "Working" tab, and it is an annoyance, but it doesn't affect the running of my strategy.

- I send a long and short OCA pair. Short order gets rejected, but long order is working. What I did: I handled the OnOrderRejected event, and if the rejected order is an entry order, I cancel all other components of my entry. For me, there is always another entry signal I can use later. If the rejected order is an exit order, the code would resubmit it at the next bar -- but that should not happen, as there would be no reason for an exit order to be rejected.

- An order is partially filled and Portfolio is not updated. What I did: I handle OnOrderPartiallyFilled event. I keep track of my position. I create partial exit orders (quantity = entryOrder.CumQty). I Replace() these exit orders as the quantity changes (either more partial fills or OnOrderFilled). As my trades are in very liquid futures and ETFs, and my quantitities are reasonably low, I either don't get filled at all (handled at the next OnBar)or get the full order filled in at most a couple partial fills. I currently keep a partially filled order in the market until the full order is filled. However, I need to figure out what to do when I increase the size of my ETF orders and a partially filled order stays that way after a certain amount of time (maybe cancel the remaining order).

This is the one remaining big gap in my code, and I was able to ignore it for now because I primarily trade futures with OQ and can achieve the $ nominal volume I want with less than 10 contracts per each instrument I trade.

- I did not have a specific problem with Replace().

I was able to reach the level of operational robustness I wanted by using OnOrderFilled, OnOrderPartiallyFilled, OnOrderRejected, and OnOrderCancelled events. I don't use OnPosition... or the remaining OnOrder... functions.

I also have a panic timer (OnTimer) that, if triggered, means the code didn't press the red button in time (something is not right). The only case I see this on a regular basis is not receiving a 1-minute bar, either because there were no trades and OQ didn't generate a bar, or IB market data was interrupted.

I keep track of my orders until I complete the full roundtrip, and reliably use IsPartiallyFilled, IsFilled, and IsDone properties of orders in my logic (being aware that IsDone is false for a cancelled order with PendingCancel status).

I initially had some silly bugs: for example once every other day or so, timing would work out that I would call Cancel() and null out my order pointer just as the order executed but before I received the OnOrderFilled event or IsFilled was true. Classical real-time coding error. In a rare moment of clarity (my v3 implementation), I finally drew the state diagram of the lifetime of my entry and exit orders, and mapped them to the above events and flags. Lucky for me, I could carry this to all my strategies.

These allowed me to deploy my strategies where IB and OQ don't get out of sync, and the trades I get are the trades I expect. I still have work to do before I can add less liquid ETFs & stocks, and to increase my order quantitites. Sorry for the long message, and I hope I was able to offer something beyond the obvious.

Interesting post dareminator. You might find the ".NET State Machine Toolkit" link of interest, posted here: http://www.smartquant.com/forums/viewto ... 8838#18838

Thank you for the pointer krn_2k. It does look interesting. I will play with it when I have some time (and will post my observations when I do).

Dareminator,

Thanks for the detailed post.

Seems like you went through a Herculean effort to work around the OQ order handling bugs.

Obviously, it would be far, far better if SmartQuant would fix these bugs so that the rest of us, and future customers, all don't have to do the same thing. I am also concerned that even if one can patch around things, and it tests OK, what other bugs are lurking in a patched-up order handling that haven't occurred yet in necessarily limited testing and live trading? Often, if code has known problems in state processing, when it is cleaned up and rigorously tested with tough and thorough test benches, multiple bugs are cleaned out, even ones that have yet to be found.

One can easily imagine devestating consequences in a high frequency long/short strategy that goes awry because of misprocessed order handling. E.G, what if a partial or full fill message gets dropped or misprocessed?

I will request an update from Smartquant on their debugging effort:
http://www.smartquant.com/forums/viewto ... highlight=

As to your post, here are some specific comments/questions:

I have done the same thing. Downside: more messages from TWS and in the audit trail, saying the cancel order has been rejected.

Why would their be a difference between the possible rejection of an entry vs. an exit order?

So you keep a side-count of partially filled orders? By using portfolio.add() or just in your internal database? In either case, and if the order does go to the orderdone state, you have to undo this as the portfolio.positions will get updated by OQ?

You may also want to look at the provider error messages using OnError(), and look for at a minimum codes 2103 (Market data farm connection is broken) and 1100 (Connectivity between IB and TWS has been lost). This way you can get a little faster response time on knowing there is a problem, and suspend placing orders and/or sound the alarm...

Another safety check I have thought of is to perdiocially use DataManager.GetBrokerInfo("IB").Accounts to check against the internal Portfolio.Positions to see if there is a mismatch, and sound the alarm if there is. Haven't coded it up yet, and there are some timing issues.

Regards,
DFT