SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Thu Jan 28, 2021 1:35 pm

All times are UTC + 3 hours




Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next
Author Message
PostPosted: Mon Apr 14, 2008 10:14 pm 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
This is a major problem.

This is OQ 2.1.3 running live with paper account at IB.

It appears that a partial fill does not immediately result in the Portolio getting updated with the new position resulting from that partial fill. Instead OQ appears to wait until the order is completed to either the Fill or Cancelled or other "Done" state before updating the Portfolio.

This is not desirable as the portfolio should immediately reflect the results of a partial fill.

It is particularly a problem when compounded by Orders that get stuck in the PendingCancel or other state, that have had partial fills, and thus never get added to the Portfolio.

IB positions and OQ positions then get permanently out-of-sync.

Regards,
DFT


Last edited by DFTrader on Tue Apr 15, 2008 5:39 am, edited 1 time in total.

Top
 Profile  
 
 Post subject:
PostPosted: Mon Apr 14, 2008 11:23 pm 
Offline

Joined: Wed Apr 27, 2005 4:41 pm
Posts: 609
Location: Helsinki, Finland
Yes, these problems have been around for quite some time. If IB TWS doesn't confirm an order or order cancellation for some reason, OQ has no way of getting rid of it. So you may have stuck orders piling up, without any method of getting rid of them.

Ideally OQ should take care of all orders confirmed or non confirmed. If this is not possible, then we should have a manual method to deleting unconfirmed orders, but that is not possible with OQ either.

At least my strategy tries all result in IB TWS and OQ being out of sync in orders. For example this evenings 288 orders yielded 8 stuck orders by the end of the trading day! My strategy tries to cancel those stuck orders all the time without success. I find this really frustrating that there is no way fix this problem even thought it has been know for a long time already. For example: http://www.smartquant.com/forums/viewto ... highlight=


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 15, 2008 12:27 am 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
I saw your post. Your situation appears to be another way that orders can get stuck. I believe my problems come from using OCA orders.

For any strategy, it is a bad situation if IB and OQ get stuck or out-of-synchronization on either positions or orders.

For my strategy, it makes the product unusable. I hope these issues can get fixed very, very soon - i.e., in the next release.

Regards,
DFT


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 15, 2008 12:26 pm 
Offline

Joined: Wed Oct 08, 2003 1:06 pm
Posts: 833
Hi DFT,

Could you please post a screenshot of the OrderManager showing this kind of stuck order and all of its execution reports?

Regards,
Sergey.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Apr 15, 2008 10:49 pm 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
Sergey,

Attached is a screen shot and test code that generated it.

If you run the code, it will generate lots of examples for you to debug of orders stuck in PendingCancel, PendingNew, and PartiallyFilled.

This test code will also result in the OQ Portfolio fairly quickly getting out-of-sync with IB.

Note that due to a bug in TWS paper trading, TWS may have to be reloaded if you observe the bid or ask for a security getting stuck.

In the attached screen shot you can see lots of order stuck in various states, even though they have all been cancelled by IB with OCA. The order.text shows the OCA Group name.

Regarding incorrect reporting, the particular order highlighting shows that the OrderQty for YHOO initated at 2:25:47 is 171. However, the order was placed with 644, and it was Replaced with 171. The report should show the original OrderQty (644), and the new one when the Replace is acknowledged. Here is the log from the code:

Quote:
4/15/2008 2:25:30 PM 205 YHOO Trade 4/15/2008 2:25:30 PM price=28.17 size=1
4/15/2008 2:25:32 PM 173 YHOO Trade 4/15/2008 2:25:32 PM price=28.18 size=1
4/15/2008 2:25:34 PM 423 YHOO Trade 4/15/2008 2:25:34 PM price=28.18 size=1
4/15/2008 2:25:35 PM 595 YHOO New LO Buy 187 28.17 2:25:28 PM-0
4/15/2008 2:25:36 PM 377 YHOO OrderStatusChanged Type=Limit Status=New
4/15/2008 2:25:36 PM 986 YHOO Cancel Buy 187 28.17 2:25:28 PM-0
4/15/2008 2:25:37 PM 377 YHOO OrderStatusChanged Type=Limit Status=PendingCancel
4/15/2008 2:25:39 PM 689 YHOO Trade 4/15/2008 2:25:39 PM price=28.17 size=2
4/15/2008 2:25:42 PM 455 YHOO Trade 4/15/2008 2:25:42 PM price=28.18 size=1
4/15/2008 2:25:46 PM 314 YHOO New LO Buy 644 28.17 2:25:28 PM-1
4/15/2008 2:25:47 PM 127 YHOO Trade 4/15/2008 2:25:47 PM price=28.18 size=3
4/15/2008 2:25:47 PM 173 YHOO Replace Buy 171 28.16 2:25:28 PM-1
4/15/2008 2:25:47 PM 236 YHOO OrderStatusChanged Type=Limit Status=New
4/15/2008 2:25:48 PM 095 YHOO OrderStatusChanged Type=Limit Status=PendingCancel
4/15/2008 2:25:51 PM 205 YHOO Trade 4/15/2008 2:25:51 PM price=28.18 size=1
4/15/2008 2:25:59 PM 267 YHOO New LO Sell 463 28.18 2:25:28 PM-1
4/15/2008 2:25:59 PM 533 YHOO New LO Buy 701 28.17 2:25:28 PM-1
4/15/2008 2:25:59 PM 861 YHOO OrderStatusChanged Type=Limit Status=Cancelled
4/15/2008 2:25:59 PM 861 YHOO OrderCancelled Type=Limit
4/15/2008 2:25:59 PM 861 YHOO OrderDone Type=Limit LastQty=0 Price=0
4/15/2008 2:26:00 PM 189 YHOO OrderStatusChanged Type=Limit Status=Cancelled
4/15/2008 2:26:00 PM 189 YHOO OrderCancelled Type=Limit
4/15/2008 2:26:00 PM 189 YHOO OrderDone Type=Limit LastQty=0 Price=0


Regards,
DFT


Attachments:
TestCode.zip [299.1 KiB]
Downloaded 390 times
Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 23, 2008 2:22 pm 
Offline

Joined: Wed Apr 27, 2005 4:41 pm
Posts: 609
Location: Helsinki, Finland
The test code downloads the positions from the broker and uses them as the basis of the portfolio. Would similar functionality be possible for open orders? This would be really excellent if it were possible.

Thanks,
Eelofi


Top
 Profile  
 
PostPosted: Wed Apr 23, 2008 9:00 pm 
Offline

Joined: Wed May 04, 2005 6:06 pm
Posts: 98
Location: SF Bay Area
DFTrader wrote:
It appears that a partial fill does not immediately result in the Portolio getting updated with the new position resulting from that partial fill. Instead OQ appears to wait until the order is completed to either the Fill or Cancelled or other "Done" state before updating the Portfolio.

I find it necessary to keep track of my positions in my own code. Then, I can use OnOrderDone() to see when an order is partially filled and add the total to my known position. This is awkward, but it works.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 23, 2008 11:01 pm 
Offline

Joined: Wed Apr 27, 2005 4:41 pm
Posts: 609
Location: Helsinki, Finland
DFTrader,

looking at your sample code it seems to fire the position adding procedure on each instrument added to the solution not only once. I think I have a ready fix for this which I'll into tomorrow and post here.

Br,
Eelofi


Top
 Profile  
 
 Post subject:
PostPosted: Thu Apr 24, 2008 12:09 am 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
Eelofi,

That is done on purpose. The code randomly, but simultaneously, adds buy/sell orders on multiple instruments, and has the potential to add them multiple times to the same instruments. This is just a test case to add lots of open orders and randomly hit them with replace and cancels.

However, any simultaneous placement of orders are all part of the same OCA Group, and should all get cancelled when anyone of them is cancelled or completely filled, according to the OCA rules.

The point is that, regardless of what orders the code places, replaces, or cancels:
1. Orders should never get stuck in PendingNew, PendingCancel, PartiallyFilled, etc. when in fact they have been cancelled
2. IB and OQ should never, never get out of sync on positions

Regards,
DFT


Top
 Profile  
 
 Post subject:
PostPosted: Thu Apr 24, 2008 11:55 am 
Offline

Joined: Fri May 06, 2005 1:40 am
Posts: 521
DFTrader wrote:
2. IB and OQ should never, never get out of sync on positions

Unfortunately this problem appears to be harder to solve than it sounds, probably due to the nature of having to rely on various third party API's to sync positions (On top of that even the most popular API IB-TWS continues to have bugs in this critical area... incredible).
DFTrader wrote:
This is a major problem.

For some time now people have been lining up to agree with you :-)
http://www.quanthouse.com/website/forum ... hp?p=12818
http://www.quanthouse.com/website/forum ... hp?p=13495
http://www.quanthouse.com/website/forum ... php?p=9227


Top
 Profile  
 
 Post subject:
PostPosted: Thu Apr 24, 2008 10:20 pm 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
Alex,

I have updated the test case code to clean up a few things: positions getting too large, unqiue order.txt when OCA disabled, and closing out all positions when strategy is stopped. I also don't recommend using GOOG as a test case instrument, as its bid/ask and volume is so different from the others, the code doesn't really exercise it very well. I am now using DELL,YHOO,MSFT as the instruments for my testing.

After further investigation, I have noticed that the problems generated by this test case are related to the use of either Replace or OCA. If I turn off using OCA orders (EnableOCA=false), and also inhibit Replace orders being sent (by if(RandomClass.NextDouble()<0.0)...), I have yet to observe OQ getting orders stuck or out-of-sync with TWS. I have run it for over an hour with no problems.

However, if you have OCA or Replace orders enabled, the simulation will very quickly generate stuck orders, especially with OCA enabled.

It must be related to the use of OCA and/or Replace - and again, there may be problems with messages occasionally not being sent by TWS, but that can't explain the near instantaneous creation of stuck orders and getting out-of-sync when Replace or OCA is used. If messages were not being sent, I would see stuck orders with just orders&cancels, but I do not. So the problem must be related to Order state processing by OQ, when orders are being replaced or cancelled by IB under OCA.

I have attached the updated code.

Regards,
DFT


Attachments:
File comment: Test Code
TestCode.zip [1.73 KiB]
Downloaded 397 times
Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 30, 2008 12:09 pm 
Offline

Joined: Fri May 25, 2007 1:57 am
Posts: 24
I observed the same OQ behavior as DFTrader. I also really wanted to deploy my strategies, so I ended up coding my error handling logic to the events I receive reliably, not the ones I had originally planned.

Here is what I did (nothing too clever, but got me going):
- When I cancel order1 in an OCA group, order2 is also cancelled in TWS, but stays in a PendingCancel status in OQ. What I did: when I want to cancel an OCA pair, I call Cancel() on both (yes, unnecessary, but code looks prettier !). I then assume that both orders were really cancelled (or wait to receive confirmation for order1 in OnOrderCancelled, which I always receive on cancellation). I null out my pointers to the orders. I forget about them. order2 stays in the OrderManager "Working" tab, and it is an annoyance, but it doesn't affect the running of my strategy.

- I send a long and short OCA pair. Short order gets rejected, but long order is working. What I did: I handled the OnOrderRejected event, and if the rejected order is an entry order, I cancel all other components of my entry. For me, there is always another entry signal I can use later. If the rejected order is an exit order, the code would resubmit it at the next bar -- but that should not happen, as there would be no reason for an exit order to be rejected.

- An order is partially filled and Portfolio is not updated. What I did: I handle OnOrderPartiallyFilled event. I keep track of my position. I create partial exit orders (quantity = entryOrder.CumQty). I Replace() these exit orders as the quantity changes (either more partial fills or OnOrderFilled). As my trades are in very liquid futures and ETFs, and my quantitities are reasonably low, I either don't get filled at all (handled at the next OnBar)or get the full order filled in at most a couple partial fills. I currently keep a partially filled order in the market until the full order is filled. However, I need to figure out what to do when I increase the size of my ETF orders and a partially filled order stays that way after a certain amount of time (maybe cancel the remaining order).

This is the one remaining big gap in my code, and I was able to ignore it for now because I primarily trade futures with OQ and can achieve the $ nominal volume I want with less than 10 contracts per each instrument I trade.

- I did not have a specific problem with Replace().

I was able to reach the level of operational robustness I wanted by using OnOrderFilled, OnOrderPartiallyFilled, OnOrderRejected, and OnOrderCancelled events. I don't use OnPosition... or the remaining OnOrder... functions.

I also have a panic timer (OnTimer) that, if triggered, means the code didn't press the red button in time (something is not right). The only case I see this on a regular basis is not receiving a 1-minute bar, either because there were no trades and OQ didn't generate a bar, or IB market data was interrupted.

I keep track of my orders until I complete the full roundtrip, and reliably use IsPartiallyFilled, IsFilled, and IsDone properties of orders in my logic (being aware that IsDone is false for a cancelled order with PendingCancel status).

I initially had some silly bugs: for example once every other day or so, timing would work out that I would call Cancel() and null out my order pointer just as the order executed but before I received the OnOrderFilled event or IsFilled was true. Classical real-time coding error. In a rare moment of clarity (my v3 implementation), I finally drew the state diagram of the lifetime of my entry and exit orders, and mapped them to the above events and flags. Lucky for me, I could carry this to all my strategies.

These allowed me to deploy my strategies where IB and OQ don't get out of sync, and the trades I get are the trades I expect. I still have work to do before I can add less liquid ETFs & stocks, and to increase my order quantitites. Sorry for the long message, and I hope I was able to offer something beyond the obvious.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 30, 2008 3:56 pm 
Offline

Joined: Fri May 06, 2005 1:40 am
Posts: 521
Interesting post dareminator. You might find the ".NET State Machine Toolkit" link of interest, posted here: http://www.smartquant.com/forums/viewto ... 8838#18838


Top
 Profile  
 
 Post subject:
PostPosted: Wed Apr 30, 2008 5:43 pm 
Offline

Joined: Fri May 25, 2007 1:57 am
Posts: 24
krn_2k wrote:
Interesting post dareminator. You might find the ".NET State Machine Toolkit" link of interest, posted here: http://www.smartquant.com/forums/viewto ... 8838#18838

Thank you for the pointer krn_2k. It does look interesting. I will play with it when I have some time (and will post my observations when I do).


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 01, 2008 8:34 pm 
Offline

Joined: Thu Oct 25, 2007 12:11 am
Posts: 86
Location: Austin, Texas
Dareminator,

Thanks for the detailed post.

Seems like you went through a Herculean effort to work around the OQ order handling bugs.

Obviously, it would be far, far better if SmartQuant would fix these bugs so that the rest of us, and future customers, all don't have to do the same thing. I am also concerned that even if one can patch around things, and it tests OK, what other bugs are lurking in a patched-up order handling that haven't occurred yet in necessarily limited testing and live trading? Often, if code has known problems in state processing, when it is cleaned up and rigorously tested with tough and thorough test benches, multiple bugs are cleaned out, even ones that have yet to be found.

One can easily imagine devestating consequences in a high frequency long/short strategy that goes awry because of misprocessed order handling. E.G, what if a partial or full fill message gets dropped or misprocessed?

I will request an update from Smartquant on their debugging effort:
http://www.smartquant.com/forums/viewto ... highlight=

As to your post, here are some specific comments/questions:
Quote:
When I cancel order1 in an OCA group, order2 is also cancelled in TWS, but stays in a PendingCancel status in OQ. What I did: when I want to cancel an OCA pair, I call Cancel() on both...

I have done the same thing. Downside: more messages from TWS and in the audit trail, saying the cancel order has been rejected.
Quote:
the rejected order is an exit order, the code would resubmit it at the next bar -- but that should not happen, as there would be no reason for an exit order to be rejected

Why would their be a difference between the possible rejection of an entry vs. an exit order?
Quote:
An order is partially filled and Portfolio is not updated. What I did: I handle OnOrderPartiallyFilled event. I keep track of my position...

So you keep a side-count of partially filled orders? By using portfolio.add() or just in your internal database? In either case, and if the order does go to the orderdone state, you have to undo this as the portfolio.positions will get updated by OQ?
Quote:
I also have a panic timer (OnTimer) that, if triggered, means the code didn't press the red button in time (something is not right). The only case I see this on a regular basis is not receiving a 1-minute bar, either because there were no trades and OQ didn't generate a bar, or IB market data was interrupted...

You may also want to look at the provider error messages using OnError(), and look for at a minimum codes 2103 (Market data farm connection is broken) and 1100 (Connectivity between IB and TWS has been lost). This way you can get a little faster response time on knowing there is a problem, and suspend placing orders and/or sound the alarm...

Another safety check I have thought of is to perdiocially use DataManager.GetBrokerInfo("IB").Accounts to check against the internal Portfolio.Positions to see if there is a mismatch, and sound the alarm if there is. Haven't coded it up yet, and there are some timing issues.

Regards,
DFT


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 21 posts ]  Go to page 1, 2  Next

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group