SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Mon Jul 23, 2018 12:41 am

All times are UTC + 3 hours




Post new topic Reply to topic  [ 32 posts ]  Go to page 1, 2, 3  Next
Author Message
PostPosted: Thu Feb 21, 2008 6:38 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Hi again,

we just wanted to know what we can do to prevent the System.OutOfMemory Exception shooting OpenQuant. Now, every time the OOM Exception shows up, the Error Message shows up again and again so we can't click the Strategy Stop Button and have to shut down OQ via the Task Manager.

We run almost a year of tickdata, with indicators that need 200 bars of 1 second bars. We tried different settings in the DataManager, such as setting Bars, Trades and Quote Arrays to values like 300, 400 or 1000. We assumed that OQ stores only this quantity of data in the memory because only the last 300 bars (for example) are needed for the calculations.

But we notice in the Task Manager how Open Quant consumes more and more memory. So in certain cases, we have to stop the Strategy before the OOM Exception shows up to prevent loosing at least the half of the results. For Example, certain test can only be made from January until August, with the rest of the year left out.

Maybe we do not exactly know how the settings in Data Manager work and you could explain it more in detail. Aditionally it would be perfect if you could give some hints how the memory issue can be solved by adjusting the code, are there maybe some clean up procedures we could insert?

Kr
Hans Moleman :shock:


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 21, 2008 6:56 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6807
Hi,

this memory problem can be caused by your strategy logic as well , so I would suggest you to start with empty strategy that just receives market data and stores it in Bars, Trades, etc. Run the strategy, check that the memory doesn't grow. If it grows, please tell us. If not, add indicators and other logic bit by bit and see what happens with memory allocation.

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Fri Feb 22, 2008 11:10 am 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Thank you Anton. One thing I can say for sure is that increasing the length of a normal SMA causes the exceptions. We are working with only two SMA Indicators, the longer one (only) needs 200 1-Second-Bars.

The backtests work well with SMA(200) but when we started experimenting with SMA(300) the exceptions occur. Of course we adapted the DataManager settings, setting Bars/Trades/Quotes to 400 in this case.

Anyway, we will try out what you recommend and let you know.

Thx H.M.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Feb 22, 2008 2:07 pm 
Offline

Joined: Wed Oct 08, 2003 1:06 pm
Posts: 833
Hi,

I have run an empty strategy on 1 year of forex quotes and 1 sec bars (3.8M quotes, 200K bars) creating two SMAs (length 200 and 300) setting Bars/Trades/Quotes to 400. Memory has been increased by around 60M and no OOM exception.

P.S. Using SMA(300) instead of SMA(200) should not increase memory usage (even decrease it because SMA(200) is longer than SMA(300)). Exception can be caused by the your strategy logic (it sends more orders when using SMA(300) etc.)

Regards,
Sergey.


Top
 Profile  
 
 Post subject:
PostPosted: Sat Feb 23, 2008 1:24 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Yes thank you, you are absolutely right. We tested the issue and came to the conclusion that it is the Order process which consumes the memory. The more Orders (and therefore Trades) are generated, the more memory is consumed.

We now set Trades to 30.000. And still, after approximately 3500 Trades, this error message appears:

System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.SortedList`2.set_Capacity(Int32 value)
at System.Collections.Generic.SortedList`2.EnsureCapacity(Int32 min)
at System.Collections.Generic.SortedList`2.Insert(Int32 index, TKey key, TValue value)
at System.Collections.Generic.SortedList`2.set_Item(TKey key, TValue value)
at SmartQuant.Data.MemorySeries`1.Add(DateTime datetime, Object obj)
at SmartQuant.Indicators.Indicator.Add(DateTime DateTime, Double Data)
at SmartQuant.Indicators.SMA.Calculate(Int32 index)
at SmartQuant.Indicators.SMA.OnInputItemAdded(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Indicators.Indicator.OnInputItemAdded2(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Series.ItemAddedEventHandler.Invoke(Object sender, DateTimeEventArgs e)
at SmartQuant.Series.TimeSeries.EmitItemAdded(DateTime dateTime)
at SmartQuant.Series.BarSeries.Add(Bar bar)
at SmartQuant.Instruments.DataManager.AG)j1eCB6qf4iopnlLW2qhjb(Object , BarEventArgs )
at SmartQuant.Providers.ProviderManager.AGQ)SMo8qDLV(Object , BarEventArgs )
at SmartQuant.Providers.BarEventHandler.Invoke(Object sender, BarEventArgs args)
at SmartQuant.Simulation.SimulationDataProvider.AQkXjYICv(Object , BarEventArgs )
at SmartQuant.Providers.BarFactory.ACYR94A1(Bar , IFIXInstrument )
at SmartQuant.Providers.BarFactory.AF99)8KlPNNG)HDxAm05IPd(ReminderEventArgs )
--- End of inner exception stack trace ---
at System.RuntimeMethodHandle._InvokeMethodFast(Object target, Object[] arguments, SignatureStruct& sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.RuntimeMethodHandle.InvokeMethodFast(Object target, Object[] arguments, Signature sig, MethodAttributes methodAttributes, RuntimeTypeHandle typeOwner)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture, Boolean skipVisibilityChecks)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at SmartQuant.Clock.ABo(jpKVUxGDuDw(DateTime )

OQ continues after clicking this message away. But several minutes later, this message appears again and Open Quant "has dedected a problem and must be closed".

So is there a limited number of Trades we can have in a backtest?


Top
 Profile  
 
 Post subject:
PostPosted: Wed Feb 27, 2008 12:10 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Do you have any suggestions for this yet?


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 2:58 pm 
Offline

Joined: Wed Oct 08, 2003 1:06 pm
Posts: 833
Hi,

Could you sum up please

- what are the settings of the Data Manager (Bar array length, Trade array length, Quote array length)

- what market data requests does your strategy uses (Market Data node in the Solution Explorer window). Whether bar requests are built from trades or not.

- how many orders the strategy generate

- how many bars are generated

you can calculate 3rd and 4th values using counters, incrementing them in OnNewOrder and OnNewBar handlers correspondingly and print these counters after each let say 100 objects.

Regards,
Sergey.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 4:50 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
- settings of the Data Manager:

Bars: 10.000.000
Trades: 10.000
Quotes: 10.000

- Market Data Requests: "Bars 1 sec". We add it like "Market Data --> Add --> Bar --> Custom --> Bartype=Time and Bar size = 1 and leave "Build Bars from Trades" checked.

- This time the Strategy generates only 750 Trades.

- The exception came after 4.194.000 Bars.

- Task Manager showed about 650.000K for OpenQuant process at the end.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 5:03 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6807
Hi,

one Bar object occupies ~200 bytes of memory. 5M bars eat ~1GB and if .NET needs to allocate additional space for bar array, it reserves an array with double size, i.e. needs ~3GB to add another entry to the bar array. Plus you have ~5M SMA entries, plus trades, plus quotes, plus orders, etc. I am not surprised that you are out of memory. The question is why do you need to keep 10M bars in memory?

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 6:39 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
OK yes we understand now, we did not know how much memory space is consumed by one bar object.

So we just tried setting Bar Array lenght to 10.000, ran the strategy (complete with creation and closing positions, aprx. 750 Trades) again and got the same exception.

Then we tried to run the strategy without sending orders again and it worked without exceptions.

Settings as described above. So the 750 Trades also consume too much memory?


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 7:08 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Sorry, the number of trades are not 750 but approximately 5000, but still, we now set the trade array to 20.000 and memory still grows to much. How much memory is consumed by one trade?

Edith: Sergey, did you try to run the strategy (you described above) with trades - lets say 5000? Would be interesting if you get the same exceptions or see the memory rise.


Top
 Profile  
 
 Post subject:
PostPosted: Thu Feb 28, 2008 8:22 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6807
Actually trades consume a lot of memory because each trade produces

- an order
- several execution reports
- portfolio transaction

so it consumes a lot of memory but 5K trades should be no problem. Sergey will check tomorrow.

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 22, 2008 12:12 am 
Offline

Joined: Thu Mar 16, 2006 12:15 pm
Posts: 184
Any update? Got the same problem.

_________________
Expect the unexpected. May your MM/RM be with you.


Top
 Profile  
 
 Post subject:
PostPosted: Thu May 22, 2008 11:04 am 
Offline

Joined: Thu Apr 17, 2008 10:57 am
Posts: 6
Hi,

I have also out of memory problem.

I am running one strategy with 1600 hundreds stocks for 10 years.
- Data Manager settings:
Bar array length = 31
Trade array length = 1
Quote array length=0

- market data are daily
- order strategy generate =150 before out of memory and then crashing

I still have an other 9 years to run and then I will add a portfolio filter using a SQL connection. So I wonder how I will be able to do that without running out of memory.

Any inputs?

Thank you,


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 02, 2008 4:14 am 
Offline

Joined: Thu Apr 17, 2008 10:57 am
Posts: 6
I am still waiting for an answer. Any help? I am wondering about the support for smarquant seems pretty slow. :(


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 32 posts ]  Go to page 1, 2, 3  Next

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 2 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group