SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Mon Nov 19, 2018 9:22 pm

All times are UTC + 3 hours




Post new topic Reply to topic  [ 32 posts ]  Go to page Previous  1, 2, 3  Next
Author Message
 Post subject:
PostPosted: Mon Jun 02, 2008 8:22 am 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
150 orders for each stock or in total?

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 02, 2008 9:05 am 
Offline

Joined: Thu Apr 17, 2008 10:57 am
Posts: 6
150 in total.

We running this strategy in a different system and we will have for the 10 years, at least 20'000 TRADES. Can SmartQuant handle that?


Top
 Profile  
 
 Post subject:
PostPosted: Mon Jun 02, 2008 12:16 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
Do you mean 20.000 trade objects (trade = market data) or 20.000 trade orders?

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Tue Jun 03, 2008 2:22 am 
Offline

Joined: Thu Apr 17, 2008 10:57 am
Posts: 6
20'000 trade orders.


Top
 Profile  
 
 Post subject:
PostPosted: Wed Jun 04, 2008 10:33 am 
Offline

Joined: Wed Oct 08, 2003 1:06 pm
Posts: 833
Hi,

I ran a test strategy on 1600 instruments each having 2300 daily bars. The strategy sends 20800 market orders and always has a position on each instrument. DataArrayLength is set to 31.

The final memory usage is 280 mb with pick usage around 620 mb.

Note, that the strategy doesn't use indicators, additional series and so on. It just sends buy and sell orders on first 13 bars.

If your strategy with the similar parameters (# of instruments, daily bars, orders, DataArrayLength) runs out of memory then probably the problem is in your strategy internal objects like indicators, arrays etc.

Regards,
Sergey.


Top
 Profile  
 
 Post subject:
PostPosted: Fri Sep 05, 2008 12:21 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
I just found a solution to several System.OutofMemory exceptions. Just raise the virtual memory in your system settings. Strategies which threw an exception earlier now do work now.


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 09, 2008 6:30 am 
Offline

Joined: Wed Aug 13, 2008 12:16 pm
Posts: 45
flotschie,

Did you ever figure out what was causing the leaks? I am experiencing similar problems running simulations of a 30-second chart. I raised my page file size, but it didn't help. The problem with looking for a memory leak is that it's small, and it takes so long to hit it -- it's like you change one thing, wait a long time, change one thing, wait a long time, etc.

What I can't figure out is if it will get through 1 iteration of the optimization cycle, why isn't everything zeroed out for the next iteration? I can get through 1 iteration, it seems like it should be able to get through all of them. This is the part I don't understand. I do wonder if it's storing a list of all the submitted orders somewhere, because I get an error in the FIX engine when it tries to submit a new order (that is where the out of memory error hits).

Is there some procedure that should be done OnStrategyStop to clear all arrays, TimeSeries, Orders, Indicators, etc. for the next loop?

Any ideas would be appreciated, otherwise I'll just keep on adding parts of my code, one section at a time, and seeing if the memory usage is growing...

Thanks,

Jeff


Top
 Profile  
 
 Post subject:
PostPosted: Tue Sep 09, 2008 3:26 pm 
Offline

Joined: Thu Jan 31, 2008 11:39 am
Posts: 166
Quote:
Did you ever figure out what was causing the leaks?


No. Eventually the number of Orders send (not executed) is too big.

Quote:
why isn't everything zeroed out for the next iteration?


Good question!

Quote:
Is there some procedure that should be done OnStrategyStop to clear all arrays, TimeSeries, Orders, Indicators, etc. for the next loop?


Also a good question, I do not have any code that clears tables etc...

I always try to minimize optimization effort. If a strategy does not work and suddenly some optimization parameters seems to work, it is not a good strategy. If a Strategy has too much parameters, it is not a good strategy. Just my 2 cents :-)


Top
 Profile  
 
 Post subject: Developers
PostPosted: Wed Sep 10, 2008 3:23 am 
Offline

Joined: Wed Aug 13, 2008 12:16 pm
Posts: 45
Do the developers have any comment on why strategies that get through 1 iteration can't get through multiple ones during optimization?

I've done a lot of looking for leaks, and I've found that in my code, if I comment out all the code that sends or cancels orders -- it will run all night just fine.

I have monitored the number orders in the Strategy.Orders.Count property, and while it grows to be quite large during 1 iteration, it resets before the next.

This to me makes me wonder if there's some sort of leak in the simulation engine that isn't freeing up all memory related to old orders before the next iteration. This could make some sense if the simulation engine isn't re-inited between iterations, but the strategy is.

I agree with flotschie that over-optimization can be a problem, but I don't think trying to run my strategy through 20 iterations is unreasonable, especially when it works just fine the first time through.


Thanks,

Jeff


Top
 Profile  
 
 Post subject:
PostPosted: Wed Sep 10, 2008 10:17 am 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
Hi,

I think it's hard to comment without looking into your strategy code. As you say, an empty strategy runs all night long, so there is no leak in OpenQuant engine.

Regards,
Anton


Top
 Profile  
 
PostPosted: Sun Apr 17, 2011 1:37 pm 
Offline

Joined: Thu Mar 10, 2011 10:09 pm
Posts: 582
Hi,

Did anybody solved the problem that OQ consumes more and more memory with each iteration run within oprimization process?
Probably you can advice some typical leaks to check ?


Top
 Profile  
 
PostPosted: Sun Apr 17, 2011 1:52 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
Basically you can send us your startegy or a smaple (your strategy without core logic for example) that causes memory leak and we will try to reproduce your issue.

Regards,
Anton


Top
 Profile  
 
PostPosted: Fri Apr 22, 2011 8:53 pm 
Offline

Joined: Thu Mar 10, 2011 10:09 pm
Posts: 582
Just to inform the community. Bug fix released for detected memory leaks with stop orders (ver 3.1.3)
Thanks guys!


Top
 Profile  
 
PostPosted: Wed Sep 14, 2011 5:40 pm 
Offline

Joined: Wed Apr 27, 2005 4:41 pm
Posts: 609
Location: Helsinki, Finland
I've been struggling with these out of memory errors for a long, long time.

It seems that regardless of how much memory or how limited the data arrays might be I never get beyond 2-3000 trades and then the out of memory hits. I naturally also use stop orders etc. Is it likely that this upgrade would help in this?

Thanks,
Eelofi


Top
 Profile  
 
PostPosted: Sun Sep 25, 2011 10:08 am 
Offline

Joined: Wed Apr 27, 2005 4:41 pm
Posts: 609
Location: Helsinki, Finland
Is the newer versions of OQ able to use the full memory available?

For example I just tested with an Amazon AC2 17GB memory machine. Before crashing with out of memory OQ is using only 1.1GB of memory leaving 14.9GB unused. This problem has in OQ for a long, long time.

I want to run longer simulations with generating bars from tick data , because that seems to give most realistic results. Now depeding on the lenght of the data contained in memory, I'm able to run from half year to 9 months, which still rather short and after that OQ crashes with out of memory regardless of how much memory the machine actually has.

    System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
    at System.Collections.Generic.SortedList`2.set_Capacity(Int32 value)
    at System.Collections.Generic.SortedList`2.EnsureCapacity(Int32 min)
    at System.Collections.Generic.SortedList`2.Insert(Int32 index, TKey key, TValue value)
    at System.Collections.Generic.SortedList`2.set_Item(TKey key, TValue value)
    at SmartQuant.Data.MemorySeries`1.set_Item(DateTime datetime, Object value)
    at SmartQuant.Series.DoubleSeries.Add(DateTime DateTime, Double Data)
    at SmartQuant.Series.DoubleSeries.set_Item(DateTime DateTime, Double value)
    at SmartQuant.Instruments.Performance.rLDcv4Mv4(DateTime , Double , Double )
    at SmartQuant.Instruments.Performance.FNhhj7AhT(DateTime )
    at SmartQuant.Instruments.Performance.rKPSv14oP(Object , PositionEventArgs )
    at SmartQuant.Instruments.Portfolio.G2NAA4jKf(Position )
    at SmartQuant.Instruments.Position.FNhhj7AhT(Object , TradeEventArgs )
    at SmartQuant.Providers.TradeEventHandler.Invoke(Object sender, TradeEventArgs args)
    at SmartQuant.Instruments.Instrument.rKPSv14oP(TradeEventArgs )
    at SmartQuant.Instruments.DataManager.cFcPv70CK(Object , TradeEventArgs )
    at SmartQuant.Providers.ProviderManager.lEAVp2xs7(Object , TradeEventArgs )
    at SmartQuant.Providers.TradeEventHandler.Invoke(Object sender, TradeEventArgs args)
    at SmartQuant.Simulation.SimulationDataProvider.GqUvMMZlL(IFIXInstrument , Trade )
    at SmartQuant.Simulation.SimulationDataProvider.dL0j4nY2c(SeriesObjectEventArgs )
    at SmartQuant.Simulation.SeriesObjectEventHandler.Invoke(SeriesObjectEventArgs args)
    at SmartQuant.Simulation.Simulator.ZfiSa3CUN(IDataSeries , IDataObject )
    at SmartQuant.Simulation.Simulator.hoSGsHSBq()
    at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
    at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
    at System.Threading.ThreadHelper.ThreadStart()


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 32 posts ]  Go to page Previous  1, 2, 3  Next

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group