SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Wed Aug 15, 2018 3:49 am

All times are UTC + 3 hours




Post new topic Reply to topic  [ 7 posts ] 
Author Message
PostPosted: Mon Jun 16, 2014 9:43 pm 
Offline

Joined: Fri Apr 25, 2014 10:56 am
Posts: 9
Hi,

I'm working with a strategy where the set of indicators & parameters is constantly changing.
Therefor I need to construct new Instances of the Indicator class on every bar (onBar).

A strategy coded in this way does run, but it is very slow and in backtesting after some hunderd bars I run into this OutOfMemory error.
I've verified that the calling the Indicator class construction at every bar is indeed the problem. If I construct the Indicators at OnStrategyStart, so with parameters kept constant during the simulation, there is no problem.

What approach would you suggest for my situation ?

Regards,

Frank

Quote:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.SortedList`2.set_Capacity(Int32 value)
at System.Collections.Generic.SortedList`2.EnsureCapacity(Int32 min)
at System.Collections.Generic.SortedList`2.Insert(Int32 index, TKey key, TValue value)
at System.Collections.Generic.SortedList`2.set_Item(TKey key, TValue value)
at SmartQuant.Data.MemorySeries`1.Add(DateTime datetime, Object obj)
at SmartQuant.Indicators.Indicator.Add(DateTime DateTime, Double Data)
at SmartQuant.Indicators.EMA.Calculate(Int32 index)
at SmartQuant.Indicators.EMA.OnInputItemAdded(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Indicators.Indicator.OnInputItemAdded2(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Series.ItemAddedEventHandler.Invoke(Object sender, DateTimeEventArgs e)
at SmartQuant.Series.TimeSeries.EmitItemAdded(DateTime dateTime)
at SmartQuant.Series.BarSeries.Add(Bar bar)
at SmartQuant.Instruments.BarSeriesList.l9GsJ1Y3L(Instrument , Bar )
at SmartQuant.Instruments.DataManager.FXeMVYXvd(Object , BarEventArgs )
at SmartQuant.Providers.ProviderManager.donsPjTgF5(Object , BarEventArgs )
at SmartQuant.Providers.BarEventHandler.Invoke(Object sender, BarEventArgs args)
at SmartQuant.Simulation.SimulationDataProvider.oB8NQOfhqM(IFIXInstrument , Bar )
at SmartQuant.Simulation.SimulationDataProvider.QGYNAiGibJ(SeriesObjectEventArgs )
at SmartQuant.Simulation.SeriesObjectEventHandler.Invoke(SeriesObjectEventArgs args)
at SmartQuant.Simulation.Simulator.dEG48R9xls(IDataSeries , IDataObject )
at SmartQuant.Simulation.Simulator.Asx4Vq8104()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()


Top
 Profile  
 
PostPosted: Tue Jun 17, 2014 9:38 pm 
Offline

Joined: Fri Apr 25, 2014 10:56 am
Posts: 9
Frank Weissbach wrote:
Hi,

I'm working with a strategy where the set of indicators & parameters is constantly changing.
Therefor I need to construct new Instances of the Indicator class on every bar (onBar).

A strategy coded in this way does run, but it is very slow and in backtesting after some hunderd bars I run into this OutOfMemory error.
I've verified that the calling the Indicator class construction at every bar is indeed the problem. If I construct the Indicators at OnStrategyStart, so with parameters kept constant during the simulation, there is no problem.

What approach would you suggest for my situation ?

Regards,

Frank

Quote:
System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
at System.Collections.Generic.SortedList`2.set_Capacity(Int32 value)
at System.Collections.Generic.SortedList`2.EnsureCapacity(Int32 min)
at System.Collections.Generic.SortedList`2.Insert(Int32 index, TKey key, TValue value)
at System.Collections.Generic.SortedList`2.set_Item(TKey key, TValue value)
at SmartQuant.Data.MemorySeries`1.Add(DateTime datetime, Object obj)
at SmartQuant.Indicators.Indicator.Add(DateTime DateTime, Double Data)
at SmartQuant.Indicators.EMA.Calculate(Int32 index)
at SmartQuant.Indicators.EMA.OnInputItemAdded(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Indicators.Indicator.OnInputItemAdded2(Object sender, DateTimeEventArgs EventArgs)
at SmartQuant.Series.ItemAddedEventHandler.Invoke(Object sender, DateTimeEventArgs e)
at SmartQuant.Series.TimeSeries.EmitItemAdded(DateTime dateTime)
at SmartQuant.Series.BarSeries.Add(Bar bar)
at SmartQuant.Instruments.BarSeriesList.l9GsJ1Y3L(Instrument , Bar )
at SmartQuant.Instruments.DataManager.FXeMVYXvd(Object , BarEventArgs )
at SmartQuant.Providers.ProviderManager.donsPjTgF5(Object , BarEventArgs )
at SmartQuant.Providers.BarEventHandler.Invoke(Object sender, BarEventArgs args)
at SmartQuant.Simulation.SimulationDataProvider.oB8NQOfhqM(IFIXInstrument , Bar )
at SmartQuant.Simulation.SimulationDataProvider.QGYNAiGibJ(SeriesObjectEventArgs )
at SmartQuant.Simulation.SeriesObjectEventHandler.Invoke(SeriesObjectEventArgs args)
at SmartQuant.Simulation.Simulator.dEG48R9xls(IDataSeries , IDataObject )
at SmartQuant.Simulation.Simulator.Asx4Vq8104()
at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()



I figured out that in OpenQuant 2014 every newly constructed indicator is assign to the BarSeries and I can control or clear them using the .Indicators[n].Clear() method.
But how does this work under the current production version?

Regards,

Frank


Top
 Profile  
 
PostPosted: Tue Jun 17, 2014 9:49 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
Hi,

the situation is even worse in case of OQ2013. Once you construct an indicatior and attach it to an input series (i.e. pass input series to indicator constructor), indicator gets subscribed to the new item added event of the input series. This means the indicator object is always attached and calcualted, even if you think you have dropped the indicator object and memory should be cleared by the garbage collector.

Regards,
Anton


Top
 Profile  
 
PostPosted: Tue Jun 17, 2014 9:54 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
PS. Let me check with our developers if there is a workaround.


Top
 Profile  
 
PostPosted: Tue Jun 17, 2014 9:58 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
By the way, one idea can be to construct a copy of bar series every time you create an inidicator and attach indicator to this bar series. Don't forget that you need to call series.Add(bar) manually in OnBar in this case. Then this bar series and attached indicator should be dropped once you drop all references to this bar series.

Regards,
Anton


Top
 Profile  
 
PostPosted: Tue Jun 17, 2014 10:00 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6808
And indicator will be calculated only when you call series.Add(bar) for its input bar series. This way you can control indicator calculations and memory usage.


Top
 Profile  
 
PostPosted: Wed Jun 18, 2014 5:58 pm 
Offline

Joined: Fri Apr 25, 2014 10:56 am
Posts: 9
Dr. Anton Fokin wrote:
And indicator will be calculated only when you call series.Add(bar) for its input bar series. This way you can control indicator calculations and memory usage.


Thank you Anton,

I make a copy of the requiered bars of the BarSeries Bars using the Bars.GetRange method.:
Code:
      if (Bars.Count > maxwarm)
      {   
         DateTime dtstart=Bars.GetDateTime(Bars.Count-maxwarm);
         DateTime dtend=Bars.Last.DateTime;
            
         BarSeries hlc=Bars.GetRange(dtstart,dtend);
                    .
                    etc

where maxwarm is the maximum warmup period of all used indicators.

I then pass hlc as input argument to my indicators.
This works well, the performance is now much better without the memory problem.

The series.Add(bar) is not necessary as the indicator is already calculated when it is constructed.


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 7 posts ] 

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 4 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group