SmartQuant Discussion

How to process large amounts of historical data
Page 1 of 1

Author:  Dr. Anton Fokin [ Fri Feb 01, 2008 5:55 pm ]
Post subject:  How to process large amounts of historical data

When you backtest a strategy, all incoming market data is stored in memory in Strategy.Bars, Strategy.Quotes and Strategy.Trades arrays. This may cause an out of memory exception if you process large enough amounts of historical data. You can set fixed sizes of these in-memory arrays, so that only last N bars, quotes or trades will be stored.

You can navigate to Tools->Options in the OpenQuant 2.x main menu and set desired number of stored market data entries. Use "-1" value to store all incoming market data.

arraylength.JPG [ 35.86 KiB | Viewed 8707 times ]

Author:  flotschie [ Sun Feb 17, 2008 12:21 pm ]
Post subject: 

Hi Anton,

when we try to backtest 1 Year intraday tick-based strategies, we have out of memory exceptions - but setting the Bars, Quotes and Trades to a certain size, the simulation gets very slow the longer the simulation lasts, until it gets so slow we have to stop.

We have 2 GB RAM, yes I think the issue might be solved with more RAM, but is there another way to solve this?

Another point of view and to discuss: Does it make sense to backtest 1 Year Intraday strategies with appx. 30 Trades a day? Do the market behavoiur changes faster? What are your experiences with Intraday Strategies?

Hans Moleman :shock:

Author:  sanjay_arora [ Thu Jun 11, 2009 11:06 pm ]
Post subject:  Streaming?

Has anyone thought of streaming the quotes and reducing the memory requirements?


Author:  Dr. Anton Fokin [ Thu Jun 11, 2009 11:26 pm ]
Post subject: 

What do you suggest exactly?


Author:  RSC [ Wed Nov 18, 2009 12:51 pm ]
Post subject: 


Working with the fixed sizes in-memory arrays I had a strange experience with the built-in ATR indicator:

The ATR indicator is in fact a SMA (TR, SMALength). When you use the ATR indicator directly in an fixed size bar array limited strategy you get unexpected (wrong) results, probably because OQ doesn't limit the TR calculation internally.

When you built your own ATR by using a TR indicator and then a SMA (TR, Length) every thing is calculated ok. Nice to know!


Author:  PBJ [ Mon Oct 31, 2011 7:47 pm ]
Post subject:  Re: How to process large amounts of historical data

Is it possible to change these 3 lookback parameters on the fly in a Script?

Author:  Dr. Anton Fokin [ Mon Oct 31, 2011 9:48 pm ]
Post subject:  Re: How to process large amounts of historical data

Which ones? Lengths of arrays? No. Why would you need such a functionality? Any user story?


Author:  PBJ [ Tue Nov 01, 2011 2:35 pm ]
Post subject:  Re: How to process large amounts of historical data

Yes Anton, these Array Lengths.
I'll certainly be running custom backtests/optimizations through scripts/scenarios instead of the classic "Run" button.
To use memory efficiently, I though I'd set Array Lengths before each backtest to the most appropriate value for each run.
For example Bar Array Lengths could be set to the Max Lookback value I use in my strategies/indicators.
But that was just a thought, nothing important for now.

As an aside, I highly appreciate the fact that backtests can be run from scripts/scenarios without having to use the GUI.
One of my goal is to be able to backtest and optimize a given strategy or set of strategies on any number of instruments, on several different timeframes (5 mn bars, 15mn, 1h for example) and store all trades in a database without having to click and configure manually each time. When backtests/optimizations are done, I could run SQL/LINQ queries in an app to extract valuable information (performance ratios, risk...) from these trades and generate a good report.
From what I understand from reading the documentation, recent developments in OQ permit just that.

Page 1 of 1 All times are UTC + 3 hours
Powered by phpBB® Forum Software © phpBB Group