SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Thu Oct 23, 2014 9:36 am

All times are UTC + 3 hours




Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next
Author Message
 Post subject: Backfilling Data
PostPosted: Thu Apr 09, 2009 12:39 pm 
Offline

Joined: Thu Apr 09, 2009 12:36 pm
Posts: 8
Hello, I have a question. I currently use Amibroker and what I like is the automatic backfill. Lets say I have all the 1minute quotes stored until end of trading today and will not run the application tomorrow while markets are open. Then when I start the application the day after tomorrow Amibroker will check which data is missing and will automatically backfill (using Esignal or Interactive Brokers) the missing data. Does Smartquant have such built-in functionality? Will it also fill missing bars up until now should there be any missing datapoints?

Thanks for the information


Top
 Profile  
 
 Post subject:
PostPosted: Fri Apr 10, 2009 1:54 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
Hi,

please take a look at this FAQ

http://www.smartquant.com/forums/viewtopic.php?t=6227

Regards,
Anton


Top
 Profile  
 
 Post subject: not applicable
PostPosted: Sat Apr 11, 2009 5:56 pm 
Offline

Joined: Thu Apr 09, 2009 12:36 pm
Posts: 8
I am afraid your link does not really address the problem. Also, I understand I can code a backfill function to esignal or IB myself but I would not be here if I wanted to code everything from scratch, I would then do that in .NET. I simply asked whether a backfill solution to IB and Esignal currently exists for Smartquant or not. Could you please specifically address that question? Thanks

What I look for is an AUTOMATIC backfill for all instruments in my instrument list or strategy. Also, i dont want to have to check which data is missing and then determine the start and end date for the data that needs to be downloaded in order to complete the backfill. I dont want to have to care when the last bar was stored in my database.

I simply want a backfill function that downloads all the missing bars till now from the last stored bar.

In that sense unless I missed out on some functionality I feel Smartquant is DEARLY missing in terms of data management in comparison to the Amibroker solutution that costs about USD 500 less.

Please feel free to correct me if this assessment is unfair but I dont seem to be able to locate any backfill function/solution that comes remotely even close to what Amibroker offers.


Top
 Profile  
 
 Post subject: Feedback?
PostPosted: Sun Apr 26, 2009 3:05 pm 
Offline

Joined: Thu Apr 09, 2009 12:36 pm
Posts: 8
Could some of you guys please provide some feedback as to which data providers flawlessly work with Smartquant? With flawless I mean

a) no additional programming required (I chose the data provider in Smart Quant, and SmartQuant hooks me up to the real-time feed and it also downloads historical data into the local database and stores the downloaded historical data
b) it backfills data. Lets say, I have historical data up until last week and now reconnect to my data provider will SQ backfill the missing data?

Any of the data providers like esignal, iqfeed, or the like that SQ FULLY supports in terms of data integration?

Thanks a lot.


Top
 Profile  
 
 Post subject:
PostPosted: Sun Apr 26, 2009 8:48 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
What's the problem with IB ?

Regards,
Anton


Top
 Profile  
 
 Post subject: Backfilling Data
PostPosted: Thu May 07, 2009 5:06 am 
Offline

Joined: Wed May 06, 2009 2:36 am
Posts: 12
Location: Canada
Anton,
I would like to ask the same question about a backfill as bbmat asks.
1. Let us consider the obvious situation:

I am collecting tick (trade) data in the real time from IB using data capture window (E-mini futures). Then my internet connection is broken and then it is restored in 3 hours. Now I need to backfill missing data (trade or tick). If I try to use "Import-Historical data-IB", then it is allowed to download time bars only and it works for 1 min bars, and from yesterday (for today it does not work).

What about to download trade data for today?
How do I and program know for what period of time data are missed?

I am aware that IB allows to download trade data for last 24 hours and the data range can be selected.

2. I am used to collect trade data for different future contract series separately. For example, I have ES H9, ES M9, TF H9, TF M9 and so on (for last 3 months of their lives). But OQ does not allow to create two different future contracts for the same underlying future. For example, it allows to create ES contract with an expiration date (Jun 19, 2009). But somewhen in June I will need to collect trade data for both June and September contracts simulteniously. How can I do that?

3. I discovered that for import of data from csv file the time bars are allowed only. But I have a big collection of tick bars for E-mini futures. How can I import my ticks bars which are in csv files?

4. With regard to API documentation, it sucks. How are customers supposed to know what classes do? You have done a great work with a development, but could not spend 10% of time for documentation!

Now my month of program evaluation is almost done, and I am not sure that OQ would satisfy me. The program looks very attractive because it allows to use .net. But I do not feel that I can go with it right now unless I am sure that I can collect trade data from IB(in terms of backfilling of missed data).



Thanks.


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 08, 2009 11:02 am 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
It's not easy to backfill missing historical data with OQ GUI but it's pretty easy to develop a maintenance script that will do this. You basically need to unerstand / use three methods in the DataManager class

1. Read historical data from local db
2. Download data from remote data provider
3. Write data to local db

We can try to improve the GUI though. If you can give us a hint / good example of such window functionality, we can add it to OQ GUI.

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Sat May 09, 2009 6:20 am 
Offline

Joined: Wed May 06, 2009 2:36 am
Posts: 12
Location: Canada
Anton, thank you. I am thinking in the same direction.

Could you please clarify the picture with regard to futures contracts?
I will remind one more time.
I can add a June ES future with an expiration date June 19 2009. But how can I add (and collect data) September ES future? I get an error message that the instrument ES exists already.


Top
 Profile  
 
 Post subject:
PostPosted: Sat May 09, 2009 1:48 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
Hi,

you should use AltSymbol technique for this. Please take a look at this FAQ thread

http://www.smartquant.com/forums/viewto ... 2036#22036

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 29, 2009 2:31 am 
Offline

Joined: Wed May 06, 2009 2:36 am
Posts: 12
Location: Canada
Dr. Anton Fokin wrote:
It's not easy to backfill missing historical data with OQ GUI but it's pretty easy to develop a maintenance script that will do this. You basically need to unerstand / use three methods in the DataManager class

1. Read historical data from local db
2. Download data from remote data provider
3. Write data to local db

We can try to improve the GUI though. If you can give us a hint / good example of such window functionality, we can add it to OQ GUI.

Regards,
Anton


Anton,

I tried to develop some script capable to back fill historical database with ticks from IB for futures.

But I got an error message:

System.ArgumentException: Unsupported data type - Trade

This is a fragment of the script:

//-----------------------------------------------------------------
private void RequestTickRefresh(Instrument instrument)
{
int i;
TradeSeries ts;
String myProvider = "IB";
String instrumentSymbol = instrument.Symbol;

ts = DataManager.GetHistoricalTrades(myProvider, instrument, dateTimePicker1.Value, dateTimePicker2.Value);
for (i=0; i<ts.Count; i++)
{
DataManager.Add(instrument, ts[i]);
}
Console.WriteLine("For Symbol "+instrumentSymbol+" Number of trades has been added: "{0}"", ts.Count);
}// RequestTickRefresh
//-----------------------------------------------------------------

Please, explain me what's wrong. The code looks pretty easy. It seems that DataManager.Add does not work with an instance of Class Trade, does it?


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 29, 2009 7:11 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
I think IB provides historical bars only, not trades. Though I might be wrong...

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 29, 2009 7:21 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
I am not sure why this method doesn't work for you, we are not able to test it until Monday, but you can try this DataManager method instead

public static void Add(Instrument instrument, DateTime datetime, double price, int size)

Regards,
Anton


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 29, 2009 8:43 pm 
Offline

Joined: Wed May 06, 2009 2:36 am
Posts: 12
Location: Canada
Dr. Anton Fokin wrote:
I am not sure why this method doesn't work for you, we are not able to test it until Monday, but you can try this DataManager method instead

public static void Add(Instrument instrument, DateTime datetime, double price, int size)

Regards,
Anton


Anton,

I have already tried to do that, but received the same error message:

System.ArgumentException: Unsupported data type - Trade

I guess, that IB does not support historical tick refresh (back fill). The smallest quant they support is 1 second bar. Because of that I think OpenQuant does not support historical tick refresh with IB.

Please, clarify for me if it is possible to override it just by historically requesting 1 second bars from IB and adding them to OpenQuant historical data base as trades (making trades (ticks) from 1 second bars artificially and programmatically)?


Top
 Profile  
 
 Post subject:
PostPosted: Fri May 29, 2009 10:33 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6468
Artificially and programmatically? I guess the answer is yes, but you should be sure you understand what you are doing. BTW, why do you need trades and why don't you want to use 1sec bars ?

Regards,
Anton


Top
 Profile  
 
 Post subject: A sad bug in OpenQuant
PostPosted: Tue Jun 16, 2009 5:53 am 
Offline

Joined: Wed May 06, 2009 2:36 am
Posts: 12
Location: Canada
Today I noticed something bad in OpenQuant.

The clock in my computer was late comparing with IB Trade Work Station at 3 minutes approximately (and comparing with accurate time). When I looked at the collected trades (by Data Captute IB) I discovered that the last trade was at 4:12:35 pm ET (within day trading hours) for future NQU9. The time stamp for the last trade is around 4:14:59 usually. And after break the first trade was at 4:27:34 pm ET (!!!).

It can be explained that OpenQuant writes trades to the historical data base with the time stamp which is the current system (computer) time. But obviously the time stamp must be that which comes from data provider. The same phenomena exists for the other futures which trade (tick) data I collect: ESU9,YMU9,TFU9. See the attached picture that is a screenshot of OpenQuant.

Anton, please, reproduce this and tell what is wrong.


Attachments:
NQU9_Trades.JPG
NQU9_Trades.JPG [ 232.22 KiB | Viewed 1527 times ]
Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 18 posts ]  Go to page 1, 2  Next

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 0 guests


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group