SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Sun Sep 20, 2020 9:22 pm

All times are UTC + 3 hours

Post new topic Reply to topic  [ 8 posts ] 
Author Message
PostPosted: Mon Dec 22, 2008 8:27 pm 

Joined: Fri May 06, 2005 1:40 am
Posts: 521
Release announcement: It is now possible to conveniently scan over all commodity, option and/or stock markets supported by CSI Data from your OpenQuant/QuantDeveloper strategy scripts in order to build market breadth indicators, portfolios, watch lists, instrument risk/correlation metrics etc. Example scripts have been attached below along with an overview and screenshots.

A quick feature overview of CSI Data from
# There are over 80 commodity futures exchanges in the world and CSI's covers them all.
# Many thousands of futures contracts are reported each and every trading day.
# All stocks traded in the U.S., Canada and the U.K. are reported.
# Options on indices, and all futures options are included.
# Single Stock Futures reflect current availability.
# The full range of 15,000+ US mutual funds are provided, complete with adjustments for capital gains and dividends.
# All futures series can be transformed into a multitude of computed contracts.
# Briese's Commitments of Traders (COT) data is included.
# Ninety-nine percent of coverage reaches back to the first day of trading for each respective market.

To use UA with OpenQuant, first make sure that you have at least version of the CSI Data provider in OpenQuant provider list, and at least version 2.10.7 of the Unfair Advantage data downloader installed (see screenshot below).
- You must Add the QuantScanner.dll to your build references before you can compile the sample scripts.
- QuantDeveloper scripts are also available on request, and mirror those in the OpenQuant pack but with a few extras including Option Chain and contract scanning using a more efficient/faster bulk data access methods that are available to QD.

A couple of caveats to keep in mind before using CSI Data:
1) The CSI Database is very large and comprehensive - depending on your CSI data subscription, running the example scripts can easily scan and access data for many hundreds of thousands of contracts, stocks, economic data etc. As of December 22nd OpenQuant (but not QD) requires you to create an instrument entry for every data series you wish to scan. Best practice to keep resource use to a minimum is to use filters wherever possible to make sure your only scanning over instruments that your strategies are actually going to be interested in using. Check comments and code in the demo scripts that give examples of simple and effective filters you can put in place to help manage the amount of instruments being created.

2) Due to CSI Data's historical "trade by hand" focus, the CSI database unfortunately contains inconsistent and redundant data design problems that will pose an unnecessary complications for automatic platforms like OpenQuant. For example the attached scripts automatically create contracts including tick sizes, factor, maturity dates strike price data fields. Due to the CSI database inconsistencies your automated strategies cannot rely on these numbers to be correct or reflect trading reality, so double checking everything against another data source "by hand" is necessary.
Some other CSI database problems include:
- Maturity dates do not include day resolution,
- Session times are incorrect or missing,
- Many database entries for the same underlying, each with different precisions (apparently introduced to support ancient third party programs that cannot handle the correct precision values!)
- Data that needs to be listed per contract for historical accuracy is instead listed only once per underlying.

Despite these problems, CSI Data does provide a useful service for maintaining a high level market overview and can make a valuable addition to your automated strategies. Some uses include building many market breadth indicators for live next-session strategy use, risk/correlation adjusted portfolio selection, daily interest rate adjusted Sharp/risk indicators etc.

Thanks goes out to CSI's lead UA programmer Josh Reed for his hard work in order to make these custom scanning features a possibility for SmartQuant users. If you do find CSI data to be of use with your automated strategies, please be sure let them know: as it will help the data service stay relevant and improve support for automated trading platforms like OpenQuant.
For those who do not already have CSI Data account and would like to try it out, there is a data limited demo version available from here

Screenshots and descriptions of example scripts follow:

DataCheck.cs runs a simple data check and warns of potential data problems. Automated trading platforms like OpenQuant are more susceptible to bad data so It is always a good idea to try and avoid data garbage in - garbage out situations with your strategies.

ScanTypes.cs simply query's for any available scanner providers and the scan types each supports. As of December 2008 there is currently only one Scanner provider: CSI Data which supports four broad categories of scan:
Building script ScanTypes.cs...
Build succeeded.
Available Scanner Providers:
CSI Data
Scan Request Types supported by CSI Data: [ScanID, UserDescription]
0 : Futures, Cash and Economic
1 : Stocks
2 : Futures Options, Put-Call Ratios
3 : Stock Options, Put-Call Ratios

In the above output, the ScanID number (0-3 shown) is all you need to specify when starting any data scan.

ScanTest.cs and ScanTest2.cs both perform the same market scan: demonstrating how to perform a full scan over UA stock markets db, filtering and ranking into the top 10 instruments by their 30 day Relative Strength indicator values - the basis of many simple momentum based auto trading strategies. The difference between the two scripts is that ScanTest.cs demonstrates using the single threaded scan events model, while the more advanced ScanTest2.cs demonstrates overriding the multi threaded helper base class "QScan" which can greatly improve scanning performance on multi-core CPU machines.

Example Output from ScanTest2.cs:
Building script ScanTest2.cs...
Build succeeded.
15/12/2008 10:49:38 "QScan" Info : Successfully requested Stocks from CSI Data, scanType=1 : Stocks
15/12/2008 10:49:40 "QScan" Info : Stocks - 1%
15/12/2008 10:49:40 "QScan" Info : Stocks - 2%
15/12/2008 11:07:08 "QScan" Info : Stocks - 99%
15/12/2008 11:07:21 "QScan" Info : Stocks - 100%
Top 10 Instruments in this scan: [RSI(30), Symbol]
100.00, BMY+ Bristol Myers Squibb Co
100.00, CZN+ Citizens Utilities Trust pf
100.00, PBI+ Pitney Bowes Inc
100.00, WYE+ Wyeth
100.00, HHM HealthShares Metabolic-Endocrine Disorde
100.00, HHN HealthShares Neuroscience Exchange Trade
100.00, HHZ HealthShares Ophthalmology Exchange Trad
100.00, HRD HealthShares Cardiology Exchange Traded
100.00, HHG HealthShares Infectious Disease Exchange
100.00, HRW HealthShares Dermatology and Wound Care
Finished Scan
Finished in 00:17:42.9775719 [HH:MM:SS.MS]

File comment: ScanViewer.cs script launches a simple windows form to help visualize instruments and data fields available to your custom scan scripts.
OQQuantScannerViewForm.JPG [ 132.68 KiB | Viewed 8819 times ]
File comment: ScanViewer.cs script visualize options instruments and data fields available to your custom scan scripts.
OQQuantScannerViewForm2.JPG [ 142.28 KiB | Viewed 8819 times ]
File comment: Script examples can automatically create instruments ready to trade (after a manual check)
OQCreateInstruments.JPG [ 93.58 KiB | Viewed 8819 times ]

Last edited by krn_2k on Mon Dec 22, 2008 8:39 pm, edited 1 time in total.
 Post subject:
PostPosted: Mon Dec 22, 2008 8:30 pm 

Joined: Fri May 06, 2005 1:40 am
Posts: 521
Extra Attachments below.

File comment: OpenQuant Script examples demonstrating Scanner features. [13.44 KiB]
Downloaded 495 times
File comment: Minimum versions necessary to use Scanner features.
ProviderVersion.JPG [ 91.33 KiB | Viewed 8818 times ]
File comment: You must Add \OpenQuant 2\Framework\bin\QuantScanner.dll to build references before you can compile the sample scripts
OQOptions.JPG [ 37.69 KiB | Viewed 8818 times ]
 Post subject:
PostPosted: Thu May 21, 2009 4:40 pm 

Joined: Fri May 06, 2005 1:40 am
Posts: 521
Just a quick note: of CSI provider/scanner is now ready to be used with Unfair Advantage version however there has been a delay getting it shipped out to OQ users (apologies to those that have emailed me on this).

Most noticeable difference in latest release: 10x faster market scans (stocks, futures, stock/fut options) due to UA API upgrades from CSI.

Latest changelog below:

    ///     <listheader>
    ///        <term>CHANGELOG</term>
    ///     </listheader>
    ///     <item>
    ///        <term></term>
    ///        <description>
    ///        - Updated scanner to use new CSI GetFastMarketProfile() method - 10x faster market scans
    ///        - Updated bulk download functions HistoricalOptionChains and HistoricalContracts to
    ///          use memory store rather than directly writing to DataManager.
    ///        - Added SPOT security type
    ///        - FltResultArray is always a double now as of
    ///        - Added two new scan types to not build detailed contract lists
    ///             - significantly faster scans if contracts not needed for example when using bulk download functions
    ///        </description>

WARNING: Do not use provider versions earlier than with latest UA releases - recent UA API changes will leave your data in a corrupted state!

 Post subject:
PostPosted: Sat Jun 13, 2009 8:04 am 

Joined: Wed Jun 10, 2009 12:06 pm
Posts: 1
What are different subscription options available to access CSI data how does it help users? :?

JN0-400 - 642-654

 Post subject:
PostPosted: Fri Sep 04, 2009 10:42 am 

Joined: Fri Sep 04, 2009 10:38 am
Posts: 1
Hi krn_2k! Thanks for posting this informative ad,by the way how come we cant use se provider versions earlier than with latest UA releases - recent UA API changes? are you sure it will leave our data in a corrupted state? Thanks!

Placement financier

 Post subject:
PostPosted: Fri Sep 04, 2009 11:23 am 

Joined: Fri May 06, 2005 1:40 am
Posts: 521
Some very useful and major CSI API upgrades (some time ago now) meant that older provider versions for QuantDeveloper/OQ became obsolete.
On data corruption: Yes is defiantly happens, its subtle/random and hard to spot. Do not use older providers as that code is now very outdated - and it is just not worth the risk.

AFAIK, once some new systems have been setup over at SmartQuant/QH, support can resume with the CSI provider for QD/OQ.

 Post subject:
PostPosted: Sun Oct 25, 2009 9:01 am 

Joined: Fri Oct 23, 2009 12:44 pm
Posts: 3
Hi krn_2k,

Can you please point me to where I can get the newer CSI provider (

 Post subject:
PostPosted: Sun Oct 25, 2009 1:53 pm 

Joined: Fri May 06, 2005 1:40 am
Posts: 521
rvarada wrote:
Can you please point me to where I can get the newer CSI provider (

Hi rvarada. The OQ team are working on a new release (not sure what their release schedule plan is yet), so hopefully when that comes out can be shipped along with it.

Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC + 3 hours

Who is online

Users browsing this forum: No registered users and 1 guest

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
Powered by phpBB® Forum Software © phpBB Group