SmartQuant Discussion
http://www.smartquant.com/forums/

Simultaneous Historical Data for Multiple Instruments
http://www.smartquant.com/forums/viewtopic.php?f=41&t=14972
Page 1 of 1

Author:  harsh [ Thu Oct 17, 2019 10:39 am ]
Post subject:  Simultaneous Historical Data for Multiple Instruments

Hi Team,

My requirement is to get historical bars for multiple instruments simultaneously (in parallel) using IQFeed. For example, if I subscribe for two instruments I1 and I2 then bars sequence should be I1.Bar1, I2.Bar1, I1.Bar2, I2.Bar2, ... I1.BarN, I2.BarN.

When I call this method for multiple instruments by calling this method multiple times in sequence, it returns bars (trigger Bar event) in the same sequence as I called the method. It would return all bars for I1 then all bars for I2 and so on.

Code:
framework.DataManager.GetHistoricalBars(
  comboBoxProvider.SelectedItem.ToString(),
  framework.InstrumentManager[symbol],
  dateTimePickerFrom.Value, dateTimePickerTo.Value,
  BarType.Time, Convert.ToInt64(textBoxBarSize.Text)
);


I also tried an alternate way using BarFactory as below:

Code:
Framework framework = Framework.Current;
IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
historicalDataProvider.Connect();
framework.DataSimulator.SubscribeBar = false;

framework.DataSimulator.DateTime1 = dateTimePickerFrom.Value;
framework.DataSimulator.DateTime2 = dateTimePickerTo.Value;

var symbols = new string[] { "AAPL", "GOOGL" };

foreach (var symbol in symbols)
{
  framework.EventManager.BarFactory.Add(framework.InstrumentManager[symbol], BarType.Time, Convert.ToInt64(textBoxBarSize.Text), providerId: historicalDataProvider.Id);
}


Could you please suggest what I might be missing?

Thanks,

Author:  skuvv [ Thu Oct 17, 2019 1:11 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

Parallel requests mean async responses, this is obvious.
From the other hand, you cannot do parallel request from single thread, so they will be queued.
Therefore, even if you get all bar series, you must sync series manually.

Author:  harsh [ Fri Oct 18, 2019 4:27 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

Can the framework not do this multi-threading for us and provide the data in same way as it is provided to Startegy? When I run (backtest) BollingerBands with AAPL and MSFT, logs from OnBar event are as below:

Code:
Bar AAPL: 204.51
Bar MSFT: 136.115
Bar AAPL: 204.58
Bar MSFT: 136.55
Bar AAPL: 204.81
Bar MSFT: 136.365
Bar AAPL: 204.75
Bar AAPL: 204.82
Bar MSFT: 136.44
Bar AAPL: 205.17
Bar AAPL: 205.16
Bar MSFT: 136.72
Bar AAPL: 205.25
Bar MSFT: 136.68


As you can see, the data appears in parallel, one bar from AAPL and then one from MSFT and so on. This is exactly what we require but we do not want to create Startegy. We want to do it directly using Framework's object. I was hoping if there is way to receive data in the form like above independently using framework's object. I think BarFactory should be able to achieve this but I am not sure what flags or settings I should set.

Author:  skuvv [ Fri Oct 18, 2019 5:13 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

Our framework processes all queued events in single thread. So this is not parallel.

Author:  harsh [ Fri Oct 18, 2019 8:35 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

Can you suggest any workaround? Do we have to create a separate framework's instance in a separate thread for each instrument if we want to receive data in parallel as I mentioned in my post earlier?

Author:  skuvv [ Mon Oct 21, 2019 10:23 am ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

I suppose parallel requests will be look like:

Code:
class Program
    {
        static string[] symbols = { "AAPL", "IBM", "MSFT", "CSCO", "AMD" };
        static Dictionary<string, BarSeries> bars = new Dictionary<string, BarSeries>();
        static void Main(string[] args)
        {
            Framework framework = new Framework("main");

            IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
            historicalDataProvider.Connect(5000); //init connection for creating iqfeed gateway

            Task[] tasks = new Task[symbols.Length];

            for (int i = 0; i < symbols.Length; i++)
            {
                string symbol = symbols[i];

                tasks[i] = new Task(() => GetBars(symbol));
                tasks[i].Start();
            }

            Task.WaitAll(tasks);

            Console.WriteLine("All tasks completed");

            Console.ReadLine();
        }

        static void GetBars(string symbol)
        {
            bars.Add(symbol, null);

            Framework framework = new Framework("framework_" + symbol);

            IHistoricalDataProvider historicalDataProvider = framework.ProviderManager.GetHistoricalDataProvider("IQFeed");
            historicalDataProvider.Connect(5000);
           
            Instrument instrument = framework.InstrumentManager[symbol];

            if (instrument == null)
                return;

            Console.WriteLine($"Sending request for {symbol}");

            BarSeries series = framework.DataManager.GetHistoricalBars(historicalDataProvider, instrument, DateTime.Now.AddDays(-100), DateTime.Now, BarType.Time, 60);

            if (series == null)
                Console.WriteLine("Series is null for {symbol}");
            else
                Console.WriteLine($"Got series with {series.Count} elements for {symbol}");

            bars[symbol] = series;
        }
    }


You can process all bars after
Code:
Task.WaitAll(tasks);

Author:  harsh [ Wed Oct 23, 2019 8:35 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

What if I need to read data from the framework's local storage not IQFeed? Can framework return data in the format (simultaneously not sequence) that I mentioned in previous post, in this case, using some built-in methods?

Author:  skuvv [ Fri Oct 25, 2019 9:54 am ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

harsh wrote:
What if I need to read data from the framework's local storage not IQFeed? Can framework return data in the format (simultaneously not sequence) that I mentioned in previous post, in this case, using some built-in methods?

No, as I said earlier, you have to process and synchronize all the data yourself.

Author:  harsh [ Sat Nov 02, 2019 2:02 pm ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

We need to read historical market data for about 600 instruments at a time in parallel (not in sequence).

It really doesn't seem a practical solution to create 600 instances of the Framework to achieve this. I am sceptical that doing so will hog CPU/Memory of the machine.

If possible, can you share APIs for reading directly from market data storage (binary/database) so we can read directly from the storage?

Or, can you suggest any other solution for this scenario? This requirement is critical to our systems.

Thanks,

Author:  skuvv [ Mon Nov 11, 2019 9:37 am ]
Post subject:  Re: Simultaneous Historical Data for Multiple Instruments

harsh wrote:
We need to read historical market data for about 600 instruments at a time in parallel (not in sequence).

It really doesn't seem a practical solution to create 600 instances of the Framework to achieve this. I am sceptical that doing so will hog CPU/Memory of the machine.

If possible, can you share APIs for reading directly from market data storage (binary/database) so we can read directly from the storage?

Or, can you suggest any other solution for this scenario? This requirement is critical to our systems.

Thanks,

I made several tests, but unfortunately I can not advise optimal way to load data in parallel for this moment.

Page 1 of 1 All times are UTC + 3 hours
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/