SmartQuant Discussion
http://www.smartquant.com/forums/

Parallelizing Historical Data Import From CSV
http://www.smartquant.com/forums/viewtopic.php?f=61&t=9788
Page 1 of 1

Author:  PBJ [ Wed Nov 16, 2011 10:01 pm ]
Post subject:  Parallelizing Historical Data Import From CSV

Hi,

I think it's quite easy now in .Net 4.0 to parallelize.
One great feature that could benefit from it would be the import of large CSV historical file.

Also, could the Historical CSV data import dialog be non-modal?
It's taking a large amount of time to import and I'd like to be able to do something else in OQ meanwhile.

EDIT: same notes for the Compress Bar feature & dialog.

Thanks!

Author:  Dr. Anton Fokin [ Wed Nov 16, 2011 10:25 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

Hi,

you can always use OQ scripts to process csv files and import data in any way you like, see this FAQ

http://smartquant.com/forums/viewtopic.php?f=64&t=6277

Regards,
Anton

Author:  PBJ [ Thu Nov 17, 2011 5:01 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

Thanks Anton. I agree it is comfortable to be able to automate common tasks via scripting, that's what I'll end up doing... but I guess it wouldn't take a lot of time to just parallelize the features mentioned either. That would make OQ a better product off the shelf.

Cheers,
PBJ

Author:  aburnettthompson [ Fri Dec 09, 2011 1:14 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

I wouldn't have thought its an easy task to parallelize. Consider that the CSV data needs to be pushed into a List. How will you do that in parallel? The list requires elements are pushed into it in order.

I suppose you could process multiple symbols at once, say load in 10 CSV files and import each on its own thread. Then the bottleneck becomes the disk IO which is serial. It's worth a try but I'm guessing you wouldn't see much benefit from using multiple threads here.

Author:  PBJ [ Tue Dec 13, 2011 9:04 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

Thank you for your input. I haven't had the time to try yet.
So far, whatever I've done with OQ, I haven't seen anything parallelized. I haven't tried optimization yet however.

Author:  newDave [ Thu Dec 15, 2011 11:03 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

I have uploaded about 120 000 000 records from about 320 files quite fast, maybe 30 min :)

Author:  PBJ [ Fri Dec 16, 2011 12:15 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

These numbers are probably correct.
I'm curious: did you do it manually from OQ's Import Historical Data dialog or from a script you developed?

Author:  newDave [ Fri Dec 16, 2011 12:27 pm ]
Post subject:  Re: Parallelizing Historical Data Import From CSV

I made import of full content using Import Wizard.
Later on I had to import again but with some additional filtering of files' content. And this I have done using Script

Page 1 of 1 All times are UTC + 3 hours
Powered by phpBB® Forum Software © phpBB Group
https://www.phpbb.com/