SmartQuant Discussion

Hi,

I think it's quite easy now in .Net 4.0 to parallelize.
One great feature that could benefit from it would be the import of large CSV historical file.

Also, could the Historical CSV data import dialog be non-modal?
It's taking a large amount of time to import and I'd like to be able to do something else in OQ meanwhile.

EDIT: same notes for the Compress Bar feature & dialog.

Thanks!

Hi,

you can always use OQ scripts to process csv files and import data in any way you like, see this FAQ

http://smartquant.com/forums/viewtopic.php?f=64&t=6277

Regards,
Anton

Thanks Anton. I agree it is comfortable to be able to automate common tasks via scripting, that's what I'll end up doing... but I guess it wouldn't take a lot of time to just parallelize the features mentioned either. That would make OQ a better product off the shelf.

Cheers,
PBJ

I wouldn't have thought its an easy task to parallelize. Consider that the CSV data needs to be pushed into a List. How will you do that in parallel? The list requires elements are pushed into it in order.

I suppose you could process multiple symbols at once, say load in 10 CSV files and import each on its own thread. Then the bottleneck becomes the disk IO which is serial. It's worth a try but I'm guessing you wouldn't see much benefit from using multiple threads here.

Thank you for your input. I haven't had the time to try yet.
So far, whatever I've done with OQ, I haven't seen anything parallelized. I haven't tried optimization yet however.

I have uploaded about 120 000 000 records from about 320 files quite fast, maybe 30 min

These numbers are probably correct.
I'm curious: did you do it manually from OQ's Import Historical Data dialog or from a script you developed?

I made import of full content using Import Wizard.
Later on I had to import again but with some additional filtering of files' content. And this I have done using Script