SmartQuant Discussion

Automated Quantitative Strategy Development, SmartQuant Product Discussion and Technical Support Forums
It is currently Thu Dec 12, 2019 8:23 pm

All times are UTC + 3 hours




Post new topic Reply to topic  [ 8 posts ] 
Author Message
PostPosted: Wed Nov 16, 2011 10:01 pm 
Offline

Joined: Sun Oct 30, 2011 12:12 am
Posts: 220
Hi,

I think it's quite easy now in .Net 4.0 to parallelize.
One great feature that could benefit from it would be the import of large CSV historical file.

Also, could the Historical CSV data import dialog be non-modal?
It's taking a large amount of time to import and I'd like to be able to do something else in OQ meanwhile.

EDIT: same notes for the Compress Bar feature & dialog.

Thanks!


Top
 Profile  
 
PostPosted: Wed Nov 16, 2011 10:25 pm 
Offline

Joined: Tue Aug 05, 2003 3:43 pm
Posts: 6816
Hi,

you can always use OQ scripts to process csv files and import data in any way you like, see this FAQ

http://smartquant.com/forums/viewtopic.php?f=64&t=6277

Regards,
Anton


Top
 Profile  
 
PostPosted: Thu Nov 17, 2011 5:01 pm 
Offline

Joined: Sun Oct 30, 2011 12:12 am
Posts: 220
Thanks Anton. I agree it is comfortable to be able to automate common tasks via scripting, that's what I'll end up doing... but I guess it wouldn't take a lot of time to just parallelize the features mentioned either. That would make OQ a better product off the shelf.

Cheers,
PBJ


Top
 Profile  
 
PostPosted: Fri Dec 09, 2011 1:14 pm 
Offline

Joined: Wed Aug 24, 2011 7:11 pm
Posts: 42
I wouldn't have thought its an easy task to parallelize. Consider that the CSV data needs to be pushed into a List. How will you do that in parallel? The list requires elements are pushed into it in order.

I suppose you could process multiple symbols at once, say load in 10 CSV files and import each on its own thread. Then the bottleneck becomes the disk IO which is serial. It's worth a try but I'm guessing you wouldn't see much benefit from using multiple threads here.


Top
 Profile  
 
PostPosted: Tue Dec 13, 2011 9:04 pm 
Offline

Joined: Sun Oct 30, 2011 12:12 am
Posts: 220
Thank you for your input. I haven't had the time to try yet.
So far, whatever I've done with OQ, I haven't seen anything parallelized. I haven't tried optimization yet however.


Top
 Profile  
 
PostPosted: Thu Dec 15, 2011 11:03 pm 
Offline

Joined: Thu Mar 10, 2011 10:09 pm
Posts: 582
I have uploaded about 120 000 000 records from about 320 files quite fast, maybe 30 min :)


Top
 Profile  
 
PostPosted: Fri Dec 16, 2011 12:15 pm 
Offline

Joined: Sun Oct 30, 2011 12:12 am
Posts: 220
These numbers are probably correct.
I'm curious: did you do it manually from OQ's Import Historical Data dialog or from a script you developed?


Top
 Profile  
 
PostPosted: Fri Dec 16, 2011 12:27 pm 
Offline

Joined: Thu Mar 10, 2011 10:09 pm
Posts: 582
I made import of full content using Import Wizard.
Later on I had to import again but with some additional filtering of files' content. And this I have done using Script


Top
 Profile  
 
Display posts from previous:  Sort by  
Post new topic Reply to topic  [ 8 posts ] 

All times are UTC + 3 hours


Who is online

Users browsing this forum: No registered users and 1 guest


You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot post attachments in this forum

Search for:
Jump to:  
cron
Powered by phpBB® Forum Software © phpBB Group