Jump to content
Medved Trader Forums

How to speed scan by backfilling only when required?


Oliver
 Share

Recommended Posts

Hi,

I run daily scans on historical data in MT on a portfolio of 1500 stocks. It takes a long time to run in Auto-Backfill mode. How can I setup the scanner to only backfill when necessary?

 

For instance, if I want to filter all stocks with 52Rng% above 50 I do not need to backfill historical data for stocks where 52Rng% is below 50. Is it possible to setup a code like :

 

else

{

BackFill;

...

}

What would be the command line to use to backfill only when necessary?

 

Regards,

Oliver

Link to comment
Share on other sites

Hi Jerry,

In my scan I do this :

 


So I would expect the code to backfill only symbols where SymbolData.Range52Percent is above 50. It does not seem to be the case. If I hit on 'Start' with 'Auto-Backfill' it takes 30 min  to compute the scan for all 1500 symbols.  Once it is computed for all symbols the other scans are faster. But I would not expect the first scan to be so slow as only around 100 stocks should be backfilled.

Also if I stop and rerun the scan it takes again around 30min to run although all symbols have normaly already been backfilled.

code example : if ( SymbolData.Range52Percent < 50) {return;} else {if (SymbolData.Last<=Open[1] or High<=High[1])...}

 

 

Link to comment
Share on other sites

Also, you're doing this on a historical chart - MT doesn't do partial historical backfills. Let me see if I can do something about it.

This will be probably in the beta after next production version (soon). For Historical scans it will still do backfills for all, but if you call a SetDaysNeeded(3) for example inside the scan code it will try to backfill only the last 3 historical days, making it a lot faster.

Link to comment
Share on other sites

Hi Jerry,

I never need more than 1 year of historical data in the scanner so it would be great to be able to set the max number of days to backfill.

Also, if with the planned release I add to the code SetDaysNeeded(260) and all historical data are already in the database for a stock, can we ensure the backfill is not performed? That we only do a historical backfill when data are missing? Also confirm that the candle of the day is not considered as historical data.

Finally, I guess it will be faster with the code above but still slow as I will always have to backfill all the symbols as th code is not evaluated. Can you consider adding a function to allow to do historical backfill on demand in the code when data are missing? This way I could run the scan with Auto-BackFill=No and only force backfill in the scan of 1500 stocks for only the 50 stocks where it is required? That will make it possible to have superfast scans on a big amount of stocks if we can control backfills.

Thanks in advance,

Oliver

Link to comment
Share on other sites

Hi Jerry,

I tested SetDaysNeeded(260) available in the latest version with a historical scan on a portfolio of 2200 stocks. I have several issues:

1- With Auto-Backfill=Yes historical backfill are performed even if it is not necessary

I do a first scan with AutoBackfill and it runs in 35 minutes. I run a second scan without AutoBackfill and it runs in 15 secondes. I then stop the current scan and rerun it again with Autobackfill and it runs in 35 minutes again. The third scan should run in 15 secons as the first backfill should have populated historical data. The third one should only use quote data of the day. Is it possible to fix that?

 

2- Availability of a ForceBackfill function

With a ForceBackfill function available in the code I could reduce the screening time from 35min to 1min by forcing the backfill only when necessary on a very small list of stocks using the quote data available for the day (AvgVol, 52w Rang%, Last, Open....). Is it possible to have such function available?

 

Kind regards,

Oliver

 

 

 

 

Link to comment
Share on other sites

I really gotta think how to do it. The SetDaysNeeded function is never executed - it is only detected in the pre-parser of the code and the flag is set that backfill is required.

To make it dynamic like you say could make it tautological/circular - what if the condition under which you execute the backfill depends on some indicator that does not have the correct value if the data is not backfilled?

Link to comment
Share on other sites

Regarding point 2, it would be up to us to ensure the code is consistent. But even if it is not it would not be an issue for the system just for the accuracy of the results.

The condition I will use will only depend on the last daily candle and the data avaiable for the symbol not of historical quotes so it will not be an issue for me. That would allow to have a faster scanner.

 

What about point 1 in my previous post? it is about ensuring that for historical data backfills only backfills historical data until yesterday and that today's quotes are taken from the realtime feed of market data. This way backfill is run only once per symbol everyday. Not each time we stop and rerun the scan.

 

 

 

 

 

Link to comment
Share on other sites

  • 1 month later...

Hi,

 

There seem to be an issue with historical backfills. It looks like at the start of a new day if the scan is done without Auto-Backfill then the scan reuslts are wrong. They do not take into account the latest quotes. But reruning the scan with Auto-Backfill does not fix the issue and backfill data. One have to clear all historical data and rerun the scans with Auto-BAckfill.

Can this be fixed?  It is annoying if the scan is both slow and unreliable.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

 Share

×
×
  • Create New...