How to speed scan by backfilling only when required?

[Oliver]

Hi,

I run daily scans on historical data in MT on a portfolio of 1500 stocks. It takes a long time to run in Auto-Backfill mode. How can I setup the scanner to only backfill when necessary?

 

For instance, if I want to filter all stocks with 52Rng% above 50 I do not need to backfill historical data for stocks where 52Rng% is below 50. Is it possible to setup a code like :

else

{

BackFill;

...

}

What would be the command line to use to backfill only when necessary?

 

Regards,

Oliver

[Jerry Medved]

MT does check the code first and does not do the backfill if there is nothing in it to require it. Also, if the data is already backfilled for a given symbol, it should skip it. Does your code reference any indicators?

[Oliver]

Hi Jerry,

In my scan I do this :

 

So I would expect the code to backfill only symbols where SymbolData.Range52Percent is above 50. It does not seem to be the case. If I hit on 'Start' with 'Auto-Backfill' it takes 30 min  to compute the scan for all 1500 symbols.  Once it is computed for all symbols the other scans are faster. But I would not expect the first scan to be so slow as only around 100 stocks should be backfilled.

Also if I stop and rerun the scan it takes again around 30min to run although all symbols have normaly already been backfilled.

code example : if ( SymbolData.Range52Percent < 50) {return;} else {if (SymbolData.Last<=Open[1] or High<=High[1])...}

 

 

[Jerry Medved]

Oliver, you said "In my scan I do this" but then there was a gap and nothing - so I think whatever you planned to post, did not come through.  Please post or email the entire scan, not just portions that you think are relevant.

[Oliver]

[Oliver]

I can reproduce the issue with the simple scan below. There are 1590 stocks in the scan but only 10 have  SymbolData.Range52Percent above 99 so only those 10 should trigger backfill to get a fast scan.

[Jerry Medved]

that is not how it works. the check for whether backfill is needed or not looks at the general code and occurs before any evaluation is done. So since you have code there looking at candle data, it means the scan requires backfill. 

[Mike Medved]

Also, you're doing this on a historical chart - MT doesn't do partial historical backfills. Let me see if I can do something about it.

This will be probably in the beta after next production version (soon). For Historical scans it will still do backfills for all, but if you call a SetDaysNeeded(3) for example inside the scan code it will try to backfill only the last 3 historical days, making it a lot faster.

[Oliver]

Hi Jerry,

I never need more than 1 year of historical data in the scanner so it would be great to be able to set the max number of days to backfill.

Also, if with the planned release I add to the code SetDaysNeeded(260) and all historical data are already in the database for a stock, can we ensure the backfill is not performed? That we only do a historical backfill when data are missing? Also confirm that the candle of the day is not considered as historical data.

Finally, I guess it will be faster with the code above but still slow as I will always have to backfill all the symbols as th code is not evaluated. Can you consider adding a function to allow to do historical backfill on demand in the code when data are missing? This way I could run the scan with Auto-BackFill=No and only force backfill in the scan of 1500 stocks for only the 50 stocks where it is required? That will make it possible to have superfast scans on a big amount of stocks if we can control backfills.

Thanks in advance,

Oliver

[Oliver]

Hi Jerry,

I tested SetDaysNeeded(260) available in the latest version with a historical scan on a portfolio of 2200 stocks. I have several issues:

1- With Auto-Backfill=Yes historical backfill are performed even if it is not necessary

I do a first scan with AutoBackfill and it runs in 35 minutes. I run a second scan without AutoBackfill and it runs in 15 secondes. I then stop the current scan and rerun it again with Autobackfill and it runs in 35 minutes again. The third scan should run in 15 secons as the first backfill should have populated historical data. The third one should only use quote data of the day. Is it possible to fix that?

 

2- Availability of a ForceBackfill function

With a ForceBackfill function available in the code I could reduce the screening time from 35min to 1min by forcing the backfill only when necessary on a very small list of stocks using the quote data available for the day (AvgVol, 52w Rang%, Last, Open....). Is it possible to have such function available?

 

Kind regards,

Oliver

 

 

 

 

[Mike Medved]

I really gotta think how to do it. The SetDaysNeeded function is never executed - it is only detected in the pre-parser of the code and the flag is set that backfill is required.

To make it dynamic like you say could make it tautological/circular - what if the condition under which you execute the backfill depends on some indicator that does not have the correct value if the data is not backfilled?

[Oliver]

Regarding point 2, it would be up to us to ensure the code is consistent. But even if it is not it would not be an issue for the system just for the accuracy of the results.

The condition I will use will only depend on the last daily candle and the data avaiable for the symbol not of historical quotes so it will not be an issue for me. That would allow to have a faster scanner.

 

What about point 1 in my previous post? it is about ensuring that for historical data backfills only backfills historical data until yesterday and that today's quotes are taken from the realtime feed of market data. This way backfill is run only once per symbol everyday. Not each time we stop and rerun the scan.

 

 

 

 

 

[Oliver]

Hi,

 

There seem to be an issue with historical backfills. It looks like at the start of a new day if the scan is done without Auto-Backfill then the scan reuslts are wrong. They do not take into account the latest quotes. But reruning the scan with Auto-Backfill does not fix the issue and backfill data. One have to clear all historical data and rerun the scans with Auto-BAckfill.

Can this be fixed?  It is annoying if the scan is both slow and unreliable.