Data loss, back fill limits, ideas?

[JamesKB2]

Hey guys,

I run Medved pretty much continuously from Sunday 6:00 pm EST to Friday 5:00 pm EST.

I do it to gather data, accumulating, for pattern use.

The most frustrating part of my use is unexpected shutdowns, whether power out or freezing, Medved data checking and recovery causes data loss collected over weeks of use.

I primarily track @ES, with SPX option strikes sometimes two months out.  "Crashing" wipes it all out, particularly the options as there's no back fill, but also ES intraday data, with limits on back fill.

Ideally, I'd like 5 minute ES for a year back intraday, BUT, I'd be happy with collected data on my machine NOT being wiped because of a crash.

Why?  I use Wave Theory in pattern and have discovered some repetitions that I like to cross check in analysis.  Killing my data base with error checking after crash prevents that.

I use AMTD data feed.

Work arounds?  Ideas?

Thanks,

Jim

[Jerry Medved]

I have a similar situation with keeping the portfolio total since that cannot be backfilled. I suggest closing MT, copying data, then restarting at least daily or every few days, but definitely less than the max backfill period.  Even if you have automatic backfill on the PC, if MT is not shut down when it is running, the data file will not be properly backed up. 

[wilburpost2]

You need to get a good extended runtime UPS for the power issues. Most have software that will close all programs before pc shutdown. You need to close MT like once a day, backup or copy MT data files, or just do a PC backup and restart MT. You would only lose one day at most. If you have a crash you load your back up. If you can't backfill the data you need you have to archive it on your own.

[JamesKB2]

UPS certainly solves the issue of power loss shutdowns.  I get that.  I suppose using some sort of automatic backup to my server (home network) would allow me to recapture long data compilations.

I didn't know if there was some setting allowing a 1 minute interval "lock" on data sets to avoid loss.

I'll fiddle around with some backup options to my main server backup drive and have it run every 12 hours, that should allow me to recover the file(s) I'd like to compile.

Will the old data (say, several months) back fill to current (new data) without truncating?  Meaning, I could just add on to the file for many months using the backup after a crash?

[wilburpost2]

In a perfect world MT or your PC would never crash and you would never lose your data. You backup your data file. It has your total collected data for many days or months. If there is a loss, you copy over the backup replacing the corrupted file and start MT. All your old data is there. When MT does a BF, auto or manually, it will check for missing data gaps and try to get that. The data you get will be missing stuff like options. You can only get what the backfill provides. MT will start to collect new data, but there will always be that gap from last backup to crash. Sometimes only a few symbols will get clobbered, the rest will be fine. In defense of MT, I might get a red error triangle on a new beta, but I can't remember MT just plain freezing or crashing. It's usually because MT happened to be running when the PC had a problem. I would first look at the PC and what's running.