My Initial Findings

[timtrader]

Jerry,

 

First of all, this new platform looks really nice.  I like all of the added flexibility and the intuitive interface.....it's coming along really nicely!

 

I downloaded, installed, and banged on MedVed Trader quite a bit this weekend, so here are my initial findings.  I'll email you my exported setting/log file shortly in case you need it:

 

1.)  Data source - Initially set to "Smart" and I have accounts for both TD Ameritrade and IQFeed set up.  I went back to change the number of days for IQFEED to 10, and changed both the Backfill source, and Historical Backfill source to IQFEED.  I then cleared all of the Intraday and Historical data.  At this point with only IQFeed selected, no chart data (Historical or intraday) will display, rather there is a message in the middle of each chart stating "There is no data to chart". I checked and re-checked my IQFeed ID and password numerous times, but this isn't the problem.  (I also went back to QuoteTracker to make sure IQFeed is up and running, and it was.)  The thing I noticed is that you can change the IQFeed ID/Password to something totally bogus, force re-login for it in the dashboard, and it will show a state of "streaming" and as status of "running".  When I change the source back to TD Ameritrade, it works fine and the chart data displays.

 

2.)  I created a 15-second chart (0.25 minutes) in OHLC format, but it is does not show the OHLC bar format, rather only a single dash.  I tried numerous stock symbols and always get the same results.  I also verified that the same works fine in QuoteTracker, so it's not a data issue.

 

3.) News is coming in fine from Yahoo and Google, but not TD Ameritrade.  Also will you eventually be adding "TheFlyonthewall" as a news data source?.....it still works fine in QuoteTracker.

 

4.)  The last bar of a historical chart in OHLC format is only showing the left arm (the open price) of the OHLC bar.  At least that is what its showing this weekend for Friday's OHLC bar on any chart.

 

5.)  When you have a chart minimized, save your current layout, close down MedVed Trader and then restart it, that chart opens back up to its "original" location....the layout doesn't open with that chart minimized.

 

6.)  I had the news window hang several times and had to kill it with Task Manager since clicking the "X" in the upper right corner didn't work.  It was acting like it was waiting for a response from something, but I couldn't just close the window when I didn't want to wait.

 

7.)  I noticed that you have 8 available color links available for the Symbol Link Groups.  With better broadband, faster PCs and more folks using multiple monitors, it might be nice to have considerably more than 8 available.

 

8.)  For Level II, I'd like to be able to move the columns around but can't.  I don't know if its not currently available, or if its because its the weekend and it is not currently displaying any Level II data.  (I prefer to show the bid/ask price before the volume.)

 

I hope this helps a bit and look forward to enhancements when they're ready.

 

Thanks,

Tim

 

 

 

[Jerry Medved]

Tim,

 

1) after clearing it?  As for IQFeed passwords, will try, but if IQConnect is already running, it may not matter what is in QT.

 

2) you are using regular OHLC Backfill. That won't work if you want 15 sec bars. You need TICK Backfill, which is an option in the backfill dropdown button on the chart if you have IQFeed selected.

 

3) TDA broke their datafeed and as far as I know, are not planning to fix that, so starting with the release I just put up, news is no longer available for TDA. As for TFOTW, we plan to implement that eventually, but right now have our hands full.

 

4) you probably have real-time updates selected for Hist charts. you can turn that of on Settings/Charts/General. Otherwise, it builds the last bar from the intraday data.

 

5) Minimized setting is not remembered intentionally

 

6) This is a similar issue to QT news window. The Embedded browser is doing the hanging when displaying some news pages from Yahoo/google News. I changed it so the page is not automatically loaded starting with this version.  Also, if you turn off the preview and have it load in external browser, that will resolve the issue

 

7) That is 8 groups, not 8 windows. Adding more colors is simple, but the problem is that the distinction between the colors becomes faint and we wanted to keep the colors very distinct.

 

8) you can pick columns, but not move them around on Level II (other than the MIRROR option). No data is because of weekend.

 

Jerry

[timtrader]

Thanks for the info.  

 

Regarding #1, I still can't get IQFeed to work no matter what I try.  Yes, I cleared all data, even rebooted my computer and only started Medved Trader after the reboot.  I'll try a fresh install on Monday afternoon unless you'd like me to try something else first.

 

Regarding #3, I had sent an email to TDA in mid-October when the news from them stopped in Quote Tracker.  They said they did make a change to the news feed, but were no longer making any updates to QuoteTracker....I don't know what they changed though.

 

Thanks,

Tim

[Jerry Medved]

IQFeed - what IQfeed version are you using?

 

You have the right password in MT now?

 

Set MT logging to 200 (on dashboard, below news buttons), then try connecting to IQfeed. If IQConnect already running, right click on it and exit. MT will force re-launch.

 

Wait for it to get quotes, news. etc... whatever data you are trying to get.

 

After it fails, go to FILE/Export Settings. Leave the right side default. On the Left side, in addition to the already checked boxes, add the LOG FILE. 

 

Email the exported file to support (It will default export to My Documents ... Medved Trader sub-folder

[Jerry Medved]

As for Ameritrade, the issue has nothing to do with QuoteTracker. They may have "changed" the datafeed (nice euphemism for "broke"), but the datafeed does not work at all at the API level, so whatever they changed, they did not update the API and I was told there is no plan to do so.  

[timtrader]

Jerry,

 

As soon as I killed IQConnect, it loaded the currently displayed charts fine, but selecting another from my worklist didn't work (the windows are linked).  Re-killing IQConnect didn't work either.

 

The export file has been emailed to you.

 

Thanks,

Tim

[timtrader]

I forgot to mention, I'm running IQFeed version 5.1.0.3.

 

Thanks,

Tim

[timtrader]

Jerry,

 

I kept fiddling with this IQFeed problem and found the following. Restarting Medved Trader, or the PC makes no difference, chart data (unless previously loaded) won't load when a new symbol is added.  When a new symbol is added all I get is the message "There is no data to chart".....this is after a fresh restart of Medved Trader.

 

However, if I go to task manager and kill iqconnect.exe, the chart data will come up (Sometimes it takes numerous kills of iqconnect.exe before this begins to work).  Once it starts working, it seems to keep working (adding new symbols and the chart data displays right away).  Once I restart Medved Trader, I have to keep killing iqconnect until it starts working again.

 

Please let me know if you'd like any other info from me.....a new log file, etc.

 

Thanks,

Tim

[Jerry Medved]

I got the log file. Will check on it tomorrow. Meanwhile, are you using any anti-virus or firewall software that may be interfering?

[timtrader]

OK, thanks.

 

I use the standard Norton Security Suite for anti-virus, so nothing special in that area.

 

Thanks,

Tim

[Jerry Medved]

Issue found and fixed. Will be in the next build - this actually impacted multiple datafeeds. 

[timtrader]

I tested your intraday build as instructed.  It works perfectly now!!!

 

Thanks Much,

Tim

[timtrader]

Jerry,

 

I think it would be a good idea to have a Chart/General option to Auto Backfill Intraday charts with TICK data.  This is because with OHLC charts of less than 1 minute (like 15-second, 0.25 minute charts), the most recent day does not show the OHLC bars....even if you select the backfill with TICK data option on the chart itself, you get the same results.  Currently the only way to show the OHLC bars for the current day on a chart of less than 1 minute bars is to first clear at least the current day's data, and then select the backfill with TICK data option....which is a bit of a pain when you're reviewing charts one at a time from a worklist or portfolio.

 

I know the speed would be slower, but not bad if only a few days are selected.

 

It might also work if the "Intraday OHLC Backfill - Number of Days" setting in "Data Sources/Configure Accounts" could be set to zero (it currently can't be set to less than 1), but that might be more confusing (perhaps other intraday charts wouldn't load any chart data if its set to zero.)

 

Hope this helps.

 

Thanks,

Tim

[Jerry Medved]

Tim, 

 

Backfill in general does not overwrite existing data. It fills in gaps. So if you already have OHLC data (like if you have auto backfill enabled), then yes, you would need to clear the data and explicitly backfill with TICK.  Assuming you have Level I quotes running, you should not have to do any backfill again, and auto backfill will basically do nothing.

 

TICK Backfill is far far slower and heavier on the bandwidth if you have a busy symbol (@ES, SPY, AAPL just to name a few).  Which is why we didn't allow it for auto backfill. We could limit it to 1 day backfill, but then we have support nightmares because people would be asking why their charts only show 1 day.

 

What you can do is
1) turn OFF Auto backfill (Settings / Charts / General - on top)

2) Set IQFeed as the Backfill Source explicitly

3) When you start MT, on the Portfolio select to BACKFILL ALL Using TICK Data.

 

That way, all your symbols have data from the time you start, and the backfill all will fill in any gaps from the time you last closed MT.  Going forward MT would be getting tick data, so no backfill is needed. And you would not have to backfill each chart individually

[timtrader]

Jerry,

 

I made the settings you suggested and it does work.  However, keep in mind that I don't really want tick data and am just trying to match my current QuoteTracker setup (7 historical charts, and 15 intraday charts - 7 are 3-minute, 7 are 1-minute, and 1 is 15-seconds.)  I pull in 5 days of intraday data.   In QuoteTracker I do not have the "Get TICK Backfill data" box checked, and the OHLC bars work on all charts, including the 15-second chart, even when a new symbol is added on the fly. (This is my goal to all of this.)

 

My concern is that with the settings you suggested, my PC was often pegged at 100% CPU usage when the market was open (QuoteTracker has a very small footprint with the same charts mentioned above).  I suspect this is because TICK data was streaming in for all of the charts in MT.

 

If I use your method above, and follow it up with a 4th step to select "BACKFILL ALL Using OHLC data" on the portfolio.....am I correct in assuming that the feed going forward wouldn't be using tick data, but OHLC data instead, and return processing requirements to normal?  (I'm sorry, its after hours so I can't try right now.)

 

I'm wondering though, from that point forward, if I'd have to do the backfill option using TICK data for any new symbols that I add, and then do it again using OHLC data so the PC resources aren't pegged.....so its a bit of a dilemma.

 

Thanks,

Tim

[Jerry Medved]

One thing I should have added - you may want to set Max # of days for the tick data to something like 1 or 2 assuming you ruin MT every day.

 

You may say that you don't want tick data, but the description of what you are trying to do implies otherwise. As far as showing 15 sec candles, QT works exactly the same way. There is no way you would be able to see those in QT on a brand new symbol without backfilling with tick data. Impossible. You either had it backfilled, or you had the symbol in the portfolio getting quotes, which would be the same thing as getting tick data. 

 

"Setting you suggested" - How exactly are you setting it up that you are using 100% cpu?  You don't have to open all charts.  You just have to have the symbols in the portfolio.  

 

If you do Backfill All with OHLC data after the tick data, then if lets say you backfilled 2 days of tick data, the rest will be OHLC.  If you run MT every day though, that will not be needed since you will already have the data. 

[timtrader]

Jerry,

I spent quite a bit of time going back and forth between QT and MT to see what is acting differently with similar settings.  To make sure my testing is accurate, I always cleared out all of the chart data before requesting new data (whether with manual backfill, or having the Auto Backfill setting turned on).

In my QuoteTracker setup, I can enter a new symbol (no data already backfilled, and the symbol not already in the portfolio) and ALL 5 days of intraday 15-second chart data, including the most recent (current day) come back showing the OHLC bars on the 15-second chart  (Whether I add a new symbol during and after market hours)  I assume this works because of the Charts/Multi-day Timeframes setting, which “I think” states to get 2 days of tick data and consolidate the data after 2 days (this is where I have my intraday charts set to pull back 5 days of data).   Perhaps this overrides the chart settings themselves which show that the chart type is OHLC, not TICK OHLC…..see below:

 

Below is an example from QuoteTracker for symbol BAC which I never trade.  Just in case, I cleared all of its data after entering the symbol in my worklist.  Then I clicked the symbol to load the linked 15-second data in the chart.  This filled 5 days of data in a second or two.  The CPU only jumped from about 5 to 30% for a second or so.  I also right-clicked the chart afterward to show that it is chart type OHLC, not TICK OHLC (though perhaps it uses tick data anyway in this case?)  You can see that the OHLC bars are showing on the chart. (I scrolled back and saw that they were there for ALL 5 days, not just the most recent 2 days.)
 

 

Below, I’m showing my QuoteTracker Backfill settings which show IQFeed and that the “Get TICK Backfill Data” box is not checked. (Again, I assume this is overridden in the Multi-Timeframes setting (5 consolidated OHLC days after 2 days)

 

 

 

 

FYI, my CPU usage for QuoteTracker normally stays well below 50% throughout the vast majority of the day, even though there are 22 charts open at all times (most on a second monitor, seven historical, seven 3-minute, seven 1-minute, and one 15-second chart).

Anyway, based on these QuoteTracker settings, is there an equivalent in MT to get the same results and not have to do anything special in order to get the OHLC bars to show on 15-second charts when a new symbol is added?  I’d like to not use any more TICK data than QuoteTracker currently uses in order to keep the CPU usage in check.

Thanks,
Tim

[timtrader]

Sorry, the images (Copied from Word) didn't come through.  I'll re-send this info in an email.

 

Thanks,

Tim

[Jerry Medved]

The picture of the QT chart you sent me was exactly what I was talking about - it is NOT showing you 15 second candles. When QT (and MT) backfill OHLC data, the data is stored as 4 points per minute - one for each: Open, High, Low, Close for that minute. If you set the chart to 15 sec candles and have backfilled data, you will just get a candle that shows a dash.  Depending on what seconds those 4 rows are entered on, you might get one candle with 2 entries and another with none, but that is purely arbitrary.   

 

What you see in MT when you set the chart to show same as QT is basically the same thing

 

As for CPU usage when you really do have tick data, I would want to know what MT settings you have.  QT is set to only store 5 days of data. Did you set MT to do the same? default is 60. Especially with 15 sec candles, that setting will be very important. (Settings/Charts/General - about middle of that screen)

[timtrader]

OK, I see what you mean now, but when a brand new 15-second chart (for a new symbol with no previous backfilled data) is entered in both QT and MT the results are always different.  In MT, you always ONLY get the dashes, but with QT, you get a mix that is about half dashes/half bars (the bars look like they are perhaps the high/low for the period).  From that point forward on both MT and QT, the full OHLC bars are generated (during market hours).  I had never noticed this mix of dashes/bars in QT when data for a new symbol is requested.  This mix of dashes and bars in QT is actually good enough to get a decent look at the current chart pattern without having to clear the data and request a backfill with TICK data. (As I said, I hadn't even noticed this mix of dashes and bars until today.)  However, in MT, since only dashes appear, you can't get a decent view of the chart pattern without first clearing the data and then refreshing it with TICK data....this would be time consuming every time you add a new symbol during the day.

 

For sake of clarity, below is an MT example of a 15-second chart (AMRN) which I looked at beginning at about 12:20 PM during market hours today.  You can see that it backfilled with dashes and then began generating full OHLC bars after that.  The chart pattern prior to 12:20 is a bit difficult to see since it is only dashes:

 

 

Below is a QT example of a 15-second chart (BAC) which I looked at beginning at about 14:03 (2:03 PM) during market hours today.  You can see that is it backfilled with a mix of both dashes and bars, but in this case it is a bit easier to decipher the chart pattern. (I guess the data isn't perfect, but its certainly more clear than only displaying dashes.)  As with MT, it generates full OHLC bars going forward:

 

 

 

Hopefully this clarifies what I've been seeing all along.  If MT is only going to show dashes on the initial load of a sub-1-minute chart, will there eventually be a quick way to clear the existing data and then backfill it with TICK data?  I was wondering if eventually, we'd be able to select commands from the Quick Access Toolbar (Like the MS Office products allow).  That would be a workable solution if I could just click a couple of icons when displaying 15-second chart data for a new symbol. (Perhaps an Icon to clear the data, and another to backfill it with TICK data)

 

Regarding CPU usage:

 

Today, during market hours I had a bit of time to use MT.  I set the MAX # of Days of Intraday Data to 5 in the general chart settings, the IQFeed Intraday OHLC Backfill to 5 days, and Intraday TICK backfill to 2 days.  (This matches my QT settings.)

 

I would say that MT is perfectly usable with these settings and it kept up without issue.  The CPU usage overall is higher, normally ranging from about 30-60% during market hours vs. only about 10-20% with QT, though both spike higher for a second when chart data for a new symbol is requested.  That's pretty good considering the MT footprint is about 10 times larger than QT (Based on memory usage as shown in the Task Manager processes).  Startup time is a bit longer....about 55-60 seconds, vs about 40-45 seconds for QT.  This is for a setup with 3 worklists and 1 portfolio for a total of about 24 symbols (varying daily volume from about 500K to 50 Million daily shares), seven historical charts, seven 3-minute charts, seven 1-minute charts, and one 15-second charts (22 charts in total)....spread out on multiple monitors.  I think this is pretty good considering I'm running on an older Pentium D CPU (2.8 GHz), 4 GB memory, with 64-bit Windows 7.

 

It will be interesting to see what you recommend for PC requirements once you're ready to go live with the product.

 

Thanks for all of your feedback. Its really appreciated!

 

Thanks,

Tim

[Jerry Medved]

the reason QT stuff looks different is purely arbitrary.  When configuring whatever source you used, we may have set the 4 points to be at 05 sec, 15 sec, 45 sec and 55 sec, so the points stored at :05 and :15 end up in the same 15 sec candle, and thus the bar.  MT happens to save the 4 points at different times within the minute so each one falls into it's own bar.  But either way, this is totally arbitrary. We picked the points where to store the data. Its not really at those points. You can try to derive some meaning from that, but there isn't any.  The only way you will get any meaningful 15 sec bars is if you have tick data. Its simply not possible any other way.

 

If you want the stuff to look better, you can try using Staircase option,  or Line charts with 15 sec resolution. However the up/down fluctuation within the minute for OHLC backfill data is NOT the same as in the original data since you will always have open, then next point higher (HIGH), then next point sharply lower (LOW) and then another point somewhere in the middle.

 

The only thing we can do is maybe allow auto backfill with tick data, but maybe just for 1 day. Will see.

 

CPU usage =  Pentium D CPU is very old. 2 Cores, no hyper-threading. MT is designed to be heavily multi-threaded. For that to be useful, its best to have more cores. It also uses more memory. Basically, we can try to be super efficient in memory and CPU usage, but then real performance will be degraded. Or we can use as much CPU and memory as available and get the fastest updates.  We chose to go with the later. If MT starts maxing out the CPU, the updates will be slowed down anyway. It is designed specifically for that.  So if CPU is maxed and you are getting 20 updates per second, the charts and quotes might slow down to 5 updates per second.However many it can push through.

 

Performance would be way better with a more modern CPU and more memory. Where QT didn't really benefit from more than a couple CPUs and memory beyond a certain point, MT will use everything.  MT can take 60 days of AAPL tick data for example, where as QT would have choked on it way before 60 days.

 

Having said that, we still need to do some basic optimizations. MT for example doesn't do consolidation at all right now. 

 

Memory Usage in Task Manager - you have to take that with a grain of salt. Not everything shown there is what you think it is.  Just because MT may be shown using 2 Gig of memory, doesn't mean it actually is using that.

[timtrader]

Thanks for all of the data details.  Actually, the Staircase option looks fine, so I'm good using that for 15-second charts instead.  (I loaded a bunch of new 15-second chart data with the Staircase option and compared it to OHLC 15-second TICK chart data (after clearing the data first.)  The initial OHLC data looks fine with the Staircase option....the moving average lines are awfully close either way.

 

Another minor thing:  In the Application/General settings, I have both of the "Separate Task Bar Icon" options NOT selected, and sure enough, when I hover over the MT Icon on the Task Bar, it only shows the dashboard and I can click it and it brings up the entire layout of all MT windows....as expected.  However, when I use Alt+Tab to switch between applications, it shows a separate entry for each MT window.  (So, I can't use it to quickly switch between MT and another application, like a browser as I can do in QT.)  I know the Windows Key+Tab works, but it just doesn't have the same look as Alt+Tab.

 

Regarding CPU, I'm surprised my older PC does as well as it has.  Nevertheless, I see plenty of PCs for sale with the newer 4th gen i7 Intel processors (4 cores/8 threads).  However, how much memory in a newer machine like this would be usefull when using MT?  I see many PCs available out there with 8, 12, 16, even 32GB of memory.  How much memory would MT actually use with about 50 symbols in portfolios and perhaps 7 historical charts and say 15 intraday OHLC charts that pull a total of 5 days of intraday OHLC backfill data, and 2 days OHLC TICK backfill data?

 

Thanks again,

Tim

[Mike Medved]

 

Thanks again,

Tim

Regarding CPU, I'm surprised my older PC does as well as it has.  Nevertheless, I see plenty of PCs for sale with the newer 4th gen i7 Intel processors (4 cores/8 threads).  However, how much memory in a newer machine like this would be usefull when using MT?  I see many PCs available out there with 8, 12, 16, even 32GB of memory.  How much memory would MT actually use with about 50 symbols in portfolios and perhaps 7 historical charts and say 15 intraday OHLC charts that pull a total of 5 days of intraday OHLC backfill data, and 2 days OHLC TICK backfill data?

 

Everything (and I mean it) that we do in MT is with the eye towards speed/efficiency of execution and reduction of memory usage and memory fragmentation. Everything that can be cached is cached. All chart updates try to draw as little on the chart as possible, and only new stuff. The indicator calculation is done using very efficient state machines. Data storage is done using memory-mapped files, which are memory-usage-efficient. And, of course, everything (like, every window for example) is in its own thread, so that the smoothness of user experience is not compromised. No hiccups (or as few as possible).

 

The actual memory usage is very hard to predict, since it depends on a LOT of factors. Every indicator you add to the chart adds memory. If your frequency is 15 sec, you use more memory than if you used 1 min. Tick data on hugely liquid stocks can take a LOT of space (when you have 100 ticks per second for 60 days, it adds up). If historical charts have their last candle updated from L1 feed, that adds to CPU usage. To add to the complexity, this is C# - so everything is done using the arcane magics of the internal memory manager, so when it runs a garbage collection you suddenly see memory usage drop etc.

 

What we do is add features, test, and see what happens. If we see bottlenecks, we solve them. It's an ongoing process.