Intraday backfill not populating all minute candles in WS API

[Benji]

I'm running into an issue in the websocket API when backfilling intraday candles (e.g. 15 minute). The first time I backfill for a symbol, the data comes in correctly. Afterward, the candles never update unless the symbol is in an active portfolio in MT.

If I start streaming L1 updates, the backfill will start returning the latest values, but it will never correct the previous candles until I clear data inside MT.

The same happens with all data streamers, and it doesn't matter if I do a "full" or "incremental" backfill. 

I could programmatically get around this if there was a "clear" command in the API, but I don't see that option. I would prefer not to have to keep every symbol I may lookup inside a MT portfolio to get around this.

Example request:

{
  "cmd": "backfillAndGetCandles",
  "reqID": "1246",
  "streamerID": "AMTD",
  "sym": "NVDA",
  "type": "OHLC",
  "backfillType": "full",
  "startDate": "2023-02-13T16:00:00.0000000-08:00",
  "candleSize": "1",
  "regHoursOnly": true
}

 

The last few response candles:

 {
        "t": "2023-02-15T19:42:00.000",
        "o": 227.27,
        "h": 227.49,
        "l": 227.2601,
        "c": 227.4462,
        "v": 37713.0
      },
      {
        "t": "2023-02-15T19:43:00.000",
        "o": 227.44,
        "h": 227.6,
        "l": 227.41,
        "c": 227.6,
        "v": 31125.0
      },
      {
        "t": "2023-02-15T19:45:00.000",
        "o": 227.65,
        "h": 227.6686,
        "l": 227.62,
        "c": 227.6399,
        "v": 6875.0
      }
 

Minute 19:44 is missing, but you can see minute 19:45 is there because I started streaming L1 updates at that time. "backfillandgetcandles" will never return minute 19:44 until I clear data inside MT window.

I've reproduced this in both the latest production and beta releases (1.1.9810.100 x64).

 

[Benji]

Upon further testing, I've determined that the issue is with "backfillAndGetCandles". If I call "backfill" first, then call "getCandles", I get the updates as expected when setting "backfillType" to "full". Backfill "full" has no effect when using backfillAndGetCandles. There is a bug, but at least I've found a workaround. This also explains why my historical charts wouldn't fully backfill until I clear data in my MT portfolio windows.

[Mike Medved]

Will check thanks.

[Mike Medved]

When I send out the backfillAndGetCandles, I first receive these messages

{
  "cmd": "backfill",
  "reqID": "1246",
  "success": "OK",
  "resultcode": 200,
  "result": {
    "action": "started",
    "type": "OHLC",
    "sym": "NVDA"
  }
}

... and then

{
  "cmd": "backfill",
  "reqID": "1246",
  "success": "OK",
  "resultcode": 200,
  "result": {
    "action": "finished",
    "type": "OHLC",
    "sym": "NVDA"
  }
}
 

... and only after that, the

{
  "cmd": "getCandles",
  "reqID": "1246",
  "success": "OK",

... etc.


Is this not what happens for you?

[Benji]

Yes, I receive those messages. And the first time I run it for a symbol, I get the correct intraday minute candles. If I try to run it later in the day for the same symbol, it gives me back the same candles I received earlier without any new updates. So it doesn't appear to be performing a backfill. To fix the issue, I have to go into MT and clear the intraday data for the symbol.

If I'm subscribed to L1 updates for the symbol, or if the symbol exists in a active MT portfolio, this issue does not occur.

Daily candles seem to incrementally backfill correctly. However, if I have bad candles in MT database (e.g. datasource has incorrect stock split data), doing a "full" backfillType has no effect and does not fix them when using 'backfillAndGetCandles'. I also have to manually clear the data in MT to fix that issue.

Is there a way I can clear data for a symbol from the API? This would actually solve all my problems. I have this occasionally when using MT directly as well - sometimes backfill just won't merge data updates until I clear.

[Mike Medved]

I fixed this - the fix will be in next release. Basically the backfillAndGetCandles's "full" method was not being passed on.

[Benji]

Thanks Mike!

[Jerry Medved]

emailed you the update