Hello,

I have a custom application that collects tick data (realtime and history) into OHLCV bars by different timeframes and analyses the price levels. As a result I want to get the same price and volume as in your history interval data. Currently I'm testing my application on history and realtime ticks and want to have a clear understanding how should I filter ticks to get the same volume (period volume) and prices (open, high, low, close) as your historical interval data?
For example, currently I do not use the following ticks:
- Realtime ticks with "Message Contents" field that is not equal "C" (Last Qualified Trade)
- History ticks with "Basis For Last" field that is not equal "C" (Last Qualified Trade)
Should I implement any additional filtering?

Thanks.

Good morning, we include C & E trades in our interval data. So you would want to add those in as well.

Tim Walter

Thank you Tim.

Anytime, while I am thinking about it, also make note our intervals are such that the data from 11:00:00 to 11:00:59.999999 is placed in the 11:01 minute bar. This can cause some people an off by one error when comparing data.

Let me know if you have any questions,

Tim

Hello Tim,

I'm checking the data matching between ticks and interval and found some inconsistencies.
Dealing with BD#C, timeframe is 1 minute, date - 19 November 2015, time 2:02.

Request of interval history data was the following:
HIT,BD#C,60,20150329 030000,20160329 160000,,,,1,HIT_BD#C

Interval data from response:
HIT_BD#C,2015-11-19 02:02:00,156.66,156.61,156.63,156.65,2242,2238,

Next, requested tick data for the interval above:
HTT,BD#C,20151119 020100,20151119 020159,,,,1,HTT_BD#C

Response and volume calculations are in the attached file (ticks_BD#C_20151119_0202.xlsx). There are tick fields in the columns from "A" to "J". Look at column "T" please, there is summary volume (sum of "last size" fields) of the tick data and and interval (period) volume of the interval data.

Could you tell me the reason why these volumes are not equal?

Thank you.
Simon

Sorry, xlsx attachments are not supported. Wrapped in a zip archive.

Hi guys, i'm finding numerous inconsistencies between interval data and tick data, could someone please get back to me on the above question today? Trying to get to the bottom of this!

Really appreciate your help

Sorry, the file that you uploaded appears to have an issue. Could you re-upload your zip file for me to look over?

Tim

Zip-file is re-uploaded. Just in case, here is a link on dropbox: https://www.dropbox.com/s/c9t8cc08jc8p4k2/ticks_BD%23C_20151119_0202.zip?dl=0

Thanks much. Looking it over now.

What protocol are you requesting the data under?

Note: If you are not calling SET PROTOCOL anywhere on this socket, it would be protocol 4.9.

On startup my application connects to the Level1 and sets protocol v5.1.

Hello,

Sorry for the delay, but this one took me a little while to track down. It appears this is related to an issue that occurred back in 2015 during our initial roll-out of microseconds. This caused the discrepancies that you are seeing in our minute data and we are looking into what it will take to get it resolved. The tick data is correct though in all of the cases that I researched, so in the interim, if you build your bars from the underlying tick data, you will have the correct information.

I will try to get more information on this in the days to come. Let me know if you have any questions beyond this though and I will be happy to impart whatever I can.

Tim

Thank you very much Tim.

Currently I'm testing my program with realtime (Level1) ticks. To build bars from tick I'm using the following felds of the Q messages: "Last", "Bid", "Ask", "Last Size", "Last TimeMS" and "Last Date".
Checks showed that value of "Last TimeMS" has a large difference sometimes (more than 5 minutes) comparing to current time. Should I use "Most Recent Trade", "Bid", "Ask", "Most Recent Size", "Most Recent TimeMS" and "Most Recent Date" instead? I want to build bars in realtime and the build bars should match your history interval data.

Thank you.

Most Recent TimeMS is what you would want. This time field only updates on a trade though, so having it go several minutes without changing, while the bid and ask values may be updating constantly can be completely normal. We would expect the time time value to update on each message that has a Message Contents of C, E, or O. Though, you should be filtering out trades with a message content value of 'O' if you want to match up to history.

Tim

Thank you again for quick reply!

So, in the other words, your historical ticks (response for any of HT* command) according to realtime ticks in the following way:
Last (history) = Most Recent Trade (realtime)
Last Size (history) = Most Recent Trade Size (realtime)
Time Stamp (history) = Most Recent Trade Date + Most Recent Trade TimeMS (realtime)
Bid (history) = Bid (realtime)
Ask (history) = Ask (realtime)

Is it correct?

Assuming you are filtering out "O" trades and that you are using protocol 5.1, the new protocol usees different time fields.

Last (history) = Most Recent Trade (realtime) = true
Last Size (history) = Most Recent Trade Size (realtime) = true
Time Stamp (history) = Most Recent Trade Date + Most Recent Trade TimeMS (realtime) = true

A snapshot of these values are taken at the time of the trade message. The individual messages are not saved.

Bid (history) = Bid (realtime)
Ask (history) = Ask (realtime)

Tim

'It appears this is related to an issue that occurred back in 2015 during our initial roll-out of microseconds'

Can you please expand on this...it kind of seems like a big deal.
How invalid is the post 2015 interval data?

There were 2 days, on the 18th and 19th of November 2015, where we received trades with a second value of 60 or more from Eurex, so we are rebuilding those intraday bars, for that exchange.

Tim

Cheers.

Hello,

Just wanted to let you know all relevant changes were made to the futures contracts affected.

Sorry for the inconvenience,

Tim

By the way: Some minutes actually do have 61 seconds. The last one was apparently at "June 30, 2015 at 23:59:60 UTC".

https://en.wikipedia.org/wiki/Leap_second

How is this handled by IQFeed? Wouldn't want my app to crash in such a case.... :)

Yep, those happen every so often. They get rolled into the previous second, luckily they happen when most everything is closed.

Tim