Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"Version 4.0.0.2 has been working well for me and I appreciate that it is now a much tighter client to work with. I feel I can go to press with my own application and rely on a stable platform" - Comment from David in IA.
"I "bracket trade" all major news releases and I have not found one lag or glitch with DTN.IQ feed. I am very comfortable with their feed under all typical news conditions (Fed releases, employment numbers, etc)." - Comment from Public Forum
"I cannot believe what a difference it makes trading with ProphetX!" - Comment from Bruce in Los Angeles
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"Previously I was using *******. IQFeed is WAY more economical, and for my charting needs is just as good, if not better." - Comment from Public Forum Post
"I just wanted to let you know how fast and easy I found it to integrate IQFeed into our existing Java code using your JNI client. In my experience, such things almost never go so smoothly - great job!" - Comment from Nate
"With HUGE volume on AAPL and RIMM for 2 days, everyone in a trading room was whining about freezes, crashes and lag with *******, RealTick, TS and Cyber. InvestorRT with IQFeed was rock solid. I mean SOLID!" - Comment from Public IRC Chat
"You are either overstaffed or people just don't have problems with your feed because customer support always answers the phone quickly." - Comment from Jay via Email
"I was with ******* for 4 years at $230 a month, this is a huge savings for me, GOD BLESS YOU PEOPLE," - Comment from T.S. via Email
"Everything is working amazing now. I'm already impressed with the true-tick feed of IQFeed and it's ability to support my 480 symbol layout." - Comment from Tyler via Email
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »IQFeed Developer »IQFeed Developer Support »historical tick data download doesn't match real time ticks
Author Topic: historical tick data download doesn't match real time ticks (17 messages, Page 1 of 1)

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 19, 2004 01:30 AM          Msg. 1 of 17
I have noticed that the historical tick data does not match the ticks I have collected in real time.
I know that the real time ticks are complete because there is a running aggregated volume per symbol as well as a counter (per symbol). However, there are often additional trades in the historical data (never the other way round).

Where are theses trades coming from?
They don't seem to exist in any form in the streaming data.


Dennis

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 19, 2004 08:34 AM          Msg. 2 of 17
Dennis,

If you could give me an example of a symbol you see this happening on, I can look into this.

Thanks,

Tim Russell
Software Engineer
DTN IQ & FinWin

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 19, 2004 10:08 AM          Msg. 3 of 17
Tim,
An example,
Date: October 12, 2004
symbol: IACI
timestamp: 13:00

The historical trades include three trades in a row of 2200, 2460 and 2800 shares.
These trades do not exist in the real time data. In the data below, the first three trades
are okay and then things go wrong:

REAL TIME DATA
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,600,3046534,4663
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046634,4664
2004-10-12 13:00:25,IACI,20.60,20.60,20.60,100,3046734,4665
2004-10-12 13:00:30,IACI,20.59,20.60,20.60,100,3046834,4666
2004-10-12 13:00:33,IACI,20.57,20.58,20.58,100,3046934,4667
2004-10-12 13:00:38,IACI,20.57,20.58,20.57,885,3047819,4668

DOWNLOADED HISTORICAL
IACI,0,20041012,13:00,20.600000,600
IACI,0,20041012,13:00,20.600000,100
IACI,0,20041012,13:00,20.600000,100
IACI,0,20041012,13:00,20.600000,2200
IACI,0,20041012,13:00,20.600000,2460
IACI,0,20041012,13:00,20.600000,2800
IACI,0,20041012,13:00,20.590000,300
IACI,0,20041012,13:00,20.590000,200
IACI,0,20041012,13:00,20.590000,885

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 19, 2004 10:28 AM          Msg. 4 of 17
Dennis,

Looking at our stored historical data vs what you sent, I see that those trades received were regular trades. Moreover, the volume on these trades (which is part of the historcal retrieval) is higher than the volume you have.

The obvious reason for this is that data is being dropped passing between our servers and you. The "why" of the dropped data is what we need to pinpoint now; I'll need a bit more information from you.

- What type of connection are you using to the Internet?

- What type of machine is your application running on?

- If possible, can you send me a private message with the list of symbols you're watching? This will give me a better idea of the bandwidth involved.

Now, some background on why these things can impact your data reception.

Our quote servers, after upgrades last month, have proven to be more than capable of keeping up with our data feed and transmitting it to customers in a timely manner. However, memory on these machines is not unlimited - we cannot queue data on our end waiting for the remote clients to be ready to receive it. Doing this would put our servers at the mercy of any client who wanted to break our servers just by not receiving data for an extended period of time. Timely (and complete) delivery of data is entirely dependant on the remote end's ability to read the data fast enough.

While I'm not certain what symbols you are watching, with 1300 symbols being watched, I'm pretty sure this is generating a significant amount of data.

Get back to me with your information, and we'll try to work from there.

Thanx,

Tim Russell
Software Engineer
DTN IQ & FinWin

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 19, 2004 08:06 PM          Msg. 5 of 17
Tim,
My trading depends on a high level of liquidity so the 1300 NASDAQ symbols I am watching are the top 1300 ordered by average traded value excluding the top two symbols (INTC & MSFT). This still leaves me with plenty of very busy symbols. (I will send my list via email)

In answer to your questions:
- Internet connection is high speed cable (in Australia). I can easily sustain 7mbps downloads.
Ping times to your login servers are around 200ms
- I am running the app on an HP xw8000 workstation with dual 3.2GHz Xeons, 4GB RAM, 15,000rpm SCSI drives, etc Windows XP SP2. IQFeed client is v2.3.0.2

I am intrigued by your response. The real time feed had no missing counts. Does that mean that these are generated by the IQFeed client rather than your server?
BTW, the historical data was downloaded by a separate application called Dynastore rather than our application.

At present, my Java application is simply receiving the data stream and writing out the data to disk. At market open the download peaks at 5.5Mbps and then settles down to around 2Mbps. The strange thing here is that the amount of data written to disk is only slightly larger than what is being received. This implies minimal compression from the servers.
I have also had problems with data corruption (missing fields, truncated records, and things like "U,F" in some fields) which may be related. There is a separate thread on this.

Dennis

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 20, 2004 09:15 AM          Msg. 6 of 17
- 7MB sustained should be more than enough - at 9am EST, I'm seeing output at just under .2Mb on your watchlist.

- Nice machine =) As I'm sure you guessed, no problems there.

- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)

Btw, as far as compression goes: I'm seeing 21KBytes/second on our server output, vs 160Kbytes/second (again, with your watchlist).

I'm actually suprised about the "U,F" you're seeing in some fields - this is something you should NEVER see - it's part of our server feed before it's converted to the text feed that IQFeed puts out. I'll pass this on to the IQFeed guys to look at.

Could be this message corruption is the root of your problem, though.

Thanks for the input,

Tim Russell
Software Engineer
DTN IQ & FinWin

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 20, 2004 02:08 PM          Msg. 7 of 17
Quote: Tim,
Some of your comments concern me. I am monitoring the bandwidth between your servers and my PC and it is very high. After the initial burst (625KBytes/sec) I am getting around 300KBytes/sec sustained for the first hour. Last week I had an almost identical watch list and was receiving around 1/8th of this bandwidth which is closer to the figures you mentioned.
The ratio of data downloaded to that output by IQFeed is around 1.2:1 which is much worse than last week.
Clearly something is wrong.
Even now (10:38am your time) I am getting ~180KB/sec from your servers.

Is there any way you can determine what is being sent to me? (ID=200643)

Dennis

PS I'm still getting the "U,F|"



Dennis - I've reposted your email here so that everybody can benefit.

I was under a slight delusion as to how IQFeed handled outgoing data - basically I looked at code that has NOT been completed yet, that fixed a problem that is most likely causing your symptoms.

Basically (and I hope you understand the basics of TCP/IP communication):
- IQFeed sets TCP send / recv buffers to 256 Kbytes.
- When incoming data is processed, a message is created for the client.
- This message is sent to the client. No checking is (currently) being done to ensure that this message is sent in its entirety. This is fine so long as the 256k TCP buffer doesn't get full.
- After the TCP buffer gets full, attempts to write to the socket will fail.
- You will get corrupted data / lose data.

The obvious problem here is that your code may not be processing messages from IQFeed fast enough. A couple of solutions:

- Make your TCP recv buffer bigger. In java.net.Socket, I believe this is accomplished via the setReceiveBufferSize() method.

- Read data from the incoming socket in one thread, and pass this data to another thread - via a queue or whatever - to be processed.

Sorry for failing to notice this earlier. The recv buffer size fix is often offered, but I wasn't aware of why until I got one of our client guys to help me find the problem in the code.

If this helps you, let me know.

Now, for the amount of data you're seeing - are you getting this via the IQConnectionManager? If not, where are you getting these values? Only curious, as what I'm seeing going to your computer doesn't account for that much data.

Thanks,

Tim Russell
Software Engineer
DTN IQ & FinWin

skunk
-DTN Evangelist-
Posts: 249
Joined: May 7, 2004


Posted: Oct 21, 2004 08:26 AM          Msg. 8 of 17
"- I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output)"

Without having seen his code, I imagine thet he is using the field "Number of Trades Today" (field 56 in the tcp buffer) to indicate the sequence number of the print.

Where does this field come from? From the description of this problem it appears that it is generated by the IQFeed client, not the server. This makes it significantly less useful, if not useless.
Edited by skunk on Oct 21, 2004 at 08:26 AM

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 21, 2004 08:32 AM          Msg. 9 of 17
Ah - trades today. Okay.

That value's maintained by the server - it's returned to you when you request it. However, it's not transmitted on a tick-by-tick basis, since it's easily calculable on the remote end.

Tim Russell
Software Engineer
DTN IQ & FinWin

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 21, 2004 08:54 AM          Msg. 10 of 17
Okay......you can forget about all my concerns to do with high bandwidth as it was my fault. My application logs just about everything and I recently changed the location of the log to a networked drive. The bandwidth measurements were being skewed by this.

Tim, I have to agree with skunk on the value of the counters. If you can't tell when you've missed a trade then what's the point. If you look at the real time data example I gave earlier then it appears that I haven't missed anything. Clearly the IQFeed client wasn't aware either.

I've made the changes to the way I read from the socket connection and increased the java buffer sizes as suggested. I now have a separate thread that takes the data stream from IQfeed and place it in a separate buffer for the application to read. It seems to be working very well with only a small number of error packets so far. I'll analyse it better at the end of trading.
I am somewhat stunned that a buffer overflow can result in corrupt data. I can understand missed message or even truncated messages, but corrupt individual fields within messages???

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 21, 2004 09:26 AM          Msg. 11 of 17
The trade counters were originally for internal use, and used by our FinWin product - since it's a web service, it doesn't do streaming data, and this information is very useful to those customers. The reason we made this available in the streaming feed was because there seemed to be a bit of interest in it.

However, it is NOT designed to be a method of letting you know data has been dropped in the feed.

As for the corrupt messages - let me know what you see.

Tim Russell
Software Engineer
DTN IQ & FinWin

skunk
-DTN Evangelist-
Posts: 249
Joined: May 7, 2004


Posted: Oct 21, 2004 10:59 AM          Msg. 12 of 17
So, to summarize (and please correct any misunderstandings)

As of 2.3.0.2 and earlier there is NO flow control between IQConnect and API socket clients. The assumption is that the client will empty the receive buffer fast enough so there are no buffer overruns.

The "num trades today" field is useless to detect missed update messages beween your server farm and IQConnect. It might be useful to detect lost messages between IQConnect and the client app though.

Is there any way to detect update messages lost beween your servers and IQConnect?

And yes, this is a thinly veiled request for server (or even exchange) generated timestamps and sequence numbers to be transmitted to API clients.

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 21, 2004 10:02 PM          Msg. 13 of 17
Okay,
I've now done the analysis of all data received from IQFeed (thank god for Linux utilities) and have mostly bad news.
At the start of market open, I received an average of 1400 messages (trades,bids,etc) per second and had no problems at all. This is the highest data rate for the day and suggests that there are no buffer overrun problems. However, I received a large number of invalid messages starting at EXACTLY 12:00pm. Now if the corruption was due to buffer overruns or lack of bandwidth or CPU then you would expect all symbols to have corrupt messages. The below analysis of these invalid messages is very telling:

Entries containing the string "U,F|" in place of expected data:
1 ADEX
1 AMZN
14821 BMET
1 BRCM
2001 CMVT
1 CNCT
2 COCO
186 EBAY
1 EMMS
6 FAST
1 FHRX
1 FRED
785 FXEN
1 JAKK
1 MLNM
2 MRCY
1 NCRX
4149 PMCS
1038 PWAV
1 PZZA
2 QLGC
13641 RSAS
1 RTIX
1 SAFC
2620 SAPE
1 TSAI
34868 VRTS
53309 YHOO
=======
127444


Trades containing more than 59 commas (invalid):
1 ADEX
1 AMZN
3391 BMET
1 BRCM
654 CMVT
1 CNCT
1 COCO
1 CSCO
169 EBAY
1 EMMS
2 FAST
1 FHRX
1 FRED
116 FXEN
1 JAKK
1 MLNM
2 MRCY
1 NCRX
1 PCAR
1510 PMCS
422 PWAV
1 PZZA
2 QLGC
4752 RSAS
1 RTIX
1 SAFC
816 SAPE
1 TSAI
13609 VRTS
18306 YHOO
=======
43768

As you can see, the 170,000+ errors affected only a handful of the 1300 symbols I'm watching. Some very busy symbols such as CSCO are not represented.
Futhermore, the error in the message is in excatly the same position for a given symbol.

eg
Q,YHOO,F,35.3800,0.89,0.025804581,16679769,100,3537U,F|,34.9000,35.3700,35.3800,3400,600,,175,3502.1,12:56b
,,35.3700,34.4900,0.01,,,,p,N,,,,10/21/2004,,35.4000,,,,0.91,-0.01,90.8,-0.212661364,0.,0.,0.01,0,98.985302
431,48137214.260000005,14,4,,16009207,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,37928,,,35.3464,

The above has a U,F| which makes it invalid. All YHOO corrupt messages have this in the same position.

This seems to point the finger directly at the IQFeed client software. Perhaps it has separate buffers for each symbol........it's just speculation.

TIM: you mentioned that you're more of a server developer. Can we get some input from a client side developer please. I am now heading off to the "IQFeed Data Wishlist" forum to ask Santa for my early Christmas present.
http://forums.iqfeed.net/index.cfm?page=forum&forumID=15

DTN_Steve_G
-DTN IQFeed-
Posts: 28
Joined: Oct 1, 2004


Posted: Oct 22, 2004 08:58 AM          Msg. 14 of 17
Dennis,

Thank you for the analysis. I will forward this on to our software engineering staff who are already investigating the "U,F|" issue.

Steve Grunberg
IQFeed Developer Support
DTN Market Access

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 22, 2004 09:18 AM          Msg. 15 of 17
Steve,

You can also send them this one which I received today (never seen this before):

TL|=8800|>18.85|?1.46|@VOZ,YYT|B585.6|C28.9|D06/01/04|E172.5|F69586|H307000,20040423|I144400,20040830|J307000,20040423|K144400,20040830|k0.50,10/13
/99|}

It looks like the output to the log.txt file from the IQFeed client.

Dennis

dhakme
-DTN Evangelist-
Posts: 150
Joined: Sep 17, 2004


Posted: Oct 26, 2004 07:49 PM          Msg. 16 of 17
Steve,
Have you heard anything from the software developers yet?
I'm keen to get acknowledgement that the problem is a bug with the IQFeed client and some indication of when a fix can be expected.
To blame this problem on an application that is too slow or has insufficiently large buffers would be unacceptable as that should result in lost messages, not corrupt messages.

Dennis

DTN_Steve_G
-DTN IQFeed-
Posts: 28
Joined: Oct 1, 2004


Posted: Oct 27, 2004 12:57 PM          Msg. 17 of 17
Dennis,

The reported issue of the character string "U,F|" occasionally appearing in streaming data has been logged in our IQFeed API Bug Database as item #2750. You can monitor its status by going out to the IQFeed Developer website and selecting the menu item "View Current IQFeed API Bug Database.

Our software support team will need to investigate it to see if they can replicate its occurence, prior to resolving it. No estimated resolution time has been set.

Steve Grunberg
IQFeed Developer Support
DTN Market Access
 

 

Time: Sat October 1, 2022 12:41 AM CFBB v1.2.0 0 ms.
© AderSoftware 2002-2003