Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 5, 2004 01:54 PM
Msg. 1 of 11
Hi, I just got some "Q" messages on port 5009 like this one: Q,LLTC,F,.LNCR,-0.07,,3121758,197,38.8300, .LNCR,37.4700,37.5000,700,200,175,173,38.83,14:24t,,38.5500,38.2400,0.03,,,,p,N,,,,11/05/2004,,.LNCR,,,,-0.07,0,33.5,-0.446889086,-0.68,-0.66,-0.38,1,0.018600996,11737427.68,14,4,,4576541,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,14414,,,38.5777, Obviously the ".LNCR" in field 10 is not correct, since this field should hold the daily low. Here are my questions: - What does ".LNCR" mean, where I expect to get a number ? - Which non-number (expect the empty string) may I experience in number fields ? Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its best
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 5, 2004 03:06 PM
Msg. 2 of 11
Here is more strange data: Q,LNCR,F, .CSCO,-0.01,,710115,100,37.8200,.CSCO,19.9700,19.9800,41200,89200,175,173,37.82,15:30t,,37.7900,37.5500,0.01,,,,p,N,,,,11/05/2004,,.CSCO,,,,-0.01,0,14.5,-0.545380922,-17.57,-17.57,-0.25,1,0.013585509,3751710.06,14,4,,1106633,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,2899,,,37.4692,' Q,GENZ,F, .INTC,0.84,,1878108,100,56.2700,.INTC,23.3300,23.3400,65400,95600,175,173,56.27,15:30t,,55.5400,54.8700,0.01,,,,t,N,,,,11/05/2004,,.INTC,,,,0.84,0,68.6,-0.14006044,-32.38,-32.38,0.17,1,0.021719619,12701322.9,14,4,,2075059,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,8819,,,55.6850, Q,LRCX,F, .MRVL,0.72,,2632057,100,26.4000,.MRVL,28.2400,28.2700,100,1000,175,173,26.4,15:30t,,25.9100,25.3800,0.03,,,,t,N,,,,11/05/2004,,.MRVL,,,,0.72,0,21.3,-0.136747458,2.16,2.17,0.19,1,0.026819923,3537019.8,14,4,,4110642,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,9714,,,26.1137, Q,QCOM,F, .AMGN,0.06,,20035175,100,38.8000,.AMGN,61.0400,61.0400,1100,500,175,173,38.8,15:30t,,38.4400,38.0700,0.,,,,t,N,,,,11/05/2004,,.AMGN,,,,0.06,0,39.2,0.315766402,22.92,22.91,-0.31,1,0.022292158,62066526.930000007,14,4,,40502115,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,50890,,,38.3189, Somehow, symbol names are in place of the price for a trade. How come ? Am I missing something ? Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its best
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 5, 2004 03:21 PM
Msg. 3 of 11
And here is more garbage data: Q,CECO,F,38.1200,7.22,0.233656958,1128708,150,38.1200,30.4100,30.6100,30U,F|,600,500,173,173,7.71, 15:30t,,30.9100,30.9000,-0.61,,,,c,N,,,,11/05/2004,,38.1200,,,,7.22,0,18.0,-0.602007052,0.,-0.64,7.21,1,0.202256034,3907185.64,14,-1,,3179301,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,4545,,,30.7194, The marked field should be at position 18 (1 based), but is at 19 ? What's going wrong there ? Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its bestEdited by Dierk Droth on Nov 5, 2004 at 03:21 PM
|
DTN_Natalie_H
-DTN Evangelist-
Posts: 175
Joined: May 10, 2004
DTN Market Access, LLC.
|
Posted: Nov 9, 2004 02:55 PM
Msg. 4 of 11
Dierk,
I'll have to look into these. Thanks for reporting them!
Natalie Hannan DTN Market Access, LLC.
|
LonnieS
-King of IQ Development-
Posts: 127
Joined: Jun 2, 2004
|
Posted: Nov 9, 2004 03:44 PM
Msg. 5 of 11
It looks like message corruption between the IQFeed component and the server. When the server cannot transmit a full message it currently discards the unsent portion. This means the next full message sent is appended to the partial message that was sent last. Corruption in a field will remain until the field is overwritten with good data.
Reasons the server could not transmit: slow network or problems in the network can cause corruption in the server TCP buffer; not reading data fast enough off of the receive TCP buffer also causes corruption. if IQFeed does not have enough CPU available to service the TCP socket it will backlog in the socket buffer and eventually corruption will occur. There are quite a few variables that affect this: number of symbols, market activity, anything else the computer may be doing.
We are aware of these problems and are currently developing fixes to prevent message corruption. In the future we will not discard the unsent portion of the message, we will send it when the socket becomes writable, this guarantees that complete messages will be sent. Until that remainder of the message is sent any new messages will be dropped for that connection.
Lonnie Shumate Development Manager, IQ Systems DTN Market ACCESS
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 9, 2004 04:06 PM
Msg. 6 of 11
Lonnie, Thanks for letting me know. Will this issue be fixed with the upcoming next release (which should be available soon, AFAIK) ? Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its best
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 10, 2004 09:52 AM
Msg. 7 of 11
Oh boy, I just let my test driver collecting market data for some 30 minutes now and got ~3 MB of screwed up data. Although the processor load was low, I had a skype session on the same internet connection. There must be an issue with bandwidth screwing up the communication. This really is a serious issue, which should be fixed with priority. Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its bestEdited by Dierk Droth on Nov 10, 2004 at 09:53 AM
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Nov 15, 2004 10:09 AM
Msg. 8 of 11
I just ran another test session to track ~100 NASDAQ stocks. There is tons of corrupted data coming in holding unexpected values. Plus some news coming in on 9100 are garbage bytes too. Could somebody pls provide a feedback when this issue will be fixed ? Thanks Dierk Droth www.trademagic.net TradeMagic - Trading at its best
|
LonnieS
-King of IQ Development-
Posts: 127
Joined: Jun 2, 2004
|
Posted: Jan 21, 2005 07:42 AM
Msg. 9 of 11
Dierk,
The degree to which you're seeing this concerns me. We're not seeing this as a systemic problem - it's just an occasional problem, usually under heavy load (100 NASDAQ symbols isn't a heavy load). We are working on a fix for the corruption issue, but if the speed of processing at the client connection falls behind, which is what causes the corruption, the result will still be dropped data. I would check cpu load, connection speed, NIC performance, etc.
Lonnie Shumate Development Manager, IQ Systems DTN Market ACCESS
|
Dierk Droth
-Interested User-
Posts: 67
Joined: Aug 11, 2004
|
Posted: Jan 21, 2005 08:51 AM
Msg. 10 of 11
Lonnie, Thanks for letting me know. However, I disagree to some of your statements: The data feed should - in no event - be corrupted. Here is my take: If the internet connection (CPU ...) just is too slow to recieve all requested data, IQFeed might drop/skip complete data messages and discard "incomplete" messages, but not pass corrupted message to the client. Regards Dierk Droth www.trademagic.net TradeMagic - Trading at its best
|
DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004
DTN Market Access, LLC.
|
Posted: Jan 27, 2005 09:57 PM
Msg. 11 of 11
Dierk,
We agree 100% on this one - data should never get corrupted. It's one of the bigger fixes I've got planned for going to QA testing in about 2 weeks. No promises yet, but that's the plan.
Tim Russell Software Engineer DTN IQ & FinWin
|
|
|
|