I've been receiving invalid update messages through the socket API;
first I thought it was a bug in my update-parsing process so I modified the console_stream example to duplicate the problem -- sent the program

- it subscribes to the Nasdaq100 stocks,
- reads 1000 updates,
- unsubscribes for the symbols,
- after suspending for a minute starts the iteration again.
/This modified console_stream uses CString so use MFC static/shared library + replace #include <windows.h> to #include <afs.h> in stdafx.h/

After some iterations I receive updates like:

Q,FLEX,F,13.6701,-0.1299,-0.00941304,1766768,181,13.8800,.PSFT,13.6700,13.6700,1100,800,,173,13.88,11:51b,,13.8100,13.8000,0.,,,,c,N,,,,10/06/2004,,13.6701,,,,-0.1299,0,1380.,-0.741625037,0.,0.,-0.1399,1,1.015354679,7602092.9811,14,4,,6610306,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,4607,,,,
Note the 9th column: ".PSFT". Similar happens in other price fields as well (bid, ask, last, etc).

Could somebody also compile/run the program and see what happens (pm me , I can send the code)? Ususally the longer it runs, the more invalid updates it receives.

I appreciate any ideas, comments!!!

Lajos
Edited by DTN_Steve_S on Sep 19, 2011 at 09:22 AM

I think you received a response via email, but for the benefit of others (since we have seen this before), this problem of corrupt data is nearly always a result of having to small a TCP buffer. Our feed is one of the fastest and most comprehensive, so while you may have been able to get away with a small TCP buffer on feeds that consolidate or filter their feed, with ours you need to open it up. I think we recommend a 64k buffer. If I am wrong, I am sure someone will correct me with the proper size they use to keep from corrupting messages.

---

Jay Froscheiser
DTN Market Access, LLC.

Thank you for the fast reply, I really appreciate it!

I've tried the 64k TCP buffer modification and double-checked the partial
message handling -- I could not find any bugs in it, so I would assume that
the invalid updates were not due to partial messages as they are handled by
the test application. But just in case I've started the 64k TCP buffered
application today shortly after market open and got invalid updates again
(lines from the test application's output):

...
Starting iteration #3
Parse error in column 3 in line:
P,SPLS,F,.SNPS,,,3185,100,0.0000,0.0000,16.7100,16.8000,200,300,,173,0.,09:29t,,0.0000,29.8700,0.09,,,,t,N,,,,10/07/2004,,.SNPS,,,,,0,,,,,,1,,,14,4,,1513610,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,23,,,,
Parse error in column 3 in line:
P,WFMI,F,.ADBE,,,1500,100,0.0000,0.0000,50.8600,50.9500,100,100,,175,0.,09:29t,,0.0000,84.8400,0.09,,,,t,N,,,,10/07/2004,,.ADBE,,,,,0,,,,,,1,,,14,4,,591682,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,12,,,,
Parse error in column 3 in line:
Q,SPLS,F,.SNPS,,,3185,100,0.0000,0.0000,29.8000,29.8000,500,300,,175,0.,09:29b,,0.0000,29.8700,0.,,,,t,N,,,,10/07/2004,,.SNPS,,,,,0,,,13.09,13.,,1,,,14,4,,1513610,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,23,,,,

...

Parse error in column 9 in line:
Q,AMGN,F,56.6700,0.11,0.001944837,126434,100,56.6700,.PAYX,56.6500,56.6800,100,200,173,173,56.67,09:31t,,56.2900,56.5600,0.03,,,,p,N,,,,10/08/2004,,56.6700,,,,0.11,0,28.9,-0.981450411,0.,0.,0.38,1,1.,71745183.390000001,14,4,,6750460,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,453,,,,
Parse error in column 9 in line:
Q,AMGN,F,56.6700,0.11,0.001944837,126434,100,56.6700,.PAYX,56.6500,56.6800,300,200,,173,56.67,09:31b,,56.2900,56.5600,0.03,,,,c,N,,,,10/08/2004,,56.6700,,,,0.11,0,28.9,-0.981450411,0.,0.,0.38,1,1.,71745183.390000001,14,4,,6750460,AMEX-BSE-CSE-CHX-PSE-NMS,,,,,453,,,,

...

I would be very happy if you could compile and run this test application (sent in email) on
some of your development machines, to see whether this error can be
duplicated in your environment.

Best Regards,
Lajos

I too have just noticed this problem and can replicate it using small buffers and large buffers.

I don't think it has anything to do with the buffer size as long as there are no bugs in your buffer handling and you are processing new data quickly enough.

It seems to be a problem of sending too many Watch request messages too quickly. If I send max 50 Watch symbol messages at a time then it doesn't happen. I haven't bothered to fine tune it and find the magic number because it will vary on different machines. I know 100 doesn't work.

Like Lajos I have received similar message corruption but with all message types - P, F and Q (Summary, Fundamental and Update). Most of the fields are returned but a few of the fields are rubbish, e.g. Last price = ".AMAT", etc...

I have used between 256 byte and 4096 byte buffers for each read and then multiple re-reads if more data is present. Whether you handle it quickly, very quickly or extremely quickly it doesn't matter. I have only been able to test reading up to 300KB per second from the socket in one hit if data is available, and don't seem to have any problems with my buffer handling.

Corruption only happens when:
1. Sending multiple Watch requests of 100 symbols, and only
2. If IQ Connection Manager is NOT already subscribed to them

I am dispatching a total of 95 Watch request messages one at a time, consecutively and asynchronously.

I don't think this is symbol related but I'm sure if you pick the busiest Nasdaq 100 symbols you have a greater chance of seeing corruption. I had 90 Nasdaq 100 and 5 Dow symbols.

I am using DTN.IQ and when it starts I can open up the IQ Connection Manager and check that there are no symbols being watched.

So the process is:
1. Close IQConnect process/es
2. Connect to TCPIP Level 1 stream
3. Send 100 Watch request messages
4. One of the first response messages (P, F or Q) of the 80-90th symbol will have bad fields
5. Maybe a few more corrupt messages come back
6. The rest of the messages from then on are OK
7. Un-Watch all these symbols and repeat from Step 2 and you will have the same problem

However, if you just kill the app without unwatching, IQConnection Manager shows there are still 100 symbols being watched. If you repeat from Step 2 you won't have a problem - at least I haven't seen it.

Alternatively you can Send only 50 Watch request messages at a time and not have this problem.

I'm using my own C# code.

A sample bad message I detected from error logs converting string fields to numeric types - I didn't check if the fields make sense at this point - just an attempt to convert to types.

=> Disney has a Last price of ".2485", High price of "U", Low price of "F|" and Bid Size of "55.95", Market Open of "0.001"
P,DIS,D,.2485,0.0010,0.004040404,,,U,F|,,55.9100,55.9500,100,700,,,,,,.2475,.2475,0.04,,,,n,,,,,,,.2485,,,,0.0010,0,,,,,0.001,1,,,,4,,,CSE-NYSE-NMS,,,,,1,,,,

The next message for Disney is OK.

Repeat and I get the problem with the first message for the same symbol Disney, similar and/or different fields. I could also get one more Update message for Disney that is also corrupt.

Regards,
Sasha
Edited by sasha on Oct 8, 2004 at 02:17 PM