Hi,

I've got invalid updates 2 days ago while having several simultaneous connections, these all had invalid last prices like:

Q,FWLT,F,0.0NRG,1.96,...
Q,SMH,E,3U,.D|,-32.69,...
Q,FRO,D,LL.TC,1.24,...

I was running the latest 4.2.1.4 IQFeed, any idea why this could have happened?

Regards,
lvarga

Hello lvarga,

The actual source of the data corruption is in the server's outgoing socket buffer. The corruption has been resolved in the upcomming server release. However, this does NOT entirely solve the problem because the cause of the problem is on the client side.

In most cases, this is caused by the client application not processing the data returned by the feed quick enough. It is also evident if your connection to our servers has fairly high latency (200+ms).

If you are seeing that the data corruption coincides with spikes in CPU or increased RAM usage, it is almost certainly an issue with not processing the data fast enough.

Now, as I mentioned above, the data corruption shouldn't happen when you are connected to the new servers but if the conditions occur that cause the problems, you will simply not receive the messages that would have been corrupted.

Hi Steve,

Thanks a lot for the prompt reply, there are few things I would like to clarify a bit:
1) when you say the client application is not processing the data fast enough, you mean my application that connects to localhost:5009 or the IQFeed client (with the icon on the tray) that connects to the IQFeed servers?
2) You say "...the data corruption shouldn't happen when you are connected to the new servers.." -- this means that after 16th's switchover this should be solved as I would automatically connect to the new servers without any changes on my side, right? (and if still not solved I should raise this issue again)

Thank you very much,
lvarga

1) There isn't a straight forward answer for this. Ultimately, it is IQFeed client that is not reading the data quick enough from the servers. However, this should only happen if your client app (connecting to localhost:5009) is not reading fast enough from the IQFeed client.

This is because the IQFeed client is designed in a way to guarantee that every message that is sent from the server is delivered to your application. If you app gets behind, then the IQFeed client has to store that data until your app is able to retrieve it. This increases the amount of CPU and RAM that the IQFeed client uses and if the situation becomes bad enough, your app and the IQFeed client will eventually start competing for computer resources. When this happens, you will almost certainly notice data corruption.

2) Once again, not quite a straight forward response here. You are correct that the issue will be resolved in the sense that you will no longer receive the corrupted messages. However, since the cause of the problem is acutally clientside, this doesn't actually solve whatever is causing the issue, it simply hides the problem. If you are experiencing corrupted messages now, you will almost certainly not be getting all of the data from the streaming feed on the new servers (essentiually some messages will be dropped to slowdown the data feed and pevent corruption). As a result, "resolved" will become a relative term that will vary from developer to developer. If your app is dependant upon receiving every tick as they happen, the issue will simply be hidden on the new servers. But if your app does not depend upon having every single tick, then you might consider the issue resolved.

Hi Steve,

I often receive similar corrupted messages. My connection is not very fast, it's latency is 200+ms. What can we do to avoid data corruption? Maybe it's possible to increase local data buffer or something else?..

As I mentioned briefly in my initial post, you need to determine what is causing the corruption.

You say that your latency is 200+ms so there is probably some corruption that will occur reguardless of how efficiently your app is processing data. The amount of corruption based on latency is going to be dependant on how many messages per second is being delivered to your app.

The more important thing at this point is to verify that the corruption isn't comming from slow processing in your app. The easiest way to verify this is by monitoring IQConnect's CPU and RAM usage. IQConnect should never use more than about 15-20MB of RAM. If you monitor the RAM usage of IQConnect.exe and see the usage consistantly more than this, then there is almost certainly some improvement in data processing that can be made on your end to reduce the amount of corruption you are seeing. CPU usage spikes should also accompany the RAM usage increases.

If IQConnect seems to be operating within normal CPU and RAM usage, then it is fairly safe to say that your corruption is going to be related to the latency. At that point, the only way you can aleviate it would be by requesting less data or by reducing your latency. Depending upon where you are located, this might mean getting a faster internet connection or simply a different ISP. If you are not located in the U.S. and unable to upgrade or change your internet connection, you might consider using a co-location service to move thier IQFeed Computer to the U.S. (reducing the physical distance to the servers will almost always reduce latency). I know that a couple of our foreign 3rd party developers have had success with this in the past.

On the other hand, keep in mind that the actual corruption will go away with the new servers. When that happens, instead of the servers corrupting the data, the data will get dropped. Your app almost certainly is dropping that data currently so the actual data that gets received for processing will be the same, the only difference is that the corrupted data isn't sent to you.

Will the IQFeed application receive a notification that "the data has been dropped"?

Not in the current version. However, I have already approached the issue with the server team and we are looking into possible implementations to get a notification to the client.

Steve (and others), I think your answers are not quite correct and therefore need clarification. I have identified three causes of corrupt data:

1) Exchange generated - this is usually simple errors such as a bid of 323.5 when it should have been 32.35. Since DTN is a raw feed these are passed through uncorrected.
2) Corrupt data being sent by the DTN servers.
3) Corrupt data produced by the IQFeed client

Type (2) corruption normally occurs when there is a spike in traffic (usually when the markets respond to an announcement). TCP/IP packets from the server normally contain multiple updates and naturally, some updates are split over packets. The corruption can occur when the last update record is split over two packets and should be completed in the next packet but isn't. Instead a new update starts in the next packet! This is unlikely to be a server application problem directly, but more likely a TCP buffer overrun on the server (why they chose TCP over UDP for streaming data I'll never understand). Hopefully, this is the problem that Steve mentioned will be fixed in the new server release ("outgoing socket buffer"). This problem is a server side problem and has nothing to do with the IQFeed client, although it should be noted that the IQFeed client completely mishandles this corruption which only exacerbates the situation.

Type (3) corruption occurs within the IQFeed client. There has been much conjecure about the causes and the known culprits are long latency times as well as insufficient bandwidth or CPU. Unfortunately, DTN thinks it's acceptable to send corrupt packets to the client through the API while I disagree. Wouldn't it be preferable for the IQFeed client to handle its buffers a little better?

Ultimately if there is insufficient bandwidth/cpu data (whole updates) will be lost. This is of course, true of any client.

The IQFeed client is also VERY inefficient when it come to CPU. A 3GHz P4 is insufficient to watch the 1300 most active symbols on either NYSE or Nasdaq. By comparison, we have comprable feeds that use less than 1/10th the cpu IQFeed uses.

I once again make the offer to write a Java IQFeed tick client that I guarantee will use far less CPU and NEVER send corrupt updates to the client. I just need approval and cooperation from IQFeed.

Dennis,

You are correct that we do not filter bad data from the exchange. This is done because many of our customers prefer to have the true data that the exchange sends. If we were to filter data, it would completely eliminate that customer base. Additionally, most of our customers who would like the data be filtered would not agree on how it should be implemented. As a result, we allow each developer to handle filtering on thier own so it is exactly to thier (or thier customers') liking.

You are also correct about the conditions for which the corruption occurs within the server (partial messages getting sent in a packet) but it still isn't entirely accurate. To clarify further (and hopefully to your satisfaction); currently when corruption occurs, it is actually 2 messages being affected (the one that was partially sent and the one immediately following it). IQFeed has no way to detect that a new message started and processes both messages as one. The new servers guarantee delivery of a full message (no more corrupted messages). Any messages that arrive while the previous message is still being sent, will get dropped. So where 2 messages are affected in the current server, only one message will be affected in the new server.


As for your Type 3:
I suppose that this all depends upon your definition of corrupt data. With IQFeed version 4.x, I have never found a case where the IQFeed client (IQConnect) actually corrupts data in the sense that this post is talking about. All of the data corruption that has occured, is sent to the client in the same form it is received from the server. I have discussed this with several of our 3rd party developers who are experiencing the corrupted messages on the current servers, and 100% of them have reported that their app does not receive the corrupted messages on the new servers. If IQConnect does somehow receive a corrupt or "bad" message, it will pass that message on to the client application. I do not agree that this is "mishandling" the data. It is functioning exactly as IQConnect is designed to function. It guarantees that every message requested by your app and sent by the server gets delivered to your app. It also means that almost all data issues can be corrected serverside without requiring a client update.

Steve,

I agree that server side errors affect two update messages. However, the second update is a complete update but is still discarded by the IQFeed client because it does not identify it as a separate update, but part of the first partial update. That's why I stated that "...IQFeed client completely mishandles this corruption which only exacerbates the situation".
A perfectly good update from the server was lost by the IQFeed client.

You wrote:


I hope either I have misunderstood or you are wrong in the above quote. Are you saying that the fix for the server side corruption is that a perfectly good update will be discarded because it arrives at an inconvenient time?


You wrote:

The start of this thread had updates such as "Q,SMH,E,3U,.D|,-32.69,...". The server did not generate this corrupt update, the IQFeed client did. You can blame latency, lack of cpu, insufficient memory, etc but the bottom line is these situations are not handled elegantly at all and IQfeed dishes out garbage. In fact, IQFeed will often continue to generate corrupt updates for a given symbol until that symbol is unwatched and rewatched.

Dennis, As I stated before, IQConnect sends to the client exactly what values the server sends for each field. Simple as that. It does no data validation whatsoever (it doesn't even do type validation as you can see). However, I believe this part of this discussion is reaching a point of uselessness. We are currently arguing about a bug that is going to be non-existant when the new servers roll out and that will start happening in less than 16hours. As a result, I am not going to argue this point with you any longer.

However, I will address your other concern and that is as follows:

Yes, that is what I am saying. If an update comes in while another is still waiting for the send to complete, the new update is dropped instead of being sent. This does NOT mean that you will be receiving any less data because on the current servers, this same scenario will result in the corrupted messages (unusable data). This is the reason that, assuming your app is keeping up with the datafeed, corrupted messages almost always occur only during fast markets and with latent connections. In fact, you will almost certainly receive MORE of your data simply because the first message will get transmitted correctly. Additionally, since the bandwidth is not being used for the corrupted messages, this will help latent connections receive more usable data.