I experienced some problems with the feed yesterday that I haven't seen before. Can anyone shed some light on what happened?

I am tracking about 500 real time symbols and did not receive any time stamps 9:32 AM-9:37 AM (inclusive) (but I did receive symbol information updates).

Also, around 9:31 AM data was received a bit too late. The following is a dump of the TCP/IP feed I received (with my own time stamps at the beginning of each line). Note the 25 seconds delay.

2008-09-15 09:31:25.858: Q,CVH,D,35.34,500,35.16,35.30,100,300,09:30:03b,35.34,36.15,0.14,c,09/15/2008,
2008-09-15 09:31:25.858: Q,JNS,D,23.71,0,22.37,22.86,300,1000,17:27:27b,0,23.71,0.49,c,09/12/2008,
2008-09-15 09:31:25.858: T,20080915 09:31:00
2008-09-15 09:31:25.858: Q,PRU,D,76.23,3122,76.09,76.11,100,100,09:30:51b,76.80,80.59,0.02,t,09/15/2008,
2008-09-15 09:31:25.858: Q,NTRS,F,85.1100,23086,85.0100,85.0900,1000,500,09:30:58b,85.8900,87.0900,0.08,t,09/15/2008,

I'm subscribing to the following fields:
S,CURRENT UPDATE FIELDNAMES,Symbol,Exchange ID,Last,Total Volume,Bid,Ask,Bid Size,Ask Size,Last Trade Time,Open,Close,Spread,Market Center,Last Trade Date

There were no mentionable delays at neither 09:30 nor 09:38. The delay could probably be attributed to the heightened market activity of yesterday, but the lost timestamps worry me. Do you stop sending them, if the delay is too large?

I would appreciate assistance. Thanks in advance.

Hello jacobi, there were no known issues yesterday with the servers that would cause a delay. Additionally, the servers do not ever filter timestamps (or any other data).

Can you give me some more information about your operating environment that will help determine what happened? How many symbols were you watching at the time (and which ones)? I assume the information you are reporting above was gathered using your own app. Is this correct? Were you also running any other 3rd party apps (or even DTN.IQ) at the same time? What was the CPU utilization at the time of the delays/missing timestamps?

This sounds like an issue where your computer got "overloaded" (for lack of a better word) by the amount of data being delivered causing IQConnect (the portion of IQFeed that runs on your computer) to delay data to your app and possibly miss data comming from the server. However, I would prefer to get more information before making that conclusion.

Thanks for your prompt reply.

My system: Windows XP Pro/SP3. Pentium D 3.20 GHz (dual core), 3.25 GB RAM.

I was watching updates on all S&P 500 stocks using the IQFeed API via TCP/IP to localhost. The 500 symbol updates probably accounted for a good deal of net traffic.

You are correct that it is my own app (C#) that uses the IQFeed API. No other applications using the feed or other intensive calculation were running (3rd party or otherwise), no DTN.IQ.

I didn't record CPU utilization. The only intensive app running was the above-mentioned app that more or less just stored the TCP/IP stream from IQFeed directly as received in a flat text file. I did not get any socket time-outs or other exceptions from the system.

Is it possible for IQFeed's internal buffer to reach its limit, and what will happen if it does?

Sorry for the delay in replying. IQConnect's internal data buffer will queue data and resize indefinately until your app can catch up. This means the only limit to the size of the buffer is the size of your available RAM and/or Virtual Memory configured on the system. Chances are that if your app gets behind far enough that windows starts paging data to disk (virtual Memory) than it won't catch back up until the market closes or you stop watching as many symbols.

Have you noticed any issues since this occurance?

Hi,

We had a second similar occurrence of the problem about a week ago.

We now subscribe to two feeds that run identical systems on two different computers (one paper trader, one actual trader) . October 14th, several time stamps were missing from both feeds.

Feed P: 9:30, 9:32-9:44, 10:46-10:15, 10:17-10:22 etc.
Feed T: 9:31-9:37, 9:39-10:30, 10:32-10:37 etc.

To be able to debug, we save each line of text that is received from the IQFeed socket (along with our own local timestamp), and the above-mentioned time stamps from IQFeed are simply missing. Also, many of the given times of trade were severely out of sync with our local timestamp (time of saving the data to the log file). As you explained it previously, I cannot see how the missing time stamps can be attributed to our software since we save every line received, as mentioned, and if IQConnect's internal buffer queues infinitely as you explained nothing should be lost along the way, or maybe I'm missing something.

The size of the text file with a day worth of text lines from the socket usually takes up about 10 GB. That particular day, the text file was only 1.2 GB large. I conclude from this that several price updates were also missing from the feed. I am pretty sure that the page file was not in action (no other application are running), and furthermore, paging shouldn't leave holes in the feed as I see it, just a premature end, right?

Our systems have had several days with working feeds, nothing changed, and then the problem appeared one day, and was gone the next.

Do you have any idea of what is happening? We lost some pretty big opportunities and have now build in several alarms that will go off if it happens again. But obviously it would be best to actually get the timestamps and data which is not delayed.

The specs of the paper trader system is as explained above, the live system runs on Vista 64 with 4GB of RAM on an Intel Core2 Duo @2.66GHz.

I hope that you can shed some light on this. Let me know if you need additional information. Thanks.

jacobi, I mentioned in a previous post that IQConnect will queue data indefinately. This works out well for IQConnect since the socket communication exists entirely on your machine with a small number of client applications serviced. However, this design is not the case for the IQFeed servers. The servers are designed with the intention that one user cannot degrade service for other users connected to the same server. As a result, the servers will drop data if it is unable to deliver that data to the client (in this case, IQConnect is the client which is running on your computer).

I checked with the server team today after the market closed, and I noticed that one of your accounts (the only one that was currently connected) is certainly not keeping up with the feed. There is a good chance that this explains the missing timestamps (but it wouldn't explain the delayed data you previously reported). The only reasons this would happen is if your computer was overloaded to the point that IQConnect was unable to get enough CPU cycles to read from the socket, or if there were network issues between our servers and your computer that could explain it.

Theoreticly, the computer specs you posted should be plenty good enough to handle the data load you are requesting. You can easily monitor your computer CPU usage using task manager (or even programaticly through code in case this is a black-box type scenario).

Since you are running Vista, I would also keep an eye on the RAM usage of your computer overall. 4GB should be plenty and IQConnect should not be using more than around 20-30MB during normal operation but as I mentioned previously, if for whatever reason, the memory that IQCoonnect is using is getting paged to disk (Virtual Memory), it will slow down the processing of your data.

Assuming the CPU usage is not the problem, the only possibility left is the network. I would recomend running a bandwidth monitoring app (there are several good free ones available) for a few days so that you get an idea of what bandwidth requirements that your app has. You mentioned that the text file is around 10GB but the internet traffic for IQFeed should be much less and it is a good idea to get a baseline (if for no other reason than to be able to reference the data at a later date when you know you are having network problems). In addition to this, you will want to run a couple speed tests (www.speedtest.net) on your broadband line to make sure that your internet connection is performing up to the specs that your ISP has advertised. Choose a test location as close to your location as possible for testing your ISP then also choose the location listed in Norfolk, Nebraska (closest test site to our servers) and run a couple tests there for comparison. Lastly, run a ping test to the server that you connect to (the IP address is shown on the connection manager) and note the times that are reported.

If you have any additional questions about this, please post them here or if you would like to contact me via chat, please log into the developer website and click the chat link.
Edited by DTN_Steve_S on Oct 23, 2008 at 05:22 PM

I'm having similar problems. I've been debugging this for awhile, and the problem could still lie in my code. However, some data I captured today is very interesting. Here's the raw data I received via the socket interface at 15:48:41 EST:

Q,SPY,81.26,491049962,200,86.87,81.03,15:48:59b,85.91,87.08,524140602,
Q,QID,89.23,43977549,100,89.65,79.12,15:48:59b,81.29,81.51,52700844,
Q,QID,89.23,43977549,100,89.65,79.12,15:48:59b,81.29,81.51,52700844,
Q,QID,89.23,43977549,100,89.65,79.12,15:48:59b,81.29,81.51,52700844,
,30.96,31.16,8719081, <<< PROBLEM STARTS HERE
Q,SPY,81.28,490873000,300,86.87,81.03,15:48:55b,85.91,87.08,524140602, <<< OLDER QUOTE: 15:48:55b
Q,QID,89.21,43973491,100,89.65,79.12,15:48:55a,81.29,81.51,52700844,
Q,QID,89.21,43973491,100,89.65,79.12,15:48:55a,81.29,81.51,52700844,

Note that the QID quotes after the problem data are older than QID quotes earlier in the same socket buffer. This problem happens frequently during the day; it does seem to be related to spikes in volume. I'm only watching about 60 or 70 symbols.

My ISP is Verizon FIOS in Washington, DC. I ran tests at speedtest.net, and my results for the Norfolk, NE server are 5219kb/s down, 1452 kb/s up, ping 72ms. Pings of 66.112.156.221 take an average of 58ms. My computer has an AMD Athlon 4400 dual-core CPU and 2GB of RAM. I do have another program, Market Delta, using the IQFeed simultaneously. Whenever I use Market Delta's "Scrub Data" command, it says it corrected many "out-of-sequence" errors in the data.

Once my program starts seeing these data problems, I seem to get many buffers with bad and/or out-of-sequence data, and I usually have to restart my program. I'll say again that it could be a bug in my code, but on the other hand, the data above is the raw data my code received via the socket interface.

I could upgrade my internet connection to one of several higher speeds. But, if that's not going to fix this problem, all I'll end up doing is paying more while I still have the problem. Is there anything you guys can do on your side to help figure out whether this problem is caused by:
1. My internet connection
2. My computer not having enough CPU cycles to keep up
3. A bug in IQConnect
4. A problem on the DTN server(s)

Thanks.

Jim

Jim, can you offer a couple clarifications please? First, on the line:

,30.96,31.16,8719081, <<< PROBLEM STARTS HERE

are you saying that you actually received an incomplete message from the feed?

Second, when you receive quotes with an older timestamp, are they duplicates of messages already received or are they simply older than previous messages? You will ocassionally get quotes with "out of order" timestamps in the feed. This is generally due to the exchange sending them this way. However, if the first part of this report is true and you are actually getting corrupted messages from the feed, then pretty much "all bets are off" until the cause of that is determined.

Steve, I figured out this afternoon that the problem was indeed due to a bug in my code. One of those obscure threading bugs. I've fixed it and haven't seen the problem since. Thanks.

Jim