My goal is to record into a database prices of an equity (QQQ) and its options by the minute.
And I'm having problems getting results that look like what I'm observing on Yahoo or PCQuote.

I cloned the console_stream C++ example. The functions memset() and recv() are called in a loop during trading hours. The string returned by recv() is parsed, and any Fundamental ("P") or Update ("Q") for QQQ is parsed for "Last" and the result is stored in a double. Then once a minute (or so), the current stored value of QQQ stored is entered into a database. When I look at the sequence at the end of a trading day, I'm finding that the recorded values aren't really matching what I've observed at Yahoo or PCQuote. In particular a significant amount of the volatility is lost. The prices in the database are frequently constant over five to ten minutes and that doesn't appear to be right. There should be more variation in the QQQ than that. My measure of volatility is the most important part of my analysis, and the results I'm getting show a volatility that is far to low. It appears to me that the Update isn't updating as frequently as it should. Is there some better way to do this? Should I be recording the "asked" rather than the "last"?

The last would be giving you the last trade price (of course), so depending upon what you are looking for, that may be the field you want to capture. It is highly possible that on QQQ the last price would be constant over a 10 minute period, although in between that 10 minute period, the price may have fluctuated up or down a few points. I would suggest pulling the 1 minute history for the day and comparing that to what you are capturing. The key is that your recording of the data is at the same "moment in time" as that of our history server or that of PcQuote/Yahoo.

---

Jay Froscheiser
DTN Market Access, LLC.

When I said constant over a 10 minute period, I mean that it was the same value for each of the minutes over the 10 minute period. It would be quite extraordinary for the fluctuations in the QQQ to have hit the same value for each of the minutes over the ten minutes. I'll try pulling the one minute history but comparison with the Yahoo intra day plots clearly show a discrepancy.

I had my program print out the buffer from the call to recv() for the first hour of the trading day and I found out what the problem might be. The Update string prints a time (field 18), and at 10:30 (an hour after trading has begun) the Update string was returning a time of 9:31. In other words, the datafeed was nearly an hour behind. The QQQ is a very highly traded equity and I suspect that the datafeed isn't able to keep up with the trading. It is trying to generate an update on every trade and there are hundreds of them in the first minutes of trading. I'm sure it's not as bad at other times of the day, but I suspect that the problems I'm having are due to the failure of the Update to keep up with the trading. This is a serious problem for my analysis. I absolutely require timely information on the QQQ.

I can assure you there isn't a delay of 1 hour in the feed (or we wouldn't have any customers). The feed should send all times in ET, so it may be off compared to your timezone. We did have some issues with time and daylight savings in the past, so maybe there is still an issue. I would suggest taking a look at IQconnect explorer at the same time running your app. This will allow you to see the updates coming through at the time you record the trade. Since QQQ is so active, there should be a trade within a second or so of every time you print your minute record.

---

Jay Froscheiser
DTN Market Access, LLC.

As I described in my post, I printed all of the updates for QQQ and QQQ options for one hour starting with the openning at 9:30 (which is when the markets open on the East Coast). The file is about one megabyte of updates. All of the updates in that file are time stamped between 9:30 and 9:31. There are hundreds (maybe thousands) of updates in that file,
and I'll admit that some of the delay is in printing to the file, but even so it should be further ahead than 9:31.

Maybe part of the problem is that I have this large watch list, QQQ, IRX.XO (interest rate for Black and Scholes), and about 600 or so QQQ option contract symbols. I might be able to get timely data on the QQQ if that was the only symbol on my watch list, but I have to have the option data as well.

I can assure you there's a problem, and what I'm seeking is help. Has anybody else implemented the kind of watch list I have? How do they do it? The only thing I can think of is to log off the feed and then log back on in 60 seconds. As you state the QQQ is so active I don't need to watch more than a few seconds to get a timely update. The only problem with that is that I might have problems getting timely option data because some of the contracts are not as active as the underlying. So I'd rather watch continuously and store into a database at a selected interval. But I can assure you the continous updating is not working.

Please forgive me as I am just trying to understand the problem. Is the issue that the timestamp isn't updating, or the data isn't updating? The reason for my previous post is that we obviously can't have delays of seconds, let alone minutes, or an hour, or we wouldn't have any customers. So I don't think there is a "delay" in the feed. Now, there may be an issue which is causing the quote message not to fire in the method you are requesting data. Or, it could simply be the timestamp not updating. That is what we need to get to the bottom of. This is why I suggest using IQconnect explorer. Or, since you are watching over 500 symbols, I assume you are subscribed to DTN.IQ. If this is the case, use the DTN.IQ client and watch the same symbols in T&S or snap quote. If you use T&S, the nice thing is you can watch 1 minute intervals and compare this to what you are seeing in your app. This would at least help narrow down where the problem may be. Once we narrow things down, we should be able to pinpoint the problem.

---

Jay Froscheiser
DTN Market Access, LLC.

This problem isn't the timestamp, nor is it that the data isn't updating. The reason, as I've tried to explain, is that the update messages can't seem to keep up with the sheer volume of transactions in the QQQ. A half hour after the market has started, the update messages are still delivering information on trades less than a minute after the market has started. The timestamps are correct and the update messages are being sent. The problem is that there are hundreds, maybe thousands of transactions in QQQ in the first few minutes of the trading day, and the updating messages take an hour to catch up.

It also appears that the update messaging gets a little bogged down throughout the day, that is, the messaging can't keep up with the large number of transactions. The result is that I can't get timely quotes on the QQQ. I can go to a lot of effort to prove this, but I can assure you that it is happening. And I believe the solution is going to be finding out a way to get the quotes, that is, I'm doing something wrong.

All I want is the QQQ option chain. It just doesn't seem to me that that shouldn't be so hard to do. Along with the option chain I have to have the value of the underlying, i.e., the QQQ, and the risk-free interest rate, the IRX.XO. And I need them on some interval, one minute would be nice.

Right now I download the option chain symbols from the data feed, and construct a watch list including those symbols and the QQQ and the IRX.X0. Then in a loop I call recv() continously throughout the trading day parsing the update messages and filling an array for the option data, and two doubles for the QQQ and IRX.X0 quotes. Then every minute the contents of the array and two doubles are uploaded to a database. The problem occurs when the update messaging gets bogged down most likely due to a large quantity of transactions resulting in a large number of update messages which the data feed has trouble keeping up with, and as a result the QQQ quote doesn't get updated in a timely fashion.

I tried setting this up different. I tried shutting down and restarting the data feed after
the one minute updating of the database, but I haven't been able to get this to work.

ronsch99,

Being a server developer, I'm not especially familiar with the ins and outs of IQFeed - but part of our testing on the internet stuff in the past has involved loading it up with the highest-activity symbols and seeing how well it kept up. Given sufficient bandwidth to the client machine, I can't see any reason why IQFeed wouldn't be able to handle this symbol load.

Now, when you throw the database updates into the mix, I'm not so sure since I've no idea how your software is structured. Is it possible that your database updates are done in-line with the quote processing, and taking too much time to complete?

I'm not familiar with the console_stream example app you mentioned - does it send a lot of output to the console? Console output is also a notorious CPU sink.

Maybe one of the other developers more familiar with IQFeed will have some input on this.

---

Tim Russell
Software Engineer
DTN IQ & FinWin

I fiddled with the program and got it down to 350 or so symbols -- I really only want three months of option chains -- but it still is bogged down. The printing to a file isn't slowing things because I had it just print the QQQ quote that it stored into the data base which happens only every minute, and the quotes still showed the stream bogged down. Nothing is getting printed to the console, if anything is printed it is to a file.

I am now wondering if my routine for parsing the update messages is the problem. I'm not a sophisticated C programmer and it could be inefficient. Another thing is that I really don't need to parse every update message. The datafeed generates hundreds of update messages in a minute and I really only need to parse some of them.