I have been hesitant to report the following problem for quite some time, since I assumed it was the fault of my own programming. But in the past few weeks, I have added copious logging to my own application, have attempted to debug it scrupulously, and now I am coming to the conclusion that perhaps the problem lies with IQFeed, not my own app.

A description: my app uses the TCP/IP (sockets) interface to IQFeed. The app runs great for some period of time. Then, at some interval which may be a few hours, or may be ten minutes, it stops working great.

Specifically, what happens is that while my app continues to receive Q, P, and F update lines perfectly well, IQFeed apparently stops listening to watch requests. I know this because symbols which are perfectly good simply don't appear in the message stream, even after multiple attempts to "watch" them. Once this condition starts, it's terminal ... no matter which symbols I attempt to add to my watch list, nothing gets added. The only way to deal with this is to shut everything down and restart it.

To address the first question DTN will ask: I am not bumping up against the symbol limit in my account. I confirm this both via my own internal application's counting scheme, and through the Connection Manager Feed Stats.

I wonder if anyone else out there is using sockets programming w/ IQFeed and facing a similar issue. Has DTN heard of anything similar? If neither answer is yes, then it may very well be a problem in my code, but if so - I can't fathom what it can be - after all, I can sit and watch the incoming message stream continue to roll into my app from IQFeed... but no new symbols get added.

Can anyone help?

Matthew

---

Matthew Klein

Matthew,

I haven't heard this issue before, and you answer the first question I would have asked

The next question I would have is when you place the watch command, do you see the connection manager increase the symbol count?

Also, maybe open a telnet session to port 5009, enter S,KEY then enter your watch command and see if the other socket is able to watch the symbol you are having issues with.

Can you give any more details as to timing, how many symbols you're watching, etc that may help. Are you churning through a large number of symbols throughout the day?

I know Natalie has identified an issue with numerous history requests, but I am not aware of anything on watching level I data.

---

Jay Froscheiser
DTN - Trading Markets

Hmm. For what it is worth, I actually do also make numerous history requests, but for minute data, not daily data. I remember reading in these forums that the problem Natalie has mentioned relates to daily history requests, right?

Let me ask this: is it possible these are both the same problem? Is one of the symptoms of the "too many history requests" problem that IQFeed stops listening to socket 'watch' requests, but it still continues to deliver updated quotes via the socket for items already in the watch list?

I would consider this great news, because it would mean the problem I am reporting is a problem that has already been reported, and has a known (or soon-to-be-known) cause, and is something which IQFeed is already presumably working on.

Matthew

---

Matthew Klein

Jay:

OK. I did some more investigation and have the following to report.

I set an alarm to sound when the socket problem occurs. (That is, when IQFeed will not respond to any more 'watch' requests, but is still delivering update messages for symbols which have already been watched on already-open socket).

When I heard the alarm this morning, I followed Jay's instructions. I telneted into port 5009, and then manually issued 'watch' commands. Lo and behold, the new socket worked fine. New symbols could be watched an unwatched.

Jay and Natalie: any ideas what this means? Any other ways I can help you investigate this?

Finally, I wonder: could I possibly be the *only* person to have this problem? If so, does this indicate that I am a lousy programmer, and I am doing something wrong; or is it merely an indication that I am one of only a few who uses the TCP/IP interface?

Matthew

---

Matthew Klein

As much as I would like to say this issue is because you are a lousy programmer , I always take the tact that we need to troubleshoot this together. If I were to blow you off and tell you its your problem, we may miss finding a bug somewhere in our code that only you have been able to identify so far. At this time we have had no other reports of this kind of problem. That doesn't mean nobody has seen it, it just means nobody has reported it. One thing I can assure you of is that we have hundreds of developers using Sockets. Only a very small number of developers are using the COM interface for IQFeed. I would guess this is more of an issue of what you are doing is different than what others do, and that is why nobody else has the issue.

The fact that you opened the socket manually and were still able to watch symbols tells us that IQFeed itself is still responding to watch commands, so we can assume its not an issue with the core IQFeed "not working". However, there may be something in the socket that you are using that is getting corrupt and thus isn't seeing the watch commands any longer.

Can you give some more specifics on how many watches/unwatches you issue and for what kind of symbols, before you see th issue occur. Does it happen during market hours or only afterhours? Again, more detail the better.

---

Jay Froscheiser
DTN - Trading Markets

Hi Matthew,

Thanks for further investigating the issue for us.

The history issue was present on all request types...not just daily. That issue would completely lock IQFeed up (if I remember correctly), so I don't think that is your issue. I believe that we have quite a few people developing to the socket interface, but that doesn't really mean anything. You could be exercising IQFeed in a different way than others do.

So, you could really help me out by giving me specific steps to recreate the problem. If I can see it happening, then I can fix it. If you have a test app that recreates the problem, you can send it to the developer support email, and I can run it to reproduce the issue. Other wise, you can just tell me the steps that you are doing, and I can write my own test app.

Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

I beginning to suspect perhaps that the problem is caused by the unusual way I use IQFeed.

First, some background. My own app is used for internal trading and trade-tracking purposes, so it doesn't need to keep a lot of symbols on the watch list at any one time. Another way to say this is that I have no need to do a lot of "scanning" of the stock universe, looking for various conditions among a large universe of symbols. Instead, I simply want to be able to pull up an up-to-date quote for a stock/future I am interested in trading. When I have looked at the quote and fundamental data, my app is generally "done" with the symbol, and doesn't need it any longer.

I tell you this to explain why I use IQFeed the way I do. I have read elsewhere on this forum that IQFeed frowns upon users doing a lot of "watching" and then "unwatching" of symbols, since each new request requires more bandwidth than a simple update message.

So when I first designed my app, I tried to be a good software citizen. If I needed a quote, I would start to watch it, pull up the quote, then process it. But I would leave the symbol in my watchlist and so continue to receive update messages for it.

But I noticed some distrubing things. First, the more quotes I had in my watchlist, the longer the latency of getting a new quote for a brand-new symbol. What's more, when I studied the IQFeed Connection Manager, I saw that the Internet Bandwidth used by my app grew quite substantially as the number of symbols in the watchlist grew. (I suppose this is obvious.) There reached a magic point (60, 70, 80 symbols, I think) where the latency for requesting a new quote on a new symbol grew too oppressive ... I suppose because my bandwidth / DTN's CPU cycles were occupied by delivering update messages for symbols I didn't really need.

So I began to experiment with an alternative software design. I would keep the number of symbols in my watch list quite low (say, 5). Thus I would ask for 5 symbols, but when the sixth was needed, I would unwatch the first one. Since I don't need to keep quotes in memory, and since I need symbols only periodically and don't need to watch them over time, this was fine with me. In fact, I believe this set-up was better from DTN's point of view, since my Internet Bandwidth usage was kept quite low (.75 K/sec during the trading day, even during periods where symbols get updated frequently).

The result of this design is: (1) I don't use a lot of DTN bandwidth (so don't get mad at me), and (2) I don't suffer latency when I need a new quote on a new symbol. In general, I can ask for a quote on a symbol, by watching it, and have the update and fundamental messages arrive almost instantly.

However, the implication of this design is that I do indeed do a lot of watching and unwatching of symbols. This might be the reason (I hypothesize) that I am one of the few who has reported a problem in which an attempt to watch a new symbol finally just fails, even though the IQFeed core is still running fine.

So, if you want to emulate my software, I would suggest you simply write a small app which will keep 5 random symbols in a watchlist, and -- when you want to watch a sixth, will unwatch symbol number 1 first. Just have the app ask for a new sixth symbol repeatedly. After some random amount of time (sometimes hours, alas!) the next attempt to watch a symbol is simply ignored, even if the watch request is repeated.

Matthew

---

Matthew Klein

Hi Matthew,

Thanks for the explanation. I will attempt to recreate the issue, and will let you know what I find. Is you app doing anything else with IQFeed when you are able to recreate the issue? News, History, etc. or are you able to reproduce it with just the Level 1 socket requests? Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Natalie:

My app will also request recent minute-bar history for symbols that are "of interest." No other IQFeed functionality is used -- no news, no optionchains, etc.

Regarding the history requests: I will sometimes get a failure when requesting history, but I believe this is caused by the too-many-history-requests bug, which is different than the problem I am reporting.

To repeat: the symptom of the problem I am reporting is that new "watch" requests are ignored, and the requested symbol never gets added to the message stream.

Matthew

---

Matthew Klein

Hi Matthew,

I haven't been able to reproduce the problem. How many symbols are you watching when this issue occurs? Have you seen the issue occur with various amounts of symbols? Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.

Natalie:

I'm adding some more extensive logging to my app, so I will be able to send you something that is intelligable to you, and helpful. As soon as I have done this (and as soon as the problem occurs again), I will email you the log. (Please let me know the email address where I should send it.)

To answer your question, though, I'm watching exactly 5 symbols when the problem occurs. However, it *seems* (need to confirm this) to happen when those five symbols are very heavily traded, during peak hours, and thus I get a very fast stream of update messages (so, the watch list might include: @YMZ5, @ESZ5, @ER2Z5, etc. during peak trading hours). What seems to happen is that my wSYMBOL requests are ignored by IQFeed when this fast and furious stream of update messages is being shoved into the socket.

Again, I'll try to capture this in a useful log for you. Hopefully I'll have something for you in the next day or two.

Matthew

---

Matthew Klein

Are you using threads? (you should be)

And if so are they synched? (they have to be)

IQ will hog a single threaded application message que when the data is flying.

Sounds like you may need to set up a blocking to squeeze your outbound message in and through.

Just a thought..

Scott

Sorry, I should have posted that Matthew and I came to a resolution via an IQFeed test build. I will get the build to QA this week. Thanks!

Natalie

---

Natalie Hannan DTN Market Access, LLC.