Is there a way of reducing the CPU usage of iqconnect.exe when subscribing to a decent size collection of symbols?

I'm subscribing to 500 fairly active symbols and I fairly often see the CPU usage spike to 25% (the maximum) even during the middle of the day. This is on a dual 2.4Ghz Xeon server system. By contrast, my application written in python is able to parse the text data and store it to a database using only about 12-15% cpu max. I'm rather worried about what will happen in a fast market. Even on my local machine, a Phenom 9550, the cpu usage gets close to maxxing out in the middle of the day.

I read elsewhere in the forums that all the Level1 socket connections use a single thread, and also that we can't run more than one instance of iqconnect.exe at a time. It feels sad that iqconnect.exe is saturating while there are three other processors on the machine that are not being utilized.

Is there any way to fix this? For example:
1) Can't I run two copies iqconnect.exe limiting each to 250 symbols?
2) Can't iqconnect.exe run each socket connection in a separate thread, allowing me to split the symbols between two or more socket connections?

What happens when iqconnect.exe maxs out the cpu for a period of time? Does it or the DTN servers start dropping ticks to keep up? Or does the feed just get buffered and delayed?

Is anybody out there running more than 500 symbols? How do you do it?

Thanks, -Russ

Russ, it is certainly possible to watch 500 active symbols with IQFeed. But it will certainly require fairly optimized processing code. Depending on where the bottleneck is in your processing, data can get queued in several places. If IQConnect (which runs on your machine) tries to send data to your app and is unable to do so (or if there is more data to send than your app or the socket buffers accept), leftover data gets queued until your app reads it off the socket. If the servers try to send data and are unable to do so, a small ammount of data will get queued but data will eventually get dropped if that queue fills and more data comes in to be sent. In short, short burts of maxed CPU usage will normally not result in dropped data but if the CPU gets maxed (or a single core that IQConnect is running on in a multi-core system) for an extended period of data, it is certainly a possibility.

If you are not already doing so, I definately recommend utilizing the Dynamic fieldsets feature of IQFeed to minimize the data that IQConnect has to process and send to your app. It will also minimize the amount of data that your app has to process comming from the feed (assuming there are fields that we send which your app ignores). In the next release (4.7 currently in beta), we have also modified the behavior of the Dynamic Fieldsets to reduce this excess data even further.


To answer your specific questions:

1) Can't I run two copies iqconnect.exe limiting each to 250 symbols?
- No, IQConnect will not allow a second instance to run.

2) Can't iqconnect.exe run each socket connection in a separate thread, allowing me to split the symbols between two or more socket connections?
- Right now, this doesn't happen. But it is certainly something we are looking into. The past few versions have had a focus on improving the efficiency of IQFeed and we will continue to do so in future versions.

3) Is anybody out there running more than 500 symbols?
- Yes. We have some customers watching 1800 symbols (the max plan we offer) without dropping any data. Unfortunately, I don't know any specifics.

Steve, thanks for the reply.

Regarding your first point, I don't see how optimizing my program will help reduce iqconnect.exe cpu usage. If my program isn't keeping up reading from the socket,
that'll just cause iqconnect.exe to block on writing (if the buffers are full). Write blocking
won't increase cpu usage. In any case, it's iqconnect.exe that's using all the cpu power,
not my program.

2nd point, I'm already using Dynamic fieldsets to reduce the quantity of data. It's possible I
might be able to pare it down by another one or two fields. I'll try that and see if it
helps.

I'm concerned not only with dropped symbols, but also with the latency. Having quotes
come in several seconds late because iqconnect.exe can't keep up is not good for trading.
Can you give me an example of a test you've run subscribing 500+ active symbols with low
cpu usage? I just don't see where the room for improvement is, since the multiprocessing
option has effectively been locked out.

Regards, -Russ

Russ, since IQConnect guarantees that any data it receives from the server will get delivered to your app (no data gets dropped once it reaches your machine), IQConnect queues any data not able to be delivered to your app and managing that queue certainly does does consume CPU cycles.

I'm not saying that this is necessarily the issue in your case (although it sounds like similar cases I've looked at before) and I am certainly willing to investigate it further with you. I was simply answering your questions.

Unfortunately, if you are already using dynamic fieldsets, trimming that down a couple more fields probably isn't going to help much. If it does, I would be interested in knowing which fields you removed.

Can you send me your symbol list that you are watching and give me a better idea of exactly what your app does? I would like to run your symbol list internally and knowing how your app is using the feed will be a huge help. Also, can you provide me with detailed the specs of the machine(s) you are using? You can post this information here if you are comfortable with it, or you can email developer support.

Thanks,
-Steve

Russ,

Here is an example watching a large number of symbols. This was on my laptop which is also running our ProphetX application, Microsoft Outlook and IE at the same time.

Specs on the laptop:
Windows XP Pro SP 3
Core 2 Duo 2 Ghz (T7250)
2 gb Ram.

See attached image for CPU utilization. I am watching 1300 symbols with IQFeed feeding into our DTN.IQ watch list application (no dynamic fieldsets).

---

Jay Froscheiser
DTN

Russ, it's my experience that when my program fell behind and IQConnect had to queue stuff up, that caused IQConnect to peg the CPU. When I rewrote my code so that it ran much faster, this problem went away. So it does seem to be true that slow client code can lead to a kind of death spiral for IQConnect. My code is now using dynamic fieldsets and running in its own thread, and it almost always keeps up. There are still spikes in volume (e.g., right after a Fed announcement) that cause my code to fall behind by a few seconds, but it catches up pretty quickly, and never causes IQConnect to peg the CPU.

Jim

Steve, I see what you're saying regarding management of the queue, although in
my experience things like buffering data take little cpu power compared to parsing
data feeds or maintaining level2 books.

I've reduced the number of fields using Dynamic Fieldsets to 8, using just
self.send2('S','SELECT UPDATE FIELDS',
'Incremental Volume','Bid','Ask','Bid Size','Ask Size','Last Trade Time',
'Market Center','Extended Trading Last')
Previously I had also included 'Total Volume' and 'Last'. Performance does seem better
than before, but that may just be due to a slow market. I'll know more by the market
close.

I've attached my symbol list in .csv format. It's a list of the futures
and stocks with the greatest dollar volume.

Basically my program just does this:
1) subscribe to Level1 quotes on a set of symbols
2) convert the data stream into my internal format, just a series of quote changes and executions
3) store the data by symbol in a set of chunks in memory (organized by a hash of the symbol)
4) when a chunk is full, write it to a SQL database

After hours, I go through and finish the sorting process, leaving me with a price database
organized by symbol.

Thanks for looking into this further for me. I look forward to seeing the results of your
tests.

Regards, -Russ

Looks like the symbol list didn't post. Here it is as a zipped .csv file.

Here's an example of what I'm seeing. It's a graph of iqconnect.exe cpu usage
versus time, made using Sysinternals Process Explorer (which I highly recommend).
As you can see, there are spikes in cpu usage, effectively maxxing out the power of
the machine. The graph on the left is my program (python.exe) archiving the data from
iqconnect.exe.

@Jay: Thanks for the screenshot and the example. I noticed you only show a
snapshot, not a graph. What's the maximum cpu usage it ever gets to?

@jimc: Thanks for relating your experiences. I'll write a version of my program
that just sucks the data down and throws it away, and see if that improves matters.

Here is a Process Explorer view of IQConnect. When I did the earlier screenshot, I had to wait for IQFeed to spike before snapping the screenshot. I didn't think you would believe that most of the time CPU is running at 0 to 1 %.

---

Jay Froscheiser
DTN

Sorry. Here is the screen shot.

---

Jay Froscheiser
DTN

Jay, that's very impressive. Now I have to figure out why I can't achieve
anything even close to that performance.

Here's another cpu usage graph, this time with a dummy version of my archiving
program, which basically does what jimc suggested. It is exactly the same, except
that it ignores all 'Q' type records. My program's cpu usage is now very low, but I'm still seeing high peaks on iqconnect.exe.

Thanks for all the suggestions, and I'd welcome more.

Ok, the plot thickens. After upgrading to the iqfeed beta version 4.7.0.2, I'm seeing dramatically different performance on two different machines. One machine, a quad core Athlon Phenom 9550, is showing perfomance similar to what Jay had. The second machine, a dual cpu 2.4 Ghz Xeon, is still showing quite poor performance.

Good:
Athlon Phenom x4 9550
Windows XP Pro SP2
4Gb ram

Bad:
Dual cpu Intel Xeon 2.4Ghz
Windows 2000 Server SP4
4gb ram


Previously, using IQFeed 4.6.1.0, the performance was only a little better on the Phenom. I'm pretty sure that the upgrade worked on both systems, because the connect dialog is displaying a prominent 4.7, and the "REQUEST STATS" command returns 4.7.0.2 on both.

Could iqconnect.exe perform better on Windows XP than on Windows 2000?
Could it have any processor specific enhancements for more recent cpus?
Any reason why the 4.7 beta version is so much better on the Phenom?

There aren't any processor specific enhancements inherently in the code. However, assuming that is an older Xeon processor (the assumption based on it being in a win2000 server), your Pheonom processor (and the hardware connected to it) is going to far outperform the Xeon processor (and the hardware connected to it).

http://www.cpubenchmark.net

I've got an older Xeon 3.2Ghz test machine here and it is only marginally better than our old 3.0ghz Pentium 4 machines. Of course, these older machines are not anywhere close to as powerful as our newer Core 2 Duo machines.

cpubenchmark.net is showing the following scores:
reported mymachine
Dual Xeon 2.8GHz 416 782
Phenom 9550 2486 2156

Yeah, at first glance the Phenom is much faster. But really that's only because
it is a quad core and Performance Test launches separate processes that use all the
cores. When I set the affinity of the pt.exe process such that it only executes on
processor #1, I get:
Dual Xeon 2.4Ghz (single processor) 296
Phenom 9550 (single processor) 576


So the real difference for a single threaded app like iqconnect.exe is about a factor of
two. What I'm seeing is that on the Phenom, the iqconnect.exe cpu usage (for 4.7 beta)
sits at zero and occasionally ticks up to perhaps 10%. On the Xeon 2.4Ghz, it runs at a
baseline level of around 5% and occasionally pins at 25%. The processing speed difference doesn't seem to be sufficient to explain how much faster the Phenom is.
Processor speed doesn't explain the dramatic improvement with the 4.7 beta version,
either.

As I mentioned above, we are making IQFeed more and more efficient with each release of the feed. Not only in IQConnect itself but also by making the feed easier to parse by client applications. If I had to venture a guess, I would say the improvement you are seeing in the 4.7 beta is related to the update in which IQConnect now only sends an update when an update is recieved for one of the fields that the client is watching when using dynamic fieldsets. As a result, I would guess that the improvement you see is related to receiving fewer actual messages to parse (which means that IQConnect didn't have to build or send that mesage either).

Sorry, what I meant to say above was that the 4.7 beta version increased performance
dramatically on the Phenom, but not on the Xeon. I don't see how the dynamic fieldsets
optimization that you referred to could explain that. Also I'm subscribing to bid/ask sizes,
which I'm guessing would be the largest part of the update bandwidth.

Thanks for all the advice and help. Getting it running smoothly on the Xeon is not looking
so hopeful at present, unfortunately. Please let me know if you have further suggestions.

As an aside, I'd like to encourage you to pursue the following enhancements for the future:
1) Allow data to be transmitted in binary form rather than ascii. In my experience, this
can save a large amount of cpu power by eliminating the need to transform the data back
and forth.
2) Have each socket connection running in its own thread. This allows load-splitting across
the different processors in a machine.