parallelguy has contributed to 12 posts out of 21175 total posts
(0.06%) in 5,154 days (0.00 posts per day).
20 Most recent posts:
Something else just noticed: IQFeed seems to be saturating one processor worth of CPU time when receiving 900+KB/s of data (8 core machine, 4gb of ram). Memory usage does not significantly fluctuate. Could this also be a problem?
On Edit: To clarify, that is IQConnect.exe seems to saturate one core. Also, this is on 4.7.1.2 (beta). Edited by parallelguy on May 14, 2010 at 10:55 AM
IQFeed seems to be having issues sustaining communications above 930-1000 KB/sec and thus seems to be losing data during very high volume periods.
I'm on a 22/6mbit cable modem, and have verified that I can sustain 22mbit speeds on both tcp and udp. I am subscribed to 576 symbols, which comprises of the S&P 500, some misc etf's and stocks, and the major futures. Regional updates are enabled for all stocks.
I suspect this may be some sort of TCP window problem, as experimenting with TCP window settings on windows 2008 r2 does seem to measurably affect the peak rate I can get out of IQFeed. Unfortunately, none of them seem to perform better then the default settings. Any ideas what I might try next?
This example might better illustrate the point:
Quote: 20100328T211459.902159>Q,@ESM10,1163.25,0,10,1163.00,1163.50,114,40,16:14:59b,, ... 20100328T212336.109902>Q,@ESM10,1163.25,0,10,1163.00,1163.50,164,40,16:14:59b,,
To clarify my earlier comment: I had also been watching in real time in quote tracker, and it seems to be a systemic problem where quotes before the globex opening hours at 17:00 central on Sunday seem to pick a rather arbitrary time rather then give the correct time. Hope this is helpful.
I'm pretty sure I also found this or a related error. It seems to be related somehow to quotes before markets open. Once globex officially opened (Sunday night), the weird timestamps went away.
From my custom log (Time is Date Time in UTC that the packet was received from IQfeed)
Quote: 20100328T211459.902159>Q,@ESU10,1158.75,0,1,1147.00,1161.50,1,1,16:14:21b,, 20100328T211459.902159>Q,@ESM10,1163.25,0,10,1163.00,1163.50,114,40,16:14:59b,, 20100328T211459.902159>Q,@NQU10,1943.25,0,5,1428.00,1956.00,5,1,15:25:33b,, 20100328T211459.902159>Q,@NQM10,1950.75,0,5,1948.50,1952.00,2,1,16:14:59b,, 20100328T211459.902159>Q,@ADM10,0.8967,0,1,0.8955,0.8970,4,1,16:59:40b,, 20100328T211459.902159>Q,@BPM10,1.4893,0,1,1.4887,1.4901,11,1,16:57:00b,, 20100328T211459.902159>Q,@ADU10,0.8862,0,3,0,0.9150,0,5,16:43:20a,, 20100328T211459.902647>Q,@BPU10,1.4883,0,1,1.4110,1.4953,1,1,16:56:36b,, 20100328T211459.902647>Q,@CDU10,0.9734,0,1,0.9700,0.9760,6,2,16:46:12b,, 20100328T211459.902647>Q,@CDM10,0.9742,0,2,0.9735,0.9749,1,1,16:59:55b,, 20100328T211459.902647>Q,@EUM10,1.3409,0,3,1.3402,1.3415,41,1,16:59:48b,, 20100328T211459.902647>Q,@MEM10,1.3414,0,1,1.3387,1.3428,1,1,16:58:30b,, 20100328T211459.902647>Q,@JYM10,0.010811,0,1,0.010799,0.010825,3,3,16:59:54b,, 20100328T211459.902647>Q,@JEM10,0.010813,0,1,0.010680,0.010869,1,1,16:44:50b,, 20100328T211459.902647>Q,@RUM10,0.33480,0,1,0,0.34200,0,1,09:42:55a,, 20100328T211459.902647>Q,@SFM10,0.9392,0,1,0.9383,0.9400,2,5,16:59:00b,,
Speaking of which: I found (I believe) another weird oddity, but in 4.6.1.0.
Quote: 20100326T014619.822024>R,IAU,PSE,106.0700,4000000,17:13:01,107.5900,400,17:13:01,14,4, 20100326T014619.822024>R,IAU,NMS,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.822024>R,IAU,BSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.822024>R,IAU,CSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.822024>R,IAU,CHX,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.822024>R,IAU,NYSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.822024>R,IAU,PSE,106.0700,4000000,17:13:01,107.5900,400,17:13:01,14,4, 20100326T014619.823000>R,IAU,NMS,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.823000>R,IAU,BSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.823000>R,IAU,CSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.823000>R,IAU,CHX,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.823000>R,IAU,NYSE,0,0,17:13:01,0,0,17:13:01,14,4, 20100326T014619.823000>R,IAU,PSE,106.0700,4000000,17:13:01,107.5900,400,17:13:01,14,4,
While 4.6.1.0 behaves correctly the vast majority of the time, it appears that when there starts to be zero bidsz/asksz regional entries, it starts sending them even if no updates have occurred. It is unclear to me if this is the same bug reported earlier in the thread. This only seems to be an issue with some stocks too, not all of them.
Any idea when there might be a beta out with a fix for this?
--Thank you
I went ahead and downgraded back down to 4.6.1.0 last night just to be on the safe side, which has nicely taken care of the regional spam and the CPU issue.
Dual Quad Core 2.66ghz, 4gb ram, Windows Server 2008 R2, 6mbit sustained, 12mbit burstable downlink, 1mbit uplink. This usually is not anywhere close to saturated, except for bursts during open/closing. The data processing side of my custom C app has a thread dedicated to decode/dispatch (with the critical sections written mostly using pointer arithmetic) before sending it to other threads for processing.
Is there any way to tell if it is having to drop server data or to up the queue size? It seems to be saturating an entire core which concerns me a bit.
If IQFeed gets overloaded does it start dropping data or does it eventually catch up? (While my application can handle the data, do I need to worry about IQFeed getting saturated trying to generate that many regional quotes? Right now it is trying to push 6-7 MB/S peak over one connection).
I have not yet had a chance to downgrade, but the big giveaway that led me down this path was the amount of data sent from IQFeed to my application. Usually it is around ~10-20gb, with at most ~29gb on a very high volume day. Yesterday ( I upgraded over the weekend) it was ~77gb. The amount of total data sent over the internet connection was the same, however, at around ~3.7gb.
This is a log snippet. The time in front is the time the message was received by my application. It is almost like it is resending the entire regional table every time anything changes.
Quote: 20100217T145523.207862>R,INTC,PSE,20.8000,16200,09:55:10,20.8100,17300,09:55:10,14,4,5, 20100217T145523.207862>R,INTC,PHLX,20.6000,2000,09:55:10,20.9500,1000,09:55:10,14,4,8, 20100217T145523.207862>R,INTC,,20.8000,1700,09:55:10,20.8100,700,09:55:10,14,4,10, 20100217T145523.207862>R,INTC,,20.8000,11800,09:55:10,20.8100,6700,09:55:10,14,4,11, 20100217T145523.207862>R,INTC,,20.8000,2400,09:55:10,20.8100,1400,09:55:10,14,4,12, 20100217T145523.207862>R,INTC,,20.7900,200,09:55:10,20.8200,100,09:55:10,14,4,13, 20100217T145523.207862>R,INTC,,20.8000,100,09:55:10,20.8100,1600,09:55:10,14,4,15, 20100217T145523.207862>R,INTC,,20.8000,3900,09:55:10,20.8100,4300,09:55:10,14,4,16, 20100217T145523.207862>R,INTC,,20.8000,9400,09:55:10,20.8100,6000,09:55:10,14,4,18, 20100217T145523.207862>Q,INTC,20.8000,6413526,100,20.8000,20.8100,17400,17300,09:55:10b,t, 20100217T145523.207862>R,INTC,PSE,20.8000,17400,09:55:10,20.8100,17300,09:55:10,14,4,5, 20100217T145523.207862>R,INTC,PHLX,20.6000,2000,09:55:10,20.9500,1000,09:55:10,14,4,8, 20100217T145523.208350>R,INTC,,20.8000,1700,09:55:10,20.8100,700,09:55:10,14,4,10, 20100217T145523.208350>R,INTC,,20.8000,11800,09:55:10,20.8100,6700,09:55:10,14,4,11, 20100217T145523.208350>R,INTC,,20.8000,2400,09:55:10,20.8100,1400,09:55:10,14,4,12, 20100217T145523.208350>R,INTC,,20.7900,200,09:55:10,20.8200,100,09:55:10,14,4,13, 20100217T145523.208350>R,INTC,,20.8000,100,09:55:10,20.8100,1600,09:55:10,14,4,15, 20100217T145523.208350>R,INTC,,20.8000,3900,09:55:10,20.8100,4300,09:55:10,14,4,16, 20100217T145523.208350>R,INTC,,20.8000,9400,09:55:10,20.8100,6000,09:55:10,14,4,18, 20100217T145523.208350>Q,INTC,20.8000,6413526,100,20.8000,20.8100,17400,17200,09:55:10b,t, 20100217T145523.208350>R,INTC,PSE,20.8000,17400,09:55:10,20.8100,17200,09:55:10,14,4,5, 20100217T145523.208350>R,INTC,PHLX,20.6000,2000,09:55:10,20.9500,1000,09:55:10,14,4,8, 20100217T145523.208350>R,INTC,,20.8000,1700,09:55:10,20.8100,700,09:55:10,14,4,10, 20100217T145523.208350>R,INTC,,20.8000,11800,09:55:10,20.8100,6700,09:55:10,14,4,11, 20100217T145523.208350>R,INTC,,20.8000,2400,09:55:10,20.8100,1400,09:55:10,14,4,12, 20100217T145523.208350>R,INTC,,20.7900,200,09:55:10,20.8200,100,09:55:10,14,4,13, 20100217T145523.208350>R,INTC,,20.8000,100,09:55:10,20.8100,1600,09:55:10,14,4,15, 20100217T145523.208350>R,INTC,,20.8000,3900,09:55:10,20.8100,4300,09:55:10,14,4,16, 20100217T145523.208350>R,INTC,,20.8000,9400,09:55:10,20.8100,6000,09:55:10,14,4,18, 20100217T145523.208350>R,INTC,PSE,20.8000,17400,09:55:10,20.8100,17200,09:55:10,14,4,5, 20100217T145523.208350>R,INTC,PHLX,20.6000,2000,09:55:10,20.9500,1000,09:55:10,14,4,8, 20100217T145523.208350>R,INTC,,20.8000,1700,09:55:10,20.8100,700,09:55:10,14,4,10, 20100217T145523.208350>R,INTC,,20.8000,11800,09:55:10,20.8100,6700,09:55:10,14,4,11, 20100217T145523.208350>R,INTC,,20.8000,2400,09:55:10,20.8100,1400,09:55:10,14,4,12, 20100217T145523.208350>R,INTC,,20.7900,200,09:55:10,20.8200,100,09:55:10,14,4,13, 20100217T145523.208350>R,INTC,,20.8000,100,09:55:10,20.8100,1600,09:55:10,14,4,15, 20100217T145523.208350>R,INTC,,20.8000,3900,09:55:10,20.8100,4300,09:55:10,14,4,16, 20100217T145523.208350>R,INTC,,20.8000,9400,09:55:10,20.8100,5900,09:55:10,14,4,18,
Regional quotes are turned on for:
Quote: MMM ADBE ANF AES AET AMD A AFL ACS AKS APD ABT ARG AKAM AYE AA ALL AGN ATI AMZN MO AEP AEE AXP AIG ABC AMT AMP AMGN APH APC ADI APA ALTR AIV APOL AAPL AMAT ADM AIZ T ADSK ADP AN AZO AVB AVY AVP BHI BLL BAC BCR BAX BBT BDX BBBY BMS BBY BIG BJS BIIB BDK HRB BMC BXP BA BSX BNI BRCM BMY CA COG CPB COF CAH CFN CCL CBG CELG CBS CNP CEPH CHRW CF CHK CAM CVX CME CB CIEN CI CAT CINF CTAS CSCO CTL CTXS C CLX CMS COH KO CCE CTSH CL CMCSA CMA CSC CPWR CAG COP ED CNX STZ CEG CVG GLW COST CVH CSX CMI CVS DHR DRI DVA DF DE DELL DNR XRAY DVN DTV DO DFS D RRD DOV DOW DV DPS DTE DUK DNB DD DYN ETFC EMN EK ETN EBAY ECL EIX EP ERTS EMR EMC ESV EOG EQT EFX ETR EQR EL EXPE EXC EXPD ESRX XOM FDO FAST FII FDX FIS FITB FHN FE FISV FLIR FLS FMC FTI FLR FRX F FO FPL FCX FTR BEN GME GCI GPS GD GE GIS GNW GPC GENZ GILD GS GR GT GOOG GWW HAL HOG HAR HRS HIG HAS HCP HCN HNZ HSY HES HD HPQ DHI HON HRL HSP HST HCBK HUM HBAN ITW RX TEG ICE INTC IPG IBM IFF IGT IP INTU ISRG IRM JBL JEC ITT IVZ JNS JDSU JCI JNJ JPM JNPR KBH K KEY KMB KIM KG KLAC KSS KFT KR LLL LH LM LEG LEN LXK LUK LIFE LLY LTD LNC LLTC LMT L LOW LO LSI M MTB MRO MMC MI MAS MAR MEE MA MAT MFE MBI MKC MCD MHP MCK MWV MHS MDT WFR MRK MDP MET PCS MCHP MU MIL MSFT MOLX TAP MWW MCO MON MOT MUR MS MYL NBR NDAQ NOV NSM NTAP NYT NWL NEM NWSA NKE GAS NI NBL JWN NSC NU NTRS NOVL NOC NUE NVDA NVLS NYX ORLY OXY ODP OMC ORCL OI PCAR PTV PLL PH PDCO PAYX BTU JCP PBCT POM PBG PKI PEP PFE PCG PM PXD PNW PBI PNC RL PPL PPG PX PCP PCL PFG PG PGN PGR PLD PRU PEG PHM QLGC PSA QCOM DGX PWR STR Q RSH RRC RTN RHT RF RSG RHI RAI ROK COL RDC R SWY SNDK SLE CRM SCG SLB SCHW SHLD SEE SHW SIAL SRE SNI SLM SPG SJM SNA SII LUV SO SWN SE S STJ SWK SPLS SBUX HOT STT SRCL SYK SUN STI SVU SYY SYMC TROW TE TGT TLAB THC TSO TER TDC TXN TXT BK TMO TIF TWC TWX TIE TJX TMK TRV TSN TSS UNH UNP UPS USB UNM VFC UTX VAR VIA VLO VMC VNO VRSN VTR WAG VZ WAT WEC WDC WFMI WFC WHR WLP WIN WMB WM WPI WPO WMT WU WY WYN WYE WYNN X XEL XLNX XL XTO XRX YUM YHOO ZION ZMH ACV CERN CHD DEO DIS HBC HSIC JOYG MICC PBE PBR RIG SY V LVS MGM MRVL FSLR BKS SPWRA AONE DAL STP MT TM POT NFLX BKC DRYS XLF FAZ FAS XLE SDS SSO XLV XLB XLK XLI XLU QQQQ SPY VXX VXZ UNG USO DBO UGA UHN DDP SGG DGL SRS SLV GDX IAU GLD
I am pretty sure this is a result of 4.7.0.9, as since then the amount of data processed seems to have risen significantly (2-3x). I seem to be getting regional quotes for a market center, and then within a few regional quotes, the exact same regional quote again. As of now almost all of my IQFEED->Application communications are being consumed by regional updates that do not convey any new data, and I know it was not this bad before I upgraded.
Thanks
|