Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"I noticed that ******* quotes locked up shortly after the interest rate announcement yesterday while yours stayed stable." - Comment from Ron in Utah
"I am very happy I changed. I love the product, but more so I am thrilled with Tech Support. You are knowledgeable, polite, pleasant and professional." - Comment from Pat
"The service is great, I see a noticeable improvement in my volume profiles over [broker]'s data feed" - Comment from Larry
"After all the anxiety I had with my previous data provider it is a relief not to have to worry about data speed and integrity." - Comment from Eamonn
"I am enjoying the feed very much - so superior to the broker provided feed I was previously using." - Comment from George
"I was with ******* for 4 years at $230 a month, this is a huge savings for me, GOD BLESS YOU PEOPLE," - Comment from T.S. via Email
"Awesome response, as usual. It is a sincere and refreshing pleasure to do business with DTN, compared to your competition." - Comment from Ryan
"Version 4.0.0.2 has been working well for me and I appreciate that it is now a much tighter client to work with. I feel I can go to press with my own application and rely on a stable platform" - Comment from David in IA.
"I used to have *******, but they are way more money for the same thing. I have had no probs with data from DTN since switching over." - Comment from Public Forum Post
"You have an excellent feed. Very few spikes for Spot Forex." - Comment from Public Forum Post
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »Product Support »IQFeed Datafeed Wish List »How about sending the Time message once per second?
Author Topic: How about sending the Time message once per second? (5 messages, Page 1 of 1)

nsolot
-DTN Guru-
Posts: 273
Joined: Sep 4, 2004


Posted: Oct 7, 2004 08:28 AM          Msg. 1 of 5
That would provide a nice way of knowing the connection is up & running.

Seems to me that it would be minimal additional bandwidth usage. Currently my app monitors ETFs (QQQ, SPY, & DIA) as well as futures (NQ, ES, YM) and a number of indices. Often I receive well over 100 messages per second, but a few times a day, I receive zero messages in a one second time slice.

DTN_Nathan_Bartsch
-IQ Developer-
Posts: 33
Joined: May 3, 2004

Telvent DTN


Posted: Oct 7, 2004 11:20 AM          Msg. 2 of 5
Greetings,

I'm a developer currently working on the DTNIQ client and I've brought this sort of issue to the server team before. They convinced me that it wasn't a very good idea based on the following:

1. Timestamps from the server (if they included second resolution) would be 20 bytes. That means for every 1000 clients we have connected to the server system we are sending out 20k worth of data every second just in time stamps

~20k per 1000 clients for each second
~1.2meg per 1000 clients for each minute
~72meg per 1000 clients for each hour
~468meg per 1000 clients for a 6.5 hour session

This amount of bandwidth for a simple timestamp would be cost prohibitive. Since we have plans to include a second resolution in trade time at some point in the future, second resolution on the timestamps seems to be overkill (and expensive for the clients since this bandwidth cost would have to be passed on to subscribers)

2. If we only sent the time stamp if the client didn't receive any data for the previous second, we would experience the following problems:

a. We would have to track what each client was watching for the previous second.
b. We would have to track if we received / transmitted any fundamental data for that client.
c. We would have to track if any of the symbols they we watching updated in the previous second.
d. We would have to track if they added / removed watches for any symbols in the previous second.
e. We would have to track if they received a news headline or any other message (like a regional update)

After we had figured out that we hadn't sent a message of any type to this client we would then send a timestamp. Again, when you get into a numbers game, only this time it's server processing...

3. TCP/IP is a reliable protocol. It guarantees delivery, unless the connection is lost.. This means if there is a problem with the quote server connection, the client is already notified. So having a second timestamp just isn't that necessary. If you really want to keep a second resolution on the client, synchronize off of the minute timestamp that we send you. This will allow you to generate your own second resolution until we provide the trade time in second resolution.

Sorry if I've been a bit verbose, but I've been through this issue myself and I wanted to explain it as thoroughly as possible.

Thanks,

Nathan W. Bartsch - Senior Software Engineer - DTN Market Access

nsolot
-DTN Guru-
Posts: 273
Joined: Sep 4, 2004


Posted: Oct 7, 2004 11:58 AM          Msg. 3 of 5
Wow, thanks for complete reply.

Just checked my code, and it I currently watch 16 tickers. By my best guess, you send me about 200MB of data a day, and it seems like an additional 0.468MB shouldn't change the bandwidth metrics all that much.

But I'm more interested in in the lost connection issue. I asked this in another area and was pointed at the thread titled "How to fugure out current connection status ?" which indicates that it takes 75 seconds for lost connection to be reported, and that's way too long for me. Is that info out of date?

DTN_Tim_Russell
-IQ Server Developer-
Posts: 41
Joined: May 3, 2004

DTN Market Access, LLC.


Posted: Oct 7, 2004 12:28 PM          Msg. 4 of 5
Well, the 75 seconds is about the maximum it's supposed to take, based on the fact that the feed should get timestamps once a minute. A failure in getting the timestamps is usually a result of network failure between the customer and our servers, not necessarily a failure in the TCP connection.

I've tried to avoid too much detail in this area up to now, but it looks like I'm not going to be able to do that this time, so, here goes

1. Detecting a connection loss because your internet connection to your computer is disconnected - this is very simple and, if IQFeed doesn't currently have this functionality I'm sure it will in the future.

2. Detecting a connection loss caused by software failure on our end is also easily detected, as the operating system will handle closing out the sockets, causing IQFeed to do whatever it is it does to notify clients and attempt to reconnect to the server way before 75 seconds has past.

3. Now, the most difficult. When a route between an IQFeed instance and our servers goes down temporarily - or becomes congested, or whatever - data may temporarily be unable to be transmitted between client and server. This is the nature of TCP, and it's designed to take these things into account, waiting a set (sometimes very long) amount of time before declaring that the connection is no longer valid. IQFeed attempts to get around this by saying, "After 75 seconds, if I still haven't received a timestamp, attempt a reconnect".

The need to wait a certain amount of time before giving up and attempting to reconnect is necessary, especially if the connection failed due to hardware problems somewhere along the line - attempts to reconnect every second are very likely going to be unsuccessful until the link is no longer congested, or the down hardware is back up, etc.

Also, having arbitrarily short timeouts tends to cause a bunch of false negatives - i.e., "No data received for 1.5 seconds, attempt to reconnect!". You can see where this would be a problem.

Let me recap a bit here:
If a connection fails (server kicks you off, you unplug your cable, software crashes), the remote TCP end should be notified immediately.

When a socket error doesn't occur, it's a very subjective thing to determine whether the TCP connection has 'failed' or is just stalled.

In a nutshell, the 75-seconds is for 'stalled' connections, not necessarily lost connections.

Tim Russell
Software Engineer
DTN IQ & FinWin

nsolot
-DTN Guru-
Posts: 273
Joined: Sep 4, 2004


Posted: Oct 8, 2004 07:21 AM          Msg. 5 of 5
Let me run this up the flagpole. How about an IQ specific ticker that would send the time (HH:MM:SS) without the date once per second? This way only clients who want this would receive it, rather than the entire client base.

or an alternative, pick some obscure index, and configure it to send an update once per second.

Looking at yesterday (Oct 7) data playback, I see a number of instances (~10) where I received zero messages from IQ, but was receiving messages from my second feed (IB). I'm not so worried about this, since they are a single second interval.

Then something "funky" happened which resolved itself at 2:02:06 EST, which has me scratching my head. For the 6 seconds prior, I received zero messages from IQ, and for the 9 seconds prior I receive zero messages from IB; there was 6 seconds of overlap where I receive zero messages from both feeds. Upon resolution, I received 98 messages from IQ and 120 messages from IB.

Ultimately, I trying to discern between instances where I get a zero message count simply because IQ had no data to send me vs. those where some part of the technology had a "hiccup".
 

 

Time: Mon May 16, 2022 12:53 AM CFBB v1.2.0 31 ms.
© AderSoftware 2002-2003