|
|
Home
Search
Register
Login
Recent Posts
| Viewing User Profile for: DTN_Tim_Russell | |||||||||||||||||||||||||
| About | Contact | ||||||||||||||||||||||||
|
|
||||||||||||||||||||||||
| Post Statistics | |||||||||||||||||||||||||
|
DTN_Tim_Russell has contributed to 41 posts out of 21120 total posts
(0.19%) in 7,162 days (0.01 posts per day).
20 Most recent posts:
Data and Content Support »
Time stamps on regional quotes May 11, 2005 08:37 AM
(Total replies: 4)
I'm looking into this. AFAIK, nothing should be sent in PST - if it is, we'll get it fixed soonest. I'll post more when I know more. Tim Russell Software Engineer DTN IQ & FinWin
Data and Content Support »
Numtrades for globex products is fubar Apr 1, 2005 09:44 AM
(Total replies: 8)
We're looking into this problem and should have a solution for this problem soon. I'll post more when I know more about what's actually causing this to not get reset any more. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
What does ".LNCR" mean, where I expect a number in the "Q" message on port 5009 ? Jan 27, 2005 09:57 PM
(Total replies: 10)
Dierk, We agree 100% on this one - data should never get corrupted. It's one of the bigger fixes I've got planned for going to QA testing in about 2 weeks. No promises yet, but that's the plan. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
Getting Regional quotes Dec 3, 2004 09:54 AM
(Total replies: 2)
Shouldn't those sizeof()'s be strlen()'s? Just wondering, not sure how IQFeed will respond to all those extraneous nulls. Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 30, 2004 12:10 AM
(Total replies: 24)
With the satellite feed, this isn't a problem. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 21, 2004 09:26 AM
(Total replies: 16)
The trade counters were originally for internal use, and used by our FinWin product - since it's a web service, it doesn't do streaming data, and this information is very useful to those customers. The reason we made this available in the streaming feed was because there seemed to be a bit of interest in it. However, it is NOT designed to be a method of letting you know data has been dropped in the feed. As for the corrupt messages - let me know what you see. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 21, 2004 08:32 AM
(Total replies: 16)
Ah - trades today. Okay. That value's maintained by the server - it's returned to you when you request it. However, it's not transmitted on a tick-by-tick basis, since it's easily calculable on the remote end. Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 20, 2004 02:08 PM
(Total replies: 16)
Quote: Tim, Some of your comments concern me. I am monitoring the bandwidth between your servers and my PC and it is very high. After the initial burst (625KBytes/sec) I am getting around 300KBytes/sec sustained for the first hour. Last week I had an almost identical watch list and was receiving around 1/8th of this bandwidth which is closer to the figures you mentioned. The ratio of data downloaded to that output by IQFeed is around 1.2:1 which is much worse than last week. Clearly something is wrong. Even now (10:38am your time) I am getting ~180KB/sec from your servers. Is there any way you can determine what is being sent to me? (ID=200643) Dennis PS I'm still getting the "U,F|" Dennis - I've reposted your email here so that everybody can benefit. I was under a slight delusion as to how IQFeed handled outgoing data - basically I looked at code that has NOT been completed yet, that fixed a problem that is most likely causing your symptoms. Basically (and I hope you understand the basics of TCP/IP communication): - IQFeed sets TCP send / recv buffers to 256 Kbytes. - When incoming data is processed, a message is created for the client. - This message is sent to the client. No checking is (currently) being done to ensure that this message is sent in its entirety. This is fine so long as the 256k TCP buffer doesn't get full. - After the TCP buffer gets full, attempts to write to the socket will fail. - You will get corrupted data / lose data. The obvious problem here is that your code may not be processing messages from IQFeed fast enough. A couple of solutions: - Make your TCP recv buffer bigger. In java.net.Socket, I believe this is accomplished via the setReceiveBufferSize() method. - Read data from the incoming socket in one thread, and pass this data to another thread - via a queue or whatever - to be processed. Sorry for failing to notice this earlier. The recv buffer size fix is often offered, but I wasn't aware of why until I got one of our client guys to help me find the problem in the code. If this helps you, let me know. Now, for the amount of data you're seeing - are you getting this via the IQConnectionManager? If not, where are you getting these values? Only curious, as what I'm seeing going to your computer doesn't account for that much data. Thanks, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 20, 2004 09:15 AM
(Total replies: 16)
- 7MB sustained should be more than enough - at 9am EST, I'm seeing output at just under .2Mb on your watchlist. - Nice machine =) As I'm sure you guessed, no problems there. - I'm concerned about this "missing counts" you're talking about - I'm not familiar with that - where are you getting it? (Remember, I'm a server guy, not real familiar with IQFeed's output) Btw, as far as compression goes: I'm seeing 21KBytes/second on our server output, vs 160Kbytes/second (again, with your watchlist). I'm actually suprised about the "U,F" you're seeing in some fields - this is something you should NEVER see - it's part of our server feed before it's converted to the text feed that IQFeed puts out. I'll pass this on to the IQFeed guys to look at. Could be this message corruption is the root of your problem, though. Thanks for the input, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 19, 2004 10:28 AM
(Total replies: 16)
Dennis, Looking at our stored historical data vs what you sent, I see that those trades received were regular trades. Moreover, the volume on these trades (which is part of the historcal retrieval) is higher than the volume you have. The obvious reason for this is that data is being dropped passing between our servers and you. The "why" of the dropped data is what we need to pinpoint now; I'll need a bit more information from you. - What type of connection are you using to the Internet? - What type of machine is your application running on? - If possible, can you send me a private message with the list of symbols you're watching? This will give me a better idea of the bandwidth involved. Now, some background on why these things can impact your data reception. Our quote servers, after upgrades last month, have proven to be more than capable of keeping up with our data feed and transmitting it to customers in a timely manner. However, memory on these machines is not unlimited - we cannot queue data on our end waiting for the remote clients to be ready to receive it. Doing this would put our servers at the mercy of any client who wanted to break our servers just by not receiving data for an extended period of time. Timely (and complete) delivery of data is entirely dependant on the remote end's ability to read the data fast enough. While I'm not certain what symbols you are watching, with 1300 symbols being watched, I'm pretty sure this is generating a significant amount of data. Get back to me with your information, and we'll try to work from there. Thanx, Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Developer Support »
historical tick data download doesn't match real time ticks Oct 19, 2004 08:34 AM
(Total replies: 16)
Dennis, If you could give me an example of a symbol you see this happening on, I can look into this. Thanks, Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 13, 2004 12:19 PM
(Total replies: 24)
Quote: Posted By: kxrdart on Today @ 11:28 AM Message: Hi Tim, I guess my requirements would be to download todays 1 or 5 minute bars on 2000 symbols in a time of less then 3 minutes. Is this realistic and currently doable ? Since I already have the previous days history local I just need a total bar snapshot for the day. Thx I really think that 2000 symbols in under 3 minutes is a bit of a stretch at this point in time. Consider even an average response time of .25 seconds - which is pretty good. .25 * 2000 / 60 = ~8.3 minutes. Realistically, I'd say double that and that would probably be more accurate. Our history system wasn't ever really designed with this kind of usage in mind (which is okay, and is why we're asking for your input). If you have a big requirement for this, though, I'd definitely suggest the satellite feed as an attractive alternative. Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 13, 2004 10:04 AM
(Total replies: 24)
jf - No need to repost them to another poll, as I'm tracking both of them. - Again, cycling through symbols after market isn't that big a deal (with the same caveats I stated before). - Number of trades, in my opinion, really belongs in the market data. If it was valuable enough to people for charting purposes, I think we could store it in end-of-day data in the future (this would be a way out). - A throttle is definitely going to happen. With that said, I seriously doubt that anybody is going to notice a significant impact. We'll test it thoroughly - and probably give 3rd party folks a chance to test it, also - before it goes live. - The symbol/token thing I think is best implemented in IQFeed, vs the historical servers (due to the nature of the IQFeed / history server communication). - We'll have to look at an end of day file. I think having individual tick and minute files that could be pulled wouldn't lessen the load / bandwidth on our servers, and I don't think it really gives us something that requesting the data through the history server doesn't already gives us. I'm always open to convincing, though. As always, thanks for the feedback. Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 12, 2004 02:05 PM
(Total replies: 24)
I'm pretty sure you can already do this via history lookup. If you're looking for something different, could you go into some more detail? Thanks! Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 11, 2004 08:16 AM
(Total replies: 24)
Lajos, I'm definitely not saying that all of the scanning that's going on is abusive. The real goal isn't to lay blame or point fingers, but to come up with solutions that will make the feed better, both from your point of view and ours. The snapshot functionality is definitely going in the next release of our quote servers / databases, so that should make life a little better for everybody who needs it. The customizable scanning will most likely require a bit development time on our part, and I don't know where Jay stands on that one. Thanks for the post! Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 8, 2004 11:01 PM
(Total replies: 24)
Allen - For the most part - and within reason, with a dash of common sense applied, and is within the scope of whatever agreements / contracts you have with DTN - we don't care too much what's done after market with regards to data retrieval as long as it's not abusive and detrimental to other customers. I haven't seen any application behavior to date that hits the system that hard after market hours. Our big concern here is, as Jay pointed out earlier, the attempt to circumvent the symbol limitations built in to the system to get realtime data for a vastly larger set of symbols - in a way that ultimately uses MUCH more bandwidth than if we just had different symbol limit subscriptions and let them watch them that way. Jay also mentioned customizable scanning service on our end - which, if this is what people are attempting to do, would be a better solution all around. The problem (and the reason I brought up these questions) is that I can only see WHAT's being done, not WHY. If the WHY is good (financially and technologically feasible), I'm 100% for attempting to take some of the strain off of our 3rd party developers by doing this on our end. Otherwise, there may be other solutions we can look at to meet your needs. Keep in mind, I don't make the sales decisions or, ultimately, the development decisions - I'm just attempting to identify the problems and the needs that you guys have for our data feed. As for the EOD recap, I'll talk this over with Lonnie and Jay - it seems like a decent idea to me, but there are some issues that'll have to be worked out if we decide to go forward with it. Thanks again, Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 8, 2004 03:50 PM
(Total replies: 24)
Quote: 2. Snapshot service I suggested this in a previous post http://forums.dtniq.com/index.cfm?page=topic&topicID=157. Efficient implementation would be on another server with support for optional batch requests (single or multiple symbols). If you keep the symbol limits in force than this using snapshots is the only other alternative to get access to more symbols. The proper use of this service is realy for instant refreshes of Level 1/Fundamental data and EOD batch summary updates. Sasha & David, the new snapshot requests will be through the same quote server, but won't be impacted by symbol limit. The IQFeed API, however, will need to be implemented to handle the new command to the quote server. Quote: Also, I'd like to see the current streaming data from registered symbols send only the dynamic changes, and not the full record for each quote as the parsing does take time and it is repeated for non-relavant data that is static. Also, no junk out to the client if the data is garbage coming into IQ. This happens on slower dial-up connections. David: Data from the IQServers to the customers is compressed. Are you talking about the data coming out of IQFeed? Thanks for the feedback so far - this definitely helps us to know if we're heading in the right direction or are totally off base with our plans. If you think of anything else, feel free to keep this thread going or start a new one (I monitor this forum a little too much  ).Tim Russell Software Engineer DTN IQ & FinWin
IQFeed Datafeed Wish List »
Add Seconds & Milliseconds to Watchlist Update message Oct 8, 2004 02:00 PM
(Total replies: 22)
Quote: Information transmitted via TCP is guaranteed to be provided in order to the application, as it is a reliable protocol. UDP, on the other hand, is unreliable and data can arrive out of order, or not at all. I stand by my first post in this thread.IQFeed utilizes TCP for all transmission, so this isn't an issue. We're working on adding seconds to the data feed currently. I don't foresee adding milliseconds - as far as I know, the exchanges only send times with a precision of seconds. --- Original message by DTN_Tim_Russell on Sep 17, 2004 11:00 AM We ARE adding seconds into the data feed. Tim Russell Software Engineer DTN IQ & FinWin
POLLS & SURVEYS »
Feature Survey Part 1 - Level 1 Data Feed Oct 8, 2004 10:11 AM
(Total replies: 24)
I really think sasha's on to something with the snapshot server. Based on the seeming demand for this - and talking to Jay and Lonnie - I'm integrating this new functionality into our quote server re-write. The quote servers most likely can handle the load, as there's not a lot of management involved with the snapshot request. If there's a problem, the functionality can easily be moved to a stand-alone server. I'm not done digesting the other suggestions here yet - but I'll respond to them after I've had time to look at them and talk them over with the other developers (and Jay  ).Thanks for the feedback so far, and keep it coming! Tim Russell Software Engineer DTN IQ & FinWin Edited by DTN_Tim_Russell on Oct 8, 2004 at 10:11 AM
IQFeed Datafeed Wish List »
How about sending the Time message once per second? Oct 7, 2004 12:28 PM
(Total replies: 4)
Well, the 75 seconds is about the maximum it's supposed to take, based on the fact that the feed should get timestamps once a minute. A failure in getting the timestamps is usually a result of network failure between the customer and our servers, not necessarily a failure in the TCP connection. I've tried to avoid too much detail in this area up to now, but it looks like I'm not going to be able to do that this time, so, here goes 1. Detecting a connection loss because your internet connection to your computer is disconnected - this is very simple and, if IQFeed doesn't currently have this functionality I'm sure it will in the future. 2. Detecting a connection loss caused by software failure on our end is also easily detected, as the operating system will handle closing out the sockets, causing IQFeed to do whatever it is it does to notify clients and attempt to reconnect to the server way before 75 seconds has past. 3. Now, the most difficult. When a route between an IQFeed instance and our servers goes down temporarily - or becomes congested, or whatever - data may temporarily be unable to be transmitted between client and server. This is the nature of TCP, and it's designed to take these things into account, waiting a set (sometimes very long) amount of time before declaring that the connection is no longer valid. IQFeed attempts to get around this by saying, "After 75 seconds, if I still haven't received a timestamp, attempt a reconnect". The need to wait a certain amount of time before giving up and attempting to reconnect is necessary, especially if the connection failed due to hardware problems somewhere along the line - attempts to reconnect every second are very likely going to be unsuccessful until the link is no longer congested, or the down hardware is back up, etc. Also, having arbitrarily short timeouts tends to cause a bunch of false negatives - i.e., "No data received for 1.5 seconds, attempt to reconnect!". You can see where this would be a problem. Let me recap a bit here: If a connection fails (server kicks you off, you unplug your cable, software crashes), the remote TCP end should be notified immediately. When a socket error doesn't occur, it's a very subjective thing to determine whether the TCP connection has 'failed' or is just stalled. In a nutshell, the 75-seconds is for 'stalled' connections, not necessarily lost connections. Tim Russell Software Engineer DTN IQ & FinWin |
|||||||||||||||||||||||||
