Join the 80,000 other DTN customers who enjoy the fastest, most reliable data available. There is no better value than DTN!

(Move your cursor to this area to pause scrolling)




"Interactive Brokers tick data was inconsistent, so I have switched to using DTN exclusively. It is great to no longer have to worry about my datafeed all day long." - Comment from Philippe
"Thank God for your Data Feed as the only Zippers I see are on my pants (LOL), and no more 200 pip spikes to mess up charts." - Comment from Spiro via Email
"I will tell others who want to go into trading that DTN ProphetX is an invaluable tool, I don't think anyone can trade without it..." - Comment from Luther
"DTN feed was the only feed that consistently matched Bloomberg feed for BID/ASK data verification work these past years......DTN feed is a must for my supply & demand based trading using Cumulative Delta" - Comment from Public Forum Post
"I used to have *******, but they are way more money for the same thing. I have had no probs with data from DTN since switching over." - Comment from Public Forum Post
"I've been using IQFeed 4 in a multi-threaded situation for the last week or two on 2600 symbols or so with 100 simultaneous daily charts, and I have had 100% responsiveness." - Comment from Scott
"There is no doubt that IQFeed is the best data provider. I am very satisfied with your services. And IQFeed is the only one that I would recommend to my friends. Now, most of them are using your product in China." - Comment from Zhezhe
"My broker in Davenport suggested I give you a try as he uses your service and says its the best." - Comment from Bill via RT Chat
"I just wanted to say how happy I am with your service. I was able to download the API docs last week and I was able to replicate Interactive Brokers historical bar queries and realtime bar queries over the weekend. That was about one of the fastest integrations that I've ever done and it works perfectly!!!!" - Comment from Jason via Email
"Everything is working great with the API. I love it." - Comment from Calvin
Home  Search  Register  Login  Recent Posts

Information on DTN's Industries:
DTN Oil & Gas | DTN Trading | DTN Agriculture | DTN Weather
Follow DTNMarkets on Twitter
DTN.IQ/IQFeed on Twitter
DTN News and Analysis on Twitter
»Forums Index »NEW IQFEED FORUMS »Miscellaneous Messages »Machine learning and tick-by-tick data
Author Topic: Machine learning and tick-by-tick data (2 messages, Page 1 of 1)

keohir808
-Interested User-
Posts: 6
Joined: Sep 16, 2019


Posted: Jan 30, 2022 08:45 PM          Msg. 1 of 2
There seems to be a lack of research regarding the use of tick-by-tick data as input to machine learning models. Has anyone experimented with machine learning and tick-by-tick data? I’ve trained an LSTM with about 2 years worth using tick-by-tick data with DTN which fit a certain criteria such as float, volume, price. The result is a model with around 69.9% accuracy. A naïve model which predicts that the bid price will be the same price as the last tick has an accuracy of around 65%. I’m wondering if I can increase my model’s accuracy through feature engineering. Could anyone share research papers regarding machine learning and tick-by-tick data? Does anyone have any insight regarding data transformations that can be applied to financial data which may result in increased accuracy if used as a feature in machine learning models?


Interests & Tools: Machine Learning, Neural Networks, Deep Learning, Python, Java, Trading, Small Caps, Interactive Brokers.
Edited by keohir808 on Jan 30, 2022 at 08:50 PM

taa_dtn
-DTN Evangelist-
Posts: 154
Joined: May 7, 2004


Posted: Jan 31, 2022 11:10 AM          Msg. 2 of 2
Yes, I experimented with this a few years ago. Your questions are relevant and insightful, but I don't have much useful information to offer in reply.

I haven't seen many published papers on the subject in recent years. Take that with a grain of salt, though, because I'm not looking actively enough. Possibly if the technique has been applied successfully, it hasn't been discussed in public for the obvious reasons. Hopefully someone else will reply with better information.

In general, I hit the same roadblocks you did. It's hard to choose the right network architecture (financial data isn't statistically stationary, so I wasn't able to design either recurrent or convolutional networks that were consistently successful). Raw tick-by-tick data has so much variability along so many dimensions that I suspect feature engineering is necessary, but that's a major research project in its own right. Techniques currently being used for natural language processing are probably where I'd start if I were to look at this again today.

Possibly the most fundamental problem I ran into is that it doesn't seem workable to use a scalar value to measure outcomes, so anything based on simple gradient descent is problematic. I think a practical outcome measurement must be at least three-dimensional -- it needs to include return, risk, and capital management. Arguably more, but the need for those three is easy to understand.
 

 

Time: Mon February 26, 2024 8:38 AM CFBB v1.2.0 12 ms.
© AderSoftware 2002-2003