7 ms·
We're in the midst of a data gold rush. People who have data are struggling to monetize it. If you're a data buyer, you're probably swamped with the quantity an
by anonu 8y ago
We're in the midst of a data gold rush. People who have data are struggling to monetize it. If you're a data buyer, you're probably swamped with the quantity and breadth of data providers out there. AI/ML techniques to make sense of this data are still only scratching the surface. I think this is where there is a lot of low-hanging fruit: creating services or tools that allow non-CS/non-Quant people to extract insights from TBs of data...
On the exchange side: these guys are always on the prowl for hot new properties to scoop up. The traditional business model of simply earning fees on exchange trading is slowly eroding away (for the last 10 years). So they need to branch out into services and other data plays...
- pplonski86 8y agoYou are right, extracting insights from data is a low-hanging fruit. From what I observe there is huge lack for proper services and tools that can automatically produce insights. There are of course automatic machine learning solutions, but they focus more on machine learning model tuning (in the kaggle style) rather that giving users understanding and awareness of the their data.
- rainboiboi 8y agoI think data scientists needs to produce more actionable insights as oppose to living in their own world. I suspect there will be an rising group of people who can understand data science techniques and communicate them effectively to drive business decisions. These people will be the ones who can clinch the top posts.
- pplonski86 8y agoI'm running automatic machine learning SaaS for 2 years, and after this time, I can tell you that it is huge problem that data scientist are living in their own world (including me! and including data science tools). I had such situation: my user created 50 ML models (xgboost, lightgbm, NN, rf) and ensemble of them. Let's say that single best model was 5% better than single best model with default hyper-params and ensemble was 2% better than tuned best single model. For me it was a huge success, but the user didn't care about model performance. He wants to have insights about data, not tuned blackbox.
- rainboiboi 8y agoI understand every single word you said and fully agree with you. Good point!
- cpb 8y agoIn an interview, Ryan Caldbeck from Circle Up describes two categories of models: brainy models and brawny models. The ensemble described above sounds like a brawny model: you don't care how it made the decision, you're glad it did the heavy lifting and you might even double check the result. However, the user's concern about the black box suggests they wanted what Ryan refers to as a brainy model, one with explicable decisions. Even within the features of the model there could be things to learn about the data. How else are data scientists stuck within their own world?
- inputcoffee 8y agoAlternative take: there isn't that much low hanging fruit there. Hear me out. "To the person who only has a hammer, everything looks like a nail." The data in front of your is the data you want to analyze, but it doesn't follow that that is the data you ought to analyze. I predict that most of the data you look at will result in nothing. The null hypothesis will not be rejected in the vast majority of cases. I think we -- machine learning learners -- have a fantasy that the signal is lurking and if we just employ that one very clever technique it will emerge. Sure random forests failed, and neural nets failed and the SVR failed but if I reduce the step size, plug the output of the SVR into the net and change the kernel... Let me put an example: suppose you want to analyze the movement of the stock market using the movement of the stars. Adding more information on the stars, and more techniques may feel like you're making progress but it isn't. Conversely, even a simple piece of simple information that requires minimal analysis (this companies sales are way up and no one else but you know it) would be very useful in making that prediction. The first data set is rich, but simply doesn't have the required signal. The second is simple, but has the required signal. The data that is widely available is unlikely to have unextracted signal left in it.
- rdlecler1 8y agoThis is where you might use a genetic algorithm or to learn which data to use on a particular prediction. Good AI won’t use all data just trim down to signal.
- pplonski86 8y agoI would like to see use case when AI selects data source to use that humans will never consider.
- jfoutz 8y agoI read a neat criticism of ai techniques. The author pointed out humans can pick out a strong signal as well or better than ai. Humans could pick out signal from an array of weak sources. Ai would identify that case with fewer weak signals required, but it was hard to trust because it was sometimes wrong. I wish I could remember the source. I’m sure it was an article here a few years ago. I want to say it was medical diagnosis based on charts. Anyway, the point was there is a very narrow valley where ai is useful beyond an expert. And that valley is expensive to explore. And, there might not be anything there.
- momentmaker 8y agoThere is Ocean Protocol (https://oceanprotocol.com/ https://oceanprotocol.com/) that lets you sell your data. There is ChainLink (https://chain.link/ https://chain.link/) that lets you sell your data via API service through decentralized oracle nodes. https://blog.goodaudience.com/the-four-biggest-use-cases-for-chainlink-a0245bd07b66 https://blog.goodaudience.com/the-four-biggest-use-cases-for... Monetization is coming soon... in a big way.
- hobofan 8y agoWe also pitched something in that direction with Rlay at Techcrunch Disrupt Berlin: https://techcrunch.com/2018/11/29/rlay-startup-battlefield/ https://techcrunch.com/2018/11/29/rlay-startup-battlefield/
- abakker 8y agoI'm calling it here: the most useful data is private, or, can't be sold due to confidentiality. The fact that data is confidential is great evidence that we know it is useful, but also that we hope others won't use that signal.
- momentmaker 8y agoWhat about banking initiatives in EU causing every bank to open up their data via API. That seems pretty private and confidential. https://www.evry.com/en/news/articles/psd2-the-directive-that-will-change-banking-as-we-know-it/ https://www.evry.com/en/news/articles/psd2-the-directive-tha... ChainLink has an option to use Intel SGX along with TownCrier to provide trusted execution at the processor level. That ensures confidentiality without exposing the data at all. http://www.town-crier.org/ http://www.town-crier.org/
- Immortalin 8y agoFwiw that's what I have been trying to do for the past few years, infrastructure for easier access to algorithmic trading. Shameless plug: https://KloudTrader.com/narwhal https://KloudTrader.com/narwhal
- pmart123 8y agoNasdaq already makes more money on data licensing than on trading fees or IPO's. Each time a professional in the financial services industry wants real-time display data, for example, they have to pay Nasdaq a monthly fee. Nasdaq and NYSE compete for listings less so now for trading fees, but because it makes its data licensing package more valuable.
- claytonjy 8y agoFor finance in particular, I'd say we're drowning in a massive volume of shitty data. A client of mine purchases several fundamental feeds from Quandl, and I email them regularly to point out errors. Not weird, hard, tricky errors, but stuff like "why are all these volumes missing" or "there's a 1-day closing price increase of 1200%" or "you divided when you should have multiplied". This tells me neither Quandl nor the original provider (e.g. Zacks) do any serious data validation, despite claiming to. If the companies people have been paying for decades for this data get it wrong this often, how can I trust any weirder data they're trying to sell me? I thought the point of buying these feeds was to let the seller worry about quality assurance.
- vasilipupkin 8y agothis doesn't matter - any sophisticated user will have their own software to clean the data anyway. Their concern is getting the data, they know how to clean it once they have it.
- claytonjy 8y agoWe're not talking about data cleaning, but about data validation. I can fix a weirdly formatted field (cleaning), but I can't reliably impute most kinds of missing data. I can detect errors, but can't fix most of them without additional information...which is exactly what I'm paying the provider for.
- vasilipupkin 8y agoyou can, there are ways to do it. interpolation, etc. Sometimes the data is missing just because it's not available, you still have to handle that case. proper way of filling in this missing data will depend on what you are using it for - so for provider to do it would be kind of wrong actually.
- claytonjy 8y agoI think we're talking past eachother here. I don't expect the provider to do imputation for me, but I shouldn't have to bug them to get the best version of the data they have. Sure, sometimes missing is missing, but in my experience with Quandl/Zacks, its usually an error on their end. The price jumps are sometimes because they conflated two different tickers. If they divided instead of multiplying (split factors), I have to have external information to even detect the error! Same goes if they get a date wrong somewhere.