5 ms·

I disagree with the whole premise that data science is the wave of the future. As soon as you collect a piece of information it is already out of date and is of

I disagree with the whole premise that data science is the wave of the future. As soon as you collect a piece of information it is already out of date and is of very limited use. Weather scientists have been collection data since the beginning of the century and they are still no better at predicting the weather than they were at the beginning of the century. Economists are in the same boat. They have tons of data on markets but they still can't figure out what makes markets tick. What we need is not more data or data centric thinking. We need generative models that explain how the data is being generated and why it is being generated in a certain way. There is only so much you can squeeze out of raw data by computing numbers from it and so far I don't think the results have been that impressive.

pvg 16y ago
Weather scientists have been collection data since the beginning of the century and they are still no better at predicting the weather than they were at the beginning of the century. Economists are in the same boat. If you're going to disagree with a premise, it might be better to pick counter-examples that are actually, you know, true.
- greenlblue 16y ago
  Huh? I'm pretty sure both economists and weather scientists have tons of data but their theories to explain that data have been stagnating for a while now. Economic crashes and booms still happen and nobody knows why even though there is a lot of historical market data people still don't know what leads to them and how to stop them. When was the last time you got an accurate rain forecast? 40% chance of rain? What does that even mean? It either rains or it doesn't and I want to know exactly which and nobody will figure out how to do so anytime soon. While I'm at it I will throw the human genome at you as well. Big pharma was promising untold breakthroughs once we figured out the stuff people are made from but the results have been unspectacular since cancer incidence and whatever other incidence you can think of has been steady and even increasing. The only time big pharmacy companies made any breakthroughs were when they were trying to figure out something and accidentally figured out something else or invented some condition, ADHD anyone? The best treatment even after all the data collection and genome extraction still is prevention and your grandma knew that and she lived during a time when the number 23 counted for big data. So I'm not holding out my breath on any promises made with big data as a powerpoint bullet. The main problem with all big data sets is that nobody knows how to handle them globally. All algorithms and theories treat the data locally and then try to collate the local pieces to get a global picture. This is fine when your data sets underlying structure is a graph or can be treated as some kind of graph. So the future doesn't belong to data scientists or statisticians but to the people that will figure out how to treat large non-graph like data sets in a meaningful way.
cageface 16y ago
On the contrary it seems like data-centric models are utterly trouncing ab-initio methods. See http://www.scribd.com/doc/13863110/The-Unreasonable-Effectiveness-of-Data http://www.scribd.com/doc/13863110/The-Unreasonable-Effectiv... .
akshayubhat 16y ago
already out of date and is of very limited use Depends on the type of the information, news: yes,information about biology, or Geo-Information or encyclopedic: not so fast. on other hand we now have access to real time data. Weather scientists have been collection data since the beginning of the century and they are still no better at predicting the weather than they were at the beginning of the century. Predicting Weather is near impossible in theory forget the practice. its a chaotic system, generally in data science we are trying to predict things which we know can be predicted, such as determining whether an article is relevant to a topic for example. Humans can do that very well but for machines to do it you need more data. They have tons of data on markets but they still can't figure out what makes markets tick. Again markets are chaotic and are affected by things like low probability events. You can do predict some patterns using information asymmetry, thats what all those traders at Goldman Sachs and other firms do. We need generative models that explain how the data is being generated and why it is being generated in a certain way. The systems that we are interested in have complex models, and generating them from first principles isnt always as easy, additionally even if you do generate you need to test them against real world data. Rather than using the normal hypothesis - experiment cycle, it makes more sense to look for predictable patterns in data. I don't think the results have been that impressive. Thats because you are out of touch with the field. Look at Google's Statistical Translation results, for example, they beat every generative model around the town. Or netflix's prize for recommendation system. As someone interested in Data Science and been quite involved with it for few years as a student and an intern let me try to explain why data science is now becoming an important area: 1. We now have a lot of data in a machine readable format: Unlike few years ago we now have huge datasets publicly available they, you have topic specific datasets such as Geo Names, Linked Geo data to much more wide encyclopedic datasets such as Wikipedia Data Dump, Freebase, Open Cyc, DBPedia. We also have huge amount of user generated data, E.g. I have on my computer right now a huge chunk of twitters follower network consisting of 35 million users(I am writing an open source whom to follow system). Additionally I also have 100 million tweets. 2. Not just data we now have access to real time data: You can access the public twitter time-line, using their streaming api, there are quite a few pubsubhubub systems out there which combine information from disparate sources and provide you unified source. 3. Moreover we have tools to handle the deluge [well sort off]: Thanks to Google, Apache, Yahoo, Facebook we now have Hadoop , Map Reduce, Pig and other tools which make job of parallelizing processing of the data easier. 5. We have a Scalable on demand infrastructure in place: Using AWS we can buy processing power as needed. It would have been impossible earlier, I recently bought a high memory instance with 17GB Ram for 50 cents an hour for 10 hours to run some jobs. It would have been impossible few years ago. We can now also deploy web apps very easily using Google App engine and dont need to even pay a penny, this enables us to create nice interfaces for visualization and querying for the data at a low initial cost. Finally we are slowly building infrastructure to sell datasets or custom apps. E.g. Amazon Dev Pay or Infochimps. The name data science can though be misleading, if you are a student that would mean taking course in Machine Learning, Data Mining, Information Retrieval, Statistics, Distributed Computing, Databases.
mturmon 16y ago
The key difference between economics and weather is that experiments with prediction change the economic system's dynamics (e.g., large-scale automated trading), but not the weather system's dynamics.

pvg 16y ago

Weather scientists have been collection data since the beginning of the century and they are still no better at predicting the weather than they were at the beginning of the century. Economists are in the same boat. If you're going to disagree with a premise, it might be better to pick counter-examples that are actually, you know, true.

greenlblue 16y ago
Huh? I'm pretty sure both economists and weather scientists have tons of data but their theories to explain that data have been stagnating for a while now. Economic crashes and booms still happen and nobody knows why even though there is a lot of historical market data people still don't know what leads to them and how to stop them. When was the last time you got an accurate rain forecast? 40% chance of rain? What does that even mean? It either rains or it doesn't and I want to know exactly which and nobody will figure out how to do so anytime soon. While I'm at it I will throw the human genome at you as well. Big pharma was promising untold breakthroughs once we figured out the stuff people are made from but the results have been unspectacular since cancer incidence and whatever other incidence you can think of has been steady and even increasing. The only time big pharmacy companies made any breakthroughs were when they were trying to figure out something and accidentally figured out something else or invented some condition, ADHD anyone? The best treatment even after all the data collection and genome extraction still is prevention and your grandma knew that and she lived during a time when the number 23 counted for big data. So I'm not holding out my breath on any promises made with big data as a powerpoint bullet. The main problem with all big data sets is that nobody knows how to handle them globally. All algorithms and theories treat the data locally and then try to collate the local pieces to get a global picture. This is fine when your data sets underlying structure is a graph or can be treated as some kind of graph. So the future doesn't belong to data scientists or statisticians but to the people that will figure out how to treat large non-graph like data sets in a meaningful way.

cageface 16y ago

On the contrary it seems like data-centric models are utterly trouncing ab-initio methods. See http://www.scribd.com/doc/13863110/The-Unreasonable-Effectiveness-of-Data http://www.scribd.com/doc/13863110/The-Unreasonable-Effectiv... .

akshayubhat 16y ago

already out of date and is of very limited use Depends on the type of the information, news: yes,information about biology, or Geo-Information or encyclopedic: not so fast. on other hand we now have access to real time data. Weather scientists have been collection data since the beginning of the century and they are still no better at predicting the weather than they were at the beginning of the century. Predicting Weather is near impossible in theory forget the practice. its a chaotic system, generally in data science we are trying to predict things which we know can be predicted, such as determining whether an article is relevant to a topic for example. Humans can do that very well but for machines to do it you need more data. They have tons of data on markets but they still can't figure out what makes markets tick. Again markets are chaotic and are affected by things like low probability events. You can do predict some patterns using information asymmetry, thats what all those traders at Goldman Sachs and other firms do. We need generative models that explain how the data is being generated and why it is being generated in a certain way. The systems that we are interested in have complex models, and generating them from first principles isnt always as easy, additionally even if you do generate you need to test them against real world data. Rather than using the normal hypothesis - experiment cycle, it makes more sense to look for predictable patterns in data. I don't think the results have been that impressive. Thats because you are out of touch with the field. Look at Google's Statistical Translation results, for example, they beat every generative model around the town. Or netflix's prize for recommendation system. As someone interested in Data Science and been quite involved with it for few years as a student and an intern let me try to explain why data science is now becoming an important area: 1. We now have a lot of data in a machine readable format: Unlike few years ago we now have huge datasets publicly available they, you have topic specific datasets such as Geo Names, Linked Geo data to much more wide encyclopedic datasets such as Wikipedia Data Dump, Freebase, Open Cyc, DBPedia. We also have huge amount of user generated data, E.g. I have on my computer right now a huge chunk of twitters follower network consisting of 35 million users(I am writing an open source whom to follow system). Additionally I also have 100 million tweets. 2. Not just data we now have access to real time data: You can access the public twitter time-line, using their streaming api, there are quite a few pubsubhubub systems out there which combine information from disparate sources and provide you unified source. 3. Moreover we have tools to handle the deluge [well sort off]: Thanks to Google, Apache, Yahoo, Facebook we now have Hadoop , Map Reduce, Pig and other tools which make job of parallelizing processing of the data easier. 5. We have a Scalable on demand infrastructure in place: Using AWS we can buy processing power as needed. It would have been impossible earlier, I recently bought a high memory instance with 17GB Ram for 50 cents an hour for 10 hours to run some jobs. It would have been impossible few years ago. We can now also deploy web apps very easily using Google App engine and dont need to even pay a penny, this enables us to create nice interfaces for visualization and querying for the data at a low initial cost. Finally we are slowly building infrastructure to sell datasets or custom apps. E.g. Amazon Dev Pay or Infochimps. The name data science can though be misleading, if you are a student that would mean taking course in Machine Learning, Data Mining, Information Retrieval, Statistics, Distributed Computing, Databases.

mturmon 16y ago

The key difference between economics and weather is that experiments with prediction change the economic system's dynamics (e.g., large-scale automated trading), but not the weather system's dynamics.