7 ms·

The money of big data is in analytics

66 points by dendory 14y ago

DrinkWater 14y ago
I'm sorry, but isn't that obvious?
capkutay 14y ago
Although the headline is stating the obvious, this article touches on some good points that companies so far have completely missed. "But now that companies can receive and store this data, everything from logs, to usage tracking, location coordinates, patterns and so on, the next step is to make sense of all of this." Shameless plug, but I'm working on those problems now and it is an extremely rewarding area to be in. If you're interested in working in that area or learning more about it feel free to ping me.
noelwelsh 14y ago
When I see analytics in industry I normally see a whole lotta graphs, big reports for the monthly board meeting, and so on. The way the industry is moving is basically replicating what people used to do with "small data" on the systems for handling "big data". Witness the surge of interest in SQL-on-Hadoop type systems. I believe a large amount of this is crap. Only actions lead to improvements. The tighter your observe/act loop the more chances to win you have in any given timeframe. Computers can act a heck of a lot faster than humans. Computers can also do a better job for balancing risk than humans in many situations. If I was a consultant I would definitely be selling my report generating ability. If I was a company that really wanted to win big I would investing in automated decision making systems. (Ob plug: My startup, Myna http://mynaweb.com http://mynaweb.com, is one small realisation of this idea.)
- deleted 14y ago
  [deleted]
- mtrimpe 14y ago
  Sorry to take it off-topic, but I've never understood how you're supposed to customize your pages based on a synchronous call to a third party API though. From what I can see it add quite a lot of latency and introduce a single point of failure, which I'm sure isn't acceptable for most cases.
  - noelwelsh 14y ago
    I'd rather have this discussion over email: noel at mynaweb.com If you're really concerned about latency, integrate on the server-side using asynchronous calls. If that isn't an option: - Our main approach to managing latency right now is making Myna really really fast (avg response time 2.5ms, 99% ~6ms). - If that isn't good enough, you can host your own Myna server(s). - We have various things in the works to improve on this in the near- and medium-term future. Happy to discuss more on email.
- blablabla123 14y ago
  Yeah within report generators is a lot of business I think. The classical report generators take ages and may even put strain on the DB servers. Writing this stuff completely from scratch on the other hand is a lot of work, so it takes time to see novel results. (I speak from experience ;))
austenallred 14y ago
It depends on how you define "analytics." The problem continues to be the same: There's too much data, and we can't make sense of it. It seems obvious, but this simple concept has huge ramifications. Different ways of approaching making use of that data are huge swings: look at how other search engines tried to make sense of the web vs. how Google did it. Look at how the dozens of analytics companies tried to make use of your web analytics vs. how Omniture did it. In a more modern example, look at how much content is generated every day on social networks, or how much data healthcare facilities have resting in different silos that is completely unusable to them. There's a lot more that needs to be done.
sgloutnikov 14y ago
Thanks for this article! As someone who is just getting into the field out of University, I love to absorb as much as possible on this topic and read different points of view.
graycat 14y ago
Before you can sell me a wrench, have to let me know what nuts it will turn, and I have to have some such nuts to turn. For big data, what nuts does it turn? I'm still waiting to hear just what nuts people want turned, especially those for which 'big data' is essential. It's easy enough to find cases where analysis has been stopped due to far too little data or far to little ability to handle more data. A classic example is R. Bellman's "curse of dimensionality" especially for his work in dynamic programming -- i.e., best decision making over time under uncertainty (with flavor quite different from the uses of dynamic programming in some computer science algorithms). Broadly, for the curse of dimensionality, we can start with the set of real numbers R, a positive integer n, and the real n-dimensional space R^n, that is, just the set of all n-tuples of real numbers. Then as n starts to grow, it takes 'big data' to start to 'fill', say, the n-dimensional 'cube' with each side 100 units long, [0, 100]^n. So, if want to describe something in such a cube and want a lot of accuracy, then can start with 1 MB of data and start multiplying by factors of 1000 over and over. We can zip past a warehouse full of 4 TB disk drives in a big hurry. Here is a general situation in 'analysis': We are looking for the value of some variable Y. Since we don't know Y, we can say we are looking for the value of a random variable. Or we can look for its distribution. For input, maybe we have many pairs (x, y) where x is in R^n and y is in R. Then, maybe we are told that in our case we have some X in R^n and want the corresponding Y or its distribution. Well, essentially there is one, just one, one to rule all the rest, way to answer this. May I have the envelope, please? Yes, here it is (drum roll): Simple, plain old cross-tabulation. Why? Because cross tabulation is just the discrete version of the joint distribution from which we use the classic Radon-Nikodym theorem (Rudin, 'Real and Complex Analysis') to say that the best answer we can get (non-linear least squares) is just the conditional distribution or its expectation the conditional expectation, that is, E[Y|X] which is the best non-linear least squares approximation of Y as a function of X, and taking an average from a cross tabulation is the discrete approximation of this. For a good approximation for a lot of values of X, we can suck up 'big data' and ask for many factors of thousands of times more. The conditional distribution of Y given X is P(Y <= y|X) and is addressed similarly. So, net, I agree that there can be some good uses for big data. Still, before proposing an answer and picking the tools, let's hear the real questions. Okay? Why? For one, as general as cross tabulation is, it commonly requires so much data that even realistic versions of 'big data' are way too small. So, typically we use methods other than just cross tabulation to make better use of our limited TBs of data, and to select such methods we really need to hear the question first. Before I select a wrench, I want to look at the nut. Is this point too much to ask in the discussion of 'big data'? I will end with one more: Suppose we want to estimate E[Y] by taking an average of n 'samples'. Under the usual assumptions, the standard deviation of our estimate goes down like 1 over the square root of n. So, to get the standard deviation 10 times smaller, we need n to be 100 times bigger. So, roughly for each additional significant digit we want in the estimate, we need another 100 times as much data. Once we start asking for more than, say, five more significant digits, we are way up on a parabola in the amount of data we need. Net, if we want really accurate estimates, then even big data has to struggle. So, really, we accept the law of diminishing returns and just use medium or small data.
jamestc 14y ago
>He says that data can be gathered showing how many people see a particular painting or share it online, and thus reach conclusions on how successful an artist is. Wouldn't this just lead to art becoming a new form of spam when people try to game this system? Not to mention that not all art is painting or can be easily shared.
antoniuschan99 14y ago
https://www.youtube.com/watch?v=G1KObNG_Wnw https://www.youtube.com/watch?v=G1KObNG_Wnw
qxf2 14y ago
>>"The problem of big data has been solved. We know how to gather data and store it." Nope. Far from it. We are still learning to gather data and store it well. This is a complex problem. The author is underestimating the difficulty in a large number of disparate people collecting data and the variety of formats it produces.
- alexatkeplar 14y ago
  Exactly this. At SnowPlow (https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow) we would love to spend more time downstream at the analysis phase (doing ML etc), but we still have to spend a ton of time working upstream on collection, storage, enrichment etc. A lot of this work is defining, testing and documenting standard protocols, data models etc (see https://github.com/snowplow/snowplow/wiki/SnowPlow-technical-documentation https://github.com/snowplow/snowplow/wiki/SnowPlow-technical... if you're interested). And this is just for eventstream analytics, working with our own data formats - ingesting and mapping third-party formats (e.g. Omniture, MailChimp, MixPanel etc) is another lot of work that needs doing... So a solved problem? Not so much.
apapli 14y ago
Big data is all the rage, but as an enterprise biz dev guy I hear a lot of talk about it and not a lot of real doing. The article states accurately that people can store huge amounts of data. What I don't think it explored sufficiently is how companies can make sense of this. Big data gives companies the ability to find new insights into relationships between data points they had no idea existed. From what I am seeing, people are however approaching the issue the same (wrong) way - they are running the same reports on larger data sets, or running new reports to confirm/disprove areas where they suspect relationships will exist. What big data needs is a generally available product (read: $$ thousands per year, not hundreds of thousands or millions) that allows companies to send in their data sets, with it proactively mining to find new trends across is entirety. I don't think this product exists yet - and if it does please tell me and I'll rapidly call them for a job - and is exactly where the big money will be.
- oscilloscope 14y ago
  The closest thing I can think of is OpenRefine. It doesn't directly find trends, but it does do machine learning to help you clean up datasets. You can also create a scatterplot matrix and do faceted search, which lets you quickly explore data and relationships between dimensions. It's an open source program (previously Google Refine). http://openrefine.org/ http://openrefine.org/ For health care, finance, enterprises and government, data quality is a huge issue. Many machine learning and statistical methods just don't work or produce misleading results with messy data.
  - aidos 14y ago
    Ahhh, so that's where Google Refine went to. I was trying to find it recently. Thanks for the link.
    - me_bx 14y ago
      You might also be interested in recline.js (http://okfnlabs.org/recline/ http://okfnlabs.org/recline/), a fork of Google refine, initially created by @maxogden, now maintained by the Open Knowledge Foundation (okfn). How their data processing features differ from openRefine is not clear to me though...
ErikAugust 14y ago
"Data is like garbage. You'd better know what you are going to do with it before you collect it." - Usually attributed to Mark Twain For an article on big data, it did not provide much data.

66 points by dendory 14y ago

I'm sorry, but isn't that obvious?

Although the headline is stating the obvious, this article touches on some good points that companies so far have completely missed. "But now that companies can receive and store this data, everything from logs, to usage tracking, location coordinates, patterns and so on, the next step is to make sense of all of this." Shameless plug, but I'm working on those problems now and it is an extremely rewarding area to be in. If you're interested in working in that area or learning more about it feel free to ping me.

noelwelsh 14y ago

When I see analytics in industry I normally see a whole lotta graphs, big reports for the monthly board meeting, and so on. The way the industry is moving is basically replicating what people used to do with "small data" on the systems for handling "big data". Witness the surge of interest in SQL-on-Hadoop type systems. I believe a large amount of this is crap. Only actions lead to improvements. The tighter your observe/act loop the more chances to win you have in any given timeframe. Computers can act a heck of a lot faster than humans. Computers can also do a better job for balancing risk than humans in many situations. If I was a consultant I would definitely be selling my report generating ability. If I was a company that really wanted to win big I would investing in automated decision making systems. (Ob plug: My startup, Myna http://mynaweb.com http://mynaweb.com, is one small realisation of this idea.)

deleted 14y ago
[deleted]
mtrimpe 14y ago
Sorry to take it off-topic, but I've never understood how you're supposed to customize your pages based on a synchronous call to a third party API though. From what I can see it add quite a lot of latency and introduce a single point of failure, which I'm sure isn't acceptable for most cases.
- noelwelsh 14y ago
  I'd rather have this discussion over email: noel at mynaweb.com If you're really concerned about latency, integrate on the server-side using asynchronous calls. If that isn't an option: - Our main approach to managing latency right now is making Myna really really fast (avg response time 2.5ms, 99% ~6ms). - If that isn't good enough, you can host your own Myna server(s). - We have various things in the works to improve on this in the near- and medium-term future. Happy to discuss more on email.
blablabla123 14y ago
Yeah within report generators is a lot of business I think. The classical report generators take ages and may even put strain on the DB servers. Writing this stuff completely from scratch on the other hand is a lot of work, so it takes time to see novel results. (I speak from experience ;))

austenallred 14y ago

It depends on how you define "analytics." The problem continues to be the same: There's too much data, and we can't make sense of it. It seems obvious, but this simple concept has huge ramifications. Different ways of approaching making use of that data are huge swings: look at how other search engines tried to make sense of the web vs. how Google did it. Look at how the dozens of analytics companies tried to make use of your web analytics vs. how Omniture did it. In a more modern example, look at how much content is generated every day on social networks, or how much data healthcare facilities have resting in different silos that is completely unusable to them. There's a lot more that needs to be done.

sgloutnikov 14y ago

Thanks for this article! As someone who is just getting into the field out of University, I love to absorb as much as possible on this topic and read different points of view.

graycat 14y ago

Before you can sell me a wrench, have to let me know what nuts it will turn, and I have to have some such nuts to turn. For big data, what nuts does it turn? I'm still waiting to hear just what nuts people want turned, especially those for which 'big data' is essential. It's easy enough to find cases where analysis has been stopped due to far too little data or far to little ability to handle more data. A classic example is R. Bellman's "curse of dimensionality" especially for his work in dynamic programming -- i.e., best decision making over time under uncertainty (with flavor quite different from the uses of dynamic programming in some computer science algorithms). Broadly, for the curse of dimensionality, we can start with the set of real numbers R, a positive integer n, and the real n-dimensional space R^n, that is, just the set of all n-tuples of real numbers. Then as n starts to grow, it takes 'big data' to start to 'fill', say, the n-dimensional 'cube' with each side 100 units long, [0, 100]^n. So, if want to describe something in such a cube and want a lot of accuracy, then can start with 1 MB of data and start multiplying by factors of 1000 over and over. We can zip past a warehouse full of 4 TB disk drives in a big hurry. Here is a general situation in 'analysis': We are looking for the value of some variable Y. Since we don't know Y, we can say we are looking for the value of a random variable. Or we can look for its distribution. For input, maybe we have many pairs (x, y) where x is in R^n and y is in R. Then, maybe we are told that in our case we have some X in R^n and want the corresponding Y or its distribution. Well, essentially there is one, just one, one to rule all the rest, way to answer this. May I have the envelope, please? Yes, here it is (drum roll): Simple, plain old cross-tabulation. Why? Because cross tabulation is just the discrete version of the joint distribution from which we use the classic Radon-Nikodym theorem (Rudin, 'Real and Complex Analysis') to say that the best answer we can get (non-linear least squares) is just the conditional distribution or its expectation the conditional expectation, that is, E[Y|X] which is the best non-linear least squares approximation of Y as a function of X, and taking an average from a cross tabulation is the discrete approximation of this. For a good approximation for a lot of values of X, we can suck up 'big data' and ask for many factors of thousands of times more. The conditional distribution of Y given X is P(Y <= y|X) and is addressed similarly. So, net, I agree that there can be some good uses for big data. Still, before proposing an answer and picking the tools, let's hear the real questions. Okay? Why? For one, as general as cross tabulation is, it commonly requires so much data that even realistic versions of 'big data' are way too small. So, typically we use methods other than just cross tabulation to make better use of our limited TBs of data, and to select such methods we really need to hear the question first. Before I select a wrench, I want to look at the nut. Is this point too much to ask in the discussion of 'big data'? I will end with one more: Suppose we want to estimate E[Y] by taking an average of n 'samples'. Under the usual assumptions, the standard deviation of our estimate goes down like 1 over the square root of n. So, to get the standard deviation 10 times smaller, we need n to be 100 times bigger. So, roughly for each additional significant digit we want in the estimate, we need another 100 times as much data. Once we start asking for more than, say, five more significant digits, we are way up on a parabola in the amount of data we need. Net, if we want really accurate estimates, then even big data has to struggle. So, really, we accept the law of diminishing returns and just use medium or small data.

jamestc 14y ago

>He says that data can be gathered showing how many people see a particular painting or share it online, and thus reach conclusions on how successful an artist is. Wouldn't this just lead to art becoming a new form of spam when people try to game this system? Not to mention that not all art is painting or can be easily shared.

antoniuschan99 14y ago

https://www.youtube.com/watch?v=G1KObNG_Wnw https://www.youtube.com/watch?v=G1KObNG_Wnw

qxf2 14y ago

>>"The problem of big data has been solved. We know how to gather data and store it." Nope. Far from it. We are still learning to gather data and store it well. This is a complex problem. The author is underestimating the difficulty in a large number of disparate people collecting data and the variety of formats it produces.

alexatkeplar 14y ago
Exactly this. At SnowPlow (https://github.com/snowplow/snowplow https://github.com/snowplow/snowplow) we would love to spend more time downstream at the analysis phase (doing ML etc), but we still have to spend a ton of time working upstream on collection, storage, enrichment etc. A lot of this work is defining, testing and documenting standard protocols, data models etc (see https://github.com/snowplow/snowplow/wiki/SnowPlow-technical-documentation https://github.com/snowplow/snowplow/wiki/SnowPlow-technical... if you're interested). And this is just for eventstream analytics, working with our own data formats - ingesting and mapping third-party formats (e.g. Omniture, MailChimp, MixPanel etc) is another lot of work that needs doing... So a solved problem? Not so much.

apapli 14y ago

Big data is all the rage, but as an enterprise biz dev guy I hear a lot of talk about it and not a lot of real doing. The article states accurately that people can store huge amounts of data. What I don't think it explored sufficiently is how companies can make sense of this. Big data gives companies the ability to find new insights into relationships between data points they had no idea existed. From what I am seeing, people are however approaching the issue the same (wrong) way - they are running the same reports on larger data sets, or running new reports to confirm/disprove areas where they suspect relationships will exist. What big data needs is a generally available product (read: $$ thousands per year, not hundreds of thousands or millions) that allows companies to send in their data sets, with it proactively mining to find new trends across is entirety. I don't think this product exists yet - and if it does please tell me and I'll rapidly call them for a job - and is exactly where the big money will be.

oscilloscope 14y ago
The closest thing I can think of is OpenRefine. It doesn't directly find trends, but it does do machine learning to help you clean up datasets. You can also create a scatterplot matrix and do faceted search, which lets you quickly explore data and relationships between dimensions. It's an open source program (previously Google Refine). http://openrefine.org/ http://openrefine.org/ For health care, finance, enterprises and government, data quality is a huge issue. Many machine learning and statistical methods just don't work or produce misleading results with messy data.
- aidos 14y ago
  Ahhh, so that's where Google Refine went to. I was trying to find it recently. Thanks for the link.
  - me_bx 14y ago
    You might also be interested in recline.js (http://okfnlabs.org/recline/ http://okfnlabs.org/recline/), a fork of Google refine, initially created by @maxogden, now maintained by the Open Knowledge Foundation (okfn). How their data processing features differ from openRefine is not clear to me though...

ErikAugust 14y ago

"Data is like garbage. You'd better know what you are going to do with it before you collect it." - Usually attributed to Mark Twain For an article on big data, it did not provide much data.