6 ms·

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever.

Another conclusion to draw from this article (which I really enjoyed, by the way) is that Big Data has been turned into one of the most abstract buzzwords ever. You thought "cloud" was bad? "Big Data" is far worse in its specificity. I can't count the number of times I'll be talking to some sales rep and they'll describe how they scan the data within whatever application they're demoing and "suggest" items using "big data techniques". In almost all cases they're talking about a few thousand or hundred thousand records, tops. I've found that when non-hardcore techies talk about Big Data, what they really mean is "they have some data" vs before, when they had zero data. From the article: "Consultants urge the data-naive to wise up to the potential of big data. A recent report from the McKinsey Global Institute reckoned that the US healthcare system could save $300bn a year – $1,000 per American – through better integration and analysis of the data produced by everything from clinical trials to health insurance transactions to smart running shoes. What these consultants mean is that by having just some data compared to the silo'd data that is the norm in US healthcare, they could save a lot, and they're right. My previous company had a large data set (20+ million patients) and we'd find millions of dollars of savings opportunities for every hospital we implemented in, but that's because we had the data, not because we were running some kind of non-causual correlation analysis like the article references. It was just because we could actually run queries on a data set. ----- Off Topic - how annoying is it that when you copy & paste from the FT, they preface your copy with the following text? High quality global journalism requires investment. Please share this article with others using the link below, do not cut & paste the article. See our Ts&Cs and Copyright Policy for more detail. Email ftsales.support@ft.com to buy additional rights. http://www.ft.com/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabdc0.html#ixzz2xSKoQYaW* http://www.ft.com/cms/s/2/21a6e7d8-b479-11e3-a09a-00144feabd...

JonLim 13y ago
Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.
- justincormack 13y ago
  Big Data used to mean petabytes, ie above the limits of performant scale up.
- ntoshev 13y ago
  I believe the accepted definition for big data is data you can't handle on a single machine and need a cluster to process. So Moore's law makes it a moving threshold.
- Malarkey73 13y ago
  I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description: On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds. When you have time to grab a coffee after hitting return then you have bigger data. If you carefully think through what you are about to ask the computer to do before pressing return then maybe you have big data. I actually think this is a better description than just size of files or data distributed across many computers as an algorithm that just streams over a massive dataset maybe in parallel can be less challenging than one that has to hold a much smaller e.g many Gb dataset fully in memory.
  - RogerL 13y ago
    So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense. It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.
    - deleted 13y ago
      [deleted]
    - Malarkey73 13y ago
      Yes to a data analyst that is big data. If you are doing some MCMC and that is really what it takes on that size of data then you have a big data problem. The more sophisticated a statistic, the more high dimensional the data, the more sampling required, or the more of the dataset it requires to memorise at once - then the smaller your big data threshold will be. It depends a lot on your point of view too. If I google something now it may bounce across lots of crazy server farms but to me I don't feel like I'm doing big data.. the person who built it all probably feels differently.
ElDiablo666 13y ago
Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."
- Houshalter 13y ago
  Who said anything about thinking, and how do you know it's hard? EDIT: I'm getting downvoted, but your statement is incredibly vague and I believe wrong. "Big Data" might be overused as a buzzword, but it's not a "nonsense concept". "Thinking is hard", I assume you are talking about strong AI, and it's not related to this at all. Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.)
  - coldtea 13y ago
    >Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.) If it takes decades of hard working geniuses to figure it out, then even if "it seems simple and obvious in retrospect", it IS hard.
  - aaron695 13y ago
    Downvoted for mentioning downvoting but totally agree. I see no connection between big data and AI. Like everything you can of course apply AI to it but I think step one is getting the analytic side down pat. And also agree thinking may not be hard. It is hard to create a thinking machine (Other than using DNA) but I don't necessary think there is anything special to it actually thinking. I'd be disappointed if Chomsky actually thought this way, would need context.
- pjmorris 13y ago
  Aiming to support the parent's claim (ElDiablo66, is this what you're referring to?), here's a quote from an article [1] about the Chomsky vs. Norvig debate a couple years ago: Chomsky critiqued the field of AI for adopting an approach reminiscent of behaviorism, except in more modern, computationally sophisticated form. Chomsky argued that the field's heavy use of statistical techniques to pick regularities in masses of data is unlikely to yield the explanatory insight that science ought to offer. For Chomsky, the "new AI" -- focused on using statistical learning techniques to better mine and predict data -- is unlikely to yield general principles about the nature of intelligent beings or about cognition. [1] http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/ http://www.theatlantic.com/technology/archive/2012/11/noam-c... [2] HN thread on [1]: https://news.ycombinator.com/item?id=4729068 https://news.ycombinator.com/item?id=4729068
STORMSPLITTER 13y ago
I enjoyed your post, nemesisj. Within your field of longitudinal patient data, if I am correct in what you have written, your large datasets simply have a new name, paranthetically Big Data, and that you could get what you need to save money without the newfangled algorithms. Within academic bioscience, I think there is great consensus on what Big Data is - I have not seen much argument at all; however, it is still very hard to define within that field. The best I can do, over this cup of coffee, is to state that there is a clear distinction between the study of a gene, up to a few pathways vs. computational analysis of multiple OMICS (genomics, metabolomics and proteomics) datasets. I know that definition is terribly lacking and I am fighting the urge to delete it for the sake of getting the post completed. Anyway, Big Data is clearly changing the academic biosciences through the funding trend. That is, grants with a computational focus, or sub focus, certainly seem to be doing comparably well. I mention this because todays academic funding trends influence the direction of tomorrows startups as those being trained are disproportionally within the better funded labs, and draw upon their previous experience when forming companies. So, I personally believe this Big Data thing, however it is best defined over all, or within a given field, is in some way something new, and will continue to shape the startup sphere for years to come, especially in the areas of genomics, metabolomics and proteomics. This is my first post: ) Thanks!
- roel_v 13y ago
  " This is my first post: )" I really don't mean to be snarky, but please try to use paragraphs. It'll make it much more likely people will read your post.
deleted 13y ago
[deleted]
nemetroid 13y ago
Re: copying from FT, if you're using Firefox you can set dom.event.clipboardevents.enabled to false to get around that. Will probably break copying in some web apps.
- greenyoda 13y ago
  NoScript also solves the problem, since they need to run JavaScript code to disable copying. (Their site displays just fine without JavaScript.)
netcan 13y ago
But, like "cloud" or "web 2.0" or any similar buzzword there obviously is some substance to it unspecific, un-novel, abused, or not. It just break into unsatisfying mush when you look at it to closely. Web 2.0 was some sort of a shift over web 1.0, the line between publisher and consumer melted. Cloud is etherealizing computing and data. There was a thread a few days ago about the film Her. "Where is Samantha" (the AI) is borderline nonsensical statement. It doesn't come up to a viewer. That's because people are used to cloud as an idea now It doesn't really matter that servers, replication, dumb clients, remote data or whatever were invented a long time ago.
owenversteeg 13y ago
They only add their preface if the text is greater or equal to 185 characters, it seems.
- salemh 13y ago
  I believe Tynt pioneered this copy/paste: http://www.tynt.com/product_copypaste http://www.tynt.com/product_copypaste http://daringfireball.net/2010/05/tynt_copy_paste_jerks http://daringfireball.net/2010/05/tynt_copy_paste_jerks
rbanffy 13y ago
Less than a month ago I was at an event about Big Data in marketing. One of the speakers spoke about how they used user data to improve their client's brand experience. If was very effective and I agree they did something extraordinary. I then asked what tools they used. He responded with a well-known relational database. I then asked the total size of their dataset, with a good idea of what the upper bounds would be. He responded "around 100 million events" since the product started, 6 months ago. It's really sad because they may end up under fire despite the effectiveness of their work. Big Data is a lot like teen sex.
- danieltillett 13y ago
  >Big Data is a lot like teen sex. LOL.
LambdaAlmighty 13y ago
I say, make hay while the sun shines, and make money while businesses throw them at vague hyped keywords. Such is the nature of primates: it's about social proofs and herd mentality. Efficient rationalism would be a dangerous chimera if you're after money.

JonLim 13y ago

Out of curiosity, when does it effectively become "big data"? I ask not to be snarky, but it might be the case that it's "big data" to someone else, but not necessarily to you. I figured it was a relative term for your industry/business, but the hacker crowd definitely seems to peg that amount in the millions of data points before calling it big data at all. Seems fair, but I'd rather clarify.

justincormack 13y ago
Big Data used to mean petabytes, ie above the limits of performant scale up.
ntoshev 13y ago
I believe the accepted definition for big data is data you can't handle on a single machine and need a cluster to process. So Moore's law makes it a moving threshold.
Malarkey73 13y ago
I think Hadley Wickham has a decent description of big data in terms of the analytical process... to expand slightly on his description: On normal data you can iteratively explore and visualise it hitting return and seeing plots or model results instantaneously or at most a few seconds. When you have time to grab a coffee after hitting return then you have bigger data. If you carefully think through what you are about to ask the computer to do before pressing return then maybe you have big data. I actually think this is a better description than just size of files or data distributed across many computers as an algorithm that just streams over a massive dataset maybe in parallel can be less challenging than one that has to hold a much smaller e.g many Gb dataset fully in memory.
- RogerL 13y ago
  So my complicated algorithm that processes 200,000 data points is big data because it takes 1/2 hour to run, but someone else's petabyte algorithm that takes 1 second on a cluster is small to "bigger" data? I don't think this makes sense. It's a measure of the size of the problem to be sure, but it is not a measure of the size of the data, or an indication of what techniques might be required to solve the problem.
  - deleted 13y ago
    [deleted]
  - Malarkey73 13y ago
    Yes to a data analyst that is big data. If you are doing some MCMC and that is really what it takes on that size of data then you have a big data problem. The more sophisticated a statistic, the more high dimensional the data, the more sampling required, or the more of the dataset it requires to memorise at once - then the smaller your big data threshold will be. It depends a lot on your point of view too. If I google something now it may bounce across lots of crazy server farms but to me I don't feel like I'm doing big data.. the person who built it all probably feels differently.

ElDiablo666 13y ago

Noam Chomsky had the best response to "big data": it's basically a nonsense concept (which I agree with) because "thinking is hard."

Houshalter 13y ago
Who said anything about thinking, and how do you know it's hard? EDIT: I'm getting downvoted, but your statement is incredibly vague and I believe wrong. "Big Data" might be overused as a buzzword, but it's not a "nonsense concept". "Thinking is hard", I assume you are talking about strong AI, and it's not related to this at all. Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.)
- coldtea 13y ago
  >Saying it's "hard" adds nothing of value, and we don't even know if it's true (in the sense that when someone does figure it out, it might seem simple and obvious in retrospect.) If it takes decades of hard working geniuses to figure it out, then even if "it seems simple and obvious in retrospect", it IS hard.
- aaron695 13y ago
  Downvoted for mentioning downvoting but totally agree. I see no connection between big data and AI. Like everything you can of course apply AI to it but I think step one is getting the analytic side down pat. And also agree thinking may not be hard. It is hard to create a thinking machine (Other than using DNA) but I don't necessary think there is anything special to it actually thinking. I'd be disappointed if Chomsky actually thought this way, would need context.
pjmorris 13y ago
Aiming to support the parent's claim (ElDiablo66, is this what you're referring to?), here's a quote from an article [1] about the Chomsky vs. Norvig debate a couple years ago: Chomsky critiqued the field of AI for adopting an approach reminiscent of behaviorism, except in more modern, computationally sophisticated form. Chomsky argued that the field's heavy use of statistical techniques to pick regularities in masses of data is unlikely to yield the explanatory insight that science ought to offer. For Chomsky, the "new AI" -- focused on using statistical learning techniques to better mine and predict data -- is unlikely to yield general principles about the nature of intelligent beings or about cognition. [1] http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/ http://www.theatlantic.com/technology/archive/2012/11/noam-c... [2] HN thread on [1]: https://news.ycombinator.com/item?id=4729068 https://news.ycombinator.com/item?id=4729068

STORMSPLITTER 13y ago

I enjoyed your post, nemesisj. Within your field of longitudinal patient data, if I am correct in what you have written, your large datasets simply have a new name, paranthetically Big Data, and that you could get what you need to save money without the newfangled algorithms. Within academic bioscience, I think there is great consensus on what Big Data is - I have not seen much argument at all; however, it is still very hard to define within that field. The best I can do, over this cup of coffee, is to state that there is a clear distinction between the study of a gene, up to a few pathways vs. computational analysis of multiple OMICS (genomics, metabolomics and proteomics) datasets. I know that definition is terribly lacking and I am fighting the urge to delete it for the sake of getting the post completed. Anyway, Big Data is clearly changing the academic biosciences through the funding trend. That is, grants with a computational focus, or sub focus, certainly seem to be doing comparably well. I mention this because todays academic funding trends influence the direction of tomorrows startups as those being trained are disproportionally within the better funded labs, and draw upon their previous experience when forming companies. So, I personally believe this Big Data thing, however it is best defined over all, or within a given field, is in some way something new, and will continue to shape the startup sphere for years to come, especially in the areas of genomics, metabolomics and proteomics. This is my first post: ) Thanks!

roel_v 13y ago
" This is my first post: )" I really don't mean to be snarky, but please try to use paragraphs. It'll make it much more likely people will read your post.

deleted 13y ago

[deleted]

nemetroid 13y ago

Re: copying from FT, if you're using Firefox you can set dom.event.clipboardevents.enabled to false to get around that. Will probably break copying in some web apps.

greenyoda 13y ago
NoScript also solves the problem, since they need to run JavaScript code to disable copying. (Their site displays just fine without JavaScript.)

netcan 13y ago

But, like "cloud" or "web 2.0" or any similar buzzword there obviously is some substance to it unspecific, un-novel, abused, or not. It just break into unsatisfying mush when you look at it to closely. Web 2.0 was some sort of a shift over web 1.0, the line between publisher and consumer melted. Cloud is etherealizing computing and data. There was a thread a few days ago about the film Her. "Where is Samantha" (the AI) is borderline nonsensical statement. It doesn't come up to a viewer. That's because people are used to cloud as an idea now It doesn't really matter that servers, replication, dumb clients, remote data or whatever were invented a long time ago.

owenversteeg 13y ago

They only add their preface if the text is greater or equal to 185 characters, it seems.

salemh 13y ago
I believe Tynt pioneered this copy/paste: http://www.tynt.com/product_copypaste http://www.tynt.com/product_copypaste http://daringfireball.net/2010/05/tynt_copy_paste_jerks http://daringfireball.net/2010/05/tynt_copy_paste_jerks

rbanffy 13y ago

Less than a month ago I was at an event about Big Data in marketing. One of the speakers spoke about how they used user data to improve their client's brand experience. If was very effective and I agree they did something extraordinary. I then asked what tools they used. He responded with a well-known relational database. I then asked the total size of their dataset, with a good idea of what the upper bounds would be. He responded "around 100 million events" since the product started, 6 months ago. It's really sad because they may end up under fire despite the effectiveness of their work. Big Data is a lot like teen sex.

danieltillett 13y ago
>Big Data is a lot like teen sex. LOL.

LambdaAlmighty 13y ago

I say, make hay while the sun shines, and make money while businesses throw them at vague hyped keywords. Such is the nature of primates: it's about social proofs and herd mentality. Efficient rationalism would be a dangerous chimera if you're after money.