5 ms·

I seem to remember a blog post one day about someone interviewing for several "Big Data" positions - where the data turned out to be some Excel sheets. Or fit o

I seem to remember a blog post one day about someone interviewing for several "Big Data" positions - where the data turned out to be some Excel sheets. Or fit on a thumb drive. The hiring companies that thought they had "big data" really didn't, was the point. The blog post was widely shared around my office at the time and I see it as the origin of this idea. Edit: I think it was https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html https://www.chrisstucchio.com/blog/2013/hadoop_hatred.html, HN discussion https://news.ycombinator.com/item?id=6398650 https://news.ycombinator.com/item?id=6398650 But the idea wasn't that data was big as soon as it didn't fit in Excel anymore - Excel was a reductio ad absurdum. If it still fits in Excel, it's laughably far away from being "big". That was the idea. Big data was about the problems you get when you need to join data together that you can't fit well in one database server, not even with the amounts of memory and disk space you can get these days. It was Google type problems, the stuff that map-reduce needed to be invented for. The kind of problem that most companies just don't have. So the term mostly disappeared from job descriptions, "data science" became popular instead, and people keep using Excel because it is great. (that blog post is from 2013 so it can't be the source of that quote from 2012. But please don't define big data in terms of Excel in a 2021 MSc thesis)

edent 6y ago
Thanks for that link. Interesting that it was written a year-or-so after the phrase became popular.
hkmurakami 6y ago
Friend in one of the big data vendors: "we mostly work with medium data"
- nerdponx 6y ago
  The way I always think of it: Medium data fits on disk but not in memory. Big data doesn't fit in either.
  - alexchamberlain 6y ago
    I like this definition too; the advantage is it scales with advances in technology. What has medium a decade ago, might be considered small (ish) now.
  - Scarblac 6y ago
    But people underestimate how much memory you can put in a server, and for how long it stays cost effective to do so.
moksly 6y ago
I work for an average sized Danish city. Even when we combined our datasets with 10 other average sized cities and accumulate the biggest national data set and let the big data wizards have a go at it, it wasn’t really large enough to amount to anything useful. Don’t get me wrong, Watson came up with a lot of BI suggestions that were useful and some of the scientists came up with semi-interesting prediction models. The thing is though, our analytics team has much better BI models and the our finance department does the prediction much more efficient and, well, legal. The prediction could become a useful aid, if it ever became legal, but not really at the license fees we were looking at. Not sure if automated BI is ever going to become good enough. It’s impressive that Watson can come up with stuff an university post-bachelor intern can, but it’s license is more than our entire analytics team, so yeah...
- Scarblac 6y ago
  I work in water management in the Netherlands and it feels the same. A human domain expert knows what kind of thing to look for. Custom tools can help to get an overview (eg we visualize water velocities of a whole regional system on a map), but generic machine learning can't add much even though we record basically everything everywhere at five minute intervals. The value of ML is perhaps in things that give little value in the individual case but can be used a huge number of times, like image classification.
  - pbhjpbhj 6y ago
    Human directed machine learning then? Presumably a domain expert who can also expertly leverage ML/AI could make some useful tools?
    - moksly 6y ago
      There are no such people though.
  - bostik 6y ago
    I can give one example where the ML/AI toolkit is useful, even if only the largest providers in the world may need to work with real Big Data[tm] volumes. Fraud detection. And you don't need to be a particularly big operator to get sufficiently large & varied data sets. In a way I think it helps that fraud is an ever-evolving, fiercely adversarial domain. New avenues are explored all the time. Occasionally old tricks are revived for a while, because they may work in the margins but become distinguishing features as soon as they see more use. And even if your ML is based on nothing more than a random forest, you can still get surprisingly useful feature combinations out of it. (Or as our data scientists said: individually meaningless features may become a valuable signal when enough of them occur at the same time.)
simias 6y ago
I think it's kind of pointless to fight for an exact meaning for these buzzwords. "Big Data", "Webscale", "Web 2.0", "NoSQL" and whatever other trendy hashtag du jour have lost most of their meaning because they hardly had any to begin with. They're marking drivel, not technical terms. And since everybody wants to look cool we all find ways to say that we're doing it, one way or the other.
- klodolph 6y ago
  Other than “webscale” I think the other terms still mean something. They don’t need to have a precise, agreed upon meaning to be useful words. “Webscale” is just a buzzword, though.
  - Quarrelsome 6y ago
    Sorry I can't resist: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
    - srswtf123 6y ago
      This video is one of the best critiques of modern software development, and I suppose IT in general, that I've ever seen. Even if I've seen it a hundred times, I always appreciate seeing it again. Thank you!
    - numpad0 6y ago
      I have suspicion that a lot of people who jumped to NoSQL just didn’t like the aesthetics of SQL syntax, in the same sense as people would not be ecstatic about Visual Basic.
- stadium 6y ago
  NoSQL was jargon speak for document store. It's really schema on read vs the traditional database where schema is defined on (or before) write. And there are plenty of ways to use sql on various data formats including json, presto being one. Around the time of peak NoSQL buzz, I went to Oracle Open world in the early-mid 2010's and there was an army of MongoDB folks outside the convention center grounds holding signs and handing out swag. NoSQL was definitely a good marketing play to try and take customers away from Oracle.
deleted 6y ago
[deleted]
GoodDreams 6y ago
A company I worked for defined big data as anything not analyzable with a relational database - a PDF or video was big data. Companies say they want the latest buzzword even though it’s against their own interest and other companies say “we can give that to you” and instead deliver something that is of actual value so the license gets renewed. Same thing happened with SOA, REST, microservices, blockchain. IBM developed a blockchain product just to get the sales conversations going and then sold something else. Imagine being on that product’s team.
- deleted 6y ago
  [deleted]
- saalweachter 6y ago
  I would go with "big data is when you have to spend more effort on the mechanism of performing the computation than on the details of the analysis itself."
cortesoft 6y ago
Luckily that blog post is out of date and there are a lot of options now to handle datasets larger than 5TB without needing Hadoop.
- jleahy 6y ago
  Awk, for example.
snidane 6y ago
Since the new breed of "data scientists" using R and Python dataframes replaced Excel "data analysts" they realized the "big" of big data hasn't disappeared and started hiring data engineers for developing and explaining the new big data platforms for stuff which didn't fit into their pandas dataframes. Still, some call it medium data since this spans the range of higher gigabytes to lower terabytes. There are still couple of business problems in the petabyte scale, but they are probably so niche that they deserve a custom solution anyway.
tragomaskhalos 6y ago
Kudos for that clarification. Years ago we did a bit of consultancy at a fashion company who drove their biannual fashion releases via data held in Excel, and their pain point was that even scrolling through the rows was agonisingly slow. Clearly all those people needed was either better desktop machines and/or an RDBMS, not an entirely new technology !
- treeman79 6y ago
  Many Years I had to drive an hour to help someone that had a corrupt CSV file. Column needed to made a little wider so it show the photo number properly. The person had a masters in CS.
treeman79 6y ago
Excel and business and computer science gets weird. I’ve seen companies spending half a million to build a report that excel and a pivot table could do in an a day. Few weeks ago company was looking at scaling up to max possible EC2 nodes in order to get something to work. I spent a couple hours tweaking the algorithm and now it’s on a micro node running 20x faster. Some data problems are truly hard and interesting and world changing. Excel covers almost everything else.

edent 6y ago

Thanks for that link. Interesting that it was written a year-or-so after the phrase became popular.

hkmurakami 6y ago

Friend in one of the big data vendors: "we mostly work with medium data"

nerdponx 6y ago
The way I always think of it: Medium data fits on disk but not in memory. Big data doesn't fit in either.
- alexchamberlain 6y ago
  I like this definition too; the advantage is it scales with advances in technology. What has medium a decade ago, might be considered small (ish) now.
- Scarblac 6y ago
  But people underestimate how much memory you can put in a server, and for how long it stays cost effective to do so.

moksly 6y ago

I work for an average sized Danish city. Even when we combined our datasets with 10 other average sized cities and accumulate the biggest national data set and let the big data wizards have a go at it, it wasn’t really large enough to amount to anything useful. Don’t get me wrong, Watson came up with a lot of BI suggestions that were useful and some of the scientists came up with semi-interesting prediction models. The thing is though, our analytics team has much better BI models and the our finance department does the prediction much more efficient and, well, legal. The prediction could become a useful aid, if it ever became legal, but not really at the license fees we were looking at. Not sure if automated BI is ever going to become good enough. It’s impressive that Watson can come up with stuff an university post-bachelor intern can, but it’s license is more than our entire analytics team, so yeah...

Scarblac 6y ago
I work in water management in the Netherlands and it feels the same. A human domain expert knows what kind of thing to look for. Custom tools can help to get an overview (eg we visualize water velocities of a whole regional system on a map), but generic machine learning can't add much even though we record basically everything everywhere at five minute intervals. The value of ML is perhaps in things that give little value in the individual case but can be used a huge number of times, like image classification.
- pbhjpbhj 6y ago
  Human directed machine learning then? Presumably a domain expert who can also expertly leverage ML/AI could make some useful tools?
  - moksly 6y ago
    There are no such people though.
- bostik 6y ago
  I can give one example where the ML/AI toolkit is useful, even if only the largest providers in the world may need to work with real Big Data[tm] volumes. Fraud detection. And you don't need to be a particularly big operator to get sufficiently large & varied data sets. In a way I think it helps that fraud is an ever-evolving, fiercely adversarial domain. New avenues are explored all the time. Occasionally old tricks are revived for a while, because they may work in the margins but become distinguishing features as soon as they see more use. And even if your ML is based on nothing more than a random forest, you can still get surprisingly useful feature combinations out of it. (Or as our data scientists said: individually meaningless features may become a valuable signal when enough of them occur at the same time.)

simias 6y ago

I think it's kind of pointless to fight for an exact meaning for these buzzwords. "Big Data", "Webscale", "Web 2.0", "NoSQL" and whatever other trendy hashtag du jour have lost most of their meaning because they hardly had any to begin with. They're marking drivel, not technical terms. And since everybody wants to look cool we all find ways to say that we're doing it, one way or the other.

klodolph 6y ago
Other than “webscale” I think the other terms still mean something. They don’t need to have a precise, agreed upon meaning to be useful words. “Webscale” is just a buzzword, though.
- Quarrelsome 6y ago
  Sorry I can't resist: https://www.youtube.com/watch?v=b2F-DItXtZs https://www.youtube.com/watch?v=b2F-DItXtZs
  - srswtf123 6y ago
    This video is one of the best critiques of modern software development, and I suppose IT in general, that I've ever seen. Even if I've seen it a hundred times, I always appreciate seeing it again. Thank you!
  - numpad0 6y ago
    I have suspicion that a lot of people who jumped to NoSQL just didn’t like the aesthetics of SQL syntax, in the same sense as people would not be ecstatic about Visual Basic.
stadium 6y ago
NoSQL was jargon speak for document store. It's really schema on read vs the traditional database where schema is defined on (or before) write. And there are plenty of ways to use sql on various data formats including json, presto being one. Around the time of peak NoSQL buzz, I went to Oracle Open world in the early-mid 2010's and there was an army of MongoDB folks outside the convention center grounds holding signs and handing out swag. NoSQL was definitely a good marketing play to try and take customers away from Oracle.

deleted 6y ago

[deleted]

GoodDreams 6y ago

A company I worked for defined big data as anything not analyzable with a relational database - a PDF or video was big data. Companies say they want the latest buzzword even though it’s against their own interest and other companies say “we can give that to you” and instead deliver something that is of actual value so the license gets renewed. Same thing happened with SOA, REST, microservices, blockchain. IBM developed a blockchain product just to get the sales conversations going and then sold something else. Imagine being on that product’s team.

deleted 6y ago
[deleted]
saalweachter 6y ago
I would go with "big data is when you have to spend more effort on the mechanism of performing the computation than on the details of the analysis itself."

cortesoft 6y ago

Luckily that blog post is out of date and there are a lot of options now to handle datasets larger than 5TB without needing Hadoop.

jleahy 6y ago
Awk, for example.

snidane 6y ago

Since the new breed of "data scientists" using R and Python dataframes replaced Excel "data analysts" they realized the "big" of big data hasn't disappeared and started hiring data engineers for developing and explaining the new big data platforms for stuff which didn't fit into their pandas dataframes. Still, some call it medium data since this spans the range of higher gigabytes to lower terabytes. There are still couple of business problems in the petabyte scale, but they are probably so niche that they deserve a custom solution anyway.

tragomaskhalos 6y ago

Kudos for that clarification. Years ago we did a bit of consultancy at a fashion company who drove their biannual fashion releases via data held in Excel, and their pain point was that even scrolling through the rows was agonisingly slow. Clearly all those people needed was either better desktop machines and/or an RDBMS, not an entirely new technology !

treeman79 6y ago
Many Years I had to drive an hour to help someone that had a corrupt CSV file. Column needed to made a little wider so it show the photo number properly. The person had a masters in CS.

treeman79 6y ago

Excel and business and computer science gets weird. I’ve seen companies spending half a million to build a report that excel and a pivot table could do in an a day. Few weeks ago company was looking at scaling up to max possible EC2 nodes in order to get something to work. I spent a couple hours tweaking the algorithm and now it’s on a micro node running 20x faster. Some data problems are truly hard and interesting and world changing. Excel covers almost everything else.