5 ms·

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real wor

This article's primary example is a single static text file with 5M lines. Sure, in that case, awk works great, but how often does that come up? In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. Further, users want real-time results, not just a few times a day when your cron script runs. Unix commands are great up to a few GBs of data, Excel is even better if you're dealing with less than a few tens of MBs. But to deal with Terabytes of data quickly and efficiently, these tools totally break down.

aidos 8y ago
Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.
- speedplane 8y ago
  > Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.
  - dredmorbius 8y ago
    Sampling or subetting your data is almost always the correct response, especially for analysis. If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.
  - hashhar 8y ago
    I have a simple bash script composed of 6 sed commands piped together to convert >100GB csvs (table dumps of recommendation data) into Redis binary protocol which are then ingested into a Redis Cluster using redis-cli --pipe. It takes somewhere around 15 minutes running on a modest bare metal server.
  - geofft 8y ago
    > Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). This doesn't follow: 1) One reason to run a cluster is to use flaky commodity hardware instead of high-reliability specialized hardware. Note the transition from specialized hardware and computer rooms of the '80s and '90s to cloud computing on preemptible instances. 2) They might just be straight-up wrong / misguided. http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... As a professional developer I've seen tons of distributed systems that could be replaced by a well-designed non-distributed system, but ELK and Mongo and etcd and Kafka and more generally cloud servers are easy off-the-shelf tools.
  - zaphar 8y ago
    Just a few weeks ago I was working with an 11TB dataset which I processed with just unix command line tools told on a single Linux VM. You can get a lot of milage out of parallel xargs and other of the shelf tools.
    - srean 8y ago
      Indeed! Gnu prallel, xargs, make -j, netcat, mawk/gawk (with C extensions if needed), jq, coreutils, textutils can get insane amount of stuff done.
JoeSmithson 8y ago
> how often does that come up? It's an important point to remember that a lot of things involved in human society have not exploded in size or complexity in the last 30 years. Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has moved from the millions into the billions. It's important, because if a government needs to, say, calculate something involving "every building in the country", or "everybody with a criminal record" they need to understand that this task, in 2019, can in fact be done by a single programmer parsing flat text files on their MBP, and does not need a new department. This is a bit like Grace Hopper always pointing out the difference between a microsecond and a nanosecond - https://www.youtube.com/watch?v=JEpsKnWZrJ8 https://www.youtube.com/watch?v=JEpsKnWZrJ8
- noir_lord 8y ago
  > Many data sets are basically proportional to the human population. Awesome point well put.
- 12298765 8y ago
  I've never seen that Grace Hopper bit, but that's awesome! Thanks for sharing. Definitely going to show it to some co-workers
curiousgal 8y ago
> Sure, in that case, awk works great, but how often does that come up? 80% of bioinformatics.
josteink 8y ago
> In the real world, those 5M lines are growing by several hundred thousand every day Or they don’t. Not all data are cloud-scale aggregations.
srean 8y ago
> But to deal with Terabytes of data quickly and efficiently, these tools totally break down Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it. Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.
patrickg_zill 8y ago
Why would you need to look at the entire history of the log file each time? Isn't it logical to have historical summary info and then the full log for say the past month?
- speedplane 8y ago
  > Isn't it logical to have historical summary info and then the full log for say the past month? What if you want to change what's contained in your summary?
geofft 8y ago
> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."
walshemj 8y ago
Users "want" vs do users "need" is the Q. Also most people don't deal with terabyte sized data sets
redsavagefiero 8y ago
In my world your case is the corner case. Don't think I want to deal with multiple terabytes of data as it's provenance is questionable and this data services industries that the human population did without until about ten/fifteen years ago.

aidos 8y ago

Well, the article does explicitly say that if you can do this then you don’t have “big data”. Maybe you see TB level processing a lot in your line of work, but most developers never will. Whenever I deal with anything a bit bigger, I break off the smallest section I need to deal with and work with that.

speedplane 8y ago
> Maybe you see TB level processing a lot in your line of work, but most developers never will. I doubt this is the case. If you're working on even a medium sized team, you'll see this level of data, whether it's internal server logs, publicly-sourced data, or a variety of other applications. Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). I don't know the percentage of developers that deal with this, but I do know that it's pretty common.
- dredmorbius 8y ago
  Sampling or subetting your data is almost always the correct response, especially for analysis. If you're literally processing TB or PB of data, you'll want to parallelise. Though shell tools do this amazingly well.
- hashhar 8y ago
  I have a simple bash script composed of 6 sed commands piped together to convert >100GB csvs (table dumps of recommendation data) into Redis binary protocol which are then ingested into a Redis Cluster using redis-cli --pipe. It takes somewhere around 15 minutes running on a modest bare metal server.
- geofft 8y ago
  > Almost by definition, any company that runs a cluster is probably dealing with TBs of data (otherwise they could probably run it on one machine). This doesn't follow: 1) One reason to run a cluster is to use flaky commodity hardware instead of high-reliability specialized hardware. Note the transition from specialized hardware and computer rooms of the '80s and '90s to cloud computing on preemptible instances. 2) They might just be straight-up wrong / misguided. http://www.frankmcsherry.org/graph/scalability/cost/2015/01/15/COST.html http://www.frankmcsherry.org/graph/scalability/cost/2015/01/... As a professional developer I've seen tons of distributed systems that could be replaced by a well-designed non-distributed system, but ELK and Mongo and etcd and Kafka and more generally cloud servers are easy off-the-shelf tools.
- zaphar 8y ago
  Just a few weeks ago I was working with an 11TB dataset which I processed with just unix command line tools told on a single Linux VM. You can get a lot of milage out of parallel xargs and other of the shelf tools.
  - srean 8y ago
    Indeed! Gnu prallel, xargs, make -j, netcat, mawk/gawk (with C extensions if needed), jq, coreutils, textutils can get insane amount of stuff done.

JoeSmithson 8y ago

> how often does that come up? It's an important point to remember that a lot of things involved in human society have not exploded in size or complexity in the last 30 years. Many data sets are basically proportional to the human population (health records, criminal records, property records etc), and these have been measured in the millions for 30+ years. In the same time the compute power of a single script has moved from the millions into the billions. It's important, because if a government needs to, say, calculate something involving "every building in the country", or "everybody with a criminal record" they need to understand that this task, in 2019, can in fact be done by a single programmer parsing flat text files on their MBP, and does not need a new department. This is a bit like Grace Hopper always pointing out the difference between a microsecond and a nanosecond - https://www.youtube.com/watch?v=JEpsKnWZrJ8 https://www.youtube.com/watch?v=JEpsKnWZrJ8

noir_lord 8y ago
> Many data sets are basically proportional to the human population. Awesome point well put.
12298765 8y ago
I've never seen that Grace Hopper bit, but that's awesome! Thanks for sharing. Definitely going to show it to some co-workers

curiousgal 8y ago

> Sure, in that case, awk works great, but how often does that come up? 80% of bioinformatics.

josteink 8y ago

> In the real world, those 5M lines are growing by several hundred thousand every day Or they don’t. Not all data are cloud-scale aggregations.

srean 8y ago

> But to deal with Terabytes of data quickly and efficiently, these tools totally break down Scaling up to 400~500GB of logs with awk and parallel has not been a problem for me. I dont think TB will be particularly hard, especially if one is reasonably proficient with the tools. Of course, if one has the mental bias that one has to throw hadoop or spark at it, thats a significant obstacle right there. Upton Sinclair effect also plays a role -- It is difficult to get a man to understand something, when his salary depends upon his not understanding it. Of course at some scale simple unix tools become impractical, but usually people reach for flashier tools even at scales where unix tools will suffice.

patrickg_zill 8y ago

Why would you need to look at the entire history of the log file each time? Isn't it logical to have historical summary info and then the full log for say the past month?

speedplane 8y ago
> Isn't it logical to have historical summary info and then the full log for say the past month? What if you want to change what's contained in your summary?

geofft 8y ago

> In the real world, those 5M lines are growing by several hundred thousand every day, and after a few months, grows beyond what a single computer or awk can handle. This is a frustrating viewpoint, and it feels to me like the same poisonous worldview as "If your company isn't growing 10% month-over-month, it's not worthwhile and we won't invest in it." There are plenty of meaningful and useful things you can do for the world at a static size, and doing continued interesting work with a dataset of the same size doesn't mean you're not part of the "real world."

walshemj 8y ago

Users "want" vs do users "need" is the Q. Also most people don't deal with terabyte sized data sets

redsavagefiero 8y ago

In my world your case is the corner case. Don't think I want to deal with multiple terabytes of data as it's provenance is questionable and this data services industries that the human population did without until about ten/fifteen years ago.