5 ms·

A programmer’s guide to big data: tools to know

79 points by mwetzler 14y ago

hayksaakian 14y ago
I still don't understand big data. Is it machine learning + analytics?
- mwetzler 14y ago
  I think of it as "trying to capture massive amounts of data and somehow get useful information out of it". This is pretty good: http://en.wikipedia.org/wiki/Big_data http://en.wikipedia.org/wiki/Big_data
- mdellabitta 14y ago
  It's having a dataset that's large enough and/or complex enough that it can't really be processed on a single machine in a reasonable amount of time.
- pnathan 14y ago
  No, it's not, although it seems to usually be linked with analytics. It's also things like sampling the power grid, photos from astronomy, and genomic data. But unfortunately those sorts of things don't seem to get the same kind of buzz.
- paulgb 14y ago
  When the press uses it, it means "data".
- TallGuyShort 14y ago
  Although I believe "big" usually refers to the size of the dataset, I originally heard the term used in the same sense as "Big Oil" - and the implication was that as companies collected massive amounts of data and figured out how to profit from it, "data" was going to become the next high-value commodity. edit: Although to answer your actual question, it's a bit like "cloud computing" - it probably refers to very scalable systems with an emphasis on reading, writing, and processing large amounts of data - but really it's a bit of a marketing term :)
  - mnicole 14y ago
    To branch off of your comment, which is the best description of it, I think it should be noted that for the past two years this subject has been the most nominated topic across all of our events for both information and security officers (the company I work for empowers F1000 C-suite executives to come up with relevant material for their peers to discuss annually). For companies that are inundated with web analytics, feedback they're getting from and trends they're seeing on social media to sales numbers and any other quantifiable data, they're trying to tie it all together so they can make changes or create new opportunities for their markets.
    - hayksaakian 14y ago
      Oh OK,that makes more sense. Big data is actually the problem then, not the solution? Meaning there's too much disparate data, and now we can try to bring it together.
      - mnicole 14y ago
        Yup!
- kirinan 14y ago
  Some big data is machine learning, a lot however is running analytics via software like Hadoop (map reduce) to gleam results from the data. Examples of both would be calculating shipping times based on previous shipping times. If you had enough data, you could predict with analytics how long a shipping process would take. You can then use machine learning to get more and more accurate results.
- dbecker 14y ago
  Unfortunately, it's a buzzword that marketers use however is convenient. The most common definition refers to the size of the dataset, but I've heard it used by so-called-experts in reference to some very small datasets.
- dkarl 14y ago
  Michael Stonebraker's answer is that big data can mean big volume (and small analytics,) big volume and big analytics, big velocity, and/or big variety. The first three aspects fit the general usage on HN, while the fourth is his own addition, I think. He's writing a blog post on each one. The first post explains what he means by each aspect: Intro plus the "big volume, small analytics" case: http://cacm.acm.org/blogs/blog-cacm/155468-what-does-big-data-mean/fulltext http://cacm.acm.org/blogs/blog-cacm/155468-what-does-big-dat... "Big volume, big analytics": http://cacm.acm.org/blogs/blog-cacm/156102-what-does-big-data-mean-part-2/fulltext http://cacm.acm.org/blogs/blog-cacm/156102-what-does-big-dat... "Big velocity": http://cacm.acm.org/blogs/blog-cacm/157589-what-does-big-data-mean-part-3/fulltext http://cacm.acm.org/blogs/blog-cacm/157589-what-does-big-dat... "Big variety": yet to come
- elchief 14y ago
  It's when you use more than 2 computers to make a chart.
briandoll 14y ago
Here's another list, sans product hype: Unix command-line tools like awk and grep. Set operation commands are essential too (http://www.catonmat.net/blog/set-operations-in-unix-shell/ http://www.catonmat.net/blog/set-operations-in-unix-shell/) Ruby/Python/Perl for more complex massaging and wrangling Excel (yes, really) for quick stats and graphs, great 1st step in understanding what you have D3.js for visualization I've used R in the past, but I found I was trying to squeeze data into R models. D3 is pretty hard to grasp initially (I'm very much still learning) but I'm finding that it's helping me think through how to visualize the data I have, rather than just forcing it into one of a few standard charts.
- walshemj 14y ago
  Excell breaks badly for trivially small data sets
  - briandoll 14y ago
    I tend to aggregate data into something manageable before I attempt to graph it, regardless of what tool I'm using. If you can deal with daily numbers, you only need 365 rows per year, and less than 9k per year if you're doing hourly numbers.
- vtuulos 14y ago
  Unix command-line tools are seriously undervalued for processing data, even big data. Need parallelization? Try xargs -P on a multi-core machine.
  - chernevik 14y ago
    Don't forget cut, sort and uniq. If you have a big csv file, you could strip out all duplicate values in the fourth fields: cut -d , -f 4 [file] | sort | uniq > values.txt You could then grep out the records with the particular values of interest. If a file is too big for an editor, you can still see the record structure with head.
- Cieplak 14y ago
elchief 14y ago
I don't get it. Why not just learn Hive (it's SQL), or use python with Hadoop.

79 points by mwetzler 14y ago

hayksaakian 14y ago

I still don't understand big data. Is it machine learning + analytics?

mwetzler 14y ago
I think of it as "trying to capture massive amounts of data and somehow get useful information out of it". This is pretty good: http://en.wikipedia.org/wiki/Big_data http://en.wikipedia.org/wiki/Big_data
mdellabitta 14y ago
It's having a dataset that's large enough and/or complex enough that it can't really be processed on a single machine in a reasonable amount of time.
pnathan 14y ago
No, it's not, although it seems to usually be linked with analytics. It's also things like sampling the power grid, photos from astronomy, and genomic data. But unfortunately those sorts of things don't seem to get the same kind of buzz.
paulgb 14y ago
When the press uses it, it means "data".
TallGuyShort 14y ago
Although I believe "big" usually refers to the size of the dataset, I originally heard the term used in the same sense as "Big Oil" - and the implication was that as companies collected massive amounts of data and figured out how to profit from it, "data" was going to become the next high-value commodity. edit: Although to answer your actual question, it's a bit like "cloud computing" - it probably refers to very scalable systems with an emphasis on reading, writing, and processing large amounts of data - but really it's a bit of a marketing term :)
- mnicole 14y ago
  To branch off of your comment, which is the best description of it, I think it should be noted that for the past two years this subject has been the most nominated topic across all of our events for both information and security officers (the company I work for empowers F1000 C-suite executives to come up with relevant material for their peers to discuss annually). For companies that are inundated with web analytics, feedback they're getting from and trends they're seeing on social media to sales numbers and any other quantifiable data, they're trying to tie it all together so they can make changes or create new opportunities for their markets.
  - hayksaakian 14y ago
    Oh OK,that makes more sense. Big data is actually the problem then, not the solution? Meaning there's too much disparate data, and now we can try to bring it together.
    - mnicole 14y ago
      Yup!
kirinan 14y ago
Some big data is machine learning, a lot however is running analytics via software like Hadoop (map reduce) to gleam results from the data. Examples of both would be calculating shipping times based on previous shipping times. If you had enough data, you could predict with analytics how long a shipping process would take. You can then use machine learning to get more and more accurate results.
dbecker 14y ago
Unfortunately, it's a buzzword that marketers use however is convenient. The most common definition refers to the size of the dataset, but I've heard it used by so-called-experts in reference to some very small datasets.
dkarl 14y ago
Michael Stonebraker's answer is that big data can mean big volume (and small analytics,) big volume and big analytics, big velocity, and/or big variety. The first three aspects fit the general usage on HN, while the fourth is his own addition, I think. He's writing a blog post on each one. The first post explains what he means by each aspect: Intro plus the "big volume, small analytics" case: http://cacm.acm.org/blogs/blog-cacm/155468-what-does-big-data-mean/fulltext http://cacm.acm.org/blogs/blog-cacm/155468-what-does-big-dat... "Big volume, big analytics": http://cacm.acm.org/blogs/blog-cacm/156102-what-does-big-data-mean-part-2/fulltext http://cacm.acm.org/blogs/blog-cacm/156102-what-does-big-dat... "Big velocity": http://cacm.acm.org/blogs/blog-cacm/157589-what-does-big-data-mean-part-3/fulltext http://cacm.acm.org/blogs/blog-cacm/157589-what-does-big-dat... "Big variety": yet to come
elchief 14y ago
It's when you use more than 2 computers to make a chart.

briandoll 14y ago

Here's another list, sans product hype: Unix command-line tools like awk and grep. Set operation commands are essential too (http://www.catonmat.net/blog/set-operations-in-unix-shell/ http://www.catonmat.net/blog/set-operations-in-unix-shell/) Ruby/Python/Perl for more complex massaging and wrangling Excel (yes, really) for quick stats and graphs, great 1st step in understanding what you have D3.js for visualization I've used R in the past, but I found I was trying to squeeze data into R models. D3 is pretty hard to grasp initially (I'm very much still learning) but I'm finding that it's helping me think through how to visualize the data I have, rather than just forcing it into one of a few standard charts.

walshemj 14y ago
Excell breaks badly for trivially small data sets
- briandoll 14y ago
  I tend to aggregate data into something manageable before I attempt to graph it, regardless of what tool I'm using. If you can deal with daily numbers, you only need 365 rows per year, and less than 9k per year if you're doing hourly numbers.
vtuulos 14y ago
Unix command-line tools are seriously undervalued for processing data, even big data. Need parallelization? Try xargs -P on a multi-core machine.
- chernevik 14y ago
  Don't forget cut, sort and uniq. If you have a big csv file, you could strip out all duplicate values in the fourth fields: cut -d , -f 4 [file] | sort | uniq > values.txt You could then grep out the records with the particular values of interest. If a file is too big for an editor, you can still see the record structure with head.
Cieplak 14y ago

elchief 14y ago

I don't get it. Why not just learn Hive (it's SQL), or use python with Hadoop.