4 ms·

Here's another list, sans product hype: Unix command-line tools like awk and grep. Set operation commands are essential too (http://www.catonmat.net/blog/set-o

Here's another list, sans product hype: Unix command-line tools like awk and grep. Set operation commands are essential too (http://www.catonmat.net/blog/set-operations-in-unix-shell/ http://www.catonmat.net/blog/set-operations-in-unix-shell/) Ruby/Python/Perl for more complex massaging and wrangling Excel (yes, really) for quick stats and graphs, great 1st step in understanding what you have D3.js for visualization I've used R in the past, but I found I was trying to squeeze data into R models. D3 is pretty hard to grasp initially (I'm very much still learning) but I'm finding that it's helping me think through how to visualize the data I have, rather than just forcing it into one of a few standard charts.

walshemj 14y ago
Excell breaks badly for trivially small data sets
- briandoll 14y ago
  I tend to aggregate data into something manageable before I attempt to graph it, regardless of what tool I'm using. If you can deal with daily numbers, you only need 365 rows per year, and less than 9k per year if you're doing hourly numbers.
vtuulos 14y ago
Unix command-line tools are seriously undervalued for processing data, even big data. Need parallelization? Try xargs -P on a multi-core machine.
- chernevik 14y ago
  Don't forget cut, sort and uniq. If you have a big csv file, you could strip out all duplicate values in the fourth fields: cut -d , -f 4 [file] | sort | uniq > values.txt You could then grep out the records with the particular values of interest. If a file is too big for an editor, you can still see the record structure with head.
Cieplak 14y ago
If you need to lots of one-off visualizations, consider Tableau (http://www.tableausoftware.com/ http://www.tableausoftware.com/). Although not as versatile as D3.js and although it's proprietary, it is really easy, essentially all click and point. My firm started using it recently and I am really jealous because MBA types can in seconds create visualizations that take me hours to do with D3.js. It supports a wide variety of data inputs, including excel and various databases. Also, Tableau has excellent support for working with maps and shapefiles.
- j_s 14y ago
  I stumbled across Gephi ('an open source graph visualization and manipulation software') and got a few comments on it about 9 months ago: http://news.ycombinator.com/item?id=3700471 http://news.ycombinator.com/item?id=3700471 https://gephi.org/ https://gephi.org/
notJim 14y ago
I feel like if you're using those tools to process raw data (and not the data produced by another job), it's not really "big" data. I would tend to define big data as data where there's so much volume that processing it in this way is not possible. Which isn't to say this article isn't full of hype—it clearly is.
dxbydt 14y ago
This isn't big data at all. This is stuff you'd use AFTER you've processed your big data & obtained a reduced dataset result that was small data, of the order of a few hundred MB. To process said big data in the first place, of the order of a few PB, typically you are going to use hdfs & at a programmatic level, an api like cascading or better yet, scalding, so you end up needing only that last item D3.js , with everything else existing as scalding source.
- saosebastiao 14y ago
  This is where "Big Data" becomes about as useful as a penis measuring contest. The vast majority of business data analysis happens at MS Excel levels of data. I have done analysis on several GBs of data using R, and those data-sets are easily the 99th percentile of data size in my field. For datasets that are bigger than that, SQL is still pretty damn good. And when you get to the point where SQL starts to break down in usefulness, you are at a level where you can start sampling and still get perfectly usable results in almost all use cases. In my experience, "Big Data" is a misnomer. What it really means is fast data. You have a repeatable calculation to perform on datasets that are perfectly fine chugging away on an Oracle cluster for 3 hours, but you want it done in 20 minutes. That is what Big Data is really used for. Anything else is just marketing hype.
  - bravura 14y ago
    No. Big data has a formal definition, but one that most people ignore: When the size of the data becomes part of the problem. (An O'Reilly author said this once, I forget his name.) For example, physicists in the 80's who had 10s of MBs of data had a big data problem. Nowadays, this typically means out-of-core data sets.
    - saosebastiao 14y ago
      That definition is but one of many supposed formal definitions. And it isn't a particularly good one, because then "Big Data" becomes a definition that can only be quantified on a person-level of granularity. According to the guy with the Marketing degree from Ho-Hum College, anything bigger than can fit in an Excel spreadsheet is "Big Data". Switch to Doug Cutting, and "Big Data" is in the hundreds of Petabytes. I don't buy the out-of-core definition either. I work using a Data Warehouse with petabytes of information...it certainly doesn't fit in memory. I don't use Hive, Pig, Cascading, etc...(okay, sometimes I use Cascalog, but not as a strategy for dealing with large amounts of data). I use SQL. And it works perfectly fine. But if you ask any of the people out there talking about "Big Data", an SQL database doesn't fit into the definition. Hell, I have done processing on a 200GB CSV file using nothing more than GAWK. Nobody is calling GAWK a big data tool. Face it. "Big Data" is a buzzword for CIOs that read magazines for CIOs but still need to find an engineer to set up their email on their iPad.
cema 14y ago
Thanks for the feedback on R. Have you felt it was more restrictive than D3 (and other technologies) or generally less comfortable to work with?

walshemj 14y ago

Excell breaks badly for trivially small data sets

briandoll 14y ago
I tend to aggregate data into something manageable before I attempt to graph it, regardless of what tool I'm using. If you can deal with daily numbers, you only need 365 rows per year, and less than 9k per year if you're doing hourly numbers.

vtuulos 14y ago

Unix command-line tools are seriously undervalued for processing data, even big data. Need parallelization? Try xargs -P on a multi-core machine.

chernevik 14y ago
Don't forget cut, sort and uniq. If you have a big csv file, you could strip out all duplicate values in the fourth fields: cut -d , -f 4 [file] | sort | uniq > values.txt You could then grep out the records with the particular values of interest. If a file is too big for an editor, you can still see the record structure with head.

Cieplak 14y ago

If you need to lots of one-off visualizations, consider Tableau (http://www.tableausoftware.com/ http://www.tableausoftware.com/). Although not as versatile as D3.js and although it's proprietary, it is really easy, essentially all click and point. My firm started using it recently and I am really jealous because MBA types can in seconds create visualizations that take me hours to do with D3.js. It supports a wide variety of data inputs, including excel and various databases. Also, Tableau has excellent support for working with maps and shapefiles.

j_s 14y ago
I stumbled across Gephi ('an open source graph visualization and manipulation software') and got a few comments on it about 9 months ago: http://news.ycombinator.com/item?id=3700471 http://news.ycombinator.com/item?id=3700471 https://gephi.org/ https://gephi.org/

notJim 14y ago

I feel like if you're using those tools to process raw data (and not the data produced by another job), it's not really "big" data. I would tend to define big data as data where there's so much volume that processing it in this way is not possible. Which isn't to say this article isn't full of hype—it clearly is.

dxbydt 14y ago

This isn't big data at all. This is stuff you'd use AFTER you've processed your big data & obtained a reduced dataset result that was small data, of the order of a few hundred MB. To process said big data in the first place, of the order of a few PB, typically you are going to use hdfs & at a programmatic level, an api like cascading or better yet, scalding, so you end up needing only that last item D3.js , with everything else existing as scalding source.

saosebastiao 14y ago
This is where "Big Data" becomes about as useful as a penis measuring contest. The vast majority of business data analysis happens at MS Excel levels of data. I have done analysis on several GBs of data using R, and those data-sets are easily the 99th percentile of data size in my field. For datasets that are bigger than that, SQL is still pretty damn good. And when you get to the point where SQL starts to break down in usefulness, you are at a level where you can start sampling and still get perfectly usable results in almost all use cases. In my experience, "Big Data" is a misnomer. What it really means is fast data. You have a repeatable calculation to perform on datasets that are perfectly fine chugging away on an Oracle cluster for 3 hours, but you want it done in 20 minutes. That is what Big Data is really used for. Anything else is just marketing hype.
- bravura 14y ago
  No. Big data has a formal definition, but one that most people ignore: When the size of the data becomes part of the problem. (An O'Reilly author said this once, I forget his name.) For example, physicists in the 80's who had 10s of MBs of data had a big data problem. Nowadays, this typically means out-of-core data sets.
  - saosebastiao 14y ago
    That definition is but one of many supposed formal definitions. And it isn't a particularly good one, because then "Big Data" becomes a definition that can only be quantified on a person-level of granularity. According to the guy with the Marketing degree from Ho-Hum College, anything bigger than can fit in an Excel spreadsheet is "Big Data". Switch to Doug Cutting, and "Big Data" is in the hundreds of Petabytes. I don't buy the out-of-core definition either. I work using a Data Warehouse with petabytes of information...it certainly doesn't fit in memory. I don't use Hive, Pig, Cascading, etc...(okay, sometimes I use Cascalog, but not as a strategy for dealing with large amounts of data). I use SQL. And it works perfectly fine. But if you ask any of the people out there talking about "Big Data", an SQL database doesn't fit into the definition. Hell, I have done processing on a 200GB CSV file using nothing more than GAWK. Nobody is calling GAWK a big data tool. Face it. "Big Data" is a buzzword for CIOs that read magazines for CIOs but still need to find an engineer to set up their email on their iPad.

cema 14y ago

Thanks for the feedback on R. Have you felt it was more restrictive than D3 (and other technologies) or generally less comfortable to work with?