4 ms·

Suggestions are more than welcome. I'm currently keeping the post up-to-date with suggested tools and repositories.

dirtyvagabond 13y ago
nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake https://github.com/Factual/drake
susi22 13y ago
http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/tools5desc.html#1 http://www.hdfgroup.org/tools5desc.html#1 The first thing I do when I deal with data which has more than --let's say-- 10,000 rows is to put it in HDF file format and work with that. Saves a ton of time while developing a script. I had a python script do a histogram and it ran ~15sec for a file with 100k rows. With converting it first to HDF it ran in ~0.5sec. The import in python is also much shorter (two lines). HDF is made for high performance numerical I/O. It's great and you can query several structures and even do slices of arrays on the command line (with h5tools). It's also widely used by Octave, Python, R, Matlab... And you don't have a drawback since you can just pipe it into existing command line tools with a h5dump. HDF5: http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf
- twistedpair 13y ago
  Wow, those benchmarks are old. 195MHz test machine? Still, 15K rows is not much. Just did that 100K rows (read/sum) bit in Octave. Took about 1/4 second to extract and sum. I assume HDF rocks for much larger sets.
susi22 13y ago
This also really helpful (pity it's relatively unknown): https://github.com/dbro/csvquote https://github.com/dbro/csvquote And: http://en.wikipedia.org/wiki/GNU_Core_Utilities http://en.wikipedia.org/wiki/GNU_Core_Utilities (section "Text utilities") http://directory.fsf.org/wiki/Textutils http://directory.fsf.org/wiki/Textutils Run "$ info coreutils"
gibrown 13y ago
Another shameless plug: http://gibrown.wordpress.com/2013/01/26/unix-bi-grams-tri-grams-and-topic-modeling/ http://gibrown.wordpress.com/2013/01/26/unix-bi-grams-tri-gr... More about Natural Language Processing.

nice post! i'd like to make a shameless plug for Drake, a 'make for data' workflow tool. https://github.com/Factual/drake https://github.com/Factual/drake

susi22 13y ago

http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/HDF5/doc/RM/Tools.html http://www.hdfgroup.org/tools5desc.html#1 http://www.hdfgroup.org/tools5desc.html#1 The first thing I do when I deal with data which has more than --let's say-- 10,000 rows is to put it in HDF file format and work with that. Saves a ton of time while developing a script. I had a python script do a histogram and it ran ~15sec for a file with 100k rows. With converting it first to HDF it ran in ~0.5sec. The import in python is also much shorter (two lines). HDF is made for high performance numerical I/O. It's great and you can query several structures and even do slices of arrays on the command line (with h5tools). It's also widely used by Octave, Python, R, Matlab... And you don't have a drawback since you can just pipe it into existing command line tools with a h5dump. HDF5: http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Performance.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf http://www.hdfgroup.org/HDF5/RD100-2002/HDF5_Overview.pdf

twistedpair 13y ago
Wow, those benchmarks are old. 195MHz test machine? Still, 15K rows is not much. Just did that 100K rows (read/sum) bit in Octave. Took about 1/4 second to extract and sum. I assume HDF rocks for much larger sets.

susi22 13y ago

This also really helpful (pity it's relatively unknown): https://github.com/dbro/csvquote https://github.com/dbro/csvquote And: http://en.wikipedia.org/wiki/GNU_Core_Utilities http://en.wikipedia.org/wiki/GNU_Core_Utilities (section "Text utilities") http://directory.fsf.org/wiki/Textutils http://directory.fsf.org/wiki/Textutils Run "$ info coreutils"

gibrown 13y ago

Another shameless plug: http://gibrown.wordpress.com/2013/01/26/unix-bi-grams-tri-grams-and-topic-modeling/ http://gibrown.wordpress.com/2013/01/26/unix-bi-grams-tri-gr... More about Natural Language Processing.