3 ms·
Efficient Tabular Storage
- owlish 11y agoHow do databases like MySQL store data efficiently for querying? It seems like something like protobuf would do well here, though you'd need to generate code for each dataset.
- kragen 11y agoTypically they use row-oriented binary storage, optionally with individual columns or subsets of columns duplicated into indices for fast querying. Have you tried protobufs? How many hundreds of megs per second do you get? I think it is remarkably slow on the scales we're talking about here.
- brudgers 11y agoTraditional DBMS's get performance by optimizing storage down to the physical layout of the data on the hardware. So MySQL makes a lot of assumptions based on the mechanics if spinning disks and buffers tailored to their physics. Database Systems: The Complete Book is a good text on the subject and the second half is all about the hardware and software used in implementing traditional systems.
- Sprint 11y agoThis is super interesting, thanks!
- TheGuyWhoCodes 11y agoVertica has all those performance enhancements, great DB can't recommend because of pricing :(
- beagle3 11y agokdb+ answers same description. And it's a 300KB executable with no dependencies (other than glib/MSVCRT).
- Ashim_Usmani 11y agoGreat share
- SloopJon 11y agoIt looks like the NYCTaxi dataset is here: http://www.andresmh.com/nyctaxitrips/ http://www.andresmh.com/nyctaxitrips/ Some background on this data: http://chriswhong.com/data-visualization/taxitechblog1/ http://chriswhong.com/data-visualization/taxitechblog1/ And data for 2014 directly from the city: https://data.cityofnewyork.us/view/gn7m-em8n https://data.cityofnewyork.us/view/gn7m-em8n
- Little_Peter 11y agoHow is it different from HDF5 (h5py and pytables)?