4 ms·
anecdota - We used fast csv tools and custom node.js scripts to wrangle import of ~500GB of geo polygon data into a large single vol postgresql+postGIS host. W
by jgord 8y ago
anecdota - We used fast csv tools and custom node.js scripts to wrangle import of ~500GB of geo polygon data into a large single vol postgresql+postGIS host.
We generate svg maps in psuedo-realtime from this data-set : 2MB maps render sub-second over the web, which feels 'responsive'.
I only mention this as many marketing people will call 50Million rows or 1TB "Big Data" and therefore suggest big / expensive / complex solutions. Recent SSD hosts have pushed up the "Big Data" watermark, and offer superb performance for many data applications.
[ yes, I know you can't beat magnetic disks for storing large videos, but thats a less common use-case ]
- sitkack 8y agoHow do you partition your data and/or build your indexes?
- jgord 8y agotwo kinds of indexes, for different purposes ... a) basically a preprocessed static inverted index on keywords [ using GIN / tsv_ etc ] b) geo location index on GIS geometry field - relying on postGIS to be performant over geo queries The data changes infrequently, so these are computed at time of import / regular update. We do have a fair amount of ram set aside for postgres [ circa 20GB ]