3 ms·

See comment below. Definitely a small database size. You hit the nail on the head. It's going to grow fast this next year, and I'd like to put off sharding a

See comment below. Definitely a small database size. You hit the nail on the head. It's going to grow fast this next year, and I'd like to put off sharding as long as possible, hence gzipped storage of the data. postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc). However I'll be storing a point reduced version of the full track in postgis, so i can move to using actual intersection queries for matching tracks, instead of the current brute force approach (check everypoint in every track sharing a bounding box) that works now. All bets are out the window though with 2-4x the traffic and data we currently have, using that brute force approach. I already run another beefy postgis setup (192gb ram, though spinning disks not SSDs) for serving OSM maps, and eventually OSM routing hence the ram.

jacques_chester 13y ago
> postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc). Dumb question: why not?
- cullenking 13y ago
  because it's faster to serve the full fidelity track, with all sensor data for every lat/lng pair, straight from disk as a gzipped file. I can store a point reduced version in postgis for actual geo operations, since that only needs lat/lng pairs, and not at 1hz sampling. Pulling 21k points (1.8mb of JSON) for a single recording out of disk into memory, turning it into JSON, sending it over the wire to the client, is slow (500ms just to query, pull off disk and turn into JSON). Serving the same data pre-compressed, is 2ms. This isn't even considering what it takes nginx to compress it on the fly for every request.
  - jacques_chester 13y ago
    So it's because you don't want to recreate the track from point data on each request. I can see why that's advantageous. OK, new dumb question. It sounds like you're serving gzip'd JSON straight off the disk with nginx (serving gzip'd files direct from disk is one of my favourite nginx features). Where does Mongo come into the picture?

jacques_chester 13y ago

> postgis isn't a good fit for the data we store in full fidelity, since it's not just geo data but also sensor data (heartrate, cadence, power in watts, temperature etc). Dumb question: why not?

cullenking 13y ago
because it's faster to serve the full fidelity track, with all sensor data for every lat/lng pair, straight from disk as a gzipped file. I can store a point reduced version in postgis for actual geo operations, since that only needs lat/lng pairs, and not at 1hz sampling. Pulling 21k points (1.8mb of JSON) for a single recording out of disk into memory, turning it into JSON, sending it over the wire to the client, is slow (500ms just to query, pull off disk and turn into JSON). Serving the same data pre-compressed, is 2ms. This isn't even considering what it takes nginx to compress it on the fly for every request.
- jacques_chester 13y ago
  So it's because you don't want to recreate the track from point data on each request. I can see why that's advantageous. OK, new dumb question. It sounds like you're serving gzip'd JSON straight off the disk with nginx (serving gzip'd files direct from disk is one of my favourite nginx features). Where does Mongo come into the picture?