3 ms·
When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're ta
by jasonwatkinspdx 13y ago
When talking about architecture at this level, you need to take a longer view. Once upon a time, a gigabyte database was mammoth beyond thought. Also, you're taking it as given that your data has its present storage footprint even though alternative architectures might have very different properties. It's worth noting that a purely additive database can be very aggressive about compression.
But also, petabyte datasets are downright cheap to deal with these days. They'll look more like that half gigabyte source code repo sooner than I think you're considering.
- RyanZAG 13y agoWe're obviously coming from different backgrounds here - do you work at Google or similar ? 'Downright cheap' is a Heroku instance or two. 'Affordable' is 20 or so servers with a few TB of drive space each in a cluster with a few backups. Petabytes of data needs a huge number of servers and dedicated personal to look after - I don't know how that can be regarded as 'downright cheap'. If you're in a position to handle petabytes of data you're probably going to roll your own data center with custom software ala Google's bigtable. This definitely isn't the market that these little startups are targeting as far as I can see. The OPs critique is very well founded.
- corresation 13y agopetabyte datasets are downright cheap to deal with these days Are you from the future? In no universe is a petabyte database remotely close to being cheap, even for large organizations. At this point I am seriously questioning your industry knowledge.
- empthought 13y agoSure it is, a few thousand entries pointing to 10-100GB pirated HD video blobs. What's the problem? /s