4 ms·
The reason people don’t start with a relational DB has nothing to do with the data model or query language - people choose databases like Cassandra because scal
by jjirsa 9y ago
The reason people don’t start with a relational DB has nothing to do with the data model or query language - people choose databases like Cassandra because scaling to 500 Postgres or MySQL instances holding a combined petabyte of data is horrific, but it’s dirt simple in things like Cassandra
The query language is a side effect of the underlying storage engine - you don’t choose it because you want a key value store, you choose it because you want horizontal scalability and cross wan HA
- 6nf 9y agoYou make it sound like many people are using petabyte scale databases. This is not even close to true. ESPECIALLY when you're just starting out. Most new projects will start with almost no data at all and the vast majority of projects will never ever get to the petabyte scales.
- jjirsa 9y agoMany people ARE using petabyte scale databases. They don't start out that way, but they will be if they're successful, and if they're not they probably shouldn't be running their own databases anyway (go RDS it up or something).
- deleted 9y ago[deleted]
- eberkund 9y agoUnless the amount of data you are collecting is increasing at a vastly unsustainable rate, you will be making megabucks and be able to afford a team of engineers to undertake a major redesign for version 2. But picking technologies based on scale at the onset is going to make getting to that point all the more unlikely.
- happymellon 9y agoWhat? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. Before that I have held positions in finance, dealing with realtime transaction processing. We did not work in petabytes. If you are working in petabytes you are storing crap in your production database, and 99% of that data is wasted.
- GrayShade 9y ago> What? No they aren't, I work with satellite imagery processing which is quite large in it raw data format, and after a decade we are not dealing with petabytes of active data, hundreds of gigs for a full earth coverage. This get large rather fast when processing time series from multiple sensors. But I agree with you otherwise, and it's not like you're storing satellite products in a database. Now I'm curious what NSA is using :).
- ddorian43 9y agohttps://accumulo.apache.org/ https://accumulo.apache.org/ See (History): https://en.wikipedia.org/wiki/Apache_Accumulo https://en.wikipedia.org/wiki/Apache_Accumulo
- GrayShade 9y agoThanks, I wasn't aware of it.
- EvilGrin 9y agoYou could just check their Github? :) https://github.com/nationalsecurityagency https://github.com/nationalsecurityagency
- altstar 9y agoYeah right. We are using nosql in manufacturing. One second data for thousands of instruments create a lot of data very quickly. And that is per plant. Having a lot of plants at various sites in the world adds up.
- jjirsa 9y agoExactly this. Whenever you collect millions of things, even at low resolution, it adds up quickly Collecting high resolution and filtering down to low resolution for lower importance is still difficult and can require petabytes for real world use cases
- bildung 9y agoPostgres (as an example) has no upper limit for database size, and the upper limit for a single table was 64TB a while ago, perhaps higher now. And you can buy stock hardware with 6 TB RAM and 76TB SSD e.g. from Dell. If you max out that single server, you could easily hire one database engineer to care about horizontal scaling.
- throwme211345 9y agoGreat. Now my 10G prod network can start replicating some fraction of 64TB at rate while you masturbate on the hardware. Utter nonsense.
- jjirsa 9y agoShow me a single person who runs 64TB on a single table on a single node, and I'll show you someone who will cry because of a prod failure sooner than later.
- jbergens 9y agoIt doesn't help when the nbr of queries and updates increases a lot. For many systems having TB of data also means having thousands or millions of users. You also probably want backups and possibly slaves that are updated inside transactions to avoid data loss. These things can lower your throughput even more.
- collyw 9y agoThe vast majority of hipster NoSQL developers I have debated with are using a single machine.
- jjirsa 9y agoThose people are objectively wrong.
- collyw 9y agoYou try and explain that to them, but whatever database they are using "is fast". Not that I have ever had a problem with relational databases being slow. Certain queries, yes, but a bit of rewriting them and proper indexing usually solves that.