4 ms·

The missing context here is the scale. What size of data and transaction volume have you been working with?

by debaserab2 8y ago

pilif 8y ago
By now the data is about 6.2 TB with overall 20k transactions per second, so I would say not small any more, but also certainly not Uber scale. Still. I have had MySQL corrupt tables with way less load and data (in the less than 10GB size range)
- jjirsa 8y ago
  6.2TB is single machine scale The company here is doing petabytes of data and millions of ops per second. That’s scale. We should be clear about what “at scale” means, and 6.2TB is not.
  - pilif 8y ago
    No matter the scale. Table corruption is never acceptable. If table corruption is acceptable, you might as well just serve data from /dev/urandom and store user data to /dev/null.
    - astine 8y ago
      Table corruption may not be 'acceptable,' but that doesn't matter. Even if your software is perfect and has no bugs, it cannot account for issues at the system and hardware level. Backups and redundant systems exist for a reason. I've have data corruption in a PG database before. I don't remember the cause--I think it was the filesystem rather than Postgres that was at fault--but it still happened and I had to deal with it.
      - manigandham 8y ago
        Obviously the situation being discussed here is corruption specifically because of the database software. If a comet destroys the entire datacenter and the data is lost, nobody is going to blame Postgres for it.
- Amezarak 8y ago
  > but also certainly not Uber scale. How many Uber rides are happening every second? I suppose it depends exactly on their setup, but I could see that being at least the right order of magnitude. EDIT: According to the Internet, there are about a billion Uber rides per year, which makes for about 31 rides per second. Even if there are 1000 transactions per ride, 21k/second sounds approximately Uber scale to me.

4 ms·

The missing context here is the scale. What size of data and transaction volume have you been working with?

by debaserab2 8y ago

The missing context here is the scale. What size of data and transaction volume have you been working with?

pilif 8y ago
By now the data is about 6.2 TB with overall 20k transactions per second, so I would say not small any more, but also certainly not Uber scale. Still. I have had MySQL corrupt tables with way less load and data (in the less than 10GB size range)
- jjirsa 8y ago
  6.2TB is single machine scale The company here is doing petabytes of data and millions of ops per second. That’s scale. We should be clear about what “at scale” means, and 6.2TB is not.
  - pilif 8y ago
    No matter the scale. Table corruption is never acceptable. If table corruption is acceptable, you might as well just serve data from /dev/urandom and store user data to /dev/null.
    - astine 8y ago
      Table corruption may not be 'acceptable,' but that doesn't matter. Even if your software is perfect and has no bugs, it cannot account for issues at the system and hardware level. Backups and redundant systems exist for a reason. I've have data corruption in a PG database before. I don't remember the cause--I think it was the filesystem rather than Postgres that was at fault--but it still happened and I had to deal with it.
      - manigandham 8y ago
        Obviously the situation being discussed here is corruption specifically because of the database software. If a comet destroys the entire datacenter and the data is lost, nobody is going to blame Postgres for it.
- Amezarak 8y ago
  > but also certainly not Uber scale. How many Uber rides are happening every second? I suppose it depends exactly on their setup, but I could see that being at least the right order of magnitude. EDIT: According to the Internet, there are about a billion Uber rides per year, which makes for about 31 rides per second. Even if there are 1000 transactions per ride, 21k/second sounds approximately Uber scale to me.