4 ms·

>S3 is the most reliable data storage mechanism in the world S3 is not the problem here. The problem is building a database on top of S3, and having to reimple

>S3 is the most reliable data storage mechanism in the world S3 is not the problem here. The problem is building a database on top of S3, and having to reimplement all the consistency, atomicity, transactions etc. on top. >no thought to a schema, no migrations to manage There is, in fact, always a schema. Some people choose to ignore it's there, to their detriment. >Always avoid the fancy option when it makes sense. It's not the 1980s. Postgres is not fancy, and Greenspunning it is a mistake. >Almost guaranteed it's cheaper. Cheaper than a 26-hour outage?

throwawaaarrgh 3y ago
> having to reimplement all the consistency, atomicity, transactions etc. on top. Did you miss where I said it's read-after-write strongly consistent?
- zetalyrae 3y ago
  Having those things at the base layer is not the same as having them in higher-level behaviour. This is why God invented SQL transactions.
  - mattigames 3y ago
    I never knew Donald Chamberlin and Raymond Boyce were God, it doesn't sound right but I don't know enough about theology to refute it.
bastawhiz 3y ago
> having to reimplement all the consistency, atomicity, transactions etc. on top. Most of those problems are moot if you're only ever writing from a single head node. If all your data is strictly ordered and you have no meaningful concurrency, this is a far, far simpler system.
- zetalyrae 3y ago
  Did I fall into a timewarp into the 70's? How on Earth, by what sane standard, is a Postgres instance too complex? If you're `fopen`ing files as a "database" you are wasting your time and lowering the world's economic productivity. Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. You never hear about such things with Postgres because Postgres is rock-solid.
  - bastawhiz 3y ago
    I'm not defending the choice to use S3. I probably wouldn't have made the same choice. But I am pointing out that it's empirically wrong to say that storing data in flat files necessitates the considerations of an ACID-capable RDBMS. But to your point, if your system requires less than a thousand lines of code to open a file, do basic parsing and processing (which no data storage system is going to do anyway), and write the output to another file, I personally can't say that Postgres or MySQL or any other solution is really worth the effort/cost to build and maintain. In the system being discussed, the benefits of an RDBMS simply don't matter: any strongly consistent key-value store would work. > Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. S3 didn't cause the outage, and from the look of it, neither did the code that processes the files. It was an application logic problem which caused issues during the restore process, and this would have been an issue regardless. You could make an argument that the recovery being slower than it could have been was a problem, but it's wild to say that and imply that traditional databases have no performance cliffs. Especially when dealing with corruption or data recovery. Raw file storage will never have a Postgres transaction id wraparound incident (see: Sentry outage for most of a day in 2015, MailChimp/Mandrill for over a day in 2019) or have to rebuild a critical index. In this case there was a hard coded concurrency limit with S3 of 250 outstanding requests. Bumping that up to 3000 would have been easy and reasonable (S3 rate limits at 5000). How confident would you be that your database can performantly handle a backfill during recovery? Have you provisioned enough iops? Are you running an RDS instance with only a burstable vCPU limit? To say Postgres is "rock-solid" (and make no mistake, I am a Postgres fanboy) dismisses the many and varying ways that it can fail in unusual and surprising ways.
  - 3y ago
nemothekid 3y ago
>Cheaper than a 26-hour outage? AFAICT, using HN theres been roughly 30 hours of non-availability over 15 years. RDS didn't even support Postgres when Tarsnap was released. EDIT: Tarsnap predates RDS.

throwawaaarrgh 3y ago

> having to reimplement all the consistency, atomicity, transactions etc. on top. Did you miss where I said it's read-after-write strongly consistent?

zetalyrae 3y ago
Having those things at the base layer is not the same as having them in higher-level behaviour. This is why God invented SQL transactions.
- mattigames 3y ago
  I never knew Donald Chamberlin and Raymond Boyce were God, it doesn't sound right but I don't know enough about theology to refute it.

bastawhiz 3y ago

> having to reimplement all the consistency, atomicity, transactions etc. on top. Most of those problems are moot if you're only ever writing from a single head node. If all your data is strictly ordered and you have no meaningful concurrency, this is a far, far simpler system.

zetalyrae 3y ago
Did I fall into a timewarp into the 70's? How on Earth, by what sane standard, is a Postgres instance too complex? If you're `fopen`ing files as a "database" you are wasting your time and lowering the world's economic productivity. Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. You never hear about such things with Postgres because Postgres is rock-solid.
- bastawhiz 3y ago
  I'm not defending the choice to use S3. I probably wouldn't have made the same choice. But I am pointing out that it's empirically wrong to say that storing data in flat files necessitates the considerations of an ACID-capable RDBMS. But to your point, if your system requires less than a thousand lines of code to open a file, do basic parsing and processing (which no data storage system is going to do anyway), and write the output to another file, I personally can't say that Postgres or MySQL or any other solution is really worth the effort/cost to build and maintain. In the system being discussed, the benefits of an RDBMS simply don't matter: any strongly consistent key-value store would work. > Complex is Greenspunning a database and having it blow up in your face and cause a twenty-six hour outage. S3 didn't cause the outage, and from the look of it, neither did the code that processes the files. It was an application logic problem which caused issues during the restore process, and this would have been an issue regardless. You could make an argument that the recovery being slower than it could have been was a problem, but it's wild to say that and imply that traditional databases have no performance cliffs. Especially when dealing with corruption or data recovery. Raw file storage will never have a Postgres transaction id wraparound incident (see: Sentry outage for most of a day in 2015, MailChimp/Mandrill for over a day in 2019) or have to rebuild a critical index. In this case there was a hard coded concurrency limit with S3 of 250 outstanding requests. Bumping that up to 3000 would have been easy and reasonable (S3 rate limits at 5000). How confident would you be that your database can performantly handle a backfill during recovery? Have you provisioned enough iops? Are you running an RDS instance with only a burstable vCPU limit? To say Postgres is "rock-solid" (and make no mistake, I am a Postgres fanboy) dismisses the many and varying ways that it can fail in unusual and surprising ways.
- 3y ago

nemothekid 3y ago

>Cheaper than a 26-hour outage? AFAICT, using HN theres been roughly 30 hours of non-availability over 15 years. RDS didn't even support Postgres when Tarsnap was released. EDIT: Tarsnap predates RDS.