5 ms·

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of thos

>The process of recovering the EC2 instance state consists of two steps: First, reading all of the metadata headers from S3; and second, "replaying" all of those operations locally. (These cannot be performed at the same time, since the use of log-structured storage means that log entries are "rewritten" to free up storage when data is deleted; log entries contain sequence numbers to allow them to be replayed in the correct order, but they must be sorted into the correct order after being retrieved before they can be replayed.) Far be it from me to tell anyone how to write software, but why build a database on top of S3 when you can just chuck the metadata into RDS with however much replication you want? The backups themselves should be in S3, but using S3 as a NoSQL append-only database seems unwise. This would benefit from being further from the metal.

amluto 3y ago
> This would benefit from being further from the metal. How, exactly, is that a good thing?
- zetalyrae 3y ago
  How is not rolling your own database a good thing? Mainly because the business of tarsnap is 1) encrypted 2) backups, not building a database storage engine.
  - catiopatio 3y ago
    It’s cute that you think implementing client-side encrypted, deduplicated backups doesn’t involve building a database storage engine.
  - catiopatio 3y ago
    Implementing client-side encrypted, deduplicated, snapshot-enabled backups with server-mediated access control inherently requires building a minimal storage engine to represent your opaque log-structured data. Embedding the log-structured representation of user data in Postgres would increase complexity and overhead without offering significant resiliency or recoverability advantages — in fact, quite the opposite.
foldr 3y ago
>Far be it from me to tell anyone how to write software, but why build a database on top of S3 when you can just chuck the metadata into RDS with however much replication you want? Cost and reliability? * Using S3 as a simple database is generally going to be much cheaper than RDS. * If you turn on point in time restore, then losing data stored in S3 is not a possibility worth worrying about on a practical level for most people. RDS replication is easy enough to use, but adds more cost and a little bit of extra infra complexity.
- zetalyrae 3y ago
  >Cost It's a bad trade. Thousands of hours of a high human capital computer scientist vs. a few tens of dollars a month for RDS. >Reliability Empirically false: none of this would have happened if Tarsnap used Postgres instead of a home-spun database.
  - nonethewiser 3y ago
    >> Cost > It's a bad trade. Maybe. But that's the reason. You never acknowledged that advantage in your question so it needed to be emphasized
    - zetalyrae 3y ago
      It never occurred to me that anyone would need it explained to them that RDS is cheaper than the time of any software engineer. The opportunity cost of building your own database is 10,000x the cost of running RDS for a year.
      - foldr 3y ago
        That sort of logic doesn't really apply here because: * RDS costs obviously scale linearly with ongoing time and probably scale linearly with the total amount of data being backed up. So depending on the revenue of the business, these extra costs could easily end up outweighing the (notional) cost of the time saved, which is mostly a one-off expense. * The cost of a software engineer's time is notional in the context of a one-person business. The author of Tarsnap isn't going to be able to employ fewer than zero additional software engineers to maintain Tarsnap because of the time saved by using RDS.
throwawaaarrgh 3y ago
Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no storage to grow, no logs to rotate, no software to upgrade on a maintenance window. Plus S3 is versioned, has multiple kinds of access control built in, is a protocol supported by many vendors and open source projects, and is (on AWS) strongly read-after-write consistent. I would also argue (though I don't have figures) that it's faster than RDS. Almost guaranteed it's cheaper. And it's integrated into many services in AWS making it easier to build more functionality into the system without programming. On a less technical note: Always avoid the fancy option when it makes sense. (From a veteran of building and maintaining large scale high performance high availability systems)
- ilyt 3y ago
  The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...
  - throwawaaarrgh 3y ago
    Fancy would be writing an application to talk to RDS, creating an RDS instance, creating a database, creating whatever IAM link is needed for auth into the db so you don't need a second set of credentials, creating a schema, creating columns with different data types, and then modifying the application to handle edge cases for the different data types, logic to insert, update, delete rows, select items, yadda yadda yadda. Dumb is `aws s3 cp` and being done in 5 minutes.
    - zetalyrae 3y ago
      [flagged]
      - HumanOstrich 3y ago
lordgilman 3y ago
FWIW Tarsnap was launched in 2008, the initial RDS for MySQL was launched in 2009.
- zetalyrae 3y ago
  You can always self-host Postgres.
  - masfuerte 3y ago
    The VM crashed, corrupting the file system. This could have made a Postgres database unrecoverable. For rock solid reliability you need more than a database instance.
    - zetalyrae 3y ago
      Make two of them.
      - masfuerte 3y ago
        Keeping some kind of Postgres cluster running for over a decade seems like a lot of work. tarsnap seems to require roughly no maintenance.
nijave 3y ago
There are some established patterns for S3 as a database. It's extremely common in "data lakes" (throw data of various schemas in and use a tool that can parse at query time). There's client libraries like Delta Lake that implement ACID on S3. Much of the Grafana stack uses S3 for storage (Mimir/metrics, Loki/logs, Tempo/traces). That said, I'm not sure about the implementation Tarsnap uses--if it's completely ad-hoc or based of other patterns/libraries.

amluto 3y ago

> This would benefit from being further from the metal. How, exactly, is that a good thing?

zetalyrae 3y ago
How is not rolling your own database a good thing? Mainly because the business of tarsnap is 1) encrypted 2) backups, not building a database storage engine.
- catiopatio 3y ago
  It’s cute that you think implementing client-side encrypted, deduplicated backups doesn’t involve building a database storage engine.
- catiopatio 3y ago
  Implementing client-side encrypted, deduplicated, snapshot-enabled backups with server-mediated access control inherently requires building a minimal storage engine to represent your opaque log-structured data. Embedding the log-structured representation of user data in Postgres would increase complexity and overhead without offering significant resiliency or recoverability advantages — in fact, quite the opposite.

foldr 3y ago

>Far be it from me to tell anyone how to write software, but why build a database on top of S3 when you can just chuck the metadata into RDS with however much replication you want? Cost and reliability? * Using S3 as a simple database is generally going to be much cheaper than RDS. * If you turn on point in time restore, then losing data stored in S3 is not a possibility worth worrying about on a practical level for most people. RDS replication is easy enough to use, but adds more cost and a little bit of extra infra complexity.

zetalyrae 3y ago
>Cost It's a bad trade. Thousands of hours of a high human capital computer scientist vs. a few tens of dollars a month for RDS. >Reliability Empirically false: none of this would have happened if Tarsnap used Postgres instead of a home-spun database.
- nonethewiser 3y ago
  >> Cost > It's a bad trade. Maybe. But that's the reason. You never acknowledged that advantage in your question so it needed to be emphasized
  - zetalyrae 3y ago
    It never occurred to me that anyone would need it explained to them that RDS is cheaper than the time of any software engineer. The opportunity cost of building your own database is 10,000x the cost of running RDS for a year.
    - foldr 3y ago
      That sort of logic doesn't really apply here because: * RDS costs obviously scale linearly with ongoing time and probably scale linearly with the total amount of data being backed up. So depending on the revenue of the business, these extra costs could easily end up outweighing the (notional) cost of the time saved, which is mostly a one-off expense. * The cost of a software engineer's time is notional in the context of a one-person business. The author of Tarsnap isn't going to be able to employ fewer than zero additional software engineers to maintain Tarsnap because of the time saved by using RDS.

throwawaaarrgh 3y ago

Forget about software or data architecture. S3 is the most reliable data storage mechanism in the world, and insanely simpler than a relational database. There is no operational failure mode to S3, other than "region went down". There is no instance to go down, no replication to fail, no worry about whether there's enough capacity for writes or too many connections, no thought to a schema, no migrations to manage, no storage to grow, no logs to rotate, no software to upgrade on a maintenance window. Plus S3 is versioned, has multiple kinds of access control built in, is a protocol supported by many vendors and open source projects, and is (on AWS) strongly read-after-write consistent. I would also argue (though I don't have figures) that it's faster than RDS. Almost guaranteed it's cheaper. And it's integrated into many services in AWS making it easier to build more functionality into the system without programming. On a less technical note: Always avoid the fancy option when it makes sense. (From a veteran of building and maintaining large scale high performance high availability systems)

ilyt 3y ago
The "fancy" option here is trying to act S3 to act like database instead of simple blob storage...
- throwawaaarrgh 3y ago
  Fancy would be writing an application to talk to RDS, creating an RDS instance, creating a database, creating whatever IAM link is needed for auth into the db so you don't need a second set of credentials, creating a schema, creating columns with different data types, and then modifying the application to handle edge cases for the different data types, logic to insert, update, delete rows, select items, yadda yadda yadda. Dumb is `aws s3 cp` and being done in 5 minutes.
  - zetalyrae 3y ago
    [flagged]
    - HumanOstrich 3y ago

lordgilman 3y ago

FWIW Tarsnap was launched in 2008, the initial RDS for MySQL was launched in 2009.

zetalyrae 3y ago
You can always self-host Postgres.
- masfuerte 3y ago
  The VM crashed, corrupting the file system. This could have made a Postgres database unrecoverable. For rock solid reliability you need more than a database instance.
  - zetalyrae 3y ago
    Make two of them.
    - masfuerte 3y ago
      Keeping some kind of Postgres cluster running for over a decade seems like a lot of work. tarsnap seems to require roughly no maintenance.

nijave 3y ago

There are some established patterns for S3 as a database. It's extremely common in "data lakes" (throw data of various schemas in and use a tool that can parse at query time). There's client libraries like Delta Lake that implement ACID on S3. Much of the Grafana stack uses S3 for storage (Mimir/metrics, Loki/logs, Tempo/traces). That said, I'm not sure about the implementation Tarsnap uses--if it's completely ad-hoc or based of other patterns/libraries.