3 ms·
I think you missed the point. GFS supported append operations and was created around 24 years ago. S3 still hasn't caught up with this particular feature. Altho
by estebarb 2y ago
I think you missed the point. GFS supported append operations and was created around 24 years ago. S3 still hasn't caught up with this particular feature. Although they clearly implemented it, as you can do a long multipart upload and S3 will join the file for you.
At a high level, yes, you can implement systems like Megastore or Bigtable over S3. However, there are many details you must take into account. You cannot simply wave away the complexity and potential failure scenarios.
For starters, how are you going to create the newest SST?
If you keep it in memory or on disk, it must be replicated to prevent data loss if a machine fails. This approach could lead to losing the most recent changes. Additionally, you end up with a hybrid system that needs to read data from multiple sources, which adds complexity. If you essentially reimplement the system, why use S3 at all?
What if the data volume gets too low and you end up writing many small, expensive files?
Using something like Kinesis for batching might work, but the data won't be visible for N minutes.
Merging partial tables also requires maintaining an external index to track availability. Transactions would be helpful, but how do you handle failures?
And we haven't even mentioned managing garbage collection. It would require an external lock or reference count system.
- klodolph 2y ago> I think you missed the point. Maybe I am thinking more broadly when imagine what it means to implement something like Spanner on top of S3. We know that Spanner on top of S3 is not going to give you the same price/performance as building Spanner on top of Colossus while giving you the same semantics. You either relax the semantics a little bit, you pay out the nose for a lot of little files, or you find a durable place outside S3 to store the newest data. > At a high level, yes, you can implement systems like Megastore or Bigtable over S3. However, there are many details you must take into account. You cannot simply wave away the complexity and potential failure scenarios. Most of the complexity is the same whether you implement Megastore on top of S3 or on top of GFS. You can’t handwave it in either scenario. > If you essentially reimplement the system, why use S3 at all? It’s highly durable, highly available, and cheap (under certain usage scenarios). GFS is not available for anyone to use, inside or outside Google. Its successor, Colossus, is not available outside Google. They’re just not available.