4 ms·

Sigh append would be such game changer for us. Pulling a big S3 object over the wire to append to it and send it back is far from ideal.

afavour 2y ago
Try Azure’s equivalent. I’ve already forgotten what it’s called but it has append operations.
- magicalhippo 2y ago
  So simple it's hard: append blobs[1]. [1]: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs https://learn.microsoft.com/en-us/rest/api/storageservices/u...
- derefr 2y ago
  The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffers, where handles can be efficiently transformed/exchanged between between different "states of matter" — immutable objects, vs append-only streamable logs, vs multi-reader multi-writer random-access disks; and where all three of these states of data-buffer can be consumed interchangeably by a consumer through the same set of APIs — where these APIs might work most efficiently for data buffers that are in the right state, but they still work for data buffers that aren't. Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases. For example, while there is a change feed for blob events, akin to S3 lifecycle event notifications, it stops exactly where S3's API stops; so there is no event generated by an append to an appendable blob, nor a write to a page in a page blob or a block in a block blob. (And even if there were, they make no guarantees of the change feed being linearized — saying that changes to some resources might arrive out-of-order or not at all; and that if you want a linearized change feed, you need to read it out of a log-multiplexer, which puts a several-minute delay and multi-minute step-granularity on reads from it.) As such, you can't use Append Blobs as the storage layer for a Kafka-alike; and nor can you use Page Blobs as the transport to enable an embedded LMDB-alike to be network-replicated. (Or rather, you can, but in both cases you won't receive timely notifications that new data has been added / that pages have been invalidated at the origin, so unless you're operating with zero caching, your cache will end up stale and your state from successive reads will end up incoherent.)
  - d0gsg0w00f 2y ago
    In my 1 year of experience with Azure 3 years ago I saw this same story play out across every product line. Layers 1-3 look amazing then layers 4-6 are total head scratcher deal breakers. Very frustrating.
- nosefrog 2y ago
  Sure, how many 9's of availability does it have again?
  - robertlagrant 2y ago
    Infinite. 9.9999999999999999999999...% uptime guaranteed in Azure.
verticalscaler 2y ago
https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove... Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.
- ericbarrett 2y ago
  Multipart uploads are not the same as append; an object being multipart-uploaded doesn't exist in the bucket namespace until the upload is finalized, and once that is done it cannot be modified without overwriting.
  - dabiged 2y ago
    They don't exist in the bucket name space, but they sure as hell exist in the bucket bill!
- joombaga 2y ago
  This doesn't really help Rowan. You cannot edit an individual part after completing the upload.
nathants 2y ago
the correct pattern is to append immutable objects into a common prefix.
- rowanseymour 2y ago
  maybe but isn't there a performance cost of searching by prefix
  - nathants 2y ago
    assuming a serial writer, use a monotonic sequence that you can walk until no more data is found. for multiple writers, you have to use uuid and pay the search cost.
- KaiserPro 2y ago
  Not if you want speed. All of those objects and working out the history takes time. The API is not speedy when it comes to key:value lookup. At that point you're better off with a local object mapping cache. But then now you've created a half arsed file system

afavour 2y ago

Try Azure’s equivalent. I’ve already forgotten what it’s called but it has append operations.

magicalhippo 2y ago
So simple it's hard: append blobs[1]. [1]: https://learn.microsoft.com/en-us/rest/api/storageservices/understanding-block-blobs--append-blobs--and-page-blobs#about-append-blobs https://learn.microsoft.com/en-us/rest/api/storageservices/u...
derefr 2y ago
The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffers, where handles can be efficiently transformed/exchanged between between different "states of matter" — immutable objects, vs append-only streamable logs, vs multi-reader multi-writer random-access disks; and where all three of these states of data-buffer can be consumed interchangeably by a consumer through the same set of APIs — where these APIs might work most efficiently for data buffers that are in the right state, but they still work for data buffers that aren't. Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases. For example, while there is a change feed for blob events, akin to S3 lifecycle event notifications, it stops exactly where S3's API stops; so there is no event generated by an append to an appendable blob, nor a write to a page in a page blob or a block in a block blob. (And even if there were, they make no guarantees of the change feed being linearized — saying that changes to some resources might arrive out-of-order or not at all; and that if you want a linearized change feed, you need to read it out of a log-multiplexer, which puts a several-minute delay and multi-minute step-granularity on reads from it.) As such, you can't use Append Blobs as the storage layer for a Kafka-alike; and nor can you use Page Blobs as the transport to enable an embedded LMDB-alike to be network-replicated. (Or rather, you can, but in both cases you won't receive timely notifications that new data has been added / that pages have been invalidated at the origin, so unless you're operating with zero caching, your cache will end up stale and your state from successive reads will end up incoherent.)
- d0gsg0w00f 2y ago
  In my 1 year of experience with Azure 3 years ago I saw this same story play out across every product line. Layers 1-3 look amazing then layers 4-6 are total head scratcher deal breakers. Very frustrating.
nosefrog 2y ago
Sure, how many 9's of availability does it have again?
- robertlagrant 2y ago
  Infinite. 9.9999999999999999999999...% uptime guaranteed in Azure.

verticalscaler 2y ago

https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuoverview.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/mpuove... Multipart upload allows you to upload a single object as a set of parts. Each part is a contiguous portion of the object's data. You can upload these object parts independently and in any order. If transmission of any part fails, you can retransmit that part without affecting other parts. After all parts of your object are uploaded, Amazon S3 assembles these parts and creates the object. In general, when your object size reaches 100 MB, you should consider using multipart uploads instead of uploading the object in a single operation.

ericbarrett 2y ago
Multipart uploads are not the same as append; an object being multipart-uploaded doesn't exist in the bucket namespace until the upload is finalized, and once that is done it cannot be modified without overwriting.
- dabiged 2y ago
  They don't exist in the bucket name space, but they sure as hell exist in the bucket bill!
joombaga 2y ago
This doesn't really help Rowan. You cannot edit an individual part after completing the upload.

nathants 2y ago

the correct pattern is to append immutable objects into a common prefix.

rowanseymour 2y ago
maybe but isn't there a performance cost of searching by prefix
- nathants 2y ago
  assuming a serial writer, use a monotonic sequence that you can walk until no more data is found. for multiple writers, you have to use uuid and pay the search cost.
KaiserPro 2y ago
Not if you want speed. All of those objects and working out the history takes time. The API is not speedy when it comes to key:value lookup. At that point you're better off with a local object mapping cache. But then now you've created a half arsed file system