3 ms·
The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffer
by derefr 2y ago
The design of Azure's blob-store abstraction is great progress toward the ideal form of an object-store: a serverless managed store for all types of data buffers, where handles can be efficiently transformed/exchanged between between different "states of matter" — immutable objects, vs append-only streamable logs, vs multi-reader multi-writer random-access disks; and where all three of these states of data-buffer can be consumed interchangeably by a consumer through the same set of APIs — where these APIs might work most efficiently for data buffers that are in the right state, but they still work for data buffers that aren't.
Sadly, Azure's implementation of its blob-store, is kind of underwhelming — especially for any kind of infrastructure-level use-cases.
For example, while there is a change feed for blob events, akin to S3 lifecycle event notifications, it stops exactly where S3's API stops; so there is no event generated by an append to an appendable blob, nor a write to a page in a page blob or a block in a block blob. (And even if there were, they make no guarantees of the change feed being linearized — saying that changes to some resources might arrive out-of-order or not at all; and that if you want a linearized change feed, you need to read it out of a log-multiplexer, which puts a several-minute delay and multi-minute step-granularity on reads from it.)
As such, you can't use Append Blobs as the storage layer for a Kafka-alike; and nor can you use Page Blobs as the transport to enable an embedded LMDB-alike to be network-replicated. (Or rather, you can, but in both cases you won't receive timely notifications that new data has been added / that pages have been invalidated at the origin, so unless you're operating with zero caching, your cache will end up stale and your state from successive reads will end up incoherent.)
- d0gsg0w00f 2y agoIn my 1 year of experience with Azure 3 years ago I saw this same story play out across every product line. Layers 1-3 look amazing then layers 4-6 are total head scratcher deal breakers. Very frustrating.