3 ms·
It’s a custom log-based block device implemented on top of Colossus/D in the user-mode lib in the virtualizer (Vanadium). Guest OS communicates using NVMe or Vi
by vl 7y ago
It’s a custom log-based block device implemented on top of Colossus/D in the user-mode lib in the virtualizer (Vanadium). Guest OS communicates using NVMe or VirtIO, Vanadium intercepts, calls PD lib.
This design has only one hop to the storage node. Low-latency workloads benefit from this design, high-bandwidth workloads sometimes actually benefit from off-loading PD to another host. To do iSCSI with one hop, you need to implement iSCSI interceptor, and basically you would have same design with less flexibility for guest OSes.
The irony, of course, that all this is a lot of legacy technologies needlessly wasting computer power: guest file-system trying to communicate with 4K blocks with “block device”, which goes through multiple layers of queues, then is re-maps to another abstraction, which goes over network to multiple hosts, etc. Not a single cloud customer ever said “we are so excited to manage volume sizes and bandwidth quotas for PD”. Better design would be to implement true data center-level filesystem to better support container workloads and leave PD for legacy cases, but Google’s storage management is so detached from reality, that it’s impossible to do cross-organizational project like this.
- ithkuil 7y agoWhat do you mean by "true data center level"? A FS with posix semantics suitable for normal apps doing file IO or something else?
- vl 7y agoI would break it down like this: 1) FS-core: provide FS-like semantics on top of D (or Colossus/D). Apart from data format and GC challenges, the piece that is missing is fast DC-level locking service. 2) Then, for VMs you can do FS driver, jump to VMM and booms, you are done, multiple legacy levels of re-packing and redirection are gone. For shared access cases you do NFS/Samba interceptor and then same code path as above. This system would be highly beneficial not only for Cloud, but for other Google properties as well: it would provide normal posix FS to Borg jobs. Amounts of equilibristics required to use any open-source package is enormous and by this time exceeded costs of developing FS multiple times, ask YouTube, MySQL, package management, etc groups. Another example of Google’s storage craziness is cross-dc storage. This should be low-level Colossus responsibility. Instead godzillion of teams implement their own, GCS, PD, Spanner, Placer, etc. Crazy.
- joatmon-snoo 7y agoHaha, yeah, I do wonder about the insanity on a pretty regular basis :) I will say though, that most of the time I don't want to use open source stuff. Observability is pretty crap, I don't want nor need software that uses write() without fsync(), and the assumptions that most OSS makes about FSs gives me nightmares on Borg.