3 ms·

I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to

• derefr 1y ago
  > which is such an obvious nice product just like BigQuery I always assumed (from outside Google) that the problem was that Colossus had to make a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does; and that therefore just exposing it directly to the public would make it possible for someone to DoS-attack the Colossus cluster. My logic was that there's actually nothing forcing [the public GCP service of] BigTable to require that a full copy of the dataset be kept hot across the nodes, with pre-reserved storage space — rather than mostly decoupling origin storage from compute† — unless it was to prevent some DoS vector. As for exactly what that DoS vector is... maybe GC/compaction policy-engine logic? (AFAICT, Colossus has pluggable "send compute to data" GC, which internal-BigTable and GCS both use. But external-BigTable forces the GC to be offloaded to the client [i.e. to the BigTable compute nodes the user has allocated] so that the user can't just load down the system with so many complex GC policies that the DC-scale Colossus cluster itself starts to fall behind its GC time budget.) --- † Where by "decouple storage from compute", I mean: • Each compute node gets a fixed-sized DAS diskset, like GCE local NVMe SSDs; • each disk in that diskset gets partitioned up at some fixed ratio, into two virtual disksets; • one virtual diskset gets RAID6'ed or ZFS'ed together, and is used as storage for non-Colossus-synced tablet-LDB nursery level SSTs; • the other virtual diskset gets RAID0'ed or LVM-JBOD-ed together and is used as a bounded-size LFU read-through cache of the Colossus-synced tablets — just like BigQuery compute nodes presumably have. (AFAIK the LDB nursery levels already get force-compacted into "full" [128MiB] Colossus-synced tablets after some quite-short finality interval, so it's not like this increases data loss likelihood by much. And BigTable doesn't guarantee durability for non-replicated keys anyway.)
  • rfoo 1y ago
    > a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does Didn't think deep into it, could this be solved with billing designs with more nuance?
• smueller1234 1y ago
  Concur, Colossus is one of the examples where Google built what almost feels like magic technology. I work on Google Storage (among other things), and I've wished for a Cloud offering that exposes Colossus for years. I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before. One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware? Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well). Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.