5 ms·

Hats off to whoever convinced management that selling Colossus via cloud was Artificial Intelligence. Bravo.

eitally 2y ago
I don't fault them for this at all. AI isn't possible without the full infra stack, which clearly includes storage (and compute, and networking, and data pipelining, and and and...). There's an entire ecosystem of ISVs that only do one of these things, very well (Pure Storage, for example, or Lamba or Coreweave, or Confluent (Kafka + Flink with LLM integration). While it might be more precisely accurate to state "AI enabling" tech, I'll give them a pass.
- rfoo 2y ago
  I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.
  - derefr 2y ago
    > which is such an obvious nice product just like BigQuery I always assumed (from outside Google) that the problem was that Colossus had to make a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does; and that therefore just exposing it directly to the public would make it possible for someone to DoS-attack the Colossus cluster. My logic was that there's actually nothing forcing [the public GCP service of] BigTable to require that a full copy of the dataset be kept hot across the nodes, with pre-reserved storage space — rather than mostly decoupling origin storage from compute† — unless it was to prevent some DoS vector. As for exactly what that DoS vector is... maybe GC/compaction policy-engine logic? (AFAICT, Colossus has pluggable "send compute to data" GC, which internal-BigTable and GCS both use. But external-BigTable forces the GC to be offloaded to the client [i.e. to the BigTable compute nodes the user has allocated] so that the user can't just load down the system with so many complex GC policies that the DC-scale Colossus cluster itself starts to fall behind its GC time budget.) --- † Where by "decouple storage from compute", I mean: • Each compute node gets a fixed-sized DAS diskset, like GCE local NVMe SSDs; • each disk in that diskset gets partitioned up at some fixed ratio, into two virtual disksets; • one virtual diskset gets RAID6'ed or ZFS'ed together, and is used as storage for non-Colossus-synced tablet-LDB nursery level SSTs; • the other virtual diskset gets RAID0'ed or LVM-JBOD-ed together and is used as a bounded-size LFU read-through cache of the Colossus-synced tablets — just like BigQuery compute nodes presumably have. (AFAIK the LDB nursery levels already get force-compacted into "full" [128MiB] Colossus-synced tablets after some quite-short finality interval, so it's not like this increases data loss likelihood by much. And BigTable doesn't guarantee durability for non-replicated keys anyway.)
    - rfoo 2y ago
      > a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does Didn't think deep into it, could this be solved with billing designs with more nuance?
  - smueller1234 2y ago
    Concur, Colossus is one of the examples where Google built what almost feels like magic technology. I work on Google Storage (among other things), and I've wished for a Cloud offering that exposes Colossus for years. I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before. One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware? Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well). Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.
klabb3 2y ago
That’s an actually impressive level of spin. Flashback to when shipping companies were slapping blockchain on international container shipments.
- Foobar8568 2y ago
  Flashback to the cabbage on the Blockchain or when they wanted to tag each fish caught in the wild as well for hu traceability!
bushbaba 2y ago
You gotta feed the GPUs & TPUs with enough data to avoid them sitting idle. Which starts to become incredibly challenging with latest gen GPU/TPU chips
re-thc 2y ago
Everything is AI these days. Does it still need convincing?
- CobrastanJorji 2y ago
  Ha, right after I read your comment, I looked at the bottom of this Hacker News page and saw their "Join us for AI Startup School" ad.
jeffbee 2y ago
I would pay serious money if they sold CFS as a service but on AWS.
- immibis 2y ago
  Hi, I'm looking for a job. Are you willing to pay me serious money to set up CFS as a service on your AWS?
  - jsnell 2y ago
    Obviously not, since you could not deliver it. It seems that you maybe don't realize what CFS is in this context, and are thinking of something else that you could just "set up"? What jeffbee is talking about is Google's proprietary Colossus File System, and all its transitive dependencies.
    - immibis 2y ago
      I meant it sarcastically, but for "serious money" you can have any software system you can dream of. You have to dream of it, though - that's one of the hard parts. It looks like every other clustered file system. What's special about Google's Colossus?
      - noahl 2y ago
        There are some semantic differences compared to POSIX filesystems. A couple big ones: - You can only append to an object, and each object can only have one writer at the time. This is useful for distributed systems - you could have one process adding records to the end of a log, and readers pulling new records from the end. - It's also possible to "finalize" an object, meaning that it can't be appended to any more. (I work on Rapid storage.)
        immibis 2y ago
        Why would you wish for a system with constraints like that, which other systems don't have?
CobrastanJorji 2y ago
They're not saying that it's AI. They're saying it's for customers who do AI. Training means lots and lots of reads from a big data store, and if you're reading from, like, big Parquet files, that probably means lots of random reads. This is for that. Speedier data access, presumably at the cost of durability and availability, which is probably a great trade-off for people doing ML training jobs.
- Zvez 2y ago
  calling everything 'for AI' is the new standard >if you're reading from, like, big Parquet files, that probably means lots of random reads and it also usually means that you shouldn't use s3 in the first place for workloads like this. Because they are usually very inefficient comparing to distributed fs. Unless you have some prefetch/cache layer, you will get both bad timings and higher costs
  - CobrastanJorji 2y ago
    But a distributed FS is far more expensive than cloud blob storage would be, and I can't imagine most workloads would need the features of a POSIX filesystem.
Rebelgecko 2y ago
Maybe they can start selling Capacitor as a file format for storing LLMs metadata or something

I don't fault them for this at all. AI isn't possible without the full infra stack, which clearly includes storage (and compute, and networking, and data pipelining, and and and...). There's an entire ecosystem of ISVs that only do one of these things, very well (Pure Storage, for example, or Lamba or Coreweave, or Confluent (Kafka + Flink with LLM integration). While it might be more precisely accurate to state "AI enabling" tech, I'll give them a pass.

rfoo 2y ago
I think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.
- derefr 2y ago
  > which is such an obvious nice product just like BigQuery I always assumed (from outside Google) that the problem was that Colossus had to make a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does; and that therefore just exposing it directly to the public would make it possible for someone to DoS-attack the Colossus cluster. My logic was that there's actually nothing forcing [the public GCP service of] BigTable to require that a full copy of the dataset be kept hot across the nodes, with pre-reserved storage space — rather than mostly decoupling origin storage from compute† — unless it was to prevent some DoS vector. As for exactly what that DoS vector is... maybe GC/compaction policy-engine logic? (AFAICT, Colossus has pluggable "send compute to data" GC, which internal-BigTable and GCS both use. But external-BigTable forces the GC to be offloaded to the client [i.e. to the BigTable compute nodes the user has allocated] so that the user can't just load down the system with so many complex GC policies that the DC-scale Colossus cluster itself starts to fall behind its GC time budget.) --- † Where by "decouple storage from compute", I mean: • Each compute node gets a fixed-sized DAS diskset, like GCE local NVMe SSDs; • each disk in that diskset gets partitioned up at some fixed ratio, into two virtual disksets; • one virtual diskset gets RAID6'ed or ZFS'ed together, and is used as storage for non-Colossus-synced tablet-LDB nursery level SSTs; • the other virtual diskset gets RAID0'ed or LVM-JBOD-ed together and is used as a bounded-size LFU read-through cache of the Colossus-synced tablets — just like BigQuery compute nodes presumably have. (AFAIK the LDB nursery levels already get force-compacted into "full" [128MiB] Colossus-synced tablets after some quite-short finality interval, so it's not like this increases data loss likelihood by much. And BigTable doesn't guarantee durability for non-replicated keys anyway.)
  - rfoo 2y ago
    > a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does Didn't think deep into it, could this be solved with billing designs with more nuance?
- smueller1234 2y ago
  Concur, Colossus is one of the examples where Google built what almost feels like magic technology. I work on Google Storage (among other things), and I've wished for a Cloud offering that exposes Colossus for years. I don't know that it took "AI branding" to convince anybody. I think these workloads potentially enabled additional demand/market for such a product that may not have been there before. One of the challenges with exposing native Colossus was always that it's just different enough from how people elsewhere are used to use Storage that there was a lot of uncertainty about the addressable market of a "native" Colossus offering. It's not a POSIX file system. Some of the specific differences (eg. no random writes) are part of what makes Colossus powerful and performant on HDDs, but it means you have to write your application to work well within its constraints. Google has been doing that for a long time. If you haven't, even if it's an amazing product, is it worth rewriting your applications or middleware? Rapid Storage basically addresses this by adding the object store API on top if it (TIL from this thread that there's a lower abstraction client in the works as well). Anyway, the team behind this is awesome. Awesome tech, awesome people. Seeing this launched at Next and seeing some appreciation on HN makes me very grateful.

klabb3 2y ago

That’s an actually impressive level of spin. Flashback to when shipping companies were slapping blockchain on international container shipments.

Foobar8568 2y ago
Flashback to the cabbage on the Blockchain or when they wanted to tag each fish caught in the wild as well for hu traceability!

bushbaba 2y ago

You gotta feed the GPUs & TPUs with enough data to avoid them sitting idle. Which starts to become incredibly challenging with latest gen GPU/TPU chips

re-thc 2y ago

Everything is AI these days. Does it still need convincing?

CobrastanJorji 2y ago
Ha, right after I read your comment, I looked at the bottom of this Hacker News page and saw their "Join us for AI Startup School" ad.

jeffbee 2y ago

I would pay serious money if they sold CFS as a service but on AWS.

immibis 2y ago
Hi, I'm looking for a job. Are you willing to pay me serious money to set up CFS as a service on your AWS?
- jsnell 2y ago
  Obviously not, since you could not deliver it. It seems that you maybe don't realize what CFS is in this context, and are thinking of something else that you could just "set up"? What jeffbee is talking about is Google's proprietary Colossus File System, and all its transitive dependencies.
  - immibis 2y ago
    I meant it sarcastically, but for "serious money" you can have any software system you can dream of. You have to dream of it, though - that's one of the hard parts. It looks like every other clustered file system. What's special about Google's Colossus?
    - noahl 2y ago
      There are some semantic differences compared to POSIX filesystems. A couple big ones: - You can only append to an object, and each object can only have one writer at the time. This is useful for distributed systems - you could have one process adding records to the end of a log, and readers pulling new records from the end. - It's also possible to "finalize" an object, meaning that it can't be appended to any more. (I work on Rapid storage.)
      - immibis 2y ago
        Why would you wish for a system with constraints like that, which other systems don't have?

CobrastanJorji 2y ago

They're not saying that it's AI. They're saying it's for customers who do AI. Training means lots and lots of reads from a big data store, and if you're reading from, like, big Parquet files, that probably means lots of random reads. This is for that. Speedier data access, presumably at the cost of durability and availability, which is probably a great trade-off for people doing ML training jobs.

Zvez 2y ago
calling everything 'for AI' is the new standard >if you're reading from, like, big Parquet files, that probably means lots of random reads and it also usually means that you shouldn't use s3 in the first place for workloads like this. Because they are usually very inefficient comparing to distributed fs. Unless you have some prefetch/cache layer, you will get both bad timings and higher costs
- CobrastanJorji 2y ago
  But a distributed FS is far more expensive than cloud blob storage would be, and I can't imagine most workloads would need the features of a POSIX filesystem.

Rebelgecko 2y ago

Maybe they can start selling Capacitor as a file format for storing LLMs metadata or something