8 ms·
Colossus for Rapid Storage
Was at the Google Next 2025 conference, and they've unveiled a zonal bucket version of GCS and what seems to be a gPRC interface over Google Colossus for Rapid Storage.
- deleted 2y ago[deleted]
- alobrah 2y agoFor some reason, text highlight didn't work, so here's the text-highlighted link: https://cloud.google.com/blog/products/compute/whats-new-with-ai-hypercomputer?e=48754805#:~:text=thinly%2Dprovisioned%20pool.-,Rapid%20Storage,-%3A%20A%20new https://cloud.google.com/blog/products/compute/whats-new-wit...
- xk3 2y agoThe gods strip off interesting bits of URLs when you submit it
- dang 2y agoif you saw that code you wouldn't deify it
- lazide 2y agoMoloch was also a god!
- SweetLlamaMyth 2y agoIt took me 4-5 attempts to not read: > If you saw that code, you wouldn't _defy_ it
- deleted 2y ago[deleted]
- dang 2y agoThat link doesn't work for me, so here's the relevant bit: Rapid Storage: A new Cloud Storage zonal bucket that enables you to colocate your primary storage with your TPUs or GPUs for optimal utilization. It provides up to 20x faster random-read data loading than a Cloud Storage regional bucket. (Normally we wouldn't allow a post like this which cherry-picks one bit of a larger article, but judging by the community response it's clear that you've put your finger on something important, so thanks! We're always game to suspend the rules when doing so is interesting.)
- alobrah 2y agoApologies! First time making a post on hacker news, and I thought this was really exciting news. FWIW, I talked to the presenter after this was revealed during the NEXT conference today, and he seems to have implied that zonal storage is quite close to what Google seems to have with Colossus.
- dang 2y agoOh no, don't apologize - this was a case where you did exactly the right thing and I'm glad you posted! (I was just adding some explanation for more seasoned users who might wonder why we were treating this a bit differently.) Also, welcome to posting on HN and we hope you'll continue!
- noahl 2y agoThere's now another blog post about Rapid storage specifically: https://cloud.google.com/blog/products/storage-data-transfer/how-the-colossus-stateful-protocol-benefits-rapid-storage https://cloud.google.com/blog/products/storage-data-transfer... . (That wasn't up yet when the original post was made.)
- dang 2y agoAh excellent—that's what we were waiting for. I've changed the URL to that from https://cloud.google.com/blog/products/compute/whats-new-with-ai-hypercomputer https://cloud.google.com/blog/products/compute/whats-new-wit... above. Thanks!
- alobrah 2y agoFYI this was unveiled at the 2025 Google Next conference, and they're apparently unveiling a gRPC client for Rapid Storage, which appears to be a very thin wrapper over Colossus itself, as this is just zonal storage.
- jauntywundrkind 2y agoStruggling to find a definition, but seemingly zonal just means there's a massive instance per cluster. Did find some interesting recent (March 28th, 2025) reads though! Colossus under the hood: How we deliver SSD performance at HDD prices https://cloud.google.com/blog/products/storage-data-transfer/how-colossus-optimizes-data-placement-for-performance https://cloud.google.com/blog/products/storage-data-transfer... I kind of thought you meant ZNS / https://zonedstorage.io/ https://zonedstorage.io/ at first, or it's more recent better awesomer counterpart Host Directed Placement (HDP). I wish someone would please please advertize support for HDP, sounds like such a free win, tackling so many write amplification issues for so little extra complexity: just say which stream you want to write to, and writes to that stream will go onto the same superblock. Duh, simple, great.
- re-thc 2y ago> Struggling to find a definition, but seemingly zonal just means there's a massive instance per cluster. There are a number of zones in a region. Region usually means city. Zone can mean data center. Rarely just means some sort of isolation (separate power / network).
- bushbaba 2y agoIt’s GCP’s answer to AWS S3 express zone https://aws.amazon.com/s3/storage-classes/express-one-zone/ https://aws.amazon.com/s3/storage-classes/express-one-zone/
- moandcompany 2y ago"Zonal" relates to the concept of "availability zones" which are the next-smallest unit below a (physical) "region." Most instances of a cloud ___ created in a region are allocated and exist at the zonal level (i.e. a specific zone of a region). A physical "region" usually consists of three or more availability zones, and each zone is physically separated from other zones, limiting the potential for foreseeable disaster events from affecting multiple zones simultaneously. Zones are close enough networking-wise to have high throughput and low latency interconnection, but not as fast as same-rack, same-cluster communications. Systems requiring high availability (or replication) generally attain this by placing instances (or replicas) in multiple availability zones. Systems requiring high-availability generally start with multi-zone replication, and Systems with even higher availability requirements may use multi-region replication, which comes at greater cost.
- hiddencost 2y agoHats off to whoever convinced management that selling Colossus via cloud was Artificial Intelligence. Bravo.
- eitally 2y agoI don't fault them for this at all. AI isn't possible without the full infra stack, which clearly includes storage (and compute, and networking, and data pipelining, and and and...). There's an entire ecosystem of ISVs that only do one of these things, very well (Pure Storage, for example, or Lamba or Coreweave, or Confluent (Kafka + Flink with LLM integration). While it might be more precisely accurate to state "AI enabling" tech, I'll give them a pass.
- rfoo 2y agoI think the joke here is that somehow management refused to sell Colossus (which is such an obvious nice product just like BigQuery) before and it takes "AI" to convince them.
- derefr 2y ago> which is such an obvious nice product just like BigQuery I always assumed (from outside Google) that the problem was that Colossus had to make a "no malicious actors" assumption in its design in order to make the performance/scaling guarantees it does; and that therefore just exposing it directly to the public would make it possible for someone to DoS-attack the Colossus cluster. My logic was that there's actually nothing forcing [the public GCP service of] BigTable to require that a full copy of the dataset be kept hot across the nodes, with pre-reserved storage space — rather than mostly decoupling origin storage from compute† — unless it was to prevent some DoS vector. As for exactly what that DoS vector is... maybe GC/compaction policy-engine logic? (AFAICT, Colossus has pluggable "send compute to data" GC, which internal-BigTable and GCS both use. But external-BigTable forces the GC to be offloaded to the client [i.e. to the BigTable compute nodes the user has allocated] so that the user can't just load down the system with so many complex GC policies that the DC-scale Colossus cluster itself starts to fall behind its GC time budget.) --- † Where by "decouple storage from compute", I mean: • Each compute node gets a fixed-sized DAS diskset, like GCE local NVMe SSDs; • each disk in that diskset gets partitioned up at some fixed ratio, into two virtual disksets; • one virtual diskset gets RAID6'ed or ZFS'ed together, and is used as storage for non-Colossus-synced tablet-LDB nursery level SSTs; • the other virtual diskset gets RAID0'ed or LVM-JBOD-ed together and is used as a bounded-size LFU read-through cache of the Colossus-synced tablets — just like BigQuery compute nodes presumably have. (AFAIK the LDB nursery levels already get force-compacted into "full" [128MiB] Colossus-synced tablets after some quite-short finality interval, so it's not like this increases data loss likelihood by much. And BigTable doesn't guarantee durability for non-replicated keys anyway.)
- acstorage 2y agoSimilar to S3 express one zone
- nodesocket 2y agoIs S3 Express One Zone performance greatly improved to standard S3 like GCP rapid storage? My understanding is S3 Express One Zone is just more cost effective. > 20x faster random-read data loading than a Cloud Storage regional bucket.
- nodesocket 2y agoUpdate: Just read this article[1] which clarifies S3 Express One Zone. Yes, performance is greatly improved, but actually storage costs are 8x more than a standard S3 bucket. The naming S3 Express One Zone is terrible and a bit misleading on pricing changes. [1] https://www.warpstream.com/blog/s3-express-is-all-you-need https://www.warpstream.com/blog/s3-express-is-all-you-need
- cowsandmilk 2y agoI understand your belief that One Zone implies less expensive, but I’m staunchly in favor of them having it in the name so people know that their data is in a single AZ. The storage class succinctly summarizes faster with lower availability.
- nodesocket 2y agoFair, how about instead of S3 Express they call it S3 Max (One Zone). It doesn’t take a rocket scientist to come up with good product names, just copy Apple. Though I suppose what happens when engineers are left up to the marketing. :-)
- onethumb 2y agoIf Apple's so great at naming things, tell me (without looking) which is bigger/better/faster for their CPUs: Max or Ultra?
- carbocation 2y agoThis could actually speed up some of my scientific computing (in some cases, data localization/delocalization is an important part of overall instance run-time). I will be interested to try it.
- bushbaba 2y agoGlad to see the zonal object store take off. Such massive bandwidth speed will re define data analytics where 99% of all queries able to run on a single node faster than what distributed compute can offer.
- pj_mukh 2y agoSuper interesting! Rapid Storage especially, very useful, but that first line: "Today's innovation isn't born in a lab or at a drafting board; it's built on the bedrock of AI infrastructure. " Uhh..No. Even as an AI developer I can tell that is some AI Comms person tripping over.
- deleted 2y ago[deleted]
- simonw 2y agoEveryone needs to learn to use a single, unique, unambiguous URL for new product announcements like this. Google aren't the only company that consistently mess this up, but given how they built a 1.95 trillion company on top of crawling URLs on the web they really should have an internal culture that values giving things unique URLs! [I had to learn this lesson myself: I used to blog "weeknotes" every week or two where I'd bundle all of my project announcements together and it sucked not being able to link to them as individual posts]
- decimalenough 2y agoGoogle's not really at fault here: the OP submitted a link to an article called "Introducing Ironwood TPUs and new innovations in AI Hypercomputer" that happens to mention Rapid Storage way down the page. In case marketing seems to move faster than documentation though, since I can't find any mention of this in the main GCS docs. https://cloud.google.com/search?hl=en&q=rapid%20storage https://cloud.google.com/search?hl=en&q=rapid%20storage
- deleted 2y ago[deleted]
- zifpanachr23 2y agoReading the press release about the "Hypercomputer" and I can't tell what part of this is real and what part is marketing. They say it comes in two configuration, 256 chips or 9,216 chips. They also say that the maximal configuration of 9,216 chips delivers 24x the compute power of the world's largest supercomputer (which they say is called El Capitan). They say that this comes to 42.6 exaFLOPs. This implies that the 9,216 chip configuration doesn't actually exist in any form in reality, or else it would now be the world's largest supercomputer (by flops) by a huge margin. Am I massively misunderstanding what the claims being made are about the TPU and the 42.6 exaFLOPs? I feel like this would be much bigger news if this was fully legit. Edit: The flops being benchmarked are not the same as regular supercomputer flops.
- phonon 2y agoSupercomputers are measured based on 64 bit floating point operations. Here they (inaptly) compared it to their 8 bit floating point operations (which are only useful for AI workloads).
- zifpanachr23 2y agoGotcha. That makes a lot more sense. I was led to believe by the wording of the comparison that they were the same operations. Appreciate the explanation.
- onlyrealcuzzo 2y agoWhy is it inapt? If all you care about is an 8-bit AI workload (there's definitely a market for that), it's nice to have 24x the speed.
- remus 2y agoIt's an apples to oranges comparison.
- onlyrealcuzzo 2y agoIt's apples to apples if you care about 8-bit (a lot of people do these days). AFAIK, there wasn't a faster 8-bit super computer to compare to - which is why they made the comparison.
- deleted 2y ago[deleted]
- wetpaws 2y ago[dead]
- acstorage 2y agoIf you want object storage faster than S3 Express One Zone or GCP Rapid Storage without the zonal limitation check out ACS: https://acceleratedcloudstorage.com https://acceleratedcloudstorage.com You can bring data in and out of the GPU quickly and improve utilization.
- deleted 2y ago[deleted]
- EE84M3i 2y agoIs this related at all the the private invite only anywhere caches? (or maybe they're GA now?) https://cloud.google.com/storage/docs/anywhere-cache https://cloud.google.com/storage/docs/anywhere-cache
- leg 2y agoAnywhere Cache and Rapid Storage share some infrastructure inside of GCS and both are good solutions for improving GCS performance, but Anywhere Cache is an SSD cache in front of the normal buckets while Rapid Storage is a new type of bucket. (I work on Google storage)
- minzi 2y agoCan you expand a bit on when it would make sense to use one versus the other?
- leg 2y agoAnywhere Cache shines in front of a multi-regional bucket. Once the data is cached, there's no egress charges and there's much better latency. This is great for someone who looks for spot compute capacity to run computations anywhere in the multi-region. It will also improve performance in front of regional buckets but as a cache, you'll see the difference between hits and misses. Rapid Storage will have all of your data local and fast, including writes. It also adds the ability to have fast durable appends, which is something you can't get from the standard buckets.
- miroljub 2y agoLike with any other new Google product, better wait a few years to see if it sticks before investing in its usage. In most cases, you'd be better off searching for an alternative from the start.
- davedx 2y agoTerrifyingly complicated and buzzword packed. I really don't know what to make of any of this or what it does, and I work with AI applications in my day job. I'm guessing the $300 of Google Cloud credit offered in this webpage wouldn't go very far using any of this stuff?
- lysecret 2y agoYou can try out everything for 300 dollars easily. Most expensive thing you can do is get a server with 8 h200s and spend 90 dollars an hour.
- devops000 2y agoIs it like PureStorage?
- deleted 2y ago[deleted]
- akshayshah 2y agoVery cool! This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, and transparently-replicated dual-region object storage - all with the same API. For infra systems, this is great: code against the GCS API, and let the user choose the cost/latency/durability tradeoffs that make sense for their use case.
- dastbe 2y ago? s3: https://aws.amazon.com/pm/serv-s3 https://aws.amazon.com/pm/serv-s3 s3 express: https://aws.amazon.com/s3/storage-classes/express-one-zone/ https://aws.amazon.com/s3/storage-classes/express-one-zone/ cross-region replication: https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html https://docs.aws.amazon.com/AmazonS3/latest/userguide/replic...
- akshayshah 2y agoThe cross-region replication I’ve seen for S3 (including the link you’ve provided) is fundamentally different from a dual-region GCS bucket. AWS is providing a way to automatically copy objects between distinct buckets, while GCS is providing a single bucket that spans multiple regions. It’s much, much easier to code against a dual-region GCS bucket because the bucket namespace and object metadata are strongly consistent across regions.
- jeffbee 2y agoThe semantics they are offering are very different from S3. In Colossus a writer can make a durable 1-byte append and other observers are able to reason about the commit point. S3 does not offer this property.
- dastbe 2y agoSure, but that's not what the parent said.
- korkybuchek 2y ago> This makes Google the only major cloud that has low-latency single-zone object storage, standard regional object storage, Absurd claim. S3 Express launched last year.
- leg 2y agoThere's a detailed blog post about Rapid Storage now available, see https://news.ycombinator.com/item?id=43645309 https://news.ycombinator.com/item?id=43645309 (I work on Google storage)
- dang 2y agoThanks! I've changed the URL of the current thread and re-upped this one. More at https://news.ycombinator.com/item?id=43646209 https://news.ycombinator.com/item?id=43646209.
- dang 2y ago(This was posted last night with https://cloud.google.com/blog/products/compute/whats-new-with-ai-hypercomputer https://cloud.google.com/blog/products/compute/whats-new-wit... above. We've changed the URL to the product-specific article.)
- nashashmi 2y agoThis link makes so much more sense than the previous link did. SSDs with high random I/o speeds are a significant contributor to the advantage. I think 20m writes per second are likely distributed over a network of drives to make that kind of speed possible.
- steveBK123 2y agoHad to go back to the classic microservices video as I was pretty sure they used Colossus but it was actually Galactus & Omega Star.
- thethimble 2y agoThis is what OP is referring to in case you haven’t been enlightened https://youtu.be/y8OnoxKotPQ?si=JAK5iPMcG1yoAhiT https://youtu.be/y8OnoxKotPQ?si=JAK5iPMcG1yoAhiT
- __turbobrew__ 2y agoI want chubby as a service so I can throw etcd and zookeeper in the trash.