4 ms·

GFS was terrible. It may have been great in the early days of google when they couldn't build an index, but I recall having to bribe hwops people (with single-

by dekhn 3y ago

GFS was terrible. It may have been great in the early days of google when they couldn't build an index, but I recall having to bribe hwops people (with single-malt) to upgrade RAM on the GFS masters, and a few times, my team (Ads) threatened to run our own filesystems that didn't fail so badly. Colossus, on the other hand, is really good.

deleted 3y ago
[deleted]
the-rc 3y ago
Among other things, GFS ran outside of Borg, so you couldn't just donate quota to let it run with more resources, as you found out. Conversely, given that Colossus ran on Borg, part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown. Then there were all the issues on the low end machines (hello, Nocona) that had not enough resources to run the storage stack AND user jobs. Stuff wouldn't schedule (and the cluster suddenly appeared to be missing some disk), but at least that was a lot more explicit than the performance issues you witnessed on GFS.
- robertlagrant 3y ago
  > part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown I've no idea what this is, but it sounds fascinating. Was this a way to auto-allocate more resource based on rules?
  - marklar423 3y ago
    What I think the GP means is previously, GFS ran on their own machines and didn't need to participate in the Google-wide quota system for CPU/RAM. However Colossus _does_ run on the general Google compute infrastructure, and so needed a way to get CPU/RAM quota where none existed (because it was all used by existing, non-storage users).
  - the-rc 3y ago
    Instead of using five masters like GFS, which typically ran on five machines dedicated to the purpose, Colossus uses a variable number of so called curators to handle all the metadata work (think of directory entries and inodes in Unix, separate from the data blocks). They're just regular Borg tasks, running along those from other services like YouTube, Gmail, Reader, etc. The more storage added to the cluster, the more traffic it gets and the busier the curators become. For very small clusters five or fewer curators might suffice, but when you hit hundreds of petabytes, you need many, many more of them. Borg requires identities, including the system account that runs the Colossus service, to have quota, so there needs to be a way to grant it an amount of CPU cores and memory somewhat proportional to the storage cluster size. (Fun things happen when the cell shrinks instead of growing.) Monitoring the entire stack also requires resources of its own, proportional to cluster size, so there were additional grants at higher priority just for those jobs.
    - crotchfire 3y ago
      I've always wondered how Google does this internally. For all the cloud companies, "quota" is literally dollars. Bandwidth, memory, cpu, these are things they sell. So they are measured in money. The existing corporate budgetary process already manages dollars (indeed, that's its main function). If you want more of one of these things, you'll be displacing some customer who stood ready to pay for them (i.e. on the spot market), so you need to replace the revenue they were providing. Very straightforward. What does google measure all these things in? Quotons? Where do quotons come from? When google becomes more profitable, does the Central Bank of Quoton print more quotons each month than they had previously? Does Google have a Chief Quoton Officer? Totally curious about this. Google is the only company operating on this scale that doesn't sell raw bandwidth/cpu/ram as their product, nor do they buy these things from underlying cloud vendors (like e.g. Cloudflare does). This makes a lot of things simpler internally and probably avoids a lot of internal conflicts, but it does mean that they have to have a parallel internal accounting system for this stuff. Is there an official exchange rate between the two accounting systems? Are different teams allowed to sell each other quotons, in exchange for budget dollars? Has the IRS ever audited Google's Quoton books? Okay that last question was not serious. But the rest were.
tytso 3y ago
Single malt was the currency of choice when bribing and/or placating SRE's and hwops folks. For example, if a SWE botched a rollout that caused a multiple SRE's to get paged at 3am, a bottle of single malt donated to the SRE bar was considered a way of apologizing.
- windenntw 3y ago
  Single Malt, specially Islays, are still very much welcomed :)

by dekhn 3y ago

deleted 3y ago

[deleted]

the-rc 3y ago

Among other things, GFS ran outside of Borg, so you couldn't just donate quota to let it run with more resources, as you found out. Conversely, given that Colossus ran on Borg, part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown. Then there were all the issues on the low end machines (hello, Nocona) that had not enough resources to run the storage stack AND user jobs. Stuff wouldn't schedule (and the cluster suddenly appeared to be missing some disk), but at least that was a lot more explicit than the performance issues you witnessed on GFS.

robertlagrant 3y ago
> part of the migration involved an elaborate system that minted CPU/RAM quota for the storage system users whenever the CFS cell was grown I've no idea what this is, but it sounds fascinating. Was this a way to auto-allocate more resource based on rules?
- marklar423 3y ago
  What I think the GP means is previously, GFS ran on their own machines and didn't need to participate in the Google-wide quota system for CPU/RAM. However Colossus _does_ run on the general Google compute infrastructure, and so needed a way to get CPU/RAM quota where none existed (because it was all used by existing, non-storage users).
- the-rc 3y ago
  Instead of using five masters like GFS, which typically ran on five machines dedicated to the purpose, Colossus uses a variable number of so called curators to handle all the metadata work (think of directory entries and inodes in Unix, separate from the data blocks). They're just regular Borg tasks, running along those from other services like YouTube, Gmail, Reader, etc. The more storage added to the cluster, the more traffic it gets and the busier the curators become. For very small clusters five or fewer curators might suffice, but when you hit hundreds of petabytes, you need many, many more of them. Borg requires identities, including the system account that runs the Colossus service, to have quota, so there needs to be a way to grant it an amount of CPU cores and memory somewhat proportional to the storage cluster size. (Fun things happen when the cell shrinks instead of growing.) Monitoring the entire stack also requires resources of its own, proportional to cluster size, so there were additional grants at higher priority just for those jobs.
  - crotchfire 3y ago
    I've always wondered how Google does this internally. For all the cloud companies, "quota" is literally dollars. Bandwidth, memory, cpu, these are things they sell. So they are measured in money. The existing corporate budgetary process already manages dollars (indeed, that's its main function). If you want more of one of these things, you'll be displacing some customer who stood ready to pay for them (i.e. on the spot market), so you need to replace the revenue they were providing. Very straightforward. What does google measure all these things in? Quotons? Where do quotons come from? When google becomes more profitable, does the Central Bank of Quoton print more quotons each month than they had previously? Does Google have a Chief Quoton Officer? Totally curious about this. Google is the only company operating on this scale that doesn't sell raw bandwidth/cpu/ram as their product, nor do they buy these things from underlying cloud vendors (like e.g. Cloudflare does). This makes a lot of things simpler internally and probably avoids a lot of internal conflicts, but it does mean that they have to have a parallel internal accounting system for this stuff. Is there an official exchange rate between the two accounting systems? Are different teams allowed to sell each other quotons, in exchange for budget dollars? Has the IRS ever audited Google's Quoton books? Okay that last question was not serious. But the rest were.

tytso 3y ago

Single malt was the currency of choice when bribing and/or placating SRE's and hwops folks. For example, if a SWE botched a rollout that caused a multiple SRE's to get paged at 3am, a bottle of single malt donated to the SRE bar was considered a way of apologizing.

windenntw 3y ago
Single Malt, specially Islays, are still very much welcomed :)