4 ms·

in a properly designed install you can actually use the compressors and fans for smoothing load spikes. won't be much, but why not. edit: otherwise I'm not get

by lstodd 1y ago

in a properly designed install you can actually use the compressors and fans for smoothing load spikes. won't be much, but why not. edit: otherwise I'm not getting what the entire article is about. it's as contrary to what I know about datacenter design as it can get. it's.. just wrong.

touisteur 1y ago
I'm thinking of sequences of 'put the sharded dataset through the ten thousand 2kW GPUs, then wait on network - all-reduce - then spike again - a mostly-synchronous all-on/all-off loop. Watching how quick they get to boost-frequency I can see where the worries come from.
- lstodd 1y ago
  does anyone actually do those kind of loads over entire dcs? because if so, I have some nice east-european guys to teach them proper load-balancing.
  - 0cf8612b2e1e 1y ago
    Wouldn’t that situation arise when a company is training their top end model? Facebook/Google/DeepSeek probably trained on thousands of collocated GPUs. The bigger the cluster, the bigger the sync delays between batches as the model data gets shunted back and forth.
    - deleted 1y ago
      [deleted]
    - lstodd 1y ago
      basically it's a load-spreading problem. this can be mitigated entirely by control plane alone. unless you think they really do a cluster-per-datacenter and in that case, I cannot believe the DC in question were not designed for peak and transient loads both in AC and power supply. Besides it would be stupud.
      - jwatte 1y ago
        Yes, the data center was designed for peak load. Unfortunately, the data center is hooked up to the grid. De-coupling that connection is what this article is literally about. Then again, some data centers just use re-generation from giant flywheels, where the grid powers the flywheel to build up inertia, and thus load can be smoothed that way. The flywheels need to keep running at 60 Hz for at least 30 seconds to give the Diesel generators time to start, should the grid fail. Run a flywheel at the main transformer, run batteries at your PDC and add supercaps at each point of load, and you may very well be able to show a much smoother load to the grid.
  - jwatte 1y ago
    Yes, they do, and no, they don't want to wait longer for the result just because you think they shouldn't.

by lstodd 1y ago

touisteur 1y ago

I'm thinking of sequences of 'put the sharded dataset through the ten thousand 2kW GPUs, then wait on network - all-reduce - then spike again - a mostly-synchronous all-on/all-off loop. Watching how quick they get to boost-frequency I can see where the worries come from.

lstodd 1y ago
does anyone actually do those kind of loads over entire dcs? because if so, I have some nice east-european guys to teach them proper load-balancing.
- 0cf8612b2e1e 1y ago
  Wouldn’t that situation arise when a company is training their top end model? Facebook/Google/DeepSeek probably trained on thousands of collocated GPUs. The bigger the cluster, the bigger the sync delays between batches as the model data gets shunted back and forth.
  - deleted 1y ago
    [deleted]
  - lstodd 1y ago
    basically it's a load-spreading problem. this can be mitigated entirely by control plane alone. unless you think they really do a cluster-per-datacenter and in that case, I cannot believe the DC in question were not designed for peak and transient loads both in AC and power supply. Besides it would be stupud.
    - jwatte 1y ago
      Yes, the data center was designed for peak load. Unfortunately, the data center is hooked up to the grid. De-coupling that connection is what this article is literally about. Then again, some data centers just use re-generation from giant flywheels, where the grid powers the flywheel to build up inertia, and thus load can be smoothed that way. The flywheels need to keep running at 60 Hz for at least 30 seconds to give the Diesel generators time to start, should the grid fail. Run a flywheel at the main transformer, run batteries at your PDC and add supercaps at each point of load, and you may very well be able to show a much smoother load to the grid.
- jwatte 1y ago
  Yes, they do, and no, they don't want to wait longer for the result just because you think they shouldn't.