4 ms·

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but

Something else I find exciting, starting with one of the reflections- The original training took 3 days on a Sun 4/260 workstation; I can't find specifics but I believe that era of early SPARC workstations would likely pull about 200 watts in total (the CPU wasn't super high powered but the whole system, running with the disks and the monitor etc would pull about that). So 200 watts * 72 hours = 14400 watt-hours of energy. Karpathy trained the equivalent on a Macbook, not even fully utilized, in 90 seconds. Likely something around 20 watts * 0.025 hours = 0.5 watt-hours. An energy efficiency improvement of nearly 30000x.

fouronnes3 3y ago
This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.
- eru 3y ago
  It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)
  - nojvek 3y ago
    To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-laptop-uses https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
    - renonce 3y ago
      It is “the greatest” because we only appreciate intelligence that we ourselves understand. A 0.0001W calculator calculates arithmetic faster than any human brain.
    - beebmam 3y ago
      You're leaving out the training requirements
    - pyinstallwoes 3y ago
      That’s not entirely fair. That’s run time cost but not the same cost to information ratio and the cost of drawing from that pool. Given a laptop is at 30W how much can a laptop do disconnected from the internet? Now how much can it do with the internet? Now how much information does the internet cost in terms of wattage? Now what’s the ratio?
      - eru 3y ago
        What can a human do disconnected from society?
  - chubot 3y ago
    I dispute that, if the metric is a chess game between an ant and a human
- quickthrower2 3y ago
  For inference that could be useful, but the energy is not for the model it is for at least the tuple of: model, model architecture and compilation, and hardware chosen.
Aardwolf 3y ago
30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then
- ben_w 3y ago
  It isn't much, but as the link says, the neural network they were reimplementing is too small to take advantage of modern hardware.
- swores 3y ago
  My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be as poor as mine: Doubling every 2 years = compound annual growth rate (CAGR) of ~41.42% CAGR = ((End Value / Start Value)^(1 / Number of Years)) - 1 ((2 / 1)^(1 / 2)) - 1 = 0.41421356237 Therefore in 34 years since then: 1 * (1 + 0.41421356237)^34 = ~131,072 So x30k is ~4.4x less than 131k. Then again, that's equivalent to ~x1.833 every two years, compared to Moore's Law of x2 every two years, so only ~8% less growth per two years, which coming back to the fact that Moore's Law is a rough estimate concept not an exact fact, doesn't seem to far off!
  - jacquesm 3y ago
    The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison. Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it, but it hasn't run in over a year and the elderly thinkpad that is now my daily driver uses far less power and still has enough compute to serve my modest needs. But if I would dive into something requiring much more compute I'd have to start the desktop again. Unfortunately power management is not such that computers can really throttle down to 'miser mode' when you don't need it, it's a good step but not as good as the jump between desktop and laptop.
Yajirobe 3y ago
> watt-hours You mean joules (up to a constant factor)?
- renonce 3y ago
  A watt-hour is 3600 joules but watt-hours or kilowatt-hours is commonly used because it’s easier to calculate.

fouronnes3 3y ago

This is very interesting, because I've always thought that all NN performance should be measured in a unit with energy in the denominator.

eru 3y ago
It totally depends on what you want to use a measure for. Just like neither height or volume alone will tell you what will fit in your car. By any measure that puts energy used by the brain in the denominator, humans are probably dumber than ants. But that doesn't mean those measures are always accurate. (For contemporary neural networks, you also have to distinguish training costs from inference costs.)
- nojvek 3y ago
  To add more context, humans are 100W biological machines. Brain is ~20% of that power - 20W. The greatest form of general intelligence at 20W. A MacBook Air is ~30W. https://www.jackery.com/blogs/knowledge/how-many-watts-a-laptop-uses https://www.jackery.com/blogs/knowledge/how-many-watts-a-lap...
  - renonce 3y ago
    It is “the greatest” because we only appreciate intelligence that we ourselves understand. A 0.0001W calculator calculates arithmetic faster than any human brain.
  - beebmam 3y ago
    You're leaving out the training requirements
  - pyinstallwoes 3y ago
    That’s not entirely fair. That’s run time cost but not the same cost to information ratio and the cost of drawing from that pool. Given a laptop is at 30W how much can a laptop do disconnected from the internet? Now how much can it do with the internet? Now how much information does the internet cost in terms of wattage? Now what’s the ratio?
    - eru 3y ago
      What can a human do disconnected from society?
- chubot 3y ago
  I dispute that, if the metric is a chess game between an ant and a human
quickthrower2 3y ago
For inference that could be useful, but the energy is not for the model it is for at least the tuple of: model, model architecture and compilation, and hardware chosen.

Aardwolf 3y ago

30k doesn't even sound like that much to me given Moore's law. I'd expect more improvement since 1989. Supercomputer performance increased more than a million since then

ben_w 3y ago
It isn't much, but as the link says, the neural network they were reimplementing is too small to take advantage of modern hardware.
swores 3y ago
My (wrong) intuition on reading your comment was that you were over-estimating the expected growth in performance over that time period, but actually after checking the maths based on Moore's Law, i.e. doubling every two years (though of course I understand that was a rough estimate, more of a concept prediction than expected to be precise) you're right so I'll share the maths for anyone else whose intuition might be as poor as mine: Doubling every 2 years = compound annual growth rate (CAGR) of ~41.42% CAGR = ((End Value / Start Value)^(1 / Number of Years)) - 1 ((2 / 1)^(1 / 2)) - 1 = 0.41421356237 Therefore in 34 years since then: 1 * (1 + 0.41421356237)^34 = ~131,072 So x30k is ~4.4x less than 131k. Then again, that's equivalent to ~x1.833 every two years, compared to Moore's Law of x2 every two years, so only ~8% less growth per two years, which coming back to the fact that Moore's Law is a rough estimate concept not an exact fact, doesn't seem to far off!
- jacquesm 3y ago
  The rest of that difference can easily be explained by the difference in the class of hardware used. A desktop made today vs a laptop is roughly that factor 4. Not sure if back then there would have been laptops that you could have done this on for a more apples-to-apples comparison. Modern laptops give great efficiency, when I went for solar power here the first thing to go was the desktop computer. I still have it, but it hasn't run in over a year and the elderly thinkpad that is now my daily driver uses far less power and still has enough compute to serve my modest needs. But if I would dive into something requiring much more compute I'd have to start the desktop again. Unfortunately power management is not such that computers can really throttle down to 'miser mode' when you don't need it, it's a good step but not as good as the jump between desktop and laptop.

Yajirobe 3y ago

> watt-hours You mean joules (up to a constant factor)?

renonce 3y ago
A watt-hour is 3600 joules but watt-hours or kilowatt-hours is commonly used because it’s easier to calculate.