7 ms·

Link to the paper in the README is broken. I believe this is the correct link to the referenced paper: https://arxiv.org/abs/2411.07231 https://arxiv.org/abs/24

by wkirby 2y ago

Link to the paper in the README is broken. I believe this is the correct link to the referenced paper: https://arxiv.org/abs/2411.07231 https://arxiv.org/abs/2411.07231

Jaxan 2y ago
There is some nice information in the appendix, like: “One training with a schedule similar to the one reported in the paper represents ≈ 30 GPU-days. We also roughly estimate that the total GPU-days used for running all our experiments to 5000, or ≈ 120k GPU-hours. This amounts to total emissions in the order of 20 tons of CO2eq.” I am not in AI at all, so I have no clue how bad this is. But it’s nice to have some idea of the costs of such projects is.
- svilen_dobrev 2y ago
  so say i have a site with 3000 images, 2M pixel each. How many GPU-months it would take to mark them? And, what gigabytes i would have to keep for the model?
  - hnuser123456 2y ago
    That amount of compute was used for training. For inference (applying the watermarks), hopefully no more than a few seconds per image. Llama 3 70B took 6.4M GPU hours to train, emitting 1900 tons of CO2 equivalent.
    - Jaxan 2y ago
      Thanks! I was not at all aware of the scale of training! To me those are crazy amounts of gpu time and resources.
      - pierrefdz 2y ago
        The amounts of gpu time in the paper are for all experiments, not just training the last model that is OSS (which is usually reported). People don't just oneshot the final model.
  - GaggiX 2y ago
    The embedder is only 1.1M parameters, so it should run extremely fast.
    - pierrefdz 2y ago
      Yes, although the number of parameters is not directly linked with the flops/speed of inference. What's nice about this AE architecture is that most of the compute (message embedding, and merging) is done at low resolution, same idea as behind latent diffusion models
- azinman2 2y ago
  It’s very interesting this is gpu time based because: 1. Different energy sources produce varyings of co2 2. This likely does not include co2 to make the GPUs or machines 3. Humans involved are not added to this at all, and all of the impact they have on the environment 4. No ability to predict future co2 from using this work. Also if it really matters, then why do it at all? If we’re saying hey this is destroying the environmental and care, then maybe don’t do that work?
  - TeMPOraL 2y ago
    > 1. Different energy sources produce varyings of co2 Yes. > 2. This likely does not include co2 to make the GPUs or machines Definitely not, nobody does that. Wish they did, in general I feel like a lot of beliefs around sustainability and environmentalism are wrong or backwards precisely because embodied energy is discounted; see e.g. stats on western nations getting cleaner, where a large - if not primary - driver of improved stats is just outsourcing manufacturing, so emissions are attributed to someone else. Anyway, embodied energy isn't particularly useful here. Energy embodied in GPUs and machines amortizes over their lifetimes and should be counted against all the things those GPUs did, do and will do, of which the training in question is just a small part. Not including it isolates the analysis to contributions from the specific task per se, and makes the results applicable to different hardware/scenarios. > 3. Humans involved are not added to this at all, and all of the impact they have on the environment This metric is so ill-defined as to be arbitrary. Even more so with conjunction with 2, as you could plausibly include a million people into it. > 4. No ability to predict future co2 from using this work. Total, no. Contribution of compute alone given similar GPU-hours per ton of CO2eq, yes.
    - _Algernon_ 2y ago
      >Definitely not, nobody does that. Except every proper Life-cycle assessment on carbon emissions ever.
      - Jerrrrrrry 2y ago
        >proper doing Scotsman-like lifting when the point was that these things are not considered, or are "externalities"
- deleted 2y ago
  [deleted]
- barbazoo 2y ago
  > This amounts to total emissions in the order of 20 tons of CO2eq. That's about 33 economy class roundtrip flights from LAX to JFK. https://www.icao.int/environmental-protection/Carbonoffset/Pages/default.aspx https://www.icao.int/environmental-protection/Carbonoffset/P...
  - kube-system 2y ago
    33 seats on a flight maybe. It's about one passenger aircraft flight, one way.
    - echelon 2y ago
      And it has produced a system superior to several engineers working full time for several years. Seems like a fair carbon trade.
      - somishere 2y ago
        But is it a trade? Feels additive. Assuming same engineers will continue spending their carbon budget elsewhere ...
      - samplatt 2y ago
        Assuming you're purchasing from someone with infinite carbon credits and you're spending it in an environment with infinite ability to re-sink the carbon. Sure.
        Tostino 2y ago
        Are you applying that same rigor to every action people undertake daily?
        samplatt 2y ago
        To a more and lesser degree depending on the action, I try to apply "that rigor" to myself, at least? And yes, I think the world would be better off if more people considered how their decisions impact others, if that's what you're getting at, but it's unrealistic to expect everyone to care about other people - and of course entirely impossible to account for ALL variables.

by wkirby 2y ago

Link to the paper in the README is broken. I believe this is the correct link to the referenced paper: https://arxiv.org/abs/2411.07231 https://arxiv.org/abs/2411.07231

Jaxan 2y ago

There is some nice information in the appendix, like: “One training with a schedule similar to the one reported in the paper represents ≈ 30 GPU-days. We also roughly estimate that the total GPU-days used for running all our experiments to 5000, or ≈ 120k GPU-hours. This amounts to total emissions in the order of 20 tons of CO2eq.” I am not in AI at all, so I have no clue how bad this is. But it’s nice to have some idea of the costs of such projects is.

svilen_dobrev 2y ago
so say i have a site with 3000 images, 2M pixel each. How many GPU-months it would take to mark them? And, what gigabytes i would have to keep for the model?
- hnuser123456 2y ago
  That amount of compute was used for training. For inference (applying the watermarks), hopefully no more than a few seconds per image. Llama 3 70B took 6.4M GPU hours to train, emitting 1900 tons of CO2 equivalent.
  - Jaxan 2y ago
    Thanks! I was not at all aware of the scale of training! To me those are crazy amounts of gpu time and resources.
    - pierrefdz 2y ago
      The amounts of gpu time in the paper are for all experiments, not just training the last model that is OSS (which is usually reported). People don't just oneshot the final model.
- GaggiX 2y ago
  The embedder is only 1.1M parameters, so it should run extremely fast.
  - pierrefdz 2y ago
    Yes, although the number of parameters is not directly linked with the flops/speed of inference. What's nice about this AE architecture is that most of the compute (message embedding, and merging) is done at low resolution, same idea as behind latent diffusion models
azinman2 2y ago
It’s very interesting this is gpu time based because: 1. Different energy sources produce varyings of co2 2. This likely does not include co2 to make the GPUs or machines 3. Humans involved are not added to this at all, and all of the impact they have on the environment 4. No ability to predict future co2 from using this work. Also if it really matters, then why do it at all? If we’re saying hey this is destroying the environmental and care, then maybe don’t do that work?
- TeMPOraL 2y ago
  > 1. Different energy sources produce varyings of co2 Yes. > 2. This likely does not include co2 to make the GPUs or machines Definitely not, nobody does that. Wish they did, in general I feel like a lot of beliefs around sustainability and environmentalism are wrong or backwards precisely because embodied energy is discounted; see e.g. stats on western nations getting cleaner, where a large - if not primary - driver of improved stats is just outsourcing manufacturing, so emissions are attributed to someone else. Anyway, embodied energy isn't particularly useful here. Energy embodied in GPUs and machines amortizes over their lifetimes and should be counted against all the things those GPUs did, do and will do, of which the training in question is just a small part. Not including it isolates the analysis to contributions from the specific task per se, and makes the results applicable to different hardware/scenarios. > 3. Humans involved are not added to this at all, and all of the impact they have on the environment This metric is so ill-defined as to be arbitrary. Even more so with conjunction with 2, as you could plausibly include a million people into it. > 4. No ability to predict future co2 from using this work. Total, no. Contribution of compute alone given similar GPU-hours per ton of CO2eq, yes.
  - _Algernon_ 2y ago
    >Definitely not, nobody does that. Except every proper Life-cycle assessment on carbon emissions ever.
    - Jerrrrrrry 2y ago
      >proper doing Scotsman-like lifting when the point was that these things are not considered, or are "externalities"
deleted 2y ago
[deleted]
barbazoo 2y ago
> This amounts to total emissions in the order of 20 tons of CO2eq. That's about 33 economy class roundtrip flights from LAX to JFK. https://www.icao.int/environmental-protection/Carbonoffset/Pages/default.aspx https://www.icao.int/environmental-protection/Carbonoffset/P...
- kube-system 2y ago
  33 seats on a flight maybe. It's about one passenger aircraft flight, one way.
  - echelon 2y ago
    And it has produced a system superior to several engineers working full time for several years. Seems like a fair carbon trade.
    - somishere 2y ago
      But is it a trade? Feels additive. Assuming same engineers will continue spending their carbon budget elsewhere ...
    - samplatt 2y ago
      Assuming you're purchasing from someone with infinite carbon credits and you're spending it in an environment with infinite ability to re-sink the carbon. Sure.
      - Tostino 2y ago
        Are you applying that same rigor to every action people undertake daily?
        samplatt 2y ago
        To a more and lesser degree depending on the action, I try to apply "that rigor" to myself, at least? And yes, I think the world would be better off if more people considered how their decisions impact others, if that's what you're getting at, but it's unrealistic to expect everyone to care about other people - and of course entirely impossible to account for ALL variables.