5 ms·

I have published an addendum to an article I wrote about AlphaChip (https://vighneshiyer.com/misc/ml-for-placement/ https://vighneshiyer.com/misc/ml-for-placeme

I have published an addendum to an article I wrote about AlphaChip (https://vighneshiyer.com/misc/ml-for-placement/ https://vighneshiyer.com/misc/ml-for-placement/) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general. In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still makes it non-competitive with commercial autofloorplanners and AutoDMP. Regardless, I think the ISPD authors owe the Nature authors an even more rigorous study that addresses all their criticisms. Even if they just try to evaluate the pre-trained checkpoint that Google published, that would be a useful piece of data to add to the debate.

wholehog 2y ago
We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the rest of the chip design process. (For context, the licensing costs alone for most commercial EDA software are in the millions of dollars.)
- bushbaba 2y ago
  h100 GPU instances are multiple orders of magnitude more expensive.
  - radq 2y ago
    Not true, H100s cost $2-3/GPU/hr on the open market.
    - menaerus 2y ago
      Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day income?
      - Jabbles 2y ago
        7 https://www.google.com/search?q=0.5e6%2F8%2F24%2F365 https://www.google.com/search?q=0.5e6%2F8%2F24%2F365
        menaerus 2y ago
        Did you really not understand rethoric nature of my question and assumed that I can't do 1st grade primary school math?
      - solidasparagus 2y ago
        Who cares? That's someone else's problem. I just pay 2-3$/hr and the H100s are usable
        menaerus 2y ago
        Look, I understand that some people are short-sighted and can hardly think out of the box and that is totally fine by me. I don't judge you for being that so I kindly ask you not to judge my question. Learn to give some benefit of the doubt.
        sangnoir 2y ago
        You should care about counterparty risks. If your business model depends on unsustainable 3rd party prices powered by VC largesse and unrealizable dreams of dominance, the very least you can do is plan for the impending reckoning, after which GPU proces will be determined by costs.
  - YetAnotherNick 2y ago
    H100 GPUs are more or less similar in price/performance. It is 2-3x more expensive per hour for 2-3x higher performance.
- vighneshiyer 2y ago
  You are correct. For commercial use, the GPUs used for training and fine-tuning aren't a problem financially. However, if we wanted to rigorously benchmark AlphaChip against simulated annealing or other floorplanning algorithms, we have to afford the same compute and runtime budget to each algorithm. With 16 GPUs running for 6 hours, you could explore a huge placement space using any algorithm, and it isn't clear if RL will outperform the other ones. Furthermore, the runtime of AlphaChip as shown in the Nature paper and ISPD was still significantly greater than Cadence's concurrent macro placer (even after pre-training, RL requires several hours of fine-tuning on the target problem instance). Arguably, the runtime could go down with more GPUs, but at this point, it is unclear how much value is coming from the policy network / problem embedding vs the ability to explore many potential placements.
  - Jabbles 2y ago
    You're saying that if the other methods were given the equivalent amount of compute they might be able to perform as well as AlphaChip? Or at least that the comparison would be fairer? Are the other methods scalable in that way?
    - pclmulqdq 2y ago
      Yes, they are. The other approaches usually look like simulated annealing, which has several hyperparameters that control how much computing is used and improve results with more compute usage.
      - wholehog 2y ago
        See my comment above - the Nature authors already did this, and tried a huge hyperparameter sweep for SA, and RL still won. See appendix of the Nature article: rdcu.be/cmedX
        pclmulqdq 2y ago
        I understand and have read the article. Running 80 experiments with a crude form of simulated annealing is at most 0.0000000001% of the effort that has been spent on making that kind of hill climb work well by traditional EDA vendors. That is also an in-sample comparison, where I would believe the Google thing pre-trained on Google chips would do well, while it might have a harder time with a chip designed by a third party (further from its pre-training). The modern versions of that hill climb also use some RL (placing and routing chips is sort of like a game), but not in the way Jeff Dean wants it to be done.
nemonemo 2y ago
In the conclusion of the article, you said: "While I concede that there are things the ISPD authors could have done better, their conclusion is still sound. The Nature authors do not address the fact that CMP and AutoDMP outperform CT with far less runtime and compute requirements." One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this? Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article toward the AlphaChip work (words like "arrogance", "disdain", "hyperbole", "belittling", "hostile" for AlphaChip authors, as opposed to "excellent" for a Synopsys VP.) Could you share where this difference in tone originates?
- vighneshiyer 2y ago
  > One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this? I believe this is a fair criticism, and it could be a reason why the ISPD Tensorboard shows divergence during training for some RTL designs. The ISPD authors provide their own justification for their substitution of training time for compute resources in page 11 of their paper (https://arxiv.org/pdf/2302.11014 https://arxiv.org/pdf/2302.11014). I do not think it changes the ISPD work's conclusions however since they demonstrate that CMP and AutoDMP outperform CT wrt QoR and runtime even though they use much fewer compute resources. If more compute resources are used and CT becomes competitive wrt QoR, then it will still lag behind in runtime. Furthermore, Google has not produced evidence that AlphaChip, with their substantial compute resources, outperforms commercial placers (or even AutoDMP). In the recent rebuttal from Google (https://arxiv.org/pdf/2411.10053 https://arxiv.org/pdf/2411.10053), the only claim on page 8 says Google VLSI engineers preferred RL over humans and commercial placers on a blind study conducted in 2020. Commercial mixed placers, if configured correctly, have become very good over the past 4 years, so perhaps another blind study is warranted. > Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article I will openly admit my bias is against the AlphaChip work. I referred to the Nature authors as 'arrogant' and 'disdainful' with respect to their statement that EDA CAD engineers are just being bitter ML-haters when they criticize the AlphaChip work. I referred to Jeff Dean as 'belittling' and 'hostile' and using 'hyperbole' with respect to his statements against Igor Markov, which I think is unbecoming of him. I referred to Shankar as 'excellent' with respect to his shrewd business acumen.

wholehog 2y ago

We're talking 16 GPUs for ~6 hrs for inference, and 48 hrs for pre-training. This is not an exorbitant amount of compute. A GPU costs $1-2/hr on the cloud market. So, ~$100-200 for inference, and ~$800-1600 for pre-training, which amortizes across chips. Cloud prices are an upper bound -- most CS labs will have way more than this available on premises. In an industry context, these costs are completely dwarfed by the rest of the chip design process. (For context, the licensing costs alone for most commercial EDA software are in the millions of dollars.)

bushbaba 2y ago
h100 GPU instances are multiple orders of magnitude more expensive.
- radq 2y ago
  Not true, H100s cost $2-3/GPU/hr on the open market.
  - menaerus 2y ago
    Yes, they even do at $1/GPU/hr. However, 8xH100 cluster at full utilization is ~8kWh of electricity and costs almost ~0.5M$. 16xH100 cluster is probably 2x of that. How many years before you break-even at ~24$/GPU/day income?
    - Jabbles 2y ago
      7 https://www.google.com/search?q=0.5e6%2F8%2F24%2F365 https://www.google.com/search?q=0.5e6%2F8%2F24%2F365
      - menaerus 2y ago
        Did you really not understand rethoric nature of my question and assumed that I can't do 1st grade primary school math?
    - solidasparagus 2y ago
      Who cares? That's someone else's problem. I just pay 2-3$/hr and the H100s are usable
      - menaerus 2y ago
        Look, I understand that some people are short-sighted and can hardly think out of the box and that is totally fine by me. I don't judge you for being that so I kindly ask you not to judge my question. Learn to give some benefit of the doubt.
      - sangnoir 2y ago
        You should care about counterparty risks. If your business model depends on unsustainable 3rd party prices powered by VC largesse and unrealizable dreams of dominance, the very least you can do is plan for the impending reckoning, after which GPU proces will be determined by costs.
- YetAnotherNick 2y ago
  H100 GPUs are more or less similar in price/performance. It is 2-3x more expensive per hour for 2-3x higher performance.
vighneshiyer 2y ago
You are correct. For commercial use, the GPUs used for training and fine-tuning aren't a problem financially. However, if we wanted to rigorously benchmark AlphaChip against simulated annealing or other floorplanning algorithms, we have to afford the same compute and runtime budget to each algorithm. With 16 GPUs running for 6 hours, you could explore a huge placement space using any algorithm, and it isn't clear if RL will outperform the other ones. Furthermore, the runtime of AlphaChip as shown in the Nature paper and ISPD was still significantly greater than Cadence's concurrent macro placer (even after pre-training, RL requires several hours of fine-tuning on the target problem instance). Arguably, the runtime could go down with more GPUs, but at this point, it is unclear how much value is coming from the policy network / problem embedding vs the ability to explore many potential placements.
- Jabbles 2y ago
  You're saying that if the other methods were given the equivalent amount of compute they might be able to perform as well as AlphaChip? Or at least that the comparison would be fairer? Are the other methods scalable in that way?
  - pclmulqdq 2y ago
    Yes, they are. The other approaches usually look like simulated annealing, which has several hyperparameters that control how much computing is used and improve results with more compute usage.
    - wholehog 2y ago
      See my comment above - the Nature authors already did this, and tried a huge hyperparameter sweep for SA, and RL still won. See appendix of the Nature article: rdcu.be/cmedX
      - pclmulqdq 2y ago
        I understand and have read the article. Running 80 experiments with a crude form of simulated annealing is at most 0.0000000001% of the effort that has been spent on making that kind of hill climb work well by traditional EDA vendors. That is also an in-sample comparison, where I would believe the Google thing pre-trained on Google chips would do well, while it might have a harder time with a chip designed by a third party (further from its pre-training). The modern versions of that hill climb also use some RL (placing and routing chips is sort of like a game), but not in the way Jeff Dean wants it to be done.

nemonemo 2y ago

In the conclusion of the article, you said: "While I concede that there are things the ISPD authors could have done better, their conclusion is still sound. The Nature authors do not address the fact that CMP and AutoDMP outperform CT with far less runtime and compute requirements." One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this? Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article toward the AlphaChip work (words like "arrogance", "disdain", "hyperbole", "belittling", "hostile" for AlphaChip authors, as opposed to "excellent" for a Synopsys VP.) Could you share where this difference in tone originates?

vighneshiyer 2y ago
> One key argument in the rebuttal against the ISPD article is that the resources used in their comparison were significantly smaller. To me, this point alone seems sufficient to question the validity of the ISPD work's conclusions. What are your thoughts on this? I believe this is a fair criticism, and it could be a reason why the ISPD Tensorboard shows divergence during training for some RTL designs. The ISPD authors provide their own justification for their substitution of training time for compute resources in page 11 of their paper (https://arxiv.org/pdf/2302.11014 https://arxiv.org/pdf/2302.11014). I do not think it changes the ISPD work's conclusions however since they demonstrate that CMP and AutoDMP outperform CT wrt QoR and runtime even though they use much fewer compute resources. If more compute resources are used and CT becomes competitive wrt QoR, then it will still lag behind in runtime. Furthermore, Google has not produced evidence that AlphaChip, with their substantial compute resources, outperforms commercial placers (or even AutoDMP). In the recent rebuttal from Google (https://arxiv.org/pdf/2411.10053 https://arxiv.org/pdf/2411.10053), the only claim on page 8 says Google VLSI engineers preferred RL over humans and commercial placers on a blind study conducted in 2020. Commercial mixed placers, if configured correctly, have become very good over the past 4 years, so perhaps another blind study is warranted. > Additionally, I noticed that the neutral tone of this comment is quite a departure from the strongly critical tone of your article I will openly admit my bias is against the AlphaChip work. I referred to the Nature authors as 'arrogant' and 'disdainful' with respect to their statement that EDA CAD engineers are just being bitter ML-haters when they criticize the AlphaChip work. I referred to Jeff Dean as 'belittling' and 'hostile' and using 'hyperbole' with respect to his statements against Igor Markov, which I think is unbecoming of him. I referred to Shankar as 'excellent' with respect to his shrewd business acumen.