4 ms·

Questions for those in the know about chip design. How are they measuring the quality of a chip design? Does the metric that Google is reporting make sense?

Questions for those in the know about chip design. How are they measuring the quality of a chip design? Does the metric that Google is reporting make sense? Or is it just something to make themselves look good? Without knowing much, my guess is that “quality” of a chip design is multifaceted and heavily dependent on the use case. That is the ideal chip for a data center would look very different from those for a mobile phone camera or automobile. So again what does “better” mean in the context of this particular problem / task.

q3k 2y ago
This is just floorplanning, which is a problem with fairly well defined quality metrics (max speed and chip area used).
- Drunk_Engineer 2y ago
  Oh man, if only it were that simple. A floorplanner has to guestimate what the P&R tools are going to do with the initial layout. That can be very hard to predict -- even if the floorplanner and P&R tool are from the same vendor.
Drunk_Engineer 2y ago
I have not read the latest paper, but their previous work was really unclear about metrics being used. Researchers trying to replicate results had a hard time getting reliable details/benchmarks out of Google. Also, my recollection is that Google did not even compute timing, just wirelength and congestion; i.e. extremely primitive metrics. Floorplanning/placement/synthesis is a billion dollar industry, so if their approach were really revolutionary they would be selling the technology, not wasting their time writing blog posts about it.
- IshKebab 2y ago
  > Floorplanning/placement/synthesis is a billion dollar industry Maybe all together, but I don't think automatic placement algorithms are a billion dollar industry. There's so much more to it than that.
  - Drunk_Engineer 2y ago
    Yes in combination. Customers generally buy these tools as a package deal. If the placer/floorplanner blows everything else out of the water, then a CAD vendor can upsell a lot of related tools.
- negativeonehalf 2y ago
  The original paper reports P&R metrics (WNS, TNS, area, power, wirelength, horizontal congestion, vertical congestion) - https://www.nature.com/articles/s41586-021-03544-w https://www.nature.com/articles/s41586-021-03544-w (no paywall): https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2021_2022/papers/Mirhoseini_NATURE_2021.pdf https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2021_2022/...
  - Drunk_Engineer 2y ago
    From what I saw in the rebuttal papers, the Google cost-function is wirelength based. You can still get good TNS from that if your timing is very simplistic -- or if you choose your benchmark carefully.
    - negativeonehalf 2y ago
      They optimize using a fast heuristic based on wirelength, congestion, and density, but they evaluate with full P&R. It is definitely interesting that they get good timing without explicitly including it in their reward function!
      - ithkuil 2y ago
        Yeah; it means the heuristic they use is a good one
      - gabegobblegoldi 2y ago
        Interesting == Suspicious? I think this is a big red flag to those in the know.
    - clickwiseorange 2y ago
      The odd thing is that they don't compute timing in RL, but claim that somehow TNS and WNS improved. Does anyone believe this? With five circuits and three wins, the results are a coin toss.
  - sijnapq 2y ago
    [dead]
- rossjudson 2y ago
  Like when Google wasted its time writing publicly about Spanner? https://research.google/pubs/spanner-googles-globally-distributed-database-2/ https://research.google/pubs/spanner-googles-globally-distri... or Bigtable? https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/ https://research.google/pubs/bigtable-a-distributed-storage-... or GFS? or MapReduce? or Borg? or...I think you get the idea.
  - bushbaba 2y ago
    Spanner research paper was in 2012. Bigtable was in 2006. GFS 2003. The last decade has been a 'lost decade' of google. Not much innovation to be honest.
    - bombita 2y ago
      Attention is all you need is 2017... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
      - Kubuxu 2y ago
        They thought it was dead end, that is why they released it :P
  - thenoblesunfish 2y ago
    I am not sure these publications were intended to generate sales of these technologies. My assumption is that they mostly help the company in terms of recruitment. This lets potential employees see cool stuff Google is doing, and see them as an industry leader.
    - vlovich123 2y ago
      Spanner is literally a Google cloud product you can buy ignoring that it underpins a good amount of Google tech internally. The same is true of other stuff. Dismissing it as a recruitment tool indicates you haven’t worked at Google or really know much about their product lines.

q3k 2y ago

This is just floorplanning, which is a problem with fairly well defined quality metrics (max speed and chip area used).

Drunk_Engineer 2y ago
Oh man, if only it were that simple. A floorplanner has to guestimate what the P&R tools are going to do with the initial layout. That can be very hard to predict -- even if the floorplanner and P&R tool are from the same vendor.

Drunk_Engineer 2y ago

I have not read the latest paper, but their previous work was really unclear about metrics being used. Researchers trying to replicate results had a hard time getting reliable details/benchmarks out of Google. Also, my recollection is that Google did not even compute timing, just wirelength and congestion; i.e. extremely primitive metrics. Floorplanning/placement/synthesis is a billion dollar industry, so if their approach were really revolutionary they would be selling the technology, not wasting their time writing blog posts about it.

IshKebab 2y ago
> Floorplanning/placement/synthesis is a billion dollar industry Maybe all together, but I don't think automatic placement algorithms are a billion dollar industry. There's so much more to it than that.
- Drunk_Engineer 2y ago
  Yes in combination. Customers generally buy these tools as a package deal. If the placer/floorplanner blows everything else out of the water, then a CAD vendor can upsell a lot of related tools.
negativeonehalf 2y ago
The original paper reports P&R metrics (WNS, TNS, area, power, wirelength, horizontal congestion, vertical congestion) - https://www.nature.com/articles/s41586-021-03544-w https://www.nature.com/articles/s41586-021-03544-w (no paywall): https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2021_2022/papers/Mirhoseini_NATURE_2021.pdf https://www.cl.cam.ac.uk/~ey204/teaching/ACS/R244_2021_2022/...
- Drunk_Engineer 2y ago
  From what I saw in the rebuttal papers, the Google cost-function is wirelength based. You can still get good TNS from that if your timing is very simplistic -- or if you choose your benchmark carefully.
  - negativeonehalf 2y ago
    They optimize using a fast heuristic based on wirelength, congestion, and density, but they evaluate with full P&R. It is definitely interesting that they get good timing without explicitly including it in their reward function!
    - ithkuil 2y ago
      Yeah; it means the heuristic they use is a good one
    - gabegobblegoldi 2y ago
      Interesting == Suspicious? I think this is a big red flag to those in the know.
  - clickwiseorange 2y ago
    The odd thing is that they don't compute timing in RL, but claim that somehow TNS and WNS improved. Does anyone believe this? With five circuits and three wins, the results are a coin toss.
- sijnapq 2y ago
  [dead]
rossjudson 2y ago
Like when Google wasted its time writing publicly about Spanner? https://research.google/pubs/spanner-googles-globally-distributed-database-2/ https://research.google/pubs/spanner-googles-globally-distri... or Bigtable? https://research.google/pubs/bigtable-a-distributed-storage-system-for-structured-data/ https://research.google/pubs/bigtable-a-distributed-storage-... or GFS? or MapReduce? or Borg? or...I think you get the idea.
- bushbaba 2y ago
  Spanner research paper was in 2012. Bigtable was in 2006. GFS 2003. The last decade has been a 'lost decade' of google. Not much innovation to be honest.
  - bombita 2y ago
    Attention is all you need is 2017... https://arxiv.org/abs/1706.03762 https://arxiv.org/abs/1706.03762
    - Kubuxu 2y ago
      They thought it was dead end, that is why they released it :P
- thenoblesunfish 2y ago
  I am not sure these publications were intended to generate sales of these technologies. My assumption is that they mostly help the company in terms of recruitment. This lets potential employees see cool stuff Google is doing, and see them as an industry leader.
  - vlovich123 2y ago
    Spanner is literally a Google cloud product you can buy ignoring that it underpins a good amount of Google tech internally. The same is true of other stuff. Dismissing it as a recruitment tool indicates you haven’t worked at Google or really know much about their product lines.