3 ms·

Well the best option is what Google's doing with its TPUs, a different architecture with far better data locality since that's the main bottleneck. https://en.

• brucethemoose2 3y ago
  I was really impressed with Microsoft's paper on llm architectures, where instead of using giant chips with giant memory busses, they pipe together a bunch of cheap, monolithic, SRAM heavy SoCs. This would work really well for something like Llama, where the layers can be easily split up into SRAM sized chunks, and the whole thing can be pipelined as long as many users are making requests.
  • marshray 3y ago
    > I was really impressed with Microsoft's paper on llm architectures Link? Please :-)
    • brucethemoose2 3y ago
      https://www.nextplatform.com/2023/07/12/microsofts-chiplet-cloud-to-bring-the-cost-of-llms-way-down/ https://www.nextplatform.com/2023/07/12/microsofts-chiplet-c... https://arxiv.org/pdf/2307.02666.pdf https://arxiv.org/pdf/2307.02666.pdf