3 ms·
AI chip contenders face daunting 'moats'
- helsinkiandrew 3y agohttps://archive.ph/x71d9 https://archive.ph/x71d9
- mikewarot 3y agoI live in a fantasy world where I've got this problem solved with one weird trick that couldn't possibly work according to all of the normal assumptions about computing and FPGAs. Imagine that you could make latency your friend, and work with it for a change. In fact, you would add a latch to every LUT just to make sure everything stays in sync. It would take almost forever to get your first result off the edge of the chip. However, then the pipeline is full and you get a new answer each and every clock cycle after that. From a system clocked at only 1 Ghz. Imagine getting 1 billion tokens per second out of GPT-4 From one server box, with about 100 trillion LUTs in it, and 100 Trillion Latches (GPT-4 is about 1.2 Trillion tokens) That's $30,000 per second of compute if this page[1] and my math are right. [1] https://help.openai.com/en/articles/7127956-how-much-does-gpt-4-cost https://help.openai.com/en/articles/7127956-how-much-does-gp...