4 ms·
Llama2.c running on a Silicon Graphics Indigo2 workstation
- xenonite 3y agoAmazing! Training the model, although it is small with only 15M parameters, takes quite some compute power. I wonder if the resources would have been sufficient at the time. Also, I wonder if disk space would have been sufficient to store the input corpus.
- andy99 3y agoIt looks like these machines run IRIX which is a unix variant on a mips processor and has some parallelization features. I wonder if the compiler he's using is taking advantage of these, or if there's potentially some speedup possible. From what I remember, llama2.c uses OMP to optionally parallelize some of the matrix-vector products which wouldn't be available on that machine. https://en.m.wikipedia.org/wiki/IRIX https://en.m.wikipedia.org/wiki/IRIX
- mechagodzilla 3y agoIt's interesting to think about some of the earliest machines we could have run a pretty great LLM on. A Cray XMP-EA from 1986 could address up to 2GW, which would have been 16GB. The XMP or earlier Cray-1 lines could also use multiple solid-state SRAM disks that were up to 1GB in size and could stream at ~1GB/sec.
- emchammer 3y agoIt's interesting to think about how computers might be different had SRAM generally developed and become prolific instead of DRAM.
- mechagodzilla 3y agoWhat do you mean? It did continue to develop (right up to the latest 3nm process!), it’s just inherently lower density due to its structure.
- emchammer 3y agoIt is not typically used outside of L1 cache. I imagine a computer using bistable memory cells as being more resilient, and a more elegant design than memory that needs circuitry to constantly refresh it.
- 1oooqooq 3y agoanswering the question in the xweet: researcher 28 years ago would have decent reproducible peer reviewed metrics so they would not even look at most of this.
- segmondy 3y agoGoes to show we have always had the "tech" just not the idea.
- nxobject 3y agoI wonder how much RAM is needed to run this – for example, would this work on the Indigo2's minimal config of 16 MB? (If so, I'll be running towards THINK Pascal to get this running on an SE/30...)
- jakupovic 3y agoThe prettiest machines ever. Also, had 3D glasses. We had one, they were old 20 years ago, but so cool.