I know much less about the large transformer models than I do about the optical hardware but the hardware ideas have been floating around since the 1980s. The d

I know much less about the large transformer models than I do about the optical hardware but the hardware ideas have been floating around since the 1980s. The development of lasers-on-a-chip made it all a lot more realizable. E.g. "On-chip optical matrix-vector multiplier for parallel computation" (2013) https://spie.org/news/4932-on-chip-optical-matrix-vector-multiplier-for-parallel-computation https://spie.org/news/4932-on-chip-optical-matrix-vector-mul... Figure 1 has a nice visual of the vector-matrix computation in terms of the diode laser array, the multiplexer, the "microring modulator matrix" and the resulting output, the new (output) vector detection system. > "We have designed and fabricated a prototype of a system capable of performing a multiplication of a M × N matrix A by a N × 1 vector B to give a M × 1 vector C. The mathematical procedure of MVM can be split into multiplications and additions, which is reflected in our design. Figure 1 shows a schematic of the architecture we propose. The elements of B are represented by the power of N modulated optical signals with N different wavelengths (λ1, λ2, …, λN), generated by N modulated laser diodes, either alone or together with N Mach-Zehnder modulators. These signals are multiplexed, passed through a common waveguide, and then projected onto M rows of the modulator matrix by a 1 × M optical splitter. Each element aij of matrix A is represented physically by the transmissivity of the microring modulator located in the ith row and the jth column of the modulator matrix. Each modulator in any one row only manipulates an optical signal with a specific wavelength." That was 10 years ago, no idea what current state-of-the-art is.