10 ms·
France's Jean Zay supercomputer now has an integrated photonic coprocessor
- 1MachineElf 5y ago>According to LightOn, its Appliance can reach a peak performance of 1.5 PetaOPS at 30W TDP and can deliver performance that 8 to 40 times higher than GPU-only acceleration. Impressive! LightOn hasn't received much discussion on here before. Some links have been submitted, and this is the only one I could find comments on: https://news.ycombinator.com/item?id=27797829 https://news.ycombinator.com/item?id=27797829
- fsh 5y agoFrom the website of the manufacturer [1] it appears that the co-processor is essentially an analog computer for matrix-vector multiplications. I am quite sceptical about the accuracy and value range of the computations. Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise. [1] https://lighton.ai/lighton-appliance/ https://lighton.ai/lighton-appliance/
- orlp 5y ago> Even puny single-precision floating point operations are accurate to something like 7 decimal digits and have a dynamic range of hundreds of dB. According to the spec sheet, the appliance only uses 6-bit inputs and 8-bit outputs, so the relative errors are probably on the percent level. This makes it hard to believe that any signal will propagate through something like a DNN without completely drowning in noise. Maybe you aren't aware, but half-precision (16 bit float) is already well-established in the AI community: https://en.wikipedia.org/wiki/Bfloat16_floating-point_format https://en.wikipedia.org/wiki/Bfloat16_floating-point_format. In context single-precision isn't all that puny! And there have already been successful experiments with stronger quantization, like 8-bit neural nets, or even 1-bit (!) neural nets. There is a lot of evidence that neural networks can be very resilient to quantization noise.
- ISL 5y agoI'd be real surprised if the neurons in our brains have ADC-equivalents better than ~4 bits.
- tgv 5y agoTrue, but ANNs are nowhere near as good as our brains, nor do they operate in the same way.
- xwolfi 5y agoI think you missed the fact we were talking about neural network, not an animal brain self replicating and branching and competing with itself for billions of years until it becomes aware of itself. Give us the the same time.
- Retric 5y agoThe earliest and simplest brains where still useful. Even insects can fly around in 3D space, I doubt you need something as complicated as a mouse brain to run a self driving car let alone a drone.
- ben_w 5y agoI’m not sure how much it matters given this thread looks like it’s going off on several successive tangents, but the important (and hard) thing with a self-driving car is making sure it doesn’t hit stuff, not the actual driving part. And drones, trivially agree: Megaphragma mymaripenne has 7400 neurones, compared to the 71/14 million in a house mouse nervous system/brain.
- robwwilliams 5y agoAnd to stay on this odd tangent: Best estimate I have for total cell numbers in mouse brain—about 75 million neurons and 35 million other cell types. This estimate is from a 476 mg brain of a C57BL/6J case—the standard mouse used by many researchers. Based on much other work with discrete neuron populations the range among different genometypes of mice probable +/- 40%. for details see: www.nervenet.org/papers/brainrev99.html Expect many more (and I hope better) estimates soon from Clarity/SHIELD whole brain lighsheet counting with Al Johnson and colleagues at Duke and team at Life Canvas Tech.
- kortex 5y agoIt's not a transmission line though, SNR does not apply in the same way. It's more like CMOS where the signal is refreshed at each gate. Each stage of an ANN applies some weight and activation. You can think of each input vector as a vector with a true value plus some noise. As long as that feature stays within some bounds, it is going to represent the same "thought vector". It may require some architecture changes to make training feasible, but it's far from a nonstarter. And that is only considering backprop learning. The brain does not use backprop, and has way higher noise levels.
- dahart 5y agoI think the parent was referring to the same noise that you are, compute precision, not transmission, and was suggesting that perhaps it won’t easily stay within bounds due to the fact that some kinds of repeated calculations lose more precision at every step. Maybe it’s application dependent, maybe NNs or other matrix-heavy domains can tolerate low precision much more easily than scientific simulations. It certainly wouldn’t surprise me if these “LightOPS” processors work well in a narrow range of applications, and won’t improve or speed up just anything that needs a matrix multiply.
- tdrdt 5y agoBut arent there other applications where this is ok? For example path tracing (ray tracing) doesn't need to be very accurate because multiple samples per pixel are used. A gaming card that uses less power is very welcome in laptops for example.
- moonchrome 5y agoWhat do you mean? What kind of worlds can you represent with 8 bits units ? Some small blocky voxel box ?
- tdrdt 5y agoI mean inacurate vector math
- moonchrome 5y agoBut that is what I'm saying if your vectors are reduced to 8 bit scalar components you can represent a 256x256x256 worth of detail in the world (doesn't need to be linear but still really limited details) ?
- montjoy 5y agoI’m totally out of my expertise here but I have a question - from my understanding ray tracing is primarily used for lighting/shadows/reflections- wouldn’t it be OK for something like shadows to be inaccurate- maybe some sort of amalgamation over frame refreshes? Real world light is messy anyway. I’m talking about a game type scenario not something scientific. Maybe another way to ask is- we’re trying to simulate a real world “analog” scene - maybe using an analog processing technique could actually be quite faithful for generating it? Or not. Like I said I don’t understand too much of this.
- moonchrome 5y agoThink about it like this : you have a spaceship model fits in 256x256x256m - to get the maximum resolution while still fitting in 8 bits you would make each axis in 1m increments and you have 256 values. So you can't have sub 1m details in geometry. Floating point is different because you have an exponent so it's not linear, you can technically have larger scale, but you sacrifice even more precision in mantissa. Now I'm not sure what this 6/8 bit precision or analog precision means so I can't say with confidence, but if your scalars are that low precision you can't really do much. You could technically encode it with some fancy tricks like instead of storing coordinates for each vertex you store the delta from previous one etc. but I think this wouldn't work if the device was just some dumb analog matrix multiplier with baked logic. Also having low detail shadows creates visual artifacts, see this for example [1] [1] https://digitalrune.github.io/DigitalRune-Documentation/media/Shadow-Resolution.jpg https://digitalrune.github.io/DigitalRune-Documentation/medi...
- bjornsing 5y agoIf there’s too much noise just lower the dropout probability from 50 to 30%. ;) Joking aside, it is interesting how much noise and quantization these neural networks can work with. I think there’s a lot of room for low precision noisy computation here.
- robert_tweed 5y agoFor a few seconds I thought it was Lite-On, best known for their cheapo CD/DVD drives. Seems to be completely unconnected though.
- Datagenerator 5y agoBrings back memories of the Plextor automatic duplicator robot we had in company back in the nineties. Great times
- zapdrive 5y agoNice. Can't wait for the new light based GPUs, all being grabbed by greedy crypto miners and me still using my 6 year old graphics card!
- JohnJamesRambo 5y agoRelief is on the horizon. Ethereum should switch to proof of stake in June 2022 and you are about to see an unholy torrent of used GPUs hit the market. I would expect you can pick up any you like for peanuts then.
- demux 5y agoEthereum PoW won't immediately disappear, and I'm sure bitcoin folk will be all too happy to grab those extra GPUs
- vidarh 5y agoGPU mining Bitcoin hasn't been reasonable for many years unless you have very cheap energy and no access to ASIC miners.
- zapdrive 5y agoI have been hearing "Ethereum is switching to proof of stake in a few months" for at least 6 years now. I don't think it's going to ever happen.
- ohgodplsno 5y agoNo, miners will just move to another valuable PoW shitcoin and/or fork ETH to stay on PoW.
- inasio 5y agoI've never heard of LightOn, and wish the website had a bit more concrete info on the specifics of the coprocessor, but I am somewhat familiar with a similar photonic coprocessor made by NTT (the Coherent Ising Machine). It's still in the research stage, the logic uses interferometry effects, and requires kilometers of fiber optic cables. Interestingly, there is a simulator based on mean field theory that runs on GPUs and FPGAs(*) that can solve some problems (e.g. SAT) with close to state of the art performance. (*) disclosure: my company helped build the simulator
- dasudasu 5y agoThere are other startups in the space that do it in semiconductors. Look up Lightelligence and Lightmatter for instance.
- 14 5y agoI remember hearing about Lifi [1] years ago and thinking we would see it everywhere but this has not been the case. I wonder what has held it back. Like this super computer Lifi promised unmatched speeds that wifi can not do. Very neat to see the field is still advancing. [1] https://lifi.co/lifi-vs-wifi/ https://lifi.co/lifi-vs-wifi/
- tomlue 5y agothere was a ted talk [1] that made me think lifi would be take off much faster than it has. [1]https://www.ted.com/talks/harald_haas_wireless_data_from_every_light_bulb https://www.ted.com/talks/harald_haas_wireless_data_from_eve...
- crate_barre 5y agoHow could LiFi be truly wireless if anything can obstruct the path of light? How’s my phone in my pocket going to get connection?
- dmm 5y agoWhy restrict yourself to visible light? There are frequencies of em waves that would easily penetrate your jeans: https://en.wikipedia.org/wiki/Gamma_ray https://en.wikipedia.org/wiki/Gamma_ray
- Denvercoder9 5y agoUsually we go to the lower-energy, longer-wavelength part of the spectrum for applications like this, since gamma rays and the high-energy, short-wavelength part of the spectrum is very hard to control and gives you cancer.
- causality0 5y agoKeeping a multi-watt transmitter of ionizing radiation in your pocket is a very bad idea.
- 5y ago
- neatze 5y agoSeems like you can try it out[1], I find this a bit funny that it is easier to get trail access to quantum CPU and light based GPU, but for Cerebras and Graphcore trial access you need spend thousands of dollars. [1]https://lighton.ai/cloud/ https://lighton.ai/cloud/
- crate_barre 5y agoWould this generate less heat? Heat is still a bottleneck in some ways right? What about size? Can we get smaller? Would we need semiconductors anymore?
- adgjlsfhk1 5y agoheat is just power. 10x less power is 10x less heat.
- goldenkey 5y agoIndeed. All electrical devices are just space heaters that do something logical with the electricity while it's heating up the wires :-)
- amelius 5y agohttps://en.wikipedia.org/wiki/Adiabatic_circuit https://en.wikipedia.org/wiki/Adiabatic_circuit
- goldenkey 5y agoFrom what I understand, most of the energy loss of our current computers is not due to using irreversible logic, but rather the actual resistance of the materials. Once we have better materials or different paradigms like photonic computing, then it would make sense to implement the https://en.wikipedia.org/wiki/Toffoli_gate https://en.wikipedia.org/wiki/Toffoli_gate https://en.wikipedia.org/wiki/Fredkin_gate https://en.wikipedia.org/wiki/Fredkin_gate and whatnot. Btw, Fredkin and Toffoli, and the rest of the kings of reversible computing were big advocates of cellular automata ;-)
- foobarian 5y ago10x less power is 10x more gear I can run off the same outlet :-)
- vmception 5y agoThere are some people that view these as a solution to Proof of Work energy use I’ve been researching OPUs for this purpose, as in, the concept of optical processors only came to my attention because of the people looking for an edge in mining and energy use. From what I can tell, it could only be a stopgap or decade long solution to PoW, as people would just hoard these over time till the energy use was the same But any slowdown is good Good for sellers of OPUs though
- adgjlsfhk1 5y agoLight based computers work best on linear operations. They are extremely unlikely to work well for computing hashes.
- vmception 5y agoRight well, thats why there is a new hashing algorithm. And they want other energy heavy cryptocurrencies to switch to it. Basically there is a whole row going on in the far far corners of crypto land, where some Cambridge and Columbia university alumni have made a new hashing algorithm (HeavyHash) that is heavy on linear operations specifically so that it would theoretically have an advantage on light based computers and be a sustainability solution [for now]. They stood up an example project back in May and forgot about it ("optical bitcoin", oBTC), but people kept mining it. Just on GPUs and FPGAs, as the low power better processors don't exist yet so there isn't really anything different at the moment. Because these are students with no funding, the management is very weak. There is at least one fork of that project that has better management ("photonic bitcoin", pBTC). But they are waiting for OPUs to exist at all, as there are a variety of vaporware companies out there with massive funding. HeavyHash itself is being used in more and more other newly launched cryptocurrencies. Hoping for OPUs to become available to actually make their networks different than others. They all so far only aspire to be examples for other major energy consuming cryptocurrencies we've heard of to switch to. The university students had submitted a proposal (BIP) to Bitcoin-core, and that slow moving network typically needs real world examples and fear of missing out for consensus to shift.
- simias 5y ago
- mikewarot 5y agoThe only actual hardware description I could find was in this arXiv link, where you have a laser that is spread then put through a light gate chip (as found in projectors), a random mask, then to a CCD camera. This does random multiplies, in parallel, which somehow prove useful. I fail to see how it actually does matrix math. https://arxiv.org/abs/1609.05204 https://arxiv.org/abs/1609.05204
- deleted 5y ago[deleted]
- daralthus 5y agoIt does random projections [1] which is really useful for dimensionality reduction. [1] https://en.wikipedia.org/wiki/Random_projection https://en.wikipedia.org/wiki/Random_projection
- faeyanpiraat 5y agoThere is an upcoming Veritasium video about the "comeback of analogue computers". Part one (teaser): https://www.youtube.com/watch?v=IgF3OX8nT0w https://www.youtube.com/watch?v=IgF3OX8nT0w
- sshlocalhost98 5y agoYup, I saw it, what is he actually trying to say ? How can analog computer supercede digital, analogue is made too specific for only 1 task
- JumpCrisscross 5y agoWe have tonnes of computing power dedicated to repeatedly solving a simply-parameterised problem.
- rawoke083600 5y agoStupid question... with todays memory capacities....At what limit/size do we stop using matrix-ops and simply use lookup tables todo 'matrix-math' ?
- jeffbee 5y agoNot a stupid question. Economically, memory density will hit a brick wall soon. Developers should prefer to waste time and save space, since parallel computation will not hit a similar limit in the foreseeable future. Memory-to-core ratio is going to be falling.
- adgjlsfhk1 5y agoTLDR is you can't. For a very simple example, storing the products of all 3x3 matrices * length 3 vectors in Float16 precision would take 2^193 bytes (which is obviously impractical).
- kortex 5y agoIf that "1 task" happens to be performing matrix multiplication (or even merely fused multiply add), you can do a heck of a lot with that. You still need digital circuitry to support the IO, but the key idea is doing linear algebra in a way that is faster and/or generates less heat per unit compute.
- mrobot 5y agoSome info on China's Jiuzhang 2 https://interestingengineering.com/chinas-new-quantum-computer-has-1-million-times-the-power-of-googles https://interestingengineering.com/chinas-new-quantum-comput...
- magicalhippo 5y agoOn a related note, the Huygens Optics channel on YouTube published a video[1] earlier this year on the attempts to make optical logic gates. He didn't get as far as making working devices, work in progress, but interesting still IMHO. He also has a lot of other great videos about optics. [1]: https://www.youtube.com/watch?v=pS1zAAD1nXI https://www.youtube.com/watch?v=pS1zAAD1nXI
- rahimiali 5y agoThe lighton device accelerates random linear projections. Its input is a vector x and its output is a vector W*x, where W is a fixed random matrix. This is useful for a class of machine learning algorithms, but not frequently used in deep learning.
- sudosysgen 5y agoCan you not decompose general matrix multiplication into the sum of vector matrix multiplications?
- deleted 5y ago[deleted]
- ww520 5y agoI'm not super well versed on this topic. But the basic physics of photon and electron make light a poor source for computing compared to electricity. Photons don't interact with each other much when crossed, while electrons interfere each other greatly. It's really difficult to build logic gates out of photons while electrons work great with semiconducting materials to build gates. Light is good for communication due to its non-interfering nature to pump up the bandwidth. There's a saying light is great for data transmission while electricity is great for computing. So when people claim making supercomputer out of optical computing, take it with a grain of salt.
- aidenn0 5y agoPhotons may not interact with themselves much, but they interact with other materials in significant and useful ways, and there are plenty of materials that change their optical properties in response to an electric field (which can be either electronic or optical in origin, given that light, particularly coherent light can have fairly strong electric fields). There are millions of things that an electronic digital computer can do that are unlikely to be replaced by photonics, but a hybrid approach may offer advantages in computing specific things. As we have slowed down getting perf/watt advantages from shrinking processes, more and more specialized hardware has been used for performing calculations. It's not that far-fetched to think that photonics might have a niche where it has performance advantages.
- ww520 5y agoYou're right that photons interact with many materials. However, few exhibit semiconducting properties that's useful for building logic gates. Silicon has a kind of perfect energy band gap for its valence band electrons jumping to conducting band free elections to allow electricity flow. The gap is not too small to introduce ambiguity or too big to require too much energy to move from non-conductive to conductive state. Photonics needs to find a semiconducting material/alloy that beats silicon to be a viable option. Most research in building logic gate with light is a combination of light and electricity. The most promising recently is using Josephson junction in a superconducting electric current loop. A photon hitting the loop adds energy to the superconducting current. With enough energy the current becomes critical current, which moves the Josephson junction from 0 voltage to a voltage. The raised voltage gives off the energy and the current falls back to non-critical. Continuous photons hitting the loop causes the Josephson junction to have an extremely high frequency AC voltage. That's a photon controlled gate. But the research is still really early and it requires low temperature superconductivity. It's still a long way to be competitive in reality.