7 ms·
The future of deep learning is photonic
- adamnemecek 5y agoI have said this many times and way considered a fool. Electricity is so bad compared with light.
- analog31 5y agoPart of my day job is optics design. There are some tradeoffs. One of them is that the structures of IC's are approaching a couple orders of magnitude smaller than the wavelengths of visible light. To illustrate this point, we have to use something other than visible light to make chips. Getting light through smaller structures becomes quite lossy. A way to make up for the size penalty might be to exploit increases in speed, but the structures for making high speed optics (such as femtosecond lasers) remain bulky. I'm certainly not trying to nay-say it, and continued research is worthwhile. The history has been that when a new thing is announced that will beat silicon, by the time that thing comes to fruition, silicon has caught up through just grinding incremental improvement. Optics can solve some interconnection issues, since it isn't confined to 2-dimensional (or "2-1/2" dimensional) structures.
- api 5y agoIf you could hit terahertz clock speeds something with only 100k transistors and hundreds of nanometers or larger feature sizes could kill modern CPUs… provided you could keep it fed with data from fast enough I/O of course.
- the__alchemist 5y agoWhy is visible-light wavelength a limitation? Energy requirements?
- analog31 5y agoJust technology. Visible light can use materials that are relatively straightforward: Glass, pure silica, and some other things can be made transparent to visible light. Silicon is friendly as a detector material. Getting materials, detectors, and light sources, to work at shorter and shorter wavelengths is a technological hurdle. Not insurmountable, but hard.
- aborsy 5y agoLoss is a main issue. It’s minimum at particular wavelengths
- mmmBacon 5y agoDude silicon absorbs below 900nm. We are talking about IR here.
- analog31 5y agoJust to clarify, so we're on the same page, silicon is the most common detector material for visible and UV. It absorbs a photon by kicking an electron into the conduction band, which is then swept to an electrical terminal and registered as a voltage or current. Silicon transmits IR, and can be used for optics and waveguides. So there's two sides to silicon, with the transition in that regime of about 900 nm. Actually, silicon photodiodes work down to about 1050 or even 1100 nm, but at a rapid loss of sensitivity and temperature stability. On the UV end, the silicon might be perfectly friendly, but the layers of stuff on top of the silicon become problematic. Silicon is also used for higher energy particles, just due to the havoc caused by absorbing an X or Gamma ray.
- mkoubaa 5y agoWhy should we care about size? As far as I understand it real estate isn't a limiting cost for data centers
- mmmBacon 5y agoUltimately because of cost and scalability. Photonic components are analog devices not digital so they do not have a “quantum” size that scales with process node. The size is the same in any node generally. It is possible to make the devices smaller by making them plasmonic but at the expense of loss. Since these are analog signal chains the signal to noise ratio at the end of the lossy device cascade becomes a significant limiting factor. For scalability, the larger the die size the more it costs. The yield also becomes a problem as the die area increases due to random defects. I’d don’t understand the visible light comment since silicon absorbs below 900nm (visible and NIR).
- analog31 5y agoOne issue is that the speed of light is 30 cm per nanosecond. This limits the size of processors that can run at, say, GHz speeds. It's also a limitation of wires, which is one reason why the original Cray supercomputer had such a funky geometry. When I was in grad school, the nuclear physics lab had racks of cables of different lengths that were conveniently labeled in nanoseconds.
- marcosdumay 5y agoAnalog computers seem to be very well fit for running neural networks (that's since the beginning, this is not the fist time normal computers are beaten), but they are a really bad choice for mostly everything else. In all likelihood, we will find many more usages for optical processors, but I really don't think they will ever be your main CPU.
- orbifold 5y agoAnalog computers can't easily be mutliplexed and the integration density of memory + other compute is nowhere near as high as current SRAM/DRAM. This might change with memristive crossbars, but that still doesn't solve the structural part of the problem, since most deep learning workloads are nowadays structurally very far from a feed-forward perceptron and dynamic execution etc. are absent from analog approaches.
- visarga 5y agoModern architectures are becoming simpler, usually just stacks of transformers, and they are good for text, vision, audio, or almost any task without special modifications. That means if they could implement an optical transformer that is 100x more efficient/faster everyone will want to have it.
- skybrian 5y agoOn the other hand, will there be dramatic improvements on the software side using improved algorithms? It seems like deep learning research is currently more focused on making things possible than making them efficient.
- varelse 5y agoYou might want to watch Nvidia's presentation on the recent MLPerf competition. There's a tremendous amount of engineering going into making this stuff more efficient on current hardware. https://developer.nvidia.com/blog/mlperf-v1-0-training-benchmarks-insights-into-a-record-setting-performance/ https://developer.nvidia.com/blog/mlperf-v1-0-training-bench...
- mirker 5y agoMost of the effort is in fine tuning the existing methods, which is important, but also leads to reinforcing the status quo in terms of algorithms. In other words, transistor scaling gets a sizable fraction of optimization, and the viable algorithms are ones most similar to dense matrix multiplication. If you unfix the algorithms (e.g., analog or very sparse models), you’ll fall off this optimization efforts and your model will eat dust.
- varelse 5y agoWhich is why you need a killer demo. New ideas are great, pure research rocks. But new ways to solve real world problems derived from that research is the killer product IMO. See an unnamed AI ASIC company proudly announcing they have achieved better perf/$ without achieving better perf for how not to do this.
- mirker 5y agoIt’s a Catch 22. New algorithms stink because conventional hardware stinks at running them and vice versa. Before AlexNet and GPU-optimized kernels, deep learning took too long to train large models, so large models were not used, and other ML methods seemed more promising. I agree a killer product is what you want, but I think the incentives are wrong for a startup to get both algorithms/hardware right before funding runs out. Just getting a working compiler is out of reach of many of these startups.
- varelse 5y agoIMO these sorts of methods become interesting the day they are demonstrated working in a conventional AI framework on a commercially important graph like Resnet or BERT. Everything else follows from a killer demo like that. The benefit is obvious if it works, but it also entails embracing non-deterministic results by design because it's analog. It's also a major shock to the tech stack. It took a half decade for GPUs to be noticed. This is a much more traumatic transition, no?
- version_five 5y agoYeah, I just came here to see if there was any discussion about this. I end up implementing stuff out of research papers pretty much exclusively. If I could train more efficiently, I would happily. But if I have to build something from scratch using an unconventional paradigm with no large body of research behind it, the value to me is negative, because before I even start I would have to convince my customers they can rely on it, that it matches state of the art, etc. Then I'd have to work with a much smaller set of tools built around it, including pretrained models with ultimately millions of dollars of R&D tied up in them. It's good to see new research, but this is not at all immediately helpful for applied work. Show me ResNet implemented this way, with a comparison of speedup and benchmarks on imagenet, and my opinion will change entirely.
- RicoElectrico 5y agoThis future always seems to be just around the corner for the past decade or so. Feels like vaporware to me.
- throwaways885 5y agoDeep learning is here, today. Not really vapourware.
- dharmaturtle 5y agoAlexNet came out 2012, less less than 10 years ago. Going from AlexNet to, say, self driving cars in under ten years sounds insane to me. This is only the beginning.
- TaylorAlexander 5y agoGood point I had not connected the two but that’s pretty significant!
- melling 5y agoGoogle has been working on the self-driving car for 12 years. https://en.m.wikipedia.org/wiki/Waymo https://en.m.wikipedia.org/wiki/Waymo They had cars on the road in 2009: “San Francisco was one of the first cities where we tested our self-driving cars, dating back to 2009 when we traveled everything from Lombard Street to the Golden Gate Bridge. Now that we have the world’s first fleet of fully self-driving cars running in Arizona, the hilly and foggy streets of San Francisco will give our cars even more practice in different terrains and environments.”
- rejectedandsad 5y agoWasn't aware that Luminous was an SNN application. That's...far less interesting, given we can't even get electronic SNN's to do anything useful.
- taneq 5y agoI think spiking neural networks are still pretty interesting, because they're the only type that we have hard evidence of being able to produce general intelligence. I agree, though, that if they're just doing 'conventional' neural network based machine learning, it's hyperbolic to call it 'the future of deep learning'. Anything these can do, by the sounds of it, GPUs can do a bit slower. Of course, 'quantity has a quality all of its own' and we wouldn't be doing things like GPT3 without GPUs... but then again the current approach of just throwing a bigger network at a problem seems to me to be a way to overfit your training data while pretending you're not.
- deleted 5y ago[deleted]
- deleted 5y ago[deleted]
- ArnoVW 5y agoBumped into this company some years ago that use light to perform 'random projection', which can be used to approximate matrix multiplication. Which is used a lot in the AI space (reduction of dimension etc) How? They do the projection quite literally: with DLP projectors that project on CCD through a 'dirty' medium, for randomness. At current DLP resolutions and refresh rates, every pixel being one bit, I'll let you imagine how much data you can push through, using only 30W. The whole thing is sold as an appliance, a sort of co-processor that can do just one operation (but a really complex one) They released their first commercial product a couple of months ago, hope it works out for them as the idea sounds pretty amazing. edit: found their website https://lighton.ai/ https://lighton.ai/
- mattbit 5y agoYeah, LightOn has been doing this for years already, it's kind of strange that there was no mention of them in the article. I know them because their offices are close to the research center where I work (in Paris, France). If I'm not wrong they were planning to offer their optical processor as a cloud service too.
- orange3xchicken 5y agofyi the founder Igor Carron runs a pretty nice academic blog on compressive sensing (a bit less academic in recent months) https://nuit-blanche.blogspot.com/ https://nuit-blanche.blogspot.com/ Also hosts the advanced matrix factorization jungle website which is a nice browse if you're interested in mf techniques. https://sites.google.com/site/igorcarron2/matrixfactorizations https://sites.google.com/site/igorcarron2/matrixfactorizatio...
- justicezyx 5y agoYou might be well informed in compressive sensing: What's the status of the compressive sensing research, and commercialization if there is any?
- andyxor 5y agohis presentation on LightOn is pretty interesting https://www.youtube.com/watch?v=0QtY4_UJF0w&ab_channel=LightOn https://www.youtube.com/watch?v=0QtY4_UJF0w&ab_channel=Light...
- shoto_io 5y agoAh… I love the new buzzword people will be throwing around soon enough. It sounds great I have to admit “Photonic AI”. Wow.
- Guest42 5y agoPretty soon Microsoft will introduce us to the P# language and have intro courses for those that don't want to deal with all that math stuff, with free credits for Azure.
- midjji 5y agoDeep learning has a lot of problems. Energy cost is not one of them.
- mkoubaa 5y agoThe neurons in our head are analog so I'm not surprised that folks are prodding there. I really don't like the emphasis here on circuit element sizes. If a computer the size of an elephant can train a large neural network more cheaply than 1000s of gpus why should we care?
- ryanthedev 5y agoI often wonder if you could use different wavelengths of light to do the same. Kind of how you could mix yellow and blue to make green.
- datameta 5y agoPhotonic in-memory ML https://www.rmit.edu.au/news/media-releases-and-expert-comments/2020/nov/light-powered-artificial-intelligence https://www.rmit.edu.au/news/media-releases-and-expert-comme...