3 ms·
Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily o
by KenoFischer 7y ago
Zygote is an orthogonal piece of technology on this front and relies on a good optimizing compiler behind it to target actual hardware. Its focus is primarily on expressability. We've been talking about automatic kernel generation for a while (and when I saw kernel generation what I mean is basically search for access patterns), but note that it's not quite as bad a problem in julia, because you can use higher order functions to get a lot of composability (e.g. if there's a hand-optimized parameterized matmul, fusing in inputs and outputs is just passing in extra functions). There's some academic work at the JuliaLab that's promising. We're also in discussions with commercial partners who care about this kind of thing to see if somebody is willing to fund it, but so far existing solutions have been good enough while we work on higher priority items. I do agree that being limited to a few hand-tuned kernels is a significant drag on research, so I hope we can do something here.
- _ks3e 7y agoIn terms of trying to break free of dependence on hand optimized kernels: a few people, myself included, have been working on some theoretical approaches to generating cache-efficient rearrangements for neutral net like problems. We've worked it out for convolution like problems [1] and have some upcoming results generalizing these techniques to other problems. Please feel free to email if you'd like to talk. [1] https://arxiv.org/abs/1802.06905 https://arxiv.org/abs/1802.06905
- steve76 7y agoGod damn you RMS. Just do this please: https://twitter.com/thisbounty_com/status/1152390739013394432 https://twitter.com/thisbounty_com/status/115239073901339443... https://twitter.com/thisbounty_com/status/1152397565901623297 https://twitter.com/thisbounty_com/status/115239756590162329...
- cs702 7y agoThank you for your response. Makes sense, and I'm happy to hear you and others are aware of the issue. PS. I now feel that I asked my question without first thinking about it a bit more; sorry about that. I temporarily "forgot" that Zygote is a source-to-source AD package because, as someone who is developing and iterating over deep learning models for potential deployment to production, I naturally tend to think in terms of monolithic software stacks -- e.g.,"the TensorFlow stack," "the PyTorch stack," "the nascent Julia stack," and so on.