5 ms·
Link to previous paper: https://arxiv.org/abs/2308.14711 https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differe
by WithinReason 3y ago
Link to previous paper:
https://arxiv.org/abs/2308.14711 https://arxiv.org/abs/2308.14711
An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the branches so they become hard branches at the end of training.
A branch is computed as branch(input, N), with a neural network N computing a scalar c=N(input), then using a sigmoid to do a soft branch by returning the weighted sum of the recursive call s(c)*branch(input, N_left) + (1-s(c)) * branch(input, N_right) (the two weights s(c) and 1-s(c) sum to 1). They only do "proper processing" using the leaf nodes.
Then they add a new loss term that encourages hard decisions by minimising the entropy of the Bernoulli distribution, making the 2 weights converge to 0 and 1, at which point only one branch needs to be taken at inference. They also state that this hardening often happens automatically though.
It's a simple idea but the loss formulation is nice, you usually want your loss terms to be a measure of information.
- WithinReason 3y agoAlso, this didn't come from OpenAI or DeepMind, or even industry. What are those guys even doing? :)
- mmaunder 3y agoMany labs doing foundational work like this and making progress don’t have the anything near the budget or compute to implement at scale. In other words they don’t have a Sam and his backers or a Zuck and his budget.
- Micoloth 3y agoThey sure as hell have no incentives to make Neural Network faster and more accessible, for starters.. (Considering they right now make more money and have more control, the less accessible and the more computation-hungry AI models are) To be fair, this approach (claims to) only speed up inference, not training, so all the GPUs are needed anyway.
- WithinReason 3y agoThey certainly have an incentive to keep these kinds of improvements in-house and not publish them, since they are commercial entities and this represents a competitive advantage.
- lawlessone 3y agoI think Nvidia might have an incentive for this not to exist. edit: but you are right for the AI companies not open sourcing their models it's an advantage to have it when others don't
- WithinReason 3y agoI'm actually not sure about Nvidia, due to https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
- ForkMeOnTinder 3y agoBut if things get too efficient for individual users, you won't need an Nvidia GPU anymore. People will use cheaper hardware instead. I'm looking forward to running good models at decent speed on a low-end CPU or whatever crappy GPU is in my phone.
- jacobsimon 3y agoI had the same thought this morning and was debating selling my nvda stock when I saw this - feels like they are well-positioned right now, as with crypto a few years ago, but if there were an efficiency breakthrough that allowed commodity CPUs to do the inference instead, this advantage could vanish quickly.
- godelski 3y agoNvidia can't make GPUs fast enough. I doubt 10xing training and/or inference efficiency would result in a decrease in demand. I would be surprised if it didn't instead increase demand. Mind you, Nvidia is pushing hard on TensorRT which optimizes models at inference time and results in major increases in throughput (not 10x though lol).
- airgapstopgap 3y ago…ETH Zurich is an illustrious research university that often cooperates with Deepmind and other hyped groups, they're right there at the frontier too, and have been for a very long time. They don't have massive training runs on their own but pound for pound I'd say they have better papers.
- godelski 3y agoETH Zurich is one of the top labs in the world. Disney Research also works with them a lot. Another "sleeper" is University of Amsterdam that has rockstars like Max Welling and his students Kingma, Salimans,van den Berg, and Hoogeboom. It's easy to get hyped up on the big tech labs because they have the most compute, but the best papers come from smaller labs and unfortunately more lately face larger challenges in getting published. It's the smaller works that create the foundations that end up in these giant models. ML is in a really weird space right now.
- brandall10 3y agoIt's great to the school of Einstein/Von Neumann is doing highly innovative work in this space.
- pr337h4m 3y agoThis from DeepMind: DiLoCo: Distributed Low-Communication Training of Language Models - https://arxiv.org/pdf/2311.08105.pdf https://arxiv.org/pdf/2311.08105.pdf From the first author on Twitter: "It could quite a big deal for people who don't have access to a colocated cluster of GPUs: e.g. with DiLoCo you could train your model, with data-parallelism, across all GPU providers, looking in real-time for the cheapest price, even if pre-emptable, even across continents" https://twitter.com/Ar_Douillard/status/1724839420820361687 https://twitter.com/Ar_Douillard/status/1724839420820361687
- quickthrower2 3y agoIt is not surprising. The assumption is that they have the best people. That you can objectively search 8 billion people for the best people globally is folly of course. There are geniuses without US citizenship / visas / green cards. And so outside brains are going to figure this out. Mix in GDP of $rest_of_world has much more resources than any company, and the luck-driven nature of making AI discoveries, and I reckon most progress will be outside of OpenAI etc. Driven by a problem the big guys don't need to solve: how do I avoiding buying a $5k graphics card.
- lawlessone 3y agoFrom the previous paper you cited >Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance. This feels like that often misinterpreted Einstein meme/qoute about humans only using a fraction of their brain power. Is this only for inference though? could it boost training?
- WithinReason 3y agoThat's an interesting question. It actually provides a nice way to parallelized training: Pretrain e.g. the first 3 branch levels, which effectively fragments the model into 8 separate parts, which you can continue training across 8 independent servers/nodes with no further communication between the nodes. A central server would run the 1st 3 levels and mark parts of the training set that each node has to train on. Maybe you could do this for the whole network and distribute the training in SETI@HOME style all over the world. Hold on, you don't even need to freeze the branches completely: each node could train 1 branch on the path to its leaf node and communicate a change in the branch node to a central server, so you can distribute training without having to pre-freeze the branches. Still would need some pre-training though, and the splits would change slowly, and the attention mechanism could complicate things. Currently distributed neural network training SETI@HOME style looks like a complete pipe dream that nobody is taking seriously. But a smart branching mechanism like this could suddenly make it possible. Folding@home reached 1.5 exaflops, which made it the world's largest supercomputer. Imagine the models we could train that way, they would far surpass whatever OpenAI or Google could train and would be public.
- alchemist1e9 3y agoThis! If this becomes true then it’s a game changer. I hope you are correct.
- richardw 3y agoAlso steps up the economic benefit of, and therefore demand for, botnets. We really need a solution to bad actors controlling vast amounts of compute.
- deleted 3y ago[deleted]
- thomasahle 3y agoSounds like hiarchial softmax from the early NLP days
- knexer 3y agoIt's mentioned briefly in the paper(1), but I'm more interested in the interpretability implications of this approach. In some respects, this marries the interpretability/editability of a small decision tree with the expressive power of a large neural network. Usually you see those two on extreme opposite ends of a tradeoff spectrum - but this approach, if it scales, might shift the pareto frontier. (1): As a byproduct, the learned regions can also be used as a partition of the input space for interpretability, surgical model editing, catastrophic forgetting mitigation, reduction of replay data budget, etc..