4 ms·
These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performanc
by owlbite 3y ago
These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude.
These things are inferior at many levels:
- Algorithmic: These things aren't feeding back to their human masters tips and tricks on how to modify the network to go faster beyond some very basic signals.
- Loss of intent: ML network designers are specifying architecture in python, and by the time it's gone through many layers of lowering, you can get some complete garbage. Highly efficient garbage, but still garbage. (recent example, we caught one of these compilers doing a slice update operation by first forming the range of all possible indices to the array, slicing that to get indices to update, and then doing a scatter; we replaced it with a single memcpy call).
- Inefficient kernels. Every time we see the output of these compilers go head-to-head with an expert assembly programmer, the compiler loses, often by 30%+. This always seems like the sort of thing that should be easy to solve, but given no-one seems to have cracked it in the past 50 years, it's obviously not as simple as it sounds.
- hotstickyballs 3y agoHardware (and performance) can always be improved without involvement of users so this is actually pretty useful.
- stabbles 3y agoTake a look at the chess engine Stockfish: they tossed out years and years of human written heuristics in board evaluation, to a small neural net that does the same but better. Now consider all the heuristics for inlining, loop unrolling, vectorization etc in compilers, certainly a neural net can be beneficial and possibly easier to maintain than tons of human written heuristics.
- asah 3y agobig +1 - IMHO the future of optimizers (and probably compilers...) are almost certainly ML-based.
- ldjkfkdsjnv 3y agoHumans designing algorithms by hand will go the way of the dodo bird
- YawningAngel 3y agoMy understanding is that stockfish retains and uses its classical evaluation model in addition to the NNUE model
- shpx 3y agoNo, they removed the code in Stockfish 16, released June 30, 2023 but it wasn't used for much or made much difference before then, after they introduced the neural net. https://github.com/official-stockfish/Stockfish/commit/af110e02ec96cdb46cf84c68252a1da15a902395 https://github.com/official-stockfish/Stockfish/commit/af110...
- owlbite 3y agoWe'll have to see. I could definitely see someone spending a lot of time training for a specific algorithmic kernel and microarchitecture and beating the best human results (by a few percent). I'd be very surprised if that can be extended to a large complex algorithmic system that is amenable to mathematical reformulations (at least within the next 10 years).
- hkmaxpro 3y agoAnother bitter lesson? http://www.incompleteideas.net/IncIdeas/BitterLesson.html http://www.incompleteideas.net/IncIdeas/BitterLesson.html
- dbecker 3y ago> These ML-compilers are being overhyped. It's all the same trade-off as a traditional compiler Funny you should say that. Because traditional compilers have been incredibly useful.
- owlbite 3y agoRight, but we still tend to sidestep the compiler and/or spend hours of human time tuning the input to get the right output for core kernels.
- JyB 3y agoComment seem extremely dismissive and close minded.
- jhardy54 3y agoExactly! Why would anyone use gcc/clang when you can just hire someone to hand-write assembly instead?
- summerlight 3y ago> It's all the same trade-off as a traditional compiler: you get a lot more throughput than hiring a specialist performance programmer, but the latter will typically outperform, possibly by orders of magnitude. That throughput is the point though? You cannot have performance specialists on every single ML workload. It's still significantly better than not having these kinds of optimization.