4 ms·
DualPipe: Bidirectional pipeline parallelism algorithm
- snake_doc 2y agoHmm weren’t there also supposed to be the SM re-allocation, doesn’t look like it was included; I may have been mis-remembering the explanation.
- danielhanchen 2y agoI attached all 3 algorithms 1F1B (1 forward 1 backward), ZB1P (zero bubble pipeline parallelism) and DualPipe as a picture here: https://x.com/danielhanchen/status/1894937006352031832 https://x.com/danielhanchen/status/1894937006352031832 for those interested :)
- Bimos 2y agoMaybe add Chimera as well? https://arxiv.org/pdf/2107.06925 https://arxiv.org/pdf/2107.06925
- danielhanchen 2y agoOh more nice pictures :)
- isoprophlex 2y agoit looks as if Chimera has marginally less bubbles than DualPipe?
- alphan0n 2y agoOff topic, but this is the Rick and Morty episode where Rick creates a perfectly level space. The symmetry is uuugh.
- danielhanchen 2y agoYou'll have to refresh my memory :) Is there like a Youtube clip for it?
- Cyphase 2y agohttps://www.youtube.com/watch?v=-MwCJpEuC44 https://www.youtube.com/watch?v=-MwCJpEuC44
- puppycodes 2y agoSorry for us utter simpletons can someone explain what it do?
- fasterergpes 2y agoIt makes it so that having more GPUs makes inference run faster. Worst case has been you can only use memory from them and gain no speed at all
- 456yetdh6r 2y ago[flagged]
- qrios 2y agoIn very simple words: it is one way to reduce the white squares in the picture from @danielhanchen[1]. In more complex words: imagine a processor which is able to process every instruction in 10 clock cycles. But also the processor is able to get new input for this instruction on every clock cycle and starts to process this new input in a pipeline. After the first input you have to wait ten clock cycles. But if you feed the input line every time you will get the output also permanently. In the case of GPUs, it is now not only a topic of a single pipeline, but multiple in parallel. Depends on your data and algorithm it can be thousands in parallel. [1] https://x.com/danielhanchen/status/1894937006352031832 https://x.com/danielhanchen/status/1894937006352031832
- xnhbx 2y ago> DualPipe was created and developed by Jiashi Li and Chengqi Deng and Wenfeng Liang. A CEO who codes.
- anonzzzies 2y agoWhen my company was still working closely with CN factories a few years ago (before the bans / clients no longer wanting to work with companies working with china etc), the CEO's of the factories we worked with all were electronic engineers at that company or another before; they all could jump in, debug schematics, sold and write firmware themselves. And they did. These were places with massive campuses with towering buildings with robots and a few (relative to the massive space) employees doing maintenance etc + prototyping.
- larodi 2y agoIt sounds so more reasonable to have a director who is actually technical, doesn't it? I'm absolutely amazed how this (to the east) contrasts to understanding (to the west) that directors rather need to know finance, strategic planning, and marketing, than the actual nuance of the work.
- tway223 2y agoTo be blunt this is exactly what is wrong with the “leadership” mindset in the west, as decisions are often made without understanding the “nuances” yet they are confident it would work.
- tantalor 2y ago"developed" and "codes" have different meanings.
- ikeashark 2y agoYes but in this context, they are very close to each other in meaning. Besides Liang does indeed code a significant amount and has contributed to almost all of their published papers.
- jpcom 2y agoDoes this remind anyone else of the Pied Piper compression algorithm?
- aqueueaqueue 2y agoMiddle out or something?
- ringer007 2y ago[dead]
- optimalplusone 2y agoI hope all the open sources Deepseek is doing encourages American labs to do more of the same. Surely they'll realize their momentum is more of a moat than their tech at any one point in time.