5 ms·
OK this was a bit funny: Top HW failure modes: * GPU falling off the bus I honestly thought "do they mean GPUs falling off a bus entering the data center
by samspenc 2y ago
OK this was a bit funny:
Top HW failure modes:
* GPU falling off the bus
I honestly thought "do they mean GPUs falling off a bus entering the data center" and then realized its actually the connectivity, as they mention in the next line
GPUs falling off: In this case, GPUs are not detected by the host on PCIe.
- cachvico 2y agoBrings a whole new meaning to bus factor
- throwup238 2y agoI’ve never met a GPU that could survive getting hit by a bus.
- ChuckMcM 2y agoThe bits on the bus go round and round! There is a lot of interesting yet unpublished work on 'data center' scale compute complexes. It was a rabbit hole I fell into several times while at Google.
- falcor84 2y agoThey do publish some of that, or at least they used to. In particular "The Datacenter as a Computer" [0] was a very interesting read. [0] https://research.google/pubs/the-datacenter-as-a-computer-an-introduction-to-the-design-of-warehouse-scale-machines-second-edition/ https://research.google/pubs/the-datacenter-as-a-computer-an...
- martin-adams 2y agoA GPU falling off the bus would be one mega flop
- m463 2y agoActually, they "fell" off the truck: https://www.theverge.com/2021/11/6/22767046/someone-stole-shipment-evga-rtx-graphics-cards https://www.theverge.com/2021/11/6/22767046/someone-stole-sh...
- sva_ 2y agoBack when EVGA was still selling GPUs ...
- TiredOfLife 2y agoAnd offering warranty. And not doing stealth total component change under same sku.
- taneq 2y agoThey did it to finance their street racing habit, I'm sure. :P
- whazor 2y agoI was imagining that some sys admin has to walk to the server, take out the GPU, blow against the PCI-E pins like a game cartridge, and put it back to try again.
- iszomer 2y agoMore to do with bent pins, material obstruction, or something as trivial as cable management (eg: bundles of qsfp weighing down the ports that are press-fitted not soldered).
- rfoo 2y ago"GPU has fallen off the bus" is an actual error message nvidia.ko prints to dmesg in this case :p
- redbell 2y ago> GPU falling off the bus I'm wondering if we could prompt llama3 with the above statement. What kind of response would it give?
- TeMPOraL 2y agoWith temperature set to 1, it recognizes the joke, but proceeds to explain what the "bus" is in computer terms, picks a problem this prompt could mean, and explains how to solve it. In ~20 tries it always gave me something along the lines of: The infamous "GPU falling off the bus" issue! This problem typically occurs when a graphics processing unit (GPU) is not properly seated or connected to its expansion slot, such as PCIe, on a motherboard. Here are some troubleshooting steps to help resolve the issue: (numbered list of steps or options follows) Tested on Llama 3 Instruct 7B Q8_0, because that one fits entirely on my GPU.
- redbell 2y ago+1, interesting findings! I like how it was able to infer the meaning from such a short phrase in a limited context.
- TeMPOraL 2y agoTo be specific, the system prompt used was (default in LM Studio config for Llama 3 V2): You are a helpful, smart, kind, and efficient AI assistant. You always fulfill the user's requests to the best of your ability. And then the query was: GPU falling off the bus And yes, I imagine it read that query as ending with an implied "pls help!".
- burkaman 2y agoIt's actually a very common phrase on forums, I think because it's an actual error that Linux will report: https://askubuntu.com/questions/868321/gpu-has-fallen-off-the-bus-nvidia https://askubuntu.com/questions/868321/gpu-has-fallen-off-th.... I've also never heard of it, but it seems like it must appear a lot in the training data and probably about 0 times is referring to a bus on the road.
- krowfromthewall 2y agolike they did to our dear Anton in Silicon Valley
- spmurrayzzz 2y agoSpeaking for myself (and I guess anyone else dealing with pcie riser hell in on-prem deep learning setups), its nice to see the massive orgs dealing with pretty much the same exact pain points as not-so-massive orgs.