11 ms·
GPUHammer: Rowhammer attacks on GPU memories are practical
- perching_aix 1y agoHW noob here, anyone here has insight on how an issue like this passes EM simulation during development? I understand that modern chips are way too complex for full formal verification, but I'd have thought memory modules would be so highly structurally regular that it might be possible there despite it.
- andyferris 1y agoI am no expert in the field, but my reading of the original rowhammer issue (and later partial hardware mitigations) was that it was seen as better to design RAM that works fast and is dense and get that to market, than to engineer something provably untamperable with greater tolerances / die size / latency. GPUs have always been squarely in the "get stuff to consumers ASAP" camp, rather than NASA-like engineering that can withstand cosmic rays and such. I also presume an EM simulation would be able to spot it, but prior to rowhammer it is also possible no-one ever thought to check for it (or more likely that they'd check the simulation with random or typical data inputs, not a hitherto-unthought-of attack vector, but that doesn't explain more modern hardware).
- privatelypublic 1y agoI seem to recall that rowhammer was known- but thought impossible for userland code to implement. This is a huge theme for vulnerabilities. I almost said "modern" but looking back I've seen the cycle (disregard attacks as strictly hypothetical. Get caught unprepared when somebody publishes something making it practical) happen more than a few times.
- Palomides 1y agosomeone did a javascript rowhammer in 2015, hardware that's vulnerable today is just manufacturers and customers deciding they don't want to pay for mitigation (personally I think all RAM in all devices should be ECC)
- grafmax 1y agoManufacturers aren’t held liable for negligence like this. It’s a classic case of economic externality.
- andyferris 1y agoYes it is - how would you go about fixing that?
- yndoendo 1y agoOnly means might be cultural. Security conferences such as DefCon or Black Hat create list of insecurely technology that is ubiquitousness and ignored by product designers and OEMs. Vote on ranking their priority and when they should be removed. News would latch on to "Hacks say all computers without ECC RAM are vulnerable and should not be purchased for their insecurity. Manufacturers like Dell, Asus, Acer, ... are selling products that help hackers steal your information." "DefCon Hackers thank Nvidia for making their jobs easier ..." Such statements would be refreshed during / after each security conference. There are over 12 conferences a year, about once a month these would be brought back into the public as a reminder. Public might stop purchasing from those manufacturers or choose the secure products to create the change.
- andyferris 1y ago> manufacturers and customers deciding they don't want to pay It's more of a tragedy-of-the-commons problem. Consumers don't know what they don't know and manufacturers need to be competitive with respect to each other. Without some kind of oversight (industry standards bodies or goverment regulation), or a level of shaming that breaks through to consumers (or e.g. class action lawsuits that impact manufacturers), no individual has any incentive to change.
- progmetaldev 1y agoShame is an underrated way towards pushing for better standards. The problem is getting people in the know, and having them vote with their wallet, or at least public sentiment (social media pressure).
- userbinator 1y agobut prior to rowhammer it is also possible no-one ever thought to check for it It was known as "pattern sensitivity" in the industry for decades, basically ever since the beginning, and considered a blocking defect. Here's a random article from 1989 (don't know why first page is missing, but look at the references): http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensitive.pdf http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensiti... Then some bastards like these came along... https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_slides.pdf https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_sli... ...and essentially said "who cares, let someone else be responsible for the imperfections while we can sell more crap", leading to the current mess we're in. The flash memory industry took a similar dark turn decades ago.
- MadnessASAP 1y agoGiven that I wasnt surprised by the headlie Inhave to imagine that nvidia engineers were also well aware. Nothing is perfect, everything has its failure conditions. The question is where do you choose to place the bar? Do you want your component to work at 60, 80, or 100C? Do you want it to work in high radiation environments? Do you want it to withstand pathological access patterns? So in other words, there isnt a sufficent market for GPUs at double the $/GB RAM but are resilient to rowhammer attacks to justify manufacturing them.
- wnoise 1y agoThe idea of pathological RAM access patterns is as ridiculous as the idea of pathological division of floating point numbers. ( https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug ). The spec of RAM is to be able to store anything in any order, reliably. They failed the spec.
- bobmcnamara 1y ago> The question is where do you choose to place the bar? In the datasheet.
- thijsr 1y agoRowhammer is an inherent problem to the way we design DRAM. It is a known problem to memory manufacturers that is very hard, if not impossible, to fix. In fact, Rowhammer only becomes worse as the memory density increases.
- sroussey 1y agoIt’s a matter of percentages… not all manufacturers fell to the rowhammer attack. The positive part of the original rowhammer report was that it gave us a new tool to validate memory (it caused failures much faster than other validation methods).
- privatelypublic 1y agoThis seems predicated on there being significant workloads that split gpu's between tenants for compute purposes. Anybody have sizable examples? Everything I can think of results in dedicated gpus.
- huntaub 1y agoMy (limited) understanding was that the industry previously knew that it was unsafe to share GPUs between tenants, which is why the major cloud providers only sell dedicated GPUs.
- im3w1l 1y agoWebgpu api taking screenshot of full desktop maybe?
- Buttons840 1y agoDo you think WebGPU would be any more of an attack vector than WebGL?
- privatelypublic 1y agoRowhammer itself is a write-only attack vector. It can, however, potentially be chained to change the write address to an incorrect region. Haven't dived into details.
- SnowflakeOnIce 1y agoHow is it a write-only attack vector?
- privatelypublic 1y agoRowhammer allows you to corrupt/alter memory physically adjacent to memory you have access to. It doesn't let you read the memory you're attacking. There's PoC's of corrupting memory _that the kernel uses to decide what that process can access_ but the process can't read that memory. It only knows that the kernel says yes where it used to say no. (Assuming it doesn't crash the whole machine first)
- userbinator 1y agoNo one really cared about the occasional bitflips in VRAM when GPUs were only used for rendering graphics. It's odd that enabling ECC can reduce performance, unless they mean that's only in the presence of ECC errors being corrected, since AFAIK for CPUs there isn't any difference in speed even when correcting errors. In a proof-of-concept, we use these bit flips to tamper with a victim’s DNN models and degrade model accuracy from 80% to 0.1%, using a single bit flip There is a certain irony in doing this to probabilistic models, designed to mimic an inherently error-prone and imprecise reality.
- hamandcheese 1y agoAvailable ECC dims are often slower than non-ECC dims. Both slower MT/s and higher latency. At least for "prosumer" ECC UDIMMs which are what I'm familiar with. So it doesn't seem that wild to me that turning on ECC might require running at lower bandwidth.
- ryao 1y agoThis is incorrect. ECC DIMMs are no slower than regular DIMMs. Instead, they have extra memory and extra memory bandwidth. A 8GB DDR4 ECC DIMM would have 9GB of memory and 9/8 the memory bandwidth. The extra memory is used to store the ECC bits while the extra memory bandwidth is to prevent performance loss when reading/writing ECC alongside the rest of the memory. The memory controller will spend an extra cycle verifying the ECC, which is a negligible performance hit. In reality, there is no noticeable performance difference. However, where you would have 128 traces to a Zen 3 CPU for DDR4 without ECC, you would need 144 traces for DDR4 with ECC. A similar situation occurs with GDDR6, except Nvidia was too cheap to implement the extra traces and pay for the extra chip, so instead, they emulate ECC using the existing memory and memory bandwidth, rather than adding more memory and memory bandwidth like CPU vendors do. This causes the performance hit when you turn on ECC on most Nvidia cards. The only exception should be the HBM cards, where the HBM includes ECC in the same way it is done on CPU memory, so there should be no real performance difference.
- bilegeek 1y ago
- iFire 1y agoDoes the ECC mode on my 4090 Nvidia rtx stop this?
- keysdev 1y agoWhat about Apple M series?
- fc417fc802 1y agoYes, but it reduces performance, and you don't need to care about this because (presumably) you aren't a cloud provider running multi-tenant workloads. Worst case scenario someone pulls this off using webgl and a website is able to corrupt your VRAM. They can't actually steal anything in that scenario (AFAIK) making it nothing more than a minor inconvenience.
- perching_aix 1y agoCouldn't it possibly lead to arbitrary code execution on the GPU, with that opening the floodgates towards the rest of the system via DMA, or maybe even enabling the dropping of some payload for the kernel mode GPU driver?
- fc417fc802 1y agoOnly if an attacker can (1) identify a piece of exploitable data that the GPU firmware or kernel mode portion of the driver relies on, (2) ascertain its location at runtime, and (3) obtain a physically adjacent block of memory. I'm not certain that something which satisfies 1, let alone 3, necessarily exists. On the CPU you flip some bits related to privilege levels. Are any analogous and similarly exploitable data structures maintained by common GPU firmware? And if so, is such data stored in the bulk VRAM? It wouldn't surprise me if it was possible but it also seems entirely unrealistic given that either you are restricted to an API such as webgl (gl;hf) or you have native access in which case you have better options available to you seeing as the driver stacks are widely rumored to be security swiss cheese (if you doubt this look at how much effort google has invested in sandboxing the "real" GPU driver away from apps on chromeos).
- PeterStuer 1y agoI always found hammering attacks to be extremely satisfying, even from a meta-physical pov. You escape a closed virtual universe by not "breaking out" in the tradidional sense, exploiting some bug in the VM hypervisor's boundary itself, but by directly manipulating the underlying physics of the universe on wich the virtual universe is founded, just by creating a pattern inside the virtual universe itself. No matter how many virtual digital layers, as long as you can impact the underlying analog substrate this might work. Makes you dream there could be an equivalent for our own universe?
- arduinomancer 1y agoI tried knocking on my wall 100,000 times and it did indeed cause a disturbance in the neighbouring cell of my apartment Turns out this whole virtualized house abstraction is a sham
- N_Lens 1y agoThanks for the sensible chuckle.
- SandmanDP 1y ago> Makes you dream there could be an equivalent for our own universe? I’ve always considered that to be what’s achieved by the LHC: smashing the fundamental building blocks of our universe together at extreme enough energies to briefly cause ripples through the substrate of said universe
- whyowhy3484939 1y agoThat's assuming there is a substrate that can be disturbed. That's where the parent's analogy breaks down. As an example of an alternative analogy: think of how many bombs need to explode in your dreams before the "substrate" is "rippled". How big do the bombs need to be? How fast does the "matter" have to "move"? I think "reality" is more along those lines. If there is a substrate - and that's a big if - IMO it's more likely to be something pliable like "consciousness". Not in the least "disturbed" by anything moving in it.
- sylware 1y agoOn the general case, that's why some optimized assembly written machine code can be an issue compared to the slow compiler generated machine code (not true all the time of course): if this machine code is 'hammering' memory, it is could happen more likely with the optimized assembly machine code than with the "actually tested" compiler genertade machine code.
- saagarjha 1y agoNo, in fact you can often Rowhammer inside an interpreter if you construct it correctly.
- sylware 1y agoPoint missed: this is not what I said.
- saagarjha 1y agoIt's slightly easier because native code is typically faster but I would not say that it would cause it to be the case where you can only do the attack by handwriting assembly. Rowhammer involves hitting memory locations which is generally easy to do from any context (most languages have arrays…). It's not like you need to do weird branch prediction stuff which might require specific gadgets to be present.
- sylware 1y agoFine-grained machine code is likely to build more efficient/successful exploit. But the goal I guess is to run that in one of the whatng cartel web engines (aka "sneaky" javascript), which are by themselves a security flaw already...