5 ms·

HW noob here, anyone here has insight on how an issue like this passes EM simulation during development? I understand that modern chips are way too complex for

HW noob here, anyone here has insight on how an issue like this passes EM simulation during development? I understand that modern chips are way too complex for full formal verification, but I'd have thought memory modules would be so highly structurally regular that it might be possible there despite it.

andyferris 1y ago
I am no expert in the field, but my reading of the original rowhammer issue (and later partial hardware mitigations) was that it was seen as better to design RAM that works fast and is dense and get that to market, than to engineer something provably untamperable with greater tolerances / die size / latency. GPUs have always been squarely in the "get stuff to consumers ASAP" camp, rather than NASA-like engineering that can withstand cosmic rays and such. I also presume an EM simulation would be able to spot it, but prior to rowhammer it is also possible no-one ever thought to check for it (or more likely that they'd check the simulation with random or typical data inputs, not a hitherto-unthought-of attack vector, but that doesn't explain more modern hardware).
- privatelypublic 1y ago
  I seem to recall that rowhammer was known- but thought impossible for userland code to implement. This is a huge theme for vulnerabilities. I almost said "modern" but looking back I've seen the cycle (disregard attacks as strictly hypothetical. Get caught unprepared when somebody publishes something making it practical) happen more than a few times.
  - Palomides 1y ago
    someone did a javascript rowhammer in 2015, hardware that's vulnerable today is just manufacturers and customers deciding they don't want to pay for mitigation (personally I think all RAM in all devices should be ECC)
    - grafmax 1y ago
      Manufacturers aren’t held liable for negligence like this. It’s a classic case of economic externality.
      - andyferris 1y ago
        Yes it is - how would you go about fixing that?
        yndoendo 1y ago
        Only means might be cultural. Security conferences such as DefCon or Black Hat create list of insecurely technology that is ubiquitousness and ignored by product designers and OEMs. Vote on ranking their priority and when they should be removed. News would latch on to "Hacks say all computers without ECC RAM are vulnerable and should not be purchased for their insecurity. Manufacturers like Dell, Asus, Acer, ... are selling products that help hackers steal your information." "DefCon Hackers thank Nvidia for making their jobs easier ..." Such statements would be refreshed during / after each security conference. There are over 12 conferences a year, about once a month these would be brought back into the public as a reminder. Public might stop purchasing from those manufacturers or choose the secure products to create the change.
    - andyferris 1y ago
      > manufacturers and customers deciding they don't want to pay It's more of a tragedy-of-the-commons problem. Consumers don't know what they don't know and manufacturers need to be competitive with respect to each other. Without some kind of oversight (industry standards bodies or goverment regulation), or a level of shaming that breaks through to consumers (or e.g. class action lawsuits that impact manufacturers), no individual has any incentive to change.
      - progmetaldev 1y ago
        Shame is an underrated way towards pushing for better standards. The problem is getting people in the know, and having them vote with their wallet, or at least public sentiment (social media pressure).
- userbinator 1y ago
  but prior to rowhammer it is also possible no-one ever thought to check for it It was known as "pattern sensitivity" in the industry for decades, basically ever since the beginning, and considered a blocking defect. Here's a random article from 1989 (don't know why first page is missing, but look at the references): http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensitive.pdf http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensiti... Then some bastards like these came along... https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_slides.pdf https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_sli... ...and essentially said "who cares, let someone else be responsible for the imperfections while we can sell more crap", leading to the current mess we're in. The flash memory industry took a similar dark turn decades ago.
MadnessASAP 1y ago
Given that I wasnt surprised by the headlie Inhave to imagine that nvidia engineers were also well aware. Nothing is perfect, everything has its failure conditions. The question is where do you choose to place the bar? Do you want your component to work at 60, 80, or 100C? Do you want it to work in high radiation environments? Do you want it to withstand pathological access patterns? So in other words, there isnt a sufficent market for GPUs at double the $/GB RAM but are resilient to rowhammer attacks to justify manufacturing them.
- wnoise 1y ago
  The idea of pathological RAM access patterns is as ridiculous as the idea of pathological division of floating point numbers. ( https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug ). The spec of RAM is to be able to store anything in any order, reliably. They failed the spec.
- bobmcnamara 1y ago
  > The question is where do you choose to place the bar? In the datasheet.
thijsr 1y ago
Rowhammer is an inherent problem to the way we design DRAM. It is a known problem to memory manufacturers that is very hard, if not impossible, to fix. In fact, Rowhammer only becomes worse as the memory density increases.
- sroussey 1y ago
  It’s a matter of percentages… not all manufacturers fell to the rowhammer attack. The positive part of the original rowhammer report was that it gave us a new tool to validate memory (it caused failures much faster than other validation methods).

andyferris 1y ago

I am no expert in the field, but my reading of the original rowhammer issue (and later partial hardware mitigations) was that it was seen as better to design RAM that works fast and is dense and get that to market, than to engineer something provably untamperable with greater tolerances / die size / latency. GPUs have always been squarely in the "get stuff to consumers ASAP" camp, rather than NASA-like engineering that can withstand cosmic rays and such. I also presume an EM simulation would be able to spot it, but prior to rowhammer it is also possible no-one ever thought to check for it (or more likely that they'd check the simulation with random or typical data inputs, not a hitherto-unthought-of attack vector, but that doesn't explain more modern hardware).

privatelypublic 1y ago
I seem to recall that rowhammer was known- but thought impossible for userland code to implement. This is a huge theme for vulnerabilities. I almost said "modern" but looking back I've seen the cycle (disregard attacks as strictly hypothetical. Get caught unprepared when somebody publishes something making it practical) happen more than a few times.
- Palomides 1y ago
  someone did a javascript rowhammer in 2015, hardware that's vulnerable today is just manufacturers and customers deciding they don't want to pay for mitigation (personally I think all RAM in all devices should be ECC)
  - grafmax 1y ago
    Manufacturers aren’t held liable for negligence like this. It’s a classic case of economic externality.
    - andyferris 1y ago
      Yes it is - how would you go about fixing that?
      - yndoendo 1y ago
        Only means might be cultural. Security conferences such as DefCon or Black Hat create list of insecurely technology that is ubiquitousness and ignored by product designers and OEMs. Vote on ranking their priority and when they should be removed. News would latch on to "Hacks say all computers without ECC RAM are vulnerable and should not be purchased for their insecurity. Manufacturers like Dell, Asus, Acer, ... are selling products that help hackers steal your information." "DefCon Hackers thank Nvidia for making their jobs easier ..." Such statements would be refreshed during / after each security conference. There are over 12 conferences a year, about once a month these would be brought back into the public as a reminder. Public might stop purchasing from those manufacturers or choose the secure products to create the change.
  - andyferris 1y ago
    > manufacturers and customers deciding they don't want to pay It's more of a tragedy-of-the-commons problem. Consumers don't know what they don't know and manufacturers need to be competitive with respect to each other. Without some kind of oversight (industry standards bodies or goverment regulation), or a level of shaming that breaks through to consumers (or e.g. class action lawsuits that impact manufacturers), no individual has any incentive to change.
    - progmetaldev 1y ago
      Shame is an underrated way towards pushing for better standards. The problem is getting people in the know, and having them vote with their wallet, or at least public sentiment (social media pressure).
userbinator 1y ago
but prior to rowhammer it is also possible no-one ever thought to check for it It was known as "pattern sensitivity" in the industry for decades, basically ever since the beginning, and considered a blocking defect. Here's a random article from 1989 (don't know why first page is missing, but look at the references): http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensitive.pdf http://web.eecs.umich.edu/~mazum/PAPERS-MAZUM/patternsensiti... Then some bastards like these came along... https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_slides.pdf https://research.ece.cmu.edu/safari/thesis/skhan_jobtalk_sli... ...and essentially said "who cares, let someone else be responsible for the imperfections while we can sell more crap", leading to the current mess we're in. The flash memory industry took a similar dark turn decades ago.

MadnessASAP 1y ago

Given that I wasnt surprised by the headlie Inhave to imagine that nvidia engineers were also well aware. Nothing is perfect, everything has its failure conditions. The question is where do you choose to place the bar? Do you want your component to work at 60, 80, or 100C? Do you want it to work in high radiation environments? Do you want it to withstand pathological access patterns? So in other words, there isnt a sufficent market for GPUs at double the $/GB RAM but are resilient to rowhammer attacks to justify manufacturing them.

wnoise 1y ago
The idea of pathological RAM access patterns is as ridiculous as the idea of pathological division of floating point numbers. ( https://en.wikipedia.org/wiki/Pentium_FDIV_bug https://en.wikipedia.org/wiki/Pentium_FDIV_bug ). The spec of RAM is to be able to store anything in any order, reliably. They failed the spec.
bobmcnamara 1y ago
> The question is where do you choose to place the bar? In the datasheet.

thijsr 1y ago

Rowhammer is an inherent problem to the way we design DRAM. It is a known problem to memory manufacturers that is very hard, if not impossible, to fix. In fact, Rowhammer only becomes worse as the memory density increases.

sroussey 1y ago
It’s a matter of percentages… not all manufacturers fell to the rowhammer attack. The positive part of the original rowhammer report was that it gave us a new tool to validate memory (it caused failures much faster than other validation methods).