8 ms·

PCIe trouble with 4TB Crucial T500 NVMe SSD for >1 power cycle on MSI PRO X670-P

167 points by transpute 2y ago

bdavbdav 2y ago
I had the same on an AORUS X570. Displayport cables with a line tied both ends (shouldn’t be, but many are) would cause BIOS resets, corruption and memory retraining.
blagie 2y ago
I was an exclusive user of Crucial for memory and storage until about a year ago. My general thought was that: - It would give me a trusted supply chain, since the company makes the silicon; and - I would have a credible standing behind it, which wasn't likely to want to tarnish its reputation cutting corners. The thinking was very much along the lines of "No one got fired buying IBM." And I think it was pretty correct for most of the past quarter-century. Historically, storage had a lot of counterfeits and shenanigans, and a credible vendor was nice. Price/performance for memory was adequate; there was a modest premium. However, post-2020, I bought a defective Crucial DIMM (and didn't find out it was defective until I was past the return window). The RMA experience was strange. Crucial said they could either: - Replace it with an inferior part with different, slower timings, which may or may not have worked in my system - Give me a quickly-expiring store credit for "fair market value" (never disclosing what that was, and stopped responding to emails when I asked) Neither of these was helpful at all. Reading online, there were many similar stories, unfortunately. They seem to be going the same direction as Sandisk / Western Digital. I replaced it with a cheap TeamGroup DIMM which worked without problems. I'm not quite sure what to do about the continued enshitification. There seem to be almost no credible brands left.
sciencesama 2y ago
I have a similar issue with nvme on a wlan slot on the lenovo thinkpad gen 8 !!
tfwnopmt 2y ago
HDMI provides power - that's how old chromecasts can work without a separate power plug. The comment about NPNs and PNPs is garbage, but there is a design fault with the board - it shouldn't allow HDMI power to flow backwards into the motherboard when the motherboard shuts off. That would likely cause a power rail sequencing issue on the board or SSD, leading to latch-up of various ICs, and non-detection on the SSD on the flowing bootup
- LeifCarrotson 2y ago
  And by "the board" I trust you mean the MSI PRO X670-P WIFI motherboard. There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. If there's a trickle (and to be clear, the 5V at 300 mA available from an HDMI cable is a trickle for a full motherboard) of current into the 3V3 bus on the ATX connector, something will be the very lowest PMIC to turn on. It's just that on this system, the SSD was the first thing. If anything, the SSD will probably be highly tolerant of brownouts because its LDO will run at around 1.9V.
  - hulitu 2y ago
    > There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. It shall set itself in Reset state.
    - LeifCarrotson 2y ago
      That would be nice, in practice, the SSD requires its power rails to start up in a particular sequence and with very particular voltages.
    - shadowpho 2y ago
      Only few devices are actually able to do that. Vast majority require require proper voltage sequencing, because to do otherwise is to add cost to your IC
  - Dylan16807 2y ago
    > There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. I'd put some more emphasis on "when", though. If it never comes back when power comes back that's not particularly correct.
    - crest 2y ago
      That's because if this theory is correct from the point of view of the SSD there was no reboot yet, because there was never any total power loss.
jauntywundrkind 2y ago
I can't get my Crucial P3+ to wake from sleep. I'd like to dig in more but I haven't had this issue with any other SSD in this system. Pretty close to saying I'm done with Crucial.
- NewJazz 2y ago
  I've had a similar experience with a crucial nvme drive, but a kernel update seems to have introduced a quirk-based fix. Not sure how much of a kludge that fix is, though.
  - wtallis 2y ago
    The quirks tables in the Linux NVMe drivers are impressive and depressing: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/core.c#n2782 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/pci.c#n3090 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/pci.c#n3583 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... And they're not even close to being comprehensive.
    - fulafel 2y ago
      Interesting that there are also some anti-quirk special cases in the vendor combo function (second link above), so a certain platform is excepted from the quirk workaround: \* \* Exclude some Kingston NV1 and A2000 devices from \* NVME_QUIRK_SIMPLE_SUSPEND. Do a full suspend to save a \* lot fo energy with s2idle sleep on some TUXEDO platforms. \*/ if (dmi_match(DMI_BOARD_NAME, "NS5X_NS7XAU") || dmi_match(DMI_BOARD_NAME, "NS5x_7xAU") || dmi_match(DMI_BOARD_NAME, "NS5x_7xPU") || dmi_match(DMI_BOARD_NAME, "PH4PRX1_PH6PRX1")) return NVME_QUIRK_FORCE_NO_SIMPLE_SUSPEND;
      - wtallis 2y ago
        I think some of those issues probably stem from the fact that there's not really any alignment between the NVMe spec and the PCIe spec with respect to power management capabilities. I've encountered drives that have implicit dependencies where certain NVMe power management features only work as intended when certain PCIe power management features are available, but there's no way for the drive to express those requirements to the host system, and no standard compliance test suite that will reveal the broken behavior that can occur in the wild. Sometimes figuring out who to blame for misbehaving hardware requires custom kernel patches, a hardware protocol analyzer at the M.2 slot, and reverse-engineering the motherboard firmware. Most of the entries in the quirks tables are based on a lot of guess-work and inferences because the kernel developers don't have the resources to fully investigate and reproduce these kinds of issues (and the hardware vendors simply don't care about thoroughly ironing out these bugs). It really sucks when you have to look at power flow out of the laptop battery and try to figure out from that whether your SSD is pulling more power than it should.
zamadatix 2y ago
On the topic of odd failure modes involving Crucial SSDs and MSI motherboards (though one that seems to actually be the drives fault) I have a t705 which at some point started only coming up as x2 lanes instead of x4 no matter which board I put it into (with no visible damage or indication as to why, though I did try to wipe down the contact side with some rubbing alcohol anyways). The particularly interesting part is I have a new x870 motherboard which supports m.2 slot 2 as being 0x, 2x, or 4x CPU direct lanes depending if you want 4x, 2x, or 0x to go to the USB 4 ports respectively. At first it sounds like a good combo - put the drive which wants to run at x2 only in the extra slot where x2 only mode is a reasonable tradeoff and still get great bandwidth because those lanes are pcie 5 and not through the chipset. For whatever reason though that drive only ever comes up in an x4 slot (at x2 speed) but not any x2 slots I've tried. I don't know enough about PCIe to assume why that is for sure but it seemed odd to me it was any way but "something is wrong with the 3rd or 4th lane and setting the slot to x2 lets the first 2 work at x2 the same as when the slot is set to x4 and it only comes up as x2".
- magicalhippo 2y ago
  PCIe devices are required to boot up using x1 lane only, and then negotiate further lanes with upstream. AFAIK it shouldn't matter if they're direct to CPU or not, at least not logically. I note the drive is Gen5 capable, does it negotiate x2 5.0 lanes or something else?
  - zamadatix 2y ago
    Negotiates to 2x 5.0 so long as the board it's plugged into supports it. 2x 4.0 or 3.0 otherwise. Hadn't tested even lower.
- tfwnopmt 2y ago
  I came across this in a manual/datasheet: >16.Link Width Negotiation in the Presence of Bad Lanes >In an effort to maximize the link width when one or more lanes of a multi-lane link are not functioning correctly (i.e., reliable communication of training sets across the lane is not possible), PES64H16G2 down-stream switch ports automatically attempt a lane reversed configuration when doing so has the potential to enhance the achievable link width.For example, if lane 1 of a x4 link is not operating correctly, the device's downstream switch port attached to the link attempts a lane reversed configuration to form a x2 link using lanes 2 and 3 (Figure 7.4(d)). If the link partner accepts the lane reversed configuration, the optimal x2 link will be formed using lanes2 and 3. If the link partner does not accept the lane reversed configuration, but instead requests a lane configuration supported by the PES64H16G2 (e.g., x1 link using lane 0), the device accepts the configuration and forms the reduced width link. Otherwise, if the lane numbering agreement fails, the device automatically re-trains the link from the Detect state. During this re-training, the PES64H16G2 port does not re-attempt a lane reversed configuration, but rather tries to form the link without reversing the lanes. As a result, a x1 link is formed using lane 0 (Figure 7.4 (e)). My guess is it's likely a bad BGA solder ball on Lane1, or possibly ESD damage if you took the SSD out and molested it or rubbed it on a cat right before it broke. Does it indicate it's using reversed lanes?
geor9e 2y ago
Why's a random tech support forum post from yesterday with 2 people replying getting reposted to HN
- aprilnya 2y ago
  I personally found it interesting.
- frantathefranta 2y ago
  Slow week but people probably enjoy the methodical troubleshooting.
- ejiblabahaba 2y ago
  For what it's worth, this post just helped me explain several years of failure to wake from sleep state, across several different MSI-based machines, when I've connected them to an HDMI port in my TV. I think this debug is interesting in its own right, and unlike 99% of the content on this website, it was directly and immediately useful to me. I doubt I'm the only one, too.
- transpute 2y ago
  This post described a rare interoperability failure with unexpected root cause, of possible interest to: Motherboard designers People upgrading PCs/laptops SSD firmware developers BIOS developers attempting PCIe device boot OS/hypervisor developers attempting PCIe device reset If you don't like this HN story, you could contribute your first story to HN.
sebazzz 2y ago
I have something similar with my webcam, which is connected to my Samsung monitor usb hub, which is connected to a usb-c dongle, which is connected to my work laptop. If my laptop crashes during a Microsoft Teams call, possibly due to the webcam, it will not show up in Windows again without it physically being disconnected from the USB hub in my Samsung monitor. I can disconnect the USB-C dongle or the monitor from USB, change ports, power off the laptop, it doesn't matter because that doesn't work. Only physically disconnecting and reconnecting it makes it show up in device manager again.
qingcharles 2y ago
I hate faults like that. Used to work in PC repair. Man brings in PC, mouse right click doesn't work. Everything else operates perfectly. Replaced in this order: mouse, IO card, hard drive with fresh OS, RAM, CPU, graphics card, motherboard. Still no right-click. Replaced the PSU last. Right-click works. FML.
- donalhunt 2y ago
  Reminds me of an old hwops story where one machine just constantly failed despite replacing every part on the tray multiple times. The conclusion was that the tray was bad. Google's definition of a server was (and still is afaik) based on the tray (chassis) so there was no way to replace it. IIRC it was "retired" with vengeance leaving a gap in the cabinet — a warning to other trays to behave.
- Frenchgeek 2y ago
  You didn't have to replace the house's wiring at least (Happened to an aunt of mine: Gave her a computer, it worked perfectly outside of her home. The electrician was a tad horrified. She still scoffed when I suggested the computer wasn't the problem first.)
  - qingcharles 2y ago
    Yeah, lucky it didn't get that far, "Sorry bro, I gotta try just replacing your house."
  - Moru 2y ago
    I plugged my old Atari into an outlet in the old basement in a different building. The HDD-cable started burning. Electric company plugged in some device to measure power over time. Turns out the power was slightly below normal but within tollerances. The OEM power supply that was powering my Atari wasn't up to standards. If I remember right, badly designed PSU's can feed too high current if the voltage is too low. Or something like that, was a very long time ago...
    - ajb 2y ago
userbinator 2y ago
This is a good cautionary story of why random parts-swapping can be a waste of time and money. Getting out the DMM and measuring voltages is something fewer and fewer people know how to do when troubleshooting electronics, but it certainly saved the OP here; I'd go a little further and figure out why the monitor seems to be leaking power into its HDMI input when switched off --- possibly an ESD-damaged MOSFET or similar? The issue does not occur when the monitor is connected via DisplayPort. https://en.wikipedia.org/wiki/DisplayPort#DP_PWR_(pin_20) https://en.wikipedia.org/wiki/DisplayPort#DP_PWR_(pin_20) Standard DisplayPort cable connections do not use the DP_PWR pin. There's also an interesting paragraph there, about some nonstandard cables connecting that pin through.
- hamandcheese 2y ago
  On the other hand, I recently fried a motherboard while trying to probe it with a multimeter. My fat fingers shorted out two adjacent pins, causing a loud spark and magic smoke.
- Arcanum-XIII 2y ago
  Not all DMM have probe small enough to connect to the lane. If it's even possible. What's more, you need to know where to put it, which can be daunting without the proper knowledge. Switching hardware is easier, faster and often the best solution in those case. Finding hardware fault is hard. Tracing it is even harder.
  - userbinator 2y ago
    I think there's something wrong with your DMM probes if you can't measure the ATX power connector with them.
bunnie 2y ago
Reading the thread it looks like the issue is leakage power on the internal 3.3v line. When the system is off 1.9v is still present. This is not uncommon, although 1.9v is a bit high. A lot of laptops have explicit active pull downs on power supplies to clamp them to zero when power is off to ensure peripherals are not accidentally powered on by stray leakage (because laptops are extremely low power by design and there is not enough stray leakage to bring the power lines down in a sleep state). My guess is main boards might not have this feature because normally there is enough off state loading that it takes care of itself. however maybe in this case the loading is not enough. A dirty fix could be to just put a static load on the 3.3v line to ground. I'd start with a 1/4w resistor around 100 ohms and just stick it from 3.3v to ground to see if that does not soak up the stray current. if it works just leave it, it's about 0.1 watts of static power and no big deal for a non portable setup. The larger picture is that the controller on the nvme might not hit its power on reset condition because it may be rated to run at 1.8v (just a guess), so 3.3v is not going low enough for the controller to perceive the system has been power cycled. Usually a supplemental power monitor is needed in those cases to ensure a reset is generated in case of leakage problems like this.
- starslab 2y ago
  Hi! I'm the OP from the Level1Techs thread. That HDMI power has some grunt behind it. During power-off state with that 1.90v phantom voltage, I put a 48ohm resistor between 3V3 and ground, the phantom voltage only dropped to 1.80v, and the SSD still didn't work when I powered the machine back on.
  - oneplane 2y ago
    Depending on the PMIC and the SSD DC conditioning, even 1.2v might be enough for it to brownout/latchup without self-resetting. (or it might power up the PHY partially or in a bad state and never link up) Try more resistors in series? (or just a bigger one if you have any -- scratch that we needed smaller ;-) ).
    - starslab 2y ago
okanat 2y ago
I bought the same model SSD for my Thinkpad P1 last month and saw the exact issue. I had to return it because it was breaking the NVMe detection completely. So it wasn't a broken unit but a design issue after all?
BearOso 2y ago
Since we're talking SSDs, I wonder if we could get some attention to the Phison E18 degradation issue [1]. Only one manufacturer, Kingston, has put out firmware containing Phison's fix, while the others just ignore it. A bunch of these drives with this controller were on sale during black Friday, so a lot more people are going to have problems in a month or so. 1. https://www.reddit.com/r/pcmasterrace/comments/1f1piwf/psa_phison_e18_based_ssd_owners_update_your/ https://www.reddit.com/r/pcmasterrace/comments/1f1piwf/psa_p...
- userbinator 2y ago
  That sounds like NAND degradation (retention failures) which can only be partially worked around in firmware (and causing more write cycles on already-marginal QLC). Unfortunately the real solution is "use better NAND", which is unlikely to happen unless enough people demand it.
  - ciupicri 2y ago
    Kingston KC3000 supposedly uses Micron 176L TLC memory [1]. The Seagate Firecuda 530 datasheet clearly says "Built with a Seagate-validated E18 controller and the latest 3D TLC NAND". A review is more precise: "Phison PS5018-E18" & "Micron B47R 176-layer 3D TLC NAND" [2]. [1]: https://www.tomshardware.com/reviews/kingston-kc3000-m2-ssd-review https://www.tomshardware.com/reviews/kingston-kc3000-m2-ssd-... [2]: https://www.kitguru.net/components/ssd-drives/simon-crisp/seagate-firecuda-530-2tb-ssd-review/all/1/ https://www.kitguru.net/components/ssd-drives/simon-crisp/se...
    - userbinator 2y ago
      B47R is indeed TLC, rated for only 1000 cycles (and 35k in SLC mode, at 1/3 the capacity.) There's also the question of whether this is "true" Micron NAND, or SpecTek which is basically Micron's rejects (and rated for even fewer cycles; only 300 in the case of their B16A.)
- ciupicri 2y ago
  Kingston doesn't seem to offer any support for Linux, so their new firmware is virtually non-existent to me. Why can't I just download the firmware and use standard nvme-cli tools to update the SSD, beats me. If Seagate (which by the way uses Phison E18 too) can do it, so can Kingston, Samsung, Crucial, Western Digital and many others. Even better would be use Linux Vendor Firmware Service (https://fwupd.org/ https://fwupd.org/).
amelius 2y ago
I have a similar problem with a Jetson board. If I turn off the power long enough (one night) and then turn it on, the only PCI card is not recognized and I have to power-cycle it to get it running.
- structural 2y ago
  Mind sharing what board/Jetson module you've seen this on? I've seen this exact symptom very intermittently on a custom board and we've wondered for a long time if was an issue with a specific type of module (or manufacturing lot of modules).
  - amelius 2y ago
    This one: https://www.avermedia.com/professional/product-detail/D315%205G https://www.avermedia.com/professional/product-detail/D315%2... My startup logic now power-cycles it until the PCI board is recognized; it works, but it's not a great solution.
    - structural 2y ago
      Interesting, we're using a completely different module (Xavier NX). And the same, disgustingly hacky, fix, of forcing a reset until it works.
      - amelius 2y ago
        I also run these commands: echo 1 > /sys/bus/pci/rescan sleep 1 Sometimes it brings the PCI card back, so I just run this as part of my boot sequence.
deleted 2y ago
[deleted]
undertaken 2y ago
anecdotal/weird computer experience: I have a rebadged Tongfang laptop (NB02 GMxRGxx w/ Ryzen 9) and upgraded it shortly after purchase. The machine arrived with lower capacity Samsung SODIMMs. Swapped in 64GB of Crucial DDR5. Shortly afterwards the machine became instable to the point of RMA. Kernel logs clogged with all sorts of panics related to NVMe, PCIe, and filesystem. Freezing. Reboots. Spent hours diagnosing it. Many permutations of kernel command line arguments; pcie, acpi tables, iommu. All for naught. The machine passed memtest86 / memtest86+ with flying colors. bonnie++ absolutely trashed it. reliably. Occasionally the NVMe drives fell off the pci bus and it wouldn't boot until I disable the slot in bios, power cycle, then re-enable the slot. Fast forward to me getting fed up with a dysfunctional system, I attmepted RMA and gave them the rundown of all the weird seemingly chipset related failures. They pushed back with "Try our RAM again." I nearly had an aneurysm when everything was stable again. After thanking the support staff profusely I bought larger capacity Samsung DIMMs in the same chip family. Still running flawlessly after almost a year. Maybe try new RAM for yucks? ;)
bb88 2y ago
So these guys [1] mention something similar where HDMI from a TV is backfeeding 40-50 volts into a cable box. This could be because of many things from electrical outlet wiring to power supply issues on the monitor to a bad component on the monitor giving a high voltage, or the monitor is badly grounded, etc, etc. I read the original thread but it doesn't look like you've measured the voltage at the HDMI port wrt motherboard ground. I think we're assuming it's 5 volts, but it could be higher, and it could have shorted (or weakened) a component on your motherboard. And that would explain why a 100 ohm resistor didn't give a meaningful voltage drop. If you need an isolation solution, Amazon sells a 50ft fiber optic one way HDMI cable [2]. The thing I don't know is if there's any actual copper to provide power over the link. There are other options which transmit the HDMI signal over pure multimode fiber as well [3]. Or you can go with a DP KVM, since you're on L1T, they sell a few DP models. I have one I purchased from L1T, and I like it a lot. Definitely though I would check out the outlets to make sure they were wired correctly. Incorrectly wired outlets because someone tried to DIY it in the US is absolutely a problem. [1] https://www.avsforum.com/threads/hdmi-cable-backfeeding-voltage-from-tv-to-cable-box.1345211/ https://www.avsforum.com/threads/hdmi-cable-backfeeding-volt... [2] https://www.amazon.com/HDMI-FURUI-HDCP2-2-18Gbps-Subsampling/dp/B077D37MT3?th=1 https://www.amazon.com/HDMI-FURUI-HDCP2-2-18Gbps-Subsampling... [3] https://fibercommand.com/products/8k-fiber-plugs?gQT=1 https://fibercommand.com/products/8k-fiber-plugs?gQT=1
- starslab 2y ago
  I already own one of those fiber-hdmi cables. Brilliant, but sometimes doesn't interoperate with DVI devices using passive DVI -> HDMI adapters. I've no idea if it has any copper conductors for HDMI power, though one end is labelled for the source and one for the display, suggesting that however it's designed it's not bi-directional. I'd love a DisplayPort KVM, but not every device that comes across my bench has a DisplayPort output, and those few that have DisplayPort but no HDMI can be accommodated with one of those commodity DisplayPort -> HDMI adapters. This situation is actually getting worse over time, not better, as many modern devices and laptops are skipping DisplayPort in favor of USB-c alt-mode. This issue has actually been going on through a monitor change on my testbench. It has happened with a Samsung SyncMaster 204T though my KVM switch, an HP ZR24w through my KVM switch, and the ZR24w directly connected. I don't think this is an issue with the rest of my equipment. This electrical was done about 15 years ago, by a ticketed electrician. One of those $5 plug testers indicates all is well, and I have no reason to believe there's any issue here. By almost pure coincidence, I have an MSI PRO X870-P motherboard on order. I'm looking forward to seeing if this same 3V3 leakage issue is present on this board too.

167 points by transpute 2y ago

I had the same on an AORUS X570. Displayport cables with a line tied both ends (shouldn’t be, but many are) would cause BIOS resets, corruption and memory retraining.

blagie 2y ago

I was an exclusive user of Crucial for memory and storage until about a year ago. My general thought was that: - It would give me a trusted supply chain, since the company makes the silicon; and - I would have a credible standing behind it, which wasn't likely to want to tarnish its reputation cutting corners. The thinking was very much along the lines of "No one got fired buying IBM." And I think it was pretty correct for most of the past quarter-century. Historically, storage had a lot of counterfeits and shenanigans, and a credible vendor was nice. Price/performance for memory was adequate; there was a modest premium. However, post-2020, I bought a defective Crucial DIMM (and didn't find out it was defective until I was past the return window). The RMA experience was strange. Crucial said they could either: - Replace it with an inferior part with different, slower timings, which may or may not have worked in my system - Give me a quickly-expiring store credit for "fair market value" (never disclosing what that was, and stopped responding to emails when I asked) Neither of these was helpful at all. Reading online, there were many similar stories, unfortunately. They seem to be going the same direction as Sandisk / Western Digital. I replaced it with a cheap TeamGroup DIMM which worked without problems. I'm not quite sure what to do about the continued enshitification. There seem to be almost no credible brands left.

sciencesama 2y ago

I have a similar issue with nvme on a wlan slot on the lenovo thinkpad gen 8 !!

tfwnopmt 2y ago

HDMI provides power - that's how old chromecasts can work without a separate power plug. The comment about NPNs and PNPs is garbage, but there is a design fault with the board - it shouldn't allow HDMI power to flow backwards into the motherboard when the motherboard shuts off. That would likely cause a power rail sequencing issue on the board or SSD, leading to latch-up of various ICs, and non-detection on the SSD on the flowing bootup

LeifCarrotson 2y ago
And by "the board" I trust you mean the MSI PRO X670-P WIFI motherboard. There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. If there's a trickle (and to be clear, the 5V at 300 mA available from an HDMI cable is a trickle for a full motherboard) of current into the 3V3 bus on the ATX connector, something will be the very lowest PMIC to turn on. It's just that on this system, the SSD was the first thing. If anything, the SSD will probably be highly tolerant of brownouts because its LDO will run at around 1.9V.
- hulitu 2y ago
  > There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. It shall set itself in Reset state.
  - LeifCarrotson 2y ago
    That would be nice, in practice, the SSD requires its power rails to start up in a particular sequence and with very particular voltages.
  - shadowpho 2y ago
    Only few devices are actually able to do that. Vast majority require require proper voltage sequencing, because to do otherwise is to add cost to your IC
- Dylan16807 2y ago
  > There's nothing incorrect about the behavior of the SSD when it's being operated outside the prescribed voltage and power thresholds. I'd put some more emphasis on "when", though. If it never comes back when power comes back that's not particularly correct.
  - crest 2y ago
    That's because if this theory is correct from the point of view of the SSD there was no reboot yet, because there was never any total power loss.

jauntywundrkind 2y ago

I can't get my Crucial P3+ to wake from sleep. I'd like to dig in more but I haven't had this issue with any other SSD in this system. Pretty close to saying I'm done with Crucial.

NewJazz 2y ago
I've had a similar experience with a crucial nvme drive, but a kernel update seems to have introduced a quirk-based fix. Not sure how much of a kludge that fix is, though.
- wtallis 2y ago
  The quirks tables in the Linux NVMe drivers are impressive and depressing: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/core.c#n2782 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/pci.c#n3090 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/drivers/nvme/host/pci.c#n3583 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/lin... And they're not even close to being comprehensive.
  - fulafel 2y ago
    Interesting that there are also some anti-quirk special cases in the vendor combo function (second link above), so a certain platform is excepted from the quirk workaround: \* \* Exclude some Kingston NV1 and A2000 devices from \* NVME_QUIRK_SIMPLE_SUSPEND. Do a full suspend to save a \* lot fo energy with s2idle sleep on some TUXEDO platforms. \*/ if (dmi_match(DMI_BOARD_NAME, "NS5X_NS7XAU") || dmi_match(DMI_BOARD_NAME, "NS5x_7xAU") || dmi_match(DMI_BOARD_NAME, "NS5x_7xPU") || dmi_match(DMI_BOARD_NAME, "PH4PRX1_PH6PRX1")) return NVME_QUIRK_FORCE_NO_SIMPLE_SUSPEND;
    - wtallis 2y ago
      I think some of those issues probably stem from the fact that there's not really any alignment between the NVMe spec and the PCIe spec with respect to power management capabilities. I've encountered drives that have implicit dependencies where certain NVMe power management features only work as intended when certain PCIe power management features are available, but there's no way for the drive to express those requirements to the host system, and no standard compliance test suite that will reveal the broken behavior that can occur in the wild. Sometimes figuring out who to blame for misbehaving hardware requires custom kernel patches, a hardware protocol analyzer at the M.2 slot, and reverse-engineering the motherboard firmware. Most of the entries in the quirks tables are based on a lot of guess-work and inferences because the kernel developers don't have the resources to fully investigate and reproduce these kinds of issues (and the hardware vendors simply don't care about thoroughly ironing out these bugs). It really sucks when you have to look at power flow out of the laptop battery and try to figure out from that whether your SSD is pulling more power than it should.

zamadatix 2y ago

On the topic of odd failure modes involving Crucial SSDs and MSI motherboards (though one that seems to actually be the drives fault) I have a t705 which at some point started only coming up as x2 lanes instead of x4 no matter which board I put it into (with no visible damage or indication as to why, though I did try to wipe down the contact side with some rubbing alcohol anyways). The particularly interesting part is I have a new x870 motherboard which supports m.2 slot 2 as being 0x, 2x, or 4x CPU direct lanes depending if you want 4x, 2x, or 0x to go to the USB 4 ports respectively. At first it sounds like a good combo - put the drive which wants to run at x2 only in the extra slot where x2 only mode is a reasonable tradeoff and still get great bandwidth because those lanes are pcie 5 and not through the chipset. For whatever reason though that drive only ever comes up in an x4 slot (at x2 speed) but not any x2 slots I've tried. I don't know enough about PCIe to assume why that is for sure but it seemed odd to me it was any way but "something is wrong with the 3rd or 4th lane and setting the slot to x2 lets the first 2 work at x2 the same as when the slot is set to x4 and it only comes up as x2".

magicalhippo 2y ago
PCIe devices are required to boot up using x1 lane only, and then negotiate further lanes with upstream. AFAIK it shouldn't matter if they're direct to CPU or not, at least not logically. I note the drive is Gen5 capable, does it negotiate x2 5.0 lanes or something else?
- zamadatix 2y ago
  Negotiates to 2x 5.0 so long as the board it's plugged into supports it. 2x 4.0 or 3.0 otherwise. Hadn't tested even lower.
tfwnopmt 2y ago
I came across this in a manual/datasheet: >16.Link Width Negotiation in the Presence of Bad Lanes >In an effort to maximize the link width when one or more lanes of a multi-lane link are not functioning correctly (i.e., reliable communication of training sets across the lane is not possible), PES64H16G2 down-stream switch ports automatically attempt a lane reversed configuration when doing so has the potential to enhance the achievable link width.For example, if lane 1 of a x4 link is not operating correctly, the device's downstream switch port attached to the link attempts a lane reversed configuration to form a x2 link using lanes 2 and 3 (Figure 7.4(d)). If the link partner accepts the lane reversed configuration, the optimal x2 link will be formed using lanes2 and 3. If the link partner does not accept the lane reversed configuration, but instead requests a lane configuration supported by the PES64H16G2 (e.g., x1 link using lane 0), the device accepts the configuration and forms the reduced width link. Otherwise, if the lane numbering agreement fails, the device automatically re-trains the link from the Detect state. During this re-training, the PES64H16G2 port does not re-attempt a lane reversed configuration, but rather tries to form the link without reversing the lanes. As a result, a x1 link is formed using lane 0 (Figure 7.4 (e)). My guess is it's likely a bad BGA solder ball on Lane1, or possibly ESD damage if you took the SSD out and molested it or rubbed it on a cat right before it broke. Does it indicate it's using reversed lanes?

geor9e 2y ago

Why's a random tech support forum post from yesterday with 2 people replying getting reposted to HN

aprilnya 2y ago
I personally found it interesting.
frantathefranta 2y ago
Slow week but people probably enjoy the methodical troubleshooting.
ejiblabahaba 2y ago
For what it's worth, this post just helped me explain several years of failure to wake from sleep state, across several different MSI-based machines, when I've connected them to an HDMI port in my TV. I think this debug is interesting in its own right, and unlike 99% of the content on this website, it was directly and immediately useful to me. I doubt I'm the only one, too.
transpute 2y ago
This post described a rare interoperability failure with unexpected root cause, of possible interest to: Motherboard designers People upgrading PCs/laptops SSD firmware developers BIOS developers attempting PCIe device boot OS/hypervisor developers attempting PCIe device reset If you don't like this HN story, you could contribute your first story to HN.

sebazzz 2y ago

I have something similar with my webcam, which is connected to my Samsung monitor usb hub, which is connected to a usb-c dongle, which is connected to my work laptop. If my laptop crashes during a Microsoft Teams call, possibly due to the webcam, it will not show up in Windows again without it physically being disconnected from the USB hub in my Samsung monitor. I can disconnect the USB-C dongle or the monitor from USB, change ports, power off the laptop, it doesn't matter because that doesn't work. Only physically disconnecting and reconnecting it makes it show up in device manager again.

qingcharles 2y ago

I hate faults like that. Used to work in PC repair. Man brings in PC, mouse right click doesn't work. Everything else operates perfectly. Replaced in this order: mouse, IO card, hard drive with fresh OS, RAM, CPU, graphics card, motherboard. Still no right-click. Replaced the PSU last. Right-click works. FML.

donalhunt 2y ago
Reminds me of an old hwops story where one machine just constantly failed despite replacing every part on the tray multiple times. The conclusion was that the tray was bad. Google's definition of a server was (and still is afaik) based on the tray (chassis) so there was no way to replace it. IIRC it was "retired" with vengeance leaving a gap in the cabinet — a warning to other trays to behave.
Frenchgeek 2y ago
You didn't have to replace the house's wiring at least (Happened to an aunt of mine: Gave her a computer, it worked perfectly outside of her home. The electrician was a tad horrified. She still scoffed when I suggested the computer wasn't the problem first.)
- qingcharles 2y ago
  Yeah, lucky it didn't get that far, "Sorry bro, I gotta try just replacing your house."
- Moru 2y ago
  I plugged my old Atari into an outlet in the old basement in a different building. The HDD-cable started burning. Electric company plugged in some device to measure power over time. Turns out the power was slightly below normal but within tollerances. The OEM power supply that was powering my Atari wasn't up to standards. If I remember right, badly designed PSU's can feed too high current if the voltage is too low. Or something like that, was a very long time ago...
  - ajb 2y ago

userbinator 2y ago

This is a good cautionary story of why random parts-swapping can be a waste of time and money. Getting out the DMM and measuring voltages is something fewer and fewer people know how to do when troubleshooting electronics, but it certainly saved the OP here; I'd go a little further and figure out why the monitor seems to be leaking power into its HDMI input when switched off --- possibly an ESD-damaged MOSFET or similar? The issue does not occur when the monitor is connected via DisplayPort. https://en.wikipedia.org/wiki/DisplayPort#DP_PWR_(pin_20) https://en.wikipedia.org/wiki/DisplayPort#DP_PWR_(pin_20) Standard DisplayPort cable connections do not use the DP_PWR pin. There's also an interesting paragraph there, about some nonstandard cables connecting that pin through.

hamandcheese 2y ago
On the other hand, I recently fried a motherboard while trying to probe it with a multimeter. My fat fingers shorted out two adjacent pins, causing a loud spark and magic smoke.
Arcanum-XIII 2y ago
Not all DMM have probe small enough to connect to the lane. If it's even possible. What's more, you need to know where to put it, which can be daunting without the proper knowledge. Switching hardware is easier, faster and often the best solution in those case. Finding hardware fault is hard. Tracing it is even harder.
- userbinator 2y ago
  I think there's something wrong with your DMM probes if you can't measure the ATX power connector with them.

bunnie 2y ago

Reading the thread it looks like the issue is leakage power on the internal 3.3v line. When the system is off 1.9v is still present. This is not uncommon, although 1.9v is a bit high. A lot of laptops have explicit active pull downs on power supplies to clamp them to zero when power is off to ensure peripherals are not accidentally powered on by stray leakage (because laptops are extremely low power by design and there is not enough stray leakage to bring the power lines down in a sleep state). My guess is main boards might not have this feature because normally there is enough off state loading that it takes care of itself. however maybe in this case the loading is not enough. A dirty fix could be to just put a static load on the 3.3v line to ground. I'd start with a 1/4w resistor around 100 ohms and just stick it from 3.3v to ground to see if that does not soak up the stray current. if it works just leave it, it's about 0.1 watts of static power and no big deal for a non portable setup. The larger picture is that the controller on the nvme might not hit its power on reset condition because it may be rated to run at 1.8v (just a guess), so 3.3v is not going low enough for the controller to perceive the system has been power cycled. Usually a supplemental power monitor is needed in those cases to ensure a reset is generated in case of leakage problems like this.

starslab 2y ago
Hi! I'm the OP from the Level1Techs thread. That HDMI power has some grunt behind it. During power-off state with that 1.90v phantom voltage, I put a 48ohm resistor between 3V3 and ground, the phantom voltage only dropped to 1.80v, and the SSD still didn't work when I powered the machine back on.
- oneplane 2y ago
  Depending on the PMIC and the SSD DC conditioning, even 1.2v might be enough for it to brownout/latchup without self-resetting. (or it might power up the PHY partially or in a bad state and never link up) Try more resistors in series? (or just a bigger one if you have any -- scratch that we needed smaller ;-) ).
  - starslab 2y ago

okanat 2y ago

I bought the same model SSD for my Thinkpad P1 last month and saw the exact issue. I had to return it because it was breaking the NVMe detection completely. So it wasn't a broken unit but a design issue after all?

BearOso 2y ago

Since we're talking SSDs, I wonder if we could get some attention to the Phison E18 degradation issue [1]. Only one manufacturer, Kingston, has put out firmware containing Phison's fix, while the others just ignore it. A bunch of these drives with this controller were on sale during black Friday, so a lot more people are going to have problems in a month or so. 1. https://www.reddit.com/r/pcmasterrace/comments/1f1piwf/psa_phison_e18_based_ssd_owners_update_your/ https://www.reddit.com/r/pcmasterrace/comments/1f1piwf/psa_p...

userbinator 2y ago
That sounds like NAND degradation (retention failures) which can only be partially worked around in firmware (and causing more write cycles on already-marginal QLC). Unfortunately the real solution is "use better NAND", which is unlikely to happen unless enough people demand it.
- ciupicri 2y ago
  Kingston KC3000 supposedly uses Micron 176L TLC memory [1]. The Seagate Firecuda 530 datasheet clearly says "Built with a Seagate-validated E18 controller and the latest 3D TLC NAND". A review is more precise: "Phison PS5018-E18" & "Micron B47R 176-layer 3D TLC NAND" [2]. [1]: https://www.tomshardware.com/reviews/kingston-kc3000-m2-ssd-review https://www.tomshardware.com/reviews/kingston-kc3000-m2-ssd-... [2]: https://www.kitguru.net/components/ssd-drives/simon-crisp/seagate-firecuda-530-2tb-ssd-review/all/1/ https://www.kitguru.net/components/ssd-drives/simon-crisp/se...
  - userbinator 2y ago
    B47R is indeed TLC, rated for only 1000 cycles (and 35k in SLC mode, at 1/3 the capacity.) There's also the question of whether this is "true" Micron NAND, or SpecTek which is basically Micron's rejects (and rated for even fewer cycles; only 300 in the case of their B16A.)
ciupicri 2y ago
Kingston doesn't seem to offer any support for Linux, so their new firmware is virtually non-existent to me. Why can't I just download the firmware and use standard nvme-cli tools to update the SSD, beats me. If Seagate (which by the way uses Phison E18 too) can do it, so can Kingston, Samsung, Crucial, Western Digital and many others. Even better would be use Linux Vendor Firmware Service (https://fwupd.org/ https://fwupd.org/).

amelius 2y ago

I have a similar problem with a Jetson board. If I turn off the power long enough (one night) and then turn it on, the only PCI card is not recognized and I have to power-cycle it to get it running.

structural 2y ago
Mind sharing what board/Jetson module you've seen this on? I've seen this exact symptom very intermittently on a custom board and we've wondered for a long time if was an issue with a specific type of module (or manufacturing lot of modules).
- amelius 2y ago
  This one: https://www.avermedia.com/professional/product-detail/D315%205G https://www.avermedia.com/professional/product-detail/D315%2... My startup logic now power-cycles it until the PCI board is recognized; it works, but it's not a great solution.
  - structural 2y ago
    Interesting, we're using a completely different module (Xavier NX). And the same, disgustingly hacky, fix, of forcing a reset until it works.
    - amelius 2y ago
      I also run these commands: echo 1 > /sys/bus/pci/rescan sleep 1 Sometimes it brings the PCI card back, so I just run this as part of my boot sequence.

deleted 2y ago

[deleted]

undertaken 2y ago

anecdotal/weird computer experience: I have a rebadged Tongfang laptop (NB02 GMxRGxx w/ Ryzen 9) and upgraded it shortly after purchase. The machine arrived with lower capacity Samsung SODIMMs. Swapped in 64GB of Crucial DDR5. Shortly afterwards the machine became instable to the point of RMA. Kernel logs clogged with all sorts of panics related to NVMe, PCIe, and filesystem. Freezing. Reboots. Spent hours diagnosing it. Many permutations of kernel command line arguments; pcie, acpi tables, iommu. All for naught. The machine passed memtest86 / memtest86+ with flying colors. bonnie++ absolutely trashed it. reliably. Occasionally the NVMe drives fell off the pci bus and it wouldn't boot until I disable the slot in bios, power cycle, then re-enable the slot. Fast forward to me getting fed up with a dysfunctional system, I attmepted RMA and gave them the rundown of all the weird seemingly chipset related failures. They pushed back with "Try our RAM again." I nearly had an aneurysm when everything was stable again. After thanking the support staff profusely I bought larger capacity Samsung DIMMs in the same chip family. Still running flawlessly after almost a year. Maybe try new RAM for yucks? ;)

bb88 2y ago

So these guys [1] mention something similar where HDMI from a TV is backfeeding 40-50 volts into a cable box. This could be because of many things from electrical outlet wiring to power supply issues on the monitor to a bad component on the monitor giving a high voltage, or the monitor is badly grounded, etc, etc. I read the original thread but it doesn't look like you've measured the voltage at the HDMI port wrt motherboard ground. I think we're assuming it's 5 volts, but it could be higher, and it could have shorted (or weakened) a component on your motherboard. And that would explain why a 100 ohm resistor didn't give a meaningful voltage drop. If you need an isolation solution, Amazon sells a 50ft fiber optic one way HDMI cable [2]. The thing I don't know is if there's any actual copper to provide power over the link. There are other options which transmit the HDMI signal over pure multimode fiber as well [3]. Or you can go with a DP KVM, since you're on L1T, they sell a few DP models. I have one I purchased from L1T, and I like it a lot. Definitely though I would check out the outlets to make sure they were wired correctly. Incorrectly wired outlets because someone tried to DIY it in the US is absolutely a problem. [1] https://www.avsforum.com/threads/hdmi-cable-backfeeding-voltage-from-tv-to-cable-box.1345211/ https://www.avsforum.com/threads/hdmi-cable-backfeeding-volt... [2] https://www.amazon.com/HDMI-FURUI-HDCP2-2-18Gbps-Subsampling/dp/B077D37MT3?th=1 https://www.amazon.com/HDMI-FURUI-HDCP2-2-18Gbps-Subsampling... [3] https://fibercommand.com/products/8k-fiber-plugs?gQT=1 https://fibercommand.com/products/8k-fiber-plugs?gQT=1

starslab 2y ago
I already own one of those fiber-hdmi cables. Brilliant, but sometimes doesn't interoperate with DVI devices using passive DVI -> HDMI adapters. I've no idea if it has any copper conductors for HDMI power, though one end is labelled for the source and one for the display, suggesting that however it's designed it's not bi-directional. I'd love a DisplayPort KVM, but not every device that comes across my bench has a DisplayPort output, and those few that have DisplayPort but no HDMI can be accommodated with one of those commodity DisplayPort -> HDMI adapters. This situation is actually getting worse over time, not better, as many modern devices and laptops are skipping DisplayPort in favor of USB-c alt-mode. This issue has actually been going on through a monitor change on my testbench. It has happened with a Samsung SyncMaster 204T though my KVM switch, an HP ZR24w through my KVM switch, and the ZR24w directly connected. I don't think this is an issue with the rest of my equipment. This electrical was done about 15 years ago, by a ticketed electrician. One of those $5 plug testers indicates all is well, and I have no reason to believe there's any issue here. By almost pure coincidence, I have an MSI PRO X870-P motherboard on order. I'm looking forward to seeing if this same 3V3 leakage issue is present on this board too.