3 ms·

+1. It's also my recent experience of running FreeBSD on AMD Ryzen, several features are missing. And so far this is a list of problems I've encountered. 1. No

+1. It's also my recent experience of running FreeBSD on AMD Ryzen, several features are missing. And so far this is a list of problems I've encountered. 1. No ECC driver for AMD Ryzen, and no ECC support in memory management. While you don't need any OS support to use ECC, it's much better to have it. * Linux has an ECC kernel driver for AMD CPUs that accurately reports the ECC status, this is extremely helpful given the uncertainty of ECC in consumer-grade hardware. On FreeBSD, it's not possible to tell whether ECC is enabled due to the lack of driver. Also, if ECC is not initialized properly by the firmware, the Linux ECC driver will also detect it and try reenabling ECC and memory scrubbing. On FreeBSD, better to check whether your BIOS is broken. * When an ECC error occurs, a Machine Check Exception is generated by the CPU. On Linux, it will be correctly decoded and recorded. But on FreeBSD, so far there's no decoder, leaving you a mysterious MCA error in dmesg. For example, a correctable DRAM ECC error will be reported as "L3 cache error" (which made many people to falsely believe that Ryzen's ECC was not working on FreeBSD). * On Linux, if an unrecoverable multibit ECC error is detected in userspace memory, the process will be killed by a SIGBUS (my username checks out?). If the error occurs in kernel memory, it triggers a kernel panic. On FreeBSD, as far as I know, none of the features is supported by the kernel, it increases the risk of data corruption, and it's not a Ryzen or MCE-specific problem. Also, Linux also supports memory offlining, which can be used to temporarily remove bad memory pages from the system after an ECC error, which is not found in FreeBSD. Fortunately the ECC-enabled flag is easy to check (I plan to make a contribution to the FreeBSD kernel), and a MCE decoder in userspace should also be easy. But implementing the rest is not trivial, the lack of ECC support in memory management is a problem. It's unfortunately that these features are not supported in a server-grade OS like FreeBSD (I guess FreeBSD runs on "real" servers, which implements all of those ECC features in the management firmware, so "real" FreeBSD users don't really need them...) 2. No USB serial system console and netconsole support. On FreeBSD, although tty login via USB serial is supported (just run a getty on it), it seems that there's no way to redirect all dmesg to a serial port (perhaps someone can correct me), FreeBSD's comconsole requires a real physical serial port, not USB, on the other hand Linux's CONFIG_USB_SERIAL_CONSOLE supports USB natively. On Linux, there's also a kernel-space netconsole that redirects all demsg via UDP/IP, this feature is not found in FreeBSD although there was an attempt in 2012.

toast0 6y ago
I ran thousands of FreeBSD servers, with ECC, and the memory management layer features would have been nice, but unrecoverable failures were rare enough that it wasn't a big deal; anyway our application ran as a single OS process using almost all the ram, so 95% chance the faulty address was OS or our application. Offlining would have been nice though; had a couple faulty addresses that were busy, so all the MCE interrupts made the system really slow, and caused a ton of application congestion. For USB serial console, I think part of the issue there is that the serial console can interface with the kernel debugger, which might be tricky if it's USB (but I guess USB keyboards probably work in the kernel debugger, so maybe there's a chance?). Thanks for working on Ryzen ECC. I don't work with a server farm anymore, and I'm too cheap to buy ECC for home use, but someday I might use that support.
adrian_b 6y ago
Yes, unfortunately FreeBSD has much better support for Intel CPUs and for Intel networking, than for alternatives. On servers with Intel CPUs, FreeBSD warned me promptly, by e-mail, about some corrected ECC errors. The errors appeared suddenly after many years of use and they kept recurring, pointing to a probable failure in the near future, but being promptly notified, I have replaced the offending DIMM and all went back to normal. So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel. There are also examples of hardware for which FreeBSD has better support than Linux, but those belong to classes of hardware less likely to be encountered by non-professional users. For example FreeBSD has better support for magnetic tapes.
- segfaultbuserr 6y ago
  > So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel. This is not just an AMD problem. For example, as I already said, there is no support of unrecoverable ECC errors handling in the FreeBSD memory management subsystem for any CPU, including Intel. On Linux, the kernel has first-class ECC support via the Error Detection And Correction subsystem (EDAC). EDAC can decide to kill a process, to trigger a kernel panic, or even to isolate the faulty memory based on the location of the memory error. On FreeBSD, the only thing that handles unrecoverable ECC errors is the userspace utility sysutils/mcelog [0], the best thing it could do after an unrecoverable error is sending you an email, at which point data corruption may have already occurred [1]. Usually correctable errors occur before unrecoverable ones, so data corruption could be prevented if it's fixed soon enough, but if you get "lucky"... And unlike writing device drivers, I don't think most people (including myself) is qualified to mess with the memory management code in the kernel. But I have a plan to do something about it, for example, there's nothing stops me from writing a simple script to decode the MCE log and kill the kernel from the userspace. And given how rare unrecoverable ECC errors are, even without graceful handling (like killing userspace process or runtime memory offlining), it should solve my concern of data integrity on FreeBSD. [0] But corrections are welcomed. I suspect FreeBSD will also halt the kernel on Intel platforms after a fatal MCE due to ECC error, but I didn't find any information and came to a negative conclusion. [1] I guess it's not really a problem for "real" FreeBSD users, on "real" servers, BIOS can kill the system after an unrecoverable ECC, which probably explains the lack of any unrecoverable error handling in FreeBSD.

toast0 6y ago

I ran thousands of FreeBSD servers, with ECC, and the memory management layer features would have been nice, but unrecoverable failures were rare enough that it wasn't a big deal; anyway our application ran as a single OS process using almost all the ram, so 95% chance the faulty address was OS or our application. Offlining would have been nice though; had a couple faulty addresses that were busy, so all the MCE interrupts made the system really slow, and caused a ton of application congestion. For USB serial console, I think part of the issue there is that the serial console can interface with the kernel debugger, which might be tricky if it's USB (but I guess USB keyboards probably work in the kernel debugger, so maybe there's a chance?). Thanks for working on Ryzen ECC. I don't work with a server farm anymore, and I'm too cheap to buy ECC for home use, but someday I might use that support.

adrian_b 6y ago

Yes, unfortunately FreeBSD has much better support for Intel CPUs and for Intel networking, than for alternatives. On servers with Intel CPUs, FreeBSD warned me promptly, by e-mail, about some corrected ECC errors. The errors appeared suddenly after many years of use and they kept recurring, pointing to a probable failure in the near future, but being promptly notified, I have replaced the offending DIMM and all went back to normal. So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel. There are also examples of hardware for which FreeBSD has better support than Linux, but those belong to classes of hardware less likely to be encountered by non-professional users. For example FreeBSD has better support for magnetic tapes.

segfaultbuserr 6y ago
> So yes, you are right, for FreeBSD it is better to stick to Intel, unless you have time to change yourself the sources, to add the missing hardware support, which is somewhat easier than attempting to make changes in the Linux kernel. This is not just an AMD problem. For example, as I already said, there is no support of unrecoverable ECC errors handling in the FreeBSD memory management subsystem for any CPU, including Intel. On Linux, the kernel has first-class ECC support via the Error Detection And Correction subsystem (EDAC). EDAC can decide to kill a process, to trigger a kernel panic, or even to isolate the faulty memory based on the location of the memory error. On FreeBSD, the only thing that handles unrecoverable ECC errors is the userspace utility sysutils/mcelog [0], the best thing it could do after an unrecoverable error is sending you an email, at which point data corruption may have already occurred [1]. Usually correctable errors occur before unrecoverable ones, so data corruption could be prevented if it's fixed soon enough, but if you get "lucky"... And unlike writing device drivers, I don't think most people (including myself) is qualified to mess with the memory management code in the kernel. But I have a plan to do something about it, for example, there's nothing stops me from writing a simple script to decode the MCE log and kill the kernel from the userspace. And given how rare unrecoverable ECC errors are, even without graceful handling (like killing userspace process or runtime memory offlining), it should solve my concern of data integrity on FreeBSD. [0] But corrections are welcomed. I suspect FreeBSD will also halt the kernel on Intel platforms after a fatal MCE due to ECC error, but I didn't find any information and came to a negative conclusion. [1] I guess it's not really a problem for "real" FreeBSD users, on "real" servers, BIOS can kill the system after an unrecoverable ECC, which probably explains the lack of any unrecoverable error handling in FreeBSD.