8 ms·
Use this kernel parameter in your kiosk
- oakwhiz 4y agoUsing a hardware watchdog is an alternative and it covers the additional case where the system hangs without panicking.
- jerrysievert 4y agoYes thank you. As someone who built these for a living I can definitely say that this is a really good solution.
- foobiekr 4y agoJust gotta be careful about how you implement it. The real problem to solve is to detect when the application is. It making forward progress. I’ve seen this implemented in awful, stupid ways (like spinning off a thread to stroke the watchdog - borderline pointless).
- ilyt 4y agoThat's generally pretty hard problem, to find a point where it triggers where app is unresponsible but not when say it is running something CPU intensive and just lags for few seconds
- kqr 4y agoIsn't that in its most general form exactly the halting problem?
- foobiekr 4y agoThe halting problem is for the general case. If you design the application to facilitate it you can demonstrate forward progress in a pretty robust way. Is it perfect? No. But you can do much, much better than “is the kernel up.”
- londons_explore 4y agoFor network connected things (which aren't designed to work offline), I like to implement a network watchdog poker... Ie. every 30 seconds, contact an update server server and, if successful, poke the hardware watchdog. That means every device in your network will either be rebooting, or correctly talking to your server with the latest version of your software. There are no other stuck/error states to consider. I also implement reboot-with-no-screen-flicker, so that I can display a basic company logo so things look respectable while there is an outage and everything is bootlooping. Some brands of screen let you upload an image to display if there is no valid signal, which is an easy way to implement this. Also worth doing a load test of your server to verify that it can withstand all your devices rebooting simultaneously and you won't suffer the thundering herd problem.
- throwaway9870 4y agoI think you are conflating hardware watchdogs and software timers. The hardware watchdog should be to catch cases where the system is no longer running your software (or the kernel or the watchdog daemon). Timeouts for software catch things like slow procedures, etc.
- foobiekr 4y agoNot at all. In most embedded devices, the point of the software on the device is some specific function which is something more than “the kernel is running” or “the kernel is scheduling the zero-io watchdog process.” You want to actually pick up the case where the kernel is up but, for example, your process is all but dead because your storage driver has wedged. The goal is to prove forward progress and the best way to do that is to come as close to proving that your userplane SW is actually working and not dead, or worse, half dead.
- noodlesUK 4y agoCould you elaborate a little about how these are implemented?
- jesse_cureton 4y agoPreface: my knowledge here is on ARM, particularly baremetal, but also embedded Linux. No idea about Windows or x86. Generally there’s a hardware watchdog implemented as a counter/timer in the processor. It can have a predefined or configurable period. It counts down, and if it times out then it initiates a hardware reset of the processor. You can ensure your software/OS is always at least executing code by having a task (in-kernel on Linux, or an RTOS task, or just in your main event loop on baremetal) that resets that timer. Then, if your code stops resetting that timer, it expires and resets the processor.
- noodlesUK 4y agoIs there some way of accessing this from user space on Linux?
- amluto 4y agohttps://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api... Many x86 systems have a built in hardware watchdog.
- jesse_cureton 4y agoYes! There’s an ioctl interface for managing the watchdog, and a character device at /dev/watchdog. The kernel docs[1] are a decent jumping off point to learn more. Upon reading these I did realize on Linux it’s implemented as a kernel device, but it’s usually a userspace task that has to notify the kernel watchdog interface to actually kick the timer. This makes sense, since userspace being functional is probably what you really care about. [1] https://www.kernel.org/doc/html/latest/watchdog/watchdog-api.html https://www.kernel.org/doc/html/latest/watchdog/watchdog-api...
- c7DJTLrn 4y agoGreat point. If the hardware is available I'd use both.
- dtgriscom 4y agoI have a network-controllable outlet switch with a timeout feature. You can configure each outlet to ping a specific IP address; if the address doesn't respond after X seconds then the outlet will be power-cycled.
- mananaysiempre 4y agoDid you get that as fabulously expensive datacenter gear, or were you able to squeeze it out of a consumer “smart outlet”? I’ve been looking for a thing like that some weeks ago (for reasons that are now moot), and the easiest option seemed to be to buy a handful of the vendor-app-locked whitelabel crap and see which can be reflashed with open source firmware. Which is not as easy as I’d wish.
- yencabulator 4y agoShelly relays are known to be easy to control via Wifi. Off the shelf, you'd need to run the actual ping+decide logic elsewhere, though you could re-flash the firmware with custom logic too... https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtocol/ https://shelly-api-docs.shelly.cloud/gen2/General/RPCProtoco... https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6FB65C0DCBCA https://www.amazon.com/stores/page/4E39C18F-DCA3-4726-A8A7-6... Plus, they're also easy to reflash with other firmwares (e.g. Tasmota, ESPHome).
- oakwhiz 4y agoI want one of these in "plug" form factor that uses a normally-closed relay rather than normally-open. The reason being that the internal power supply and microcontroller could fail by shutting down, preventing things from working at all. There's nothing that can be done about the microcontroller failing while holding the circuit open, though, but power supply failure has been a reoccurring but infrequent thing for me lately and it would be nice to cover that case well. Edit: For small do-it-yourself computer clusters you probably want it to fail normally-open, this would be for non-redundant stuff
- metadat 4y ago
- throwaway9870 4y agoExactly. Every Intel machine has a hardware watchdog built into the ICH chipset. Make sure it is turned on.
- smithcoin 4y agoAny good software solutions that can do the equivalent for windows? I have systems that end up hanging at black error screen due to some kind of system restore/failure. The only way I have found to reboot them is with human intervention using a keyboard.
- dataflow 4y agoDoes this setting not work? https://cdn.mos.cms.futurecdn.net/wj8raAJEtvzSR3pav36KYV-1200-80.png https://cdn.mos.cms.futurecdn.net/wj8raAJEtvzSR3pav36KYV-120...
- lrvick 4y ago
- izacus 4y agoThis statement has no connection to reality.
- MrStonedOne 4y ago
- wardedVibe 4y agoThere's a comment after yours that says you can do it via: wmic RecoverOS set AutoReboot = True
- 13of40 4y agoIt might be worth noting that a BSOD is exactly the Windows equivalent of a kernel panic. Also, this apparently toggles the option in Windows: wmic RecoverOS set AutoReboot = True
- suprjami 4y agoAddendum: If you are using something like kdump to capture a vmcore of those kernel panics, then ensure you set `kernel.panic` large enough that the vmcore can actually be written to the dump target. Setting it to 60 seconds is not much use when it takes 90 seconds to capture the core file.
- mkarliner 4y agoYou are a god. Thank you.
- JonChesterfield 4y agoHow robust is either the proposed: `panic=60` or the Linux API described in sibling comments? I run into kernel panics a lot and usually power cycle the machine. In particular for this to reboot after the kernel dies it seems to need real hardware support. Ping this register every N seconds or I reboot style. Is that what we're working with here, vs some other part of the kernel which might survive part of it dying? > docs here suggest the system can get in a state which ignores the watchdog, https://mjmwired.net/kernel/Documentation/nmi_watchdog.txt https://mjmwired.net/kernel/Documentation/nmi_watchdog.txt > thread here suggests a bug in the bios can stop a watchdog working https://community.intel.com/t5/Intel-NUCs/Issue-with-Linux-watchdog-timer-where-system-did-not-restart/td-p/1243160 https://community.intel.com/t5/Intel-NUCs/Issue-with-Linux-w...
- ilyt 4y ago> I run into kernel panics a lot Where ? The most I saw was when trying to boot something untypical. > Is that what we're working with here, vs some other part of the kernel which might survive part of it dying? I'd imagine it's same path for panic. There is probably still ways to crash (especially with hardware faults) to get around it so it isn't replacement for watchdog but it looks like an improvement > docs here suggest the system can get in a state which ignores the watchdog, https://mjmwired.net/kernel/Documentation/nmi_watchdog.txt https://mjmwired.net/kernel/Documentation/nmi_watchdog.txt I'd like to say you need "proper" external watchdog like IPMI based one on server but * They still need to get thru the boot for kernel to activate them in the first place. I think I saw some servers that had some extra protection to crash on boot but definitely not anything common * I've seen weird bugs around those too. Like older IBM servers could trigger watchdog when you synchronized IPMI time You really kinda need 2 watchdog loops, long one (say 5-10 minutes) that runs from start catering to "something crashed during boot", and short one (few seconds to maybe a minute) that gets activated once kernel and OS is up and running
- someweirdperson 4y ago> If you’re building a kiosk, it’s hopefully obvious to avoid having any read-write mounted filesystems at all costs. I wish there was support for this out of the box in linux distributions. Either by providing a (sub)set of software that doesn't need to write, and/or adding an overlay. Even for rasperries which are often used in a way where they would benefit it requires some work to set things up. Having that kernel parameter set automatically when in that mode would be nice, too. Or maybe I'm just not aware that distributions for this specific use-case exist.
- c7DJTLrn 4y agoTiny Core and Alpine run in memory as far as I remember. You could also build a custom image of say, Ubuntu, with a tmpfs overlay on top.
- dinom 4y agoI've been meaning to try Ubuntu's overlayroot package but since I tend to stick with Debian I haven't gotten around to it. Sounds interesting though... https://blog.dustinkirkland.com/2012/08/introducing-overlayroot-overlayfs.html https://blog.dustinkirkland.com/2012/08/introducing-overlayr...
- LargoLasskhyfv 4y agoLook into AntiX and/or MX-Linux. They both use the same toolset, are Debian-based, while AntiX is more 'spartan', and MX more full desktop. Both can be remastered into oblivion with their delivered tools, and installed in all sorts of ways, including running from RAM. Running as full XFCE Desktop with FF, occassional updates except kernel, all in RAM, with quick logout and telinit 1 as root from console, then back to telinit 5(with autologin back into X): $ uptime 09:44:35 up 160 days, 46 min, 3 users, load average: 0.02, 0.08, 0.31 Most hassle-free Linux-distro I've ever had, while being compact, but fully usable for my needs. There may be others, but I'm tired of trying. Edit: Obviously not running with this kernel-parameter or watchdog, because I don't see panics here. But could be easily integrated, because Debian. OFC you'd need to have a remastered and prepared(to your needs) boot-device available after a reboot, while running from RAM. Could even be an USB-keychain. Anyways, they have all the tools to do this within a few clicks from X and a few function key-presses and menu choices during first boot. Works for me, YMMV. https://antixlinux.com/ https://antixlinux.com/ https://mxlinux.org/ https://mxlinux.org/
- jesprenj 4y agoPerhaps the title could be updated so it isn't as clickbaity: Use kernel parameter panic=60 in your kiosk
- stingraycharles 4y agoI don’t think the title is too clickbaity — it would do the article injustice compared to all the actual clickbait content online.
- an_ko 4y agoIt's a matter of degrees. The title here still gives less information than it could have in the same amount of space, forcing more people to read the article to find out, than would otherwise have.
- c7DJTLrn 4y agoSorry, I didn't mean for it to be clickbaity. I usually refrain from titles like this.
- mikedelago 4y agoI don't think it's too bad. Kiosks aren't overly prevalent, so unlike particularly egregious clickbait examples (Dentists hate this one simple trick for perfect teeth, etc), I had less stake in the result. I was still curious about the content (which is how a title is supposed to work, right?) so I clicked it anyway, and learned about a kernel parameter I wasn't aware of. It's a good article, thanks for writing it.
- jesprenj 4y agoI did not mean to negatively judge your post. After all the title does explain that it's about a kernel parameter, which is probably enough information; telling which specific one it is could even possibly be less estetic.
- dezgeg 4y agoOn the kernel side please use kernel.sysrq=0 also to disable magic sysrq (usually under key combo Alt+PrintScreen).
- boramalper 4y agoWhy?
- dezgeg 4y agoSo that people cannot press these key combinations behind Alt+PrintScreen: https://en.wikipedia.org/wiki/Magic_SysRq_key https://en.wikipedia.org/wiki/Magic_SysRq_key (assuming there is a keyboard, of course).
- boramalper 4y agoAh, I assumed touchscreen-only right away.
- LinuxBender 4y agoAnother useful setting one could potentially use in conjunction with the panic timer is vm.panic_on_oom = 2 In my experience, most systems get into a precarious state when not engineered to only use a percentage of available memory, factoring in the apps that spike memory using when forking child processes. This is most useful on NFS diskless machines, any machine that is managed as cattle vs kittens i.e. treating them like disposable VM's, dev/qa machines and especially machines used for application load testing. One could even use in production after a significant amount of testing but it requires a few more settings and having automation doing a little bit of math as OOM/vmscan often picks the wrong processes to terminate and leaves the machine in a broken and sticky state and increases downtime impact on SLA. The probability of an OOM panic can be reduced by using: vm.overcommit_ratio = 0 The default of 50 or 50%, 1.5X ram+swap allows the kernel to permit an application to spawn even when there is no memory available. This is handy on development systems but can lead to problems when automation does not properly calculate how much memory is available and permits an application to request too much memory as the kernel does not protect it's own future memory requirements. One must then use a little bit of math based on the total memory installed in the system to calculate the values of vm.admin_reserve_kbytes, vm.user_reserve_kbytes, and most importantly vm.min_free_kbytes to prevent applications from grabbing too much memory. This also requires adjusting automation to deal with grabbing enough information when an application does not spawn correctly. This can be further improved by containerizing applications or using cgroup memory constraints when feasible. A percentage of memory must remain unallocated by applications for the kernel to use. If using idrac/ilo/ilom cards, some of them have a feature to grab screenshots and will save the screenshots that led up to the panic, though will not always get the right moments. [Edit] I should also add that the math calculations assume all deployment teams and all of their automation tools are aware of one another and math-all-the-things. e.g. production code, monitoring tools, analytic teams all add their numbers together plus overhead. If this is a Kiosk then maybe that is just one team.
- jgerrish 4y agoBefore deploying this, do some analysis of your system and failure modes. And document it. For example, do you have read-write filesystems or swap? If it gets corrupted, will a reboot cycle with fsck corrupt it further, preventing post-mortem forensics? This may have security implications too. What about network access to a shared system? Will a power outage cause a stampede on your servers? Should you stagger that 60 value? Just a few questions to ask.
- jgerrish 4y agoAnd exponential backoff can help in a larger context. And coordination and yada yada. The bigger thing is asking yourself these questions should reduce the pain. When it doesn't, what are we doing?