7 ms·

I've grown to love systemd. It solves a lot of my problems with init scripts, particularly the ones involving environment / PATH at boot time. I've made init

I've grown to love systemd. It solves a lot of my problems with init scripts, particularly the ones involving environment / PATH at boot time. I've made init scripts for things before which work when they are invoked manually, but not at boot time because PATH was different. With systemd I am confident that if it works through systemctl it will work at boot. Maybe I am not tuning Linux appropriately, but I have been in situations where a Linux system is overwhelmed / overloaded and I am unable to ssh to it. I have never had that experience with FreeBSD -- somehow ssh is always quick and responsive even if the OS is out of memory. Most of the systems that I deal with are Linux, but I still have a few FreeBSD systems around and they are extraordinarily stable.

eptcyka 5y ago
I think there's something pathological in the I/O subsystems on Linux that make it a bad experience - I've experienced horrible U/I latencies on desktop and server settings with Linux when there was any kind of I/O load, and found FreeBSD to always be a breath of fresh and quick air in this regard.
- pedrocr 5y ago
  There are definitely pathological cases around. Here's an 8 year old bug, still valid, that's a common extreme slowdown when copying to/from a USB drive: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993
  - rogers18445 5y ago
    It sometimes manifests as audio stuttering (the kind which sounds like unstable sample rate) which suggests a lot of context switching, perhaps a scheduler problem. Had this problem intermittently and seemingly at random for many years. Sometimes would happen all day, then nothing for months. Quite Bizarre.
- skeptical1 5y ago
  For a long time after creating my own Linux distro, I had the same kind of problems also. It turns out the Linux kernel is horribly tuned by default. After a number of tweaks and adjustments, I finally got all those bugs ironed out. Now my (four core) desktop is perfectly smooth and responsive under all loads, even playing video and running multiple builds while copying files around. Here's the important parts of what I've done: * set disk i/o schedulers to 'bfd' for spinning drives and 'deadline' for solid state, and 'none' for nvme, by creating a file in /etc/udev/rules.d . kernel must have deadline and bfd schedulers compiled in. * turned on SCSI block multiqueue in kernel config. requires kernel command line option scsi_mod.use_blk_mq=1 to actually enable it. this helped, but did not completely cure the disk i/o problem. * patched kernel source file ./block/blk-mq-sched.c to hard limit number of queued block device requests to 2, instead of default which is like 32. this absolutely cured the problem. no more disk i/o dragging the system down. doesn't seem to have a major effect on throughput. * kernel is configured for full preemption, with 1000hz timer frequency. * for architectures which will boot using the muqss cpu scheduler patch, i enable that with a 100hz timer freq instead. * overcommit is disabled, as well as swap, and i use earlyoom to ensure process destruction proceeds in a controlled manner in event of memory exhaustion. That's the bulk of it. No real magic involved; just un-fuck-ifying the default kernel config, which is garbage even for server use IMO. (This is on a 4.x kernel btw, and I have no plans to downgrade to the 5.x series.) The fact that these some or all of these tweaks aren't done by default would seem, along with other evidence, to support my belief that Linux is actively being sabotaged by people who don't want it to succeed.
  - z3t4 5y ago
    Software have a hard time keeping up with hardware architecture, mostly because of backwards compatibility... Imagine running a restaurant, normally you can take 32 orders and have the customer sit and wait. One day you get new chefs that can make food 100x faster, but now you can only take one or two orders before the chefs have the food ready and you have to give it to the customer that ordered it. So despite the chefs being 100x faster it now takes much longer to place an order, and the waiting line can grow long with impatient customers.
travisgriggs 5y ago
> I've grown to love systemd. I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum. The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome. With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.
- fullstop 5y ago
  The RC system was simple, I'll give you that. I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary. The RC system was simple and also full of exceptions and corner cases.
  - tluyben2 5y ago
    What is the use case for that? I never had these needs on servers. Maybe you are doing something very different from me, so I am curious. I use RC on servers and systemd on clients; things run robustly for 15+ years on many servers for me (with security updates included).
    - fullstop 5y ago
      Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process! This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill. This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance. If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on. It's all very error prone and the process for each daemon is often different.
tored 5y ago
On my personal linux workstation I experienced multiple times that I couldn't ssh into it after the desktop froze, it was due to too small swap file.
- themodelplumber 5y ago
  That's interesting, what proportion are you using to determine swap size? For example 1x RAM, 2x RAM, etc.
  - tored 5y ago
    I run at 1xRAM. Seems that my problems went away after that. Ubuntu by default sets it to 2GB even if you have something like 16GB RAM. Swap file can then fill up pretty quickly.
    - themodelplumber 5y ago
      > Ubuntu by default Yep, one of my systems got this treatment. It's annoying because OOM has been an issue ever since. I'm playing with EarlyOOM and trying to remember if it'll be a huge pain to resize the partition for more swap space. Thanks for your reply.
      - tored 5y ago
        Thanks for the tip on EarlyOOM. Swap file is at least easier to handle than swap partition. Some run their Linux without swap at all, however I think that is better for a server setup where you run a small set of specific set of binaries, thus you know your load. That is not the case on a desktop where you running a variable set of binaries depending on your task. I’m somewhat surprised how bad this works on Linux, I don’t think I ever experienced a swap problem on Windows (not even in the 9x days), I guess this is because "Year of the Linux desktop" never happened, Linux is primarily a server and embedded OS, not a desktop OS.
        deleted 5y ago
        [deleted]
saurik 5y ago
If it makes you feel any better, I have been using Linux for like 25-30 years now and I still experience "sigh. and now I can't SSH into the box to kill the thing I shouldn't have started".

eptcyka 5y ago

I think there's something pathological in the I/O subsystems on Linux that make it a bad experience - I've experienced horrible U/I latencies on desktop and server settings with Linux when there was any kind of I/O load, and found FreeBSD to always be a breath of fresh and quick air in this regard.

pedrocr 5y ago
There are definitely pathological cases around. Here's an 8 year old bug, still valid, that's a common extreme slowdown when copying to/from a USB drive: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993 https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1208993
- rogers18445 5y ago
  It sometimes manifests as audio stuttering (the kind which sounds like unstable sample rate) which suggests a lot of context switching, perhaps a scheduler problem. Had this problem intermittently and seemingly at random for many years. Sometimes would happen all day, then nothing for months. Quite Bizarre.
skeptical1 5y ago
For a long time after creating my own Linux distro, I had the same kind of problems also. It turns out the Linux kernel is horribly tuned by default. After a number of tweaks and adjustments, I finally got all those bugs ironed out. Now my (four core) desktop is perfectly smooth and responsive under all loads, even playing video and running multiple builds while copying files around. Here's the important parts of what I've done: * set disk i/o schedulers to 'bfd' for spinning drives and 'deadline' for solid state, and 'none' for nvme, by creating a file in /etc/udev/rules.d . kernel must have deadline and bfd schedulers compiled in. * turned on SCSI block multiqueue in kernel config. requires kernel command line option scsi_mod.use_blk_mq=1 to actually enable it. this helped, but did not completely cure the disk i/o problem. * patched kernel source file ./block/blk-mq-sched.c to hard limit number of queued block device requests to 2, instead of default which is like 32. this absolutely cured the problem. no more disk i/o dragging the system down. doesn't seem to have a major effect on throughput. * kernel is configured for full preemption, with 1000hz timer frequency. * for architectures which will boot using the muqss cpu scheduler patch, i enable that with a 100hz timer freq instead. * overcommit is disabled, as well as swap, and i use earlyoom to ensure process destruction proceeds in a controlled manner in event of memory exhaustion. That's the bulk of it. No real magic involved; just un-fuck-ifying the default kernel config, which is garbage even for server use IMO. (This is on a 4.x kernel btw, and I have no plans to downgrade to the 5.x series.) The fact that these some or all of these tweaks aren't done by default would seem, along with other evidence, to support my belief that Linux is actively being sabotaged by people who don't want it to succeed.
- z3t4 5y ago
  Software have a hard time keeping up with hardware architecture, mostly because of backwards compatibility... Imagine running a restaurant, normally you can take 32 orders and have the customer sit and wait. One day you get new chefs that can make food 100x faster, but now you can only take one or two orders before the chefs have the food ready and you have to give it to the customer that ordered it. So despite the chefs being 100x faster it now takes much longer to place an order, and the waiting line can grow long with impatient customers.

travisgriggs 5y ago

> I've grown to love systemd. I think I’ve grown to appreciate it, but not love it. It feels like both systems are at opposite ends of a pendulum. The RC/unit system was very approachable for me. You could explain it easily, and you could get inside of it and mess around very easily. That affordance/discoverability was awesome. With systemd, simple things are nicely templated. I can find an example and tweak it to achieve what I want. Complicated things get complicated real fast.

fullstop 5y ago
The RC system was simple, I'll give you that. I love not having to track down pid files. I love not having to check if the pid file contents match a valid instance of the expected binary. The RC system was simple and also full of exceptions and corner cases.
- tluyben2 5y ago
  What is the use case for that? I never had these needs on servers. Maybe you are doing something very different from me, so I am curious. I use RC on servers and systemd on clients; things run robustly for 15+ years on many servers for me (with security updates included).
  - fullstop 5y ago
    Let's say that you're starting Apache Tomcat. You have to dig into the tomcat startup scripts and figure out where, if anywhere, it writes the pid file to disk so that you can later use it to determine if the process is running or not. If java happened to crash, though, this pid file is stale and might not point to a valid process. There's a chance that the pid has been reused, and it could have been reused by anything -- even another java process! This is important, because this pid file is used to determine which process receives the kill signal. If you get it wrong, and have the right permissions, you can accidentally kill something that you did not intend to kill. This is further complicated if you want to run multiple instances of Tomcat because now you need to have a unique path for this pid file per tomcat instance. If the thing that you're trying to run doesn't fork, you then have to execute it in the background and then store the result of $! somewhere so that you know how to kill the process later on. It's all very error prone and the process for each daemon is often different.

tored 5y ago

On my personal linux workstation I experienced multiple times that I couldn't ssh into it after the desktop froze, it was due to too small swap file.

themodelplumber 5y ago
That's interesting, what proportion are you using to determine swap size? For example 1x RAM, 2x RAM, etc.
- tored 5y ago
  I run at 1xRAM. Seems that my problems went away after that. Ubuntu by default sets it to 2GB even if you have something like 16GB RAM. Swap file can then fill up pretty quickly.
  - themodelplumber 5y ago
    > Ubuntu by default Yep, one of my systems got this treatment. It's annoying because OOM has been an issue ever since. I'm playing with EarlyOOM and trying to remember if it'll be a huge pain to resize the partition for more swap space. Thanks for your reply.
    - tored 5y ago
      Thanks for the tip on EarlyOOM. Swap file is at least easier to handle than swap partition. Some run their Linux without swap at all, however I think that is better for a server setup where you run a small set of specific set of binaries, thus you know your load. That is not the case on a desktop where you running a variable set of binaries depending on your task. I’m somewhat surprised how bad this works on Linux, I don’t think I ever experienced a swap problem on Windows (not even in the 9x days), I guess this is because "Year of the Linux desktop" never happened, Linux is primarily a server and embedded OS, not a desktop OS.
      - deleted 5y ago
        [deleted]

saurik 5y ago

If it makes you feel any better, I have been using Linux for like 25-30 years now and I still experience "sigh. and now I can't SSH into the box to kill the thing I shouldn't have started".