8 ms·

So a few milliseconds in total. Big whoop. OpenBSD's ahci(4) driver stalls 5-6 seconds for each SATA device on the bus, just to make sure the device really is t

So a few milliseconds in total. Big whoop. OpenBSD's ahci(4) driver stalls 5-6 seconds for each SATA device on the bus, just to make sure the device really is there and that the link speed is correct...or something. My two OpenBSD machines, which incidentally have 3 SATA drives each, spend almost 20 seconds on that one segment alone during kernel startup.

jhalstead 3y ago
Colin (and others?) have been working to reduce the boot time of FreeBSD for a while now. At this point shaving off a few milliseconds is probably a nice win for them. https://twitter.com/cperciva/status/1659391725427720195?t=0y_F1T03dyGuBb8PjECm7Q&s=19 https://twitter.com/cperciva/status/1659391725427720195?t=0y...
- taeric 3y ago
  I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? Note that I want to stress that that is no reason for some folks to stop focusing on it. And kudos on the improvement. Criticism, if any, here is that statistics once again rears its head to make something feel bigger by using a percent.
  - nextaccountic 3y ago
    > I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? If whoever is fixing this depend on quickly launching instances in Firecracker, it's their problem. And that's how open source is usually done
    - taeric 3y ago
      Sorta. They have to rely on this on a repeated basis such that it matters. And there, if startup time is still 10x this, roughly, why push for solutions that require that many kernel startups?
      - vasco 3y ago
        Why do anything, we're all going to die!! Super fast boots of containers is a good effort, would be cool to be as fast as spawning a process or a thread!
        taeric 3y ago
        That is silly. I already said I do not intend this as a criticism of the folks working on it. Kudos on the improvement. That said... for most people, you are better off learning to use threads or processes. It is almost certainly less effort than spinning up a VM. Not to mention everything else you will, by necessity, have to do if you are using VMs for everything. Unless you find ways to do shared network connections across VMs and such, but... I'm curious how we can make things closer to threads, without eventually bringing all of the dangers of threads along with it?
        rcoveson 3y ago
        Why make "for most people" points when we're already talking about FreeBSD? There's a sort of uni-/light-kernel race quietly happening right now since the industry doesn't really fully trust process isolation within an OS for running completely untrusted code. At least not as much as it trusts VT-X/AMD-V. In that race, kernel startup time is just like JVM startup time, or Julia startup time, or Python startup time, all of which are things that people work on shaving milliseconds from.
        taeric 3y ago
        Ouch. That feels even harsher to the idea than what I'm saying. :D To your point, my "for most people" is aimed at this forum. Not the folks doing the work. Is incredibly cool that the folks doing this are also on the forum, but I think it is safe to say they are not most of the folks on here. That not the case?
        rcoveson 3y ago
        Say the post had been "How Netflix uses FreeBSD to achieve network latencies less than 10 microseconds" or something. How helpful would it be to comment about how "for most people" this doesn't matter? Did anybody who read the tweet think it mattered to them when it didn't? It mentions Firecracker explicitly. How many people on HN do you think upvote this because they themselves are running FreeBSD on Firecracker, versus the number who upvoted because it's just interesting in and of itself?
    - lacrosse_tannin 3y ago
      why not boot a good os in firecracker instead?
  - maccard 3y ago
    4.5ms on what hardware, in what scenario? Would I like to save 5ms off the startup time of my VMs? You betcha. Does that 5ms turn into 200ms on a low power device? Probably
    - taeric 3y ago
      I'm not even clear that I care on my VMs. I don't start up enough for that to save me more than... a second a month or so? Maybe. Probably not even.
      - maccard 3y ago
        Great! Then this doesn't affect you!
        taeric 3y ago
        Right, but again pointing this at most people on this forum, that answer is probably the same. Very few of us are in a situation where this can save seconds a year, much less seconds a month. For the folks this could potentially save time, I'm curious if it is better than taking an alternative route. Would be delighted to see an analysis showing impact. And again, kudos to the folks for possibly speeding this up. I'm assuming qsort or some such will be faster, but that isn't a given. For that matter, if it is being sorted so that searches are faster later, than the number of searches should be considered, as you could probably switch out to sentinel scanning to save the sort time, and then you are down to how many scans you are doing times the number of elements you are looking against.
    - daneel_w 3y ago
      How would you even discern this delay from all the other delays happening before you're done booting, when there are so many other natural variances of a couple of milliseconds here and there? Every "on metal" boot is unique, no two are exactly as fast, and certainly never within 1.98 milliseconds of eachother even in the case of a VM with lots of spare resources on the host. You're painting a too pretty picture of this.
  - aidenn0 3y ago
    My math says ~2ms (28ms*0.07 = 1.96ms)? Still, if it mattered to get it down to 28ms, it might matter to get it down to 26ms...
    - taeric 3y ago
      Agreed, but my math also says that if this is your pain point, you'd save even more time by not firing off more VMs? That is, skip the startup time entirely by just not doing it that often.
      - aidenn0 3y ago
        I don't startup VMs multiple times per hour, much less per minute, so I don't assume to know what tradeoffs the people using firecracker are making when deciding how often to startup VMs.
        taeric 3y ago
        Fair. I still feel fine pushing back on this. The amount of other resources getting wasted to support this sort of workload is kind of easy to imagine. I will put back the perspective that they were not hunting for 2ms increases. They were originally chopping off at seconds on startup. The progress they made is laudable. And how they found it and acted on it is great.
  - MuffinFlavored 3y ago
    if 4.5ms is 7%, the entire boot time is 64ms?
    - taeric 3y ago
      Ha, I thought I saw 4.5ms from another post. Not sure where I got that number. :( Realizing this is why someone said the number was more like 2ms. I don't think that changes much of my view here. Again, kudos on making it faster, if they do.
EuropeOverlords 3y ago
OS should save all serial numbers of devices and after they are all found continue. we should make it declarative or how do you want to call it. also why cant freaking boot process be optimized after first boot? bsd is essentialy sorting SAME thing every boot, THAT is ridiculous. sort once, save order ( list, sysinit000000000000 ), boot fast next time. hardware or other change or failed boot, can trigger start sorting bull for safety. you know what youre booting into, so sort it once then run it from saved order next time. how many times you change hardware on computer ? and if you do, you can just restart with grub flag, toggle switch in control panel before restart, etc
- exmadscientist 3y ago
  The code to manage "do I sort or do I cache" is probably worse than the code to just sort it unconditionally. And you really want to do this automatically, with no manual anything, because (among other reasons) if you need to swap a part on a broken machine, you do not want that to needlessly break the software. So you're sorting regardless, and so you might as well just be unconditional about it.
  - EuropeOverlords 3y ago
    on my machine TPM is checking state of machine for security reasons, so if this runs ANYWAY then why not use that for one more useful thing ... ( it runs before even machine thinks about booting anything ) you can define shutdown flag for hardware change: shutdown -rF XD, some things are hot swappable and for that there has to be logic (in kernel) to know you have new hardware... if you have total hardware failure that can be detected too, on next boot...
- Denvercoder9 3y ago
  > toggle switch in control panel before restart Ah yes, let's brick the OS when hardware fails and has to be replaced without advance notice.
  - EuropeOverlords 3y ago
kevingadd 3y ago
If you boot OpenBSD thousands of times per day it adds up. I can imagine this being the case for people running big server farms or doing virtualization stuff.
- daneel_w 3y ago
  These are servers and I reboot them only when necessary. It's a pet peeve, but a big one.
  - ilyt 3y ago
    In my experience time to boot everything else on server far exceeds any OS shenaningans. I had servers that took up to 5 minutes from start/reboot to GRUB prompt. And they were not some 60 drive monsters but typical dual CPU 1U servers...
- trzy 3y ago
  At 7% of 28ms, if you boot it a thousand times per day you get a whopping 2 seconds.
  - ilyt 3y ago
    no, if your SATA/SAS stalls for 20s you get 20s
MichaelZuo 3y ago
How do the other OS's get around this?
- daneel_w 3y ago
  I don't know, but I don't think there's an actual case that Linux/FreeBSD/etc. happens to have a workaround for. I think it's just OpenBSD charging towards a windmill. It's not impossible that it's some archaic and since long invalid legacy thing along the lines of "let's wait 5 seconds here just in case the drive hasn't managed to spin up, despite the computer having been powered on for 15 seconds already, because that happened once in 2003 on someone's 1989 HPPA system so we need to support that case". I'm not joking, this is really what I imagine it being about.
  - adrianmonk 3y ago
    That's believable and maybe even reasonable. There were (or still are?) SCSI drives that would not spin up until told to over the bus. I think the idea is to not do them all at once since the motor draws a lot of power. I'm fairly sure I've run into drives that would not respond on the bus while spinning up. And if it happens with SCSI drives, it may happen with other types.
    - daneel_w 3y ago
      >"There were (or still are?) SCSI drives that would not spin up until told to over the bus" Surely they'd spin up as soon as the BIOS started probing the bus to init the hardware, long before the kernel was running, or they'd be infamous for being "the HDDs you cannot boot an OS from"... In my 25 years of using PCs I have not once come across a drive that did not spin up as soon as the computer was powered on. But whatever the case is, Linux and FreeBSD never had this behavior. Waiting some arbitrary amount of time isn't an appropriate solution (to what I insist is just an imagined problem), it's just a poor bandaid.
      - toast0 3y ago
        > Surely they'd spin up as soon as the BIOS started probing the bus to init the hardware, long before the kernel was running, or they'd be infamous for being "the HDDs you cannot boot an OS from"... Depends on the market they sell into, might not be a big deal for enterprise drives that you don't expect to boot from; especially if you're attaching them to a controller that you also don't expect to boot from. I had some giant quad processor pentium II beast after it was retired, and the BIOS could not boot from the fancy drive array at all; and even if it could, the drives were on a staggered startup system, so it would have had to wait forever for that.
0x457 3y ago
Well, FreeBSD has other issues too. I had re-cable all of my USB devices because it was taking forever (linux and windows booted a lot faster).
- ilyt 3y ago
  Huh ? Were they resetting under FreeBSD or something ?
  - 0x457 3y ago
    I had it setup like this: some devices plugged into PC directly, KVM plugged into the PC, hub plugged into PC, hub on monitor plugged into hub and a bunch of things plugged into hubs. Half of the devices were just powered by USB and not using any data, but only needed to be ON when the PC is ON. The way it worked: - FreeBSD would scan all USB devices - get held up on some for no reason - find a hub - scan everything on a hub - find another hub - scan everything on that hub as well Anytime during the scan it could run into a device that takes longer (no clue which or why, only happened on FreeBSD and never on Linux or Windows) I'm sure windows and linux did the same thing, but both shapely continued booting while FreeBSD waited and waited and waited and waited. Bias light on a monitor would sometimes completely stop the process until unplugged. USB stack on FreeBSD is forever cursed, I still remember kernel panics when you unplug a keyboard or a mouse.
    - toast0 3y ago
      > I'm sure windows and linux did the same thing, but both shapely continued booting while FreeBSD waited and waited and waited and waited. My understanding is FreeBSD will by default wait until it's found all the fixed disks before it figures out which one to mount as root. This could be a USB drive, so it needs to enumerate all the USB first. Adding hw.usb.no_boot_wait=1 to /boot/loader.conf.local or thereabouts if you don't need that will skip that wait and speed up the boot process a lot (especially on hardware where USB is slow to finish enumeration).
      - 3y ago
ilyt 3y ago
...wat? How something that bad could linger for so long?
- aidenn0 3y ago
  It adds 2ms to the boot time on modern hardware, and the list it is sorting has grown by about 2 orders of magnitude since it was introduced. Seems reasonable to me that it was unnoticed.
  - mardifoufs 3y ago
    I think the comment you are replying to was referring to the openbsd issue that apparently adds 10-20secs per boot in the GP comment.
  - hinkley 3y ago
    I am currently at work fighting the same class of problem. Bad pattern was used a few times, seemed to work. So it was used twenty times, still no problems. Now it’s used 100 times and it was 20% of response time. Got that down to half in a few months, now it’s a long slog to halve it again. It should never have been allowed to get past 3% of request startup time.
- tedunangst 3y ago
  Because none of the people afflicted have done anything about it.
  - daneel_w 3y ago
    Likely because none have the required proficiency. I certainly don't. "So? It's open source. Stop complaining and fix it." is never a good response to a bug report. I know this quirky behavior has been brought up a few times by other users than me on the openbsd-bugs@ mailing list during the 10 or so years since I first observed it.
count 3y ago
Colin's focus has been on speeding up EC2 boot time. You pay per second from time on EC2. A few milliseconds at scale probably ads up to a decent amount of savings - easily imaginable it's enough to cover the work it took to find the time savings.
- taeric 3y ago
  Yes, you pay per second. But, per other notes, this is slated to save at most 2ms. Realistically, it won't save that much, as they are still doing a sort, not completely skipping it, but lets assume qsort does get this down to 0 and they somehow saved it all. That is trivially 500 boots before you see a second. Which... is a lot of reboots. If you have processes that are spawning off 500 instances, you would almost certainly see better savings by condensing those down to fewer instances. So, realistically, assuming that Colin is a decently paid software engineer, I find it doubtable the savings from this particular change will ever add up to mean anything even close to what it costs to have just one person assigned to it. Now, every other change they found up to this point may help sway that needle, but at this point, though 7% is a lot of percent, they are well into the area where savings they are finding are very unlikely to ever be worth finding. Edit: I saw that qsort does indeed get it into basically zero at this range of discussion. I'd love to see more of the concrete numbers.
  - vidarh 3y ago
    Missing the point, which is to shave off cold start times for things like Lambdas, where shorter start times means you can more aggressively shut them down when idle, which means you can pack more onto the same machine.
    - taeric 3y ago
      See, the binpacking that this implies for lambda instances is insane to me. Certainly sounds impressive and great, and they may even pull it off. And, certainly, for the Lambda team, this at least sounds like it makes sense. I'll even root for them to hope it works out. It just reminds me of several failed attempts to move to JVM based technologies to replace perl, because "just think of the benefits if we get JVM startup so that each request is its own JVM?" They even had papers showing resource advantages that you would gain. Turned out, many times, that competing against how cheap perl processes were was not something you wanted to do. At least, not if you wanted to succeed.

jhalstead 3y ago

Colin (and others?) have been working to reduce the boot time of FreeBSD for a while now. At this point shaving off a few milliseconds is probably a nice win for them. https://twitter.com/cperciva/status/1659391725427720195?t=0y_F1T03dyGuBb8PjECm7Q&s=19 https://twitter.com/cperciva/status/1659391725427720195?t=0y...

taeric 3y ago
I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? Note that I want to stress that that is no reason for some folks to stop focusing on it. And kudos on the improvement. Criticism, if any, here is that statistics once again rears its head to make something feel bigger by using a percent.
- nextaccountic 3y ago
  > I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? If whoever is fixing this depend on quickly launching instances in Firecracker, it's their problem. And that's how open source is usually done
  - taeric 3y ago
    Sorta. They have to rely on this on a repeated basis such that it matters. And there, if startup time is still 10x this, roughly, why push for solutions that require that many kernel startups?
    - vasco 3y ago
      Why do anything, we're all going to die!! Super fast boots of containers is a good effort, would be cool to be as fast as spawning a process or a thread!
      - taeric 3y ago
        That is silly. I already said I do not intend this as a criticism of the folks working on it. Kudos on the improvement. That said... for most people, you are better off learning to use threads or processes. It is almost certainly less effort than spinning up a VM. Not to mention everything else you will, by necessity, have to do if you are using VMs for everything. Unless you find ways to do shared network connections across VMs and such, but... I'm curious how we can make things closer to threads, without eventually bringing all of the dangers of threads along with it?
        rcoveson 3y ago
        Why make "for most people" points when we're already talking about FreeBSD? There's a sort of uni-/light-kernel race quietly happening right now since the industry doesn't really fully trust process isolation within an OS for running completely untrusted code. At least not as much as it trusts VT-X/AMD-V. In that race, kernel startup time is just like JVM startup time, or Julia startup time, or Python startup time, all of which are things that people work on shaving milliseconds from.
        taeric 3y ago
        Ouch. That feels even harsher to the idea than what I'm saying. :D To your point, my "for most people" is aimed at this forum. Not the folks doing the work. Is incredibly cool that the folks doing this are also on the forum, but I think it is safe to say they are not most of the folks on here. That not the case?
        rcoveson 3y ago
        Say the post had been "How Netflix uses FreeBSD to achieve network latencies less than 10 microseconds" or something. How helpful would it be to comment about how "for most people" this doesn't matter? Did anybody who read the tweet think it mattered to them when it didn't? It mentions Firecracker explicitly. How many people on HN do you think upvote this because they themselves are running FreeBSD on Firecracker, versus the number who upvoted because it's just interesting in and of itself?
  - lacrosse_tannin 3y ago
    why not boot a good os in firecracker instead?
- maccard 3y ago
  4.5ms on what hardware, in what scenario? Would I like to save 5ms off the startup time of my VMs? You betcha. Does that 5ms turn into 200ms on a low power device? Probably
  - taeric 3y ago
    I'm not even clear that I care on my VMs. I don't start up enough for that to save me more than... a second a month or so? Maybe. Probably not even.
    - maccard 3y ago
      Great! Then this doesn't affect you!
      - taeric 3y ago
        Right, but again pointing this at most people on this forum, that answer is probably the same. Very few of us are in a situation where this can save seconds a year, much less seconds a month. For the folks this could potentially save time, I'm curious if it is better than taking an alternative route. Would be delighted to see an analysis showing impact. And again, kudos to the folks for possibly speeding this up. I'm assuming qsort or some such will be faster, but that isn't a given. For that matter, if it is being sorted so that searches are faster later, than the number of searches should be considered, as you could probably switch out to sentinel scanning to save the sort time, and then you are down to how many scans you are doing times the number of elements you are looking against.
  - daneel_w 3y ago
    How would you even discern this delay from all the other delays happening before you're done booting, when there are so many other natural variances of a couple of milliseconds here and there? Every "on metal" boot is unique, no two are exactly as fast, and certainly never within 1.98 milliseconds of eachother even in the case of a VM with lots of spare resources on the host. You're painting a too pretty picture of this.
- aidenn0 3y ago
  My math says ~2ms (28ms*0.07 = 1.96ms)? Still, if it mattered to get it down to 28ms, it might matter to get it down to 26ms...
  - taeric 3y ago
    Agreed, but my math also says that if this is your pain point, you'd save even more time by not firing off more VMs? That is, skip the startup time entirely by just not doing it that often.
    - aidenn0 3y ago
      I don't startup VMs multiple times per hour, much less per minute, so I don't assume to know what tradeoffs the people using firecracker are making when deciding how often to startup VMs.
      - taeric 3y ago
        Fair. I still feel fine pushing back on this. The amount of other resources getting wasted to support this sort of workload is kind of easy to imagine. I will put back the perspective that they were not hunting for 2ms increases. They were originally chopping off at seconds on startup. The progress they made is laudable. And how they found it and acted on it is great.
- MuffinFlavored 3y ago
  if 4.5ms is 7%, the entire boot time is 64ms?
  - taeric 3y ago
    Ha, I thought I saw 4.5ms from another post. Not sure where I got that number. :( Realizing this is why someone said the number was more like 2ms. I don't think that changes much of my view here. Again, kudos on making it faster, if they do.

EuropeOverlords 3y ago

OS should save all serial numbers of devices and after they are all found continue. we should make it declarative or how do you want to call it. also why cant freaking boot process be optimized after first boot? bsd is essentialy sorting SAME thing every boot, THAT is ridiculous. sort once, save order ( list, sysinit000000000000 ), boot fast next time. hardware or other change or failed boot, can trigger start sorting bull for safety. you know what youre booting into, so sort it once then run it from saved order next time. how many times you change hardware on computer ? and if you do, you can just restart with grub flag, toggle switch in control panel before restart, etc

exmadscientist 3y ago
The code to manage "do I sort or do I cache" is probably worse than the code to just sort it unconditionally. And you really want to do this automatically, with no manual anything, because (among other reasons) if you need to swap a part on a broken machine, you do not want that to needlessly break the software. So you're sorting regardless, and so you might as well just be unconditional about it.
- EuropeOverlords 3y ago
  on my machine TPM is checking state of machine for security reasons, so if this runs ANYWAY then why not use that for one more useful thing ... ( it runs before even machine thinks about booting anything ) you can define shutdown flag for hardware change: shutdown -rF XD, some things are hot swappable and for that there has to be logic (in kernel) to know you have new hardware... if you have total hardware failure that can be detected too, on next boot...
Denvercoder9 3y ago
> toggle switch in control panel before restart Ah yes, let's brick the OS when hardware fails and has to be replaced without advance notice.
- EuropeOverlords 3y ago

kevingadd 3y ago

If you boot OpenBSD thousands of times per day it adds up. I can imagine this being the case for people running big server farms or doing virtualization stuff.

daneel_w 3y ago
These are servers and I reboot them only when necessary. It's a pet peeve, but a big one.
- ilyt 3y ago
  In my experience time to boot everything else on server far exceeds any OS shenaningans. I had servers that took up to 5 minutes from start/reboot to GRUB prompt. And they were not some 60 drive monsters but typical dual CPU 1U servers...
trzy 3y ago
At 7% of 28ms, if you boot it a thousand times per day you get a whopping 2 seconds.
- ilyt 3y ago
  no, if your SATA/SAS stalls for 20s you get 20s

MichaelZuo 3y ago

How do the other OS's get around this?

daneel_w 3y ago
I don't know, but I don't think there's an actual case that Linux/FreeBSD/etc. happens to have a workaround for. I think it's just OpenBSD charging towards a windmill. It's not impossible that it's some archaic and since long invalid legacy thing along the lines of "let's wait 5 seconds here just in case the drive hasn't managed to spin up, despite the computer having been powered on for 15 seconds already, because that happened once in 2003 on someone's 1989 HPPA system so we need to support that case". I'm not joking, this is really what I imagine it being about.
- adrianmonk 3y ago
  That's believable and maybe even reasonable. There were (or still are?) SCSI drives that would not spin up until told to over the bus. I think the idea is to not do them all at once since the motor draws a lot of power. I'm fairly sure I've run into drives that would not respond on the bus while spinning up. And if it happens with SCSI drives, it may happen with other types.
  - daneel_w 3y ago
    >"There were (or still are?) SCSI drives that would not spin up until told to over the bus" Surely they'd spin up as soon as the BIOS started probing the bus to init the hardware, long before the kernel was running, or they'd be infamous for being "the HDDs you cannot boot an OS from"... In my 25 years of using PCs I have not once come across a drive that did not spin up as soon as the computer was powered on. But whatever the case is, Linux and FreeBSD never had this behavior. Waiting some arbitrary amount of time isn't an appropriate solution (to what I insist is just an imagined problem), it's just a poor bandaid.
    - toast0 3y ago
      > Surely they'd spin up as soon as the BIOS started probing the bus to init the hardware, long before the kernel was running, or they'd be infamous for being "the HDDs you cannot boot an OS from"... Depends on the market they sell into, might not be a big deal for enterprise drives that you don't expect to boot from; especially if you're attaching them to a controller that you also don't expect to boot from. I had some giant quad processor pentium II beast after it was retired, and the BIOS could not boot from the fancy drive array at all; and even if it could, the drives were on a staggered startup system, so it would have had to wait forever for that.

0x457 3y ago

Well, FreeBSD has other issues too. I had re-cable all of my USB devices because it was taking forever (linux and windows booted a lot faster).

ilyt 3y ago
Huh ? Were they resetting under FreeBSD or something ?
- 0x457 3y ago
  I had it setup like this: some devices plugged into PC directly, KVM plugged into the PC, hub plugged into PC, hub on monitor plugged into hub and a bunch of things plugged into hubs. Half of the devices were just powered by USB and not using any data, but only needed to be ON when the PC is ON. The way it worked: - FreeBSD would scan all USB devices - get held up on some for no reason - find a hub - scan everything on a hub - find another hub - scan everything on that hub as well Anytime during the scan it could run into a device that takes longer (no clue which or why, only happened on FreeBSD and never on Linux or Windows) I'm sure windows and linux did the same thing, but both shapely continued booting while FreeBSD waited and waited and waited and waited. Bias light on a monitor would sometimes completely stop the process until unplugged. USB stack on FreeBSD is forever cursed, I still remember kernel panics when you unplug a keyboard or a mouse.
  - toast0 3y ago
    > I'm sure windows and linux did the same thing, but both shapely continued booting while FreeBSD waited and waited and waited and waited. My understanding is FreeBSD will by default wait until it's found all the fixed disks before it figures out which one to mount as root. This could be a USB drive, so it needs to enumerate all the USB first. Adding hw.usb.no_boot_wait=1 to /boot/loader.conf.local or thereabouts if you don't need that will skip that wait and speed up the boot process a lot (especially on hardware where USB is slow to finish enumeration).
    - 3y ago

ilyt 3y ago

...wat? How something that bad could linger for so long?

aidenn0 3y ago
It adds 2ms to the boot time on modern hardware, and the list it is sorting has grown by about 2 orders of magnitude since it was introduced. Seems reasonable to me that it was unnoticed.
- mardifoufs 3y ago
  I think the comment you are replying to was referring to the openbsd issue that apparently adds 10-20secs per boot in the GP comment.
- hinkley 3y ago
  I am currently at work fighting the same class of problem. Bad pattern was used a few times, seemed to work. So it was used twenty times, still no problems. Now it’s used 100 times and it was 20% of response time. Got that down to half in a few months, now it’s a long slog to halve it again. It should never have been allowed to get past 3% of request startup time.
tedunangst 3y ago
Because none of the people afflicted have done anything about it.
- daneel_w 3y ago
  Likely because none have the required proficiency. I certainly don't. "So? It's open source. Stop complaining and fix it." is never a good response to a bug report. I know this quirky behavior has been brought up a few times by other users than me on the openbsd-bugs@ mailing list during the 10 or so years since I first observed it.

count 3y ago

Colin's focus has been on speeding up EC2 boot time. You pay per second from time on EC2. A few milliseconds at scale probably ads up to a decent amount of savings - easily imaginable it's enough to cover the work it took to find the time savings.

taeric 3y ago
Yes, you pay per second. But, per other notes, this is slated to save at most 2ms. Realistically, it won't save that much, as they are still doing a sort, not completely skipping it, but lets assume qsort does get this down to 0 and they somehow saved it all. That is trivially 500 boots before you see a second. Which... is a lot of reboots. If you have processes that are spawning off 500 instances, you would almost certainly see better savings by condensing those down to fewer instances. So, realistically, assuming that Colin is a decently paid software engineer, I find it doubtable the savings from this particular change will ever add up to mean anything even close to what it costs to have just one person assigned to it. Now, every other change they found up to this point may help sway that needle, but at this point, though 7% is a lot of percent, they are well into the area where savings they are finding are very unlikely to ever be worth finding. Edit: I saw that qsort does indeed get it into basically zero at this range of discussion. I'd love to see more of the concrete numbers.
- vidarh 3y ago
  Missing the point, which is to shave off cold start times for things like Lambdas, where shorter start times means you can more aggressively shut them down when idle, which means you can pack more onto the same machine.
  - taeric 3y ago
    See, the binpacking that this implies for lambda instances is insane to me. Certainly sounds impressive and great, and they may even pull it off. And, certainly, for the Lambda team, this at least sounds like it makes sense. I'll even root for them to hope it works out. It just reminds me of several failed attempts to move to JVM based technologies to replace perl, because "just think of the benefits if we get JVM startup so that each request is its own JVM?" They even had papers showing resource advantages that you would gain. Turned out, many times, that competing against how cheap perl processes were was not something you wanted to do. At least, not if you wanted to succeed.