6 ms·

Colin (and others?) have been working to reduce the boot time of FreeBSD for a while now. At this point shaving off a few milliseconds is probably a nice win fo

Colin (and others?) have been working to reduce the boot time of FreeBSD for a while now. At this point shaving off a few milliseconds is probably a nice win for them. https://twitter.com/cperciva/status/1659391725427720195?t=0y_F1T03dyGuBb8PjECm7Q&s=19 https://twitter.com/cperciva/status/1659391725427720195?t=0y...

taeric 3y ago
I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? Note that I want to stress that that is no reason for some folks to stop focusing on it. And kudos on the improvement. Criticism, if any, here is that statistics once again rears its head to make something feel bigger by using a percent.
- nextaccountic 3y ago
  > I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? If whoever is fixing this depend on quickly launching instances in Firecracker, it's their problem. And that's how open source is usually done
  - taeric 3y ago
    Sorta. They have to rely on this on a repeated basis such that it matters. And there, if startup time is still 10x this, roughly, why push for solutions that require that many kernel startups?
    - vasco 3y ago
      Why do anything, we're all going to die!! Super fast boots of containers is a good effort, would be cool to be as fast as spawning a process or a thread!
      - taeric 3y ago
        That is silly. I already said I do not intend this as a criticism of the folks working on it. Kudos on the improvement. That said... for most people, you are better off learning to use threads or processes. It is almost certainly less effort than spinning up a VM. Not to mention everything else you will, by necessity, have to do if you are using VMs for everything. Unless you find ways to do shared network connections across VMs and such, but... I'm curious how we can make things closer to threads, without eventually bringing all of the dangers of threads along with it?
        rcoveson 3y ago
        Why make "for most people" points when we're already talking about FreeBSD? There's a sort of uni-/light-kernel race quietly happening right now since the industry doesn't really fully trust process isolation within an OS for running completely untrusted code. At least not as much as it trusts VT-X/AMD-V. In that race, kernel startup time is just like JVM startup time, or Julia startup time, or Python startup time, all of which are things that people work on shaving milliseconds from.
        taeric 3y ago
        Ouch. That feels even harsher to the idea than what I'm saying. :D To your point, my "for most people" is aimed at this forum. Not the folks doing the work. Is incredibly cool that the folks doing this are also on the forum, but I think it is safe to say they are not most of the folks on here. That not the case?
        rcoveson 3y ago
        Say the post had been "How Netflix uses FreeBSD to achieve network latencies less than 10 microseconds" or something. How helpful would it be to comment about how "for most people" this doesn't matter? Did anybody who read the tweet think it mattered to them when it didn't? It mentions Firecracker explicitly. How many people on HN do you think upvote this because they themselves are running FreeBSD on Firecracker, versus the number who upvoted because it's just interesting in and of itself?
    - nextaccountic 3y ago
      Isn't fast startup the point of firecracker? It's supposed to scale down to zero when there are no users, and quickly ramp up when needed.
      - taeric 3y ago
        I mean... sorta? Yes, it is the point. But, the goal is to make it so that you can do this scale up/down at the VM level. We have been able to do this at a thread or even process level for a long long time, at this point.
  - lacrosse_tannin 3y ago
    why not boot a good os in firecracker instead?
- maccard 3y ago
  4.5ms on what hardware, in what scenario? Would I like to save 5ms off the startup time of my VMs? You betcha. Does that 5ms turn into 200ms on a low power device? Probably
  - taeric 3y ago
    I'm not even clear that I care on my VMs. I don't start up enough for that to save me more than... a second a month or so? Maybe. Probably not even.
    - maccard 3y ago
      Great! Then this doesn't affect you!
      - taeric 3y ago
        Right, but again pointing this at most people on this forum, that answer is probably the same. Very few of us are in a situation where this can save seconds a year, much less seconds a month. For the folks this could potentially save time, I'm curious if it is better than taking an alternative route. Would be delighted to see an analysis showing impact. And again, kudos to the folks for possibly speeding this up. I'm assuming qsort or some such will be faster, but that isn't a given. For that matter, if it is being sorted so that searches are faster later, than the number of searches should be considered, as you could probably switch out to sentinel scanning to save the sort time, and then you are down to how many scans you are doing times the number of elements you are looking against.
  - daneel_w 3y ago
    How would you even discern this delay from all the other delays happening before you're done booting, when there are so many other natural variances of a couple of milliseconds here and there? Every "on metal" boot is unique, no two are exactly as fast, and certainly never within 1.98 milliseconds of eachother even in the case of a VM with lots of spare resources on the host. You're painting a too pretty picture of this.
- aidenn0 3y ago
  My math says ~2ms (28ms*0.07 = 1.96ms)? Still, if it mattered to get it down to 28ms, it might matter to get it down to 26ms...
  - taeric 3y ago
    Agreed, but my math also says that if this is your pain point, you'd save even more time by not firing off more VMs? That is, skip the startup time entirely by just not doing it that often.
    - aidenn0 3y ago
      I don't startup VMs multiple times per hour, much less per minute, so I don't assume to know what tradeoffs the people using firecracker are making when deciding how often to startup VMs.
      - taeric 3y ago
        Fair. I still feel fine pushing back on this. The amount of other resources getting wasted to support this sort of workload is kind of easy to imagine. I will put back the perspective that they were not hunting for 2ms increases. They were originally chopping off at seconds on startup. The progress they made is laudable. And how they found it and acted on it is great.
- MuffinFlavored 3y ago
  if 4.5ms is 7%, the entire boot time is 64ms?
  - taeric 3y ago
    Ha, I thought I saw 4.5ms from another post. Not sure where I got that number. :( Realizing this is why someone said the number was more like 2ms. I don't think that changes much of my view here. Again, kudos on making it faster, if they do.

taeric 3y ago

I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? Note that I want to stress that that is no reason for some folks to stop focusing on it. And kudos on the improvement. Criticism, if any, here is that statistics once again rears its head to make something feel bigger by using a percent.

nextaccountic 3y ago
> I mean, I get it. But shaving 4.5ms does seem to fall into the realm of not most peoples problems? If whoever is fixing this depend on quickly launching instances in Firecracker, it's their problem. And that's how open source is usually done
- taeric 3y ago
  Sorta. They have to rely on this on a repeated basis such that it matters. And there, if startup time is still 10x this, roughly, why push for solutions that require that many kernel startups?
  - vasco 3y ago
    Why do anything, we're all going to die!! Super fast boots of containers is a good effort, would be cool to be as fast as spawning a process or a thread!
    - taeric 3y ago
      That is silly. I already said I do not intend this as a criticism of the folks working on it. Kudos on the improvement. That said... for most people, you are better off learning to use threads or processes. It is almost certainly less effort than spinning up a VM. Not to mention everything else you will, by necessity, have to do if you are using VMs for everything. Unless you find ways to do shared network connections across VMs and such, but... I'm curious how we can make things closer to threads, without eventually bringing all of the dangers of threads along with it?
      - rcoveson 3y ago
        Why make "for most people" points when we're already talking about FreeBSD? There's a sort of uni-/light-kernel race quietly happening right now since the industry doesn't really fully trust process isolation within an OS for running completely untrusted code. At least not as much as it trusts VT-X/AMD-V. In that race, kernel startup time is just like JVM startup time, or Julia startup time, or Python startup time, all of which are things that people work on shaving milliseconds from.
        taeric 3y ago
        Ouch. That feels even harsher to the idea than what I'm saying. :D To your point, my "for most people" is aimed at this forum. Not the folks doing the work. Is incredibly cool that the folks doing this are also on the forum, but I think it is safe to say they are not most of the folks on here. That not the case?
        rcoveson 3y ago
        Say the post had been "How Netflix uses FreeBSD to achieve network latencies less than 10 microseconds" or something. How helpful would it be to comment about how "for most people" this doesn't matter? Did anybody who read the tweet think it mattered to them when it didn't? It mentions Firecracker explicitly. How many people on HN do you think upvote this because they themselves are running FreeBSD on Firecracker, versus the number who upvoted because it's just interesting in and of itself?
  - nextaccountic 3y ago
    Isn't fast startup the point of firecracker? It's supposed to scale down to zero when there are no users, and quickly ramp up when needed.
    - taeric 3y ago
      I mean... sorta? Yes, it is the point. But, the goal is to make it so that you can do this scale up/down at the VM level. We have been able to do this at a thread or even process level for a long long time, at this point.
- lacrosse_tannin 3y ago
  why not boot a good os in firecracker instead?
maccard 3y ago
4.5ms on what hardware, in what scenario? Would I like to save 5ms off the startup time of my VMs? You betcha. Does that 5ms turn into 200ms on a low power device? Probably
- taeric 3y ago
  I'm not even clear that I care on my VMs. I don't start up enough for that to save me more than... a second a month or so? Maybe. Probably not even.
  - maccard 3y ago
    Great! Then this doesn't affect you!
    - taeric 3y ago
      Right, but again pointing this at most people on this forum, that answer is probably the same. Very few of us are in a situation where this can save seconds a year, much less seconds a month. For the folks this could potentially save time, I'm curious if it is better than taking an alternative route. Would be delighted to see an analysis showing impact. And again, kudos to the folks for possibly speeding this up. I'm assuming qsort or some such will be faster, but that isn't a given. For that matter, if it is being sorted so that searches are faster later, than the number of searches should be considered, as you could probably switch out to sentinel scanning to save the sort time, and then you are down to how many scans you are doing times the number of elements you are looking against.
- daneel_w 3y ago
  How would you even discern this delay from all the other delays happening before you're done booting, when there are so many other natural variances of a couple of milliseconds here and there? Every "on metal" boot is unique, no two are exactly as fast, and certainly never within 1.98 milliseconds of eachother even in the case of a VM with lots of spare resources on the host. You're painting a too pretty picture of this.
aidenn0 3y ago
My math says ~2ms (28ms*0.07 = 1.96ms)? Still, if it mattered to get it down to 28ms, it might matter to get it down to 26ms...
- taeric 3y ago
  Agreed, but my math also says that if this is your pain point, you'd save even more time by not firing off more VMs? That is, skip the startup time entirely by just not doing it that often.
  - aidenn0 3y ago
    I don't startup VMs multiple times per hour, much less per minute, so I don't assume to know what tradeoffs the people using firecracker are making when deciding how often to startup VMs.
    - taeric 3y ago
      Fair. I still feel fine pushing back on this. The amount of other resources getting wasted to support this sort of workload is kind of easy to imagine. I will put back the perspective that they were not hunting for 2ms increases. They were originally chopping off at seconds on startup. The progress they made is laudable. And how they found it and acted on it is great.
MuffinFlavored 3y ago
if 4.5ms is 7%, the entire boot time is 64ms?
- taeric 3y ago
  Ha, I thought I saw 4.5ms from another post. Not sure where I got that number. :( Realizing this is why someone said the number was more like 2ms. I don't think that changes much of my view here. Again, kudos on making it faster, if they do.