14 ms·

We achieved a 6-fold increase in Podman startup speed

403 points by DrinkWater 3y ago

crest 3y ago
Adding a `sleep $DELAY` to a startup script isn't something to brag about :-P. scnr
- Hendrikto 3y ago
  They removed it.
rwmj 3y ago
> If the backup camera or other sensors were to run as containers, we needed to improve the starting speed significantly. I think I see the problem already. Why does anyone think its a good idea to put everything in an embedded system into a container? Particularly as everything comes from a single vendor and so the usual argument about "but libraries are too hard!" doesn't apply.
- GordonS 3y ago
  Playing Devil's advocate here, but there are some good reasons you might do this. For example, Docker's "pull" system is great for updates, and means it's trivial to rollback to an earlier version if something went wrong. A Docker registry also means you can easily switch to another version when needed. You also get a supervision of containers, with automatic restarts (yes, I know you can do this nowadays with systemd :)
  - qprofyeh 3y ago
    Yup, I can imagine complete images are easier to version-manage and to tag as minor/major/hotfix than each individual part of the stack. I think people underestimate the amount of software that is ALREADY running in their cars/airplanes/helicopters and even elevators.
  - andix 3y ago
    Sure, but if you provide the possibility to deploy a lot of components separately, at some point they will run in different versions. And who knows if rear view camera 1.3.22-44 works with turn signal 4.86.233-stable and break pedal 0.6.9876-beta?
    - wiseowise 3y ago
      How is that different from shared libraries?
      - andix 3y ago
        It’s the same problem. But library authors usually take extra care to stay compatible. And if you create your own shared libraries, they are normally not deployed separately, usually you bundle them with your main executable.
      - solatic 3y ago
        Shared libraries are bundled into one container, then the one container is submitted to a gauntlet of tests (service / API-level tests, e2e tests, etc.). If you're submitting an embedded device to a gauntlet of tests, you're anyway "containerizing" all the shared libraries when you build the embedded image. Trying to build containers within the embedded image has questionable merit.
      - aaravchen 3y ago
        Because you can prove it. In the auto industry you can't just claim it conforms, you have to prove it does for all possible conditions. With containers enforcing that separation, you can prove it much more easily.
    - bluGill 3y ago
      Those 3 for the most part don't have to work together. They need to run on the same CPU without taking more than their allowed portion of the CPU. If they have to work together the communication protocol is clearly defined well in advanced and limited to exactly what they need to say. Thus we are reasonably sure if any one combination works all possible combinations will work. Even then there is typically higher level control to only release combinations that are tested to work together.
  - ClumsyPilot 3y ago
    A statically compiled executable is even easier, and can be hosted on your webserver
    - aaravchen 3y ago
      But unfortunately they also become very space inefficient when a lot of processes need the same (relatively) large blocks of code. But if you carefully curate your containers to use the same base image that contains those libraries already, you don't have to duplicate it. Comparable would be to having an inode de-duplicating file systems, and deterministic binary generation. But it's hard to prove "the correctness" of inode de-duplicating file systems in extreme environments like auto is required to, and deterministic binary generation is hard to control 3ven if it is possible with the specific build tools (it usually isn't).
  - rwmj 3y ago
    "yum", "apt" etc have registries, roll back etc and have for a good few decades.
    - talonx 3y ago
      <sarcasm>But..but..they are not "cool" </sarcasm> This forced application of new technologies into every possible domain can have real security and reliability consequences - and all because some VP somewhere decided they needed to use the latest shiny thing in cars or fridges or ACs or whatever.
      - doubled112 3y ago
        Imagine we could build a dishwasher that didn't throw water on the floor? Oh well, we got WiFi instead. That's fun, right? If a dishwasher needs a firmware update, I might simply argue it was defective. Not everything needs to be secure or updated constantly. It shouldn't have network access to begin with.
        talonx 3y ago
        Exactly. When will vendors stop touting WiFi access as a "feature"?
    - ilyt 3y ago
      And when you upgrade ssl lib now 200 apps are not vulnerable to exploit, vs updating 200 different containers
  - interactivecode 3y ago
    unless the update requires version updates on multiple services which means the versioning and rollback has the same effect as a more monolithic codebase. but with the added complexity of not knowing how it all impacts each other. In my experience updates to smaller services are often trivial, or for updates that are actually impactful it would be way easier to coordinate in a monolithic codebase.
    - steveBK123 3y ago
      yes my first job over 15 years ago was essentially micro services, and we ran into this type of problem.. trivial updates to individual services could be iterated extremely quickly systematic changes to behavior across services were so hard they became incredibly uncommon
- andix 3y ago
  I agree. Microservices are a great concept for a lot of problems. But now I constantly see way too small services. Every tiny piece of software gets its own service and its own software lifecycle (including versioning and deployment). And what happens now is, that you need a huge effort to integrate all those components. End to end system tests get much more important, but are still harder to do than simple unit/integration tests. traditional testing strategies start to get pointless, because most bugs now only appear when combining services in a production setup. Yes, development gets easier, because every team can just develop, without aligning too much with other teams. But the deployment/ops/acceptance step often gets impossibly complex.
  - zokier 3y ago
    You are blaming cultural issues on technology. You have the same integration pains with libraries too. In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways
    - ilyt 3y ago
      The old method of forcing everyone to freeze at certain dependencies also means that any bug in those dependencies is fixed in all components at once. Obviously going too big here is problematic as it can slow it down when tens or hundreds of people are involved in every update, but going too small have similar problems, on top of generally more smaller services eating more resources. We don't need "front reflector LED setting app" being called from "lighting setting app" called from "car setting app", it can probably just be one service. Smaller services also mean more services to update if some commonly used lib gets a security bug. Updating SSL lib in big monolith is just update, run tests, but in microservices that's multiplied by amount of teams and services. > In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways Moot point. We pick the tools to make the job easier. Good team with bad tools will still be slower and less efficient than good team with good tools.
    - goodpoint 3y ago
- erinaceousjones 3y ago
  To be honest I think we should be adopting the full ecosystem we've been busy building around containerization. Imagine calling up breakdown assistance because your car won't start, mechanic comes out, cracks the hood and is like "ah there's your problem right there, ignition service has only 1/2 pods healthy because the node went into NotReady due to DiskPressure. I can clear up some log files so it goes underneath 80% disk usage again but sucks teeth it's gonna cost ya. I'd recommend throwing the whole car out and getting a new one. You shouldn't get an attachment to these things, they're cattle not pets." Truly breathtaking.
  - fredsted 3y ago
    Looks like your cars Kubernetes certificates have expired after a year, we'll need to SSH in and run kubeadm to refresh them. Wait, the 5G pod isn't starting ...
    - wpietri 3y ago
      You joke, but I decided I wanted to try out Kubernetes, so I set it up at home and moved some of my local services into it, one of them being my custom lighting automation software. Eventually my lights just stopped working, and after a lot of rummaging it turned out to be expired certificates. I promptly put the software back where it was before, removed Kubernetes, and decided there were better things for me to play around with.
  - yellowapple 3y ago
    Can't wait to plug in an ODB2 scanner and get a k9s screen.
  - deleted 3y ago
    [deleted]
- ilyt 3y ago
  Especially a thing that should be just "a device that puts video stream on whatever bus it uses". infotainment already uses html/js based UIs, just embed video player in there...
- spyremeown 3y ago
  https://www.toradex.com/torizon https://www.toradex.com/torizon they do and it works quite well. Super handy to have CI/CD and easy OTA updates with containers.
- jpgvm 3y ago
  I think if you open up a Tesla infotainment system you would find something eerily similar to containers. Remember that what you and most programmers think of containers is merely one possible assembly of a bunch of kernel features. There exists not just a gradient between plain processes and containers but a whole solution space with different tradeoffs. I happen to know a small amount of the Tesla internals and they are using cgroups, namespaces, app armour and ebpf based syscall filtering to secure various processes on the car. You almost certainly should not use docker or podman to manage processes on a car but that doesn't mean you shouldn't embrace the subsystems they are built on in order to increase security resilience and defense in depth.
- vbezhenar 3y ago
  Do you oppose running camera software in a separate process? I think it makes sense. Camera process might crash and will be restarted, this should not cause restart of the entire shell. What people should understand is that container in Linux is just a separate process running in powerful chroot (which isolates not just file tree, but also process tree and other things). So the same reasoning which applies to running some code in a separate process also applies to running some code in a separate container. I'd even argue that in an ideal world, almost every process should run in a separate container. The tooling is not here, but concept is just fine.
  - garaetjjte 3y ago
    Containers usually ship their own libraries, which means less sharing, more disk usage and higher memory pressure.
    - vbezhenar 3y ago
      That depends on implementation. Shared libraries which use the same inode will be shared AFAIK. If containers use different libraries, they'll not be shared, of course, but that's a deliberate choice of container creator.
      - bonzini 3y ago
        Containers can use the same base image for the OS.
    - rusk 3y ago
      I see these as a relatively straightforward set of problems to identify, quantify, and remedy. It’s a tradeoff between static memory usage and stability. If that additional memory footprint becomes an issue you can make plans to align dependencies.
    - goodpoint 3y ago
      And much lower chances of getting security updates, now that everything is a huge blob.
- deleted 3y ago
  [deleted]
- deleted 3y ago
  [deleted]
- galangalalgol 3y ago
  Everything never comes from a single vendor. Even microcontrollers the hardware abstraction layer is hiding peripheral support packages for all the i2c and spi things it talks to, each from a different vendor potentially. I'm not saying we should be running k8 on an 8051 or even a cortex m33, but on an arm7? Maybe. Cult of Ferris time, static linking in rust means your binary is your container, particularly if you statically link in musl.
- bluGill 3y ago
  Not everything is developed by the single vendor. Sometimes they buy a program from a vendor they don't fully trust. There is also security, I don't care if someone hacks my radio system nearly as much if someone hacks the brake system. Containers is one part of the total package to isolate parts so if there is a hack the whole system isn't taken out. Computers are a large system. Someone in "the other group" making a mistake can bring your part of the system down. Much of the code is written in C or C++, and so one "old school" programmer can make a mistake and write past their memory into a data structure you are using, and the bug report goes to you not them. If you have the above system, when splitting the monolith apart you will discover libfoo.so that both depend on, and the two groups will want to upgrade separately: containers allow this to happen, without modifying your build system to give them different versions or otherwise allow two different libraries with the same name to sit on your system. The above is what is obvious enough that I can talk about it. (I work on an embedded system at John Deere so I cannot comment more than the above even if you ask.)
hghid 3y ago
I'm all for improvements to pod startup times etc, but the general idea of putting more software into cars is not that appealing. I recently broke down in the highlands of Scotland in a fairly new car with the family - it was a horrible experience. It was made worse by the fact that there was nobody close that had a clue what to do with the car. The breakdown service arrived promptly, plugged the diagnostic tool into the car, proclaimed it broken, called a tow truck and left - two days later we arrived home. Had I been in a less complex car, a local garage could most likely have fixed the problem and sent us on our way. The sophistication and gadgets in modern cars are great until something goes wrong then they fail hard. Small local garages that used to be a life saver are next to useless now as they don't have the tools and knowledge to fix a mobile data centre.
- dtx1 3y ago
  That's imo a right to repair issue. We can build easily diagnosable and easily fixable hardware. Big Corps just don't
  - carlmr 3y ago
    It's also a liability issue. If a company allows tinkering with the software in the car it opens itself up to massive lawsuits. If we have a right to repair here, we also need to see how to handle liability here. If you flash your own software on the motor controller and subsequently mow through a group of people because you forgot to do a plausibility check on the accelerator pedal value who takes responsibility then? Even if you just get the original software, how do we ensure you flash it correctly? If you get the schematics, how do we know used the right parts that are rated for 125°C temperatures.
    - anaganisk 3y ago
      Lol, this is absolute funny, every example you came up with has already been there for years. Aren't cars being modded every day, ECU tuning, engine mods, etc? Go ahead sue the company, companies aren't some innocent babies, they can afford to quickly dismiss the claim by just pointing towards the mod. Auto Companies have never been held liable for a car that has been modded. Does it waste money to be sued? Yes! But does it save a lot of money for consumers and is much better for the environment? Yes! If companies are greedy/selfish about their profits then consumers don't need to think about how right to repair hurts those companies.
IceWreck 3y ago
Why would you run podman inside a vehicle's computer. Cool nonetheless.
- imp0cat 3y ago
  Perhaps you want to somehow isolate different parts of the car to protect against somehting like the CAN bus attack?
  - blueflow 3y ago
    Containers do isolation in userspace, its a pure software thing. Its not doing any kind of hardware isolation nor is it able to.
    - TheCondor 3y ago
      Don’t they use cgroups? It’s software but the kernel helps.
- ElectricalUnion 3y ago
  Well you can ignore all sorts of Compartmentalization and run everything in the same cgroup, same chroot, same user, like how it is on conventional x86_64/aarch64 computers. It just isn't safe.
perlgeek 3y ago
Do we know which cars specifically run their applications in containers?
- steveBK123 3y ago
  agreed, id like to know so I can avoid buying them
  - ElectricalUnion 3y ago
    Do we know which cars specifically run their applications without any sort of containers, just everything thrown in the same userland without any care for security?
    - stfutechbros 3y ago
      If the security of it is a concern, then it's already doing too much.
manojlds 3y ago
Why does a car even need to run containers? It's a known hardware, so why containers? Feels like lazy engineering.
- lmz 3y ago
  Isolation berween apps? Although not sure what that would buy over just having separate UIDs.
  - cowl 3y ago
    easier done with dedicated Controllers instead of one BIG controller that needs to containerise its software? Why does the rear camera and lights need to use the same controller as the Engine sensors? This way you even avoid the latest "CAN bus injection attack" that are using the lights connection to inject Key Crypto attacks. not everything needs to be integrated.
    - ElectricalUnion 3y ago
      > easier done with dedicated Controllers instead of one BIG controller This bring costs and supply chain issues, and we had plenty of supply chain issues earlier this year.
  - jerf 3y ago
    It buys you conformance to Conway's Law. The team building the media center is that much more certain that the climate control code is fully isolated, up to and including the ability to have their own fully isolated filesystem so updating a single library won't take anything else down (and updating a single library doesn't require buy-in from everybody who works on the car), and that they only communicate exactly and only on the published API specs and not via dropping undocumented files on the file system or other such things. (Or if they do, you have a place to see that they have a weird bind mount they really shouldn't, etc.) I wouldn't consider this a night & day change, but an incremental one. But a good incremental one overall; I wouldn't drop everything to implement this but I'd definitely see it as a good thing even in the absence of functionality improvements. There's other benefits too like being able to update just one container in case of some problem, and having the blast radius more thoroughly contained than it would be with everything installed into one big base system.
mongol 3y ago
I didn't know they used containers in cars. I guess it makes sense when you think about it but it always felt like more of an "enterprise" solution to me.
- pqb 3y ago
  AFAIK, Red Hat is working collaboratively with General Motors [0]. There is one article [1] on their blog regarding the containers in cars. I haven't found many technical information in it, but maybe you will find anything pleasing your eye. [0]: https://www.redhat.com/en/about/press-releases/red-hat-and-general-motors-collaborate-trailblaze-future-software-defined-vehicles https://www.redhat.com/en/about/press-releases/red-hat-and-g... [1]: https://www.redhat.com/en/blog/running-containers-cars https://www.redhat.com/en/blog/running-containers-cars
scientism 3y ago
This should be part of the "How to Overengineer Anything and Everything" series.
mkoubaa 3y ago
All this because nobody wants to change the ELF loader to better isolate dependencies of binaries
- IshKebab 3y ago
  I think it's more because Linux OS developers have never bothered to move away from Unix's "all apps get mixed together at absolute locations" filesystem model. If apps were like on Mac - self-contained directories that can be installed at any path - then Docker would probably be a footnote.
  - never_inline 3y ago
    > mixed together at absolute locations There is DT_RUNPATH probably since before I was born. The problem is it's not always utilised, distributions prefer to share libraries over isolating applications, and loading shared libraries isn't the only host-dependent thing done by application code. Also you realise that docker provides more functionality than a tarball, right?
    - mkoubaa 3y ago
      DT_RUNPATH is not nearly flexible enough to matter
      - never_inline 3y ago
        Can you elaborate what more function you were expecting? There's appImage and a variety of home directory package managers on *NIX platforms. None of them caught up.
        mkoubaa 3y ago
        I'm not really expecting anything, it's just my experience developing commercial desktop applications on Linux that you inevitably end up having a startup script that sets LD_LIBRARY path before the main process starts. And even then global symbols with the same name collide so you have to be really careful about what gets loaded into the process.
hamdouni 3y ago
Nice to see improvements in podman startup and how the team achieve it... Constraints (car env) drive creativity !
anaganisk 3y ago
Who said AI is killing dev jobs, now Devs have an alternative employment as a car mechanic. Drop kubernetes or VMs into the car and DevOps guys can also join. It would be so fun to hear "Umm your ingress seems to use older API, I have to update it for the gearbox to engage" and then see them run kubectl apply.
nickjj 3y ago
I'm happy to see improvements like this. In 2018 I opened a github issue around container startup time[0] with Docker. A couple of things have changed since that issue but generally speaking we are talking about ~5s (containers) vs 150ms (no containers) to start a realistic process that depends on networking and other things you'd expect in a typical web app. [0]: https://github.com/moby/moby/issues/38077 https://github.com/moby/moby/issues/38077
shrubble 3y ago
So it was poorly thought out or poorly designed all this time, then they fixed the obvious errors? That was my impression after reading the article...
- MuffinFlavored 3y ago
  makes you wonder what the aspects other than startup time probably look like?
- q845712 3y ago
  i don't know if they fixed _all_ the errors -- there's still apparently a bunch of containers running in somebody's car...
com2kid 3y ago
> we did was analyze what happens when Podman starts a container and why it takes so long. It turns out there was a lot of low-hanging fruit. I've done this analysis for lots of software before, Windows has a really nice tool called Process Monitor that I've used to find huge slow downs before. Point it at a process and it'll tell you the every bit of IO that the application does, and at that point you can just start digging and opening bugs. IMHO almost every piece of software of any significant size horribly misbehaves. Opening the same file again and again, writing out lots of tiny temp files, reading the same key from the registry (or some config file) over and over again , and worst of all, using fixed timeouts waiting for a task to complete instead of doing the work to make things properly event based. On that last note, timeouts instead of event callbacks (or some sort of signaling system) is the cause of so much slowness across our entire industry. If something you are doing on a computer takes a human noticable amount of time, and you aren't throwing around tens of GBs of data or waiting on a slow network, then you are flat out doing something wrong.
jensenbox 3y ago
I am very likely one of only a few people but it really irritates me when the term "fold" is used when they really mean "times". Folding a piece of paper (just like binary numbering) 6 times will provide you with a stack of 64 sheets. They did not have a performance increase of 64 times. This is identical to the idea of stating "magnitude" as being the number of times based on 10. How wrong am I?
- nunuvit 3y ago
  Haha I like that idea, but it's your own personal definition. "-fold" is multiplicative and it's used to flexibly change the part of speech since at least Old English [1]. You can think of folds as referring to the individual layers of something folded, rather than the action of folding. It can be either depending on context. I can make 64 folds (layers) by making 6 folds (actions). [1] https://www.etymonline.com/word/-fold https://www.etymonline.com/word/-fold
kobalsky 3y ago
podman rootless and startup speed was what lured me in, sadly after a couple of years I've switched back to docker, bit happily in rootless mode now too. podman works fine until it doesn't. My hypothesis is that it has some fundamental design philosophy that makes it brittle. Properly cleaning up doesn't exist in their vocabulary. For example, a cancelled download or image extraction can bring the whole thing down at the worst time, you have to hunt down the corrupted folder and remove so that anything works again. A failed compose startup can leave the network in a undefined state hard to diagnose and impossible to recover without wiping some folders within /run/user and killing some rogue processes manually. This is further cemented by the fact that a lot of minor issues are answered with: podman system reset, which reeks of rm -fr node_modules. docker was always a pleasure to work with, I still don't understand why I suffered with podman so long.
- pikelet 3y ago
  That's pretty much my experience! I've tried switching to podman a few times now and I really wanted to like it. Each time lasting a few months, and it always ends with frustration. At this point it's like btrfs for me. Perhaps it has improved, but I've been burned too many times, the trust is gone and I just will not go back. Some software just seems to have fundamental design issues (too fast, too early?), and when that's the case, more often than not it doesn't matter how many years of development go into it, it will always have problems. Docker isn't perfect. I wish they would put more development into rootless mode. But it has never given me the kind of issues podman has. It just does what I ask it to and gets out of the way.
polskibus 3y ago
Can podman be used instead of docker on Windows for minkube? Would it work faster than docker in such case?
bighoki2885000 3y ago
[dead]
44791i4 3y ago
[flagged]

403 points by DrinkWater 3y ago

crest 3y ago

Adding a `sleep $DELAY` to a startup script isn't something to brag about :-P. scnr

Hendrikto 3y ago
They removed it.

rwmj 3y ago

> If the backup camera or other sensors were to run as containers, we needed to improve the starting speed significantly. I think I see the problem already. Why does anyone think its a good idea to put everything in an embedded system into a container? Particularly as everything comes from a single vendor and so the usual argument about "but libraries are too hard!" doesn't apply.

GordonS 3y ago
Playing Devil's advocate here, but there are some good reasons you might do this. For example, Docker's "pull" system is great for updates, and means it's trivial to rollback to an earlier version if something went wrong. A Docker registry also means you can easily switch to another version when needed. You also get a supervision of containers, with automatic restarts (yes, I know you can do this nowadays with systemd :)
- qprofyeh 3y ago
  Yup, I can imagine complete images are easier to version-manage and to tag as minor/major/hotfix than each individual part of the stack. I think people underestimate the amount of software that is ALREADY running in their cars/airplanes/helicopters and even elevators.
- andix 3y ago
  Sure, but if you provide the possibility to deploy a lot of components separately, at some point they will run in different versions. And who knows if rear view camera 1.3.22-44 works with turn signal 4.86.233-stable and break pedal 0.6.9876-beta?
  - wiseowise 3y ago
    How is that different from shared libraries?
    - andix 3y ago
      It’s the same problem. But library authors usually take extra care to stay compatible. And if you create your own shared libraries, they are normally not deployed separately, usually you bundle them with your main executable.
    - solatic 3y ago
      Shared libraries are bundled into one container, then the one container is submitted to a gauntlet of tests (service / API-level tests, e2e tests, etc.). If you're submitting an embedded device to a gauntlet of tests, you're anyway "containerizing" all the shared libraries when you build the embedded image. Trying to build containers within the embedded image has questionable merit.
    - aaravchen 3y ago
      Because you can prove it. In the auto industry you can't just claim it conforms, you have to prove it does for all possible conditions. With containers enforcing that separation, you can prove it much more easily.
  - bluGill 3y ago
    Those 3 for the most part don't have to work together. They need to run on the same CPU without taking more than their allowed portion of the CPU. If they have to work together the communication protocol is clearly defined well in advanced and limited to exactly what they need to say. Thus we are reasonably sure if any one combination works all possible combinations will work. Even then there is typically higher level control to only release combinations that are tested to work together.
- ClumsyPilot 3y ago
  A statically compiled executable is even easier, and can be hosted on your webserver
  - aaravchen 3y ago
    But unfortunately they also become very space inefficient when a lot of processes need the same (relatively) large blocks of code. But if you carefully curate your containers to use the same base image that contains those libraries already, you don't have to duplicate it. Comparable would be to having an inode de-duplicating file systems, and deterministic binary generation. But it's hard to prove "the correctness" of inode de-duplicating file systems in extreme environments like auto is required to, and deterministic binary generation is hard to control 3ven if it is possible with the specific build tools (it usually isn't).
- rwmj 3y ago
  "yum", "apt" etc have registries, roll back etc and have for a good few decades.
  - talonx 3y ago
    <sarcasm>But..but..they are not "cool" </sarcasm> This forced application of new technologies into every possible domain can have real security and reliability consequences - and all because some VP somewhere decided they needed to use the latest shiny thing in cars or fridges or ACs or whatever.
    - doubled112 3y ago
      Imagine we could build a dishwasher that didn't throw water on the floor? Oh well, we got WiFi instead. That's fun, right? If a dishwasher needs a firmware update, I might simply argue it was defective. Not everything needs to be secure or updated constantly. It shouldn't have network access to begin with.
      - talonx 3y ago
        Exactly. When will vendors stop touting WiFi access as a "feature"?
  - ilyt 3y ago
    And when you upgrade ssl lib now 200 apps are not vulnerable to exploit, vs updating 200 different containers
- interactivecode 3y ago
  unless the update requires version updates on multiple services which means the versioning and rollback has the same effect as a more monolithic codebase. but with the added complexity of not knowing how it all impacts each other. In my experience updates to smaller services are often trivial, or for updates that are actually impactful it would be way easier to coordinate in a monolithic codebase.
  - steveBK123 3y ago
    yes my first job over 15 years ago was essentially micro services, and we ran into this type of problem.. trivial updates to individual services could be iterated extremely quickly systematic changes to behavior across services were so hard they became incredibly uncommon
andix 3y ago
I agree. Microservices are a great concept for a lot of problems. But now I constantly see way too small services. Every tiny piece of software gets its own service and its own software lifecycle (including versioning and deployment). And what happens now is, that you need a huge effort to integrate all those components. End to end system tests get much more important, but are still harder to do than simple unit/integration tests. traditional testing strategies start to get pointless, because most bugs now only appear when combining services in a production setup. Yes, development gets easier, because every team can just develop, without aligning too much with other teams. But the deployment/ops/acceptance step often gets impossibly complex.
- zokier 3y ago
  You are blaming cultural issues on technology. You have the same integration pains with libraries too. In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways
  - ilyt 3y ago
    The old method of forcing everyone to freeze at certain dependencies also means that any bug in those dependencies is fixed in all components at once. Obviously going too big here is problematic as it can slow it down when tens or hundreds of people are involved in every update, but going too small have similar problems, on top of generally more smaller services eating more resources. We don't need "front reflector LED setting app" being called from "lighting setting app" called from "car setting app", it can probably just be one service. Smaller services also mean more services to update if some commonly used lib gets a security bug. Updating SSL lib in big monolith is just update, run tests, but in microservices that's multiplied by amount of teams and services. > In general if your teams are not aligned, it doesn't really matter if they are developing microservices, libraries, or one spaghetti ball mess, its going to be problem anyways Moot point. We pick the tools to make the job easier. Good team with bad tools will still be slower and less efficient than good team with good tools.
  - goodpoint 3y ago
erinaceousjones 3y ago
To be honest I think we should be adopting the full ecosystem we've been busy building around containerization. Imagine calling up breakdown assistance because your car won't start, mechanic comes out, cracks the hood and is like "ah there's your problem right there, ignition service has only 1/2 pods healthy because the node went into NotReady due to DiskPressure. I can clear up some log files so it goes underneath 80% disk usage again but sucks teeth it's gonna cost ya. I'd recommend throwing the whole car out and getting a new one. You shouldn't get an attachment to these things, they're cattle not pets." Truly breathtaking.
- fredsted 3y ago
  Looks like your cars Kubernetes certificates have expired after a year, we'll need to SSH in and run kubeadm to refresh them. Wait, the 5G pod isn't starting ...
  - wpietri 3y ago
    You joke, but I decided I wanted to try out Kubernetes, so I set it up at home and moved some of my local services into it, one of them being my custom lighting automation software. Eventually my lights just stopped working, and after a lot of rummaging it turned out to be expired certificates. I promptly put the software back where it was before, removed Kubernetes, and decided there were better things for me to play around with.
- yellowapple 3y ago
  Can't wait to plug in an ODB2 scanner and get a k9s screen.
- deleted 3y ago
  [deleted]
ilyt 3y ago
Especially a thing that should be just "a device that puts video stream on whatever bus it uses". infotainment already uses html/js based UIs, just embed video player in there...
spyremeown 3y ago
https://www.toradex.com/torizon https://www.toradex.com/torizon they do and it works quite well. Super handy to have CI/CD and easy OTA updates with containers.
jpgvm 3y ago
I think if you open up a Tesla infotainment system you would find something eerily similar to containers. Remember that what you and most programmers think of containers is merely one possible assembly of a bunch of kernel features. There exists not just a gradient between plain processes and containers but a whole solution space with different tradeoffs. I happen to know a small amount of the Tesla internals and they are using cgroups, namespaces, app armour and ebpf based syscall filtering to secure various processes on the car. You almost certainly should not use docker or podman to manage processes on a car but that doesn't mean you shouldn't embrace the subsystems they are built on in order to increase security resilience and defense in depth.
vbezhenar 3y ago
Do you oppose running camera software in a separate process? I think it makes sense. Camera process might crash and will be restarted, this should not cause restart of the entire shell. What people should understand is that container in Linux is just a separate process running in powerful chroot (which isolates not just file tree, but also process tree and other things). So the same reasoning which applies to running some code in a separate process also applies to running some code in a separate container. I'd even argue that in an ideal world, almost every process should run in a separate container. The tooling is not here, but concept is just fine.
- garaetjjte 3y ago
  Containers usually ship their own libraries, which means less sharing, more disk usage and higher memory pressure.
  - vbezhenar 3y ago
    That depends on implementation. Shared libraries which use the same inode will be shared AFAIK. If containers use different libraries, they'll not be shared, of course, but that's a deliberate choice of container creator.
    - bonzini 3y ago
      Containers can use the same base image for the OS.
  - rusk 3y ago
    I see these as a relatively straightforward set of problems to identify, quantify, and remedy. It’s a tradeoff between static memory usage and stability. If that additional memory footprint becomes an issue you can make plans to align dependencies.
  - goodpoint 3y ago
    And much lower chances of getting security updates, now that everything is a huge blob.
deleted 3y ago
[deleted]
deleted 3y ago
[deleted]
galangalalgol 3y ago
Everything never comes from a single vendor. Even microcontrollers the hardware abstraction layer is hiding peripheral support packages for all the i2c and spi things it talks to, each from a different vendor potentially. I'm not saying we should be running k8 on an 8051 or even a cortex m33, but on an arm7? Maybe. Cult of Ferris time, static linking in rust means your binary is your container, particularly if you statically link in musl.
bluGill 3y ago
Not everything is developed by the single vendor. Sometimes they buy a program from a vendor they don't fully trust. There is also security, I don't care if someone hacks my radio system nearly as much if someone hacks the brake system. Containers is one part of the total package to isolate parts so if there is a hack the whole system isn't taken out. Computers are a large system. Someone in "the other group" making a mistake can bring your part of the system down. Much of the code is written in C or C++, and so one "old school" programmer can make a mistake and write past their memory into a data structure you are using, and the bug report goes to you not them. If you have the above system, when splitting the monolith apart you will discover libfoo.so that both depend on, and the two groups will want to upgrade separately: containers allow this to happen, without modifying your build system to give them different versions or otherwise allow two different libraries with the same name to sit on your system. The above is what is obvious enough that I can talk about it. (I work on an embedded system at John Deere so I cannot comment more than the above even if you ask.)

hghid 3y ago

I'm all for improvements to pod startup times etc, but the general idea of putting more software into cars is not that appealing. I recently broke down in the highlands of Scotland in a fairly new car with the family - it was a horrible experience. It was made worse by the fact that there was nobody close that had a clue what to do with the car. The breakdown service arrived promptly, plugged the diagnostic tool into the car, proclaimed it broken, called a tow truck and left - two days later we arrived home. Had I been in a less complex car, a local garage could most likely have fixed the problem and sent us on our way. The sophistication and gadgets in modern cars are great until something goes wrong then they fail hard. Small local garages that used to be a life saver are next to useless now as they don't have the tools and knowledge to fix a mobile data centre.

dtx1 3y ago
That's imo a right to repair issue. We can build easily diagnosable and easily fixable hardware. Big Corps just don't
- carlmr 3y ago
  It's also a liability issue. If a company allows tinkering with the software in the car it opens itself up to massive lawsuits. If we have a right to repair here, we also need to see how to handle liability here. If you flash your own software on the motor controller and subsequently mow through a group of people because you forgot to do a plausibility check on the accelerator pedal value who takes responsibility then? Even if you just get the original software, how do we ensure you flash it correctly? If you get the schematics, how do we know used the right parts that are rated for 125°C temperatures.
  - anaganisk 3y ago
    Lol, this is absolute funny, every example you came up with has already been there for years. Aren't cars being modded every day, ECU tuning, engine mods, etc? Go ahead sue the company, companies aren't some innocent babies, they can afford to quickly dismiss the claim by just pointing towards the mod. Auto Companies have never been held liable for a car that has been modded. Does it waste money to be sued? Yes! But does it save a lot of money for consumers and is much better for the environment? Yes! If companies are greedy/selfish about their profits then consumers don't need to think about how right to repair hurts those companies.

IceWreck 3y ago

Why would you run podman inside a vehicle's computer. Cool nonetheless.

imp0cat 3y ago
Perhaps you want to somehow isolate different parts of the car to protect against somehting like the CAN bus attack?
- blueflow 3y ago
  Containers do isolation in userspace, its a pure software thing. Its not doing any kind of hardware isolation nor is it able to.
  - TheCondor 3y ago
    Don’t they use cgroups? It’s software but the kernel helps.
ElectricalUnion 3y ago
Well you can ignore all sorts of Compartmentalization and run everything in the same cgroup, same chroot, same user, like how it is on conventional x86_64/aarch64 computers. It just isn't safe.

perlgeek 3y ago

Do we know which cars specifically run their applications in containers?

steveBK123 3y ago
agreed, id like to know so I can avoid buying them
- ElectricalUnion 3y ago
  Do we know which cars specifically run their applications without any sort of containers, just everything thrown in the same userland without any care for security?
  - stfutechbros 3y ago
    If the security of it is a concern, then it's already doing too much.

manojlds 3y ago

Why does a car even need to run containers? It's a known hardware, so why containers? Feels like lazy engineering.

lmz 3y ago
Isolation berween apps? Although not sure what that would buy over just having separate UIDs.
- cowl 3y ago
  easier done with dedicated Controllers instead of one BIG controller that needs to containerise its software? Why does the rear camera and lights need to use the same controller as the Engine sensors? This way you even avoid the latest "CAN bus injection attack" that are using the lights connection to inject Key Crypto attacks. not everything needs to be integrated.
  - ElectricalUnion 3y ago
    > easier done with dedicated Controllers instead of one BIG controller This bring costs and supply chain issues, and we had plenty of supply chain issues earlier this year.
- jerf 3y ago
  It buys you conformance to Conway's Law. The team building the media center is that much more certain that the climate control code is fully isolated, up to and including the ability to have their own fully isolated filesystem so updating a single library won't take anything else down (and updating a single library doesn't require buy-in from everybody who works on the car), and that they only communicate exactly and only on the published API specs and not via dropping undocumented files on the file system or other such things. (Or if they do, you have a place to see that they have a weird bind mount they really shouldn't, etc.) I wouldn't consider this a night & day change, but an incremental one. But a good incremental one overall; I wouldn't drop everything to implement this but I'd definitely see it as a good thing even in the absence of functionality improvements. There's other benefits too like being able to update just one container in case of some problem, and having the blast radius more thoroughly contained than it would be with everything installed into one big base system.

mongol 3y ago

I didn't know they used containers in cars. I guess it makes sense when you think about it but it always felt like more of an "enterprise" solution to me.

pqb 3y ago
AFAIK, Red Hat is working collaboratively with General Motors [0]. There is one article [1] on their blog regarding the containers in cars. I haven't found many technical information in it, but maybe you will find anything pleasing your eye. [0]: https://www.redhat.com/en/about/press-releases/red-hat-and-general-motors-collaborate-trailblaze-future-software-defined-vehicles https://www.redhat.com/en/about/press-releases/red-hat-and-g... [1]: https://www.redhat.com/en/blog/running-containers-cars https://www.redhat.com/en/blog/running-containers-cars

scientism 3y ago

This should be part of the "How to Overengineer Anything and Everything" series.

mkoubaa 3y ago

All this because nobody wants to change the ELF loader to better isolate dependencies of binaries

IshKebab 3y ago
I think it's more because Linux OS developers have never bothered to move away from Unix's "all apps get mixed together at absolute locations" filesystem model. If apps were like on Mac - self-contained directories that can be installed at any path - then Docker would probably be a footnote.
- never_inline 3y ago
  > mixed together at absolute locations There is DT_RUNPATH probably since before I was born. The problem is it's not always utilised, distributions prefer to share libraries over isolating applications, and loading shared libraries isn't the only host-dependent thing done by application code. Also you realise that docker provides more functionality than a tarball, right?
  - mkoubaa 3y ago
    DT_RUNPATH is not nearly flexible enough to matter
    - never_inline 3y ago
      Can you elaborate what more function you were expecting? There's appImage and a variety of home directory package managers on *NIX platforms. None of them caught up.
      - mkoubaa 3y ago
        I'm not really expecting anything, it's just my experience developing commercial desktop applications on Linux that you inevitably end up having a startup script that sets LD_LIBRARY path before the main process starts. And even then global symbols with the same name collide so you have to be really careful about what gets loaded into the process.

hamdouni 3y ago

Nice to see improvements in podman startup and how the team achieve it... Constraints (car env) drive creativity !

anaganisk 3y ago

Who said AI is killing dev jobs, now Devs have an alternative employment as a car mechanic. Drop kubernetes or VMs into the car and DevOps guys can also join. It would be so fun to hear "Umm your ingress seems to use older API, I have to update it for the gearbox to engage" and then see them run kubectl apply.

nickjj 3y ago

I'm happy to see improvements like this. In 2018 I opened a github issue around container startup time[0] with Docker. A couple of things have changed since that issue but generally speaking we are talking about ~5s (containers) vs 150ms (no containers) to start a realistic process that depends on networking and other things you'd expect in a typical web app. [0]: https://github.com/moby/moby/issues/38077 https://github.com/moby/moby/issues/38077

shrubble 3y ago

So it was poorly thought out or poorly designed all this time, then they fixed the obvious errors? That was my impression after reading the article...

MuffinFlavored 3y ago
makes you wonder what the aspects other than startup time probably look like?
q845712 3y ago
i don't know if they fixed _all_ the errors -- there's still apparently a bunch of containers running in somebody's car...

com2kid 3y ago

> we did was analyze what happens when Podman starts a container and why it takes so long. It turns out there was a lot of low-hanging fruit. I've done this analysis for lots of software before, Windows has a really nice tool called Process Monitor that I've used to find huge slow downs before. Point it at a process and it'll tell you the every bit of IO that the application does, and at that point you can just start digging and opening bugs. IMHO almost every piece of software of any significant size horribly misbehaves. Opening the same file again and again, writing out lots of tiny temp files, reading the same key from the registry (or some config file) over and over again , and worst of all, using fixed timeouts waiting for a task to complete instead of doing the work to make things properly event based. On that last note, timeouts instead of event callbacks (or some sort of signaling system) is the cause of so much slowness across our entire industry. If something you are doing on a computer takes a human noticable amount of time, and you aren't throwing around tens of GBs of data or waiting on a slow network, then you are flat out doing something wrong.

jensenbox 3y ago

I am very likely one of only a few people but it really irritates me when the term "fold" is used when they really mean "times". Folding a piece of paper (just like binary numbering) 6 times will provide you with a stack of 64 sheets. They did not have a performance increase of 64 times. This is identical to the idea of stating "magnitude" as being the number of times based on 10. How wrong am I?

nunuvit 3y ago
Haha I like that idea, but it's your own personal definition. "-fold" is multiplicative and it's used to flexibly change the part of speech since at least Old English [1]. You can think of folds as referring to the individual layers of something folded, rather than the action of folding. It can be either depending on context. I can make 64 folds (layers) by making 6 folds (actions). [1] https://www.etymonline.com/word/-fold https://www.etymonline.com/word/-fold

kobalsky 3y ago

podman rootless and startup speed was what lured me in, sadly after a couple of years I've switched back to docker, bit happily in rootless mode now too. podman works fine until it doesn't. My hypothesis is that it has some fundamental design philosophy that makes it brittle. Properly cleaning up doesn't exist in their vocabulary. For example, a cancelled download or image extraction can bring the whole thing down at the worst time, you have to hunt down the corrupted folder and remove so that anything works again. A failed compose startup can leave the network in a undefined state hard to diagnose and impossible to recover without wiping some folders within /run/user and killing some rogue processes manually. This is further cemented by the fact that a lot of minor issues are answered with: podman system reset, which reeks of rm -fr node_modules. docker was always a pleasure to work with, I still don't understand why I suffered with podman so long.

pikelet 3y ago
That's pretty much my experience! I've tried switching to podman a few times now and I really wanted to like it. Each time lasting a few months, and it always ends with frustration. At this point it's like btrfs for me. Perhaps it has improved, but I've been burned too many times, the trust is gone and I just will not go back. Some software just seems to have fundamental design issues (too fast, too early?), and when that's the case, more often than not it doesn't matter how many years of development go into it, it will always have problems. Docker isn't perfect. I wish they would put more development into rootless mode. But it has never given me the kind of issues podman has. It just does what I ask it to and gets out of the way.

polskibus 3y ago

Can podman be used instead of docker on Windows for minkube? Would it work faster than docker in such case?

bighoki2885000 3y ago

[dead]

44791i4 3y ago

[flagged]