18 ms·

“It works on my machine” turns to “it works in my container” (2019)

• somat 3y ago
  Programmer: "I don't know whats wrong, it works on my machine" Manager: "Fine, then we will ship your machine" And thus docker was born.
  • salawat 3y ago
    I have never been able to realize the alleged ergonomic gains of containers. Ever. It always adds more friction to actually getting something initially stood up, prototyped, and deployed. I'm guessing it may be one of these things where it only starts to make sense after something has matured enough to warrant being replicated en-masse in a data-center environment. Then again, I tend to live in a world where I'm testing the ever-loving crap out of everything; and all that instrumentation has to go somewhere!
    • turtlebits 3y ago
      Being able to run self contained postgres with a single command is an easy ergonomic win. You don't necessarily need to be an engineer or write code for containers to be useful.
      • ilyt 3y ago
        How you inform your backup system where to get backups ? How you set pg_hba and other configs? Few other "how?" and you're doing what you'd be doing on VM anyway
        • deleted 3y ago
          [deleted]
        • rad_gruchalski 3y ago
          > How you inform your backup system where to get backups ? How you set pg_hba and other configs? Simple answer: with configuration. In Docker Compose or Kubernetes. Less often in Mesos. Maybe I want to run it on a fleet of VMs, maybe on bare metal. > and you're doing what you'd be doing on VM anyway Right. But with containers I can have different apps using different dependency versions. Some things use nginx, some use some other web server. Some things run with node 12, some with node 16, some things use MySQL, some other things need Postgres. This is so easy with containers.
      • saurik 3y ago
        So my issue is that now I have PostgreSQL running... inside a container? So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up. And like, that's just minimum: I generally care deeply about the filesystem that PostgreSQL is running on and I will want to ensure the transaction log is on a different disk than the data files... and I now have to configure all of this stuff multiple times. At some point I am going to have to edit the configuration file for PostgreSQL... is it inside of the container? Did I have to manually map it from the outside of the container? The way you access PostgreSQL locally--maybe if you find yourself ready to add a copy of pgbouncer, but also just to run the admin tools--is via unix domain sockets. Are those correctly mapped outside of the container, and where did I put them? I honestly don't get it for something like PostgreSQL. I even use containers, but I can only see downsides for this particular use case. You know how easy it is to run PostgreSQL in some reasonable default confirmation? It is effectively 0 commands as you just install it and your distribution generally already has it running.
        • rad_gruchalski 3y ago
          > So my issue is that now I have PostgreSQL running... inside a container? So I need to figure out where the data for it is stored and figure out how to ensure it is being backed up. And how is that different from running directly on the OS?
          • serf 3y ago
            the OS will have an understood abstraction behind the filesystem structure, whereas container-systems often create an entirely new abstraction that they view as the best way. this makes sense when you're trying to be deployable universally, but it increases the amount of onboarding that someone needs to receive before they're proficient with the container system; onboarding they may have not actually needed to get the software working and well understood, simply 'docker overhead'. from personal experience : i'm a long time old linux person, the insistence on going 'all in' on Docker (or whatever) just to run a python script that has two or three common shared dependencies gains me nothing but the hassle of now having to maintain and understand a container system. if you're shipping truly fragile software that is dependent on version 1.29382828 rather than version 1.29382827 then I understand the benefits gained, but just to containerize something very simple in order to follow industry trends is obnoxious, increasingly common, and seemingly has soured a lot of people on a good idea. p.s. : I can also understand the idea of containerizing very simple things as parts of a larger mechanism; I just don't get it with the promise that it'll reduce end-user complexity, it isn't that simple.
      • lelanthran 3y ago
        > Being able to run self contained postgres with a single command is an easy ergonomic win. You don't necessarily need to be an engineer or write code for containers to be useful. Postgres is a poor example; it's far far easier to do `apt-get install postgresql` than to run it from a container. The latter needs a container set up with correct ports, plus it needs to be pointed at the correct data directory before it is as usable as the former.
        • imtringued 3y ago
          Is this some kind of joke? Those things are absolutely trivial and once you wrote them in your bash script or compose file you can forget about these things.
    • j1elo 3y ago
      For me, it solved A LOT of cases where users said "your software install fails", "the software doesn't run", or similar claims. They tend to be complicated to troubleshoot: people try setting up new software in an already customized development environments, and all sorts of mistakes are done. It was refreshing being able to say: this works perfectly from a base Ubuntu 20.04 container, launching this and that commands to install and run the software. Anything that diverges from there, you're on uncharted territory and/or is a problem in your system, not a defect in the software.
    • ilyt 3y ago
      Well there is one unarguable one, although with consequences. Instead of keeping some old machine with old dependencies because project is on life support and doing update to latest version of language/tools is unviable, you can just keep a docker container for it while machine it is running on is up to date. It doesn't solve the problem of application, but it is no longer ops problem that app is old. You can also do that partially, like keep old PHP version running with just fcgi socket exposd in a container, while rest of the app lives on "normal" VM (or other container, if that's what you want).
      • MilStdJunkie 3y ago
        That's what got me on the Docker kick. Old unsupported programs that are tightly coupled to a kernel version, a Windows thing, some combination of Java / JDK shenanigans, an old SoundBlaster DLL, "The Python Carnival", or, really, what-have-you. Honestly, every big ERP demo ever should be done via container, because - and I swear I am not making this up - I have sat in months-long meetings with dozens and dozens of staff, both mine and theirs, trying to get Big ERP X up and running on-prem under whatever weird-ass policies the customer has on all their environments. Granted, a lot of THAT benefits from the fact that the container actually has to be a documented flavor of good solid *NIX, unlike the "Windows Image X" which is usually just a big "whatever" of old iron, management fad spoor, and seven different kinds of antivirus software. Hell, last time I did this, IT couldn't even find me any kind of description of what the "official" windows image actually was. So no fault of Windows there, it's just that, as the big bus everyone rides in, it gets all the goop from everyone wrenching on it.
        • deleted 3y ago
          [deleted]
        • pxc 3y ago
          > seven different kinds of antivirus software The last time I counted up all the different persistent, security-related agents running on my corporate Windows box, it was actually 14.
    • _8j50 3y ago
      To me, containers are like OOP. Nice ideas on paper but a pain the ass in practice. but again, like OOP, a lot of people do great things with them but I don't think they can't do the same things with VMs. My theory is eventually things will come full circle and there will entire app ecosystems running in a unikernel executable which is running on physical hardware and the hardware itself is segmented. I can imagine companies like netflix and cloudflare having racks of servers with 1TB ram with no disk, booting a unikernel from the network for maximum throughput and latency and then everyone in tech follows along.
      • imtringued 3y ago
        I think you don't understand the problem. What people want is the ability to do all these things with a single command that they put inside a bash script.
    • golergka 3y ago
      Have you tried setting up a python project with native dependencies on a different version of OS than original developer?
      • auggierose 3y ago
        If you cannot even set up your dependencies, a container might look like a solution, but I can assure you, it isn't.
        • golergka 3y ago
          It is the best solution. In my particular situation, at last. Another solution would find a senior developer who can manage his dependencies properly, but I can't afford him. However, I can pay a junior ML developer wage and just throw what he develops in a docker image instead of wasting time and money on doing things the hard way for a prototype. As engineers, we're supposed to be rational people, but unfortunately, we tend to forget that time and money are often happen to be the most important constraints we have to work around.
      • twelve40 3y ago
        No, i haven't. (not OP but) The last decade or more 90% of people i collaborated with were on the same operating system. Which moves the docker development setup solidly into "nice to have" category for me, especially when working on small teams <10.
        • golergka 3y ago
          Things easily break even between minor releases of the same system.
    • Karunamon 3y ago
      I guess we live in entirely different worlds, because, for me, containers have been a godsend. Standing up the average application is a matter of reading a compose file and modifying it according to instructions, or using it to construct a proper manifest for something like k8s for full production. For toy/evaluation use, it's hard to beat "tweak compose file, docker-compose up". You now have a golden source of truth for how the entire application and all of its moving parts were set up. Going back to deploying applications on bare metal feels positively medieval by comparison. From the admin side, configuration drift is effectively not a thing anymore. That's huge.
      • wruza 3y ago
        It feels like admins just offloaded the job of deploying apps back to developers. And as a developer I woudn’t even mind, if not for the f-ing slowdown of, well, the pace of development. All external teams that I worked with and who were using containers for development were like “oh you need that trivial endpoint or a bugfix? No problem, we’ll do it in few hours and it will autodeploy tomorrow”. And I didn’t ask why, because I know. As if we weren’t snail-paced money burners already. Provided all that, I don’t get why “devops” is even a high-paying job at most places except few really web scale ones. They are basically former sysadmins who reject anything except plugging colored squares into square sockets.
        • ChoHag 3y ago
          [dead]
    • rad_gruchalski 3y ago
      I'm on the other end of the spectrum. I don't want to deal with written instructions. A shell program will do but a container is the best. Give me an environment with all dependencies, software, and some decent configuration method. A config file over a volume will do, env vars are better. Dockerfile tells me everything I need to know about the environment the app requires. I can run that container in CI, in tests (with stuff like testcontainers), on my laptop, in production, on Kubernetes, in Mesos, Lambda, whatever. Instrumentation usually ends up in Prometheus and Jaeger.
    • Danjoe4 3y ago
      You answered your own question. When you join a large project and there's a perfect dev environment available in 3 minutes with `make init`, it makes sense. If you're the solo dev, it doesn't make sense to spend 5 hours figuring out the docker build system.
      • earthling8118 3y ago
        In my experience it never ends up being this simple. At least not with a combo of make and docker.
        • ehnto 3y ago
          I have yet to be handed a containerized dev env that "just works". One company came close, they had someone dedicated to maintaining the dev tooling including the containers. It still didn't "just work" but they handled the troubleshooting, and the fix made it into the repo for the container so it wasn't just an unwritten adhoc fix I needed to remember. Close enough. I totally get the concept, I wish it worked so seamlessly that the promise was realized. But since I usually work with tooling I am familiar with, there is no point for me personally. I can stand up a local env faster than I can troubleshoot a broken container. Since I am not interested in the infrastructure and just want to get to work, that is what I do.
      • computerfriend 3y ago
        Every time I see this, I peek at the init target and it's always doing something weird like writing to my SSH config or something. I have learned to never trust installers written by colleagues.
    • flatline 3y ago
      Instead of composing libraries into an executable you can use containers to compose services into an application [0]. As noted in the sibling comment, this still can have a high utility for just a single workstation/service. Also as you noted, scaling to a datacenter is already partly automated. The entire environment is captured in code and reproducible/portable. But you need to be familiar with the tools and ecosystem to make use of them. [0] We don’t have concise terms for the types of distributed, complex systems which have become standard.
      • muxator 3y ago
        > Instead of composing libraries into an executable you can use containers to compose services into an application This is the most concise and exact definition I've heard lately. My only problem with the consequences of this approach is that the amount of overhead for performing the same operations is astonishing. Messages pass on a network instead of stsying in RAM, CPU context switches every other ms to handle IO for what could have been a simple function call, etc. It's amazing when it's needed, but seeing this approach becoming the standard for a big part of the industry almost turns it into an environmental problem. How much energy is wasted juggling bits around?
        • lmz 3y ago
          It doesn't have to be a library replacement. It could be that your app uses a DB server and a message queue for async tasks. Containers allow you to easily run those servers locally for dev if you don't care about performance and maybe in prod you'll have a dedicated server for them, or use some DB as a service offering.
          • muxator 3y ago
            Yeah, agreed: in that case spinning a container is evidently an operational advantage. However, I was specifically answering to the different case highlighted by the parent comment: how a potentially cohesive application is cut along some of its internal APIs, and some functionality is allocated to different processes living in different containers. It is an extreme point in the continuum "single thread" -> "multi thread" -> "multiple processes" -> "fully distributed". In that continuum scalability increases, while efficiency progressively decreases. Cornering oneself to a specific point in the design space is problematic, and for some cases has direct implications on how many resources are wasted. A very didactic experience is, for example, running a simple local application under a microarchitecture profiler, such as Intel Vtune. It is not uncommon seeing that even straightforward C/C++ programs use a core resources less than 10%. What I am reflecting about is that the choice of fragmenting that program among tens of systems (maybe in a scripting langiage) should be conscious, and done after encountering performance or scalability bottlenecks. How much of the resulting total system workload would be useful work? The quantity of potentially wasted resources is astonishing if you think about it.
    • lmm 3y ago
      They make sense if you're using Python, because Python dependency management is awful. For anything else they're not really worth it IME.
      • coderenegade 3y ago
        Not at all. They make cross compiling fairly trivial, and give teams a way of managing cross compiled dependencies. This was vital a few years ago for things like torch and even opencv, where deploying on anything other than x86 meant you were probably on your own, and you almost certainly wanted control over the compiler flags. That's just one example, though. It also lets you standardize the dev environment (work directly out of the build container directly, or use your host OS and run unit tests in a local build), and it allows for easy, standardized testing within the same environment.
        • lmm 3y ago
          > Not at all. They make cross compiling fairly trivial, and give teams a way of managing cross compiled dependencies. How so? A container is the same architecture as the host, how does that help with cross compiling?
          • coderenegade 3y ago
            You combine qemu with docker. You can then run arm containers and compile your code locally for deployment to an arm board.
            • lmm 3y ago
              Qemu is the part that's making that possible though. What's docker adding?
              • imtringued 3y ago
                The ability to kickstart the whole process with a few commands, possibly just a single bash script.
    • time0ut 3y ago
      I can’t live without it. The power of docker compose for the development process is unparalleled. Being able to declare my entire local environment in a single file in a consistent way, create and destroy it with a simple command, and share it with my co-workers eliminates so many issues. Then being able to package my application in way that is repeatable and predictable is great. Then that same artifact is run across multiple environments spanning data centers around the world and on other developers’ machines via their compose files if needed. We test the crap out of all of this as well. Docker helps there to generate ephemeral environments to run the bulk of the integration tests against every time we push to git. Its just so damn powerful compared to what we had before…
      • stavros 3y ago
        Agreed, and the fact that I can launch and develop my web app from seven years ago, no matter how much my environment has changed, is a lifesaver. There's a lot of code I've written that's not deployable any more because the versions of the dependencies it was using don't exist on my OS any more, and are too old to install.
        • deleted 3y ago
          [deleted]
        • friendzis 3y ago
          As long as you have saved the container image and contents of all mapped volumes somewhere. Which is not much different from snapshotting a VM.
          • imtringued 3y ago
            Snapshotting a VM is a pain in the ass. It takes forever and the tooling sucks. The concept of snapshotting is also completely wrong. Snapshots are tied to a VM with the specific VM being front and center, you know, the thing that is disposable with containers. Also, from my misadventures with Ubuntu VMs I always ran into the problem of them breaking over time.
      • 3y ago
    • marcosdumay 3y ago
      It solves the "we can't possibly package the software for all versions of libc out there" and the "there are all those programs that must run at the same time, it's hard to correctly install them all" problems. People push it to solve the "this program needs executable A version 1, that program needs executable A version 2, both programs must run at the same time" one, that should never be a big issue anyway. But then, we have none of those problems at work and people still believe it's a panacea. I don't know what people believe they gain by it.
      • 8organicbits 3y ago
        Dependency management is one thing Docker handles, and I don't really think it does a great job of that. To me containers solve the challenge of getting a development environment that closely matches production. It standardizes how deployments are performed, how testing environments are set up. Before this we had complex, immutable ansible scripts, READMEs, install scripts that had to be babysat, snowflake servers that no-one knew how they were configured. It's also an extremely widely used technology, so if you use it in your stack you can expect many people you hire to know how to use it from day one versus everyone learning a bespoke set of custom scripts.
      • jjjfdjunnmko23 3y ago
        This was already trivially solvable by establishing a new sysroot, which in all fairness is 90% of the core benefit of containers in my view. The rest of the container features are bloat for most of my use cases.
      • nerdponx 3y ago
        It also solves the "we have limited developer-hours available and just want to deploy some stuff" problem. Docker is amazing for teams with limited resources. If you can deploy one image, you can deploy any image.
        • marcosdumay 3y ago
          That's my point. I hear a lot of what you say, but I don't understand it. In general, without some kind of qualification, deploying an image is way worse than a flat binary, or a tar from some interpreted language.
    • hinkley 3y ago
      Containers aren't about you. They're about the person you're trying to hand things to. I got stuck creating VMs for testers for a distributed workflow (several services, several tools), and keeping those VMs up to date and working was a PITA, but far less painful than dealing with them filing bug reports based on thinking they were doing X when they were doing Y. I ended up creating a workflow that felt like layers, so when Docker came along I didn't need a salespitch. I could just do what I'd done with a script instead of a runbook. Where do I sign up?
      • paulddraper 3y ago
        Exactly. Containers are packaging for executables.
        • transcriptase 3y ago
          Another decade or two and perhaps the Unix community will finally reinvent the .exe or .app
          • coderenegade 3y ago
            It's more than just creating a distributable, though. It's also a dev fixed environment and a cross-compiler when paired with qemu, and a test suite all rolled into one. It's hugely powerful for how simple it is.
          • paulddraper 3y ago
            That's just.......wrong. MacOS makes static linking quite difficult. And Windows is infamous for "DLL Hell."
            • pjmlp 3y ago
              Windows 9X/2000/NT was long time ago, nowadays that it more a GNU/Linux problem than Windows one.
          • nerdponx 3y ago
    • imtringued 3y ago
      I have an app that needs a media server, a webrtc STUN/TURN server, a PostgreSQL database, a JVM based web server, a worker that may or may not run on the same host and a way to run liquibase for migrations and certbot for letsencrypt certificates. The whole app can be started with a single command and it works on most Linux distributions. I can't imagine wasting the time of users or newbie developers demanding that they install all these things separately and with no easy way to clean it up if they want to undo everything.
  • andrewedstrom 3y ago
    Honestly, a pretty reasonable solution to that problem. It's cool that we have the technology to make that work.
  • treeman79 3y ago
    Owner hired an extremely “senior” developer. Was told to let him do his thing. After he spent 3 months building a web app, I asked him how he wanted to deploy it. Perfectly straight face he said we would take his developer machine to a local data center and plug it in. We could then buy him a new developer machine. It went downhill from there. I ended up writing the application from scratch and deploying it that same evening. Owner hired a lot Of strange people.
  • bandrami 3y ago
    An idea meant to lighten the load on sysadmins now means I have seven different OS versions to worry about
    • coding123 3y ago
      I mean 10-15 years ago we had developers that copied their JAR from their system to prod through the network share.
      • piyh 3y ago
        We used to, we still do too
      • Ygg2 3y ago
        I did that 5 years ago.
  • dunham 3y ago
    > "I don't know whats wrong, it works on my machine" I had one of these years ago where QA had an issue that I couldn't reproduce. I walked over to his desk, watched the repro and realized that he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select.
    • voyagerfan5761 3y ago
      I do not even want to know what sort of weird-ass GUI code or behavior made those two operations dissimilar.
      • dunham 3y ago
        Yeah, it's been way too long to remember (maybe a decade ago). Best guess is the click event was propagating up the dom and triggering something. These days, it looks like chrome generates a click even if you don't let up on the mouse.
    • ParetoOptimal 3y ago
      > he was someone who clicked to open a dropdown and then clicked again to select, while I would hold the mouse button down and then let up to select. I didn't know the latter was possible, but I also don't know which of these I do.
  • RF_Savage 3y ago
    Friend found some developers vacation photos on an industrial controller. Turns out they did ship a 1:1 image of his machine.
  • marcus_holmes 3y ago
    We used to do literally this back in the day. Dev would get the thing working on their machine configured for a customer. We'd take their machine and put it in the server room, and use it as the server for that customer. Dev would get a new machine. Yes, I know it's stupid. But if it's stupid and it works, it isn't stupid. DLL Hell was real. Spending days trying to get the exact combination of runtimes and DLL's that made the thing spring into life wasn't fun, especially with a customer waiting and management breathing down our necks. This became the easiest option. We started speccing dev machines with half an eye on "this might end up in the server room".
    • salawat 3y ago
      ...what? nm -D is not hard. Debugging missing symbols isn't even that hard. Biggest damn challenge I see is we've completely dropped the ball at educating people about how linkers, loaders, and dynamic library symbols work. ...and so people understand something here... I've transplanted/frankensteined userspaces that were completely hosed/disjoint into working before. In fact, every now and again I do it just to remain in practice. It's actually gotten to the point I don't even worry about dependency hell anymore. I just find the right version of the library/source and expand my archives just in case I need it down the road.
  • rahoulb 3y ago
    That's basically what Smalltalk was back in the 20th Century. The OS, the development environment and the application (both code and live objects) where one and the same thing. To ship an "app" you would export the image and the user would load it into their Smalltalk VM.
• Ilasky 3y ago
  This is something I've been trying to fix with PingQuick[0]. I got tired of spinning up containers, dealing with backends and setups only for it to be broken somewhere along the line, which then turns into me 4 hours deep in googling docker commands. I just want my code somewhere that someone else can ping it with no setup - that's it. [0] https://www.pingquick.dev https://www.pingquick.dev
  • rad_gruchalski 3y ago
    I'd love to hear one of your war stories.
  • blown_gasket 3y ago
    Is this different than functions-as-a-service?
  • xwowsersx 3y ago
    FYI this doesn't seem to work (Chrome on Android). When I click the button, it says "creating..." and then.... nothing
• xrd 3y ago
  There are definitely footguns with docker. I still feel like if I build the image I have a lot more control over it than when I'm trying to document build scripts, create the right package files or lock files, etc. I don't buy this argument and stopped reading when it was obvious that these issues could be solved by using a clearly specific build tag for the base image, etc.
• ilyt 3y ago
  That's the thing I like about self-contained binaries (Of Go or any other sort). Just FROM scratch COPY this-or-that LABEL prometheus.port=9100 LABEL prometheus.path=/metrics EXPOSE 3001 EXPOSE 9100 and nothing breaks. Only feeble component is CA bundle for SSL-related stuff as that by nature is changeable.
  • lmm 3y ago
    Why bother with a container at that point? Doesn't it introduce as many problems as it solves?
    • deleted 3y ago
      [deleted]
    • femiagbabiaka 3y ago
      The same reason you'd use one to begin with, primarily isolation of processes. That doesn't go away just because the binary is statically compiled. But you don't have to, of course, plenty of people don't.
    • xyst 3y ago
      Now you are back to the “it works on my machine” issues. Seen a couple of times where a precompiled binary works okay on one OS but the behavior changes when ran on another OS. In my case, the bottom bid contracting firm that delivered the code had special logic for windows but otherwise wouldn’t happen on unix based machines.
      • lmm 3y ago
        > Now you are back to the “it works on my machine” issues. Seen a couple of times where a precompiled binary works okay on one OS but the behavior changes when ran on another OS. Sure, but containers don't protect you from that - you're still exposed to differences in the host kernel. For that kind of issue you'd want a full VM rather than a container.
        • 3y ago
• dsr_ 3y ago
  All of these problems are about dependencies. And dependencies are about the way that we went from a blank slate to a working system. If you can't retrace that path, you can't debug. If you don't have tools to track the path, you will make mistakes. At best, you can exactly replicate the system that you need to fix -- and fixing is changing.
  • ttymck 3y ago
    If I understand correctly, Dockerfile, and image layers, encode that path, making it retrace-able, yes?
    • dsr_ 3y ago
      If I tell you that there's a remote code execution in libfoobar-1.03 through 1.15, how long does it take you to verify where libfoobar is installed, and what versions are in use? Remember, nobody ships an image layer of libfoobar, it's a common component, though not as common as openssl. Is there one command, or one script, which can do that? You need that, basically daily. Is there one command to rebuild with the new libfoobar-1.17, and at most one more to test for basic functionality? You need that, too.
      • Spivak 3y ago
        I mean you're not gonna like the answer but in real life when herding cats the answer is setting up an image scanner and renovate, and calling it a day. It's not like OS images are any better in this respect. I have bit my teeth long enough on software that depends on the base OS and not being able to infra upgrades. Bifurcating responsibility into app/platform is a breath of fresh air by comparison.
      • alexchantavy 3y ago
        Yup this is a hard problem. Shameless plug to my blogpost on how we built something to do this: https://eng.lyft.com/vulnerability-management-at-lyft-enforcing-the-cascade-part-1-234d1561b994 https://eng.lyft.com/vulnerability-management-at-lyft-enforc...
      • 3y ago
• ggm 3y ago
  the memory in my s/w development history was a long-lived build host which had masses of install-loop dependencies (things which need earlier versions of themselves to exist, to bootstrap installing the one you want) as well as dependencies hidden inside things. Basically, "it built on <x>" became a stock joke because it was demonstrably true: anything would build on a host with almost all dependencies resolved, if badly. one thing containers do is provide an audit/reproducible basis to say WHY something builds. it may still be an implicit dependency: something in a specific version of an underlying substrate like the OS or a boot time act which works on that container and not on another. You may not "see" it at first but its a lower bar to understanding it, if the revision control behind the build chain is good enough. a build chain of :latest is probably not good. reproducible builds drive against that. I don't build things for a living any more. Probably this is all well known and argued better by others.
• bogota 3y ago
  Although this was always a problem until the mac M1 chips it didn’t matter much. Now it happens almost every week. I would prefer to have at least the same architecture between my local and prod environments.
  • rad_gruchalski 3y ago
    docker build --platform linux/amd64 alternatively: docker buildx build --platform linux/arm64,linux/amd64 # or whatever you need to target...
    • bogota 3y ago
      Yeah i have used this. It’s about 10x slower though. For instance unit tests that took 2 minutes on mac take almost 20 minutes. Good for one off testing but not a full solution
      • rad_gruchalski 3y ago
        I’d assume it’s no different than cross-compiling on amd64 to arm64. At least it’s possible.
  • pornel 3y ago
    Hetzner has ARM VPSes now :)
  • wmf 3y ago
    So change your prod environment to ARM64.
    • bogota 3y ago
      Yes change the entire production environment to accommodate the development environment… it’s a lot cheaper to just switch developer laptops at next refresh
      • snvzz 3y ago
• coding123 3y ago
  I'll take it works in my container a billion times before it works in my machine. Then I'll know its just configuration. Yes configuration can be hard, but at the end of the day, docker forces all injection points to be super explicit.
  • matthewcroughan 3y ago
    > docker forces all injection points to be super explicit. L O L
• favflam 3y ago
  The next state is "It works in web assembly runtime (WASI)", no?
  • rockwotj 3y ago
    I mean you still have to have reproducible builds either way, which is really what all this is about. Build in an hermetic environment! :shakes-fist-at-bazel-for-not-being-easier-to-use:
    • matthewcroughan 3y ago
      "Easier to use" comes at a cost. I have to wonder whether there's a lower limit to how "easy" reproducibility can be. It is a matter of physics at some level, and the ease of reproducing software can often depend upon what its dependencies are and how easy *they* are to reproduce. I think that using less software, and better software is part of the solution to making reproducibility easy. But FWIW, I do think that Nix lowers the barrier to making reproducible software.
• LoganDark 3y ago
  Looks like this domain name is suddenly just deregistered completely? "dwdraju.medium.com’s server IP address could not be found."
  • ylere 3y ago
    Works for me, maybe an be an issue with your DNS or routing? They're using Cloudfares Reverse Proxy. dig +short dwdraju.medium.com 162.159.152.4 162.159.153.4
    • ddtaylor 3y ago
      Works on my machine.
    • LoganDark 3y ago
      Apparently, my firewall wasn't responding and couldn't answer the DNS requests. What the hell portmaster.
• stavros 3y ago
  One thing I've learned when deploying: Pin absolutely everything. From the exact Docker base image version, to the package installer (e.g. Poetry) version, to all dependencies.
  • remram 3y ago
    Some debian images use snapshot.debian.org, making `apt-get install` reproducible. It's a nice trick. Otherwise distro package installs are not reproducible even if you lock the base image (and apt-get with a specific version will most likely fail).
  • greatpostman 3y ago
    Yup. Always in for a world of pain if you don’t explicitly declare dependency versions
• dekhn 3y ago
  I once worked with a scientist who could only replicate their results on a single computer which had been extensively modified over time. Like, hot patching the C library and stuff. They were absolutely stuck on this machine, and never once considered the possibility that their results were due to the local modifications. In retrospect this is not completely surprising given the incentive system in science.
  • analog31 3y ago
    How do you specify the need for reproducible software installations in an incentive system? Also, what kind of scientist was this? I'm a physicist. I'm deeply concerned about the reproducibility of my results. I periodically try to rebuild my software systems on a clean computer to make sure it's both possible and well documented.
    • dennis_jeeves1 3y ago
      >I'm a physicist. I'm deeply concerned about the reproducibility of my results. By way of comparison, biology and related 'sciences' ( psychology, medicine, nutrition, microbiology etc.) will appear to be nearly a scam...
    • dekhn 3y ago
      A biophysicist- a person who built 3d structural models. Most scientists from tyhe era I'm describing use computers as tools and see periodically moving from system to system as an interruption/distraction from them doing science.
      • analog31 3y ago
        Were they paying for their own computer? This is a problem I've seen in academic science. Grants and departments don't want to pay for computers, so there's a tacit expectation that you'll buy one yourself. My spouse had to do this. In turn there's no consistent IT management of these computers. My wife's attitude is typical: If they want to touch my computer, they can buy one that they own.
• IanCal 3y ago
  It's a nice list of problems and tricks to help, and it is a fun take on issues. For those focusing on the title, a nice way of seeing a difference is less "my container does X" and more "when I run the production container...". Sure your local build might do one thing but you can get what's built in production and try running it. Maybe you can quickly swap between multiple combinations of service versions to replicate a bug that happened during deployments.
• chmod775 3y ago
  Especially for linux hosts this misses that containers will still run on the same kernel (version) as your host OS, inheriting a lot of settings and limitations from there as well (for example net.core.somaxconn). The most obvious ones are sysctl settings, many of which must be set system-wide. Common ones which can drastically change characteristics of database software are vm.nr_hugepages (postgres - why are we OOM or latency is spotty?) and vm.overcommit_memory (redis - why does background saving not work?).
• crooked-v 3y ago
  If you really want the most infuriating version, do enough web dev and you'll eventually run into "it works in my country".
  • KingMob 3y ago
    My favorite are bugs caused by a team distributed on opposite sides of the prime meridian, so you get "Works on my half of earth"
• _9za9 3y ago
  I got -3 points on a comment on a different post where I stated what I didn't like about docker. I was bullied by docker fan boys. Glad to see many here agree with me. It sucks to be surrounded by a mob.
  • kaaaate 3y ago
    took a peek at your comment, and i do agree with you. docker can add unnecessary bloat to projects. docker shouldn't be used for everything. if you provide a docker version of something, it's a smart idea to also publish (for example) an appimage or deb file for people who can't or don't want to use docker. like for example, at my work we don't want to use docker because we will have to get approval from corporate for every little script we want to run in a container because corporate identifies a container as a separate application so it must go through the approval process (which takes 4-8 weeks).
• UniqueUsername0 3y ago
  Nix.
• eikenberry 3y ago
  The article seems to miss the point that we are able talk about these differences in terms of containers as they are abstracted into a purely software system that are reproducible. Versus a customized hardware+software system with no means to reproduce it. Containers were a huge step forward because they raised so much more of the software stack into a simple, repeatable, defined systems than were previously much harder to obtain.
  • paulddraper 3y ago
    Exactly. "It works on my machine" "Okay well here is the exact image digest and configuration from docker inspect" "Thanks I can reproduce the problem now"
    • yjftsjthsd-h 3y ago
      I have hit problems where the host was out of date and that stopped certain newer containers from working. So it's good but not perfect.
• angarg12 3y ago
  The difference is that it doesn't work on MY container, it works on A container. And I can run the same container byte by byte in my machine, yours, or a production cluster. Most/all of the issues highlighted in this article can be mitigated by making your system more self-contained (less dependencies), or using some layers of abstraction on top (e.g. docker compose). Despite these hot takes containers are a massive step in simplifying system deployments.
• nunez 3y ago
  For everyone that went straight into the comments, this article is meant to guide people through _preventing_ situations like "it works in my container."
• msm_ 3y ago
  I feel like most of the problems raised in this blog post can be solved with a proper reproducible build system - for example NixOs (or guix if you will) derivations. It's true that Dockerfiles are not reproducible, but at least they're human friendly and easy to deploy. If you need something more deterministic, I really encourage you to try NixOs. It's (almost) 100% reproducible and works for any real-world use-case that I've ever had. Dockerfiles have a different use case - they are a formal version of a installation instuction that you would give to a new hire in the older times.
  • pkulak 3y ago
    And if you use flakes, it _is_ 100% reproducible.
    • TheDong 3y ago
      That is not the case. Consider the following flake: # flake.nix { outputs = { self, nixpkgs }: { packages.x86_64-linux.default = with nixpkgs.legacyPackages.x86_64-linux; dockerTools.buildImage { name = "nonreproducible-image"; copyToRoot = (runCommand "not-reproducible" {} '' mkdir -p $out head -c 100 /dev/urandom > $out/random ''); }; }; } That is "pure" in the sense that "nix build" will let you build that in pure mode (i.e. in a flake). It will produce a store path with the same hash on any machine you 'nix build' it on, but the contents of 'random' will be different each time. Flakes do not make nix reproducible since flakes do not prevent derivations from running arbitrary commands and producing their own impurities.
      • matthewcroughan 3y ago
        Reproducibility and build determinism are slightly different. Nix is not really about bit-for-bit reproducibility, that is a side-goal, and one that is being worked towards. It is far more pragmatic than that. If you use `--rebuild` like `nix build nixpkgs#hello --rebuild` it *will* warn you if the output is not deterministic FWIW. Instead, Nix ensures the inputs are fetched deterministically via Fixed Output Derivations (FODs) that guarantee the output of building/compiling against that input can only vary so much, and in ways often insignificant to code flow in the output program, (timestamps/byte layout, etc). In the Nix thesis this is referred to as "the extensional model" in Section 5 (Page 87), otherwise known as input-addressing rather than the intensional model which is in reference to content-addressing. Of course we'd all like build outputs to be deterministic, but that's not very pragmatic, or achievable. Many compilers and toolchains don't produce the same results twice, but this often does not effect the behavior of the program. There's some tracking of that goal on r13y.com, and it's probably a goal that will always be limited to a small subset of deterministic outputs, as it's unlikely that all toolchains used in the dependency graph will exhibit deterministic behavior. Sometimes things that do effect code flow leak in, but are squashed by the Nix stdenv where possible, and also by the culture of fixing/patching those problems in Nixpkgs. The result is a pragmatic set of 100,000+ packages that work properly and can be reproduced anywhere.
• realjhol 3y ago
  Nix. The solution is Nix
• yeck 3y ago
  Imo, the strength of containers is portability, not reproducible builds. Even portability isn't perfect, since images are sensitive to CPU architectures and containers can still actually rely on host system configurations (mounting part of fs, privileged mode).
• jchw 3y ago
  The main reason why containers are (a lot) better than the former status quo is shockingly simple: Dockerfiles. They list the steps that you need to build a working image. And yep, there are some caveats, but most of the time they do not cause problems (example: I have containers where :latest has been fine for over 5 years.) I'll go as far as to say that if you want reproducible images that don't unexpectedly break without some kind of ability to trace it back to a change, always use sha256 digests in your `FROM` clauses, never fetch files directly from URLs (and/or check sha256 digests of things that you do fetch,) be thoughtful about the way your container is designed, and favor multi-stage builds and other OCI image builders to writing silly shell scripts that wrangle the build command in unusual ways. But generally? It's still a massive improvement just having the steps written out somewhere. When I first started, it seemed the best you could get is a Vagrantfile that basically worked after you stepped on a bunch of rakes to figure out exactly what quirks you had to work around. With containers, things break a lot more predictably in my experience.
  • Corsome 3y ago
    Agreed with most of the points raised here although reproducible images are currently hard to achieve due to technical reasons on how docker builder operates. See https://medium.com/nttlabs/bit-for-bit-reproducible-builds-with-dockerfile-7cc2b9faed9f https://medium.com/nttlabs/bit-for-bit-reproducible-builds-w...
    • jchw 3y ago
      Yeah, I understand what you mean; bit-for-bit reproducibility is a great idea, and to be clear, I was not talking about that sort of reproducibility, though we should certainly strive for it.
  • pmontra 3y ago
    Yes I remember the problems with Vagrant. I'm unsure about what's making Docker more predictable across machines than Vagrant. Possible reasons - it's usually headless - it comes with a layer that mounts the host file system, instead of installing extensions - better testing of that layer on all platforms, especially the ones that need to add a Linux kernel? (Windows and Mac) - it's harder to ssh into a container, manually fix things and persist the changes without updating the Dockerfile. We can do that with a Vagrant machine. Anything else?
• newman314 3y ago
  There are a number of incorrect statements in this post. 1) One should neither be using the "latest" nor just the "version" tag as the version can still vary depending on when it is pulled. Instead, one should use a combination of version + hash, say alpine:3.18.2@sha256:82d1e9d7ed48a7523bdebc18cf6290bdb97b82302a8a9c27d4fe885949ea94d1 for reproducibility reasons. This provides for human readable versions as well as the specific hash. 2) Next, afaik, Compose has removed the need for version tags. All of the compose.yml files that I now use do not specify versions. See https://github.com/compose-spec/compose-spec/blob/master/04-version-and-name.md https://github.com/compose-spec/compose-spec/blob/master/04-...
  • dikei 3y ago
    "version + hash" is ugly though. I trust the publisher of my base image to keep compatibility even if they update their image and trust my test suites to detect any issues, so I just use version without the hash nowadays.
    • cratermoon 3y ago
      > I trust the publisher of my base image to keep compatibility even if they update their image That's how we get hacks like SolarWinds and MOVEit.
      • dikei 3y ago
        Using hash doesn't protect you from supply chain attack either. If the publisher is compromise, any updates could potentially be malicious. The alternative is to never update at all, which can be even worse.
        • cratermoon 3y ago
          It doesn't completely protect, no. Nothing does. Like much in security, defense in depth is the byword. Not checking the hash throws away a layer.
• wildpeaks 3y ago
  In short, make sure to pin the exact version of dependencies, set the user and keep in mind OS-specific restrictions. Containers are still a great improvement over running on dev machines directly: besides cleanly separating environment per project, an unsung benefit is being able to get rid of the environment when you're done working on a project instead of accumulating bloat over time.
• deleted 3y ago
  [deleted]
• mshekow 3y ago
  I also looked at this topic, see [1]. Some points are similar to the article posted by OP. My findings were: - Docker Desktop and Docker engine (CE) behave differently, e.g. bind mounts, or file system ownerships. - CPU/Platform differences (ARM vs. AMD64): many devs don't realize they use ARM on their mac, thus ARM images are used by default, and tools you run in it (or want to install) may be have differently, or may be missing entirely - Incompatible Linux kernel APIs (when containerized binaries make syscalls not supported by the the host's kernel, for whatever reason) - Using the same version tags, expecting the same result (--> insanity, as you know it :D) - Different engines (e.g. Docker Desktop vs. colima) change the execution behavior (RUNNING containers) - Different build engines (e.g. kaniko vs. BuildKit vs. buildah) change the BUILD behavior For anyone who is interested: more details in [1]. [1] https://www.augmentedmind.de/2023/04/02/docker-portability-issues/ https://www.augmentedmind.de/2023/04/02/docker-portability-i...
  • nerdponx 3y ago
    I think a lot of this comes down to a broader difference between Mac/Windows Docker Desktop and "plain" Docker on Linux. The former is actually backed by a VM, so a lot of the painless simplicity comes from having a true virtual machine involved, rather than just a layer of namespacing. A lot of people are in here complaining about how Docker is not reproducible enough. But reproducibility of image builds is a matter of diminishing returns, and there are other problems to worry about, like the ones you are pointing out. Speaking of which, it's probably good to get in the habit of installing some Linux OS in a VM and trying to run your container images inside that (with "plain" Docker, no inner VM), before pushing it to your cloud host and waiting for it to fail there.
• zokier 3y ago
  While the comments here talk lot about pinning and locking everything down, I'll offer alternative viewpoint: test your application against wider range of environments and versions. Docker is great for that too, you can easily spin up your application in Debian oldstable or latest Fedora and see how it behaves. Your software will become less fragile as a result and better in the long term. Somehow the old adage of portable software being good software seems to have been lost to the ages now that we as developers have attained such precise control over the environment our software runs in.
• frankreyes 3y ago
  Casey Muratory was spot on about using containers was not a solution to any problem, but just more of the same complexity increase. Full presentation at: The Only unbreakable Law https://youtu.be/5IUj1EZwpJY https://youtu.be/5IUj1EZwpJY
  • dns_snek 3y ago
    He says the same about virtual machines, package managers, engines and even libraries. That presentation is a borderline-psychotic rant unless viewed through a very narrow lens where the only thing that matters is maximum-performance systems programming. He's disregarding every productivity improvement in pursuit of maximum performance. He makes that very clear, and in that context it makes sense, but most people won't have the same priorities because we don't live in a world where we can afford to write our own TCP/IP stack to maximize the throughput on our shitty REST APIs that C/R/U/D our customers' TODO items in our database.
    • frankreyes 3y ago
      Yes, there's a trade off between labour productivity and runtime performance. And it's true that not everyone has to run their app at 120hz latency, but it's also true that software has become slower. And it's not something new. JWZ from Netscape established the Zawinski's Law: https://softwareengineering.stackexchange.com/questions/150254/what-does-jamie-zawinskis-law-mean https://softwareengineering.stackexchange.com/questions/1502... >> Every program attempts to expand until it can read mail. Those programs which cannot so expand are replaced by ones which can. But also N Wirth said something similar: https://en.m.wikipedia.org/wiki/Wirth%27s_law https://en.m.wikipedia.org/wiki/Wirth%27s_law >> Wirth's law is an adage on computer performance which states that software is getting slower more rapidly than hardware is becoming faster. Wikipedia from Wirth Law surveys other variations of the statement. So no, it's not just some crack pot hypothesis. It's a real world problem that we have to actually deal with.
• brmgb 3y ago
  Controversial opinion but I still deeply believe that any building tools offering a “latest” option in its build configuration file and not as an option to update said file is doing something deeply wrong and just poorly designed. Everything pooled from outside should be using a checksum. People want and need the ability to pin their environment. If you want to avoid silly surprises when going to production or through your CI system, you need everything to be identical: same versions, same configuration. Tooling should help you do that not introduce new way to shoot you in the foot.
• matthewcroughan 3y ago
  Nix. Nix. Nix. Nix. Nix. Nix. https://www.youtube.com/watch?v=0uixRE8xlbY https://www.youtube.com/watch?v=0uixRE8xlbY https://www.youtube.com/watch?v=6Le0IbPRzOE https://www.youtube.com/watch?v=6Le0IbPRzOE
• ParetoOptimal 3y ago
  https://devenv.sh/ https://devenv.sh/ is my answer. Nix plus good UX that can even build vscode compatible devcontainers that mirror the nix shell.
• ParetoOptimal 3y ago
  Docker performance hit can also matter. Recently changed a codebase from a devcontainer to Nix shell: Test runtime docker: 13s Test runtime nix: 6s