38 ms·

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continou

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is that it frees us from troubleshooting "works on my machine" issues, or from someone pushing an update somewhere and breaking our build. That's not important to everyone if they have few dependencies that don't change often, but for an internet company, the trend is accelerating towards bigger and more complex dependency graphs. Kubernetes has mostly focused on stateless applications so far. That's the easy part! The hard part is managing databases. We don't use Kubernetes, but there's little attraction because it would be addressing something that's already effortless for us to manage. What works for us is to do the simplest thing that works, then iterate. I remember being really intimidated about all the big data technologies coming out a decade ago, thinking they are so complex that they must know what they're doing! But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I was in a sort of paralysis of what we'd do after we outgrew postgresql, and never found a good answer. Here we are years later, with a dozen+ postgresql databases, some measuring up to 30 terabytes each, and it's still the best solution for us. Perhaps I've read too far into the intent of the question, but maybe you can afford to drop the research project into containers and kubernetes, and do something simple that works for now, and get back to focusing on product?

higeorge13 5y ago
It sounds interesting. You mentioned big data; what is your tech stack other than postgres?
- smilliken 5y ago
  Python, PostgreSQL, Nix, Linux. Javascript and Typescript for frontend. Everything else is effectively libraries. After 300k lines of Python, with the occasional C extension, we haven't found any limit.
brimble 5y ago
> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime) is what's tricky. Spinning up new workers, reliable- and repeatable-enough deployment with rollbacks and such, that's easy, until it touches the disk (or the DB schema, that's its own whole messy dance to get right)
- fer 5y ago
  Sometimes I think K8s is largely pushed for Amazon/Google/Microsoft to sell the disk that goes along with it.
  - zomglings 5y ago
    Don't forget network costs (especially for "high availability" clusters spanning multiple regions).
  - swiftcoder 5y ago
    Admittedly it's 5 years since I worked at AWS, but at the time there was basically no internal usage of containers. ECS was pushing customers towards containers, but the the feeling towards containers at the time inside the company could be summed up as "we have a decade worth of operational tooling, why would we throw that all away?"
- zthrowaway 5y ago
  People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.
  - mountainriver 5y ago
    Yes you can run DBs on kube now, much of people thinking this isn’t good comes from years back when it wasn’t
claytonjy 5y ago
is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.
- silviogutierrez 5y ago
  It's a hurdle to be able to write Nix stuff, but to consume it is pretty easy. Shameless plug here: https://www.reactivated.io/documentation/why-nix/ https://www.reactivated.io/documentation/why-nix/
  - olifante 5y ago
    I just tried Nix yesterday for the first time in order to play with reactivated. Was surprised by how simple it was to get it running. Made me think that perhaps containers are not as indispensable as I thought. And reactivated is awesome, by the way. I just worry about how brittle it will be as both Django and React evolve.
    - silviogutierrez 5y ago
      Thanks! Hopefully not brittle at all. Specially since both React and Django are very stable projects, the latter even more than the former.
  - deleted 5y ago
    [deleted]
- smilliken 5y ago
  In my case, adopting Nix was a response to having a poor onboarding process for new engineers. It was always fully automated, but it wasn't reliable before nix. So somebody would join the team, and it was embarrassing because the first day would be them troubleshooting a complex build process to get it to work on their machine. Not a great first impression. So I adopted Nix "in anger" and now new machines always build successfully on the first try. For me it was easy to set up. 80% done on the first day. It helped that I understood the conceptual model and benefits first. There's a lot of directions you can take it, I'd recommend getting something really simple working then iterating. Don't try to build the Sistine Chapel with all the different things integrated on day one. It's hard to overstate how well that decision has worked out for me. Computers used to stress me out a lot more because of dependencies and broken builds. Now I have a much healthier relationship with computers.
barbazoo 5y ago
> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the continuous build?
- q_eng_anon 5y ago
  https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless This is the container community’s response to this ambiguity I think.
- monocasa 5y ago
  I think that's what they're saying, that containers are orthogonal to reproducible builds.
- jffry 5y ago
  > Containers only give reproducible deployments, not builds I think that means containers alone are insufficient for creating reproducible builds, not that containers make reproducible builds impossible.
  - lazide 5y ago
    I believe that is correct. It’s also hard to make container builds really reproducible (even with other parts of the build being so). Some sibling comments have talked about why. Containers are to ops like C is to programming languages. Man is it useful, but boy are there footguns everywhere. Usually still better than the ops equivalent of assembly, which is what came before though - all the custom manual processes. Kubernetes is maybe like early C++. A bit weird, still has footguns, and is it really helping? Mostly it is in most cases, but we’re also discovering many more new failure modes and having to learn some new complex things. As it matures, new classes of problems will be way easier to solve with less work, and most of the early footguns will be found and fixed. There will be new variants that hide the details and de-footgun things too, and that will have various trade offs. No idea what will be the equivalent of Rust or Java yet.
disintegore 5y ago
I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](https://drewdevault.com/2017/09/08/Complicated.html https://drewdevault.com/2017/09/08/Complicated.html) isn't particularly hard. What I don't see a lot of, however, is alternatives. "Learn how to do ops" is not exactly pertinent when most documentation on the subject will point you towards containers. In addition the whole principle, while clearly proven, does strike me as a patch on existing platforms and executable formats that weren't designed to solve the sorts of problems that we have today. While efforts could have been made to make software universally more portable it seems we opted for finding the smallest feasible packaging method with already existing technology and have been rolling with it for a decade. So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table (eg "we work with petabytes of data just fine but deploying updates is a nightmare"), how much manual effort they put into it, etc. New greenfield projects attempting to replace the fundamentals rather than re-orchestrate them are also very pertinent here.
- SQueeeeeL 5y ago
  >So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table Decades of work typically. That or just standard parallel deployments, most things Docker is good at would take the average developer many unnecessary hours to reproduce.
  - pojzon 5y ago
    Currently there is a maltitude of solutions available on the market that make containers and k8s just a complex and hard to maintain choice. It all boils down to “what bullshit was my management sold on”. Promises of “easy to use, easy to replace, cloud-agnostic, doesnt require domain knowledge” etc etc etc. Its all bullshit. At some point/scale you need experts in the field that cash in a month more than average dev per year.
ixxie 5y ago
I would love to hear more about your architecture and deployment... do your services run as NixOS modules? Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers?
- smilliken 5y ago
  For the first 6 years of using Nix, it was depoyed on Ubuntu. We recently migrated to NixOS. NixOS is fantastic for other reasons that are similar but separate from Nix as a package manager/build system. It's easier to incrementally switch to Nix first then NixOS later. We don't use systemd services for our code. We only use NixOS as an operating system, not for application layer. Our code works just the same on any linux distribution, or even macos minus linux-only stuff. That way we don't need to insist that everyone uses the same OS for development, and no one is forced into VMs. Everything can be run and tested locally.
  - smilliken 5y ago
    > Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers? Not using NixOps or Morph. I recall considering a few others as well. In each case, I wasn't able to understand what they were doing that I couldn't do with a simple script. Instead, there's a python program that does an rsync, installs some configuration files, and runs the nix build. It deploys to any linux OS, but it does some stuff differently for NixOS, and the differences account for about 100 lines while increasing scope (managing filesystems, kernel versions, kernel modules, os users, etc). A deploy to all hosts takes about a minute because it runs in parallel. Deploys are zero downtime, including restarting a bunch of web apps. The NixOS configuration itself is another ~200 lines, plus a bit per host for constants like hostname and ip addresses. It's really neat being able to define an entire operating system install in a single configuration file that gets type checked.
    - osener 5y ago
      For others interested in deploying like this, this approach sounds exactly like what krops does. Krops is similar to Morph, except it does the derivative building on the remote host. It is very simple and works great. Deploying from macOS to NixOS is possible as well. https://github.com/krebs/krops https://github.com/krebs/krops https://tech.ingolf-wagner.de/nixos/krops/ https://tech.ingolf-wagner.de/nixos/krops/
rootusrootus 5y ago
> But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement until you are doing a lot of the same thing. And then it costs $$$$$$.
- jmt_ 5y ago
  Totally agree. Something I find myself thinking/saying more and more is "don't underestimate what you can accomplish with a $5 VPS". Maybe a little hyperbolic, but computers are really fast these days. Elastic resources are pretty magical when you need them, but the majority of software projects just don't end up requiring as many resources as devs/management often seem to assume. Granted, migrating to something like AWS, if you do end up needing it, is a pain, but so is adding unnecessary layers of AWS complexity for your CRUD app/API backend. Also wonder how many devs came up on AWS and prefer it for familiarity and not having to know and worry too much about aspects of deployment outside of the code itself (i.e not having to know too much about managing postgres, nginx, etc).
- JohnBooty 5y ago
  Yeah, I feel this way a lot. In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem. There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that. The devops guy at a previous gig thought I was some kind of naive simp for insisting we didn't need a cluster of Amazon's version of Redis. We were storing like, a megabyte of data in there. That's not a typo. It's not that he was an idiot. Smart guy actually. Not sure if he was just padding his resume with experience ("sure, I've admin'd Redis clusters!") or if he was locked into some kind of default thinking that wouldn't even allow him to consider some kind of minimal setup.
  - throwbigdata 5y ago
sideway 5y ago
Out of curiosity, what industry is your company in?
- smilliken 5y ago
  Details in bio.
Thaxll 5y ago
The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version to build then the result is reproducible. That's what most people should be doing, you pin versions in your build image that's it. > Kubernetes has mostly focused on stateless applications so far. That's the easy part Kubernetes can run database, and stateless application is not a solved problem, I mean how does your services get restarted on your system, is it something that you had to re-create as well?
- cj 5y ago
  > the rest of the world is improving on those tech Does that matter if the current stack works just fine? Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks. If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code. I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.
  - Thaxll 5y ago
    At some point you just don't want to re-invent the wheel, reminds me a lot of: https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
  - hinkley 5y ago
    The people who are invested in their home-grown solutions are usually pretty bad at doing tech support for them. They enjoy writing code, and it doesn't take long before you've created enough surface area that you couldn't possibly keep up with requests even if you wanted to. Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail to use it. They are a mirror and people don't always like what they see in it. That guy who keeps talking trash about how stupid everyone else is for not understanding his beautiful code is deflecting. If they're right then my code does not have value/isn't smart which means I don't have value/am not smart, so clearly the problem is that I'm surrounded by assholes. I don't think we would be using as much 3rd party software without StackOverflow, and almost nobody creates a StackOverflow for internal tools. Nobody writes books for internal tools. Nobody writes competitors for internal tools which either prove or disprove the utility of the original tool. All of those options are how some people learn, and in some cases how they debug. You're cutting off your own nose when you reinvent something and don't invest the time to do all of these other things. If I lose someone important or get a big contract I didn't think I could win, I can't go out and hire anyone with 3 years experience in your internal tool, trading money for time. I have to hire noobs and sweat it out while we find out if they will ever learn the internal tools or not. The older I get the more I see NIH people as working a protection racket, because it amplifies the 'value' of tenure at the company. The externalities created really only affect new team members, delaying the date when you can no longer dismiss their ideas out of hand. You have in effect created an oligarchy, whereas most of the rest of us prefer a representative republic (actual democracy is too many meetings).
tharne 5y ago
> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two had developed much better BS-detectors than 20-something me.
- usaphp 5y ago
  Or it could be that you just became old and don't want to learn new things anymore ))
  - jenkstom 5y ago
    This is the antivax sentiment of the IT world. It's new! It's shiny! Give me some Ivermectin because I know things other people don't! (Yes, I realize you're probably joking. I'm not, particularly.)
    - ThePowerOfFuet 5y ago
      What does ivermectin have to do with anything? Are you suffering from chronic parasitic infection?
    - rattlesnakedave 5y ago
      You have it reversed. “I don’t want the new vaccine! Who knows what the long term side effects are!” is equivalent to the technical curmudgeon. The voluntary early trial test group are the bleeding edge tech people. The Ivermectin crowd is the guy that suggests rewriting the backend in Haskell for no good reason.
    - nurettin 5y ago
      It's like you're just repeating the last inane discussion YouTube got you hooked on in order to get some ad clicks.
  - 5y ago
david38 5y ago
Ehhh, I run a high performance database system in Kubernetes and it works great. It’s distributed, uses EBS volumes. That’s about as opposite of stateless as it gets.
- tetha 5y ago
  Do note that you have offloaded a good chunk of state management into EBS volumes via CSI. Attaching the CSI volumes is one thing, running the disks hosting the volumes is another thing.
rapsey 5y ago
I've seen comments on here from people with large Nix deployments with like hundreds of thousands of Nix script code and saying it is a complete nightmare to manage.
mamcx 5y ago
Any hints in how use nix with DBs? I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple. I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....
- smilliken 5y ago
  Yeah, we built a tool for managing postgresql databases with nix. It's called schematic: https://gitlab.com/deltaex/schematic https://gitlab.com/deltaex/schematic We've been using it in prod for a couple years. There's a couple dozen production deployments outside the company as well. It's open source, MIT licensed. It doesn't have documentation yet, so it's currently only for people that don't mind reading the source or have talked to us directly about it. It uses Nix because postgresql has C extensions, which can depend on anything in the software universe. Schema depends on extensions, so it's not technically possible to separate schema migrations from Nix without duct tape abd glue. So schematic is a sort of "distribution" of PostgreSQL that has a package manager (for extensions, schema, content, etc), and manages revisions. If this is interesting to others here, I can do a "Show HN" post after getting the docs in order.
  - mamcx 5y ago
    Yeah, managing the DBs is the major complications on doing this. A little discrepancy in binaries are not that always problematic, but a fail on the DB is catastrophic :)
deathanatos 5y ago
> Kubernetes has mostly focused on stateless applications so far. That's the easy part! The hard part is managing databases. Kubernetes can absolutely host stateful applications. A StatefulSet is the app-level construct pretty much intended for exactly that use case. My company runs a distributed database on top of Kubernetes. (We have another app that uses StatefulSets to maintain a cache across restarts, so that it can come up quickly, as it otherwise needs to sync a lot of data. But, it is technically stateless: we could just sync the entire dataset, which is only ~20 GiB, each time it starts, but that is wasteful, and it makes startup, and thus deployments of new code, quite slow. It would also push the limits of what an emptyDir can accommodate in our clusters' setup.) (The biggest issue we've had, actually, with that, is that Azure supports NVMe, and Azure has AKS, but Azure — for God only knows what reason — apparently doesn't support combining the two. We'd love to do that, & have managed k8s & nice disks, but Azure forces us to choose. This is one of my chief gripes about Azure's services in general: they do not compose, and trying to compose them is just fraught with problems. But, that's an Azure issue, not a Kubernetes one.)
- pojzon 5y ago
  For anyone looking for confirmations look for Zalando’s Postgres Operator. They host 100++ db clusters using that also for production workloads.
geoduck14 5y ago
Well, I was about to ask if you were looking for a job - but then I saw you were the founder and CTO. So... are you hiring?
- smilliken 5y ago
  Yes, we're hiring! :) Reach out to dev@mixrank.com. Hiring globally for pretty much all roles, including junior roles.
deleted 5y ago
[deleted]
n42 5y ago
I'm leading my team through a migration to Nix environments right now, and everything you've said has rung true to my own experiences so far. I'd love to hear more about the hard lessons you've learned, if you have any thoughts there
ComradePhil 5y ago
If you really believed what you pretend to, you would be programming in .NET on Windows computers and running your apps on Windows... or even Java. Or take that problem to another level and use golang. Instead, you use python and try to fix the problems with another immature solution: Nix, which has uniform builds... but not really in certain cases... and it is the same on every Linux... except not really so you kind of have to use NixOS.
fnordpiglet 5y ago
I think containers and k8s start to make sense when you need to impose logical isolation on your flat topology networks, often for security but also for preventing everything becoming overly complex in their dependencies. But this IMO isn’t often a concern until your enterprise is very large and complex with some very crufty dependencies that you realize eventually you can never practically demise because you can’t even fully determine what all depends on it at any given time. That said, I would prefer managing that stuff through SDNs rather than trying to make user space overly complex software replication of commodity capabilities. Or, just use a cloud provider to start managing the isolation for the parts that benefit for it. What really baffles me is the use of k8s in aws. Why? BTW - congrats on effective use of Nix. I use it extensively at home for my home services. It’s got some weirdness to it for sure but it’s a heck of a lot better than almost anything I’ve ever seen to date. I can’t wait to see what evolves out of it.

higeorge13 5y ago

It sounds interesting. You mentioned big data; what is your tech stack other than postgres?

smilliken 5y ago
Python, PostgreSQL, Nix, Linux. Javascript and Typescript for frontend. Everything else is effectively libraries. After 300k lines of Python, with the occasional C extension, we haven't found any limit.

brimble 5y ago

> That's the easy part! The hard part is managing databases. Ding ding ding ding ding. "But what about disk?" (so, relatedly, databases) is the hard part. Balancing performance (network disks suuuuuck) and flexibility ("just copy the disk image to another machine" is fine when it's a few GB—less useful when it's lots of GB and you might need to migrate it to another city and also you'd rather not have much downtime) is what's tricky. Spinning up new workers, reliable- and repeatable-enough deployment with rollbacks and such, that's easy, until it touches the disk (or the DB schema, that's its own whole messy dance to get right)

fer 5y ago
Sometimes I think K8s is largely pushed for Amazon/Google/Microsoft to sell the disk that goes along with it.
- zomglings 5y ago
  Don't forget network costs (especially for "high availability" clusters spanning multiple regions).
- swiftcoder 5y ago
  Admittedly it's 5 years since I worked at AWS, but at the time there was basically no internal usage of containers. ECS was pushing customers towards containers, but the the feeling towards containers at the time inside the company could be summed up as "we have a decade worth of operational tooling, why would we throw that all away?"
zthrowaway 5y ago
People still think stateful things are impossible on k8s but Stateful sets and persistent volumes solves a lot of this. You should be relying on out of the box DB replication to make sure data is available in multiple areas. This is no different on other platforms.
- mountainriver 5y ago
  Yes you can run DBs on kube now, much of people thinking this isn’t good comes from years back when it wasn’t

claytonjy 5y ago

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

silviogutierrez 5y ago
It's a hurdle to be able to write Nix stuff, but to consume it is pretty easy. Shameless plug here: https://www.reactivated.io/documentation/why-nix/ https://www.reactivated.io/documentation/why-nix/
- olifante 5y ago
  I just tried Nix yesterday for the first time in order to play with reactivated. Was surprised by how simple it was to get it running. Made me think that perhaps containers are not as indispensable as I thought. And reactivated is awesome, by the way. I just worry about how brittle it will be as both Django and React evolve.
  - silviogutierrez 5y ago
    Thanks! Hopefully not brittle at all. Specially since both React and Django are very stable projects, the latter even more than the former.
- deleted 5y ago
  [deleted]
smilliken 5y ago
In my case, adopting Nix was a response to having a poor onboarding process for new engineers. It was always fully automated, but it wasn't reliable before nix. So somebody would join the team, and it was embarrassing because the first day would be them troubleshooting a complex build process to get it to work on their machine. Not a great first impression. So I adopted Nix "in anger" and now new machines always build successfully on the first try. For me it was easy to set up. 80% done on the first day. It helped that I understood the conceptual model and benefits first. There's a lot of directions you can take it, I'd recommend getting something really simple working then iterating. Don't try to build the Sistine Chapel with all the different things integrated on day one. It's hard to overstate how well that decision has worked out for me. Computers used to stress me out a lot more because of dependencies and broken builds. Now I have a much healthier relationship with computers.

barbazoo 5y ago

> Containers only give reproducible deployments, not builds Could someone elaborate on this please? Doesn't it depend entirely on your stack how reproducible your build is? Say I have a Python app with its OS level packages installed into a (base) image and its Python dependencies specified in a Pipfile, doesn't that make it pretty reproducible? Is the weak spot here any OS dependencies being installed as part of the continuous build?

q_eng_anon 5y ago
https://github.com/GoogleContainerTools/distroless https://github.com/GoogleContainerTools/distroless This is the container community’s response to this ambiguity I think.
monocasa 5y ago
I think that's what they're saying, that containers are orthogonal to reproducible builds.
jffry 5y ago
> Containers only give reproducible deployments, not builds I think that means containers alone are insufficient for creating reproducible builds, not that containers make reproducible builds impossible.
- lazide 5y ago
  I believe that is correct. It’s also hard to make container builds really reproducible (even with other parts of the build being so). Some sibling comments have talked about why. Containers are to ops like C is to programming languages. Man is it useful, but boy are there footguns everywhere. Usually still better than the ops equivalent of assembly, which is what came before though - all the custom manual processes. Kubernetes is maybe like early C++. A bit weird, still has footguns, and is it really helping? Mostly it is in most cases, but we’re also discovering many more new failure modes and having to learn some new complex things. As it matures, new classes of problems will be way easier to solve with less work, and most of the early footguns will be found and fixed. There will be new variants that hide the details and de-footgun things too, and that will have various trade offs. No idea what will be the equivalent of Rust or Java yet.

disintegore 5y ago

I asked the question without context because I didn't want to fire the thread off in the wrong direction but I suppose in a comment chain it's fine. For what it's worth we do use containers and K8s heavily at my current job. I know that there are quite a few people opposed to the state of containers and the technologies revolving around them. I don't think the arguments they present are bad. [Attacking the premise](https://drewdevault.com/2017/09/08/Complicated.html https://drewdevault.com/2017/09/08/Complicated.html) isn't particularly hard. What I don't see a lot of, however, is alternatives. "Learn how to do ops" is not exactly pertinent when most documentation on the subject will point you towards containers. In addition the whole principle, while clearly proven, does strike me as a patch on existing platforms and executable formats that weren't designed to solve the sorts of problems that we have today. While efforts could have been made to make software universally more portable it seems we opted for finding the smallest feasible packaging method with already existing technology and have been rolling with it for a decade. So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table (eg "we work with petabytes of data just fine but deploying updates is a nightmare"), how much manual effort they put into it, etc. New greenfield projects attempting to replace the fundamentals rather than re-orchestrate them are also very pertinent here.

SQueeeeeL 5y ago
>So essentially I'm interesting in knowing how people find ways to reproduce the value these technologies offer, what they instead rely on, which things they leave on the table Decades of work typically. That or just standard parallel deployments, most things Docker is good at would take the average developer many unnecessary hours to reproduce.
- pojzon 5y ago
  Currently there is a maltitude of solutions available on the market that make containers and k8s just a complex and hard to maintain choice. It all boils down to “what bullshit was my management sold on”. Promises of “easy to use, easy to replace, cloud-agnostic, doesnt require domain knowledge” etc etc etc. Its all bullshit. At some point/scale you need experts in the field that cash in a month more than average dev per year.

ixxie 5y ago

I would love to hear more about your architecture and deployment... do your services run as NixOS modules? Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers?

smilliken 5y ago
For the first 6 years of using Nix, it was depoyed on Ubuntu. We recently migrated to NixOS. NixOS is fantastic for other reasons that are similar but separate from Nix as a package manager/build system. It's easier to incrementally switch to Nix first then NixOS later. We don't use systemd services for our code. We only use NixOS as an operating system, not for application layer. Our code works just the same on any linux distribution, or even macos minus linux-only stuff. That way we don't need to insist that everyone uses the same OS for development, and no one is forced into VMs. Everything can be run and tested locally.
- smilliken 5y ago
  > Are you using NixOps or Morph or something else? How does your world look like without K8s and Containers? Not using NixOps or Morph. I recall considering a few others as well. In each case, I wasn't able to understand what they were doing that I couldn't do with a simple script. Instead, there's a python program that does an rsync, installs some configuration files, and runs the nix build. It deploys to any linux OS, but it does some stuff differently for NixOS, and the differences account for about 100 lines while increasing scope (managing filesystems, kernel versions, kernel modules, os users, etc). A deploy to all hosts takes about a minute because it runs in parallel. Deploys are zero downtime, including restarting a bunch of web apps. The NixOS configuration itself is another ~200 lines, plus a bit per host for constants like hostname and ip addresses. It's really neat being able to define an entire operating system install in a single configuration file that gets type checked.
  - osener 5y ago
    For others interested in deploying like this, this approach sounds exactly like what krops does. Krops is similar to Morph, except it does the derivative building on the remote host. It is very simple and works great. Deploying from macOS to NixOS is possible as well. https://github.com/krebs/krops https://github.com/krebs/krops https://tech.ingolf-wagner.de/nixos/krops/ https://tech.ingolf-wagner.de/nixos/krops/

rootusrootus 5y ago

> But I'd so often dive in to understand the details and be disillusioned about how much complexity there is for relatively little benefit. I feel similarly about many cloud services in e.g. AWS. I get where it's sometimes handy, but the management overhead can get pretty insane. I used to hear people say "just do X in AWS and ta-da you're done!" only to find out that in many cases it isn't actually a net improvement until you are doing a lot of the same thing. And then it costs $$$$$$.

jmt_ 5y ago
Totally agree. Something I find myself thinking/saying more and more is "don't underestimate what you can accomplish with a $5 VPS". Maybe a little hyperbolic, but computers are really fast these days. Elastic resources are pretty magical when you need them, but the majority of software projects just don't end up requiring as many resources as devs/management often seem to assume. Granted, migrating to something like AWS, if you do end up needing it, is a pain, but so is adding unnecessary layers of AWS complexity for your CRUD app/API backend. Also wonder how many devs came up on AWS and prefer it for familiarity and not having to know and worry too much about aspects of deployment outside of the code itself (i.e not having to know too much about managing postgres, nginx, etc).
JohnBooty 5y ago
Yeah, I feel this way a lot. In recent years I've done work for companies where it feels like the end result of a lot of very expensive AWS infra is... more expensive, and not really superior to things we had ten years ago when hosting on-prem. There's a level of scale for which distributed AWS/Azure is necessary, but I'm not sure how many shops need that. The devops guy at a previous gig thought I was some kind of naive simp for insisting we didn't need a cluster of Amazon's version of Redis. We were storing like, a megabyte of data in there. That's not a typo. It's not that he was an idiot. Smart guy actually. Not sure if he was just padding his resume with experience ("sure, I've admin'd Redis clusters!") or if he was locked into some kind of default thinking that wouldn't even allow him to consider some kind of minimal setup.
- throwbigdata 5y ago

sideway 5y ago

Out of curiosity, what industry is your company in?

smilliken 5y ago
Details in bio.

Thaxll 5y ago

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version to build then the result is reproducible. That's what most people should be doing, you pin versions in your build image that's it. > Kubernetes has mostly focused on stateless applications so far. That's the easy part Kubernetes can run database, and stateless application is not a solved problem, I mean how does your services get restarted on your system, is it something that you had to re-create as well?

cj 5y ago
> the rest of the world is improving on those tech Does that matter if the current stack works just fine? Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks. If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code. I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.
- Thaxll 5y ago
  At some point you just don't want to re-invent the wheel, reminds me a lot of: https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
- hinkley 5y ago
  The people who are invested in their home-grown solutions are usually pretty bad at doing tech support for them. They enjoy writing code, and it doesn't take long before you've created enough surface area that you couldn't possibly keep up with requests even if you wanted to. Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail to use it. They are a mirror and people don't always like what they see in it. That guy who keeps talking trash about how stupid everyone else is for not understanding his beautiful code is deflecting. If they're right then my code does not have value/isn't smart which means I don't have value/am not smart, so clearly the problem is that I'm surrounded by assholes. I don't think we would be using as much 3rd party software without StackOverflow, and almost nobody creates a StackOverflow for internal tools. Nobody writes books for internal tools. Nobody writes competitors for internal tools which either prove or disprove the utility of the original tool. All of those options are how some people learn, and in some cases how they debug. You're cutting off your own nose when you reinvent something and don't invest the time to do all of these other things. If I lose someone important or get a big contract I didn't think I could win, I can't go out and hire anyone with 3 years experience in your internal tool, trading money for time. I have to hire noobs and sweat it out while we find out if they will ever learn the internal tools or not. The older I get the more I see NIH people as working a protection racket, because it amplifies the 'value' of tenure at the company. The externalities created really only affect new team members, delaying the date when you can no longer dismiss their ideas out of hand. You have in effect created an oligarchy, whereas most of the rest of us prefer a representative republic (actual democracy is too many meetings).

tharne 5y ago

> What works for us is to do the simplest thing that works, then iterate. The older I get, the more often I'm reminded that this un-sexy approach is really the best way to go. When I was younger, I always thought the old guys pushing boring solutions just didn't want to learn new things. Now I'm starting to realize that after several decades of experience, they simply got burned enough times to learn a thing or two had developed much better BS-detectors than 20-something me.

usaphp 5y ago
Or it could be that you just became old and don't want to learn new things anymore ))
- jenkstom 5y ago
  This is the antivax sentiment of the IT world. It's new! It's shiny! Give me some Ivermectin because I know things other people don't! (Yes, I realize you're probably joking. I'm not, particularly.)
  - ThePowerOfFuet 5y ago
    What does ivermectin have to do with anything? Are you suffering from chronic parasitic infection?
  - rattlesnakedave 5y ago
    You have it reversed. “I don’t want the new vaccine! Who knows what the long term side effects are!” is equivalent to the technical curmudgeon. The voluntary early trial test group are the bleeding edge tech people. The Ivermectin crowd is the guy that suggests rewriting the backend in Haskell for no good reason.
  - nurettin 5y ago
    It's like you're just repeating the last inane discussion YouTube got you hooked on in order to get some ad clicks.
- 5y ago

david38 5y ago

Ehhh, I run a high performance database system in Kubernetes and it works great. It’s distributed, uses EBS volumes. That’s about as opposite of stateless as it gets.

tetha 5y ago
Do note that you have offloaded a good chunk of state management into EBS volumes via CSI. Attaching the CSI volumes is one thing, running the disks hosting the volumes is another thing.

rapsey 5y ago

I've seen comments on here from people with large Nix deployments with like hundreds of thousands of Nix script code and saying it is a complete nightmare to manage.

mamcx 5y ago

Any hints in how use nix with DBs? I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple. I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....

smilliken 5y ago
Yeah, we built a tool for managing postgresql databases with nix. It's called schematic: https://gitlab.com/deltaex/schematic https://gitlab.com/deltaex/schematic We've been using it in prod for a couple years. There's a couple dozen production deployments outside the company as well. It's open source, MIT licensed. It doesn't have documentation yet, so it's currently only for people that don't mind reading the source or have talked to us directly about it. It uses Nix because postgresql has C extensions, which can depend on anything in the software universe. Schema depends on extensions, so it's not technically possible to separate schema migrations from Nix without duct tape abd glue. So schematic is a sort of "distribution" of PostgreSQL that has a package manager (for extensions, schema, content, etc), and manages revisions. If this is interesting to others here, I can do a "Show HN" post after getting the docs in order.
- mamcx 5y ago
  Yeah, managing the DBs is the major complications on doing this. A little discrepancy in binaries are not that always problematic, but a fail on the DB is catastrophic :)

deathanatos 5y ago

> Kubernetes has mostly focused on stateless applications so far. That's the easy part! The hard part is managing databases. Kubernetes can absolutely host stateful applications. A StatefulSet is the app-level construct pretty much intended for exactly that use case. My company runs a distributed database on top of Kubernetes. (We have another app that uses StatefulSets to maintain a cache across restarts, so that it can come up quickly, as it otherwise needs to sync a lot of data. But, it is technically stateless: we could just sync the entire dataset, which is only ~20 GiB, each time it starts, but that is wasteful, and it makes startup, and thus deployments of new code, quite slow. It would also push the limits of what an emptyDir can accommodate in our clusters' setup.) (The biggest issue we've had, actually, with that, is that Azure supports NVMe, and Azure has AKS, but Azure — for God only knows what reason — apparently doesn't support combining the two. We'd love to do that, & have managed k8s & nice disks, but Azure forces us to choose. This is one of my chief gripes about Azure's services in general: they do not compose, and trying to compose them is just fraught with problems. But, that's an Azure issue, not a Kubernetes one.)

pojzon 5y ago
For anyone looking for confirmations look for Zalando’s Postgres Operator. They host 100++ db clusters using that also for production workloads.

geoduck14 5y ago

Well, I was about to ask if you were looking for a job - but then I saw you were the founder and CTO. So... are you hiring?

smilliken 5y ago
Yes, we're hiring! :) Reach out to dev@mixrank.com. Hiring globally for pretty much all roles, including junior roles.

deleted 5y ago

[deleted]

n42 5y ago

I'm leading my team through a migration to Nix environments right now, and everything you've said has rung true to my own experiences so far. I'd love to hear more about the hard lessons you've learned, if you have any thoughts there

ComradePhil 5y ago

If you really believed what you pretend to, you would be programming in .NET on Windows computers and running your apps on Windows... or even Java. Or take that problem to another level and use golang. Instead, you use python and try to fix the problems with another immature solution: Nix, which has uniform builds... but not really in certain cases... and it is the same on every Linux... except not really so you kind of have to use NixOS.

fnordpiglet 5y ago

I think containers and k8s start to make sense when you need to impose logical isolation on your flat topology networks, often for security but also for preventing everything becoming overly complex in their dependencies. But this IMO isn’t often a concern until your enterprise is very large and complex with some very crufty dependencies that you realize eventually you can never practically demise because you can’t even fully determine what all depends on it at any given time. That said, I would prefer managing that stuff through SDNs rather than trying to make user space overly complex software replication of commodity capabilities. Or, just use a cloud provider to start managing the isolation for the parts that benefit for it. What really baffles me is the use of k8s in aws. Why? BTW - congrats on effective use of Nix. I use it extensively at home for my home services. It’s got some weirdness to it for sure but it’s a heck of a lot better than almost anything I’ve ever seen to date. I can’t wait to see what evolves out of it.