5 ms·

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your ho

by Thaxll 5y ago

The thing is, you spent probably a lot of time on things that were granted elsewhere and while the rest of the world is improving on those tech you keep your home grown solution that is harder and harder to maintain. Plus the knowledge that is not transferable. > Containers only give reproducible deployments, not builds, so they would be a step down. This is not true, if you use a docker image A with specific version to build then the result is reproducible. That's what most people should be doing, you pin versions in your build image that's it. > Kubernetes has mostly focused on stateless applications so far. That's the easy part Kubernetes can run database, and stateless application is not a solved problem, I mean how does your services get restarted on your system, is it something that you had to re-create as well?

cj 5y ago
> the rest of the world is improving on those tech Does that matter if the current stack works just fine? Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks. If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code. I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.
- Thaxll 5y ago
  At some point you just don't want to re-invent the wheel, reminds me a lot of: https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
- hinkley 5y ago
  The people who are invested in their home-grown solutions are usually pretty bad at doing tech support for them. They enjoy writing code, and it doesn't take long before you've created enough surface area that you couldn't possibly keep up with requests even if you wanted to. Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail to use it. They are a mirror and people don't always like what they see in it. That guy who keeps talking trash about how stupid everyone else is for not understanding his beautiful code is deflecting. If they're right then my code does not have value/isn't smart which means I don't have value/am not smart, so clearly the problem is that I'm surrounded by assholes. I don't think we would be using as much 3rd party software without StackOverflow, and almost nobody creates a StackOverflow for internal tools. Nobody writes books for internal tools. Nobody writes competitors for internal tools which either prove or disprove the utility of the original tool. All of those options are how some people learn, and in some cases how they debug. You're cutting off your own nose when you reinvent something and don't invest the time to do all of these other things. If I lose someone important or get a big contract I didn't think I could win, I can't go out and hire anyone with 3 years experience in your internal tool, trading money for time. I have to hire noobs and sweat it out while we find out if they will ever learn the internal tools or not. The older I get the more I see NIH people as working a protection racket, because it amplifies the 'value' of tenure at the company. The externalities created really only affect new team members, delaying the date when you can no longer dismiss their ideas out of hand. You have in effect created an oligarchy, whereas most of the rest of us prefer a representative republic (actual democracy is too many meetings).
  - p10_user 5y ago
    Nice post. I wish I was reading this 6 years ago, but at least I quickly learned to be humble about the quality of my code.
    - hinkley 5y ago
      I'm not quite sure how to interpret what you've said exactly. Some of what I said is a performance piece. If I insist that I am human and thus make mistakes, then it makes space both for others to make mistakes and for us to have a frank conversation of how, in our more lucid moments, we can do things that fight/counteract our basic nature. It's morally equivalent to putting a plastic cover over the Big Red Button so that pushing it becomes a conscious and considered act. Deciding to push the button is step 1, moving the cover is step 2, and step 1.5 or 2.5 is realizing this is in fact not what you want to do after all, instead of a momentary bit of inattention, bad judgement, or the Imp of the Perverse.. It's not that I won't make mistakes, but that (hopefully) I a make them less often. But making them less often, I'm entrusted with bigger potential for mistakes, and so my mistakes tend to cause more harm. In a Precautionary Principle sense, we end up roughly within the same order of magnitude. When I make 5x fewer mistakes I'm allowed to be responsible for problems that are twice as risky. If you want candor about mistakes you have to start by giving people permission to make them (or at least, to make new ones).
      - p10_user 5y ago
        Everything you are talking about seems centered around cultivating humility. You are absolutely right. It’s ultimately the ability to allow yourself to make mistakes. Being wrong is no fun. Best to get it over with, accept it and work forward, rather than to bring yourself into some delusion.
- Thaxll 5y ago
  It works fine until the core people that know that stack leave your compagny and no one wants to touch it. Then the new people comes in a pich a solution that is widely used.
lnxg33k1 5y ago
I also don’t agree with the concept, and I’ve had problems about with docket using a specific OS version but pulling in some minor software version which broke, so I think even if being slightly true, that statement represent something that might be solved by specifying minor versions for the packages you depend on, which is a level of effort compatible with the one you need on nix to specify packages hashes, so you can have both with docker, a strict and non strict approach, while I guess nix only supports strict
jandrewrogers 5y ago
Sure, Kubernetes can run a database, but not efficiently. Companies with intensive data infrastructure frequently operate at scale without containers, either VMs or bare metal. The larger the data volume, the less likely they are to use containers because efficiency is more important. It is also simpler to manage this kind of thing outside containers, frankly, since you are running a single process per server. People have been building this infrastructure since long before containers, thousands of servers in single operational clusters. Highly automated data infrastructure without containers is pretty simple and straightforward if this is your business.
- anonymousDan 5y ago
  Can you point to any resources on the perf implications of docker for databases?
  - Art9681 5y ago
    OP is likely making assumptions on obsolete knowledge. Tech moves really fast which means the knowledge we gained last week might no longer be relevant. Here is an article showing it is perfectly acceptable to deploy Postgres in Kubernetes: https://www.redhat.com/en/resources/crunchy-data-and-openshift-container-storage-for-PostgreSQL-detail https://www.redhat.com/en/resources/crunchy-data-and-openshi... The great majority of Kubernetes experts I have met use a managed service and have not been exposed to the internals or the control plane back end. An abstracted version of Kubernetes that works well for most use cases but makes it more difficult to solve some of the problems listed in this thread. Most folks dont really know Kubernetes. They know how to deploy X application using X abstraction (Helm, Operators, CD tooling) to some worker node in a public cloud using public sources and thats basically it. It's no wonder they cannot solve some of the common problems because they didnt really deploy the apps. They filled out a form and pushed a button.
  - jandrewrogers 5y ago
    High-performance database engines take complete control of their resources, I/O, and scheduling, completely bypassing the OS kernel. Linux is explicitly designed to allow this. This architecture enables integer factor improvements in throughput on the same hardware, which is why you would design your software this way. Older, slower database engine designs (e.g. Postgres) don't take explicit control of resources in this way and will work reasonably well in containers. The original Linux container design was never intended with this type of software in mind, and tacitly reduces the control a process has over its resources. Invariants required for performance are violated, and therefore performance suffers relative to bare metal (or conditionally VMs). The v2 container implementation, which isn't widely available yet, recognizes these problems and attempts to remedy some of these poor behaviors of the v1 container implementation. If your database does not assume strict control of its underlying hardware resources then this does not affect you. However, this kind of strict resource control is idiomatic in all the highest performing databases architectures, to great effect, so it is the most performant databases that are most adversely affected by containerization. FWIW, virtual machines used to have similar terrible performance for high-performance databases for similar reasons. Now there are mechanisms for the database to effectively bypass the VM similar to the kernel, and the loss of performance is minimal. Container environments typically don't offer similar bypass mechanisms.
zomglings 5y ago
The OP is talking about reproducible builds. A docker image is a build artifact. If I gave you a Dockerfile for a node.js service that installed its dependencies using npm, and the service used non-trivial dependencies, and I gave you this file 10 years after it was first written, chances are you would not be able to build a new image using that Dockerfile. This would be a problem for you if you had to make some small change to that node.js service (without changing any dependencies).
- Art9681 5y ago
  There are other container build tools such as Kaniko that have solved the reproducible builds issue right? If we're operating at scale, it probably means we are using some flavor of Kubernetes and the Docker runtime is no longer relevant. Would Redhat OpenShift's source-to-image (S2I) build process solve the reproducible builds requirement? This space moves quickly and the assumptions we had last week may no longer be relevant. For example Crunchy Postgres offers an enterprise supported Kubernetes Operator that leverages stateful sets. Yet I am reading in these comments that it is an unsolved issue.
- Thaxll 5y ago
  Docker is also used to build things not just run them, multistage build with dependency vendoring, you have a 100% reproducible build. It's very easy to do in Go. You pin the version used to build your app ( the runtime version ) In your git repo you have dependcies That's it.
smilliken 5y ago
Dan Luu wrote an essay on this topic recently: https://danluu.com/nothing-works/ https://danluu.com/nothing-works/ He ponders why it is that big websites inevitably have kernel developers. Way out of their domain of expertise, right? If you adopt a technology, you're responsible for it. When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it. PostgreSQL is used by probably millions of people, but we've had to patch it on multiple occasions and run our fork in production. Nobody wants to do that, but sometimes you have to. The point is, you can't just say "oh, we use kubernetes so we don't have to think about it". No. You added it to your stack, and now you're responsible for it. Pick technologies that you're able to support if they're abandoned, unresponsive to your feature requests and bug reports, or not interested in your use case. Pick open source, obviously. This is another reason I like Nix. It's a one-line change to add a patch file to a build to fix an issue. So I can contribute a fix upstream to some project, and then I don't have to wait for their release process, I can use the patch immediately and let go of it whenever the upstream eventually integrates it. It lowers the cost of being responsible for third party software that we depend on.
- Thaxll 5y ago
  > When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it. I doubt that this happen in reality because Kubernetes cover uses cases for pretty much everyone, now I would doubt the knowledge of a regular dev to try to mimic a solution that k8s already does.

by Thaxll 5y ago

cj 5y ago

> the rest of the world is improving on those tech Does that matter if the current stack works just fine? Imagine someone who built a calculator app with jQuery javascript 10 years ago, and all it does is add and subtract numbers. You could spend time porting it to Ember, and then migrating to Angular, and then porting it to React, and then porting it to React with SSR and hooks. If the calculator app worked with 15 year old code, you can either leave it be, or try to keep up with latest tech trends and continuously refactor and port the code. I think there are a lot of "calculator app" type components of many systems that, at a certain point, can be considered "done, complete" without ever needing a rebuild or rewrite. Even if they were built on technologies that might now be considered antiquated.

Thaxll 5y ago
At some point you just don't want to re-invent the wheel, reminds me a lot of: https://en.wikipedia.org/wiki/Not_invented_here https://en.wikipedia.org/wiki/Not_invented_here
hinkley 5y ago
The people who are invested in their home-grown solutions are usually pretty bad at doing tech support for them. They enjoy writing code, and it doesn't take long before you've created enough surface area that you couldn't possibly keep up with requests even if you wanted to. Which they don't, because nothing has convinced me of the crappiness of some of my code more thoroughly than watching other people try and fail to use it. They are a mirror and people don't always like what they see in it. That guy who keeps talking trash about how stupid everyone else is for not understanding his beautiful code is deflecting. If they're right then my code does not have value/isn't smart which means I don't have value/am not smart, so clearly the problem is that I'm surrounded by assholes. I don't think we would be using as much 3rd party software without StackOverflow, and almost nobody creates a StackOverflow for internal tools. Nobody writes books for internal tools. Nobody writes competitors for internal tools which either prove or disprove the utility of the original tool. All of those options are how some people learn, and in some cases how they debug. You're cutting off your own nose when you reinvent something and don't invest the time to do all of these other things. If I lose someone important or get a big contract I didn't think I could win, I can't go out and hire anyone with 3 years experience in your internal tool, trading money for time. I have to hire noobs and sweat it out while we find out if they will ever learn the internal tools or not. The older I get the more I see NIH people as working a protection racket, because it amplifies the 'value' of tenure at the company. The externalities created really only affect new team members, delaying the date when you can no longer dismiss their ideas out of hand. You have in effect created an oligarchy, whereas most of the rest of us prefer a representative republic (actual democracy is too many meetings).
- p10_user 5y ago
  Nice post. I wish I was reading this 6 years ago, but at least I quickly learned to be humble about the quality of my code.
  - hinkley 5y ago
    I'm not quite sure how to interpret what you've said exactly. Some of what I said is a performance piece. If I insist that I am human and thus make mistakes, then it makes space both for others to make mistakes and for us to have a frank conversation of how, in our more lucid moments, we can do things that fight/counteract our basic nature. It's morally equivalent to putting a plastic cover over the Big Red Button so that pushing it becomes a conscious and considered act. Deciding to push the button is step 1, moving the cover is step 2, and step 1.5 or 2.5 is realizing this is in fact not what you want to do after all, instead of a momentary bit of inattention, bad judgement, or the Imp of the Perverse.. It's not that I won't make mistakes, but that (hopefully) I a make them less often. But making them less often, I'm entrusted with bigger potential for mistakes, and so my mistakes tend to cause more harm. In a Precautionary Principle sense, we end up roughly within the same order of magnitude. When I make 5x fewer mistakes I'm allowed to be responsible for problems that are twice as risky. If you want candor about mistakes you have to start by giving people permission to make them (or at least, to make new ones).
    - p10_user 5y ago
      Everything you are talking about seems centered around cultivating humility. You are absolutely right. It’s ultimately the ability to allow yourself to make mistakes. Being wrong is no fun. Best to get it over with, accept it and work forward, rather than to bring yourself into some delusion.
Thaxll 5y ago
It works fine until the core people that know that stack leave your compagny and no one wants to touch it. Then the new people comes in a pich a solution that is widely used.

lnxg33k1 5y ago

I also don’t agree with the concept, and I’ve had problems about with docket using a specific OS version but pulling in some minor software version which broke, so I think even if being slightly true, that statement represent something that might be solved by specifying minor versions for the packages you depend on, which is a level of effort compatible with the one you need on nix to specify packages hashes, so you can have both with docker, a strict and non strict approach, while I guess nix only supports strict

jandrewrogers 5y ago

Sure, Kubernetes can run a database, but not efficiently. Companies with intensive data infrastructure frequently operate at scale without containers, either VMs or bare metal. The larger the data volume, the less likely they are to use containers because efficiency is more important. It is also simpler to manage this kind of thing outside containers, frankly, since you are running a single process per server. People have been building this infrastructure since long before containers, thousands of servers in single operational clusters. Highly automated data infrastructure without containers is pretty simple and straightforward if this is your business.

anonymousDan 5y ago
Can you point to any resources on the perf implications of docker for databases?
- Art9681 5y ago
  OP is likely making assumptions on obsolete knowledge. Tech moves really fast which means the knowledge we gained last week might no longer be relevant. Here is an article showing it is perfectly acceptable to deploy Postgres in Kubernetes: https://www.redhat.com/en/resources/crunchy-data-and-openshift-container-storage-for-PostgreSQL-detail https://www.redhat.com/en/resources/crunchy-data-and-openshi... The great majority of Kubernetes experts I have met use a managed service and have not been exposed to the internals or the control plane back end. An abstracted version of Kubernetes that works well for most use cases but makes it more difficult to solve some of the problems listed in this thread. Most folks dont really know Kubernetes. They know how to deploy X application using X abstraction (Helm, Operators, CD tooling) to some worker node in a public cloud using public sources and thats basically it. It's no wonder they cannot solve some of the common problems because they didnt really deploy the apps. They filled out a form and pushed a button.
- jandrewrogers 5y ago
  High-performance database engines take complete control of their resources, I/O, and scheduling, completely bypassing the OS kernel. Linux is explicitly designed to allow this. This architecture enables integer factor improvements in throughput on the same hardware, which is why you would design your software this way. Older, slower database engine designs (e.g. Postgres) don't take explicit control of resources in this way and will work reasonably well in containers. The original Linux container design was never intended with this type of software in mind, and tacitly reduces the control a process has over its resources. Invariants required for performance are violated, and therefore performance suffers relative to bare metal (or conditionally VMs). The v2 container implementation, which isn't widely available yet, recognizes these problems and attempts to remedy some of these poor behaviors of the v1 container implementation. If your database does not assume strict control of its underlying hardware resources then this does not affect you. However, this kind of strict resource control is idiomatic in all the highest performing databases architectures, to great effect, so it is the most performant databases that are most adversely affected by containerization. FWIW, virtual machines used to have similar terrible performance for high-performance databases for similar reasons. Now there are mechanisms for the database to effectively bypass the VM similar to the kernel, and the loss of performance is minimal. Container environments typically don't offer similar bypass mechanisms.

zomglings 5y ago

The OP is talking about reproducible builds. A docker image is a build artifact. If I gave you a Dockerfile for a node.js service that installed its dependencies using npm, and the service used non-trivial dependencies, and I gave you this file 10 years after it was first written, chances are you would not be able to build a new image using that Dockerfile. This would be a problem for you if you had to make some small change to that node.js service (without changing any dependencies).

Art9681 5y ago
There are other container build tools such as Kaniko that have solved the reproducible builds issue right? If we're operating at scale, it probably means we are using some flavor of Kubernetes and the Docker runtime is no longer relevant. Would Redhat OpenShift's source-to-image (S2I) build process solve the reproducible builds requirement? This space moves quickly and the assumptions we had last week may no longer be relevant. For example Crunchy Postgres offers an enterprise supported Kubernetes Operator that leverages stateful sets. Yet I am reading in these comments that it is an unsolved issue.
Thaxll 5y ago
Docker is also used to build things not just run them, multistage build with dependency vendoring, you have a 100% reproducible build. It's very easy to do in Go. You pin the version used to build your app ( the runtime version ) In your git repo you have dependcies That's it.

smilliken 5y ago

Dan Luu wrote an essay on this topic recently: https://danluu.com/nothing-works/ https://danluu.com/nothing-works/ He ponders why it is that big websites inevitably have kernel developers. Way out of their domain of expertise, right? If you adopt a technology, you're responsible for it. When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it. PostgreSQL is used by probably millions of people, but we've had to patch it on multiple occasions and run our fork in production. Nobody wants to do that, but sometimes you have to. The point is, you can't just say "oh, we use kubernetes so we don't have to think about it". No. You added it to your stack, and now you're responsible for it. Pick technologies that you're able to support if they're abandoned, unresponsive to your feature requests and bug reports, or not interested in your use case. Pick open source, obviously. This is another reason I like Nix. It's a one-line change to add a patch file to a build to fix an issue. So I can contribute a fix upstream to some project, and then I don't have to wait for their release process, I can use the patch immediately and let go of it whenever the upstream eventually integrates it. It lowers the cost of being responsible for third party software that we depend on.

Thaxll 5y ago
> When Kubernetes inevitably has an issue that is a blocker for us, I don't have confidence in my ability to fix it. When an internal python or shell program has an issue that is a blocker for us, I change it. I doubt that this happen in reality because Kubernetes cover uses cases for pretty much everyone, now I would doubt the knowledge of a regular dev to try to mimic a solution that k8s already does.