6 ms·

Except that state on kubernetes seems really hard to do well, and data is inherently state. Also the complexity cost significantly adds to the 'configuration t

Except that state on kubernetes seems really hard to do well, and data is inherently state. Also the complexity cost significantly adds to the 'configuration that outperforms a single thread'. 99% of the time you'll be better off putting a docker container on a single box.

j4hdufd8 5y ago
> 99% of the time you'll be better off putting a docker container on a single box. Sorry, what did you mean here? This doesn't make a lot of sense.
- deleted 5y ago
  [deleted]
- spicyusername 5y ago
  I think they mean that 99% of the time you're better off using a single container on a single server and wiring them together using traditional tools (load balancers, network-attached storage, etc) than you would be if you adopt Kubernetes. This gives you many of the same application-development benefits (standard packaging format, write once run anywhere, etc) that Kubernetes does without the overhead of managing the infrastructure complexity. Kubernetes is so complicated and many organizations do not really think about what problems they are trying to solve before diving in. Kubernetes solves so many problems that most IT organizations don't actually have or that are perfectly solvable in a simpler way with the existing virtual-machine ecosystem.
  - superkuh 5y ago
    99% of the time for 99% of people you're better off not using a container at all and actually running your application on your OS instead of under $alot of layers of extra abstraction. Of course if you're not a human person it's different. Corporate people, being made up of multiple different people, care more about their human elements being able to interact with each other's work. There containers might help.
    - beckingz 5y ago
      Microservices do allow scaling the development of an application on the human side. This is true! A kickass CICD setup and good developer tools probably should be done first for the typical company, otherwise the majority of the added engineering time will go into scaling the infrastructure setup.
    - zozbot234 5y ago
      Containers do not add extra abstraction layers, they're about proper namespacing of all resources. They simplify the OS's existing abstractions and remove a lot of pointless attack surface.
      - zekrioca 5y ago
        Exactly. Some people think containers are a whole new layer, akin to a VM. I think it is just their mental model that are used to equate containers and VMs.
      - ElectricalUnion 5y ago
        > They simplify the OS's existing abstractions I am pretty sure they're supposed to use a somewhat "Cooperative Namespacing" schema, that makes both the application and the OS/container abstraction layer more complex, while attempting to deliver lower overhead for said namespacing. Sorta like Cooperative Multitasking, compared to Preemptive Multitasking. And even that doesn't work for some situations; as an example, old versions of Java (7 and older) will use the incorrectly namespaced Linux file system APIs to incorrectly read whole-system limitations instead of current container limitations to setup memory and CPU parameters. > remove a lot of pointless attack surface. Not really, containers are for lower overhead cooperative namespacing, not security. You want bare minimum lightweight VMs for security.
    - pengaru 5y ago
      Unless severely resource constrained, it makes sense to at least toss the application under something light like an nspawn container. If you're using systemd hosts it's already there. Even if you don't use user or network namespaces it at least confers a desirable separation of application and its dependencies vs. underlying host at the filesystem level. It simplifies administration and allows your app and its dependencies to move at a pace decoupled from the host kernel and its minimal dependencies required for providing ssh access and systemd.
- beckingz 5y ago
  spicyusername has it right. There's a concept called COST - Configuration that Outperforms a Single Thread, which basically asks the question: "how many cores/servers does a system need to outperform a single thread/core". Often this means that to outperform a single beefy server you need to get to 100+ servers to overcome the coordination costs and actually do more processing or process more faster. Lots of workload specific details here, but the general application in data engineering is "How many nodes does your spark cluster need to outperform my laptop, or how many nodes does your cluster need to outperform a single turbo.large.beef instance?" My belief is that most companies chase scalability wrong in data. That said, there are some companies that handle telemetry at scale and actually need to record and analyze billions of events per second. In these cases you will trivially hit the scaling limits of a single server. 99% of companies are not here. I could be convinced that this number is actually as low as 90% :)
  - candiddevmike 5y ago
    On a lot of platforms you'll see better cost savings scaling vertically too. With a Go app, it'll run just as good (or better?) on one (or two for HA?) beefy server vs a fleet, the http router will saturate all of your cores just fine.
nomadiccoder 5y ago
It took me a long time to come to terms with this fact. I was once asked during an interview what I wouldnt run in k8s and I said legacy apps and hardware specific apps. Now I think that for many applications cloud run or cloud function can reduce the complexity of software delivery. It takes a team just to manage the k8s infra while an app team can be responsible for their own cloud run app.
- raesene9 5y ago
  When I talk to people about k8s, if they haven't containerized their apps yet, I generally recommend starting with a simple CAAS setup and then, if you need some of the flexibility and power that k8s provides, then move those apps into a cluster. It can do a lot of cool stuff, but there's plenty of complexity there, and there are quite a few potential nasty surprises in there (from a security standpoint anyway, which is where I focus)
- politelemon 5y ago
  By cloud function, do you mean something like AWS Lambda?
  - candiddevmike 5y ago
    OP is using google cloud/GCP, cloud functions and cloud run are google cloud serverless offerings. Cloud functions is comparable to lambda, cloud run is basically a kubernetes-based PaaS.
    - fulafel 5y ago
      CFs are higher level than Lambda, they can implement HTTP endpoints directly. More like Serverless framework, or a combination of API Gateway + Lambda. And they have good container support, Lambda container support is a bit iffy.
      - politelemon 5y ago
        Thanks both for explaining. Now I understand why, recently AWS Lambda added support for invocation URLs. Or I thought they did but the page seems to have been taken down now.
taeric 5y ago
This is an interesting facet of software that feels lost. The naive way can often scale remarkably well. Even better, if you leave an approach stable, as in not changing, you can often get a lot of improvements by focusing on other sections. This isn't too say that throwing more code and complexity can't help. But it is probably no different than any other help, in that most gains are probably slow to realize and more marginal than you'd hope.
hodgesrm 5y ago
> Except that state on kubernetes seems really hard to do well, and data is inherently state. This is changing. For databases the usual approach is to write an operator. Most databases these days typically have one (or more).
maccard 5y ago
the problem with container on a single box is that you still need to solve a bunch of problems thay k8s solves for you with that. By the time you've handled deploying updated containers, a sidecar for logging and/or metrics, service discovery, and decided you _maybe_ want two instances of your app, you likely should have just used k8s from the get go. You get all of that straight out of the box, and it's going to be much easier to hire someone to manage that than the mess of bash scripts you end up with otherwise.
- deleted 5y ago
  [deleted]

j4hdufd8 5y ago

> 99% of the time you'll be better off putting a docker container on a single box. Sorry, what did you mean here? This doesn't make a lot of sense.

deleted 5y ago
[deleted]
spicyusername 5y ago
I think they mean that 99% of the time you're better off using a single container on a single server and wiring them together using traditional tools (load balancers, network-attached storage, etc) than you would be if you adopt Kubernetes. This gives you many of the same application-development benefits (standard packaging format, write once run anywhere, etc) that Kubernetes does without the overhead of managing the infrastructure complexity. Kubernetes is so complicated and many organizations do not really think about what problems they are trying to solve before diving in. Kubernetes solves so many problems that most IT organizations don't actually have or that are perfectly solvable in a simpler way with the existing virtual-machine ecosystem.
- superkuh 5y ago
  99% of the time for 99% of people you're better off not using a container at all and actually running your application on your OS instead of under $alot of layers of extra abstraction. Of course if you're not a human person it's different. Corporate people, being made up of multiple different people, care more about their human elements being able to interact with each other's work. There containers might help.
  - beckingz 5y ago
    Microservices do allow scaling the development of an application on the human side. This is true! A kickass CICD setup and good developer tools probably should be done first for the typical company, otherwise the majority of the added engineering time will go into scaling the infrastructure setup.
  - zozbot234 5y ago
    Containers do not add extra abstraction layers, they're about proper namespacing of all resources. They simplify the OS's existing abstractions and remove a lot of pointless attack surface.
    - zekrioca 5y ago
      Exactly. Some people think containers are a whole new layer, akin to a VM. I think it is just their mental model that are used to equate containers and VMs.
    - ElectricalUnion 5y ago
      > They simplify the OS's existing abstractions I am pretty sure they're supposed to use a somewhat "Cooperative Namespacing" schema, that makes both the application and the OS/container abstraction layer more complex, while attempting to deliver lower overhead for said namespacing. Sorta like Cooperative Multitasking, compared to Preemptive Multitasking. And even that doesn't work for some situations; as an example, old versions of Java (7 and older) will use the incorrectly namespaced Linux file system APIs to incorrectly read whole-system limitations instead of current container limitations to setup memory and CPU parameters. > remove a lot of pointless attack surface. Not really, containers are for lower overhead cooperative namespacing, not security. You want bare minimum lightweight VMs for security.
  - pengaru 5y ago
    Unless severely resource constrained, it makes sense to at least toss the application under something light like an nspawn container. If you're using systemd hosts it's already there. Even if you don't use user or network namespaces it at least confers a desirable separation of application and its dependencies vs. underlying host at the filesystem level. It simplifies administration and allows your app and its dependencies to move at a pace decoupled from the host kernel and its minimal dependencies required for providing ssh access and systemd.
beckingz 5y ago
spicyusername has it right. There's a concept called COST - Configuration that Outperforms a Single Thread, which basically asks the question: "how many cores/servers does a system need to outperform a single thread/core". Often this means that to outperform a single beefy server you need to get to 100+ servers to overcome the coordination costs and actually do more processing or process more faster. Lots of workload specific details here, but the general application in data engineering is "How many nodes does your spark cluster need to outperform my laptop, or how many nodes does your cluster need to outperform a single turbo.large.beef instance?" My belief is that most companies chase scalability wrong in data. That said, there are some companies that handle telemetry at scale and actually need to record and analyze billions of events per second. In these cases you will trivially hit the scaling limits of a single server. 99% of companies are not here. I could be convinced that this number is actually as low as 90% :)
- candiddevmike 5y ago
  On a lot of platforms you'll see better cost savings scaling vertically too. With a Go app, it'll run just as good (or better?) on one (or two for HA?) beefy server vs a fleet, the http router will saturate all of your cores just fine.

nomadiccoder 5y ago

It took me a long time to come to terms with this fact. I was once asked during an interview what I wouldnt run in k8s and I said legacy apps and hardware specific apps. Now I think that for many applications cloud run or cloud function can reduce the complexity of software delivery. It takes a team just to manage the k8s infra while an app team can be responsible for their own cloud run app.

raesene9 5y ago
When I talk to people about k8s, if they haven't containerized their apps yet, I generally recommend starting with a simple CAAS setup and then, if you need some of the flexibility and power that k8s provides, then move those apps into a cluster. It can do a lot of cool stuff, but there's plenty of complexity there, and there are quite a few potential nasty surprises in there (from a security standpoint anyway, which is where I focus)
politelemon 5y ago
By cloud function, do you mean something like AWS Lambda?
- candiddevmike 5y ago
  OP is using google cloud/GCP, cloud functions and cloud run are google cloud serverless offerings. Cloud functions is comparable to lambda, cloud run is basically a kubernetes-based PaaS.
  - fulafel 5y ago
    CFs are higher level than Lambda, they can implement HTTP endpoints directly. More like Serverless framework, or a combination of API Gateway + Lambda. And they have good container support, Lambda container support is a bit iffy.
    - politelemon 5y ago
      Thanks both for explaining. Now I understand why, recently AWS Lambda added support for invocation URLs. Or I thought they did but the page seems to have been taken down now.

taeric 5y ago

This is an interesting facet of software that feels lost. The naive way can often scale remarkably well. Even better, if you leave an approach stable, as in not changing, you can often get a lot of improvements by focusing on other sections. This isn't too say that throwing more code and complexity can't help. But it is probably no different than any other help, in that most gains are probably slow to realize and more marginal than you'd hope.

hodgesrm 5y ago

> Except that state on kubernetes seems really hard to do well, and data is inherently state. This is changing. For databases the usual approach is to write an operator. Most databases these days typically have one (or more).

maccard 5y ago

the problem with container on a single box is that you still need to solve a bunch of problems thay k8s solves for you with that. By the time you've handled deploying updated containers, a sidecar for logging and/or metrics, service discovery, and decided you _maybe_ want two instances of your app, you likely should have just used k8s from the get go. You get all of that straight out of the box, and it's going to be much easier to hire someone to manage that than the mess of bash scripts you end up with otherwise.

deleted 5y ago
[deleted]