3 ms·

Yuck. We blew a ton of money and time, despite a great infra/hpc/etc team (PhD, exGoogle, exNetflix, ex-scale video, etc), on a skilled but overambitious infra

Yuck. We blew a ton of money and time, despite a great infra/hpc/etc team (PhD, exGoogle, exNetflix, ex-scale video, etc), on a skilled but overambitious infra subteam where we should have (and ended on) docker / compose. K8s makes sense if you have a ton of servers or scale or devs, and want to pay employees and team in perpetuity to focus on it, but for half the folks out there, stick with the pieces or you have the cost of openstack and no better off for the increment over just a few of the pieces you can carve out without k8s itself. The article that leadership needs, vs vendor / fanboy bullshit, is "only do k8s if you have these exact scale and ops problems and your stack looks exactly like this and your entire app team looks like this and you have fulltime k8s people now and forever, and you are willing to tax each team member to now deal with k8s glue/ops/perf problems, and xyz other things , otherwise if any of these simplifying or legacy assumptions apply, perfect: you get to focus on these way faster and cheaper agile orchestration options and actions".

reqres 7y ago
> where we should have (and ended on) docker / compose We ended up doing this too. Do you have any links to deploying docker-compose in production? We've not been able to find out much. However our solution seems to work well - am keen to find out how other people are managing host setup, updating and remotely controlling docker-compose In our setup, we essentially use docker-compose as a process manager: - Updates with docker-compose.yml via git syncs - Logging via journald (which in turn is forwarded) - Simple networking with `network: host` and managing firewalls at the host level with dashboard/labelling - Restarts / Launch on startup with `restart: always` policy IMO, more straightforward in the past 1) Entrusting everything to a package manager or build script which might break on a new version release 2) Maintaining ansible script repository to do the host configuration, package management and updates for you - these too always need manual intervention on major version updates
- lmeyerov 7y ago
  Fairly similar. Docker/docker-compose takes care of launch, healthchecks / soft restarts, replication, GPU virtualization & isolation, log forwarding, status checks, and a bunch of other things. Most of our users end up on-prem, so the result is that _customers_ can admin relatively easily, not just us, despite weird stuff like use of GPU virtualization. I've had to debug folks in airgapped rooms over the phone: ops simplicity is awesome. Some key things beyond your list from an ops view: -- containers/yml parameterized by version tag (and most things we gen): simplifies a lot -- packer + ~50 line shell script for airgapped tarball/AMI/VM/etc. generation + tagged git/binary store copies = on-prem + multiple private cloud releases are now at a biweekly cadence, and for our cloud settings, turning on an AMI will autolaunch it -- low down-time system upgrades are now basically launching new instances (auto-healthcheck) + running a small data propagation, and upon success, it dns flips. -- That same script will next turn into our auto-updater on-prem / private cloud users without much difference. They generally are single-node, which `docker-compose -p` solves. -- secrets are a bit wonkier, but essentially docker-compose passes .envs, and dev uses keybase (= encrypted gitfs) and prod is something else Some cool things around GPUs happening that I can't talk about for a bit unfortunately, and supporting the dev side is a longer story. Some of these patterns and the tools involved are normal part of the k8s life... which is my point: going incrementally from docker / docker-compose or equiv lightweight tooling will save your team + business time / money / heartache. Sometimes it's worth blowing months/years/millions and taxing the folks who'd be otherwise uninvolved, but easily for over half the folks out there, probably 90%+.. so not worth it. Instead, as we need a thing, you can see how we incrementally add it from a as-simple-as-possible baseline.
- dc2447 7y ago
  Curious if you tested POC for simple COE?

reqres 7y ago

> where we should have (and ended on) docker / compose We ended up doing this too. Do you have any links to deploying docker-compose in production? We've not been able to find out much. However our solution seems to work well - am keen to find out how other people are managing host setup, updating and remotely controlling docker-compose In our setup, we essentially use docker-compose as a process manager: - Updates with docker-compose.yml via git syncs - Logging via journald (which in turn is forwarded) - Simple networking with `network: host` and managing firewalls at the host level with dashboard/labelling - Restarts / Launch on startup with `restart: always` policy IMO, more straightforward in the past 1) Entrusting everything to a package manager or build script which might break on a new version release 2) Maintaining ansible script repository to do the host configuration, package management and updates for you - these too always need manual intervention on major version updates

lmeyerov 7y ago
Fairly similar. Docker/docker-compose takes care of launch, healthchecks / soft restarts, replication, GPU virtualization & isolation, log forwarding, status checks, and a bunch of other things. Most of our users end up on-prem, so the result is that _customers_ can admin relatively easily, not just us, despite weird stuff like use of GPU virtualization. I've had to debug folks in airgapped rooms over the phone: ops simplicity is awesome. Some key things beyond your list from an ops view: -- containers/yml parameterized by version tag (and most things we gen): simplifies a lot -- packer + ~50 line shell script for airgapped tarball/AMI/VM/etc. generation + tagged git/binary store copies = on-prem + multiple private cloud releases are now at a biweekly cadence, and for our cloud settings, turning on an AMI will autolaunch it -- low down-time system upgrades are now basically launching new instances (auto-healthcheck) + running a small data propagation, and upon success, it dns flips. -- That same script will next turn into our auto-updater on-prem / private cloud users without much difference. They generally are single-node, which `docker-compose -p` solves. -- secrets are a bit wonkier, but essentially docker-compose passes .envs, and dev uses keybase (= encrypted gitfs) and prod is something else Some cool things around GPUs happening that I can't talk about for a bit unfortunately, and supporting the dev side is a longer story. Some of these patterns and the tools involved are normal part of the k8s life... which is my point: going incrementally from docker / docker-compose or equiv lightweight tooling will save your team + business time / money / heartache. Sometimes it's worth blowing months/years/millions and taxing the folks who'd be otherwise uninvolved, but easily for over half the folks out there, probably 90%+.. so not worth it. Instead, as we need a thing, you can see how we incrementally add it from a as-simple-as-possible baseline.
dc2447 7y ago
Curious if you tested POC for simple COE?