4 ms·

I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a serv

I disagree that production is hard in itself, I think the way people approach production makes it needlessly more difficult. It seems that when launching a service, everyone seems to try to get the entire stack in one shot, which is difficult. A much better approach is "1. Make it work 2. Make it good 3. Make it fast". Your initial prototypes need to define the core functionality, and then you incrementally build stuff on top of that.

Veuxdo 4y ago
Security as well. Now step 0 is "Make it secure". When using microservices connected via the internet, you have no choice.
- madeofpalk 4y ago
  > When using microservices 1. Make it work
- dgroshev 4y ago
  Security is a spectrum. You can go from out of the box Debian listening on 22/80/443 directly on internet to WAF, intrusion detection, encrypted overlay networks with CA hierarchy and offline route keys, encryption at rest, elaborate IAM, hardware signed commits, full supply chain audits with every dependency vetted and vendored, multi-access-level logs, mandatory spyware on all corporate gear, and so on and so forth. For most businesses it's more of the former than the latter, and there's definitely no such thing as categorically "secure".
lcw 4y ago
> 1. Make it work 2. Make it good 3. Make it fast Anecdotally, everytime someone tells me Kubernetes is overkill and then follows your approach. In a year they end up building the capabilities that come out of the box with Kubernetes, and of course they aren't as well thought out. Because they were done adhoc and as needed.
- dgroshev 4y ago
  Why is that bad? If the business survived for long enough to have that problem, that's a win, not a failure. Being killed or at least hobbled by unnecessary complexity, on the other hand, is a thing in way more businesses than we like to admit.
  - Spivak 4y ago
    Yep, you can grow to be a huge company with just autoscaling groups and some terraform. Absolutely painfully boring stack. I’m working at a place now with that stack and we haven’t had a single page in over a year now and we do 5 deploys a day ish.
YetAnotherNick 4y ago
> 1. Make it work 2. Make it good 3. Make it fast Disagree. The first two objective clashes very much with each other. I have worked in teams that did that. And generally if anything non trivial is done without planning to just "make it work", it would be more often than not, ridden with many design issues which doesn't allow it to be good.
- ActorNightly 4y ago
  If you start designing into the future, you are now trying to make it good before making it work. Making it work means you start with barebones, just to have a proof of concept that runs, is accessible, is secured, even if you have to click things together and manually ssh into EC2s to launch code. Once you have everything working, then the bigger design should be a lot clearer, in terms of what you have to do to make it good.
- nerdix 4y ago
  I've never seen anything that couldn't be incrementally improved. There are definitely tons of situations where the additional effort to "make it good" is worth incurring upfront. But you can definitely make something work then make it better. And a lot of times, you'll find that your carefully planned out "good" solution ends up not being all that good and needs incremental improvement. It can be hard to get something right on the first try especially if you haven't done it before.
  - jakupovic 4y ago
    Fully agree here. Not being abe to incrementally improve something that works would be a contradiction as the original thing never worked at all then.
  - rektide 4y ago
    I see way more making-poor-decisions underbidding-complexity, then spending years hacking on shit cause it's kind of jank. When folks have license to do good work, I usually see it going well. If they have some informed opinions & group wisdom at their disposal & which they leverage.
yourapostasy 4y ago
When I see developers use k8s as “programmable infra replacing on-prem functionality as a baseline”, some time after step 3 is, “make it as cheap as on-prem”. Unfortunately, many of the “move fast and break things” tenor that got the services built in k8s in the first place, baked in decisions that will cost as much to re-engineer into an elastic cloud-savvy form that also saves money compared to on-prem. This is often from organizational or project management mis-coordination than technical choices the developers are fully aware of. Like a system’s owners not prioritizing putting events of interest into a message queue and insisting upon polling for the information on one end. And an offshore operations team saying learning how to support AWS Lambda/Azure Automation/GCP Cloud Functions is an added cost scope to the contract. So the developer is backed into a corner to run cron/SystemD-based polling off a small EC2. Thousands of decisions like this add up fast, and take a major executive committee commitment to root out tech debt and prevent new tech debt from accumulating to unsustainable levels to prevent cloud initiatives from sinking beneath a mire of expensive small decisions in pursuit of new accomplishments. It is a tough balancing act.

Veuxdo 4y ago

Security as well. Now step 0 is "Make it secure". When using microservices connected via the internet, you have no choice.

madeofpalk 4y ago
> When using microservices 1. Make it work
dgroshev 4y ago
Security is a spectrum. You can go from out of the box Debian listening on 22/80/443 directly on internet to WAF, intrusion detection, encrypted overlay networks with CA hierarchy and offline route keys, encryption at rest, elaborate IAM, hardware signed commits, full supply chain audits with every dependency vetted and vendored, multi-access-level logs, mandatory spyware on all corporate gear, and so on and so forth. For most businesses it's more of the former than the latter, and there's definitely no such thing as categorically "secure".

lcw 4y ago

> 1. Make it work 2. Make it good 3. Make it fast Anecdotally, everytime someone tells me Kubernetes is overkill and then follows your approach. In a year they end up building the capabilities that come out of the box with Kubernetes, and of course they aren't as well thought out. Because they were done adhoc and as needed.

dgroshev 4y ago
Why is that bad? If the business survived for long enough to have that problem, that's a win, not a failure. Being killed or at least hobbled by unnecessary complexity, on the other hand, is a thing in way more businesses than we like to admit.
- Spivak 4y ago
  Yep, you can grow to be a huge company with just autoscaling groups and some terraform. Absolutely painfully boring stack. I’m working at a place now with that stack and we haven’t had a single page in over a year now and we do 5 deploys a day ish.

YetAnotherNick 4y ago

> 1. Make it work 2. Make it good 3. Make it fast Disagree. The first two objective clashes very much with each other. I have worked in teams that did that. And generally if anything non trivial is done without planning to just "make it work", it would be more often than not, ridden with many design issues which doesn't allow it to be good.

ActorNightly 4y ago
If you start designing into the future, you are now trying to make it good before making it work. Making it work means you start with barebones, just to have a proof of concept that runs, is accessible, is secured, even if you have to click things together and manually ssh into EC2s to launch code. Once you have everything working, then the bigger design should be a lot clearer, in terms of what you have to do to make it good.
nerdix 4y ago
I've never seen anything that couldn't be incrementally improved. There are definitely tons of situations where the additional effort to "make it good" is worth incurring upfront. But you can definitely make something work then make it better. And a lot of times, you'll find that your carefully planned out "good" solution ends up not being all that good and needs incremental improvement. It can be hard to get something right on the first try especially if you haven't done it before.
- jakupovic 4y ago
  Fully agree here. Not being abe to incrementally improve something that works would be a contradiction as the original thing never worked at all then.
- rektide 4y ago
  I see way more making-poor-decisions underbidding-complexity, then spending years hacking on shit cause it's kind of jank. When folks have license to do good work, I usually see it going well. If they have some informed opinions & group wisdom at their disposal & which they leverage.

yourapostasy 4y ago

When I see developers use k8s as “programmable infra replacing on-prem functionality as a baseline”, some time after step 3 is, “make it as cheap as on-prem”. Unfortunately, many of the “move fast and break things” tenor that got the services built in k8s in the first place, baked in decisions that will cost as much to re-engineer into an elastic cloud-savvy form that also saves money compared to on-prem. This is often from organizational or project management mis-coordination than technical choices the developers are fully aware of. Like a system’s owners not prioritizing putting events of interest into a message queue and insisting upon polling for the information on one end. And an offshore operations team saying learning how to support AWS Lambda/Azure Automation/GCP Cloud Functions is an added cost scope to the contract. So the developer is backed into a corner to run cron/SystemD-based polling off a small EC2. Thousands of decisions like this add up fast, and take a major executive committee commitment to root out tech debt and prevent new tech debt from accumulating to unsustainable levels to prevent cloud initiatives from sinking beneath a mire of expensive small decisions in pursuit of new accomplishments. It is a tough balancing act.