12 ms·

Distracting Infrastructure

136 points by kristianp 8y ago

perfunctory 8y ago
> ‘Engineers get sidetracked by things that make engineers excited, not that solve real problems for users’ The real question is why business allows them to. I find it entertaining to try to substitute another profession for engineer in the sentence above and see if it still works. Lawyers, teachers, doctors, civil architects...
- pacifika 8y ago
  Well perhaps it’s the business doing pie in the sky thinking resulting in these solutions. It’s not without a reason the Indy dev is highlighted here.
  - DyslexicAtheist 8y ago
    in most companies (I know) the business stakeholders have little idea about what makes sense within and for engineering. So they rely on their 'subject matter expert' engineers to find the best decisions. (and a place that doesn't trust their engineers to come up with the right solution is doomed imo). This problem usually indicates a weak team, or lack of leadership within engineering. But granted, selecting a weak CTO (who picks snowflake & diva engineers) should be attributed to the business.
- arethuza 8y ago
  My experiences with lawyers suggests that they can have a tendency to get sidetracked that is most definitely "not that solve real problems for users" - the real difference with lawyers being that they will then charge you vast amounts of money for their sidetracks.
- cturner 8y ago
  The precious commodity with our work is cognitive load. We can build tooling that reduces future cognitive load. As a result, we have access to multipliers. There is no analogy to this with the other fields.
  - philwelch 8y ago
    The idea that tooling "reduces" cognitive load is based on the assumption that tooling lets you work in terms of abstractions rather than worrying about what the tooling actually does. This assumption breaks down as soon as the tooling itself breaks down. At that point, not only do you have to worry about the underlying problems that the tooling is intended to solve, but you also have to worry about the implementation details of the tooling itself. Tooling breaks down more often when it's more complicated, and you need more complexity when your tooling abstracts over a large amount of complexity, so the more that your tooling is supposed to reduce your cognitive load, the more likely it will break in such a way that it actually increases your cognitive load.
    - deleted 8y ago
      [deleted]
- dasKrokodil 8y ago
  Pilots!
- TheOtherHobbes 8y ago
  'CEOs get excited by things that make CEOs excited, not that solve real problems for real users...' Still works, mostly. Certainly true of architects, partially true of lawyers and doctors, less true of teachers. The key difference is the amount of f2f time with users/customers/clients. If you add a dumb feature to your site and you have to talk to users about it f2f - not over Twitter or Slack or Medium, but in person - you're going to give yourself a much more realistic idea of what matters to them.
  - mr_toad 8y ago
    > CEOs get excited by things that make CEOs excited They seem particularly enarmoured with re-arranging the org-charts.
- Jordanpomeroy 8y ago
  > `you’re on Medium after all’ Ahhhhh! You got me!
AstralStorm 8y ago
Usually,. the thing that you lose is scalability, availability any reliability. You will need a remote backup and a secondary located service, just in case. Potentially a third development one to not push breaking changes directly into production. Skipping any of this will make your service brittle in addition to not being scalable. And it already hits the rule of three... (of "don't repeat yourself") that makes a lightweight Kubernetes deployment worthwhile. (Perhaps without all the bells and whistles.) Remember, downtime is usually lost money. A day's worth can be thousands of dollars at least, and that's two nines already. (99.99%)
- chii 8y ago
  Or, you ship 2 months earlier, and the trade off being suffering a day's worth of downtime. It makes no sense to start with a more complicated setup, until there is clear evidence of a need.
  - AstralStorm 8y ago
    The main problem is that usually low quality infra bites you when you get exposure. Which is exactly when you don't want it to fail... For a start, Heroku or AWS instance will do and they scale. (As the other commenter said) If you're doing your own infra, you need at least basic fallbacks and safety.
- theblackcat1002 8y ago
  Well if the first day you shipped your product and it's already worth 1k, yeah it makes sense to do everything the "correct" way. But for most of the side projects and startup products this makes no sense at all.
warp_factor 8y ago
I saw so many places falling for this fallacy. Instead on working on the core products, engineers would have "fun" adding more microservices, redundant Kubernetes clusters, istio controllers etc etc. (all of that with less than 10 users).
- otabdeveloper1 8y ago
  Nobody ever got promoted or a raise because the core product 'just works'. As an engineer, you need growth in headcount or complexity if you want a raise. It's a people management problem that flows from the CEO and form the HR department, not a technical one.
  - expertentipp 8y ago
    One gets fired when everything „just works”. Everything works (untils it stops working) so your job is done, what are you doing even during your office hours? I wish having a „technical manager” would avoid such situations but it’s enough if this „technical manager” hates the whole domain you work in.
- collyw 8y ago
  When people start adding micro services (or whatever the fad is this month) ask "why?" "Because it scales" "But we don't have a scaling problem, and if we did it could easily be solved with some caching". Unfortunately there is also the bullshit which is hiring in this industry, where solving things the simple way doesn't add anything interesting to your CV.
  - VBprogrammer 8y ago
    People who underestimate the challenges adding caching can cause make me a bit nervous. Especially when you start putting caches on top of caches. You can end up with a hunt which cache has the incorrect value problem.
  - jordanbeiber 8y ago
    Just my 2c regarding ”microservices” or SOA, whatever you want to call it, is that, sure it can enable you to scale horizontally from a technical perspective, but few have to deal with those problems for real. What it does though when organically implemented, is it allows you to scale dev teams more easilly and it will also let you build functionallity and features more rapidly for an ever evolving business. This is my take-away from two massive undertakings at two separate medium-to-large scale enterprises that I’ve been involved in. One was a complete failure and one an enormous success that both have had business altering implications.
carlisle_ 8y ago
>Your users to don’t care how your code gets onto your server. I don't like arguments like this. This is true for any number of tasks that a layman wouldn't care to understand. You still will do them anyway, regardless of whether or not your user cares. It doesn't take a lot of experience to see the correlation between poor engineering practices and poor user experience. Now, I don't think the author was using this argument to pitch poor engineering, but there are good reasons for setting up resilient and consistent build deploys, as he addresses. I think the only real takeaway is "dont over-engineer your infrastructure" which is just an extension of generally "don't over-engineer." Every time infrastructure gets talked about on HN the conversation gets so sidetracked. The real problem is that people don't think about infrastructure in the context of a holistic view of a project. The hours you spend setting up your project with resilient and consistent CI builds are many times more valuable down the road. I think people need to not be so afraid of spending time on infrastructure. People already err on the side of spending less than more time on infrastructure. I don't think it's wrong at all that the author spent hours building an automatic build process for his project, especially because now that he's set it up he can refer to it in the future. Maybe with his newfound knowledge and experience on the problem means that he can leverage it with little effort in the future. None of this is a bad thing especially if you think about your engineering career on a macro scale instead of week to week.
- 0db532a0 8y ago
  I agree that CI is a brilliant investment. Also it can initially be as simple as a git hook. However, I agree with the author’s point that you do not need a Kubernetes distributed NoSQL megagalaxy when you have a number of customers which you can count on one hand. I’ve seen so much time wasted on this sort of thing. One obvious benefit of the whole single/simple server idea is that you can make changes to the system so quickly and easily, which is very important at the beginning.
  - davidjnelson 8y ago
    Ya nosql can be useful at scale, but it's not a great choice for most new projects. It really locks you into your schema and makes querying the data at scale difficult outside of use cases you originally designed for. I've seen aws architects recommending just starting with sql, and adding nosql later as needed. Definitely agree with this approach for most use cases.
reacharavindh 8y ago
Good old KISS policy. Sure, Kubernetes, fully automated CI/CD, autoscaling, lambda functions are all cool and very useful if you have a problem and paying users in that scale. Majority of the applications even today in 2019 would be served well by the reliable Nginx + (Python, C++, Rust, Go or other language of choice), PostgreSQL and backups. If you keep things simple, you can even have a standby mirror of the same infrastructure to take over if needed, and it would still cost you less than spinning up nodes after nodes for the complexity of running your own Kubernetes.
- scarface74 8y ago
  Or just set up Elastic Beanstalk with AWS or something similar, hand it a zip file of your code and let AWS configure your load balancer, autoscaling, patching your OS and runtime, managing your database and backups, etc.
  - reacharavindh 8y ago
    What you suggest may be "simple" in terms of the amount of work the user needs to do, but my comment is about reduced complexity and $$ because you don't need them yet. When you set up such an infrastructure like you proposed, there are still too many wheels in motion(complexity), and even worse they are abstracted away from you further and further. And, I'm sure the AWS bill is not going to be pretty considering they do all those bells and whistles.
    - scarface74 8y ago
      You literally click a few buttons and it sets up all of the infrastructure. Then you give it a zip file. The EC2 instance that it creates is a regular VM. Elastic Beanstalk itself is free.
      - reacharavindh 8y ago
        It is true that Elastic Beanstalk is free, and is very easy for a developer to use to begin with. However, If I were to advice a new startup on their infrastructure, I would not recommend it for the following reasons. It is my personal judgement. Others may prefer to go all in on Amazon. 1. It is clearly a vendor lock in. You build your business upon all AWS services, using an AWS orchestrator - good luck ever getting out. 2. Abstraction. It is a controversial opinion. I'd rather have less magic when possible and know what goes on underneath, and possibly have control over. Now, a startup may choose to offload the duties of operating a database(RDS), storing files(S3), performing backups(S3/Glacier) in the interest of time and trade-off control, optimisation and $$. However, considering the generalisation of simpler startups, IMHO operating a small scale Postgres on a standard EC2 instance is not really much of a hassle... 3. Cost - however small it may be in terms of actual dollars, AWS managed services(S3, RDS, SES, SQS etc..) come with added cost over simply running their open source equivalent yourselves. Again, Keeping it simple - not in terms of what the user needs to do to deploy their application, but in terms of what components do you need to get your application running in production.
lbill 8y ago
I agree with the author : start simple, then use more complex tools when and if you need to. I believe it is true about infrastructure, and about features and code as well. When my team needs to release a functionality for the "product"[1] that we maintain, we have a very simple strategy : we go step by step. The users tell us what they need, and we start to deliver as soon as we can. Our first release only solves a portion of the user's problem, but the second release solves more, and so does the next, and so on. I devised this strategy with the users to make sure they are fine with it: they expect a perfect product... eventually. But in the mean time, they having only some portions of their needs met, and we avoid the feeling of being overwhelmed by a great apparent complexity. [1] The context is a bit complicated... and irrelevant. We don't exactly maintain a product, but it's close enough for the argument here.
- lbill 8y ago
  Oops! Sorry for the huge typo in my comment. "they having only some portions of their needs met" should be read "they are fine with having only some portions of their needs met". (It is a bit more than a typo)
mdekkers 8y ago
You don't need it if you are a one-man band doing a small side project. > I spent hours setting up a simple and free continuous deployment pipeline. Commit to git, Semaphore CI builds the Docker image, uploads the image to a container registry, then pulls & restarts the container on my server. It’s really cool. I felt like a wizard once I got it working … and it was free! I actually sat down here to write all about how to do it yourself, until I realised how long I spent on it and how much value it delivered to my users If you have a team of some size, the time you gain over the lifecycle of the project can be very significant when you have a smoothly running automated dev/test/UA/live pipeline. > Not build the fanciest deployment pipelines, or multi-zone, multi-region, multi-cloud Nuclear Winter proof high availability setup. If your downtime can be measured in appreciable lost revenue or reputational cost, then yes, you need all of that, as well as a solid SRE or 5 to keep it all running.
- sleepychu 8y ago
  > Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great. You have the funds to do things “right”.
  - mdekkers 8y ago
    It doesn't have to be "vast quantities of cash", it only has to balance out the cost of the additional work. And reputational cost is notoriously difficult to turn into a cash value. I really have an issue with how the author presents this as a "Almost nobody needs this stuff", where the truth is that it is a lot more nuanced. I typically advise my clients what uptime will cost them extra, per "9" (as in the 5-nines), and let them make their own decisions.
  - scarface74 8y ago
    You don’t have to be a FAANG. You can be a B2B startup with one or two major clients who if they see you aren’t highly available will jump ship taking a lot of your revenue with them. I’ve been in enough meetings working for small startups where our clients grilled us about our resiliency, availability, redundancy etc. and where they forced us to put our code in escrow as part of our contract.
sbhn 8y ago
It depends on who your customer is. If your customer is the investor, then there is no limit to the amount of hardware, infrastructure, and experts sitting behind a desk and multiple monitors in every concevable separation of concern you can think of in every major city across the globe. Your product is amazing, and its going to change the world and how people percieve things. You need everything you can throw at it. Investors want to be the best. And thats if you customer is the investor.
alfonsodev 8y ago
When you are building a startup, there is always the question, "will this scale?" or "in the event of an investment or going viral are we ready to scale automatically ?" I think that there is a lot of worries around this, it conditions technology choices, sometimes pushing early optimisations, and it is hard because you don't have the data on how many concurrent users your product will have. The best you can do is to extrapolate from you current data or other products data. So many people end up following a recipe, it could be Firebase recipe, AWS Fargate, Serverless... you name it. Recipes that promise limitless autoscale. I think indeed is distracting and premature, but again is hard to avoid it, if the premise is that there will be a massive growth overnight, which is imposible to predict that will potentially kill us, and we have to be ready.
- jcadam 8y ago
  I've wasted a lot of time during my attempts as a solo developer to build a product with a scalable backend architecture for the hordes of users that never materialize. Next idea I have, I swear I'm going to throw all my effort into the frontend and go with as minimal a backend as I can get away with (heck, depending on the application, human-in-the-loop might be fine early on) until I get some validation. It'll feel "wrong", but whatever.
m90 8y ago
While I think you can definitely get too fancy with all of that sweet infra, I feel this is missing an important point: Once you have a working pipeline in place, you can focus on the features, that's the whole point of the exercise: enabling developers to ship features for users. If I have to SSH into some machine and restart a set of services manually in the correct order, it will keep me from releasing features. I am afraid / lazy / whatever. Spending time on this is not something that will never ever reach the user in any way. The user does not care if something is using Heroku, K8s, or something else, yes, but the user definitely appreciates a product that can reliably iterate on things without major downtime or similar problems. It's kind of like saying, "well unit tests are nice and stuff, but does the user ever notice?". I would strongly argue the user will notice unit tests in the long run as they ensure the software keeps being maintainable.
- afarrell 8y ago
  When you walk into a restaurant, do you notice if they have an efficiently laid-out kitchen with mise-en-place, good relationships with their suppliers, or a refrigerator at the proper temperature? No. But you do notice when you food is slow to arrive, the food doesn't taste fresh, or the chicken gives you food poisoning.
- ctnaxx 8y ago
  But many applications were better around 2002-2004 than now. The only thing that's better now for users who don't care about privacy at all is the opaque sign-in-with-X feature. Email web interfaces got worse, login forms got worse, documentation got worse, search got worse (peaked around 2004).
  - m90 8y ago
    Which ones? Which feature?
  - ox_cable 8y ago
    I've never seen a more biased, unsubstantiated, "back in my day" comment on HN. Well done. Give specifics if you want to be taken seriously instead of just saying "well, back in my day everything was better".
jdietrich 8y ago
There's an obvious and overlooked middle ground between "massively complicated infrastructure" and "a cheap VPS" - charge more and use PaaS. Heroku and Engine Yard are expensive, but they deliver a huge amount of value. You've got better things to do than patch Nginx or deal with a crashed server at 4am, just pay someone a sensible amount of money to run your infrastructure for you.
- pseudoramble 8y ago
  I agree. There are also options to make some of the things from the article relatively easy. Off the top of my head, a CI/CD pipeline can be set up using Gitlab , Azure DevOps, Bitbucket Pipelines, and I'm sure tons of others. They do some of the heavy lifting and you still get the benefit of an organized build that's easy to deploy to one of the services listed above. And yeah, this may be layered on later when it seems like the idea has legs and you'd benefit from this. It most likely doesn't need to be the first step.
- bufferoverflow 8y ago
  > an obvious and overlooked middle ground between "massively complicated infrastructure" and "a cheap VPS" It's called a dedicated server. These days you can get a monster of a machine. Even on AWS you can get a machine with 2TB of RAM and 128 cores.
- davidjnelson 8y ago
  Heroku is fine, but their lack of an autoscaling database for spiky load and minimum $250/mo for autoscaling web servers are really very far behind what using aws directly offers these days. Heroku seems to have stopped innovating.
cdoxsey 8y ago
Please at least use a managed database. Treat that linode box like it might disappear at any moment. It is so easy to shoot yourself in the foot with a database and lose everything. Also FWIW configuring Linux machines manually over SSH and building and deploying your code too it, isn't always easier. Make sure you write everything you did down, because in 6 months you will completely forget how you built this thing, where it was deployed, etc...
- mark_l_watson 8y ago
  I have not done this for a number of years, but in the past for my own and customer projects I would just run Postgres on the same VPS as the web app but use a cronjob to write incremental backups every few hours to S3. As other people here have said, agreeing with you, paying for multiple availability zone RDS or paying Heroku is also a good plan.
- Cthulhu_ 8y ago
  And whatever you choose, TEST YOUR BACKUPS. Spin up a new instance and practice recovery. This needs to be done at least quarterly, both to test your backups and train yourself and/or your people in this process.
- mcshaggins 8y ago
  Please write everything down and be kind to 'Future You'. I am very anti-spending money on software that isn't directly related to a project but a $10 confluence wiki lifetime license is one of the best purchases I have ever made.
mark_l_watson 8y ago
Right on! I am enjoying the book “A Company of One” right now and the author makes the same point of only doing what it takes to provide great service to a few customers. Another point is to spend more effort servicing existing customers than spending time trying to get new customers. For deployment: if you can tolerate rare service downtimes then it is difficult to beat Linode or Digital Ocean. I am probably in a minority, but I also really like GCP’s AppEngine: turn on billing, set the minimum instance count to one and the maximum to two or three, and you have a resilient system. That said, I run my primary web site on a single tiny GCP VPS instance with no fault tolerance. For heavier compute requirements, I also really like a physical server from Hetzner- for about $100/month you can get a GPU, i7, and lots of RAM and bandwidth.
- robodale 8y ago
  Thanks for the "A Company of One" book recommendation. I just ordered it.
jpttsn 8y ago
I run a small team, but we do a large number of projects. Each project is developed fast and then maintained at low intensity for a long time. Have found great use for a nifty complex DevOps pipeline. Before, when I built the same app for two years, the nifty pipeline was overwrought. I think the context switching inherent in what I do now makes the nifty pipeline useful.
q-base 8y ago
I see this "Push for new and shiny solutions"[1] across everything from startups to large corporations. And I am genuinely surprised at how often I see people falling for this good old trap of over-optimization/engineering. I genuinely cannot fathom why people want the complexity. It is by no means constrained to infrastructure, it is architecture, technology, management the whole lot. [1]http://jesperreiche.com/the-push-for-new-and-shiny-solutions-to-old-known-problems/ http://jesperreiche.com/the-push-for-new-and-shiny-solutions...
jb3689 8y ago
The problem with Kubernetes (and microservices in general) is getting a team productive with it, and that requires lots of time, dedication, and tooling. It's not the right solution for most teams, but it could be if we streamline it more
ratling 8y ago
This is a pretty dumb article. Like, no shit don't build more infrastructure than you need. If you're a one or two man team you don't need anything more complicated than 'push container here, pull down here' or 'push code here' et. al. Once you get above that having a repeatable process (repeatable by someone besides the person who built it) becomes valuable. The 'every second spent on infrastructure isn't spent on features' line makes me think I'm going to have to blackmail into working on security updates or paying down technical debt. And the picture you picked is cheeky. I will agree with this statement, "You need the most simple, least time consuming infrastructure that gets you to that point." But that WILL become more than just 'throw it on a linode instance' after you get some customers and more than 2 people working on whatever uber for dogs project you have going on.
draw_down 8y ago
Your users don’t care about the text editor you use to write code, but I bet you use something more complex than nano. Tools are to be valued differently.
SnacksOnAPlane 8y ago
One of my web sites is run by a "bundle exec rails s" in a tmux window on an EC2 instance. New deploy? Run "git pull" (and maybe "rake db:migrate") in tmux pane and restart the "rails s" command. Easy peasy. I'd make it more complicated and resilient but why?
davidjnelson 8y ago
> Kubernetes clusters? Load balancers across multiple cloud regions? Zero-click, Blue-Green deployment based on real-time metrics from a continuous deployment pipeline? Kubernetes, multi region, canary deployment, immutable infrastructure, even infrastructure as code is overengineering for a brand new indie project, for sure. BUT, there is a certain minimum amount of quality, availability and scalability any product needs, especially a new product that might show up on hacker news or product hunt and get 100-300 concurrent requests a second for multiple hours at unpredictable times. To not use fargate because you have to dockerize your app and build it during a codepipeline stage is silly. That takes what, a week, max, the first time you ever do it? Once you've done it, you can do it faster for future products. To not use an autoscaling and high availability database when it costs the same and is so simple to set up makes no sense either. To not use ci/cd to test your db migrations and new code on staging before pushing to prod is just dumb and asking for pain and suffering. Why put yourself through that when products have been built to make these things _so_ easy and inexpensive now?

136 points by kristianp 8y ago

> ‘Engineers get sidetracked by things that make engineers excited, not that solve real problems for users’ The real question is why business allows them to. I find it entertaining to try to substitute another profession for engineer in the sentence above and see if it still works. Lawyers, teachers, doctors, civil architects...

pacifika 8y ago
Well perhaps it’s the business doing pie in the sky thinking resulting in these solutions. It’s not without a reason the Indy dev is highlighted here.
- DyslexicAtheist 8y ago
  in most companies (I know) the business stakeholders have little idea about what makes sense within and for engineering. So they rely on their 'subject matter expert' engineers to find the best decisions. (and a place that doesn't trust their engineers to come up with the right solution is doomed imo). This problem usually indicates a weak team, or lack of leadership within engineering. But granted, selecting a weak CTO (who picks snowflake & diva engineers) should be attributed to the business.
arethuza 8y ago
My experiences with lawyers suggests that they can have a tendency to get sidetracked that is most definitely "not that solve real problems for users" - the real difference with lawyers being that they will then charge you vast amounts of money for their sidetracks.
cturner 8y ago
The precious commodity with our work is cognitive load. We can build tooling that reduces future cognitive load. As a result, we have access to multipliers. There is no analogy to this with the other fields.
- philwelch 8y ago
  The idea that tooling "reduces" cognitive load is based on the assumption that tooling lets you work in terms of abstractions rather than worrying about what the tooling actually does. This assumption breaks down as soon as the tooling itself breaks down. At that point, not only do you have to worry about the underlying problems that the tooling is intended to solve, but you also have to worry about the implementation details of the tooling itself. Tooling breaks down more often when it's more complicated, and you need more complexity when your tooling abstracts over a large amount of complexity, so the more that your tooling is supposed to reduce your cognitive load, the more likely it will break in such a way that it actually increases your cognitive load.
  - deleted 8y ago
    [deleted]
dasKrokodil 8y ago
Pilots!
TheOtherHobbes 8y ago
'CEOs get excited by things that make CEOs excited, not that solve real problems for real users...' Still works, mostly. Certainly true of architects, partially true of lawyers and doctors, less true of teachers. The key difference is the amount of f2f time with users/customers/clients. If you add a dumb feature to your site and you have to talk to users about it f2f - not over Twitter or Slack or Medium, but in person - you're going to give yourself a much more realistic idea of what matters to them.
- mr_toad 8y ago
  > CEOs get excited by things that make CEOs excited They seem particularly enarmoured with re-arranging the org-charts.
Jordanpomeroy 8y ago
> `you’re on Medium after all’ Ahhhhh! You got me!

AstralStorm 8y ago

Usually,. the thing that you lose is scalability, availability any reliability. You will need a remote backup and a secondary located service, just in case. Potentially a third development one to not push breaking changes directly into production. Skipping any of this will make your service brittle in addition to not being scalable. And it already hits the rule of three... (of "don't repeat yourself") that makes a lightweight Kubernetes deployment worthwhile. (Perhaps without all the bells and whistles.) Remember, downtime is usually lost money. A day's worth can be thousands of dollars at least, and that's two nines already. (99.99%)

chii 8y ago
Or, you ship 2 months earlier, and the trade off being suffering a day's worth of downtime. It makes no sense to start with a more complicated setup, until there is clear evidence of a need.
- AstralStorm 8y ago
  The main problem is that usually low quality infra bites you when you get exposure. Which is exactly when you don't want it to fail... For a start, Heroku or AWS instance will do and they scale. (As the other commenter said) If you're doing your own infra, you need at least basic fallbacks and safety.
theblackcat1002 8y ago
Well if the first day you shipped your product and it's already worth 1k, yeah it makes sense to do everything the "correct" way. But for most of the side projects and startup products this makes no sense at all.

warp_factor 8y ago

I saw so many places falling for this fallacy. Instead on working on the core products, engineers would have "fun" adding more microservices, redundant Kubernetes clusters, istio controllers etc etc. (all of that with less than 10 users).

otabdeveloper1 8y ago
Nobody ever got promoted or a raise because the core product 'just works'. As an engineer, you need growth in headcount or complexity if you want a raise. It's a people management problem that flows from the CEO and form the HR department, not a technical one.
- expertentipp 8y ago
  One gets fired when everything „just works”. Everything works (untils it stops working) so your job is done, what are you doing even during your office hours? I wish having a „technical manager” would avoid such situations but it’s enough if this „technical manager” hates the whole domain you work in.
collyw 8y ago
When people start adding micro services (or whatever the fad is this month) ask "why?" "Because it scales" "But we don't have a scaling problem, and if we did it could easily be solved with some caching". Unfortunately there is also the bullshit which is hiring in this industry, where solving things the simple way doesn't add anything interesting to your CV.
- VBprogrammer 8y ago
  People who underestimate the challenges adding caching can cause make me a bit nervous. Especially when you start putting caches on top of caches. You can end up with a hunt which cache has the incorrect value problem.
- jordanbeiber 8y ago
  Just my 2c regarding ”microservices” or SOA, whatever you want to call it, is that, sure it can enable you to scale horizontally from a technical perspective, but few have to deal with those problems for real. What it does though when organically implemented, is it allows you to scale dev teams more easilly and it will also let you build functionallity and features more rapidly for an ever evolving business. This is my take-away from two massive undertakings at two separate medium-to-large scale enterprises that I’ve been involved in. One was a complete failure and one an enormous success that both have had business altering implications.

carlisle_ 8y ago

>Your users to don’t care how your code gets onto your server. I don't like arguments like this. This is true for any number of tasks that a layman wouldn't care to understand. You still will do them anyway, regardless of whether or not your user cares. It doesn't take a lot of experience to see the correlation between poor engineering practices and poor user experience. Now, I don't think the author was using this argument to pitch poor engineering, but there are good reasons for setting up resilient and consistent build deploys, as he addresses. I think the only real takeaway is "dont over-engineer your infrastructure" which is just an extension of generally "don't over-engineer." Every time infrastructure gets talked about on HN the conversation gets so sidetracked. The real problem is that people don't think about infrastructure in the context of a holistic view of a project. The hours you spend setting up your project with resilient and consistent CI builds are many times more valuable down the road. I think people need to not be so afraid of spending time on infrastructure. People already err on the side of spending less than more time on infrastructure. I don't think it's wrong at all that the author spent hours building an automatic build process for his project, especially because now that he's set it up he can refer to it in the future. Maybe with his newfound knowledge and experience on the problem means that he can leverage it with little effort in the future. None of this is a bad thing especially if you think about your engineering career on a macro scale instead of week to week.

0db532a0 8y ago
I agree that CI is a brilliant investment. Also it can initially be as simple as a git hook. However, I agree with the author’s point that you do not need a Kubernetes distributed NoSQL megagalaxy when you have a number of customers which you can count on one hand. I’ve seen so much time wasted on this sort of thing. One obvious benefit of the whole single/simple server idea is that you can make changes to the system so quickly and easily, which is very important at the beginning.
- davidjnelson 8y ago
  Ya nosql can be useful at scale, but it's not a great choice for most new projects. It really locks you into your schema and makes querying the data at scale difficult outside of use cases you originally designed for. I've seen aws architects recommending just starting with sql, and adding nosql later as needed. Definitely agree with this approach for most use cases.

reacharavindh 8y ago

Good old KISS policy. Sure, Kubernetes, fully automated CI/CD, autoscaling, lambda functions are all cool and very useful if you have a problem and paying users in that scale. Majority of the applications even today in 2019 would be served well by the reliable Nginx + (Python, C++, Rust, Go or other language of choice), PostgreSQL and backups. If you keep things simple, you can even have a standby mirror of the same infrastructure to take over if needed, and it would still cost you less than spinning up nodes after nodes for the complexity of running your own Kubernetes.

scarface74 8y ago
Or just set up Elastic Beanstalk with AWS or something similar, hand it a zip file of your code and let AWS configure your load balancer, autoscaling, patching your OS and runtime, managing your database and backups, etc.
- reacharavindh 8y ago
  What you suggest may be "simple" in terms of the amount of work the user needs to do, but my comment is about reduced complexity and $$ because you don't need them yet. When you set up such an infrastructure like you proposed, there are still too many wheels in motion(complexity), and even worse they are abstracted away from you further and further. And, I'm sure the AWS bill is not going to be pretty considering they do all those bells and whistles.
  - scarface74 8y ago
    You literally click a few buttons and it sets up all of the infrastructure. Then you give it a zip file. The EC2 instance that it creates is a regular VM. Elastic Beanstalk itself is free.
    - reacharavindh 8y ago
      It is true that Elastic Beanstalk is free, and is very easy for a developer to use to begin with. However, If I were to advice a new startup on their infrastructure, I would not recommend it for the following reasons. It is my personal judgement. Others may prefer to go all in on Amazon. 1. It is clearly a vendor lock in. You build your business upon all AWS services, using an AWS orchestrator - good luck ever getting out. 2. Abstraction. It is a controversial opinion. I'd rather have less magic when possible and know what goes on underneath, and possibly have control over. Now, a startup may choose to offload the duties of operating a database(RDS), storing files(S3), performing backups(S3/Glacier) in the interest of time and trade-off control, optimisation and $$. However, considering the generalisation of simpler startups, IMHO operating a small scale Postgres on a standard EC2 instance is not really much of a hassle... 3. Cost - however small it may be in terms of actual dollars, AWS managed services(S3, RDS, SES, SQS etc..) come with added cost over simply running their open source equivalent yourselves. Again, Keeping it simple - not in terms of what the user needs to do to deploy their application, but in terms of what components do you need to get your application running in production.

lbill 8y ago

I agree with the author : start simple, then use more complex tools when and if you need to. I believe it is true about infrastructure, and about features and code as well. When my team needs to release a functionality for the "product"[1] that we maintain, we have a very simple strategy : we go step by step. The users tell us what they need, and we start to deliver as soon as we can. Our first release only solves a portion of the user's problem, but the second release solves more, and so does the next, and so on. I devised this strategy with the users to make sure they are fine with it: they expect a perfect product... eventually. But in the mean time, they having only some portions of their needs met, and we avoid the feeling of being overwhelmed by a great apparent complexity. [1] The context is a bit complicated... and irrelevant. We don't exactly maintain a product, but it's close enough for the argument here.

lbill 8y ago
Oops! Sorry for the huge typo in my comment. "they having only some portions of their needs met" should be read "they are fine with having only some portions of their needs met". (It is a bit more than a typo)

mdekkers 8y ago

You don't need it if you are a one-man band doing a small side project. > I spent hours setting up a simple and free continuous deployment pipeline. Commit to git, Semaphore CI builds the Docker image, uploads the image to a container registry, then pulls & restarts the container on my server. It’s really cool. I felt like a wizard once I got it working … and it was free! I actually sat down here to write all about how to do it yourself, until I realised how long I spent on it and how much value it delivered to my users If you have a team of some size, the time you gain over the lifecycle of the project can be very significant when you have a smoothly running automated dev/test/UA/live pipeline. > Not build the fanciest deployment pipelines, or multi-zone, multi-region, multi-cloud Nuclear Winter proof high availability setup. If your downtime can be measured in appreciable lost revenue or reputational cost, then yes, you need all of that, as well as a solid SRE or 5 to keep it all running.

sleepychu 8y ago
> Obviously if you’re FAANG-level or some established site where that 0.1% downtime translates into vast quantities of cash disappearing from your books, this stuff is all great. You have the funds to do things “right”.
- mdekkers 8y ago
  It doesn't have to be "vast quantities of cash", it only has to balance out the cost of the additional work. And reputational cost is notoriously difficult to turn into a cash value. I really have an issue with how the author presents this as a "Almost nobody needs this stuff", where the truth is that it is a lot more nuanced. I typically advise my clients what uptime will cost them extra, per "9" (as in the 5-nines), and let them make their own decisions.
- scarface74 8y ago
  You don’t have to be a FAANG. You can be a B2B startup with one or two major clients who if they see you aren’t highly available will jump ship taking a lot of your revenue with them. I’ve been in enough meetings working for small startups where our clients grilled us about our resiliency, availability, redundancy etc. and where they forced us to put our code in escrow as part of our contract.

sbhn 8y ago

It depends on who your customer is. If your customer is the investor, then there is no limit to the amount of hardware, infrastructure, and experts sitting behind a desk and multiple monitors in every concevable separation of concern you can think of in every major city across the globe. Your product is amazing, and its going to change the world and how people percieve things. You need everything you can throw at it. Investors want to be the best. And thats if you customer is the investor.

alfonsodev 8y ago

When you are building a startup, there is always the question, "will this scale?" or "in the event of an investment or going viral are we ready to scale automatically ?" I think that there is a lot of worries around this, it conditions technology choices, sometimes pushing early optimisations, and it is hard because you don't have the data on how many concurrent users your product will have. The best you can do is to extrapolate from you current data or other products data. So many people end up following a recipe, it could be Firebase recipe, AWS Fargate, Serverless... you name it. Recipes that promise limitless autoscale. I think indeed is distracting and premature, but again is hard to avoid it, if the premise is that there will be a massive growth overnight, which is imposible to predict that will potentially kill us, and we have to be ready.

jcadam 8y ago
I've wasted a lot of time during my attempts as a solo developer to build a product with a scalable backend architecture for the hordes of users that never materialize. Next idea I have, I swear I'm going to throw all my effort into the frontend and go with as minimal a backend as I can get away with (heck, depending on the application, human-in-the-loop might be fine early on) until I get some validation. It'll feel "wrong", but whatever.

m90 8y ago

While I think you can definitely get too fancy with all of that sweet infra, I feel this is missing an important point: Once you have a working pipeline in place, you can focus on the features, that's the whole point of the exercise: enabling developers to ship features for users. If I have to SSH into some machine and restart a set of services manually in the correct order, it will keep me from releasing features. I am afraid / lazy / whatever. Spending time on this is not something that will never ever reach the user in any way. The user does not care if something is using Heroku, K8s, or something else, yes, but the user definitely appreciates a product that can reliably iterate on things without major downtime or similar problems. It's kind of like saying, "well unit tests are nice and stuff, but does the user ever notice?". I would strongly argue the user will notice unit tests in the long run as they ensure the software keeps being maintainable.

afarrell 8y ago
When you walk into a restaurant, do you notice if they have an efficiently laid-out kitchen with mise-en-place, good relationships with their suppliers, or a refrigerator at the proper temperature? No. But you do notice when you food is slow to arrive, the food doesn't taste fresh, or the chicken gives you food poisoning.
ctnaxx 8y ago
But many applications were better around 2002-2004 than now. The only thing that's better now for users who don't care about privacy at all is the opaque sign-in-with-X feature. Email web interfaces got worse, login forms got worse, documentation got worse, search got worse (peaked around 2004).
- m90 8y ago
  Which ones? Which feature?
- ox_cable 8y ago
  I've never seen a more biased, unsubstantiated, "back in my day" comment on HN. Well done. Give specifics if you want to be taken seriously instead of just saying "well, back in my day everything was better".

jdietrich 8y ago

There's an obvious and overlooked middle ground between "massively complicated infrastructure" and "a cheap VPS" - charge more and use PaaS. Heroku and Engine Yard are expensive, but they deliver a huge amount of value. You've got better things to do than patch Nginx or deal with a crashed server at 4am, just pay someone a sensible amount of money to run your infrastructure for you.

pseudoramble 8y ago
I agree. There are also options to make some of the things from the article relatively easy. Off the top of my head, a CI/CD pipeline can be set up using Gitlab , Azure DevOps, Bitbucket Pipelines, and I'm sure tons of others. They do some of the heavy lifting and you still get the benefit of an organized build that's easy to deploy to one of the services listed above. And yeah, this may be layered on later when it seems like the idea has legs and you'd benefit from this. It most likely doesn't need to be the first step.
bufferoverflow 8y ago
> an obvious and overlooked middle ground between "massively complicated infrastructure" and "a cheap VPS" It's called a dedicated server. These days you can get a monster of a machine. Even on AWS you can get a machine with 2TB of RAM and 128 cores.
davidjnelson 8y ago
Heroku is fine, but their lack of an autoscaling database for spiky load and minimum $250/mo for autoscaling web servers are really very far behind what using aws directly offers these days. Heroku seems to have stopped innovating.

cdoxsey 8y ago

Please at least use a managed database. Treat that linode box like it might disappear at any moment. It is so easy to shoot yourself in the foot with a database and lose everything. Also FWIW configuring Linux machines manually over SSH and building and deploying your code too it, isn't always easier. Make sure you write everything you did down, because in 6 months you will completely forget how you built this thing, where it was deployed, etc...

mark_l_watson 8y ago
I have not done this for a number of years, but in the past for my own and customer projects I would just run Postgres on the same VPS as the web app but use a cronjob to write incremental backups every few hours to S3. As other people here have said, agreeing with you, paying for multiple availability zone RDS or paying Heroku is also a good plan.
Cthulhu_ 8y ago
And whatever you choose, TEST YOUR BACKUPS. Spin up a new instance and practice recovery. This needs to be done at least quarterly, both to test your backups and train yourself and/or your people in this process.
mcshaggins 8y ago
Please write everything down and be kind to 'Future You'. I am very anti-spending money on software that isn't directly related to a project but a $10 confluence wiki lifetime license is one of the best purchases I have ever made.

mark_l_watson 8y ago

Right on! I am enjoying the book “A Company of One” right now and the author makes the same point of only doing what it takes to provide great service to a few customers. Another point is to spend more effort servicing existing customers than spending time trying to get new customers. For deployment: if you can tolerate rare service downtimes then it is difficult to beat Linode or Digital Ocean. I am probably in a minority, but I also really like GCP’s AppEngine: turn on billing, set the minimum instance count to one and the maximum to two or three, and you have a resilient system. That said, I run my primary web site on a single tiny GCP VPS instance with no fault tolerance. For heavier compute requirements, I also really like a physical server from Hetzner- for about $100/month you can get a GPU, i7, and lots of RAM and bandwidth.

robodale 8y ago
Thanks for the "A Company of One" book recommendation. I just ordered it.

jpttsn 8y ago

I run a small team, but we do a large number of projects. Each project is developed fast and then maintained at low intensity for a long time. Have found great use for a nifty complex DevOps pipeline. Before, when I built the same app for two years, the nifty pipeline was overwrought. I think the context switching inherent in what I do now makes the nifty pipeline useful.

q-base 8y ago

I see this "Push for new and shiny solutions"[1] across everything from startups to large corporations. And I am genuinely surprised at how often I see people falling for this good old trap of over-optimization/engineering. I genuinely cannot fathom why people want the complexity. It is by no means constrained to infrastructure, it is architecture, technology, management the whole lot. [1]http://jesperreiche.com/the-push-for-new-and-shiny-solutions-to-old-known-problems/ http://jesperreiche.com/the-push-for-new-and-shiny-solutions...

jb3689 8y ago

The problem with Kubernetes (and microservices in general) is getting a team productive with it, and that requires lots of time, dedication, and tooling. It's not the right solution for most teams, but it could be if we streamline it more

ratling 8y ago

This is a pretty dumb article. Like, no shit don't build more infrastructure than you need. If you're a one or two man team you don't need anything more complicated than 'push container here, pull down here' or 'push code here' et. al. Once you get above that having a repeatable process (repeatable by someone besides the person who built it) becomes valuable. The 'every second spent on infrastructure isn't spent on features' line makes me think I'm going to have to blackmail into working on security updates or paying down technical debt. And the picture you picked is cheeky. I will agree with this statement, "You need the most simple, least time consuming infrastructure that gets you to that point." But that WILL become more than just 'throw it on a linode instance' after you get some customers and more than 2 people working on whatever uber for dogs project you have going on.

draw_down 8y ago

Your users don’t care about the text editor you use to write code, but I bet you use something more complex than nano. Tools are to be valued differently.

SnacksOnAPlane 8y ago

One of my web sites is run by a "bundle exec rails s" in a tmux window on an EC2 instance. New deploy? Run "git pull" (and maybe "rake db:migrate") in tmux pane and restart the "rails s" command. Easy peasy. I'd make it more complicated and resilient but why?

davidjnelson 8y ago

> Kubernetes clusters? Load balancers across multiple cloud regions? Zero-click, Blue-Green deployment based on real-time metrics from a continuous deployment pipeline? Kubernetes, multi region, canary deployment, immutable infrastructure, even infrastructure as code is overengineering for a brand new indie project, for sure. BUT, there is a certain minimum amount of quality, availability and scalability any product needs, especially a new product that might show up on hacker news or product hunt and get 100-300 concurrent requests a second for multiple hours at unpredictable times. To not use fargate because you have to dockerize your app and build it during a codepipeline stage is silly. That takes what, a week, max, the first time you ever do it? Once you've done it, you can do it faster for future products. To not use an autoscaling and high availability database when it costs the same and is so simple to set up makes no sense either. To not use ci/cd to test your db migrations and new code on staging before pushing to prod is just dumb and asking for pain and suffering. Why put yourself through that when products have been built to make these things _so_ easy and inexpensive now?