15 ms·

Some day I would like a powwow with all you hackers about whether 99% of apps need more than a $5 droplet from Digital Ocean, set up the old-fashioned way, LAMP

Some day I would like a powwow with all you hackers about whether 99% of apps need more than a $5 droplet from Digital Ocean, set up the old-fashioned way, LAMP --- though feel free to switch out the letters: BSD instead of Linux, Nginx instead of Apache, PostgreSQL instead of MySQL, Ruby or Python instead of PHP. I manage dozens of apps for thousands of users. The apps are all on one server, its load average around 0.1. I know, it isn't web-scale. Okay, how about Hacker News? It runs on one server. Moore's Law rendered most of our impressive workloads to a golf ball in a football field, years ago. I understand these companies needing many, many servers: Google, Facebook, Uber, and medium companies like Basecamp. But to the rest I want to ask, what's the load average on the Kubernetes cluster for your Web 2.0 app? If it's high, is it because you are getting 100,000 requests per second, or is it the frameworks you cargo-culted in? What would the load average be if you just wrote a LAMP app? EDIT: Okay, a floating IP and two servers.

deleted 8y ago
[deleted]
tshannon 8y ago
Does hacker news really run on one server? What if server goes down? I've always though high availability was the more important reason for multiple servers, rather than performance. Even if you have only two paying customers, they are probably paying for the right to hit your website / service 24/7.
- lostapathy 8y ago
  What if hacker news goes down, more work gets done for the day? It’s really common to over estimate the cost of being down, while underestimating the costs of resiliency. And in the end if you can’t fully afford the resiliency, you end up with a system that’s more complicated and thus less stable than it could have been had you just accepted a tiny bit of risk.
  - _drFaust 8y ago
    extremely valid point. The initial setup isn't the challenge, it's the endless tweaks to make it fault tolerant. For anyone looking into k8s, take time to research best practices for monitoring and readiness probes.
- wpietri 8y ago
  > What if server goes down? Productivity of Y Combinator startups goes up. So YC benefits either way.
- matuszeg 8y ago
  Has hacker news ever gone down? I cant remember it ever going dkwnt but I assume it has to have at some point.
  - iBelieve 8y ago
    I've seen it happen at least once in the few years that I've been on Hacker News, see https://news.ycombinator.com/item?id=17228704 https://news.ycombinator.com/item?id=17228704
  - freehunter 8y ago
    Here's the official status twitter: https://twitter.com/hnstatus https://twitter.com/hnstatus
  - collyw 8y ago
    I got a server can't respond to your request type message this morning. OK, the server wasn't down, but it was loaded enough that it couldn't serve me. A page refresh and it was working.
- jimnotgym 8y ago
  An interesting case study of an app at scale is Stack Overflow. Rather more than one server, but rather less than you might expect. Edit: fixed url https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-...
  - branko_d 8y ago
    Key point: "The primary reason the utilization is so low is efficient code."
- paulddraper 8y ago
  One application server, yes. HN is fronted by Cloudflare for CDN + DDoS protection, which of course is a lot more than one server. That's why if you get a particularly long thread (1k+ comments), admins will beg people to log out so that the responses can be served from the CDN cache. Example: American 2016 presidential election https://news.ycombinator.com/item?id=12909752 https://news.ycombinator.com/item?id=12909752 (1,700 comments)
  - tambre 8y ago
    HN hasn't been fronted by Cloudflare since July.
    - paulddraper 8y ago
      Hm, interesting. I noticed HN started responding to HEAD requests with 405s recently; perhaps that is the cause.
    - ksec 8y ago
      Why did they stop using Cloudflare?
      - tambre 8y ago
        It was part of their networking rework. Presumably to improve stability of the website. I've also been told that it was a big step towards enabling IPv6 for HN. There are other parts of Y Combinator that still use Cloudflare though.
  - detaro 8y ago
    Not Cloudflare anymore. Traffic goes to somewhere in San Diego from Europe, which it wouldn't if it was Cloudflare (different IP range too)
- vesak 8y ago
  Well, there's another important question: What percentage of all services need high availability? Stetson-Harrison method shows that it's less than 5%.
  - kodablah 8y ago
    Need is determined by the customer. You might be able to explain to them that they don't need it on their way to a competitor. May not be fair or rational, but redundancy and its peace-of-mind has real benefits in most systems compared to cost.
- deleted 8y ago
  [deleted]
devman0 8y ago
I think for smaller use cases its more about high availability than load balancing.
gambler 8y ago
I realize there is a need for multi-server applications with automated deployment and scaling. However, the accidental complexity of serverless setups and container orchestration tools is just off the charts. When reading these articles I get roughly the same feeling I got when reading J2EE articles back when J2EE was "the future" and "the only way to build scalable infrastructure".
- collyw 8y ago
  Pepole say that serverless keeps thing simpler. In reality its just moving the complexity to different areas (devops as opposed to application complexity). We have been writing applications for longer, so it should be easier to keep the complexity lower if you keep the logic there. Are there similar well known patterns / best practices / ways of structuring devops that have been developed for applications over the years?
citilife 8y ago
For reference, this runs on a $5 AWS instance: https://hnprofile.com/ https://hnprofile.com/ The database is $600 per month, but that data runs five different websites (and it's a few hundred Gb of data). EDIT: for those mentioning the 502 gateway error, it does auto scales - Now it's costing more per month, at least temporarily.
- supakeen 8y ago
  Not a good example as it is currently 502'ing: https://i.imgur.com/sU8Zn5v.png https://i.imgur.com/sU8Zn5v.png
- wnevets 8y ago
  >502 Bad Gateway maybe a $5 instance isn't enough
- kayoone 8y ago
  $5 AWS instance ? aren't they all substantially more than that ?
  - maxerickson 8y ago
    Nope. https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/ https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...
SEJeff 8y ago
FreeBSD, Apache, Python - the FAP stack Ok. I'll see myself out.
- movedx 8y ago
  I laughed ;-)
  - collyw 8y ago
    Me too, surprised this hasn't been down-voted to hell on here.
mdekkers 8y ago
Not everyone uses K8s for webapps. You would be surprised at the level of enterprise penetration of K8s. Those enterprises do boring stuff like datawarehousing etc.
- jimmy1 8y ago
  One of the more interesting use cases I have read in recent memory: Chick-fil-A used it to set up a bare metal edge compute network between all it's restaurants The how: https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... The why: https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2 https://medium.com/@cfatechblog/edge-computing-at-chick-fil-...
pythonaut_16 8y ago
I personally use Docker combined with a $5 droplet on Digital Ocean. This makes it easy to spin up multiple applications and sites without worrying about conflicting dependencies, and docker-compose gives me most of the benefits of orchestration tools (e.g. Kubernetes) that actually matter for my small scale usage. Also Traefik makes a nice load balancer for this uage
- bdcravens 8y ago
  > docker-compose gives me most of the benefits of orchestration I feel this is a very unappreciated feature of docker-compose. I've gotten pretty far with setting restart: always, baking a machine image, using cloud scaling and load balancers.
- ngrilly 8y ago
  Interesting! When you rollout a new version, how do you coordinate docker-compose and Traefik to avoid any user-facing downtime (no 502)?
  - pythonaut_16 8y ago
    The simplest answer is I am the only user currently so it doesn't matter. Even with users, a short down time to bring down the old images and bring up the new images should be manageable. If I ever got to a point where that wasn't sufficient then I would consider that project successful enough to merit investment into zero downtime strategies and probably its own VPS as well. From a more technical side, I haven't seen an issue with Traefik when I take a container down and back up again. The only delay I would anticipate is when you bring a new service online for the first time and Traefik detects it and configures + fetches LetsEncrypt certs for it.
    - ngrilly 8y ago
      Thanks.
wpietri 8y ago
As somebody who has his own colocated server (and has since Bubble 1.0), I definitely agree that the old-fashioned way still works just fine. On the other hand, I've been building a home Kubernetes cluster to check out the new hotness. And although I don't think Kubernetes provides huge benefits to small-scale operators, I would still probably recommend that newbs look at some container orchestration approach instead of investing in learning old-school techniques. The problem for me with the old big-server-many-apps approach is the way it becomes hard to manage. 5 years on, I know that I did a bunch of things for a bunch of reasons, but I don't really remember what or why. It mixes intention with execution in a way that gets muddled over time. Moving to a new server or OS is more archaeology than engineering. The rise of virtual servers and tools like Chef and Puppet provided some ways to manage that complexity. But "virtual server" is like "horseless carriage". The term itself indicates that some transition is happening, but that we don't really understand it yet. I believe containers are at least the next step in that direction. Done well, I think containers are a much cleaner way of separating intent from implementation than older approaches. Something like Kubernetes strongly encourages patterns that make scaling easier, sure. But even if the scaling never happens, it makes people better prepared for operational issues that certainly will happen. Migrations, upgrades, hardware failures, transfers of control.
- heipei 8y ago
  This. Kubernetes (or whatever other container scheduler) might feel like overkill, but if all they do is force you to adopt a container-centric / 12-factor way of building your applications it was worth trying them. And once you've adopted that workflow it's a no-brainer to go from a single node to a cluster which will dynamically allocate the workloads it runs. Running a small container cluster at work has even changed how I setup single-host projects in my spare time: I will build everything into a container, bind-mount whatever it might need, create a simple systemd unit that just runs / rms the docker container on start and stop. Bliss.
- 3pt14159 8y ago
  I've found it just pushes the complexity elsewhere or opens up (or silences) performance or security problems you wouldn't have had if you'd stuck to the old fashion way of doing things. Keep checklists and script what you can. I find it helpful to follow edge whenever I can so if I hit a snag the developers that made the change still have that change fresh in their mind. It really doesn't take that much time to keep things updated if you don't go hog-wild on different libraries and if your stuff is reasonably well tested. As an aside, I've been thinking that there should be a stack that is designed and built for the sole purpose of staying stable over decades. Something with a bunch of stripped down technology. As robust and stable as possible. Only allow security updates. Only allow certain character sets. Something built on a language that is just stupid simple and secure. A Swift or Rust subset maybe? Lua? Lisp without the macro insanity? If the future needs some sort of tech that we didn't anticipate (say, something to handle quantum computers breaking cryptography) then the stack should be setup in such a way to decouple the varying layers with minimal work. I liken it to building codes. We should have pre-setup combinations of technologies that are stable, simple, and combinable. Sure, go outside them for skyscrapers, but for the day-to-day buildings things are getting too complicated.
dcosson 8y ago
I really think that running a LAMP server for the average beginning developer these days would be just as complicated, maybe more complicated, than running a single deployment on Google Kubernetes Engine. You have to know about package managers and init systems and apache/nginx config files and keep track of security updates for your stack and rotate the logs so the hard drive doesn't fill up. If you already know how do this stuff in your sleep because you've done it for years, then yeah, don't fix what isn't broken. But if you're starting with no background, there's nothing inherently wrong with using a more advanced tool if that tool has good resources to get you started easily. Just because there's more complexity in the entirety of the stack when running an orchestration system doesn't necessarily mean more complexity for the end user. Side note - couldn't you make a similar argument about any kind of further abstraction? "Question for all you hackers out there - do you really need HTTP requests with their complicated headers and status codes and keepalive timeouts? I run several apps just sending plain text over TCP sockets and it works fine."
- gambler 8y ago
  >I really think that running a LAMP server for the average beginning developer these days would be just as complicated, maybe more complicated, than running a single deployment on Google Kubernetes Engine. Back when I first started doing web dev I went from knowing nothing about server setups or Unix (i.e. running off managed hosting) to a reasonably secure FreeBSD server with a working content management system in 3 days. This included installing the OS. The same FAMP setup (with modifications and updates, obviously) continued to work perfectly fine for the next decade. Re-edit: In the original version of the post I drew a parallel with several teams at my previous job "figuring out" AWS Lambda for several month, and stumbling over gotchas, multiple ways of doing everything and a myriad of conflicting tools. Since there is a reply to that statement, I guess I will add this note.
  - pacala 8y ago
    Containers are a mechanism to run your old machine[s], but with a reproducible setup script. A machine packed in a container happens to also run on your dev/CI environments. There isn't much logical difference between a physical machine, a VM and a container [0]. Serverless offers a large surface of APIs, some of them proprietary, tangled in an ever evolving dependency hell. Historical note: Google Cloud started serverless with AppEngine, then focused on GCE [and later GKE] _because_ serverless was hard and AWS was eating their launch with VMs. [0] For example, we can argue about security isolation issues in containers vs VMs. Eventually this will become a moot point as technology advances far enough that we can run each container inside a hardware-backed VM.
draw_down 8y ago
Well, I don't know where you work, I guess at the company that makes little bespoke apps that all run on a single point of failure, but I actually do work somewhere that runs at the scale of the sort of companies you mention. So it's good to know about techniques and tools that are useful in that context. Hacker News is great and all, but it ain't a business. And if it goes down, well, surf another website for a while. Many of us work in situations that are significantly more... constrained than that. We aren't able to say, "Hey, we're down at the moment so just use one of our competitors for a while, is that chill?" So it's maybe not the best example of what we apparently should all be doing. I don't know this mythical person is, who's always talking about "webscale" and overengineering the shit out of everything. Kind of a tired strawman imo
gizmodo59 8y ago
I think of these in this way: 99% of the apps are developed by the developers who are not in the top 1%. The cheap access to computing power has led to a growth of developers beyond the highly skilled ones who can milk out everything available in a less powerful computer. I'd like to believe we are at an Electron development phase where we just want to ship as much as possible easily without worrying about hiring great talent (And yeah I hate that its inefficient in terms of memory usage). This has led to the explosion of so many frameworks that does a lot of things easily which requires such complex devops pipeline.
StavrosK 8y ago
They do not, I run tens of low-traffic projects very successfully on a $10/mo Hetzner server on Dokku. Dokku is amazing and so is Hetzner, I don't know why people always go for the high-scalability, expensive options just to end up with 0 utilization.
- devhead 8y ago
  if you have zero utilization, then you should scale down or lower your instance type until you are optimized for performance and cost. Which is easier to do when you are in an environment such as gce or aws.
  - StavrosK 8y ago
    There's a minimum when you have deployed each one of your hundred microservices to a server, though.
- Aeolun 8y ago
  Because my company is risk-averse, and perfectly happy to drop $1000 a month for a ridiculously overprovisioned database instance just to ensure an issue with the database will never cause their contracts to be lost.
grey-area 8y ago
I think you're right - at least 90% of servers on the web would be fine with a couple of instances at most backed by a decent db. It can get more compex depending on your resiliance requirements but it really doesn't have to be. I guess I run a CPG stack - Coreos, Postgresql, Go. Don't bother with containers as Go produces one binary which can be run under systemd. It is far simpler than kubernetes and the only real reason for other servers is redundancy. The only bit of complexity is I usually run the db servers as a separate instance or use a managed service. You can go a long way with very boring tech. I've run a little HN clone written in go on one $5 digital ocean droplet for years - it handles moderate traffic spikes with little effort.
meritt 8y ago
As someone who runs a very successful data business on a simple stack (php, cron, redis, mariadb), I definitely agree. We've avoided the latest trends/tools and just keep humming along while outperforming and outdelivering our competitors. We're also revenue-funded so no outside VC pushing us to be flashy, but I will definitely admit it makes hiring difficult. Candidates see our stack as boring and bland, which we make up for that in comp, but for a lot of people that's not enough. If you want to run a reliable, simple, and profitable business, keep your tech stack equally simple. If you want to appeal to VCs and have an easy time recruiting, you need to be cargo cult, even if it's not technically necessary.
- movedx 8y ago
  Good on you mate. I love this approach. I'll be doing the same thing my self with a few products I'm developing with my brother. "Keep it simple, keep it stable" is what I like to say.
- jdhawk 8y ago
  I work in the same environment, and hiring is the only downside I see. Resume Driven Development and "Dev Sexy" has made it difficult to find developers who are willing to come on board, despite the sanity provided by simplicity & comp.
  - Aeolun 8y ago
    It will now, but how many people are going to be interested in my LAMP experience 5 or 10 years down the line? While everyone else has been working with the cloud/kubernetes/aws/gcp/serverless technologies.
- nineteen999 8y ago
  Hear hear. I lead a team that builds and manages critical emergency services infrastructure. Our stack is pretty boring, but then it has to be running 99.999% of the time. Rather than wasting time chasing the latest flavour-of-the-month tool or framework we invest our time in plugging any kind of gap that could ever bring our service down. We don't need people who are only looking to pad their resume with the hottest buzzwords, we look for people who want to make critical services run all that time, that rarely fail and when they do, they handle failure gracefully. The number of devops/agile/SaaS style shops I have seen where the product falls over for several hours at a time is astounding, and it can often be attributed to rushed/inexperienced engineering or unnecessary complexity. Lucky for them it's usually just the shareholders bottom line that is affected. If the services my team provides don't work, ambulances and fire engines are either not arriving fast enough or at all.
ownagefool 8y ago
You probably don't need kubernetes. Lets be fair, it offers:- > Orchestration of block storage > Resource Management > Rolling Deploys > Cloud provider agnostic APIs* If you don't need any of these things, and your stack fits on a single server or two, and you aren't already familiar with it, I'm not sure why you bother other than an interest. That said, there's a world of companies that aren't FAANG, ub3r and Basecamp, and many of those paying reasonble sums of money have more complicated and resource intensive requirements that don't fit on a single server. Government Departments, Retail Companies and Banks all likely have a number of different software development projects where giving a number of developers API access to a platform that offers the above advatanges is, in my opinion, a good thing. Once you get to FAANG level, who knows whether kube itself will actually help or hinder at that level. * Personally I'd rather use the kube APIs than talking to any of the cloud providers directly. I imagine that's somewhat personal preference and somewhat because I've been able to easily run it in my basement. *2 Namespaces also makes making more environments for CI/CD easier, so as soon as you have a team of developers and you want to do that sort of thing, it also makes sense. Not so much for a loan developer and his server.
- sleepybrett 8y ago
  There is also overhead in the form of instances to run and maintain the backend datastore and control plane components. You should already be at a certain scale before considering kubernetes.
  - ownagefool 8y ago
    This is free on GKE and I believe AKS, but of course if you're doing this yourself for some reason, you need to compare it with the alternatives.
aeriklawson 8y ago
> I manage dozens of apps for thousands of users. The apps are all on one server, its load average around 0.1. If you're at this scale you can do whatever you want. Most of the stuff I've made has been with simple building blocks like you've described, maybe thrown in with some caches and a load balancer. Although I've worked with other teams who really did have the high scale request flows that require you think about using a different architecture. Even so, K8s is not the end game and you can make something work even just extending the LAMP stack.
staticassertion 8y ago
I think that kub and, generally, cloud providers have allowed for more ambitious projects to be generally accessible. My side project is intended to handle > 1 billion events per day, with fairly low latency. That's well over 10k events per second. I doubt I could do this easily on a single box, and I wouldn't really want to try. Why constrain myself that way? Is it worth just doing this the standard LAMP way? More and more problems are available to be solved using commodity systems, so we have more and more people solving those problems with these new systems.
- toredash 8y ago
  Depends on the box. Per core performance is probably your metric there if the app can utilize multiple cores.
kodablah 8y ago
> Some day I would like a powwow with all you hackers about whether 99% of apps need more than [...] Close. But I also need it HA w/ automatic failover, auto-SSL certs (meaning I might need to give my DNS provider creds depending upon LE approach), notifications on outages, easy viewing of logs past and present, easy metric viewing, automatic backups, and updates that are easy for me to sign off on to then run. I'll do the plugging on the app side (that is exporting metrics, logs, etc). And no vendor-specific solutions (even if they are repackaged common components like RDS is for Postgres), I should be able to run on a couple VMs on my laptop if I want and it be the exact same. And I may want to add an MQ/stream (e.g. Kafka), in-mem DB (e.g. Redis), etc later and still want log aggregation, metrics, backups, etc. Really, that's not asking too much but it's definitely more than LAMP. We need a pithy name for this startup-in-a-box (again, that's NOT a PaaS, but a self-hosted management on an existing set of servers). Nobody wants to fumble w/ Ansible/Puppet/Salt/Chef/whatever all over the place or hire an ops guy, and people don't want to use vendor-specific solutions. I agree with the "you only need this"...but we need just a bit more to handle outages and auditing.
movedx 8y ago
Spot on, friend. So recently I started writing a simple web application for my family. They send emails to each other with gift wish lists in them and we all have to juggle those emails around. I figured some products would exist already to solve this problem, but I wanted to make my own. When it came time to make it I thought: "This has to be a REST API with a JS front end" and then further down the line, "Man I should use Flutter and only make it a mobile app!" I had other thoughts about making it Serverless and doing $thisCoolThing and using $thatNewTech. In the end nothing got done at all. Fast forward to today and it's a monolith Go application that renders Bootstrap 4 templates server-side, serves some static CSS directly, sits on a single server (DigitalOcean) and uses a single PostgreSQL instance (on the same server). The Bootstrap 4 CSS and JS come from their CDN. I made the technology simpler and the job got done. It's an MVP with basic database backups in place, using Docker to deploy the app. It just works. Lessons for me from this: * Server-side template rendering is perfectly fine and actually easier, frankly * JS can still be used client-side to improve the experience without replacing the above or making the entire rendering process client-side * Although Go compiles to a single static binary I still need other assets, so it went into a Docker container for the added security benefits not to mention portability * Serverless is nice, but unless it's replaced the above day-to-day, there's always a steep learning curve around something you haven't done with it yet, but need * Picking the latest and greatest tech tends to stagnate progress or halt it entirely, in most cases * An a software MVP needs an MVP infrastructure to go with it Just my thoughts.
- Enginerd3 8y ago
  "I figured some products would exist already to solve this problem, but I wanted to make my own." But why? You could've used so many different products. You could've even used Google Docs.
  - movedx 8y ago
    There's a bit more to the story than simply solving a problem. I have other plans for the software and we have ideas about how we want to move it forward. One of the key features is being able to "tag" an item on someone's wish list as "bought" or "buying". This allows others viewing the list to know that item has been taken. But there's also a requirement that the original author of the list/item cannot see that it has been bought otherwise there's no element of surprise for them come time to open the gift(s) :-) Spreadsheets don't enable that privacy/secrecy.
treis 8y ago
>I manage dozens of apps for thousands of users. The apps are all on one server, How are these backed up?
- kodablah 8y ago
  And how do they fail over when the server dies (at least the non-user-app part like their DBs)?
manigandham 8y ago
It's not just about scaling. That seems to be the only thing people talk about because it sounds sexy but the reality is about operations. Kubernetes makes deployments, rolling upgrades, monitoring, load balancing, logging, restarts, and other ops very easy. It can be as simple as a 1-line command to run a container or several YAML files to run complex applications and even databases. Once you become familiar with the options, you tend to think of running all software that way and it becomes really easy to deploy and test anything. So yes, for personal projects a single server with SSH/Docker is fine, but any business can save time and IT overhead with Kubernetes. Considering how easy the clouds have made it to spin up a cluster, it's a great trade-off for most companies.
- ecoqba11 8y ago
  Exactly!
- wetpaste 8y ago
  Exactly. It solves some of the most important problems that come up of working with microservice based architectures, and establishes mature patterns around the ability for multiple developer teams to update and scale each piece of a distributed application. Also, what do you do when your digital ocean droplets needs hundreds or thousands of customers on it? Maybe each customer needs it's own database (well in my case they do), configuration, storage requirements, and multi-node requirements. How do you keep track of all that, how do you automatate it and QUICKLY recover in failure scenarios. You need to be able to deal with failure on a node, or if there's a bad actor you need to be able to move them off easily without downtime or affecting other customers, automatically, seamlessly. You need to be able see an overview of your resources across all nodes and where apps are placed and have something decide if the hardware your new container is being added to can handle another JVM or whatever. For cost effectiveness, you want to be able to overcommit resources, so you want containers. You want those to translate to other platforms, AWS, google, azure, on-prem. You have a single declarative language that works anywhere you can deploy a k8s cluster. You need to deal with growth, good patterns for rolling back and updating versions of parts of the stack. You want all of your deployments to be declarative and to be able to tightly control the the options for each one and get back to where you were. I agree that it doesn't make sense for everything, and it requires fundamental understanding of linux and software before it even makes sense to try shoehorning onto, but it solves real world problems for many people, it's not just a hype thing. I would say docker itself was more of a hype thing than k8s, the maturity and features of k8s and other orchestration systems that came out of the docker model is there for a reason, because it solves all of the real-world problems people couldn't solve with vanilla docker without tons of custom scripting and hacky workarounds. Docker solved the big problem by providing the isolating environments for each app and splitting things out into microservices that way, without having to commit a full statically resourced VM or bare metal per service. K8s solves all of the other problems that came out of that (pods, stateful sets, init containers, jobs, cronjobs, service definitions, deployments, volume claims.)
derefr 8y ago
> If it's high, is it because you are getting 100,000 requests per second, or is it the frameworks you cargo-culted in? Mine's high because our business model involves blockchain stuff, and 1. blockchain nodes are CPU+memory+disk hogs; 2. ETL pipelines that feed historical data in from blockchains produce billions of events in their catch-up phase. (And we're constantly re-running the catch-up phase as we change parameters.) Sadly, we need several fast servers even without any traffic :/
tptacek 8y ago
The reliability and performance story for Hacker News is not great, and that's despite the fact that its design has lots of simplifying assumptions. I wouldn't call HN a success story for the "just drop it on a server" approach. Of course, HN is a kind of art project, and its scaling and performance goals are not typical of most applications.
z3t4 8y ago
On a service that has been up for almost 20 years, same code base, thousands of daily users, the first server was constantly on 100% CPU. The second server, averaged around 10% CPU with lots of spikes. The third server, now average below 1% CPU usage. Next time I need to upgrade I will probably get a "NUC", or a smarphone, or something even smaller. But it's not only CPU's that has gotten better. The first server also maxed out the bandwidth! And now, although with less users, the bandwidth usage is less then 1% Started out on 0.5Mbit DSL, and it's now on a Gbit fiber.
johngalt 8y ago
Initially I was skeptical as well. One server in a colocation will handle enough traffic until you can afford to hire all the people to make you web scale. But then I started playing with the various tools and seeing how people used them, and it totally changed my view. The key point is that many of the new technologies in operations are about simplicity rather than speed. Standing up a stack in AWS can be flipped on and off like a light switch, and all the configuration steps can be much more easily automated/shared/updated/documented etc... It's not about any of these technologies being more efficient; it is about spending more in order to abstract away many of the headaches that slow down development. Certainly there are some people who are prematurely planning for a deluge of traffic and spending waayyy too many engineering resources on a 'web scale' stack, but that's not the majority.
- jammygit 8y ago
  This is a really interesting comment, thanks
Aeolun 8y ago
The load average on my kubernetes cluster is actually around 3-4 without it even doing anything. There’s a bunch of apps running in there, but nothing that would justify the load. It’s also generating roughly 20 log lines per second. I’m really not sure what it’s doing...
navinsylvester 8y ago
Hacker news running on a single server is a very bad precedent. Wish people running the show address it quickly since its being used as a bad example. When building a business you should take care of having an environment which is resilient. I agree it's not for everyone. But its quite essential when you have a huge customer base and care about unpleasant experience. If someone is running an important business and leaving it to chance - its just pure arrogance or gross incompetence.
wilsonnb3 8y ago
I did a programming project for a job interview recently at a company called Willowtree that makes iOS and Android apps for other companies. It was a pretty simple project, basically wrap a rest API around some JSON data provided to you. I ended up deploying mine to Google Cloud Platform onto a VM running Ubuntu and Apache, and they seemed rather concerned that I took that approach instead of leveraging some kind of containerization or PaaS approach. My API definitely had problems, as I don’t have much back end experience, but I found it strange that they would look down upon deploying to a cloud VM. It doesn’t seem like it was that long ago that a VM hosted on AWS or Digital Ocean was the latest and greatest and it seemed like a logical choice for something that would only ever be used by about five people.
- sdotsen 8y ago
  You prob dodged a bullet then. We give out take home exercises (not my idea but whatever), and we tell the client we don't care which config mgmt you use just pick something you're comfortable with. We use TF+Ansible but we would never frown upon work that's something using Salt, Chef or CFEngine.
oftenwrong 8y ago
It's convenient to be able to trivially create new production-like environments. Great for reproducing bugs or simulating deploys or running demos. My company's setup and scale doesn't necessitate kubernetes, but I still find it useful. It was fairly straightforward to set up.
solatic 8y ago
Stop thinking of Kubernetes as an easy way to scale ops for a single app, and start thinking of Kubernetes as an easy way to scale ops for non-trivial amount n apps. If you're a startup with a monolith then sure, you probably don't need Kubernetes. If you're not using Heroku/GAE/etc. then you generate a machine image from your app, deploy it behind a load balancer (start with two servers), and use some managed database for the backend. That's pretty simple. You can scale development without scaling the size of your ops team (1-2 people, only need two if you're trying to avoid bus factor 1), at least until you need to outscale a monolith. If you need to run a bunch of applications, made by a bunch of different teams (let alone when they don't work for you - i.e. an off-the-shelf product from a vendor), then using a managed Kubernetes provider makes this relatively simple without needing more people. If you try to do that without containers and orchestration, and want to keep a rapid pace of deployment, and not hire tons more people, you will go crazy.

deleted 8y ago

[deleted]

tshannon 8y ago

Does hacker news really run on one server? What if server goes down? I've always though high availability was the more important reason for multiple servers, rather than performance. Even if you have only two paying customers, they are probably paying for the right to hit your website / service 24/7.

lostapathy 8y ago
What if hacker news goes down, more work gets done for the day? It’s really common to over estimate the cost of being down, while underestimating the costs of resiliency. And in the end if you can’t fully afford the resiliency, you end up with a system that’s more complicated and thus less stable than it could have been had you just accepted a tiny bit of risk.
- _drFaust 8y ago
  extremely valid point. The initial setup isn't the challenge, it's the endless tweaks to make it fault tolerant. For anyone looking into k8s, take time to research best practices for monitoring and readiness probes.
wpietri 8y ago
> What if server goes down? Productivity of Y Combinator startups goes up. So YC benefits either way.
matuszeg 8y ago
Has hacker news ever gone down? I cant remember it ever going dkwnt but I assume it has to have at some point.
- iBelieve 8y ago
  I've seen it happen at least once in the few years that I've been on Hacker News, see https://news.ycombinator.com/item?id=17228704 https://news.ycombinator.com/item?id=17228704
- freehunter 8y ago
  Here's the official status twitter: https://twitter.com/hnstatus https://twitter.com/hnstatus
- collyw 8y ago
  I got a server can't respond to your request type message this morning. OK, the server wasn't down, but it was loaded enough that it couldn't serve me. A page refresh and it was working.
jimnotgym 8y ago
An interesting case study of an app at scale is Stack Overflow. Rather more than one server, but rather less than you might expect. Edit: fixed url https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-stack-overflow/ https://nickcraver.com/blog/2013/11/22/what-it-takes-to-run-...
- branko_d 8y ago
  Key point: "The primary reason the utilization is so low is efficient code."
paulddraper 8y ago
One application server, yes. HN is fronted by Cloudflare for CDN + DDoS protection, which of course is a lot more than one server. That's why if you get a particularly long thread (1k+ comments), admins will beg people to log out so that the responses can be served from the CDN cache. Example: American 2016 presidential election https://news.ycombinator.com/item?id=12909752 https://news.ycombinator.com/item?id=12909752 (1,700 comments)
- tambre 8y ago
  HN hasn't been fronted by Cloudflare since July.
  - paulddraper 8y ago
    Hm, interesting. I noticed HN started responding to HEAD requests with 405s recently; perhaps that is the cause.
  - ksec 8y ago
    Why did they stop using Cloudflare?
    - tambre 8y ago
      It was part of their networking rework. Presumably to improve stability of the website. I've also been told that it was a big step towards enabling IPv6 for HN. There are other parts of Y Combinator that still use Cloudflare though.
- detaro 8y ago
  Not Cloudflare anymore. Traffic goes to somewhere in San Diego from Europe, which it wouldn't if it was Cloudflare (different IP range too)
vesak 8y ago
Well, there's another important question: What percentage of all services need high availability? Stetson-Harrison method shows that it's less than 5%.
- kodablah 8y ago
  Need is determined by the customer. You might be able to explain to them that they don't need it on their way to a competitor. May not be fair or rational, but redundancy and its peace-of-mind has real benefits in most systems compared to cost.
deleted 8y ago
[deleted]

devman0 8y ago

I think for smaller use cases its more about high availability than load balancing.

gambler 8y ago

I realize there is a need for multi-server applications with automated deployment and scaling. However, the accidental complexity of serverless setups and container orchestration tools is just off the charts. When reading these articles I get roughly the same feeling I got when reading J2EE articles back when J2EE was "the future" and "the only way to build scalable infrastructure".

collyw 8y ago
Pepole say that serverless keeps thing simpler. In reality its just moving the complexity to different areas (devops as opposed to application complexity). We have been writing applications for longer, so it should be easier to keep the complexity lower if you keep the logic there. Are there similar well known patterns / best practices / ways of structuring devops that have been developed for applications over the years?

citilife 8y ago

For reference, this runs on a $5 AWS instance: https://hnprofile.com/ https://hnprofile.com/ The database is $600 per month, but that data runs five different websites (and it's a few hundred Gb of data). EDIT: for those mentioning the 502 gateway error, it does auto scales - Now it's costing more per month, at least temporarily.

supakeen 8y ago
Not a good example as it is currently 502'ing: https://i.imgur.com/sU8Zn5v.png https://i.imgur.com/sU8Zn5v.png
wnevets 8y ago
>502 Bad Gateway maybe a $5 instance isn't enough
kayoone 8y ago
$5 AWS instance ? aren't they all substantially more than that ?
- maxerickson 8y ago
  Nope. https://aws.amazon.com/ec2/pricing/reserved-instances/pricing/ https://aws.amazon.com/ec2/pricing/reserved-instances/pricin...

SEJeff 8y ago

FreeBSD, Apache, Python - the FAP stack Ok. I'll see myself out.

movedx 8y ago
I laughed ;-)
- collyw 8y ago
  Me too, surprised this hasn't been down-voted to hell on here.

mdekkers 8y ago

Not everyone uses K8s for webapps. You would be surprised at the level of enterprise penetration of K8s. Those enterprises do boring stuff like datawarehousing etc.

jimmy1 8y ago
One of the more interesting use cases I have read in recent memory: Chick-fil-A used it to set up a bare metal edge compute network between all it's restaurants The how: https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at-chick-fil-a-scale-7b0607bd3541 https://medium.com/@cfatechblog/bare-metal-k8s-clustering-at... The why: https://medium.com/@cfatechblog/edge-computing-at-chick-fil-a-7d67242675e2 https://medium.com/@cfatechblog/edge-computing-at-chick-fil-...

pythonaut_16 8y ago

I personally use Docker combined with a $5 droplet on Digital Ocean. This makes it easy to spin up multiple applications and sites without worrying about conflicting dependencies, and docker-compose gives me most of the benefits of orchestration tools (e.g. Kubernetes) that actually matter for my small scale usage. Also Traefik makes a nice load balancer for this uage

bdcravens 8y ago
> docker-compose gives me most of the benefits of orchestration I feel this is a very unappreciated feature of docker-compose. I've gotten pretty far with setting restart: always, baking a machine image, using cloud scaling and load balancers.
ngrilly 8y ago
Interesting! When you rollout a new version, how do you coordinate docker-compose and Traefik to avoid any user-facing downtime (no 502)?
- pythonaut_16 8y ago
  The simplest answer is I am the only user currently so it doesn't matter. Even with users, a short down time to bring down the old images and bring up the new images should be manageable. If I ever got to a point where that wasn't sufficient then I would consider that project successful enough to merit investment into zero downtime strategies and probably its own VPS as well. From a more technical side, I haven't seen an issue with Traefik when I take a container down and back up again. The only delay I would anticipate is when you bring a new service online for the first time and Traefik detects it and configures + fetches LetsEncrypt certs for it.
  - ngrilly 8y ago
    Thanks.

wpietri 8y ago

As somebody who has his own colocated server (and has since Bubble 1.0), I definitely agree that the old-fashioned way still works just fine. On the other hand, I've been building a home Kubernetes cluster to check out the new hotness. And although I don't think Kubernetes provides huge benefits to small-scale operators, I would still probably recommend that newbs look at some container orchestration approach instead of investing in learning old-school techniques. The problem for me with the old big-server-many-apps approach is the way it becomes hard to manage. 5 years on, I know that I did a bunch of things for a bunch of reasons, but I don't really remember what or why. It mixes intention with execution in a way that gets muddled over time. Moving to a new server or OS is more archaeology than engineering. The rise of virtual servers and tools like Chef and Puppet provided some ways to manage that complexity. But "virtual server" is like "horseless carriage". The term itself indicates that some transition is happening, but that we don't really understand it yet. I believe containers are at least the next step in that direction. Done well, I think containers are a much cleaner way of separating intent from implementation than older approaches. Something like Kubernetes strongly encourages patterns that make scaling easier, sure. But even if the scaling never happens, it makes people better prepared for operational issues that certainly will happen. Migrations, upgrades, hardware failures, transfers of control.

heipei 8y ago
This. Kubernetes (or whatever other container scheduler) might feel like overkill, but if all they do is force you to adopt a container-centric / 12-factor way of building your applications it was worth trying them. And once you've adopted that workflow it's a no-brainer to go from a single node to a cluster which will dynamically allocate the workloads it runs. Running a small container cluster at work has even changed how I setup single-host projects in my spare time: I will build everything into a container, bind-mount whatever it might need, create a simple systemd unit that just runs / rms the docker container on start and stop. Bliss.
3pt14159 8y ago
I've found it just pushes the complexity elsewhere or opens up (or silences) performance or security problems you wouldn't have had if you'd stuck to the old fashion way of doing things. Keep checklists and script what you can. I find it helpful to follow edge whenever I can so if I hit a snag the developers that made the change still have that change fresh in their mind. It really doesn't take that much time to keep things updated if you don't go hog-wild on different libraries and if your stuff is reasonably well tested. As an aside, I've been thinking that there should be a stack that is designed and built for the sole purpose of staying stable over decades. Something with a bunch of stripped down technology. As robust and stable as possible. Only allow security updates. Only allow certain character sets. Something built on a language that is just stupid simple and secure. A Swift or Rust subset maybe? Lua? Lisp without the macro insanity? If the future needs some sort of tech that we didn't anticipate (say, something to handle quantum computers breaking cryptography) then the stack should be setup in such a way to decouple the varying layers with minimal work. I liken it to building codes. We should have pre-setup combinations of technologies that are stable, simple, and combinable. Sure, go outside them for skyscrapers, but for the day-to-day buildings things are getting too complicated.

dcosson 8y ago

I really think that running a LAMP server for the average beginning developer these days would be just as complicated, maybe more complicated, than running a single deployment on Google Kubernetes Engine. You have to know about package managers and init systems and apache/nginx config files and keep track of security updates for your stack and rotate the logs so the hard drive doesn't fill up. If you already know how do this stuff in your sleep because you've done it for years, then yeah, don't fix what isn't broken. But if you're starting with no background, there's nothing inherently wrong with using a more advanced tool if that tool has good resources to get you started easily. Just because there's more complexity in the entirety of the stack when running an orchestration system doesn't necessarily mean more complexity for the end user. Side note - couldn't you make a similar argument about any kind of further abstraction? "Question for all you hackers out there - do you really need HTTP requests with their complicated headers and status codes and keepalive timeouts? I run several apps just sending plain text over TCP sockets and it works fine."

gambler 8y ago
>I really think that running a LAMP server for the average beginning developer these days would be just as complicated, maybe more complicated, than running a single deployment on Google Kubernetes Engine. Back when I first started doing web dev I went from knowing nothing about server setups or Unix (i.e. running off managed hosting) to a reasonably secure FreeBSD server with a working content management system in 3 days. This included installing the OS. The same FAMP setup (with modifications and updates, obviously) continued to work perfectly fine for the next decade. Re-edit: In the original version of the post I drew a parallel with several teams at my previous job "figuring out" AWS Lambda for several month, and stumbling over gotchas, multiple ways of doing everything and a myriad of conflicting tools. Since there is a reply to that statement, I guess I will add this note.
- pacala 8y ago
  Containers are a mechanism to run your old machine[s], but with a reproducible setup script. A machine packed in a container happens to also run on your dev/CI environments. There isn't much logical difference between a physical machine, a VM and a container [0]. Serverless offers a large surface of APIs, some of them proprietary, tangled in an ever evolving dependency hell. Historical note: Google Cloud started serverless with AppEngine, then focused on GCE [and later GKE] _because_ serverless was hard and AWS was eating their launch with VMs. [0] For example, we can argue about security isolation issues in containers vs VMs. Eventually this will become a moot point as technology advances far enough that we can run each container inside a hardware-backed VM.

draw_down 8y ago

Well, I don't know where you work, I guess at the company that makes little bespoke apps that all run on a single point of failure, but I actually do work somewhere that runs at the scale of the sort of companies you mention. So it's good to know about techniques and tools that are useful in that context. Hacker News is great and all, but it ain't a business. And if it goes down, well, surf another website for a while. Many of us work in situations that are significantly more... constrained than that. We aren't able to say, "Hey, we're down at the moment so just use one of our competitors for a while, is that chill?" So it's maybe not the best example of what we apparently should all be doing. I don't know this mythical person is, who's always talking about "webscale" and overengineering the shit out of everything. Kind of a tired strawman imo

gizmodo59 8y ago

I think of these in this way: 99% of the apps are developed by the developers who are not in the top 1%. The cheap access to computing power has led to a growth of developers beyond the highly skilled ones who can milk out everything available in a less powerful computer. I'd like to believe we are at an Electron development phase where we just want to ship as much as possible easily without worrying about hiring great talent (And yeah I hate that its inefficient in terms of memory usage). This has led to the explosion of so many frameworks that does a lot of things easily which requires such complex devops pipeline.

StavrosK 8y ago

They do not, I run tens of low-traffic projects very successfully on a $10/mo Hetzner server on Dokku. Dokku is amazing and so is Hetzner, I don't know why people always go for the high-scalability, expensive options just to end up with 0 utilization.

devhead 8y ago
if you have zero utilization, then you should scale down or lower your instance type until you are optimized for performance and cost. Which is easier to do when you are in an environment such as gce or aws.
- StavrosK 8y ago
  There's a minimum when you have deployed each one of your hundred microservices to a server, though.
Aeolun 8y ago
Because my company is risk-averse, and perfectly happy to drop $1000 a month for a ridiculously overprovisioned database instance just to ensure an issue with the database will never cause their contracts to be lost.

grey-area 8y ago

I think you're right - at least 90% of servers on the web would be fine with a couple of instances at most backed by a decent db. It can get more compex depending on your resiliance requirements but it really doesn't have to be. I guess I run a CPG stack - Coreos, Postgresql, Go. Don't bother with containers as Go produces one binary which can be run under systemd. It is far simpler than kubernetes and the only real reason for other servers is redundancy. The only bit of complexity is I usually run the db servers as a separate instance or use a managed service. You can go a long way with very boring tech. I've run a little HN clone written in go on one $5 digital ocean droplet for years - it handles moderate traffic spikes with little effort.

meritt 8y ago

As someone who runs a very successful data business on a simple stack (php, cron, redis, mariadb), I definitely agree. We've avoided the latest trends/tools and just keep humming along while outperforming and outdelivering our competitors. We're also revenue-funded so no outside VC pushing us to be flashy, but I will definitely admit it makes hiring difficult. Candidates see our stack as boring and bland, which we make up for that in comp, but for a lot of people that's not enough. If you want to run a reliable, simple, and profitable business, keep your tech stack equally simple. If you want to appeal to VCs and have an easy time recruiting, you need to be cargo cult, even if it's not technically necessary.

movedx 8y ago
Good on you mate. I love this approach. I'll be doing the same thing my self with a few products I'm developing with my brother. "Keep it simple, keep it stable" is what I like to say.
jdhawk 8y ago
I work in the same environment, and hiring is the only downside I see. Resume Driven Development and "Dev Sexy" has made it difficult to find developers who are willing to come on board, despite the sanity provided by simplicity & comp.
- Aeolun 8y ago
  It will now, but how many people are going to be interested in my LAMP experience 5 or 10 years down the line? While everyone else has been working with the cloud/kubernetes/aws/gcp/serverless technologies.
nineteen999 8y ago
Hear hear. I lead a team that builds and manages critical emergency services infrastructure. Our stack is pretty boring, but then it has to be running 99.999% of the time. Rather than wasting time chasing the latest flavour-of-the-month tool or framework we invest our time in plugging any kind of gap that could ever bring our service down. We don't need people who are only looking to pad their resume with the hottest buzzwords, we look for people who want to make critical services run all that time, that rarely fail and when they do, they handle failure gracefully. The number of devops/agile/SaaS style shops I have seen where the product falls over for several hours at a time is astounding, and it can often be attributed to rushed/inexperienced engineering or unnecessary complexity. Lucky for them it's usually just the shareholders bottom line that is affected. If the services my team provides don't work, ambulances and fire engines are either not arriving fast enough or at all.

ownagefool 8y ago

You probably don't need kubernetes. Lets be fair, it offers:- > Orchestration of block storage > Resource Management > Rolling Deploys > Cloud provider agnostic APIs* If you don't need any of these things, and your stack fits on a single server or two, and you aren't already familiar with it, I'm not sure why you bother other than an interest. That said, there's a world of companies that aren't FAANG, ub3r and Basecamp, and many of those paying reasonble sums of money have more complicated and resource intensive requirements that don't fit on a single server. Government Departments, Retail Companies and Banks all likely have a number of different software development projects where giving a number of developers API access to a platform that offers the above advatanges is, in my opinion, a good thing. Once you get to FAANG level, who knows whether kube itself will actually help or hinder at that level. * Personally I'd rather use the kube APIs than talking to any of the cloud providers directly. I imagine that's somewhat personal preference and somewhat because I've been able to easily run it in my basement. *2 Namespaces also makes making more environments for CI/CD easier, so as soon as you have a team of developers and you want to do that sort of thing, it also makes sense. Not so much for a loan developer and his server.

sleepybrett 8y ago
There is also overhead in the form of instances to run and maintain the backend datastore and control plane components. You should already be at a certain scale before considering kubernetes.
- ownagefool 8y ago
  This is free on GKE and I believe AKS, but of course if you're doing this yourself for some reason, you need to compare it with the alternatives.

aeriklawson 8y ago

> I manage dozens of apps for thousands of users. The apps are all on one server, its load average around 0.1. If you're at this scale you can do whatever you want. Most of the stuff I've made has been with simple building blocks like you've described, maybe thrown in with some caches and a load balancer. Although I've worked with other teams who really did have the high scale request flows that require you think about using a different architecture. Even so, K8s is not the end game and you can make something work even just extending the LAMP stack.

staticassertion 8y ago

I think that kub and, generally, cloud providers have allowed for more ambitious projects to be generally accessible. My side project is intended to handle > 1 billion events per day, with fairly low latency. That's well over 10k events per second. I doubt I could do this easily on a single box, and I wouldn't really want to try. Why constrain myself that way? Is it worth just doing this the standard LAMP way? More and more problems are available to be solved using commodity systems, so we have more and more people solving those problems with these new systems.

toredash 8y ago
Depends on the box. Per core performance is probably your metric there if the app can utilize multiple cores.

kodablah 8y ago

> Some day I would like a powwow with all you hackers about whether 99% of apps need more than [...] Close. But I also need it HA w/ automatic failover, auto-SSL certs (meaning I might need to give my DNS provider creds depending upon LE approach), notifications on outages, easy viewing of logs past and present, easy metric viewing, automatic backups, and updates that are easy for me to sign off on to then run. I'll do the plugging on the app side (that is exporting metrics, logs, etc). And no vendor-specific solutions (even if they are repackaged common components like RDS is for Postgres), I should be able to run on a couple VMs on my laptop if I want and it be the exact same. And I may want to add an MQ/stream (e.g. Kafka), in-mem DB (e.g. Redis), etc later and still want log aggregation, metrics, backups, etc. Really, that's not asking too much but it's definitely more than LAMP. We need a pithy name for this startup-in-a-box (again, that's NOT a PaaS, but a self-hosted management on an existing set of servers). Nobody wants to fumble w/ Ansible/Puppet/Salt/Chef/whatever all over the place or hire an ops guy, and people don't want to use vendor-specific solutions. I agree with the "you only need this"...but we need just a bit more to handle outages and auditing.

movedx 8y ago

Spot on, friend. So recently I started writing a simple web application for my family. They send emails to each other with gift wish lists in them and we all have to juggle those emails around. I figured some products would exist already to solve this problem, but I wanted to make my own. When it came time to make it I thought: "This has to be a REST API with a JS front end" and then further down the line, "Man I should use Flutter and only make it a mobile app!" I had other thoughts about making it Serverless and doing $thisCoolThing and using $thatNewTech. In the end nothing got done at all. Fast forward to today and it's a monolith Go application that renders Bootstrap 4 templates server-side, serves some static CSS directly, sits on a single server (DigitalOcean) and uses a single PostgreSQL instance (on the same server). The Bootstrap 4 CSS and JS come from their CDN. I made the technology simpler and the job got done. It's an MVP with basic database backups in place, using Docker to deploy the app. It just works. Lessons for me from this: * Server-side template rendering is perfectly fine and actually easier, frankly * JS can still be used client-side to improve the experience without replacing the above or making the entire rendering process client-side * Although Go compiles to a single static binary I still need other assets, so it went into a Docker container for the added security benefits not to mention portability * Serverless is nice, but unless it's replaced the above day-to-day, there's always a steep learning curve around something you haven't done with it yet, but need * Picking the latest and greatest tech tends to stagnate progress or halt it entirely, in most cases * An a software MVP needs an MVP infrastructure to go with it Just my thoughts.

Enginerd3 8y ago
"I figured some products would exist already to solve this problem, but I wanted to make my own." But why? You could've used so many different products. You could've even used Google Docs.
- movedx 8y ago
  There's a bit more to the story than simply solving a problem. I have other plans for the software and we have ideas about how we want to move it forward. One of the key features is being able to "tag" an item on someone's wish list as "bought" or "buying". This allows others viewing the list to know that item has been taken. But there's also a requirement that the original author of the list/item cannot see that it has been bought otherwise there's no element of surprise for them come time to open the gift(s) :-) Spreadsheets don't enable that privacy/secrecy.

treis 8y ago

>I manage dozens of apps for thousands of users. The apps are all on one server, How are these backed up?

kodablah 8y ago
And how do they fail over when the server dies (at least the non-user-app part like their DBs)?

manigandham 8y ago

It's not just about scaling. That seems to be the only thing people talk about because it sounds sexy but the reality is about operations. Kubernetes makes deployments, rolling upgrades, monitoring, load balancing, logging, restarts, and other ops very easy. It can be as simple as a 1-line command to run a container or several YAML files to run complex applications and even databases. Once you become familiar with the options, you tend to think of running all software that way and it becomes really easy to deploy and test anything. So yes, for personal projects a single server with SSH/Docker is fine, but any business can save time and IT overhead with Kubernetes. Considering how easy the clouds have made it to spin up a cluster, it's a great trade-off for most companies.

ecoqba11 8y ago
Exactly!
wetpaste 8y ago
Exactly. It solves some of the most important problems that come up of working with microservice based architectures, and establishes mature patterns around the ability for multiple developer teams to update and scale each piece of a distributed application. Also, what do you do when your digital ocean droplets needs hundreds or thousands of customers on it? Maybe each customer needs it's own database (well in my case they do), configuration, storage requirements, and multi-node requirements. How do you keep track of all that, how do you automatate it and QUICKLY recover in failure scenarios. You need to be able to deal with failure on a node, or if there's a bad actor you need to be able to move them off easily without downtime or affecting other customers, automatically, seamlessly. You need to be able see an overview of your resources across all nodes and where apps are placed and have something decide if the hardware your new container is being added to can handle another JVM or whatever. For cost effectiveness, you want to be able to overcommit resources, so you want containers. You want those to translate to other platforms, AWS, google, azure, on-prem. You have a single declarative language that works anywhere you can deploy a k8s cluster. You need to deal with growth, good patterns for rolling back and updating versions of parts of the stack. You want all of your deployments to be declarative and to be able to tightly control the the options for each one and get back to where you were. I agree that it doesn't make sense for everything, and it requires fundamental understanding of linux and software before it even makes sense to try shoehorning onto, but it solves real world problems for many people, it's not just a hype thing. I would say docker itself was more of a hype thing than k8s, the maturity and features of k8s and other orchestration systems that came out of the docker model is there for a reason, because it solves all of the real-world problems people couldn't solve with vanilla docker without tons of custom scripting and hacky workarounds. Docker solved the big problem by providing the isolating environments for each app and splitting things out into microservices that way, without having to commit a full statically resourced VM or bare metal per service. K8s solves all of the other problems that came out of that (pods, stateful sets, init containers, jobs, cronjobs, service definitions, deployments, volume claims.)

derefr 8y ago

> If it's high, is it because you are getting 100,000 requests per second, or is it the frameworks you cargo-culted in? Mine's high because our business model involves blockchain stuff, and 1. blockchain nodes are CPU+memory+disk hogs; 2. ETL pipelines that feed historical data in from blockchains produce billions of events in their catch-up phase. (And we're constantly re-running the catch-up phase as we change parameters.) Sadly, we need several fast servers even without any traffic :/

tptacek 8y ago

The reliability and performance story for Hacker News is not great, and that's despite the fact that its design has lots of simplifying assumptions. I wouldn't call HN a success story for the "just drop it on a server" approach. Of course, HN is a kind of art project, and its scaling and performance goals are not typical of most applications.

z3t4 8y ago

On a service that has been up for almost 20 years, same code base, thousands of daily users, the first server was constantly on 100% CPU. The second server, averaged around 10% CPU with lots of spikes. The third server, now average below 1% CPU usage. Next time I need to upgrade I will probably get a "NUC", or a smarphone, or something even smaller. But it's not only CPU's that has gotten better. The first server also maxed out the bandwidth! And now, although with less users, the bandwidth usage is less then 1% Started out on 0.5Mbit DSL, and it's now on a Gbit fiber.

johngalt 8y ago

Initially I was skeptical as well. One server in a colocation will handle enough traffic until you can afford to hire all the people to make you web scale. But then I started playing with the various tools and seeing how people used them, and it totally changed my view. The key point is that many of the new technologies in operations are about simplicity rather than speed. Standing up a stack in AWS can be flipped on and off like a light switch, and all the configuration steps can be much more easily automated/shared/updated/documented etc... It's not about any of these technologies being more efficient; it is about spending more in order to abstract away many of the headaches that slow down development. Certainly there are some people who are prematurely planning for a deluge of traffic and spending waayyy too many engineering resources on a 'web scale' stack, but that's not the majority.

jammygit 8y ago
This is a really interesting comment, thanks

Aeolun 8y ago

The load average on my kubernetes cluster is actually around 3-4 without it even doing anything. There’s a bunch of apps running in there, but nothing that would justify the load. It’s also generating roughly 20 log lines per second. I’m really not sure what it’s doing...

navinsylvester 8y ago

Hacker news running on a single server is a very bad precedent. Wish people running the show address it quickly since its being used as a bad example. When building a business you should take care of having an environment which is resilient. I agree it's not for everyone. But its quite essential when you have a huge customer base and care about unpleasant experience. If someone is running an important business and leaving it to chance - its just pure arrogance or gross incompetence.

wilsonnb3 8y ago

I did a programming project for a job interview recently at a company called Willowtree that makes iOS and Android apps for other companies. It was a pretty simple project, basically wrap a rest API around some JSON data provided to you. I ended up deploying mine to Google Cloud Platform onto a VM running Ubuntu and Apache, and they seemed rather concerned that I took that approach instead of leveraging some kind of containerization or PaaS approach. My API definitely had problems, as I don’t have much back end experience, but I found it strange that they would look down upon deploying to a cloud VM. It doesn’t seem like it was that long ago that a VM hosted on AWS or Digital Ocean was the latest and greatest and it seemed like a logical choice for something that would only ever be used by about five people.

sdotsen 8y ago
You prob dodged a bullet then. We give out take home exercises (not my idea but whatever), and we tell the client we don't care which config mgmt you use just pick something you're comfortable with. We use TF+Ansible but we would never frown upon work that's something using Salt, Chef or CFEngine.

oftenwrong 8y ago

It's convenient to be able to trivially create new production-like environments. Great for reproducing bugs or simulating deploys or running demos. My company's setup and scale doesn't necessitate kubernetes, but I still find it useful. It was fairly straightforward to set up.

solatic 8y ago

Stop thinking of Kubernetes as an easy way to scale ops for a single app, and start thinking of Kubernetes as an easy way to scale ops for non-trivial amount n apps. If you're a startup with a monolith then sure, you probably don't need Kubernetes. If you're not using Heroku/GAE/etc. then you generate a machine image from your app, deploy it behind a load balancer (start with two servers), and use some managed database for the backend. That's pretty simple. You can scale development without scaling the size of your ops team (1-2 people, only need two if you're trying to avoid bus factor 1), at least until you need to outscale a monolith. If you need to run a bunch of applications, made by a bunch of different teams (let alone when they don't work for you - i.e. an off-the-shelf product from a vendor), then using a managed Kubernetes provider makes this relatively simple without needing more people. If you try to do that without containers and orchestration, and want to keep a rapid pace of deployment, and not hire tons more people, you will go crazy.