4 ms·

In 2003 I was a developer on a website that served a million pages per hour using 2 databases, 4 webservers, and associated networking. Most of that hardware w

by btilly 7y ago

In 2003 I was a developer on a website that served a million pages per hour using 2 databases, 4 webservers, and associated networking. Most of that hardware was there for automated failover. At that scale, orchestration is simply not a challenge. We wrote a deployment script to roll out code changes and life was good. 2020 hardware is not worse than 2004 hardware. And while that business eventually failed, it was not due to technology. (And it made a lot of money before it failed.) As I said, most here are not doing something harder now than we did then. The same class of solution works. As the old saying goes, KISS. Know when it works, and when it doesn't. As another saying says, YAGNIY. (You Ain't Gonna Need It Yet.) Deal with the limits of being monolithic after you encounter them. Don't complicate your life to tackle problems that don't exist.

mschuster91 7y ago
> In 2003 I was a developer on a website that served a million pages per hour using 2 databases, 4 webservers, and associated networking. Most of that hardware was there for automated failover. fwiw, Stackoverflow runs on a similarly sized/deployed stack. But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that works (I can't remember the last SO outage!), and their employees are also happy with this. Others, especially startups? Gotta chase these hipster credentials and use the free AWS/Azure/GCP credits somehow... and the employees don't want boring old school stuff, they want the newest hipster crap that's barely alpha tested, but in production. Utterly ridiculous. Most of them would be fine with three sub-$10k AMD servers and a bit of work by a seasoned Linux sys-admin. But eww. Linux management. Not cool/hip.
- heavenlyblue 7y ago
  Did that stack include daily traffic analytics hardware?
  - btilly 7y ago
    See https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... for an overview of their stack. They do NOT need a stack of daily traffic analytics hardware. As I said above, remote RPCs take an order of magnitude more resources than function calls. If you are used to using various distributed tools to do analytics, you are used to requiring an order of magnitude more hardware than a simpler solution. And if there are inefficiencies in your solution, it is easy for a second order of magnitude to sneak in. Therefore your experience is going to seriously mislead you about the likely limits of a simpler solution. Making this concrete, if your job requires a Hadoop cluster of less than 50 machines, odds are good that it could be run just fine on a single machine with a different technology choice. But if you are fully utilizing hundreds of machines in your cluster, then you actually need to be distributed. (Though you could achieve considerable savings for some of your jobs could be run in a more efficient architecture. Amazon in particular makes excellent use of this kind of hybrid architecture in places.) Google is making excellent decisions for the scale that they operate on. You just are unlikely to operate at a scale that makes their decisions make much sense for you.
  - mschuster91 7y ago
    For what the hell do you need traffic analytics beyond parsing access logs? Current ad-tech/analytics is a mixture of outright fraud, fraud sold as "AI", fraud sold as "big data", middlemen claiming to solve said fraud, and privacy invasions on a scale that would make the Stasi drool. Stackoverflow doesn't need this kind of shit and frankly, no one else does. The GDPR was a first start in the right direction and I'm seriously hoping coronavirus has at least one positive side and eliminates those fraudsters who survived the GDPR wave.
  - manigandham 7y ago
    Stackoverflow logs every request into SQL Server which uses columnstore tables to handle all their analytics. It's fast, efficient and more than enough.
- btilly 7y ago
  But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that works (I can't remember the last SO outage!), and their employees are also happy with this. Great example! Far too often we introduce complexity to manage complexity. Then we fail to manage the complexity that we created, and conclude that we need even more complexity! Taking the simple approach seems impossible and utterly scary. But it works more often than most would believe.
castillar76 7y ago
As someone said, "Scale is the problem (almost) everyone wants and (almost) no one has." Technologies like Kubernetes were developed from managing monster-sized applications like Facebook, Amazon, and Google. Devs really want to play with the cool tech, but unless they work for a FAANG company, the odds are good that not only do they not have those problems now, they will never have those problems. That's not to say that things like containers are useless. I've seen a lot of benefits from developers working on containerization because it forces them to think about making their code reproducible and automating their development and delivery models. That has huge benefits from a security perspective, even if the end result is that you take the artifacts produced and rsync them onto a remote VM in the datacenter and restart services. There's lots of good to be gained there—I just wish people would quit trying to forklift everything into containers (and thence into the cloud) when they don't have to.
- mschuster91 7y ago
  There do exist things that make it sense for even small shops putting their stuff into containers and then small-ish local/bare metal or even cloud k8s environments: 1) automatic restarts of services when they fail: yes, this is possible with systemd but it's a nightmare to set up compared to a simple HEALTHCHECK command in Docker or an appropriate config section in k8s/rancher. 2) dependency chains (e.g. a java app that depends on mysql database and an external elasticsearch): again, this is possible with systemd, but way easier in k8s. 3) hardware/OS maintenance: in ye olde days, the admin would have to manually shut down services, copy data to new servers, start services there and then pray everything would work to prepare for a large-ish OS update or a longer hardware maintenance without having too much downtime (or buy more expensive software versions such as JIRA/Confluence DC). With k8s/rancher, assuming shared NFS storage, it's as easy as draining and cordoning the node, minimal downtime (essentially the time the software needs to stop/start) and no network/firewall/LB surprises along the way. 4) related to 3), in the cloud, assuming that services have horizontal scalability in mind (which most popular PHP environments have), it's relatively easy and straightforward to use autoscaling to save costs on one side but also prevent your site from going belly-up in case of HN/Reddit mentions. With manual deployments? Ouch. 5) Software upgrades or the creation of stage environments are so much easier, assuming well-written Docker images and software. Again let's take JIRA and Confluence - manually installed it's a lot of fumbling around to upgrade an instance, but with Docker? As easy as incrementing the version tag in the k8s config and 5min later it's updated.

by btilly 7y ago

mschuster91 7y ago

> In 2003 I was a developer on a website that served a million pages per hour using 2 databases, 4 webservers, and associated networking. Most of that hardware was there for automated failover. fwiw, Stackoverflow runs on a similarly sized/deployed stack. But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that works (I can't remember the last SO outage!), and their employees are also happy with this. Others, especially startups? Gotta chase these hipster credentials and use the free AWS/Azure/GCP credits somehow... and the employees don't want boring old school stuff, they want the newest hipster crap that's barely alpha tested, but in production. Utterly ridiculous. Most of them would be fine with three sub-$10k AMD servers and a bit of work by a seasoned Linux sys-admin. But eww. Linux management. Not cool/hip.

heavenlyblue 7y ago
Did that stack include daily traffic analytics hardware?
- btilly 7y ago
  See https://nickcraver.com/blog/2016/02/17/stack-overflow-the-architecture-2016-edition/ https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... for an overview of their stack. They do NOT need a stack of daily traffic analytics hardware. As I said above, remote RPCs take an order of magnitude more resources than function calls. If you are used to using various distributed tools to do analytics, you are used to requiring an order of magnitude more hardware than a simpler solution. And if there are inefficiencies in your solution, it is easy for a second order of magnitude to sneak in. Therefore your experience is going to seriously mislead you about the likely limits of a simpler solution. Making this concrete, if your job requires a Hadoop cluster of less than 50 machines, odds are good that it could be run just fine on a single machine with a different technology choice. But if you are fully utilizing hundreds of machines in your cluster, then you actually need to be distributed. (Though you could achieve considerable savings for some of your jobs could be run in a more efficient architecture. Amazon in particular makes excellent use of this kind of hybrid architecture in places.) Google is making excellent decisions for the scale that they operate on. You just are unlikely to operate at a scale that makes their decisions make much sense for you.
- mschuster91 7y ago
  For what the hell do you need traffic analytics beyond parsing access logs? Current ad-tech/analytics is a mixture of outright fraud, fraud sold as "AI", fraud sold as "big data", middlemen claiming to solve said fraud, and privacy invasions on a scale that would make the Stasi drool. Stackoverflow doesn't need this kind of shit and frankly, no one else does. The GDPR was a first start in the right direction and I'm seriously hoping coronavirus has at least one positive side and eliminates those fraudsters who survived the GDPR wave.
- manigandham 7y ago
  Stackoverflow logs every request into SQL Server which uses columnstore tables to handle all their analytics. It's fast, efficient and more than enough.
btilly 7y ago
But that is because Stackoverflow is fine with their employees running a cheap, un-complex system that works (I can't remember the last SO outage!), and their employees are also happy with this. Great example! Far too often we introduce complexity to manage complexity. Then we fail to manage the complexity that we created, and conclude that we need even more complexity! Taking the simple approach seems impossible and utterly scary. But it works more often than most would believe.

castillar76 7y ago

As someone said, "Scale is the problem (almost) everyone wants and (almost) no one has." Technologies like Kubernetes were developed from managing monster-sized applications like Facebook, Amazon, and Google. Devs really want to play with the cool tech, but unless they work for a FAANG company, the odds are good that not only do they not have those problems now, they will never have those problems. That's not to say that things like containers are useless. I've seen a lot of benefits from developers working on containerization because it forces them to think about making their code reproducible and automating their development and delivery models. That has huge benefits from a security perspective, even if the end result is that you take the artifacts produced and rsync them onto a remote VM in the datacenter and restart services. There's lots of good to be gained there—I just wish people would quit trying to forklift everything into containers (and thence into the cloud) when they don't have to.

mschuster91 7y ago
There do exist things that make it sense for even small shops putting their stuff into containers and then small-ish local/bare metal or even cloud k8s environments: 1) automatic restarts of services when they fail: yes, this is possible with systemd but it's a nightmare to set up compared to a simple HEALTHCHECK command in Docker or an appropriate config section in k8s/rancher. 2) dependency chains (e.g. a java app that depends on mysql database and an external elasticsearch): again, this is possible with systemd, but way easier in k8s. 3) hardware/OS maintenance: in ye olde days, the admin would have to manually shut down services, copy data to new servers, start services there and then pray everything would work to prepare for a large-ish OS update or a longer hardware maintenance without having too much downtime (or buy more expensive software versions such as JIRA/Confluence DC). With k8s/rancher, assuming shared NFS storage, it's as easy as draining and cordoning the node, minimal downtime (essentially the time the software needs to stop/start) and no network/firewall/LB surprises along the way. 4) related to 3), in the cloud, assuming that services have horizontal scalability in mind (which most popular PHP environments have), it's relatively easy and straightforward to use autoscaling to save costs on one side but also prevent your site from going belly-up in case of HN/Reddit mentions. With manual deployments? Ouch. 5) Software upgrades or the creation of stage environments are so much easier, assuming well-written Docker images and software. Again let's take JIRA and Confluence - manually installed it's a lot of fumbling around to upgrade an instance, but with Docker? As easy as incrementing the version tag in the k8s config and 5min later it's updated.