5 ms·

I hate when stack overflow is held up as an example of how we can run any system on “a few servers” - stacknoverflow has like 3 features and has an engineering

I hate when stack overflow is held up as an example of how we can run any system on “a few servers” - stacknoverflow has like 3 features and has an engineering focus on the single goal of performance and keeping on running on the small subset of servers. Every other project as different constraints.

Etheryte 5y ago
I don't think this really does justice to what Stack Overflow does. They're probably the most visited engineering-related site on any given day. They have their widely known Q&A, live updates and notifications, chat rooms, a blog, job listings with email updates, review queues, moderator tools, and so forth. Perhaps you only use three features, but the site does a tremendous amount, and that's not even thinking about all the cross Stack Exchange network functionalities.
- ed_elliott_asc 5y ago
  Perhaps I was being a little dismissive but it doesn’t have the same features of something like Facebook or LinkedIn or google
  - Cthulhu_ 5y ago
    Mind you, those companies probably shouldn't have as many features as they have.
  - shakna 5y ago
    > but it doesn’t have the same features of something like Facebook or LinkedIn or google But it does seem to have most of the features of something like Reddit.
lmilcin 5y ago
I think number of features have little to do with the situation. Nowadays, everybody insists on putting stuff on K8s regardless of how large or small it is. An application is an application for the purpose of running it on a server. It doesn't really matter how much functionality it has. It is microservices "revolution" (quotes intentionally) that caused larger applications to be split a lot of small ones and complicated the execution environment to the point that a lot of people spend a lot of time just trying to figure out how to run their applications reliably. That is not necessary. If you can have multiple microservices, more likely than not you can have them as separate libraries used by single application or a separate modules of a single application. Just make sure to put the same thought into modularizing it as you would designing microservices APIs and you can have the same but much easier and with much better performance (no serialization/deserialization, no network hops, no HTTP stack, no auth, etc.)
- cameronh90 5y ago
  "Nowadays, everybody insists on putting stuff on K8s regardless of how large or small it is." But if you already have the tooling, experience and support for k8s, why wouldn't you use it? I can fire up a k8s cluster on any major cloud provider in minutes (or bare metal in slightly longer), and deploy almost any web app using a standard interface that has extremely wide adoption. K8s supports a ton of things, but you don't have to use them. It can be complicated, but usually it's not. It feels a bit like saying why use Linux for your simple app when it could run just fine on DOS or QNX. How many years of my life have I wasted debugging networking issues on Linux or Windows that turned out to be caused by a feature I wasn't even (intentionally) using...
  - lmilcin 5y ago
    > It can be complicated, but usually it's not. The usual story. Everything works, until it doesn't. If you are a huge corp with good engineering you can have people dedicated to understanding k8s and then it kinda makes sense. They can spend time to learn it really well so that they have necessary chops to deal with problems when they happen. On the other hand, if you are smaller company, you are more likely embracing this new idea of developers running everything including k8s, you are in for a trouble. They will know how to make it work but that's about it. Because if you need to learn everything you actually learn nothing very well. And there certainly isn't enough time in the world to learn everything in development. My philosophy is applications must be built for when it breaks and it is unacceptable to run an application with a team that will not be able to fix it if it breaks. ** Couple of years ago I joined a small group of teams of developers (together about 40 devs) who together maintained a collection of 140 services, all with same or very similar stack (Java, Spring, REST, RabbitMQ). They had trouble delivering anything to prod because of complex dependencies, requirements, complex networking, complex process to find out where stuff broke between 7 layers of services between the original user call and the actual execution of their action. I rolled my sleeves and put everything in a single repo, single build, single set of dependencies, single version, single deployment process, single layer of static app servers. I left after the team was reduced from 40 to 5. There was no problem delivering anything (we had 60 successful deployments in a row when I left) and the guys who were left admitted they are bored and underutilized because of how easy it is to make changes to the system. These were still the same guys that were advocating for microservices. From what I heard they are not advocating for microservices anymore. Can microservices be done well? Sure they can. But it takes additional effort and experience to do it well. Why make your life difficult when it is not needed?
tluyben2 5y ago
I hate it when people act like every other project has different constraints. For most web projects that is simply not the case. Maybe in some cases there are small differences which won't be noticed by any run of the mill setup. If your constraints are different but you have 100 members and never more (like most startups) your constraints generally don't matter even if they are radically different. Most people here are overarchitecting for the "chance of becoming Facebook"; personally I rather make a lot more profit (from day 1) by using cheap stuff and simple setups and when we accidentally make it to Facebook size, we will revisit. We can move it to Amazon to keep running and in the meanwhile rearchitect. But chances are against us, and all of you here, that we will ever reach anything close to this. Even with investor money (just did 2 raises) we have better stuff to do.
- p_l 5y ago
  Or maybe we're running a more complex infra not because of scaling, but because declarative setup free from expense of cloud services (esp. important outside of SV/rUSA bubble) can mean that hosting the services that are needed to run the project can be made cheaper, both in time and materiel.
pjmlp 5y ago
Just like there are people that love to setup an Hadoop cluster for data that fits into a USB pen, there are others that love to ramp up a Kubernetes cluster for what is a typical three tiered application.

Etheryte 5y ago

I don't think this really does justice to what Stack Overflow does. They're probably the most visited engineering-related site on any given day. They have their widely known Q&A, live updates and notifications, chat rooms, a blog, job listings with email updates, review queues, moderator tools, and so forth. Perhaps you only use three features, but the site does a tremendous amount, and that's not even thinking about all the cross Stack Exchange network functionalities.

ed_elliott_asc 5y ago
Perhaps I was being a little dismissive but it doesn’t have the same features of something like Facebook or LinkedIn or google
- Cthulhu_ 5y ago
  Mind you, those companies probably shouldn't have as many features as they have.
- shakna 5y ago
  > but it doesn’t have the same features of something like Facebook or LinkedIn or google But it does seem to have most of the features of something like Reddit.

lmilcin 5y ago

I think number of features have little to do with the situation. Nowadays, everybody insists on putting stuff on K8s regardless of how large or small it is. An application is an application for the purpose of running it on a server. It doesn't really matter how much functionality it has. It is microservices "revolution" (quotes intentionally) that caused larger applications to be split a lot of small ones and complicated the execution environment to the point that a lot of people spend a lot of time just trying to figure out how to run their applications reliably. That is not necessary. If you can have multiple microservices, more likely than not you can have them as separate libraries used by single application or a separate modules of a single application. Just make sure to put the same thought into modularizing it as you would designing microservices APIs and you can have the same but much easier and with much better performance (no serialization/deserialization, no network hops, no HTTP stack, no auth, etc.)

cameronh90 5y ago
"Nowadays, everybody insists on putting stuff on K8s regardless of how large or small it is." But if you already have the tooling, experience and support for k8s, why wouldn't you use it? I can fire up a k8s cluster on any major cloud provider in minutes (or bare metal in slightly longer), and deploy almost any web app using a standard interface that has extremely wide adoption. K8s supports a ton of things, but you don't have to use them. It can be complicated, but usually it's not. It feels a bit like saying why use Linux for your simple app when it could run just fine on DOS or QNX. How many years of my life have I wasted debugging networking issues on Linux or Windows that turned out to be caused by a feature I wasn't even (intentionally) using...
- lmilcin 5y ago
  > It can be complicated, but usually it's not. The usual story. Everything works, until it doesn't. If you are a huge corp with good engineering you can have people dedicated to understanding k8s and then it kinda makes sense. They can spend time to learn it really well so that they have necessary chops to deal with problems when they happen. On the other hand, if you are smaller company, you are more likely embracing this new idea of developers running everything including k8s, you are in for a trouble. They will know how to make it work but that's about it. Because if you need to learn everything you actually learn nothing very well. And there certainly isn't enough time in the world to learn everything in development. My philosophy is applications must be built for when it breaks and it is unacceptable to run an application with a team that will not be able to fix it if it breaks. ** Couple of years ago I joined a small group of teams of developers (together about 40 devs) who together maintained a collection of 140 services, all with same or very similar stack (Java, Spring, REST, RabbitMQ). They had trouble delivering anything to prod because of complex dependencies, requirements, complex networking, complex process to find out where stuff broke between 7 layers of services between the original user call and the actual execution of their action. I rolled my sleeves and put everything in a single repo, single build, single set of dependencies, single version, single deployment process, single layer of static app servers. I left after the team was reduced from 40 to 5. There was no problem delivering anything (we had 60 successful deployments in a row when I left) and the guys who were left admitted they are bored and underutilized because of how easy it is to make changes to the system. These were still the same guys that were advocating for microservices. From what I heard they are not advocating for microservices anymore. Can microservices be done well? Sure they can. But it takes additional effort and experience to do it well. Why make your life difficult when it is not needed?

tluyben2 5y ago

I hate it when people act like every other project has different constraints. For most web projects that is simply not the case. Maybe in some cases there are small differences which won't be noticed by any run of the mill setup. If your constraints are different but you have 100 members and never more (like most startups) your constraints generally don't matter even if they are radically different. Most people here are overarchitecting for the "chance of becoming Facebook"; personally I rather make a lot more profit (from day 1) by using cheap stuff and simple setups and when we accidentally make it to Facebook size, we will revisit. We can move it to Amazon to keep running and in the meanwhile rearchitect. But chances are against us, and all of you here, that we will ever reach anything close to this. Even with investor money (just did 2 raises) we have better stuff to do.

p_l 5y ago
Or maybe we're running a more complex infra not because of scaling, but because declarative setup free from expense of cloud services (esp. important outside of SV/rUSA bubble) can mean that hosting the services that are needed to run the project can be made cheaper, both in time and materiel.

pjmlp 5y ago

Just like there are people that love to setup an Hadoop cluster for data that fits into a USB pen, there are others that love to ramp up a Kubernetes cluster for what is a typical three tiered application.