4 ms·

Startups don't always want the most cost-effective solution. A lot of startups' intentions is to look "serious". If everyone has an engineering playground full

Startups don't always want the most cost-effective solution. A lot of startups' intentions is to look "serious". If everyone has an engineering playground full of microservices with 5 different programming languages all hosted on AWS, you have to be the same if you want to join the "big boys" club. Sometimes startups don't even care about solving the business problem. The objective is instead to get the "startup CEO" lifestyle for the connections and reputation regardless of whether you end up solving your business problem in the end. Same for a lot of engineers - they'll happily join and build the engineering playground while enjoying their salary regardless of whether it's the right option from an engineering point of view. If anything, an engineering playground gives you the opportunity to manage lots of people and solve (self-inflicted) problems that will look good on your resume in a way that a boring solution with a handful of engineers won't. On the other hand, if you live and die by solving the business problem the most cost-effective way, all the "scalability" goes out the window and a simple stack on top of bare-metal makes much more sense, because the only thing that pays you is solving the business problem (in the cheapest way possible) as opposed to bragging about your self-inflicted technical complexity at AWS conferences.

marginalia_nu 5y ago
I wouldn't discount Hanlon's razor. There seems to be many people who simply fall for the marketing and think that kubernetes and hundreds of microservices are somehow necessary to run a small to medium sized web service. They can't comprehend how a technology stack with a 20 year track record could be even nearly as stable and reliable as a grab bag of 2 weeks old javascript microframeworks. It's absolutely inconceivable how something that performed well a decade ago on hardware that was a hundred times slower could keep up on modern servers.
- anderspitman 5y ago
  And unfortunately the effect flows both ways. I'm having a heck of a time finding a simple, modern log aggregator not designed for kubernetes and infinite scale. I have 4 load balanced VMs and literally just need to pull the logs into a central location sorted by time. There's a shocking amount of complexity around all the popular tools in this space.
  - marginalia_nu 5y ago
    Seems like an opportunity to build one ;-) The neat thing about simple tools is that they're often relatively simple to build.
  - ThrashBeard 5y ago
    Define modern? RSYSLOG has existed for a long time and is probably the simplest solution for sending logs somewhere.
  - icedchai 5y ago
    You can use logstash, without elasticsearch, without kibana. Example: use the logstash file output plugin and have it collect logs (from filebeat, syslog, or whatever you prefer) and dump them into a directory named /logs/<date>/service ... then just use grep or whatever.
Dave3of5 5y ago
I interviewed with a company that was like this. Their Engineering Manager took me through a systems design quiz. The problem was simple: * 100k users updating their gas / electricity meter readings every second My first question was what unit was the meter reading in, he said it was the current total (called a totalizer in the biz). I'd worked for a Flow Metering company before. So I said: "There's no need to upload the total every second reduce the frequency and the problem becomes easier" Didn't accept that and I said ok a few optimized servers behind a HAProxy load balancer would the trick as the processing is very simple here. The database is the harder part as you can't have a simple approach here is 100k requests / second is going to cause contention for most DB's. You'd need to partition the database in a clever way to reduce locking / contention. This answer was unacceptable he ignored it and asked me to start building some insane processing system where you use Azure Functions behind an api gateway to process every request into a queue. Then you have a scheduled job in Azure of which the name has changed by now. This job looks at the queue in batches of say up to 1000 records at a time and writes these in bulk into the DB Once it's finished it then immediately reinvokes itself to create a "job loop". You would create multiple of the "job loops" that run in parallel. Then you would need to scale up these jobs to drain the queue at a rate that meant we could process 100k requests per second. You would also need something to handle errors, for example requests that are broken somehow. These would go into a broken request queue and could be retried by the system automatically a certain number of times if they went over the number of retries then they would go into a dead letter queue to be looked at manually. I'm pretty sure this was the actual process he would take in designing this type of system. Use every option the cloud gives him to make some overly complex thing which is probably quite unreliable. I also suspect that's the reason I wasn't hired as I was just looking at the thing sensibly rather than in the "cloud native" way. He doesn't work for that company anymore and is now back to a normal (non-senior) software developer.
- Nextgrid 5y ago
  In his hypothetical system with queues and tasks, I wonder how much resources are being wasted on the management overhead (receiving tasks from the network, scheduling functions, the cold start of the Azure function, etc) compared to a boring solution in a cron on a high-performance bare-metal machine costing a fraction of his solution?

marginalia_nu 5y ago

I wouldn't discount Hanlon's razor. There seems to be many people who simply fall for the marketing and think that kubernetes and hundreds of microservices are somehow necessary to run a small to medium sized web service. They can't comprehend how a technology stack with a 20 year track record could be even nearly as stable and reliable as a grab bag of 2 weeks old javascript microframeworks. It's absolutely inconceivable how something that performed well a decade ago on hardware that was a hundred times slower could keep up on modern servers.

anderspitman 5y ago
And unfortunately the effect flows both ways. I'm having a heck of a time finding a simple, modern log aggregator not designed for kubernetes and infinite scale. I have 4 load balanced VMs and literally just need to pull the logs into a central location sorted by time. There's a shocking amount of complexity around all the popular tools in this space.
- marginalia_nu 5y ago
  Seems like an opportunity to build one ;-) The neat thing about simple tools is that they're often relatively simple to build.
- ThrashBeard 5y ago
  Define modern? RSYSLOG has existed for a long time and is probably the simplest solution for sending logs somewhere.
- icedchai 5y ago
  You can use logstash, without elasticsearch, without kibana. Example: use the logstash file output plugin and have it collect logs (from filebeat, syslog, or whatever you prefer) and dump them into a directory named /logs/<date>/service ... then just use grep or whatever.

Dave3of5 5y ago

I interviewed with a company that was like this. Their Engineering Manager took me through a systems design quiz. The problem was simple: * 100k users updating their gas / electricity meter readings every second My first question was what unit was the meter reading in, he said it was the current total (called a totalizer in the biz). I'd worked for a Flow Metering company before. So I said: "There's no need to upload the total every second reduce the frequency and the problem becomes easier" Didn't accept that and I said ok a few optimized servers behind a HAProxy load balancer would the trick as the processing is very simple here. The database is the harder part as you can't have a simple approach here is 100k requests / second is going to cause contention for most DB's. You'd need to partition the database in a clever way to reduce locking / contention. This answer was unacceptable he ignored it and asked me to start building some insane processing system where you use Azure Functions behind an api gateway to process every request into a queue. Then you have a scheduled job in Azure of which the name has changed by now. This job looks at the queue in batches of say up to 1000 records at a time and writes these in bulk into the DB Once it's finished it then immediately reinvokes itself to create a "job loop". You would create multiple of the "job loops" that run in parallel. Then you would need to scale up these jobs to drain the queue at a rate that meant we could process 100k requests per second. You would also need something to handle errors, for example requests that are broken somehow. These would go into a broken request queue and could be retried by the system automatically a certain number of times if they went over the number of retries then they would go into a dead letter queue to be looked at manually. I'm pretty sure this was the actual process he would take in designing this type of system. Use every option the cloud gives him to make some overly complex thing which is probably quite unreliable. I also suspect that's the reason I wasn't hired as I was just looking at the thing sensibly rather than in the "cloud native" way. He doesn't work for that company anymore and is now back to a normal (non-senior) software developer.

Nextgrid 5y ago
In his hypothetical system with queues and tasks, I wonder how much resources are being wasted on the management overhead (receiving tasks from the network, scheduling functions, the cold start of the Azure function, etc) compared to a boring solution in a cron on a high-performance bare-metal machine costing a fraction of his solution?