12 ms·

A recurring pattern that I have noticed: 1) a company the size of Amazon, Facebook, or Google runs into a problem that comes from their gigantic scale, or the f

A recurring pattern that I have noticed: 1) a company the size of Amazon, Facebook, or Google runs into a problem that comes from their gigantic scale, or the fact that they're trying to do something very different (e.g. Google Maps compared to old-style web pages) 2) Big Company comes up with a new technology to solve their problem 3) Many, many small companies that will never become big enough to need this, and are doing far less exotic things that require it, will try to make their ordinary web page into a microservice single page app with a NoSQL backend, because that's what successful companies do. Imitating success often makes sense; after all, we don't want to imitate failure. But your company is almost certainly not facing issues at all like Amazon, Facebook, or Google, and also does not have nearly as many programmers to do it.

_pmf_ 8y ago
Adding to 1): more often than not, the "scaling" issues are 95% social and 5% technical.
GiorgioG 8y ago
Kubernetes comes to mind.
mmt 8y ago
> also does not have nearly as many programmers to do it. Nor the underlying infrastructure, such as, for example, physical server hardware that's a fraction of the cost of cloud.
pmiller2 8y ago
This doesn’t only apply to technology. It applies to hiring and other organizational processes as well.
- SideburnsOfDoom 8y ago
  e.g. "the spotify model"
- rossdavidh 8y ago
  Rings true. I once worked a contract at a place with proprietary hardware, that wanted their software extras to be purchased separately, and accessible through an "app store". This seemed to imply they thought somebody else would develop apps for their store, which was odd because if their proprietary hardware (which was quite expensive, justifiably so) sold 1000 units it would be considered successful. Basically nobody made apps for their app store except the people who worked for the same company that made the hardware, and I had a hard time understanding why they didn't use the (much simpler) model of "if you pay us $X more we'll throw in this bit of software". Instead, they had to develop all of the software for running and maintaining an app store, including the issues of making sure that apps uploaded to their store were not malware. Then, I realized what was going on: Apple had introduced their app store a few years before, and it was a big deal. Steve Jobs was the CEO on the cover of every magazine. The CEO looked at Apple, and said, "we must have an app store", and that was all there was to it, even though Apple sold several orders of magnitude more units, making the concept of an app store way more suitable for them than for the company I was working for.
swoorup 8y ago
Thank you for saying this. Its a personal battle preventing me from working on my own startup trying to imitate the likes of Google, Amazon.
realusername 8y ago
I've seen this so many times unfortunately. Additionally, you don't need a big data architecture when you get 20GB of data/year.
- mmt 8y ago
  > you don't need a big data architecture when you get 20GB of data/year Probably no even if it's 50x that at 1TB/year, unless it's in very small chunks and/or requires an outsized amount of (possibly realtime) processing. Current commodity servers will scale up to 12TiB of main memory, and if you're willing to use previous generation CPUs, those servers can twice the RAM (24TiB). Under half a megabuck without storage.
latch 8y ago
I call this: Headline Driven Development Another good example are monorepos. A decision maker sees a headline about Google and Facebook using monorepos and mandates his company to switch. Unfortunately, he or she didn't read or didn't understand the actual article that explains that it only works because of strict code reviews and extensive integration tests.
- Joky 8y ago
  > Another good example are monorepos. A decision maker sees a headline about Google and Facebook using monorepos and mandates his company to switch. Unfortunately he didn't know that `git` does not scale well with the size of the repo and Google have built their custom repository system to tackle this.
  - tom_ 8y ago
    Google used Perforce, which has a somewhat diferent model from git and scales a lot better. A monorepo is also the obvious way to use it, as you can’t (I don’t think?) branch between depots.
- xab9 8y ago
  We call it Hype Driven Development which is just as bad as Marketing Driven Development... and yes, monorepos, don't even mention it, good heavens. Another hype that keeps coming back is the magic of trunk based development (with a random po reading some flashy article about feature switches as the wonderous solution for a-b testing AND faster development). Nowdays I even consider putting react and angular into this pack, since, you know, "if it's good enough for facebook/google then it must be really good" - anyone ever tried to increase performance for a react site (and realizing that precompiling templates was not exactly a bad idea years ago) or hitting the wall with inline async helpers and rx magic might know my voes. But then again, give me a fast and reliable server rendered site over an unsteable trainwreck spa anytime and I will be happy.
  - blotter_paper 8y ago
    >anyone ever tried to increase performance for a react site (and realizing that precompiling templates was not exactly a bad idea years ago) I'm not sure I'm parsing this sentence correctly. Are you saying that precompiling templates and rehydrating them doesn't cut it anymore? If so, why not? I haven't used React much, but I've done some work in a framework with similar principles and I felt like proper precompiling, with basic HTML fallbacks cooked in where possible, provided all the performance of server rendered sites with the added bells and whistles of SPAs (including that most subsequent requests can load quicker than a full rerender, provided JS is present).
dcow 8y ago
I feel similar sometimes and my next thought is usually, “what part of the system is breaking down such that engineers at small companies don’t understand the scope of their projects?”. Is it a lack of experienced/authoritative technical leadership? Is it bad blood from the rails monolith days? Is it a symptom of a high density of top-tier minds mob-engineering solutions? Are engineers just lazy and prefer the industry status quo framework and some config scripts to the work of actually building software?... Is it even a problem that can be quantified (small company x could be here instead of here if they hadn’t spent resources following the giants)?
- pjmlp 8y ago
  It fashion driven development. Many cannot get to the grips that majority of work is just plumbing, yet another CRUD app or just plain maintenance. So it is cooler to keep pushing for things like those companies, even if there is no need, after all the cv needs to match the new buzzword soup filter from HR.
- amirouche 8y ago
  > Are engineers just lazy and prefer the industry status quo framework and some config scripts to the work of actually building software? It's worse than lazy, it's the new way of doing engineering that is glueing together ready made components.
  - icebraining 8y ago
    Isn't that the method of every mature branch of engineering? Few projects involve inventing new tools, materials or processes in civil or automative engineering, for example.
    - dcow 8y ago
      My point here was not that people shouldn’t reuse good solutions and avoid redundant work if possible. It was that if this approach is “overly” desirable, then we will naturally see solutions deployed that are not 100% tailored to their application. Writing and designing good business is _work_.
ci5er 8y ago
Wet Streets Make Rain(tm). Not the same thing, but the psychology is related. Ref: Michael Crighton; Gell-Mann
_pdp_ 8y ago
Well this is quite bit of generalisation. My business eradicated servers and containers and replaced them with AWS Lambda Functions composed in microservices architecture backed by DynamoDB tables. The result was an order of magnitude reduction in operational complexity, increased stability and reduced cost. This is not fad. This is real.
- lubos 8y ago
  Lucky you. There are very few established businesses which can 100% squeeze themselves into the constrains of serverless architecture.
invaliduser 8y ago
Exactly my feeling. Also, when I hire people coming from the startup industry I have to explain why we don't use kubernetes (we manage 2 servers for our production) or redux (good old publish-and-subscribe works great for us). Some of them just don't get it. Others feel relieved.
- akie 8y ago
  Ok, fair point - but redux is great even if your app is small.
  - invaliduser 8y ago
    Exactly that. Our app is >60kloc of typescript/react, we target android,ios,macos,win32 sharing >85% of the same source code with react native and react web, many abstractions, a lot of local storage and sync with remote server, and so on. Not a small app. Still, we don't use redux, just good ol' publish and subcribe with appropriate separation of concern. My point is there are many ways to manage an app state, redux is one of them, and people can't get to understand how there are many other possible software architectures that provide maintability and scalabity of the source code, and yet does not involve what is seen as the all-incompassing pattern/tool called flux/redux.
- RealDinosaur 8y ago
  I agree, it does sometimes feel that everyone is taking crazy pills. Why did everyone start caring about mutability all of a sudden? I really like MobX ATM. It's a lot easier to explain to new people as well. Redux is like... too much...
  - legulere 8y ago
    People started caring about mutability out of two reasons I would say: - It’s a source of thread-safety bugs. With multiple cores even in phones and async programming rising this has become a bigger issue. - It’s a source of general program logic bugs. I wouldn’t say that it has come all of a sudden. Like other modern techniques it has been standard in functional programming for years. In mainstream programming languages it’s also not totally new, for instance immutable strings were a design decision in java.
zimpenfish 8y ago
> also does not have nearly as many programmers to do it I tried to make this point at the last place because they were insisting that because Amazon et al used microservices, they were obviously the right way. They were planning on 19 microservices to replace the monolith with a dev team of ... 3 perms and 2 contractors.
- smadge 8y ago
  > 19 microservices ... with a dev team of ... 3 perms and 2 contractors. I thought the main benefit of a microservice architecture is to break up an application into smaller services that are about the size that a small development team can build and maintain on their own. I'm not experienced in this area, but more than one microservice per developer seems like a major "architecture smell."
  - _pmf_ 8y ago
    > I'm not experienced in this area, but more than one microservice per developer seems like a major "architecture smell." It's not a smell; it's a hot, steaming turd.
  - zimpenfish 8y ago
    > more than one microservice per developer seems like a major "architecture smell." Yeah, it's normally that you need N developers per microservice (I've seen N=5 recommended, for example.)
    - corpMaverick 8y ago
      The prefix micro in Microservice is very misleading. It is a microservice in the context of a >1,000KLOC monolith. I do like N=5 guideline as a way to make you think twice before making too many little services.
  - repolfx 8y ago
    That's not the main benefit. The reason Google et al split services up into independent servers is for physical reasons. They have large datasets that don't fit on a single machine, so they have to shard the servers that handle it, and they have want to maximise their use of memory and CPU, so running unrelated tasks on the cores is a bad idea - at which point why link in the code? Before you know it you've got "microservices" except they aren't micro, they're machine sized. Somehow this got mangled into "microservices are good because they help teams work together" which isn't what it's about at all. At least at Google the smallest hello world server was something like 50mb when I worked there, just because of the huge dependency graph. It was common for servers to pull in hundreds of modules maintained by nearly as many different teams, all in a single binary. Each server was essentially a "monolith" except for the stuff that had to run in separate backends because of resource constraints.
gigatexal 8y ago
+100 votes to this. My last job didn’t need to go Microservices and Kubernetes. But we did from the beginning. And the overhead of getting the greenfield project off the ground was immense. I kept thinking a aimple albeit inelegant monolith would have helped us prove the concept better and that we could have used it to then refactor once operational. If I take nothing else from my time at that startup I’ll tske this: sometimes you don’t need all the fancy tech that is hyped from the incumbent platforms with a vested interest in getting you onto their platform, sometimes it’s best to role out that tried and true Ruby on Rails or LAMP stack to focus on the product.
- wink 8y ago
  I'm more annoyed by the extremes. Services are awesome (not using the SOA term...) but how about a monolith for most stuff and splitting out a few key things? I guess it's so boring that no one writes about it...
- sieabahlpark 8y ago
  Kubernetes sure makes deployments easy though.
jwillmer 8y ago
I hear this quite often but the case is that if you are looking for financing the main technical concern of the investors is how (and if) your application does scale - talking about SaaS. If a SaaS gets popular the new user load can be quite heavy and in the past there were a lot of story's about companies that needed to rebuild there system because of this (I think pipedrive was the last example I heard of). Although it makes sense to ignore scaling at the beginning you need to design your application in a way that you can easily transition to something that can scale.
- mmt 8y ago
  > the main technical concern of the investors is how (and if) your application does scale How often is this borne of ignorance, though? > in the past there were a lot of story's about companies that needed to rebuild there system More specifically, are investors merely remembering the struggles of the dot-com boom, when transistor density was one thousandth that of today? Internet usage (number of users and number of services each one uses) has grown, too, but I haven't seen statistics that suggest this would be more than 100x of 20 years ago. > Although it makes sense to ignore scaling at the beginning you need to design your application in a way that you can easily transition to something that can scale. This presents a false dichotomy. One doesn't have to ignore scaling at the beginning to make the decision to trade initial time-to-market for eventual "horizontal" scalability (usually what is meant by "can scale" [1]), but that may be what happens in practice. [1] even though one can buy a remarkably large single server 3-6x price premiums over mid-range single servers, as well as use other, more traditional techniques, before "needing" to rebuild the entire system.
  - jwillmer 8y ago
    The investors I know and have to work with are mostly no technicians. They scan the news for investments and reported problems in growing business. If you can convince them that you thought about scaling issues and know how to solve them then they come up you should be fine :)
    - mmt 8y ago

_pmf_ 8y ago

Adding to 1): more often than not, the "scaling" issues are 95% social and 5% technical.

GiorgioG 8y ago

Kubernetes comes to mind.

mmt 8y ago

> also does not have nearly as many programmers to do it. Nor the underlying infrastructure, such as, for example, physical server hardware that's a fraction of the cost of cloud.

pmiller2 8y ago

This doesn’t only apply to technology. It applies to hiring and other organizational processes as well.

SideburnsOfDoom 8y ago
e.g. "the spotify model"
rossdavidh 8y ago
Rings true. I once worked a contract at a place with proprietary hardware, that wanted their software extras to be purchased separately, and accessible through an "app store". This seemed to imply they thought somebody else would develop apps for their store, which was odd because if their proprietary hardware (which was quite expensive, justifiably so) sold 1000 units it would be considered successful. Basically nobody made apps for their app store except the people who worked for the same company that made the hardware, and I had a hard time understanding why they didn't use the (much simpler) model of "if you pay us $X more we'll throw in this bit of software". Instead, they had to develop all of the software for running and maintaining an app store, including the issues of making sure that apps uploaded to their store were not malware. Then, I realized what was going on: Apple had introduced their app store a few years before, and it was a big deal. Steve Jobs was the CEO on the cover of every magazine. The CEO looked at Apple, and said, "we must have an app store", and that was all there was to it, even though Apple sold several orders of magnitude more units, making the concept of an app store way more suitable for them than for the company I was working for.

swoorup 8y ago

Thank you for saying this. Its a personal battle preventing me from working on my own startup trying to imitate the likes of Google, Amazon.

realusername 8y ago

I've seen this so many times unfortunately. Additionally, you don't need a big data architecture when you get 20GB of data/year.

mmt 8y ago
> you don't need a big data architecture when you get 20GB of data/year Probably no even if it's 50x that at 1TB/year, unless it's in very small chunks and/or requires an outsized amount of (possibly realtime) processing. Current commodity servers will scale up to 12TiB of main memory, and if you're willing to use previous generation CPUs, those servers can twice the RAM (24TiB). Under half a megabuck without storage.

latch 8y ago

I call this: Headline Driven Development Another good example are monorepos. A decision maker sees a headline about Google and Facebook using monorepos and mandates his company to switch. Unfortunately, he or she didn't read or didn't understand the actual article that explains that it only works because of strict code reviews and extensive integration tests.

Joky 8y ago
> Another good example are monorepos. A decision maker sees a headline about Google and Facebook using monorepos and mandates his company to switch. Unfortunately he didn't know that `git` does not scale well with the size of the repo and Google have built their custom repository system to tackle this.
- tom_ 8y ago
  Google used Perforce, which has a somewhat diferent model from git and scales a lot better. A monorepo is also the obvious way to use it, as you can’t (I don’t think?) branch between depots.
xab9 8y ago
We call it Hype Driven Development which is just as bad as Marketing Driven Development... and yes, monorepos, don't even mention it, good heavens. Another hype that keeps coming back is the magic of trunk based development (with a random po reading some flashy article about feature switches as the wonderous solution for a-b testing AND faster development). Nowdays I even consider putting react and angular into this pack, since, you know, "if it's good enough for facebook/google then it must be really good" - anyone ever tried to increase performance for a react site (and realizing that precompiling templates was not exactly a bad idea years ago) or hitting the wall with inline async helpers and rx magic might know my voes. But then again, give me a fast and reliable server rendered site over an unsteable trainwreck spa anytime and I will be happy.
- blotter_paper 8y ago
  >anyone ever tried to increase performance for a react site (and realizing that precompiling templates was not exactly a bad idea years ago) I'm not sure I'm parsing this sentence correctly. Are you saying that precompiling templates and rehydrating them doesn't cut it anymore? If so, why not? I haven't used React much, but I've done some work in a framework with similar principles and I felt like proper precompiling, with basic HTML fallbacks cooked in where possible, provided all the performance of server rendered sites with the added bells and whistles of SPAs (including that most subsequent requests can load quicker than a full rerender, provided JS is present).

dcow 8y ago

I feel similar sometimes and my next thought is usually, “what part of the system is breaking down such that engineers at small companies don’t understand the scope of their projects?”. Is it a lack of experienced/authoritative technical leadership? Is it bad blood from the rails monolith days? Is it a symptom of a high density of top-tier minds mob-engineering solutions? Are engineers just lazy and prefer the industry status quo framework and some config scripts to the work of actually building software?... Is it even a problem that can be quantified (small company x could be here instead of here if they hadn’t spent resources following the giants)?

pjmlp 8y ago
It fashion driven development. Many cannot get to the grips that majority of work is just plumbing, yet another CRUD app or just plain maintenance. So it is cooler to keep pushing for things like those companies, even if there is no need, after all the cv needs to match the new buzzword soup filter from HR.
amirouche 8y ago
> Are engineers just lazy and prefer the industry status quo framework and some config scripts to the work of actually building software? It's worse than lazy, it's the new way of doing engineering that is glueing together ready made components.
- icebraining 8y ago
  Isn't that the method of every mature branch of engineering? Few projects involve inventing new tools, materials or processes in civil or automative engineering, for example.
  - dcow 8y ago
    My point here was not that people shouldn’t reuse good solutions and avoid redundant work if possible. It was that if this approach is “overly” desirable, then we will naturally see solutions deployed that are not 100% tailored to their application. Writing and designing good business is _work_.

ci5er 8y ago

Wet Streets Make Rain(tm). Not the same thing, but the psychology is related. Ref: Michael Crighton; Gell-Mann

_pdp_ 8y ago

Well this is quite bit of generalisation. My business eradicated servers and containers and replaced them with AWS Lambda Functions composed in microservices architecture backed by DynamoDB tables. The result was an order of magnitude reduction in operational complexity, increased stability and reduced cost. This is not fad. This is real.

lubos 8y ago
Lucky you. There are very few established businesses which can 100% squeeze themselves into the constrains of serverless architecture.

invaliduser 8y ago

Exactly my feeling. Also, when I hire people coming from the startup industry I have to explain why we don't use kubernetes (we manage 2 servers for our production) or redux (good old publish-and-subscribe works great for us). Some of them just don't get it. Others feel relieved.

akie 8y ago
Ok, fair point - but redux is great even if your app is small.
- invaliduser 8y ago
  Exactly that. Our app is >60kloc of typescript/react, we target android,ios,macos,win32 sharing >85% of the same source code with react native and react web, many abstractions, a lot of local storage and sync with remote server, and so on. Not a small app. Still, we don't use redux, just good ol' publish and subcribe with appropriate separation of concern. My point is there are many ways to manage an app state, redux is one of them, and people can't get to understand how there are many other possible software architectures that provide maintability and scalabity of the source code, and yet does not involve what is seen as the all-incompassing pattern/tool called flux/redux.
RealDinosaur 8y ago
I agree, it does sometimes feel that everyone is taking crazy pills. Why did everyone start caring about mutability all of a sudden? I really like MobX ATM. It's a lot easier to explain to new people as well. Redux is like... too much...
- legulere 8y ago
  People started caring about mutability out of two reasons I would say: - It’s a source of thread-safety bugs. With multiple cores even in phones and async programming rising this has become a bigger issue. - It’s a source of general program logic bugs. I wouldn’t say that it has come all of a sudden. Like other modern techniques it has been standard in functional programming for years. In mainstream programming languages it’s also not totally new, for instance immutable strings were a design decision in java.

zimpenfish 8y ago

> also does not have nearly as many programmers to do it I tried to make this point at the last place because they were insisting that because Amazon et al used microservices, they were obviously the right way. They were planning on 19 microservices to replace the monolith with a dev team of ... 3 perms and 2 contractors.

smadge 8y ago
> 19 microservices ... with a dev team of ... 3 perms and 2 contractors. I thought the main benefit of a microservice architecture is to break up an application into smaller services that are about the size that a small development team can build and maintain on their own. I'm not experienced in this area, but more than one microservice per developer seems like a major "architecture smell."
- _pmf_ 8y ago
  > I'm not experienced in this area, but more than one microservice per developer seems like a major "architecture smell." It's not a smell; it's a hot, steaming turd.
- zimpenfish 8y ago
  > more than one microservice per developer seems like a major "architecture smell." Yeah, it's normally that you need N developers per microservice (I've seen N=5 recommended, for example.)
  - corpMaverick 8y ago
    The prefix micro in Microservice is very misleading. It is a microservice in the context of a >1,000KLOC monolith. I do like N=5 guideline as a way to make you think twice before making too many little services.
- repolfx 8y ago
  That's not the main benefit. The reason Google et al split services up into independent servers is for physical reasons. They have large datasets that don't fit on a single machine, so they have to shard the servers that handle it, and they have want to maximise their use of memory and CPU, so running unrelated tasks on the cores is a bad idea - at which point why link in the code? Before you know it you've got "microservices" except they aren't micro, they're machine sized. Somehow this got mangled into "microservices are good because they help teams work together" which isn't what it's about at all. At least at Google the smallest hello world server was something like 50mb when I worked there, just because of the huge dependency graph. It was common for servers to pull in hundreds of modules maintained by nearly as many different teams, all in a single binary. Each server was essentially a "monolith" except for the stuff that had to run in separate backends because of resource constraints.

gigatexal 8y ago

+100 votes to this. My last job didn’t need to go Microservices and Kubernetes. But we did from the beginning. And the overhead of getting the greenfield project off the ground was immense. I kept thinking a aimple albeit inelegant monolith would have helped us prove the concept better and that we could have used it to then refactor once operational. If I take nothing else from my time at that startup I’ll tske this: sometimes you don’t need all the fancy tech that is hyped from the incumbent platforms with a vested interest in getting you onto their platform, sometimes it’s best to role out that tried and true Ruby on Rails or LAMP stack to focus on the product.

wink 8y ago
I'm more annoyed by the extremes. Services are awesome (not using the SOA term...) but how about a monolith for most stuff and splitting out a few key things? I guess it's so boring that no one writes about it...
sieabahlpark 8y ago
Kubernetes sure makes deployments easy though.

jwillmer 8y ago

I hear this quite often but the case is that if you are looking for financing the main technical concern of the investors is how (and if) your application does scale - talking about SaaS. If a SaaS gets popular the new user load can be quite heavy and in the past there were a lot of story's about companies that needed to rebuild there system because of this (I think pipedrive was the last example I heard of). Although it makes sense to ignore scaling at the beginning you need to design your application in a way that you can easily transition to something that can scale.

mmt 8y ago
> the main technical concern of the investors is how (and if) your application does scale How often is this borne of ignorance, though? > in the past there were a lot of story's about companies that needed to rebuild there system More specifically, are investors merely remembering the struggles of the dot-com boom, when transistor density was one thousandth that of today? Internet usage (number of users and number of services each one uses) has grown, too, but I haven't seen statistics that suggest this would be more than 100x of 20 years ago. > Although it makes sense to ignore scaling at the beginning you need to design your application in a way that you can easily transition to something that can scale. This presents a false dichotomy. One doesn't have to ignore scaling at the beginning to make the decision to trade initial time-to-market for eventual "horizontal" scalability (usually what is meant by "can scale" [1]), but that may be what happens in practice. [1] even though one can buy a remarkably large single server 3-6x price premiums over mid-range single servers, as well as use other, more traditional techniques, before "needing" to rebuild the entire system.
- jwillmer 8y ago
  The investors I know and have to work with are mostly no technicians. They scan the news for investments and reported problems in growing business. If you can convince them that you thought about scaling issues and know how to solve them then they come up you should be fine :)
  - mmt 8y ago