4 ms·

When I read something like this I realise just how immense of a lead tech companies are building vs 'normal' companies. There's just layer of expertise in how t

When I read something like this I realise just how immense of a lead tech companies are building vs 'normal' companies. There's just layer of expertise in how to streamline and organise themselves that anyone without a 1000 strong engineering team and huge engineering budget just are never going to think about.

ramraj07 6y ago
Ill argue that you don't need it for a small company - just don't overcomplicate your life by using technology that's designed for big companies. Stick to simple systems like elastic beanstalk or app engine + RDS and something like snowflake and you get powerful, manageable, auditable projects. I'm also finding great benefit in launching each project in its own AWS account. This means you know exactly what project costs how much!
- jonplackett 6y ago
  I was thinking more in terms of large but traditional companies. The kind tech companies put out of business regularly!
QUFB 6y ago
This is where things are heading, especially for business or corporate 'consumers': Need web hosting? Certainly you can't deal with a 2Tbps DDoS attack, so let's route that traffic through Cloudflare. Sending email? Can't send it out with those /23s you have, they're on some blacklist from a shady DNSBL provider. Need to pay for GSuite or Office365. Can't protect your CEO against a targeted attack? GSuite Advanced Protection or Microsoft ATP has you covered.
- organsnyder 6y ago
  Some of those have been true for quite some time. I ran my own mail server back in the 2000s to early 2010s, and incoming mail was especially hellish (effective spam filtering was basically impossible). These are really just variations on the theme of economies of scale, aren't they?
  - johnthescott 6y ago
    commercial dns black lists can be very effective at filtering spam and not expensive for personal use.
hibikir 6y ago
Having worked at some of those companies, along with small startups, the biggest difference isn't really plain old sophistication, but expenses so high as to make the engineering effort worthwhile. If your production system is 10 instances on some random cloud, a 10% efficiency savings saves me 1 instance, so maybe $2k a year. Taking into account opportunity costs vs doing things that raise revenue, said startup would consider the effort a waste of time unless it took a few hours. With the same architecture, but instead 20K instances, then suddenly that 10% is saving 2000 instances, and 4 million a year. Unless there's a major engineering shortage, chances are that spending a over a month on 4 million in yearly savings will be completely justified, and would even be a highlight in someone SRE's review. Chances are that the optimization wasn't even any harder in the big tech company: It's just that small savings on big piles of money are suddenly worth it. It's not possible to find millions of dollars in loose change between the proverbial couch cushions if you didn't have the chance to spend hundreds of millions in the first place. Heck, in a growth company, even at that size, 4 million might not be enough savings, as there might be even bigger things. Same with dev tooling: Saving a company 5% CI times is not great with 10 developers, but if you are making five thousand developers more productive, suddenly you can hire an entire team of very specialized developers, and it's not a luxury. This is also why often copying what large companies is foolish when you are small: The tradeoffs are going to be completely different. What would be an unacceptable flaw in a large company is just fine in your startup. The real trick IMO is the 200-800 range: Large enough that the simple solutions for small companies have probably broken down and are causing pain, but nowhere near the staffing to hire yourself a team to, say, add types to Ruby, get a team to build around the weaknesses of your database, or whichever other problem you have that any member of FAANG would just throw 10 million dollars in staffing costs without batting an eye. I've seen way too many companies that stopped being able to grow their company at those intermediate sizes, and get stuck in technical hell.
- kabes 6y ago
  The problem is that managers at small companies with some, but insufficient, technical knowledge also read these articles and think they need to do the same. I come across overengineereed stuff at small companies all the time for things they don't need but only do because it's what Google/Facebook is doing.
jacques_chester 6y ago
If it makes you feel better, normal companies have "cost accountants", whose entire job is to worry about this kind of problem. Cost accounting a fairly well-developed field with a long history. There are several major approaches and some less well known ones. I think the most applicable in our field is "resource cost accounting", which looks superficially like what Netflix described. You keep "resource accounts" in the units of the actual stuff consumed by your business processes, then map backwards from the final product to costs. Of course I am not an accountant. Just took a course and found it enlightening.
- jonplackett 6y ago
  I think the difference is that Netflix sees these problems and has the person-power to solve them and make themselves leaner. I guess it comes down to how far ahead you're looking really and tech companies are incentivised to look further ahead than your average big company. They're all planning for world domination (whether that actually happens or not)
  - jacques_chester 6y ago
    I don't disagree, I'm just trying to convey that in some sense the forward-looking effort would be accelerated by looking sideways as well. Cost accounting is practiced widely in many industries and appears in medium-sized businesses (tens of employees) as well as large businesses (tens of thousands of employees).

ramraj07 6y ago

Ill argue that you don't need it for a small company - just don't overcomplicate your life by using technology that's designed for big companies. Stick to simple systems like elastic beanstalk or app engine + RDS and something like snowflake and you get powerful, manageable, auditable projects. I'm also finding great benefit in launching each project in its own AWS account. This means you know exactly what project costs how much!

jonplackett 6y ago
I was thinking more in terms of large but traditional companies. The kind tech companies put out of business regularly!

QUFB 6y ago

This is where things are heading, especially for business or corporate 'consumers': Need web hosting? Certainly you can't deal with a 2Tbps DDoS attack, so let's route that traffic through Cloudflare. Sending email? Can't send it out with those /23s you have, they're on some blacklist from a shady DNSBL provider. Need to pay for GSuite or Office365. Can't protect your CEO against a targeted attack? GSuite Advanced Protection or Microsoft ATP has you covered.

organsnyder 6y ago
Some of those have been true for quite some time. I ran my own mail server back in the 2000s to early 2010s, and incoming mail was especially hellish (effective spam filtering was basically impossible). These are really just variations on the theme of economies of scale, aren't they?
- johnthescott 6y ago
  commercial dns black lists can be very effective at filtering spam and not expensive for personal use.

hibikir 6y ago

Having worked at some of those companies, along with small startups, the biggest difference isn't really plain old sophistication, but expenses so high as to make the engineering effort worthwhile. If your production system is 10 instances on some random cloud, a 10% efficiency savings saves me 1 instance, so maybe $2k a year. Taking into account opportunity costs vs doing things that raise revenue, said startup would consider the effort a waste of time unless it took a few hours. With the same architecture, but instead 20K instances, then suddenly that 10% is saving 2000 instances, and 4 million a year. Unless there's a major engineering shortage, chances are that spending a over a month on 4 million in yearly savings will be completely justified, and would even be a highlight in someone SRE's review. Chances are that the optimization wasn't even any harder in the big tech company: It's just that small savings on big piles of money are suddenly worth it. It's not possible to find millions of dollars in loose change between the proverbial couch cushions if you didn't have the chance to spend hundreds of millions in the first place. Heck, in a growth company, even at that size, 4 million might not be enough savings, as there might be even bigger things. Same with dev tooling: Saving a company 5% CI times is not great with 10 developers, but if you are making five thousand developers more productive, suddenly you can hire an entire team of very specialized developers, and it's not a luxury. This is also why often copying what large companies is foolish when you are small: The tradeoffs are going to be completely different. What would be an unacceptable flaw in a large company is just fine in your startup. The real trick IMO is the 200-800 range: Large enough that the simple solutions for small companies have probably broken down and are causing pain, but nowhere near the staffing to hire yourself a team to, say, add types to Ruby, get a team to build around the weaknesses of your database, or whichever other problem you have that any member of FAANG would just throw 10 million dollars in staffing costs without batting an eye. I've seen way too many companies that stopped being able to grow their company at those intermediate sizes, and get stuck in technical hell.

kabes 6y ago
The problem is that managers at small companies with some, but insufficient, technical knowledge also read these articles and think they need to do the same. I come across overengineereed stuff at small companies all the time for things they don't need but only do because it's what Google/Facebook is doing.

jacques_chester 6y ago

If it makes you feel better, normal companies have "cost accountants", whose entire job is to worry about this kind of problem. Cost accounting a fairly well-developed field with a long history. There are several major approaches and some less well known ones. I think the most applicable in our field is "resource cost accounting", which looks superficially like what Netflix described. You keep "resource accounts" in the units of the actual stuff consumed by your business processes, then map backwards from the final product to costs. Of course I am not an accountant. Just took a course and found it enlightening.

jonplackett 6y ago
I think the difference is that Netflix sees these problems and has the person-power to solve them and make themselves leaner. I guess it comes down to how far ahead you're looking really and tech companies are incentivised to look further ahead than your average big company. They're all planning for world domination (whether that actually happens or not)
- jacques_chester 6y ago
  I don't disagree, I'm just trying to convey that in some sense the forward-looking effort would be accelerated by looking sideways as well. Cost accounting is practiced widely in many industries and appears in medium-sized businesses (tens of employees) as well as large businesses (tens of thousands of employees).