5 ms·

The one thing I see in my current company, and a growing trend with SaaS apps is that companies are forgetting how to actually engineer. Like Boeing- the more y

by lykr0n 6y ago

The one thing I see in my current company, and a growing trend with SaaS apps is that companies are forgetting how to actually engineer. Like Boeing- the more you outsource the less able you're able to react to changing market forces and fix issues. We run Hadoop & Spark internally, but the team is underfunded and stuck in a constant cycle of fighting fires. And the result (and part of a larger push of the company due to the same cycle of under-funding and culture issues) is that we're moving our petabytes of data to cloud providers into their systems. Not only is the cost of doing this dwarfing that it would take to actually fix our issues, but we're going to lose the people who know how to design and manage petabyte scale hadoop clusters. We wind up in a situation where we locked up data fundamental to our company and our position in the market with a 3rd party, and losing the talent that would allow us to maintain full control over the data. If the service increases prices, changes it's offering, or we get to a point where the offering doesn't meet our needs- we're fucked. It's nice that Databricks has a nice "offramp" that you can take to go somewhere else, but the general idea is the same.

random314 6y ago
This happens with every technology stack from steam engines to software. There was a time when programmers could solder together an ALU using transistor gates.
- doctor_eval 6y ago
  Well yeah I could probably have done that with some 7400 series TTL ... but the MIPS wouldn't nearly cut it in today's world :)
- tluyben2 6y ago
  It might be the same with 'trains' (steam engines); I don't know much about that. But losing skills in 'the cloud' makes you prisoner to PaaS/SaaS providers; if these skills almost completely disappear (let's say, akin to your transistor gates, that almost no-one can install or maintain server setups anymore), the cloud providers really can do whatever they want price wise. For instance, if a cloud provider misconfigures something at scale in this great future, there is no way for anyone to verify that; it might be that you are paying 1000x what you should be paying for a particular database query that triggers a perfect storm in your set-up. As no-one in your company knows and no-one can check what or even if this is going on, it'll be considered 'normal' and you just pay. This is already happening, at scale, but not due to the cloud provider's fault, but because people are already not learning skills. If I would have $100 for every missing index I find per month in client's setups, I would be working a few hours/month for a very nice wage. Heck, most startup databases I see these days don't have any indexes at all; and then they wonder why they are paying $10k/month at aws while I can get them the same or better performance for a few bucks/month on a vps. Not because I'm so smart, but just doing everything 'low-level' wrong because no-one simply knows anymore how a database works (it's the cloud! we shouldn't have to!). It's not restricted to relational dbs either; sometimes I cry when I see dynamodb setups made by people who think this is just the cloud way of working; 'throw in whatever data, it'll always optimise itself'. So this is already happening at many companies and it's getting worse.
  - random314 6y ago
    Similar thing happened in the past when application developers (most apps were written assembly in the past) built a dependency on instruction sets. If the computer manufacturer underperforms in future years, you are stuck. This dynamic can never be erased even as we keep climbing higher on the stack.
foobiekr 6y ago
As you say, this de-skilling problem is broader than SaaS, it is tied to oursourcing of critical competencies or, in younger companies, never having them in the first place. If you want to see de-skilling in action, hard core, go look into the service providers, wireless and fixed. They are running on fumes and attrition-victorious teams that last had a new technology in the late 70s/early 80s because everyone good at networking went to the FANG predecessors and then FANG proper.
- mlthoughts2018 6y ago
  It’s also a problem with compensation. Many companies take an incredibly stupid and paternalistic attitude that employees should be happy with 3% - 5% raises, no bonuses, and refresher equity way less than their original grant. Anyone actually talented just leaves after 2-4 years, so you get zero cohesion or snowballing effect of talented tech leaders who other good people actually want to work with. These places become total career death. Eventually it becomes a place where only people with kids or other significant life obligation go, specifically because they value no-ambition bureaucracy that trades off compensation and risk taking in exchange for work life balance. Those places are so soul crushing. Avoid at all costs.
lumost 6y ago
It's incredibly difficult to fund internal platform teams appropriately. Usually one of three failure patterns emerges 1) The team is competent but picks up migration work to arbitrary technologies, and approaches with no clear ROI. These migrations block feature development and never seem to end e.g. teams ceaselessly migrating from GCP to AWS, to kubernetes, to podman from mysql to postgresql etc. 2) The team is operationally heavy and generates arbitrary requirements for everyone else to follow. The toolchain seems to get worse over time, and the number of hoops to jump through to get anything done endlessly grows e.g. Wait 2 weeks and get three business approvals for a server which you aren't allowed to have root access to. 3) The team has big ideas, but the business constantly under invests. The team is called to fight every fire but unable to stop the fires through any meaningful project. The company ends up on a platform thats constantly on fire. When weighing these execution risks building an internal platform for just about anything looks incredibly expensive. I've only been at 1 company out of 6 which nailed the internal platform tooling requirements. The only reason I can attribute to their success was through quarterly NPS surveys on the developer experience for every major piece of the companies toolchain + hard to meet SLAs for uptime.
- Spivak 6y ago
  I think you need to also include the reality that you graduate to 2 to stop the bleeding from 1 and 3. I’ve been at companies where we have had literally no choice but to implement 2 because the ops team was drowning due to every part of the business demanding to be priority 1 and not being able to actually reason about prod because devs would manually make changes. The underlying symptom is bad project management and more work than the team can handle but convincing the business that the ops team actually needs to be twice the size and have dedicated managers falls on deaf ears.
david38 6y ago
Using third party tools doesn’t lock you out of them. Nothing stops you from the collecting that data before sending it off. Every company outsources something fundamental. Does Google mine its own metal? Generate its own electricity? Even if you did it on prem, that doesn’t save you from license renewal costs or upgrades. You can write the software yourself, but that’s not cheap either.
- paulryanrogers 6y ago
  Is metal and electricity Google's core competency though? I don't think anyone is arguing one should maintain their own silica, atoms, independent universe, etc .
  - mr_toad 6y ago
    Using Hadoop doesn’t mean your core competency is maintaining Hadoop. Analysts don’t want to be engineers.
  - B1FF_PSUVM 6y ago
    > independent universe, Solipsystem, Inc. begs to differ. Having it your way is a booming business nowadays. (j/k, borrowed an SF title https://en.wikipedia.org/wiki/Solip:System https://en.wikipedia.org/wiki/Solip:System )
hodgesrm 6y ago
> we're going to lose the people who know how to design and manage petabyte scale hadoop clusters. Why is that different from "lose the people who know how to design and write accounting systems from scratch?" That was what happened when packaged accounting systems showed up. I'm not sure why you would want to preserve knowledge of Hadoop if other technologies are more efficient.
- sombremesa 6y ago
  This is a good point. By the same token, most of us have no idea how to milk our own cows or forge our own knives. Having businesses dedicated to one craft and providing their services to those who'd rather be doing other things seems like a natural model humans - and nature itself - have been following for millenia. It's not like stem cells worry about losing the ability to excrete bile when they become neurons.
tluyben2 6y ago
Another thing I noticed that adds to this point; the investors (well their tech DD people) seem to really push this. I have seen more and more teams not advancing to the next round because they wanted (with good reason imho) to engineer their own tech than 'just go with aws' blindly. AWS costing literally over 10x (that's lowballing, more often 20-30x and sometimes 100x) as much. As a startup, that's not always the best choice imho, but I see people getting forced into it as investors (again, their tech dd peeps) reason that it scales better than people (it does, but that's usually not needed at all at seed and even far beyond for many types of businesses) but in that, they lose sight of that these skilled people might not join at all because the money you could've paid them in the early days to get/keep them on board, now disappears into the cloud. It seems most investor's (here in the EU at least) dream to run companies with only MBA's and this is a solid step in that direction.

by lykr0n 6y ago

random314 6y ago

This happens with every technology stack from steam engines to software. There was a time when programmers could solder together an ALU using transistor gates.

doctor_eval 6y ago
Well yeah I could probably have done that with some 7400 series TTL ... but the MIPS wouldn't nearly cut it in today's world :)
tluyben2 6y ago
It might be the same with 'trains' (steam engines); I don't know much about that. But losing skills in 'the cloud' makes you prisoner to PaaS/SaaS providers; if these skills almost completely disappear (let's say, akin to your transistor gates, that almost no-one can install or maintain server setups anymore), the cloud providers really can do whatever they want price wise. For instance, if a cloud provider misconfigures something at scale in this great future, there is no way for anyone to verify that; it might be that you are paying 1000x what you should be paying for a particular database query that triggers a perfect storm in your set-up. As no-one in your company knows and no-one can check what or even if this is going on, it'll be considered 'normal' and you just pay. This is already happening, at scale, but not due to the cloud provider's fault, but because people are already not learning skills. If I would have $100 for every missing index I find per month in client's setups, I would be working a few hours/month for a very nice wage. Heck, most startup databases I see these days don't have any indexes at all; and then they wonder why they are paying $10k/month at aws while I can get them the same or better performance for a few bucks/month on a vps. Not because I'm so smart, but just doing everything 'low-level' wrong because no-one simply knows anymore how a database works (it's the cloud! we shouldn't have to!). It's not restricted to relational dbs either; sometimes I cry when I see dynamodb setups made by people who think this is just the cloud way of working; 'throw in whatever data, it'll always optimise itself'. So this is already happening at many companies and it's getting worse.
- random314 6y ago
  Similar thing happened in the past when application developers (most apps were written assembly in the past) built a dependency on instruction sets. If the computer manufacturer underperforms in future years, you are stuck. This dynamic can never be erased even as we keep climbing higher on the stack.

foobiekr 6y ago

As you say, this de-skilling problem is broader than SaaS, it is tied to oursourcing of critical competencies or, in younger companies, never having them in the first place. If you want to see de-skilling in action, hard core, go look into the service providers, wireless and fixed. They are running on fumes and attrition-victorious teams that last had a new technology in the late 70s/early 80s because everyone good at networking went to the FANG predecessors and then FANG proper.

mlthoughts2018 6y ago
It’s also a problem with compensation. Many companies take an incredibly stupid and paternalistic attitude that employees should be happy with 3% - 5% raises, no bonuses, and refresher equity way less than their original grant. Anyone actually talented just leaves after 2-4 years, so you get zero cohesion or snowballing effect of talented tech leaders who other good people actually want to work with. These places become total career death. Eventually it becomes a place where only people with kids or other significant life obligation go, specifically because they value no-ambition bureaucracy that trades off compensation and risk taking in exchange for work life balance. Those places are so soul crushing. Avoid at all costs.

lumost 6y ago

It's incredibly difficult to fund internal platform teams appropriately. Usually one of three failure patterns emerges 1) The team is competent but picks up migration work to arbitrary technologies, and approaches with no clear ROI. These migrations block feature development and never seem to end e.g. teams ceaselessly migrating from GCP to AWS, to kubernetes, to podman from mysql to postgresql etc. 2) The team is operationally heavy and generates arbitrary requirements for everyone else to follow. The toolchain seems to get worse over time, and the number of hoops to jump through to get anything done endlessly grows e.g. Wait 2 weeks and get three business approvals for a server which you aren't allowed to have root access to. 3) The team has big ideas, but the business constantly under invests. The team is called to fight every fire but unable to stop the fires through any meaningful project. The company ends up on a platform thats constantly on fire. When weighing these execution risks building an internal platform for just about anything looks incredibly expensive. I've only been at 1 company out of 6 which nailed the internal platform tooling requirements. The only reason I can attribute to their success was through quarterly NPS surveys on the developer experience for every major piece of the companies toolchain + hard to meet SLAs for uptime.

Spivak 6y ago
I think you need to also include the reality that you graduate to 2 to stop the bleeding from 1 and 3. I’ve been at companies where we have had literally no choice but to implement 2 because the ops team was drowning due to every part of the business demanding to be priority 1 and not being able to actually reason about prod because devs would manually make changes. The underlying symptom is bad project management and more work than the team can handle but convincing the business that the ops team actually needs to be twice the size and have dedicated managers falls on deaf ears.

david38 6y ago

Using third party tools doesn’t lock you out of them. Nothing stops you from the collecting that data before sending it off. Every company outsources something fundamental. Does Google mine its own metal? Generate its own electricity? Even if you did it on prem, that doesn’t save you from license renewal costs or upgrades. You can write the software yourself, but that’s not cheap either.

paulryanrogers 6y ago
Is metal and electricity Google's core competency though? I don't think anyone is arguing one should maintain their own silica, atoms, independent universe, etc .
- mr_toad 6y ago
  Using Hadoop doesn’t mean your core competency is maintaining Hadoop. Analysts don’t want to be engineers.
- B1FF_PSUVM 6y ago
  > independent universe, Solipsystem, Inc. begs to differ. Having it your way is a booming business nowadays. (j/k, borrowed an SF title https://en.wikipedia.org/wiki/Solip:System https://en.wikipedia.org/wiki/Solip:System )

hodgesrm 6y ago

> we're going to lose the people who know how to design and manage petabyte scale hadoop clusters. Why is that different from "lose the people who know how to design and write accounting systems from scratch?" That was what happened when packaged accounting systems showed up. I'm not sure why you would want to preserve knowledge of Hadoop if other technologies are more efficient.

sombremesa 6y ago
This is a good point. By the same token, most of us have no idea how to milk our own cows or forge our own knives. Having businesses dedicated to one craft and providing their services to those who'd rather be doing other things seems like a natural model humans - and nature itself - have been following for millenia. It's not like stem cells worry about losing the ability to excrete bile when they become neurons.

tluyben2 6y ago

Another thing I noticed that adds to this point; the investors (well their tech DD people) seem to really push this. I have seen more and more teams not advancing to the next round because they wanted (with good reason imho) to engineer their own tech than 'just go with aws' blindly. AWS costing literally over 10x (that's lowballing, more often 20-30x and sometimes 100x) as much. As a startup, that's not always the best choice imho, but I see people getting forced into it as investors (again, their tech dd peeps) reason that it scales better than people (it does, but that's usually not needed at all at seed and even far beyond for many types of businesses) but in that, they lose sight of that these skilled people might not join at all because the money you could've paid them in the early days to get/keep them on board, now disappears into the cloud. It seems most investor's (here in the EU at least) dream to run companies with only MBA's and this is a solid step in that direction.