15 ms·

The many lies about reducing complexity part 2: Cloud

224 points by rapnie 6y ago

deleted 6y ago
[deleted]
ChicagoDave 6y ago
Reducing complexity should never be about platform (on-prem vs cloud). It should be about constructing software in partnership with the business and reducing complexity with modeled boundaries. You can leverage the cloud to do some interesting things, but the true benefit in is _what_ you construct, not _how_.
- sanp 6y ago
  There is an element of _how_ as well. You could create simple monoliths or overengineered microservices. Or, complex monoliths with heavy coupling vs cleanly designed microservices with clear separations of concern.
  - the-smug-one 6y ago
    Are microservices meant to separate data too? As in, each service has its own database. Wouldn't that lead to non-normalisation of the data or a lot of expensive network lookups to get what I want/need? What is the point of micro services anyway :-)?
    - kqr 6y ago
      > Are microservices meant to separate data too? As in, each service has its own database. Yes. > Wouldn't that lead to non-normalisation of the data Yes. But it's not as bad as it sounds. That is how data on paper used to work, after all. Business rules (at least ones that have been around for more than 5--10 years) are written with intensely non-normalised data in mind. Business people tend to be fine with eventual consistency on the scale of hours or even days. Non-normalised data also makes total data corruption harder, and forensics in the case of bugs easier, in some ways: you find an unexpected value somewhere? Check the other versions that ought to exist and you can probably retrace at what point it got weird. The whole idea of consistent and fully normalised data is a, historically speaking, very recent innovation, and I'm not convinced it will last long in the real world. I think this is a brief moment in history when our software is primitive enough, yet optimistic enough, to even consider that type of data storage. And come on, it's not like the complete consistency of the data is worth that many dollars in most cases, if we actually bother to compute the cost.
    - ratww 6y ago
      > Are microservices meant to separate data too? As in, each service has its own database. Ideally yes, to scale. Sometimes you have a service with obvious and easy-to-split boundaries, and microservices are a breeze. Some things that are easy to turn into microservices: "API Wrapper" to a complex and messy third-party API. Logging and data collection. Sending emails/messages. User authentication. Search. Anything in your app that could become another app. However, when your data model is tightly coupled and you need to choose between tradeoffs (data duplication), having bigger services, or even keeping it as a monolith. Btw, if you don't care about scalability, sharing a database is still still not the best idea. But you can have a microservice that wraps the database in a service, for example. Tools like Hasura can be used for that.
    - NicoJuicy 6y ago
      Microservices is a solution for an organisational problem ( multiple employees in one project), not a technical one. If you're flying solo, just use DDD for example. It will give you the same patterns without the devops complexity
    - ChicagoDave 6y ago
      There's a progression ever developer that grew up on SQL Server/relational data needs to go through... 1. The models (plural intended) of a business are not necessarily a relational. There could be an upstream/downstream relationship. There could be an event-based relationship. There may be no relations at all (documents are handy in these scenarios). Stop assuming you start with an entity relation diagram. That's an immediate limiting factor when listening to the business describe their processes. 2. There is no such thing as immediate updates to any database. There is _always_ latency. Build software understanding this. 3. Operational data and Analytical data are TWO DIFFERENT THINGS. (sorry for the shouting) Operationally, I only need concern myself with the immediate needs of the user or process. If I'm doing "something" to the customer domain, I don't need to know or do anything else. If I'm doing something to the order domain, I may need to notify some other domains of what I'm doing or have done, but that's secondary and not _immediately important_. Inventory systems should have built-in mechanisms for levels and never need to know the exact up-to-date figures. My operational domains can notify whole other systems on changes in data. So your analytical system can subscribe to these changes and normalize that data all it wants. I can even build user interfaces that display both operational and analytical data. Micro-services are brilliant at operational domain boundary adherence. Events are brilliant at notifying external boundaries of change. The caveat I point out to my clients is that thinking in this way is very different than we're used to and often comfortable with. It takes time to identify the best boundaries and events for the models of a business. But if you put in that time, the result will be software that your business personnel can actually understand.
- mumblemumble 6y ago
  I honestly believe that hiding complexity behind a closed door does not eliminate it. However, a lot of software and service vendors have a vested interest in convincing people otherwise. And, historically, they've had all sorts of great platforms for doing so. Who doesn't enjoy a free day out of the office, with lunch provided? It's also much easier to hide complexity than it is to remove it. One can be accomplished with (relative) turnkey solutions, generally without ever having to leave the comfort of your computer. Whereas the other generally requires long hours standing in front of a chalkboard and scratching your head.
  - SpicyLemonZest 6y ago
    On the other hand, hiding complexity behind closed doors can be a very valuable thing, if it lets you keep track of who knows about the complexity behind each. I can't count the number of issues I've encountered that would have taken minutes instead of hours if only I'd known which specific experts I needed to talk to.
    - mumblemumble 6y ago
      Agreed. Though, that to comes at a cost, so I don't want to do it except when it's worth it. http://yosefk.com/blog/redundancy-vs-dependencies-which-is-worse.html http://yosefk.com/blog/redundancy-vs-dependencies-which-is-w...
  - hinkley 6y ago
    > It's also much easier to hide complexity than it is to remove it. There are multiple ways to hide complexity. Some of them make it easier to remove (eg, refactoring), others make it nearly impossible to remove. In a service market there’s a perverse incentive to move toward the latter.
bob1029 6y ago
This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and S3. I.e. the actual compute/memory/storage/network capacity, and the mapping of the outside world to it. Because we are a software company, we write most of our own software. There is no value to our customers in us stringing together a pile of someone else's products, especially in a way that we cannot guarantee will be sustainable for >5 years. We cannot afford to constantly rework completed product installations. We strongly feel that any deeper buy-in with 3rd party technology vendors would compromise our agility and put us at their total mercy. Where we are currently positioned in the lock-in game, we could pull the ripcord and be sitting in a private datacenter within a week. All we need to do is move VMs, domain registrations and DNS nameservers if we want to kill the AWS bill. I feel for those who are up to their eyeballs in cloud infrastructure. Perhaps you made your own bed, but you shouldn't have to suffer in it. These are very complex decisions. We didn't get it right at first either. Maybe consider pleading with your executive management for mercy now. Perhaps you get a shot at a complete redo before it all comes crashing down. We certainly did. It's amazing what can happen if you have the guts to own up to bad choices and start an honest conversation. I would also be interested to hear the other side of the coin. Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it?
- rvanmil 6y ago
  We carefully select and use PaaS and managed cloud services to construct our infrastructure with. This allows us to maximize our focus on what our customers are paying for: creating software for them which will typically be in use for 5+ years. We spend close to zero time on infrastructure maintenance and management, we pay others to do this for us, cheaper and more reliable. Having to swap out one service for another hasn't given us any trouble or unreasonable costs yet in the past 5 years. Unlike the article is trying to convince us of, it has massively reduced complexity for us.
- bane 6y ago
s3tz 6y ago
Anyone manage to find part 1? It's not on their site, can't seem to find it.
- jvanderbot 6y ago
  It's linked in the article https://ea.rna.nl/2016/01/10/a-tale-of-application-rationalization-not/ https://ea.rna.nl/2016/01/10/a-tale-of-application-rationali... "This was actually part 1 of this story: A tale of application rationalisation (not)."
CalChris 6y ago
Isn't this just Jevon's Paradox applied to software? when technological progress ... increases the efficiency with which a resource is used .., but the rate of consumption of that resource rises due to increasing demand [1] [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox
mrkramer 6y ago
Microsoft summarized it nice [1] : Advantages of public clouds: Lower costs No maintenance Near-unlimited scalability High reliability Advantages of a private cloud: More flexibility More control More scalability (compared to pure on-prem solution) [1] https://azure.microsoft.com/en-us/overview/what-are-private-public-hybrid-clouds/ https://azure.microsoft.com/en-us/overview/what-are-private-...
- indymike 6y ago
  Hmm... So Azure for unlimited scalability... But private clouds have more scalability?
  - beaconstudios 6y ago
    Presumably both options are written relative to non cloud setups
  - 8K832d7tNmiQ 6y ago
    probably meant that you can just request hundreds of servers in other part of the world in one single setup compared to manually building your server there.
    - mrkramer 6y ago
      "More scalability — private clouds often offer more scalability compared to on-premises infrastructure." I think they meant private cloud (renting 3rd party servers and using/maintaining your private cloud) vs on-prem (buying servers and building your own data centers).
- cccc4all 6y ago
  Notice that neither list has "Reduce Complexity" as benefit. Cloud can abstract away complexity. Shift managing the complexity behind the cloud. But, it's more likely that complexity increases rather than decreases.
jvanderbot 6y ago
Part 1 is linked in article https://ea.rna.nl/2016/01/10/a-tale-of-application-rationalization-not/ https://ea.rna.nl/2016/01/10/a-tale-of-application-rationali... "This was actually part 1 of this story: A tale of application rationalisation (not)."
deleted 6y ago
[deleted]
arendtio 6y ago
I wonder who tells the story that cloud computing has something to do with reducing complexity. In my world, cloud computing is a bout scalability and making things as complex as they need to be to be scalable. This rarely means that complexity is being reduced.
- didericis 6y ago
  The simplicity of having one cloud based product rather than several native products built for different systems is an argument I’ve heard a lot.
  - ratww 6y ago
    This is an advantage of the web platform, not exactly related to cloud. You can get this advantage with an on-premises web product, or with old school hosting.
    - didericis 6y ago
      Very true, just trying to explain the source of the “cloud reduces complexity” argument. There are a number of small operations that don’t want to manage all their own hardware, so cloud and web are conflated, and you get the web platform simplicity argument being used to justify a cloud platform.
- twistedpair 6y ago
  But doesn't cloud computing reduce complexity? I used to write my own SystemV boot scripts, service installation and discovery logic, round robin updating tooling, and much more to handle live app deploy/cut over. Now I have two dozen lines of YAML and GKE JustWorks™, deploying my services and taking care of everything I used to do with my bespoke scripts. My world is much simpler now.
candiddevmike 6y ago
I see a lot of mentions in the comments about just using the basic storage/networking/compute from AWS/AZ/GCP--if that's all you're using, you should really consider other providers. Linode, Digital Ocean, and Vultr will be far more competitive and offer faster machines, cheaper, and with better bandwidth pricing. The point of using AWS/AZ/GCP is to leverage their managed service portfolio and be locked in. If you aren't doing that, there are better companies that want your business and will treat you much better.
- ithrow 6y ago
  IME, the network of AWS is much better than that of DO, Linode.
  - dilyevsky 6y ago
    How so?
    - ithrow 6y ago
      Less hiccups and downtime. It's faster and with better latency to other third party services. Superior internal control. Ex: In, linode a private IP address gives EVERYONE on the same data center access to your Linode server. Also, last time I used them they didn't have a Firewall.
      - dilyevsky 6y ago
        Right linode is basically old school dedicated servers afaik but DO should be in different class
        dijit 6y ago
        AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests). Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance. For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between those nodes often and sometimes congestion which drops users. (IRC is highly stateful TCP sessions). I've only known two truly good networking suppliers, GCP (and their magic BGP<->PoP<->Dark Fibre networks) and.. Tilaa.. (which is only hosted in Netherlands.. which is why I can't use them for my global network)
skohan 6y ago
I have to say, at my current company we are using Serverless, and it really does feel like it reduces complexity. No runtime/framework to set up, no uptime monitoring or management required on the application layer, and scaling is essentially solved for us. I mean you do pay for what you get, but it does feel like one of those technologies which really lowers the barrier to entry in terms of being able to release a production web application. In terms of building an MVP, a single developer really can deploy an application without any dev-ops support, and it will serve 10 million users if you manage to get that much traffic. I'm sure it's not optimal for every case, but for an awful lot of cases it seems pretty darned good, and you can save on dev ops hiring.
- ratww 6y ago
  I used to be very excited about serverless, and I still have high hopes for it. But for me it ended up replacing the complexity of runtime and frameworks with the complexity of configuring auxiliary srevices like Gateway API, Amazon VPC, etc. We needed to move the complexity to some tool that configured the services around Lambda, like Terraform or Cloud Formation, or at best to a framework like Claudia or Serverless.com. Configuring it by hand looks fine in tutorials, but is madness: it's still complex, and makes it all way too fragile. There are however some products that make the experience better by simplifying configuration, like Vercel and Netlify.
  - skohan 6y ago
    Yeah I certainly agree that the complexity doesn't really go away completely, and sometimes it's much more frustrating to have to configure poorly documented services rather than just having access to the host OS. I guess my overall point would be that two of the hardest things to do in terms of making a production-ready application are scaling and security, and Serverless pretty much obviates them. So it's not a magic wand, but it does take away some of the significant barriers to entry.
    - ratww 6y ago
      Yes, I agree with that point. I think my point was more that Serverless is a good idea, but the current implementations are still not good at removing complexity. But I can see this easily changing, with open standards and the such.
ehnto 6y ago
I don't understand what people are building in order to need half of this decoupled and managed elsewhere anyway. It wasn't all that challenging to self manage it five years ago, what's changed? My guess is that the average small to medium project has drank the enterprise coolaid, and they are suffering the configuration and complexity nightmares that surround managing cloud infrastructure before they really needed to. As the article is pointing out, you don't forgo managing these things by doing it in the cloud, you just manage it inside a constantly changing Web UI instead of something likely familiar to your developers.
- ratww 6y ago
  There are products (Terraform, CloudFormation) that help managing without an UI, but they also add complexity, so our point definitely stills stands.
  - dijit 6y ago
    I've used Terraform for 4~ years. the providers are ever changing (and so is the terraform language/features). So the "ever-changing" part is certainly still true for Terraform.
- dkarl 6y ago
  I guess it's Kool-Aid? I don't know; I don't remember being lied to when I started using cloud services. I think of cloud resources as being amazing and basically magical, but I know there's a limit to the magic, and the rest is work. People using (for example) AWS S3 should not be surprised that they still have to work to manage the naming, organization, access control, encryption, retention, etc. of their data, and they might encounter problems if they try to load a 100GB S3 object into a byte array in a container provisioned with 1GB of RAM. But they are. I don't know if that's human nature or if they're being lied to by consultants and marketers.
nelsonenzo 6y ago
This person has never worked in a data center. He thinks he's managing a network because he sets a few vpc ips, that's an itsy tiny fragment of networking, and the cloud has indeed removex a great deal you previously had to manage on prem.
- rubatuga 6y ago
  I'm guessing you mean things like STP, firewalls, rate-limiting, routing, DHCP, and link aggregation?
  - LimaBearz 6y ago
    Or going back even further I started my career worrying about all that, plus HVAC, hardware acquisition, capacity planning, power management, distribution and managing gensets(fuel and maintenance); and all the fun crap associated with it like a burst water pipe in your data center on Christmas Eve
  - nelsonenzo 6y ago
    to start, yes :)
zoomablemind 6y ago
Subjectively, it increasingly feels that while the complexity has been increasing, the notion of longevity of the underlying products and services has been degrading. While updates to software were expected, general outlook would be that they would not be breaking the core features. The emphasis on backwards compatibility was in a way an assurance to businesses that building their operations on vendor's products is not risky. Even then, some mission-critical elements would be defensively abstracted to avoid the dependency risks (at least theoretically...) Now, we all witness the "eternal-beta" paradigm across the most of the major software products. Frequent builds with automatic updates, when new features could be suddenly pushed, and old features removed. Sure, it's still possible to spec out a "hard-rock" steady platform, postpone updates, abstract dependencies and just focus on business. But...such approach won't be approved, as it's widely acknowledged that the presence of critical bugs is rather a "feature" of all software. Postponing the updates is not prudent, it's a liability. So the rock-solid expectations are just an illusion or perhaps a fantasy promoted widely, just to get the foot in the door. Ironically, the most stable elements are the so much dreaded "legacy", too often in charge of the business-critical logic.
- twistedpair 6y ago
  > the notion of longevity of the underlying products and services has been degrading. Is eternal longevity even a relevant concept in computing ops any more? I think businesses must grok that all services are living things that will require eventual maintenance. If you roll your own service, eventually the OS, even an LTS, will need upgrading when the prior release goes EOL. If you're higher up on a FaaS, the language/runtime will still age out, like Python 2, Node.js 6, or Java 5. These will inevitably come with changes. You application will need to adapt. In extreme cases you can pay for ancient language support and customer IBM mainframe hardware, but the reality is the software's hosting will always need to be updated as time marches on.
  - bdavis__ 6y ago
    Kind of depends on "eternal". I propose 10 years is possible, and realistic. 5 years should not be rare. 20 years is probably a fantasy.
blabitty 6y ago
For simple document or file type storage for an application I think solutions like S3 or azure blob storage are really great. In my real world experience we replaced a highly finicky way too heavyweight COTS CMS and all the associated database and backend SAN complexity with S3 and it ended up being way cheaper, easier and more reliable. To me object storage is one place where the cloud hype does deliver. Have had mixed results with the other services, even with things like RDS you have to do a lot of tuning and backup / DR work yourself.

224 points by rapnie 6y ago

deleted 6y ago

[deleted]

Reducing complexity should never be about platform (on-prem vs cloud). It should be about constructing software in partnership with the business and reducing complexity with modeled boundaries. You can leverage the cloud to do some interesting things, but the true benefit in is _what_ you construct, not _how_.

sanp 6y ago
There is an element of _how_ as well. You could create simple monoliths or overengineered microservices. Or, complex monoliths with heavy coupling vs cleanly designed microservices with clear separations of concern.
- the-smug-one 6y ago
  Are microservices meant to separate data too? As in, each service has its own database. Wouldn't that lead to non-normalisation of the data or a lot of expensive network lookups to get what I want/need? What is the point of micro services anyway :-)?
  - kqr 6y ago
    > Are microservices meant to separate data too? As in, each service has its own database. Yes. > Wouldn't that lead to non-normalisation of the data Yes. But it's not as bad as it sounds. That is how data on paper used to work, after all. Business rules (at least ones that have been around for more than 5--10 years) are written with intensely non-normalised data in mind. Business people tend to be fine with eventual consistency on the scale of hours or even days. Non-normalised data also makes total data corruption harder, and forensics in the case of bugs easier, in some ways: you find an unexpected value somewhere? Check the other versions that ought to exist and you can probably retrace at what point it got weird. The whole idea of consistent and fully normalised data is a, historically speaking, very recent innovation, and I'm not convinced it will last long in the real world. I think this is a brief moment in history when our software is primitive enough, yet optimistic enough, to even consider that type of data storage. And come on, it's not like the complete consistency of the data is worth that many dollars in most cases, if we actually bother to compute the cost.
  - ratww 6y ago
    > Are microservices meant to separate data too? As in, each service has its own database. Ideally yes, to scale. Sometimes you have a service with obvious and easy-to-split boundaries, and microservices are a breeze. Some things that are easy to turn into microservices: "API Wrapper" to a complex and messy third-party API. Logging and data collection. Sending emails/messages. User authentication. Search. Anything in your app that could become another app. However, when your data model is tightly coupled and you need to choose between tradeoffs (data duplication), having bigger services, or even keeping it as a monolith. Btw, if you don't care about scalability, sharing a database is still still not the best idea. But you can have a microservice that wraps the database in a service, for example. Tools like Hasura can be used for that.
  - NicoJuicy 6y ago
    Microservices is a solution for an organisational problem ( multiple employees in one project), not a technical one. If you're flying solo, just use DDD for example. It will give you the same patterns without the devops complexity
  - ChicagoDave 6y ago
    There's a progression ever developer that grew up on SQL Server/relational data needs to go through... 1. The models (plural intended) of a business are not necessarily a relational. There could be an upstream/downstream relationship. There could be an event-based relationship. There may be no relations at all (documents are handy in these scenarios). Stop assuming you start with an entity relation diagram. That's an immediate limiting factor when listening to the business describe their processes. 2. There is no such thing as immediate updates to any database. There is _always_ latency. Build software understanding this. 3. Operational data and Analytical data are TWO DIFFERENT THINGS. (sorry for the shouting) Operationally, I only need concern myself with the immediate needs of the user or process. If I'm doing "something" to the customer domain, I don't need to know or do anything else. If I'm doing something to the order domain, I may need to notify some other domains of what I'm doing or have done, but that's secondary and not _immediately important_. Inventory systems should have built-in mechanisms for levels and never need to know the exact up-to-date figures. My operational domains can notify whole other systems on changes in data. So your analytical system can subscribe to these changes and normalize that data all it wants. I can even build user interfaces that display both operational and analytical data. Micro-services are brilliant at operational domain boundary adherence. Events are brilliant at notifying external boundaries of change. The caveat I point out to my clients is that thinking in this way is very different than we're used to and often comfortable with. It takes time to identify the best boundaries and events for the models of a business. But if you put in that time, the result will be software that your business personnel can actually understand.
mumblemumble 6y ago
I honestly believe that hiding complexity behind a closed door does not eliminate it. However, a lot of software and service vendors have a vested interest in convincing people otherwise. And, historically, they've had all sorts of great platforms for doing so. Who doesn't enjoy a free day out of the office, with lunch provided? It's also much easier to hide complexity than it is to remove it. One can be accomplished with (relative) turnkey solutions, generally without ever having to leave the comfort of your computer. Whereas the other generally requires long hours standing in front of a chalkboard and scratching your head.
- SpicyLemonZest 6y ago
  On the other hand, hiding complexity behind closed doors can be a very valuable thing, if it lets you keep track of who knows about the complexity behind each. I can't count the number of issues I've encountered that would have taken minutes instead of hours if only I'd known which specific experts I needed to talk to.
  - mumblemumble 6y ago
    Agreed. Though, that to comes at a cost, so I don't want to do it except when it's worth it. http://yosefk.com/blog/redundancy-vs-dependencies-which-is-worse.html http://yosefk.com/blog/redundancy-vs-dependencies-which-is-w...
- hinkley 6y ago
  > It's also much easier to hide complexity than it is to remove it. There are multiple ways to hide complexity. Some of them make it easier to remove (eg, refactoring), others make it nearly impossible to remove. In a service market there’s a perverse incentive to move toward the latter.

bob1029 6y ago

This shared responsibility principle that underlies cloud marketing speak sounds a lot like the self-driving mess we find ourselves in today - I.e. the responsibility boundary between parties exists in a fog of war and results in more exceptions than if one or the other were totally responsible. We have been a customer of Amazon AWS for ~6 years now, and we still really only use ~3 of their products: EC2, Route53 and S3. I.e. the actual compute/memory/storage/network capacity, and the mapping of the outside world to it. Because we are a software company, we write most of our own software. There is no value to our customers in us stringing together a pile of someone else's products, especially in a way that we cannot guarantee will be sustainable for >5 years. We cannot afford to constantly rework completed product installations. We strongly feel that any deeper buy-in with 3rd party technology vendors would compromise our agility and put us at their total mercy. Where we are currently positioned in the lock-in game, we could pull the ripcord and be sitting in a private datacenter within a week. All we need to do is move VMs, domain registrations and DNS nameservers if we want to kill the AWS bill. I feel for those who are up to their eyeballs in cloud infrastructure. Perhaps you made your own bed, but you shouldn't have to suffer in it. These are very complex decisions. We didn't get it right at first either. Maybe consider pleading with your executive management for mercy now. Perhaps you get a shot at a complete redo before it all comes crashing down. We certainly did. It's amazing what can happen if you have the guts to own up to bad choices and start an honest conversation. I would also be interested to hear the other side of the coin. Who out there is using 20+ AWS/Azure/GCP products to back a single business app and is having a fantastic time of it?

rvanmil 6y ago
We carefully select and use PaaS and managed cloud services to construct our infrastructure with. This allows us to maximize our focus on what our customers are paying for: creating software for them which will typically be in use for 5+ years. We spend close to zero time on infrastructure maintenance and management, we pay others to do this for us, cheaper and more reliable. Having to swap out one service for another hasn't given us any trouble or unreasonable costs yet in the past 5 years. Unlike the article is trying to convince us of, it has massively reduced complexity for us.
bane 6y ago

s3tz 6y ago

Anyone manage to find part 1? It's not on their site, can't seem to find it.

jvanderbot 6y ago
It's linked in the article https://ea.rna.nl/2016/01/10/a-tale-of-application-rationalization-not/ https://ea.rna.nl/2016/01/10/a-tale-of-application-rationali... "This was actually part 1 of this story: A tale of application rationalisation (not)."

CalChris 6y ago

Isn't this just Jevon's Paradox applied to software? when technological progress ... increases the efficiency with which a resource is used .., but the rate of consumption of that resource rises due to increasing demand [1] [1] https://en.wikipedia.org/wiki/Jevons_paradox https://en.wikipedia.org/wiki/Jevons_paradox

mrkramer 6y ago

Microsoft summarized it nice [1] : Advantages of public clouds: Lower costs No maintenance Near-unlimited scalability High reliability Advantages of a private cloud: More flexibility More control More scalability (compared to pure on-prem solution) [1] https://azure.microsoft.com/en-us/overview/what-are-private-public-hybrid-clouds/ https://azure.microsoft.com/en-us/overview/what-are-private-...

indymike 6y ago
Hmm... So Azure for unlimited scalability... But private clouds have more scalability?
- beaconstudios 6y ago
  Presumably both options are written relative to non cloud setups
- 8K832d7tNmiQ 6y ago
  probably meant that you can just request hundreds of servers in other part of the world in one single setup compared to manually building your server there.
  - mrkramer 6y ago
    "More scalability — private clouds often offer more scalability compared to on-premises infrastructure." I think they meant private cloud (renting 3rd party servers and using/maintaining your private cloud) vs on-prem (buying servers and building your own data centers).
cccc4all 6y ago
Notice that neither list has "Reduce Complexity" as benefit. Cloud can abstract away complexity. Shift managing the complexity behind the cloud. But, it's more likely that complexity increases rather than decreases.

jvanderbot 6y ago

Part 1 is linked in article https://ea.rna.nl/2016/01/10/a-tale-of-application-rationalization-not/ https://ea.rna.nl/2016/01/10/a-tale-of-application-rationali... "This was actually part 1 of this story: A tale of application rationalisation (not)."

deleted 6y ago

[deleted]

arendtio 6y ago

I wonder who tells the story that cloud computing has something to do with reducing complexity. In my world, cloud computing is a bout scalability and making things as complex as they need to be to be scalable. This rarely means that complexity is being reduced.

didericis 6y ago
The simplicity of having one cloud based product rather than several native products built for different systems is an argument I’ve heard a lot.
- ratww 6y ago
  This is an advantage of the web platform, not exactly related to cloud. You can get this advantage with an on-premises web product, or with old school hosting.
  - didericis 6y ago
    Very true, just trying to explain the source of the “cloud reduces complexity” argument. There are a number of small operations that don’t want to manage all their own hardware, so cloud and web are conflated, and you get the web platform simplicity argument being used to justify a cloud platform.
twistedpair 6y ago
But doesn't cloud computing reduce complexity? I used to write my own SystemV boot scripts, service installation and discovery logic, round robin updating tooling, and much more to handle live app deploy/cut over. Now I have two dozen lines of YAML and GKE JustWorks™, deploying my services and taking care of everything I used to do with my bespoke scripts. My world is much simpler now.

candiddevmike 6y ago

I see a lot of mentions in the comments about just using the basic storage/networking/compute from AWS/AZ/GCP--if that's all you're using, you should really consider other providers. Linode, Digital Ocean, and Vultr will be far more competitive and offer faster machines, cheaper, and with better bandwidth pricing. The point of using AWS/AZ/GCP is to leverage their managed service portfolio and be locked in. If you aren't doing that, there are better companies that want your business and will treat you much better.

ithrow 6y ago
IME, the network of AWS is much better than that of DO, Linode.
- dilyevsky 6y ago
  How so?
  - ithrow 6y ago
    Less hiccups and downtime. It's faster and with better latency to other third party services. Superior internal control. Ex: In, linode a private IP address gives EVERYONE on the same data center access to your Linode server. Also, last time I used them they didn't have a Firewall.
    - dilyevsky 6y ago
      Right linode is basically old school dedicated servers afaik but DO should be in different class
      - dijit 6y ago
        AWS networking isn't _great_ but it's decidedly better than DO (which is actually the worst of those listed based on my own TCP connection tests). Linode is pretty stable if not very exciting, Vultr is "better than DO", but their networks are almost always in maintenance. For a little context; I maintain IRC servers and those are currently hosted in vultr (with linked nodes in 5 regions), I notice ping spikes between those nodes often and sometimes congestion which drops users. (IRC is highly stateful TCP sessions). I've only known two truly good networking suppliers, GCP (and their magic BGP<->PoP<->Dark Fibre networks) and.. Tilaa.. (which is only hosted in Netherlands.. which is why I can't use them for my global network)

skohan 6y ago

I have to say, at my current company we are using Serverless, and it really does feel like it reduces complexity. No runtime/framework to set up, no uptime monitoring or management required on the application layer, and scaling is essentially solved for us. I mean you do pay for what you get, but it does feel like one of those technologies which really lowers the barrier to entry in terms of being able to release a production web application. In terms of building an MVP, a single developer really can deploy an application without any dev-ops support, and it will serve 10 million users if you manage to get that much traffic. I'm sure it's not optimal for every case, but for an awful lot of cases it seems pretty darned good, and you can save on dev ops hiring.

ratww 6y ago
I used to be very excited about serverless, and I still have high hopes for it. But for me it ended up replacing the complexity of runtime and frameworks with the complexity of configuring auxiliary srevices like Gateway API, Amazon VPC, etc. We needed to move the complexity to some tool that configured the services around Lambda, like Terraform or Cloud Formation, or at best to a framework like Claudia or Serverless.com. Configuring it by hand looks fine in tutorials, but is madness: it's still complex, and makes it all way too fragile. There are however some products that make the experience better by simplifying configuration, like Vercel and Netlify.
- skohan 6y ago
  Yeah I certainly agree that the complexity doesn't really go away completely, and sometimes it's much more frustrating to have to configure poorly documented services rather than just having access to the host OS. I guess my overall point would be that two of the hardest things to do in terms of making a production-ready application are scaling and security, and Serverless pretty much obviates them. So it's not a magic wand, but it does take away some of the significant barriers to entry.
  - ratww 6y ago
    Yes, I agree with that point. I think my point was more that Serverless is a good idea, but the current implementations are still not good at removing complexity. But I can see this easily changing, with open standards and the such.

ehnto 6y ago

I don't understand what people are building in order to need half of this decoupled and managed elsewhere anyway. It wasn't all that challenging to self manage it five years ago, what's changed? My guess is that the average small to medium project has drank the enterprise coolaid, and they are suffering the configuration and complexity nightmares that surround managing cloud infrastructure before they really needed to. As the article is pointing out, you don't forgo managing these things by doing it in the cloud, you just manage it inside a constantly changing Web UI instead of something likely familiar to your developers.

ratww 6y ago
There are products (Terraform, CloudFormation) that help managing without an UI, but they also add complexity, so our point definitely stills stands.
- dijit 6y ago
  I've used Terraform for 4~ years. the providers are ever changing (and so is the terraform language/features). So the "ever-changing" part is certainly still true for Terraform.
dkarl 6y ago
I guess it's Kool-Aid? I don't know; I don't remember being lied to when I started using cloud services. I think of cloud resources as being amazing and basically magical, but I know there's a limit to the magic, and the rest is work. People using (for example) AWS S3 should not be surprised that they still have to work to manage the naming, organization, access control, encryption, retention, etc. of their data, and they might encounter problems if they try to load a 100GB S3 object into a byte array in a container provisioned with 1GB of RAM. But they are. I don't know if that's human nature or if they're being lied to by consultants and marketers.

nelsonenzo 6y ago

This person has never worked in a data center. He thinks he's managing a network because he sets a few vpc ips, that's an itsy tiny fragment of networking, and the cloud has indeed removex a great deal you previously had to manage on prem.

rubatuga 6y ago
I'm guessing you mean things like STP, firewalls, rate-limiting, routing, DHCP, and link aggregation?
- LimaBearz 6y ago
  Or going back even further I started my career worrying about all that, plus HVAC, hardware acquisition, capacity planning, power management, distribution and managing gensets(fuel and maintenance); and all the fun crap associated with it like a burst water pipe in your data center on Christmas Eve
- nelsonenzo 6y ago
  to start, yes :)

zoomablemind 6y ago

Subjectively, it increasingly feels that while the complexity has been increasing, the notion of longevity of the underlying products and services has been degrading. While updates to software were expected, general outlook would be that they would not be breaking the core features. The emphasis on backwards compatibility was in a way an assurance to businesses that building their operations on vendor's products is not risky. Even then, some mission-critical elements would be defensively abstracted to avoid the dependency risks (at least theoretically...) Now, we all witness the "eternal-beta" paradigm across the most of the major software products. Frequent builds with automatic updates, when new features could be suddenly pushed, and old features removed. Sure, it's still possible to spec out a "hard-rock" steady platform, postpone updates, abstract dependencies and just focus on business. But...such approach won't be approved, as it's widely acknowledged that the presence of critical bugs is rather a "feature" of all software. Postponing the updates is not prudent, it's a liability. So the rock-solid expectations are just an illusion or perhaps a fantasy promoted widely, just to get the foot in the door. Ironically, the most stable elements are the so much dreaded "legacy", too often in charge of the business-critical logic.

twistedpair 6y ago
> the notion of longevity of the underlying products and services has been degrading. Is eternal longevity even a relevant concept in computing ops any more? I think businesses must grok that all services are living things that will require eventual maintenance. If you roll your own service, eventually the OS, even an LTS, will need upgrading when the prior release goes EOL. If you're higher up on a FaaS, the language/runtime will still age out, like Python 2, Node.js 6, or Java 5. These will inevitably come with changes. You application will need to adapt. In extreme cases you can pay for ancient language support and customer IBM mainframe hardware, but the reality is the software's hosting will always need to be updated as time marches on.
- bdavis__ 6y ago
  Kind of depends on "eternal". I propose 10 years is possible, and realistic. 5 years should not be rare. 20 years is probably a fantasy.

blabitty 6y ago

For simple document or file type storage for an application I think solutions like S3 or azure blob storage are really great. In my real world experience we replaced a highly finicky way too heavyweight COTS CMS and all the associated database and backend SAN complexity with S3 and it ended up being way cheaper, easier and more reliable. To me object storage is one place where the cloud hype does deliver. Have had mixed results with the other services, even with things like RDS you have to do a lot of tuning and backup / DR work yourself.