6 ms·

Cloud computing seems to be a winner take all scenario. For example, if you use AWS and need a message broker service then you'll use this. If you use Azure, yo

by bot41 6y ago

Cloud computing seems to be a winner take all scenario. For example, if you use AWS and need a message broker service then you'll use this. If you use Azure, you'll use their version. Development seems like just hooking up this components. I can't tell if this is a good thing or a bad thing.

ggoo 6y ago
good - my job is really easy bad - my job is really boring
- deanCommie 6y ago
  Somehow I don't think physical laborers ever complain when they get new and more powerful tools to make their job easier. It's only software engineers that bemoan their lives getting easier, so they can spend more time working on other problems higher up the abstraction chain.
  - marta_morena_28 6y ago
    And you can spend that time thinking about how not to use a message broker... And maybe just use lower level MANAGED services, like SQS or SNS.
  - dudul 6y ago
    How about when the new powerful tools help get rid of some of them due to productivity gain?
  - OJFord 6y ago
    The 'problems higher up the abstraction chain' are the ones that are closer to labour, or factory work. Mundane and repetitive, relatively speaking easy - requiring less thought and being to die extent trainable as working within a pattern/template.
- methodin 6y ago
  I beg to differ on your second point - at my company we've fully embraced AWS and putting vendor lock-in issues aside, the end result is focusing more on the application and less on the minutiae of operational issues which is a big win. This makes things much more interesting since you can get there faster and consequently take on more impactful projects in the same timeframe. This in general is a boon for developers in my experience.
  - ignoramous 6y ago
    Agreed. The flip side isn't vendor lock-in though, it is building complex systems under the guise of scale.
- bot41 6y ago
  > good - my job is really easy Longterm isn't this bad? For example, if you can do it with 5 years experience. At 15 years experience you'll likely be too expensive for companies to want to hire you. They'll just hire more junior people.
qz2 6y ago
Nope. If only. There are four considerations to look at which are rarely if ever mentioned between the mountains of well organised hype: The first is application complexity. A lot of real workloads are quite complex but do not need to scale out a lot. The cloud and all the hype is organised around simple workloads that need to scale out easily which is an easy win. So for example Netflix or a SaaS application with a few tens of endpoints and a React front end or something. The wide sprawling real businesses are a terrible fit and tend to get rather expensive rather quickly when you start putting their workloads into the cloud. There is marginal aggregate cost benefit over actually buying hardware ($4m a year SQL server clusters are a reality in the cloud), the real benefit being only agility. The second is simply "hooking up components" sounds really easy. But it's not. I think perhaps 50% of my time is working out why X won't talk to Y or why Z is broken and finding some opaque abstraction which doesn't allow me to get to the bottom of the problem. It's very very easy to turn your deployment into a complete tangle of chaos and circular dependencies which are very hard to rationalise and automate even with state of the art automation tools (which I will say tend to melt in your hands). This is existing layering on top of the same concerns you had before rather than a different one. Thirdly we have to work out the difference between mature products and hype. Nearly all solutions are described in little blog snippets that make things look really easy for a specific and narrow use case but realistically things are really fucking complicated and in some cases absolutely awfully described in documentation. In a lot of cases, including AWS, it's actually hard to find someone at the cloud vendor who knows how something works when you break it. And sometimes there are solutions which are just absolutely dire. Again pointing the finger here at Amazon's managed ElasticSearch. Fourthly, you end up being perpetual bean counter afraid of the rube goldberg machine waking in the middle of the night due to some event you didn't anticipate and drinking the content of your credit card in a few minutes. Some of the cost management and spot instance management software automates this rather nicely into a whole cluster of new failure modes as well just as if the complexity wasn't enough already. A trite version of this is "saving money costs money and sometimes the benefits are less than the costs" So what you end up doing is trading your original problems for a set of new and shiny ones which are possibly even more complicated. But at least you only have one vendor to shout at, which is a net win if you've ever tried to get HPE and Cisco to work out what fucked up mess is going on between their two lumps of iron. I digress but be careful with assumptions about it being magical unicorns. They poop and you have to shovel it.
awinder 6y ago
AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology. This is more likely to do with opening doors for large and complex applications that can’t be rewritten to come into the AWS cloud. Unsexy stuff but there’s some fun engineering to be done in that realm, if you find like puzzles and shitshows to be fun anyways :-)
- manigandham 6y ago
  Neither SQS nor Kinesis support the same functionality as RabbitMQ.
- avinassh 6y ago
  > AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology. Why are they better?
  - awinder 6y ago
    They’ve been around for quite some time so they have a much wider customer base & bigger teams supporting them. AWS services can be a bit choppy in the beginning so imo, especially with queues, I’d wait for it to bake.
    - pmiller2 6y ago
      I agree. SQS has done nothing but improve over time.
  - whatsmyusername 6y ago
    They aren't. They're technically more correct but not always the practical best choice. RabbitMQ is a smart play as Rabbit is very easy to use, understand, and troubleshoot at the low end (which is where I suspect the vast majority of queue systems live). It also has a feature which is actually really hard to do (and sqs doesn't do). Guaranteed delivery of a message once. That was THE reason we never migrated to SQS, there are scenarios where SQS can double deliver. Our codebase was built up from nothing over time and couldn't gracefully handle double delivery of messages in all scenarios. We could have refactored, but it wasn't worth the work when we were already doing a half billion in revenue without getting even close to the limitations of rabbit AND were close to selling (which we ultimately did). AWS is great at selling multiple slight variations of the same product. If you look you can usually find ONE variation that works for you. The real test will be if the billing isn't garbage (garbage billing is why we didn't use their other AMQP service and part of the reason why we don't use things like EKS or Managed SFTP despite having the need).
malux85 6y ago
It’s a bad thing - whenever you cannot make a choice you become a slave to tyranny. Infra lock-in means you’re a slave to their whims (financial, legal, competitivenes, whatever) This is justified by “less maintenance, and easier deployment” but the reality of the situation is, it’s not worth giving your freedom up for, and to a lesser degree - if your platform becomes popular, you end up spending the same amount of time tweaking and optimising to match the idiosyncrasies of their implementation anyway. But the most important part is vendor lock-in, it’s bad.
- sofixa 6y ago
  How is using AWS or Azure's managed RabbitMQ service lock-in? You can easily switch to someone else's managed service, or roll your own, since it's still just good old RabbitMQ.
  - malux85 6y ago
    I was responding to the parents more general point about cloud services being "winner take all"
- nijave 6y ago
  How many cases are you truely not locked into anything? Even if you host your own RabbitMQ you're still coupled to the software. I'm not sure being coupled to cloud service X is any worse than OSS product Y. For the latter, you tend to need to have more expertise to run it yourself. At the end of the day, you can still rewrite your code and switch in both cases. You can end up in a tough spot if the OSS community loses interest in the software you've already bet your complicated app on, as well

by bot41 6y ago

ggoo 6y ago

good - my job is really easy bad - my job is really boring

deanCommie 6y ago
Somehow I don't think physical laborers ever complain when they get new and more powerful tools to make their job easier. It's only software engineers that bemoan their lives getting easier, so they can spend more time working on other problems higher up the abstraction chain.
- marta_morena_28 6y ago
  And you can spend that time thinking about how not to use a message broker... And maybe just use lower level MANAGED services, like SQS or SNS.
- dudul 6y ago
  How about when the new powerful tools help get rid of some of them due to productivity gain?
- OJFord 6y ago
  The 'problems higher up the abstraction chain' are the ones that are closer to labour, or factory work. Mundane and repetitive, relatively speaking easy - requiring less thought and being to die extent trainable as working within a pattern/template.
methodin 6y ago
I beg to differ on your second point - at my company we've fully embraced AWS and putting vendor lock-in issues aside, the end result is focusing more on the application and less on the minutiae of operational issues which is a big win. This makes things much more interesting since you can get there faster and consequently take on more impactful projects in the same timeframe. This in general is a boon for developers in my experience.
- ignoramous 6y ago
  Agreed. The flip side isn't vendor lock-in though, it is building complex systems under the guise of scale.
bot41 6y ago
> good - my job is really easy Longterm isn't this bad? For example, if you can do it with 5 years experience. At 15 years experience you'll likely be too expensive for companies to want to hire you. They'll just hire more junior people.

qz2 6y ago

Nope. If only. There are four considerations to look at which are rarely if ever mentioned between the mountains of well organised hype: The first is application complexity. A lot of real workloads are quite complex but do not need to scale out a lot. The cloud and all the hype is organised around simple workloads that need to scale out easily which is an easy win. So for example Netflix or a SaaS application with a few tens of endpoints and a React front end or something. The wide sprawling real businesses are a terrible fit and tend to get rather expensive rather quickly when you start putting their workloads into the cloud. There is marginal aggregate cost benefit over actually buying hardware ($4m a year SQL server clusters are a reality in the cloud), the real benefit being only agility. The second is simply "hooking up components" sounds really easy. But it's not. I think perhaps 50% of my time is working out why X won't talk to Y or why Z is broken and finding some opaque abstraction which doesn't allow me to get to the bottom of the problem. It's very very easy to turn your deployment into a complete tangle of chaos and circular dependencies which are very hard to rationalise and automate even with state of the art automation tools (which I will say tend to melt in your hands). This is existing layering on top of the same concerns you had before rather than a different one. Thirdly we have to work out the difference between mature products and hype. Nearly all solutions are described in little blog snippets that make things look really easy for a specific and narrow use case but realistically things are really fucking complicated and in some cases absolutely awfully described in documentation. In a lot of cases, including AWS, it's actually hard to find someone at the cloud vendor who knows how something works when you break it. And sometimes there are solutions which are just absolutely dire. Again pointing the finger here at Amazon's managed ElasticSearch. Fourthly, you end up being perpetual bean counter afraid of the rube goldberg machine waking in the middle of the night due to some event you didn't anticipate and drinking the content of your credit card in a few minutes. Some of the cost management and spot instance management software automates this rather nicely into a whole cluster of new failure modes as well just as if the complexity wasn't enough already. A trite version of this is "saving money costs money and sometimes the benefits are less than the costs" So what you end up doing is trading your original problems for a set of new and shiny ones which are possibly even more complicated. But at least you only have one vendor to shout at, which is a net win if you've ever tried to get HPE and Cisco to work out what fucked up mess is going on between their two lumps of iron. I digress but be careful with assumptions about it being magical unicorns. They poop and you have to shovel it.

awinder 6y ago

AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology. This is more likely to do with opening doors for large and complex applications that can’t be rewritten to come into the AWS cloud. Unsexy stuff but there’s some fun engineering to be done in that realm, if you find like puzzles and shitshows to be fun anyways :-)

manigandham 6y ago
Neither SQS nor Kinesis support the same functionality as RabbitMQ.
avinassh 6y ago
> AWS has sqs & kinesis which are much better queueing options in that scenario where you’re in AWS doing new dev and can pick a technology. Why are they better?
- awinder 6y ago
  They’ve been around for quite some time so they have a much wider customer base & bigger teams supporting them. AWS services can be a bit choppy in the beginning so imo, especially with queues, I’d wait for it to bake.
  - pmiller2 6y ago
    I agree. SQS has done nothing but improve over time.
- whatsmyusername 6y ago
  They aren't. They're technically more correct but not always the practical best choice. RabbitMQ is a smart play as Rabbit is very easy to use, understand, and troubleshoot at the low end (which is where I suspect the vast majority of queue systems live). It also has a feature which is actually really hard to do (and sqs doesn't do). Guaranteed delivery of a message once. That was THE reason we never migrated to SQS, there are scenarios where SQS can double deliver. Our codebase was built up from nothing over time and couldn't gracefully handle double delivery of messages in all scenarios. We could have refactored, but it wasn't worth the work when we were already doing a half billion in revenue without getting even close to the limitations of rabbit AND were close to selling (which we ultimately did). AWS is great at selling multiple slight variations of the same product. If you look you can usually find ONE variation that works for you. The real test will be if the billing isn't garbage (garbage billing is why we didn't use their other AMQP service and part of the reason why we don't use things like EKS or Managed SFTP despite having the need).

malux85 6y ago

It’s a bad thing - whenever you cannot make a choice you become a slave to tyranny. Infra lock-in means you’re a slave to their whims (financial, legal, competitivenes, whatever) This is justified by “less maintenance, and easier deployment” but the reality of the situation is, it’s not worth giving your freedom up for, and to a lesser degree - if your platform becomes popular, you end up spending the same amount of time tweaking and optimising to match the idiosyncrasies of their implementation anyway. But the most important part is vendor lock-in, it’s bad.

sofixa 6y ago
How is using AWS or Azure's managed RabbitMQ service lock-in? You can easily switch to someone else's managed service, or roll your own, since it's still just good old RabbitMQ.
- malux85 6y ago
  I was responding to the parents more general point about cloud services being "winner take all"
nijave 6y ago
How many cases are you truely not locked into anything? Even if you host your own RabbitMQ you're still coupled to the software. I'm not sure being coupled to cloud service X is any worse than OSS product Y. For the latter, you tend to need to have more expertise to run it yourself. At the end of the day, you can still rewrite your code and switch in both cases. You can end up in a tough spot if the OSS community loses interest in the software you've already bet your complicated app on, as well