7 ms·

Most of the microservice code I see is response = fetch(url, payload) if (response.error) ... but 99% of the folks I ask what is going to happen when the

by tra3 3y ago

Most of the microservice code I see is response = fetch(url, payload) if (response.error) ... but 99% of the folks I ask what is going to happen when the fetch does NOT error out but instead takes 10 seconds look at me like I'm speaking gibberish. This is the single biggest reason for cascading failures I see. Netflix has dealt with it via their Hystrix library (open source). These days it seems like a proxy like Consul is the way to go. It encapsulates all of the fancy logic (like circuit breakers and flow control) so your service doesn't have to.

jiggawatts 3y ago
As an ardent fan of monoliths and how they generally avoid such tar pits, I have to acknowledge that service-oriented architectures have their uses. So do we all have to keep reinventing these wheels, but only after a production outage? Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? I keep seeing the same patterns being re-engineered over and over. Maybe it’s time to refactor these out…
- klooney 3y ago
  It's more work, is the simple answer.
  - jiggawatts 3y ago
    Once. It's more work once instead of over and over. That's the point of operating systems, standard libraries, and modules! I see this weird back-lash in modern development against having common, standard platforms. I suspect it comes from the Python and JavaScript world, where having "no batteries included" is seen as a good thing, instead of a guaranteed mess of dozens of half-complete incompatible frameworks. I'm coming from the perspective of Windows and comparing it to, say, Azure or AWS. All three have some concepts of access control, log collection, component systems, processes, etc... But all three are proprietary. Kubernetes goes a long way, but it isn't a user-mode system that can be directly accessed from code. Compare with Service Fabric, which has a substantial SDK component that integrates into the applications. As an example, here's a really basic thing that is actually absurdly difficult to solve well: web application session state. If you have sticky load balancing using cookies, then the session state is accessed on one VM something like 99.99% of the time... except for that 1% of the time when it isn't. This could be due to a restart, load rebalancing, or whatever. If you put the session state into something external like Redis, then a zone-redundant deployment will eat a ~1ms delay on every page render, every time. Service Fabric uses a model where it keeps three replicas of the state: one in the original web server, and two replicas in elsewhere. This way, reads are in process on the same VM most of the time, resulting in nanosecond latencies. Writing the state can occur asynchronously after the page response is already being sent. I'd like to see concepts like this, along with all sorts of service-to-service communication patterns, consolidated into an "operating system like platform" designed for the mid-2020s clouds instead of 1990s server farms.
    - lstamour 3y ago
      We’re getting there but it takes time to agree on what the best implementation of a reinvented wheel looks like? A good example is OpenTelemetry, which is an obvious idea in hindsight but looks like it will take about a decade to ship. Or how we move the goalposts when we reach a goal, for example Kubernetes standardized certain aspects of cloud but now that we have that, instead of celebrating we bemoan its complexity and lack of utility at solving actual application or organization challenges such that we still need to use cloud APIs plus container images plus all this other complexity. But hey, we did solve the problem of distributing code to run on machines, it’s just in hindsight it doesn’t seem like it was that hard? We adjust pretty quick to the “new normal” when it’s not even a decade yet since Docker and Kubernetes appeared on the scene.
      - intelVISA 3y ago
        the main critique of k8s is that its poor design adds complexity to an already challenging concept (for non CS folks)
    - thevagrant 3y ago
      " I see this weird back-lash in modern development against having common, standard platforms. " I think it has always been that way. It comes down to personality types. Many devs I've met think that the implementation they wrote themselves is simpler and easier to understand vs learning a platform api or existing library. They tend to shrug off when I point out security or other potential problems
      - parasti 3y ago
        At least in web development rolling your own is usually the pragmatic choice. It won't break opaquely upon update, you can fix it yourself, it only does what you need. Library and platform updates have a much higher chance of breaking something because of the large impact surface, feature updates being conflated with security updates, insufficient testing, and such breakages are much harder to resolve because they are a black box to you. Really nothing to do with personalities.
- kneebonian 3y ago
  > Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? May I introduce you to inferno or Plan 9 my friend. Elegant OSes from a more civilized age.
- OkayPhysicist 3y ago
  > Or is it time someone started work on a distributed operating system? For stuff like this, we've had it since the 80's: Erlang's (and it's sleek offspring, Elixir) BEAM VM. A virtual machine with concurrent, parallel, and distributed systems in mind? Check. A standard library containing batteries-included solutions to most design and technical challenges you'll run into while building such systems? Check. Tooling for stuff like deployments, diagnostics of running systems, and the ability to pull open a REPL for hands-in-the-meat debugging? Check, Check, Check.
- raincole 3y ago
  > So do we all have to keep reinventing these wheels, but only after a production outage? We live in a world where programmers' "consensus" is that checked exceptions are bad and we need to remove them from Java. People generally just don't care anything except the happies path.
  - speed_spread 3y ago
    That's not the reason to remove checked exceptions from Java. The reason is that exceptions don't compose - they don't play well with functional style code which otherwise works pretty well in Java.
    - Groxx 3y ago
      Yeah. Checked exceptions are fine - it's just an early returning version of error/result-ADTs (which are widely used and loved). Java's implementation isn't generic over exception types, so you're forced to make Bad Decisions constantly.
      - PhilipRoman 3y ago
        Fun fact: you can actually parameterize functions and types over exception types in Java, <T extends Throwable> ... throws T will type check as expected. Of course, if you solve a problem using generics, you will now have 2 problems instead of 1...
        Thiez 3y ago
        And stuff like the Stream api does not use these generics, so you end up wrapping exceptions in RuntimeException anyway, which... again defeats the point of checked exceptions.
        Groxx 3y ago
        I have never tried that, thanks! Does it work for a variable list of things, or would supporting two different checked exceptions require two generic params? If it's one per param I think that's neat but probably limited in use to "your exception can contain a generic value", like `throw NewInsertFailedException(value)`.
- jerf 3y ago
  "So do we all have to keep reinventing these wheels, but only after a production outage?" Lotta cynical replies, and mine is going to sound like one of them at first, but I actually mean it in a relatively deep and profound way: Time is hard. You can even see it in pure math, where Logic is all fun and everyone's having a great time being clever and making all sorts of exciting systems and inferences in those systems... and then you try to build Temporal Logic and all the pretty just goes flying out the door. Even "what if the reply takes ten seconds" is the beginning. By the very nature of the question itself I can infer the response is expected to be small. What if it is large? What if it might legitimately take more than ten seconds to transfer even under ideal circumstances, but you need to know that it's not working as quickly as possible? Is your entry point open to the public? How does it do with slowloris attacks [1]? What if your system simply falls behind due to lack of resources? The difference between 97% capacity and 103% capacity in your real, time-bound systems can knock your socks off in ways you'd never model in an atemporal system that ignored how long things take to happen. Programming would be grungy enough even if we didn't have these considerations, but I'm not even scratching the surface on the number of ways that adding time as a real-world consideration complexifies a ton of things. Our most common response is often just to ignore it. This is... actually often quite rational, a lot of the failure cases can be feasibly addressed by various human interventions, e.g., while writing your service to be robust to "a slow internal network" might be a good idea, there's also a sense in which the only real solution is to speed up the internal network. But still, time is always sitting there crufting things up. One of my favorites is the implicit dependency graph you accidentally start creating once your business systems guys start doing "daily processes" of this and that. We're going to do a daily process to run the bills, but that depends on the four daily dumps that feed the billing process to all have been done first. By the way, did you check that the dumps are actually done and not actually in progress as you're trying to use them? And those four daily dumps each have some other daily processes behind them, and if you're not very careful you'll create loops in those processes which introduce all sorts of other problems... in the end, a set of processes that in perfect atemporal logic land wouldn't be too difficult to deal with becomes something very easy to sleepwalk into a nightmare world, where your dump is scheduled to run between 2:12 and 2:16 and it damned well better not fail for any reason, in your control or out of it, or we're not doing billing today. (Or even the nightmare world where your dump is scheduled to run after 3pm but before 1pm every day... that is, these dependency graphs don't have to get very complicated before literally impossible constraints start to appear if you're not careful!) Trying to explain this to a large number of teams at every level of engineering capability level (frequently going all the down to "a guy who distrusts and doesn't like computers who, against his will, maintains a spreadsheet, which is also one of the vital pillars of our business") is the sort of thing that may make you want to consider becoming a monk. [1]: https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security)
- Sylamore 3y ago
  That's kind of what HP NonStop is, a distributed operating system operating as a huge cluster. If you followed their coding practices and used their native libraries you could almost always do things like freeze a process, move it to an entirely different CPU (which could be a totally different physical server) then restart it without losing the work in progress, processes could auto-restart and resume from the last checkpoint, add more processes to handle the messages in the queue and all kinds of built into the OS and layered services niceties that everyone keeps reinventing.
deleted 3y ago
[deleted]
Sylamore 3y ago
As a capacity planner I tried to argue in favor of tools like Hystrix being built into our middleware services because when we had large IPPV events cascading failures was the biggest risk to our availability and it happened because 99% of the time our services could process any queues before downstream timeouts occured but during high volume events the queues would grow due to nearly instant demand occurring (normally within 5 minutes of the PPV event start time) and causing the queues to go deeper than our timeouts. Combine that with the queues not being durable if a process restart was needed and things got real ugly real fast under extreme load. Automatic retries + deep queues + short timeouts = service issues that take hours to unwind often requiring a coordinated cold restart of the entire middleware pipeline and millions in lost revenue. To compensate we had to scale our systems for the absolute instantaneous peak demand because being legacy systems (pre containers) with a lot of rigid plumbing in them we couldn't just scale on demand. Things did not degrade gracefully once you hit the timeout limits in the composite API calls.
guideamigo 3y ago
Even better would be the scenario when it is stuck forever. Most languages have no default timeout for sending network requests!
intelVISA 3y ago
True but such foundational code is rarely kept synchronous in prod, you'd usually have a coroutine (or equiv) and await the reply, or timeout after X seconds? Installing a random, kinda crappy, third party service like Consul seems overkill imo
- spiffytech 3y ago
  Depending on your request rate and how much higher the latency is than usual, I could see wanting a tool with full-system visibility rather than just independent timeouts at the call site. Some examples of holistic problems: How many partially-processed requests can you hold in memory at once as they pile up under that delay? If the downstream service suddenly fulfills all the pending requests at once, does the thundering herd cause your service to overload other systems?

by tra3 3y ago

jiggawatts 3y ago

As an ardent fan of monoliths and how they generally avoid such tar pits, I have to acknowledge that service-oriented architectures have their uses. So do we all have to keep reinventing these wheels, but only after a production outage? Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? I keep seeing the same patterns being re-engineered over and over. Maybe it’s time to refactor these out…

klooney 3y ago
It's more work, is the simple answer.
- jiggawatts 3y ago
  Once. It's more work once instead of over and over. That's the point of operating systems, standard libraries, and modules! I see this weird back-lash in modern development against having common, standard platforms. I suspect it comes from the Python and JavaScript world, where having "no batteries included" is seen as a good thing, instead of a guaranteed mess of dozens of half-complete incompatible frameworks. I'm coming from the perspective of Windows and comparing it to, say, Azure or AWS. All three have some concepts of access control, log collection, component systems, processes, etc... But all three are proprietary. Kubernetes goes a long way, but it isn't a user-mode system that can be directly accessed from code. Compare with Service Fabric, which has a substantial SDK component that integrates into the applications. As an example, here's a really basic thing that is actually absurdly difficult to solve well: web application session state. If you have sticky load balancing using cookies, then the session state is accessed on one VM something like 99.99% of the time... except for that 1% of the time when it isn't. This could be due to a restart, load rebalancing, or whatever. If you put the session state into something external like Redis, then a zone-redundant deployment will eat a ~1ms delay on every page render, every time. Service Fabric uses a model where it keeps three replicas of the state: one in the original web server, and two replicas in elsewhere. This way, reads are in process on the same VM most of the time, resulting in nanosecond latencies. Writing the state can occur asynchronously after the page response is already being sent. I'd like to see concepts like this, along with all sorts of service-to-service communication patterns, consolidated into an "operating system like platform" designed for the mid-2020s clouds instead of 1990s server farms.
  - lstamour 3y ago
    We’re getting there but it takes time to agree on what the best implementation of a reinvented wheel looks like? A good example is OpenTelemetry, which is an obvious idea in hindsight but looks like it will take about a decade to ship. Or how we move the goalposts when we reach a goal, for example Kubernetes standardized certain aspects of cloud but now that we have that, instead of celebrating we bemoan its complexity and lack of utility at solving actual application or organization challenges such that we still need to use cloud APIs plus container images plus all this other complexity. But hey, we did solve the problem of distributing code to run on machines, it’s just in hindsight it doesn’t seem like it was that hard? We adjust pretty quick to the “new normal” when it’s not even a decade yet since Docker and Kubernetes appeared on the scene.
    - intelVISA 3y ago
      the main critique of k8s is that its poor design adds complexity to an already challenging concept (for non CS folks)
  - thevagrant 3y ago
    " I see this weird back-lash in modern development against having common, standard platforms. " I think it has always been that way. It comes down to personality types. Many devs I've met think that the implementation they wrote themselves is simpler and easier to understand vs learning a platform api or existing library. They tend to shrug off when I point out security or other potential problems
    - parasti 3y ago
      At least in web development rolling your own is usually the pragmatic choice. It won't break opaquely upon update, you can fix it yourself, it only does what you need. Library and platform updates have a much higher chance of breaking something because of the large impact surface, feature updates being conflated with security updates, insufficient testing, and such breakages are much harder to resolve because they are a black box to you. Really nothing to do with personalities.
kneebonian 3y ago
> Or is it time someone started work on a distributed operating system? Vaguely like Kubernetes but full-featured? May I introduce you to inferno or Plan 9 my friend. Elegant OSes from a more civilized age.
OkayPhysicist 3y ago
> Or is it time someone started work on a distributed operating system? For stuff like this, we've had it since the 80's: Erlang's (and it's sleek offspring, Elixir) BEAM VM. A virtual machine with concurrent, parallel, and distributed systems in mind? Check. A standard library containing batteries-included solutions to most design and technical challenges you'll run into while building such systems? Check. Tooling for stuff like deployments, diagnostics of running systems, and the ability to pull open a REPL for hands-in-the-meat debugging? Check, Check, Check.
raincole 3y ago
> So do we all have to keep reinventing these wheels, but only after a production outage? We live in a world where programmers' "consensus" is that checked exceptions are bad and we need to remove them from Java. People generally just don't care anything except the happies path.
- speed_spread 3y ago
  That's not the reason to remove checked exceptions from Java. The reason is that exceptions don't compose - they don't play well with functional style code which otherwise works pretty well in Java.
  - Groxx 3y ago
    Yeah. Checked exceptions are fine - it's just an early returning version of error/result-ADTs (which are widely used and loved). Java's implementation isn't generic over exception types, so you're forced to make Bad Decisions constantly.
    - PhilipRoman 3y ago
      Fun fact: you can actually parameterize functions and types over exception types in Java, <T extends Throwable> ... throws T will type check as expected. Of course, if you solve a problem using generics, you will now have 2 problems instead of 1...
      - Thiez 3y ago
        And stuff like the Stream api does not use these generics, so you end up wrapping exceptions in RuntimeException anyway, which... again defeats the point of checked exceptions.
      - Groxx 3y ago
        I have never tried that, thanks! Does it work for a variable list of things, or would supporting two different checked exceptions require two generic params? If it's one per param I think that's neat but probably limited in use to "your exception can contain a generic value", like `throw NewInsertFailedException(value)`.
jerf 3y ago
"So do we all have to keep reinventing these wheels, but only after a production outage?" Lotta cynical replies, and mine is going to sound like one of them at first, but I actually mean it in a relatively deep and profound way: Time is hard. You can even see it in pure math, where Logic is all fun and everyone's having a great time being clever and making all sorts of exciting systems and inferences in those systems... and then you try to build Temporal Logic and all the pretty just goes flying out the door. Even "what if the reply takes ten seconds" is the beginning. By the very nature of the question itself I can infer the response is expected to be small. What if it is large? What if it might legitimately take more than ten seconds to transfer even under ideal circumstances, but you need to know that it's not working as quickly as possible? Is your entry point open to the public? How does it do with slowloris attacks [1]? What if your system simply falls behind due to lack of resources? The difference between 97% capacity and 103% capacity in your real, time-bound systems can knock your socks off in ways you'd never model in an atemporal system that ignored how long things take to happen. Programming would be grungy enough even if we didn't have these considerations, but I'm not even scratching the surface on the number of ways that adding time as a real-world consideration complexifies a ton of things. Our most common response is often just to ignore it. This is... actually often quite rational, a lot of the failure cases can be feasibly addressed by various human interventions, e.g., while writing your service to be robust to "a slow internal network" might be a good idea, there's also a sense in which the only real solution is to speed up the internal network. But still, time is always sitting there crufting things up. One of my favorites is the implicit dependency graph you accidentally start creating once your business systems guys start doing "daily processes" of this and that. We're going to do a daily process to run the bills, but that depends on the four daily dumps that feed the billing process to all have been done first. By the way, did you check that the dumps are actually done and not actually in progress as you're trying to use them? And those four daily dumps each have some other daily processes behind them, and if you're not very careful you'll create loops in those processes which introduce all sorts of other problems... in the end, a set of processes that in perfect atemporal logic land wouldn't be too difficult to deal with becomes something very easy to sleepwalk into a nightmare world, where your dump is scheduled to run between 2:12 and 2:16 and it damned well better not fail for any reason, in your control or out of it, or we're not doing billing today. (Or even the nightmare world where your dump is scheduled to run after 3pm but before 1pm every day... that is, these dependency graphs don't have to get very complicated before literally impossible constraints start to appear if you're not careful!) Trying to explain this to a large number of teams at every level of engineering capability level (frequently going all the down to "a guy who distrusts and doesn't like computers who, against his will, maintains a spreadsheet, which is also one of the vital pillars of our business") is the sort of thing that may make you want to consider becoming a monk. [1]: https://en.wikipedia.org/wiki/Slowloris_(computer_security) https://en.wikipedia.org/wiki/Slowloris_(computer_security)
Sylamore 3y ago
That's kind of what HP NonStop is, a distributed operating system operating as a huge cluster. If you followed their coding practices and used their native libraries you could almost always do things like freeze a process, move it to an entirely different CPU (which could be a totally different physical server) then restart it without losing the work in progress, processes could auto-restart and resume from the last checkpoint, add more processes to handle the messages in the queue and all kinds of built into the OS and layered services niceties that everyone keeps reinventing.

deleted 3y ago

[deleted]

Sylamore 3y ago

As a capacity planner I tried to argue in favor of tools like Hystrix being built into our middleware services because when we had large IPPV events cascading failures was the biggest risk to our availability and it happened because 99% of the time our services could process any queues before downstream timeouts occured but during high volume events the queues would grow due to nearly instant demand occurring (normally within 5 minutes of the PPV event start time) and causing the queues to go deeper than our timeouts. Combine that with the queues not being durable if a process restart was needed and things got real ugly real fast under extreme load. Automatic retries + deep queues + short timeouts = service issues that take hours to unwind often requiring a coordinated cold restart of the entire middleware pipeline and millions in lost revenue. To compensate we had to scale our systems for the absolute instantaneous peak demand because being legacy systems (pre containers) with a lot of rigid plumbing in them we couldn't just scale on demand. Things did not degrade gracefully once you hit the timeout limits in the composite API calls.

guideamigo 3y ago

Even better would be the scenario when it is stuck forever. Most languages have no default timeout for sending network requests!

intelVISA 3y ago

True but such foundational code is rarely kept synchronous in prod, you'd usually have a coroutine (or equiv) and await the reply, or timeout after X seconds? Installing a random, kinda crappy, third party service like Consul seems overkill imo

spiffytech 3y ago
Depending on your request rate and how much higher the latency is than usual, I could see wanting a tool with full-system visibility rather than just independent timeouts at the call site. Some examples of holistic problems: How many partially-processed requests can you hold in memory at once as they pile up under that delay? If the downstream service suddenly fulfills all the pending requests at once, does the thundering herd cause your service to overload other systems?