6 ms·

One think that would make wonder is a company wide event queue. Just imagine all the things that it would be possible to make automatic if all the events that

by siscia 6y ago

One think that would make wonder is a company wide event queue. Just imagine all the things that it would be possible to make automatic if all the events that happens inside a business environment where in a queue. And don't even think about how it would be great if the same queue were open to outside providers.

capableweb 6y ago
Solid idea, worth thinking more about. A FIFO queue of tasks that anyone can grab from at any time. Main drawback is that each task would have to have a proper description, context and definition of done, otherwise it's bound to fail. Once you have those, any tool would do. But usually those three things are the hard part, not the tool/technology.
janekm 6y ago
I once had the idea for a global event queue, to enable new types of application interaction. Then twitter came out and I thought "oh well, that basically does what I was thinking of". Didn't quite turn out that way when they messed up the APIs, and then IFTTT did a better version of that concept...
ojhughes 6y ago
Enterprise Service Bus had this goal, and it mostly failed as a standard due to its complexity. Arguably, HATEOS has a similar goal by making APIs universally discoverable
- the_af 6y ago
  A similar concept lives on in the form of event streaming and pub/sub. I'm not sure how HATEOAS is related though? This is about publishing and subscribing to events (or streams of data), not about API discovery or "resources" as documents. Or did you mean something else?
Galanwe 6y ago
That's called an ESB (enterprise software bus), a lot of companies have that, and there's a lot of softwares already on that space (IBM has quite a good market share there). Some companies do roll their own though, I've seen quite a few implementing ESBs on top of 0mq for instance. My previous company was doing trading and made the outgoing orders available on the ESB. It's was quite handy since all teams could subscribe to the trade topic and do their own processing: trade checks, accounting, risk, etc. It's a bit old school though, I mainly see old corporations with that architecture. The thing is, it ages quite badly most the time. Fast forward 5 years, and every single team is publishing stuff on the ESB "cause it's super easy to publish and consume data from the ESB". Before you know it, you have a full team dedicated to scaling and maintaining the monster.
- alexisread 6y ago
  Sounds like Kafka? (Only half joking)
  - the_af 6y ago
    It's not a joke. An ESB and event-streaming like in Kafka are very related concepts. Some details vary, but they share a lot of the same ideas and live in the same "problem space".
- golergka 6y ago
  Do you mean that these scaling problems hapenned because ESB was misused? Or would it occur with any architecture or architectures (if different teams used different instruments) just because of sheer volume of data that needs to be processed by business logic? May be I'm bad at phrasing this, but the point is, sometimes you have perfomance problems because you could write better code, or use better tech, or tweak requirements a little bit, or do something else a little smarted. But sometimes you have perfomance problems simply because you just do a lot of actual work.
  - Galanwe 6y ago
    I think these problems occurred because an ESB is too easy to over-use. Once you have it, it's very tempting to use it for everything. But of course the more you use it, the more features it needs to have. Let me illustrate that with a example compiling some of the patterns I have come to see at companies that overly used their ESBs: Say Bob is working at a trading company and sets up an ESB where he publishes the daily trades of the company. Soon, the trading desk learns about that, and decide that it's actually cool to use this for trading, they just have to listen on the ESB, and send the trades accordingly. That's the error pattern 1: making the ESB business critical. The next day, the risk team learns about that ESB thing, and decide its very handy to perform post-trading checks by just listening to the trades flowing on the ESB. So they setup a system that listens for trades on the ESB, check that the trade is compliant with some limits, and send an other message on the ESB to let everyone know that this trade is validated. This is error pattern 2: cascading messages triggered from other messages. The week after that, the security team learns about the ESB, and decides its very insecure to let anyone see the trades of the company, so they start implementing access control on the ESB. This is error pattern 3: now you have an overly complex layer on top of the ESB to decide who can see what and who can publish what. Rince and repeat patterns 1, 2,and 3 for 5 years and here is the situation you end up with: - The ESB is not the easy and handy system it was in the beginning. Since it has become the de-facto standard for publishing information in the whole company, it has to support features for _all_ the company use cases. There is access control to publish per topic, access control to listen per topic, multiple bindings of varying quality for each technological stack/language that each team in the company is using. The company of course is not capable / prepared to maintain a software of this scope, so the ESB is crippled with bugs that nobody can fix, because, you know, the infrastructure team cannot fix their groovy scripts using the ESB cause the guy that wrote them left. And the marketing team has some interns using the excel plug-in but they don't have time to rewrite them this year. The ESB is now partly un maintained, because the company relied totally on it without having the capacity / willingness / foresight of understanding how intricate it can be to update something that everyone use. - The ESB is now very slow, because it was so tempting to publish anything of various interest on it that everyone did it. The problem is that the ESB is also critical for the company, so the whole flow of message is now slowly moving and overflowing, requiring endless tuning and tentatives at scaling it better. Of course 80% of the messages on the ESB are actually not listened to by anyone, but since nobody really knows who listens to the published messages, it's very tempting to just _not_ stop programs from publishing, ever, because god knows if some random team at the other end of the company might have a program reading these messages. - You most likely have now an IT team dedicated to maintain the ESB. They are squeezed and pressured by the business teams to keep the ESB fast and easy to stable without requiring them to recode all the crap they plugged on it. On the other end, the other IT teams are pressuring them to update the ESB to support <place your language/stack here>. Of course the ESB team has no incentive to make any improvement whatsoever to the ESB, because that would definitely crash most of the crap the less technical teams of the company plugged to it. But the ESB team is the de facto guardian of the temple of the ESB, so everyone ends up frustrated by the situation. --- I'm not sure I did a good job at explaining the various problems here, but basically, the one size fits all that ESBs are promoting is often not a future proof choice. The reality is that you don't want your whole company coupled to a single system like that. Otherwise your system will be as good as the worst user of it.
stevekemp 6y ago
I think that was the motivation behind the project-wide event-bus/queue that the Fedora distribution started to use: https://lwn.net/Articles/608915/ https://lwn.net/Articles/608915/ https://fedmsg2.readthedocs.io/en/latest/ https://fedmsg2.readthedocs.io/en/latest/ I remember having a play with it a couple of years ago, and it was nice to see the various messages flowing over it.
11235813213455 6y ago
That's how most job queues work (typically for processing periodic, asynchronous and heavy tasks, e.g. emails, ..)
bob1029 6y ago
ESBs and other abstractions are the only way many sorts of highly-complex businesses can be practically tied together and integrated. Think about an automated factory with tens of thousands of tools, each with hundreds or thousands of events, all from differing vendors. The only rational way to tie this stuff together is with some sort of event-driven messaging architecture.
blargmaster33 6y ago
Aka "The backlog"
adamkl 6y ago
That is exactly the premise behind Kafka and LinkedIn's (creator of Kafka) systems architecture. Here's an excellent article on how that architecture looks in practice: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...
- the_af 6y ago
  I also recommend this article! It's entertaining and informative, and a good primer on distributed logs. The reference at the end (which I won't spoil in this comment) made me chuckle.
hoorayimhelping 6y ago
You're describing an Enterprise Service Bus, a concept that has been around and implemented for decades: https://en.wikipedia.org/wiki/Enterprise_service_bus https://en.wikipedia.org/wiki/Enterprise_service_bus Spoiler alert: it's not the beautiful, decoupled silver bullet it seems like. The concept makes a ton of sense, but good execution of it is difficult. The biggest issue I've run into using systems that use this pattern is throughput and latency. Most shops design and deploy their systems in such a way that as soon as there's a little bit of traffic beyond the norm, the whole thing gets clogged and backed up for hours.
bsenftner 6y ago
Visual effects production companies have a studio wide queue, typically called "the queue", and entire film productions are created by digital artists and developers running jobs on their queue that create, polish or otherwise manipulate every shot of every frame of the film, plus trailers never in the film at all.
mrighele 6y ago
This has been possible the healthcare sector for many years with HL7 and later FHIR [1]. Essentially you have a standard protocol and a set of message to exchange health-related information between the different softwares even if they from different vendors. This allows (for example) a doctor with software A to ask for a test on a patient, have a nurse see it on software B, have the sample sent to a laboratory which uses software C and then the result sent back to software A where the doctor can also check it. Meanwhile the signed document has been stored on software D. That said FHIR especially is more than that and allows different approaches. [1] https://www.hl7.org/fhir/ https://www.hl7.org/fhir/
- chapium 6y ago
  HL7 is a shared messaging standard, much like XML or JSON. The shared standard does allow for communication between applications, but its rather unruly.
  - mrighele 6y ago
    I would say that HL7 is more like a XML + a set of XML schemas. In any case, you're right that it is only a messaging standard, but I have always seen it used in the contest of having a central broker like Mirth passing messages around so it is in my opinion a good example of what the parent poster was thinking about. And yes, you're right, it is quite unruly (information being present in one field or another depending on the software and the like). In my experience luckily most of those issues could be handled at the broker level.
Jtsummers 6y ago
Most of the responses are about ESB and similar, but see also the Linda language [0]. Using a shared associative memory called a tuplespace to coordinate between different processes. Each process can be run concurrently (on the same or separate computer systems) and respond as things appear in the memory by transforming the data, triggering actions, etc. [0] https://en.wikipedia.org/wiki/Linda_(coordination_language) https://en.wikipedia.org/wiki/Linda_(coordination_language)
Jolter 6y ago
Look into the Eiffel protocol if you want further examples besides Kafka and ESB.
chapium 6y ago
I think you've described a service ticket management system.

by siscia 6y ago

capableweb 6y ago

Solid idea, worth thinking more about. A FIFO queue of tasks that anyone can grab from at any time. Main drawback is that each task would have to have a proper description, context and definition of done, otherwise it's bound to fail. Once you have those, any tool would do. But usually those three things are the hard part, not the tool/technology.

janekm 6y ago

I once had the idea for a global event queue, to enable new types of application interaction. Then twitter came out and I thought "oh well, that basically does what I was thinking of". Didn't quite turn out that way when they messed up the APIs, and then IFTTT did a better version of that concept...

ojhughes 6y ago

Enterprise Service Bus had this goal, and it mostly failed as a standard due to its complexity. Arguably, HATEOS has a similar goal by making APIs universally discoverable

the_af 6y ago
A similar concept lives on in the form of event streaming and pub/sub. I'm not sure how HATEOAS is related though? This is about publishing and subscribing to events (or streams of data), not about API discovery or "resources" as documents. Or did you mean something else?

Galanwe 6y ago

That's called an ESB (enterprise software bus), a lot of companies have that, and there's a lot of softwares already on that space (IBM has quite a good market share there). Some companies do roll their own though, I've seen quite a few implementing ESBs on top of 0mq for instance. My previous company was doing trading and made the outgoing orders available on the ESB. It's was quite handy since all teams could subscribe to the trade topic and do their own processing: trade checks, accounting, risk, etc. It's a bit old school though, I mainly see old corporations with that architecture. The thing is, it ages quite badly most the time. Fast forward 5 years, and every single team is publishing stuff on the ESB "cause it's super easy to publish and consume data from the ESB". Before you know it, you have a full team dedicated to scaling and maintaining the monster.

alexisread 6y ago
Sounds like Kafka? (Only half joking)
- the_af 6y ago
  It's not a joke. An ESB and event-streaming like in Kafka are very related concepts. Some details vary, but they share a lot of the same ideas and live in the same "problem space".
golergka 6y ago
Do you mean that these scaling problems hapenned because ESB was misused? Or would it occur with any architecture or architectures (if different teams used different instruments) just because of sheer volume of data that needs to be processed by business logic? May be I'm bad at phrasing this, but the point is, sometimes you have perfomance problems because you could write better code, or use better tech, or tweak requirements a little bit, or do something else a little smarted. But sometimes you have perfomance problems simply because you just do a lot of actual work.
- Galanwe 6y ago
  I think these problems occurred because an ESB is too easy to over-use. Once you have it, it's very tempting to use it for everything. But of course the more you use it, the more features it needs to have. Let me illustrate that with a example compiling some of the patterns I have come to see at companies that overly used their ESBs: Say Bob is working at a trading company and sets up an ESB where he publishes the daily trades of the company. Soon, the trading desk learns about that, and decide that it's actually cool to use this for trading, they just have to listen on the ESB, and send the trades accordingly. That's the error pattern 1: making the ESB business critical. The next day, the risk team learns about that ESB thing, and decide its very handy to perform post-trading checks by just listening to the trades flowing on the ESB. So they setup a system that listens for trades on the ESB, check that the trade is compliant with some limits, and send an other message on the ESB to let everyone know that this trade is validated. This is error pattern 2: cascading messages triggered from other messages. The week after that, the security team learns about the ESB, and decides its very insecure to let anyone see the trades of the company, so they start implementing access control on the ESB. This is error pattern 3: now you have an overly complex layer on top of the ESB to decide who can see what and who can publish what. Rince and repeat patterns 1, 2,and 3 for 5 years and here is the situation you end up with: - The ESB is not the easy and handy system it was in the beginning. Since it has become the de-facto standard for publishing information in the whole company, it has to support features for _all_ the company use cases. There is access control to publish per topic, access control to listen per topic, multiple bindings of varying quality for each technological stack/language that each team in the company is using. The company of course is not capable / prepared to maintain a software of this scope, so the ESB is crippled with bugs that nobody can fix, because, you know, the infrastructure team cannot fix their groovy scripts using the ESB cause the guy that wrote them left. And the marketing team has some interns using the excel plug-in but they don't have time to rewrite them this year. The ESB is now partly un maintained, because the company relied totally on it without having the capacity / willingness / foresight of understanding how intricate it can be to update something that everyone use. - The ESB is now very slow, because it was so tempting to publish anything of various interest on it that everyone did it. The problem is that the ESB is also critical for the company, so the whole flow of message is now slowly moving and overflowing, requiring endless tuning and tentatives at scaling it better. Of course 80% of the messages on the ESB are actually not listened to by anyone, but since nobody really knows who listens to the published messages, it's very tempting to just _not_ stop programs from publishing, ever, because god knows if some random team at the other end of the company might have a program reading these messages. - You most likely have now an IT team dedicated to maintain the ESB. They are squeezed and pressured by the business teams to keep the ESB fast and easy to stable without requiring them to recode all the crap they plugged on it. On the other end, the other IT teams are pressuring them to update the ESB to support <place your language/stack here>. Of course the ESB team has no incentive to make any improvement whatsoever to the ESB, because that would definitely crash most of the crap the less technical teams of the company plugged to it. But the ESB team is the de facto guardian of the temple of the ESB, so everyone ends up frustrated by the situation. --- I'm not sure I did a good job at explaining the various problems here, but basically, the one size fits all that ESBs are promoting is often not a future proof choice. The reality is that you don't want your whole company coupled to a single system like that. Otherwise your system will be as good as the worst user of it.

stevekemp 6y ago

I think that was the motivation behind the project-wide event-bus/queue that the Fedora distribution started to use: https://lwn.net/Articles/608915/ https://lwn.net/Articles/608915/ https://fedmsg2.readthedocs.io/en/latest/ https://fedmsg2.readthedocs.io/en/latest/ I remember having a play with it a couple of years ago, and it was nice to see the various messages flowing over it.

11235813213455 6y ago

That's how most job queues work (typically for processing periodic, asynchronous and heavy tasks, e.g. emails, ..)

bob1029 6y ago

ESBs and other abstractions are the only way many sorts of highly-complex businesses can be practically tied together and integrated. Think about an automated factory with tens of thousands of tools, each with hundreds or thousands of events, all from differing vendors. The only rational way to tie this stuff together is with some sort of event-driven messaging architecture.

blargmaster33 6y ago

Aka "The backlog"

adamkl 6y ago

That is exactly the premise behind Kafka and LinkedIn's (creator of Kafka) systems architecture. Here's an excellent article on how that architecture looks in practice: https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying https://engineering.linkedin.com/distributed-systems/log-wha...

the_af 6y ago
I also recommend this article! It's entertaining and informative, and a good primer on distributed logs. The reference at the end (which I won't spoil in this comment) made me chuckle.

hoorayimhelping 6y ago

You're describing an Enterprise Service Bus, a concept that has been around and implemented for decades: https://en.wikipedia.org/wiki/Enterprise_service_bus https://en.wikipedia.org/wiki/Enterprise_service_bus Spoiler alert: it's not the beautiful, decoupled silver bullet it seems like. The concept makes a ton of sense, but good execution of it is difficult. The biggest issue I've run into using systems that use this pattern is throughput and latency. Most shops design and deploy their systems in such a way that as soon as there's a little bit of traffic beyond the norm, the whole thing gets clogged and backed up for hours.

bsenftner 6y ago

Visual effects production companies have a studio wide queue, typically called "the queue", and entire film productions are created by digital artists and developers running jobs on their queue that create, polish or otherwise manipulate every shot of every frame of the film, plus trailers never in the film at all.

mrighele 6y ago

This has been possible the healthcare sector for many years with HL7 and later FHIR [1]. Essentially you have a standard protocol and a set of message to exchange health-related information between the different softwares even if they from different vendors. This allows (for example) a doctor with software A to ask for a test on a patient, have a nurse see it on software B, have the sample sent to a laboratory which uses software C and then the result sent back to software A where the doctor can also check it. Meanwhile the signed document has been stored on software D. That said FHIR especially is more than that and allows different approaches. [1] https://www.hl7.org/fhir/ https://www.hl7.org/fhir/

chapium 6y ago
HL7 is a shared messaging standard, much like XML or JSON. The shared standard does allow for communication between applications, but its rather unruly.
- mrighele 6y ago
  I would say that HL7 is more like a XML + a set of XML schemas. In any case, you're right that it is only a messaging standard, but I have always seen it used in the contest of having a central broker like Mirth passing messages around so it is in my opinion a good example of what the parent poster was thinking about. And yes, you're right, it is quite unruly (information being present in one field or another depending on the software and the like). In my experience luckily most of those issues could be handled at the broker level.

Jtsummers 6y ago

Most of the responses are about ESB and similar, but see also the Linda language [0]. Using a shared associative memory called a tuplespace to coordinate between different processes. Each process can be run concurrently (on the same or separate computer systems) and respond as things appear in the memory by transforming the data, triggering actions, etc. [0] https://en.wikipedia.org/wiki/Linda_(coordination_language) https://en.wikipedia.org/wiki/Linda_(coordination_language)

Jolter 6y ago

Look into the Eiffel protocol if you want further examples besides Kafka and ESB.

chapium 6y ago

I think you've described a service ticket management system.