7 ms·

On Building Glue Systems

72 points by poga 5y ago

deleted 5y ago
[deleted]
john-doe 5y ago
> Since a glue system is about connecting many isolated pieces into a coordinated abomination (…) Sounds like modern web development.
- Joker_vD 5y ago
  Or shell scripting, legacy or modern, doesn't matter.
krossitalk 5y ago
This starts out strong and then trails off without a real conclusion or opinion on glue systems. A bad build system or lack of maintenance can happen to any project.
- xwowsersx 5y ago
  Agreed. I'm kind of surprised to see this post so high up on the front page here.
shireboy 5y ago
Not sure if this is exactly what the article refers to, but at one of my customers I helped create an app literally called Glue. It has things like a task scheduler for scheduling http posts to developers apps, or monitoring email boxes and posting to web hooks when email comes in. It _does_ need maintenance but I think it beats putting that logic in each app.
- nicktorba 5y ago
  There seem to be a lot of components like this that need to be developed. For example, there are plenty of tools to create ML model endpoints (send data, receive prediction) but many times, those models are not customer/user facing. They will be hit by some other program/system (often on a regularly timed schedule). One solution is to embed the model in the application that needs the predictions, but this can get hairy when it comes to updates/maintanence/etc. Over the next few years, we will see some generalized "model invocation" components that integrate with common data sources for these use cases. You can see an inkling of this with the seldon-core kafka integration: https://docs.seldon.io/projects/seldon-core/en/stable/streaming/kafka.html https://docs.seldon.io/projects/seldon-core/en/stable/stream...
sigmonsays 5y ago
is kubernetes considered a glue system?
- bovermyer 5y ago
  Not really, in my opinion. It's a platform. I think a glue system is meant to connect disparate pieces that otherwise could not interact with each other.
- chubot 5y ago
  I'd definitely say so. Combined with Docker, its job is to "tame" / coordinate / connect extremely heterogeneous software, much of which wasn't designed to run on clusters. I think of shell as the language for glue (and we need a better one). and I explored this relationship in a followup to a popular post [1]: http://www.oilshell.org/blog/2021/07/cloud-review.html http://www.oilshell.org/blog/2021/07/cloud-review.html [1] https://news.ycombinator.com/item?id=27903720 https://news.ycombinator.com/item?id=27903720
- dehrmann 5y ago
  It's large and designed to the point that no, it isn't, but interestingly, it usually replaces a lot of glue systems.
- coding123 5y ago
  Seems like people think so, but I would say no, it's just a platform that runs VMs. It doesn't really care about the data running through services other than making sure it makes it to the destination. That being said, it's quickly becoming easier to plug and play traditional message busses or queuing systems by deploying a container to it. So it helps simplify the addition of some kind of glue system.
PaulHoule 5y ago
The article makes the interesting claim that Clojure, Elixir and similar languages are good for "glue" but doesn't make any effort to develop the argument. For a long-time we've heard the opinion that complex systems are best written in a combination of a "systems" language (say C++ or Java) and a "scripting" language (say Tcl, Lua or Clojure.) That's how video games are written, for instance. Clojure and Elixir have the particular angle of being suitable for concurrency, which more conventional scripting languages (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various global interpreter locks.
- Lammy 5y ago
  > which more conventional scripting languages (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various global interpreter locks For what it's worth, each instance of the new "Ractor" concurrency primitive in Ruby 3 has its own interpreter lock: https://docs.ruby-lang.org/en/master/doc/ractor_md.html https://docs.ruby-lang.org/en/master/doc/ractor_md.html
  - ptx 5y ago
    Something similar is on the way for Python with subinterpreters: https://pythondev.readthedocs.io/subinterpreters.html https://pythondev.readthedocs.io/subinterpreters.html
- marcinzm 5y ago
  >global interpreter locks. Does this really matter for a glue language in a distributed system? Async for non-blocking IO and then just run multiple processes. Allows for things like killing a process and letting the OS cleanup. For example, in Java you're not supposed to just kill a thread.
  - PaulHoule 5y ago
    It's an interesting question. I've written a lot of "glue" in Python using asyncio, websockets, AQMP and similar things for my IoT system at home. The coding is fun and the performance seems excellent, but that's because the workload is low. If I was at the point where I needed more than one core worth of performance I wouldn't be so happy. One case where I am a little frustrated with Python now is a "batch job" that processes and resizes images. There are numerous half-baked ways to farm out the work to separate processes. These would be "good enough" but they are still half baked, and the frameworks that give a little structure to the process have one thing in common: they don't run on Windows. In a pure "glue" case there is also an Amdahl's law kind of situation: it's not crazy to have a 32-core or more system today and a single-threaded "glue" system that dispatches tasks will have a hard time keeping those cores busy if it is spending even 2% of the CPU effort that it takes to do the real work.
tonyhb 5y ago
Been thinking about glue systems a bunch - the platform I'm working on (Inngest [0]) solves this kind of problem. I think "glue systems" can be generalized by saying: systems that consolidate events, and react to events in real-time. Most everything that happens in your system is an event, and things need to react to this. For example: - A glue system will almost certainly plumb stripe events info a billing system, plus other systems for things like failed payments. This system shouldn't go down (HA). It should be able to react to changes easily - as biz ops change frequently. It should be auditable, easy to debug, and easy to deploy. The first pass at this in startups is usually... just doing stuff in the API request, or building webhook endpoints that manage this for you within the API. Then, you might add Sidekiq, or SQS/Lambda. Then, you might build out event-driven architecture using SQS/Kafka. It's kind of a pain, and still it's not easy to grok, debug, or architect. Inngest handles all of this for you, though. It consolidates events from every system (internal and external, via APIs, webhooks, oauth, etc.) and then allows you to run DAG-based workflows in real-time whenever events are received. With full logging, debugging (step-over debugging), retries, user-auditing, changelogs, version control, etc. I think that event-driven systems and glue-based systems overall haven't had much love in the developer UX, but I'm hopeful we can change that. If you're interested in using us or working with us on building the platform, ping me :) [0] https://www.inngest.com https://www.inngest.com
- tut-urut-utut 5y ago
  So many words to just say ESB. I know it's not fancy and falling out of fashion, but what you describe here is what different ESB/MIddleware systems have been doing for ages.
  - tonyhb 5y ago
    Yeah, agree, there's a large sector of overlap. Basically that, but modern, and built for engineers with a polished experience.
- lukeramsden 5y ago
  Inngest looks very cool - is there any chance of this being an "open-source SaaS" / open-core type thing like, for example, TimescaleDB or Temporal? Would love to contribute to something like this, but not as paid employment.
zaphar 5y ago
I don't understand what he means with the build systems comment? What does that have to do with Glue systems? I could speculate that he wishes the build systems accounted for all the supposedly siloed dependent systems in some way but it's not clear what exactly his complaint here is.
pphysch 5y ago
The top two comments as of now, paraphrasing: 1) "Complex systems should be designed like video games" 2) "Complex systems should be designed around a huge event bus SaaS" Video games/engines are architected around responsiveness (low latency) & content creation. Sending every little event to a remote SaaS for processing is sure to add tremendous latency. To be fair, many video games due make heavy use of event processing/message passing. But the architecture of a video game event system & SaaS event bus are quite different for performance, security, reliability reasons. Complex distributed systems are, well, complex, and can't be reduced to a simple formula or architecture. Better to spend time accurately understanding the problem that is being solved.
jusonchan81 5y ago
We have been using Netflix Conductor and it has a ton of features that helps with glueing systems.

72 points by poga 5y ago

deleted 5y ago

[deleted]

> Since a glue system is about connecting many isolated pieces into a coordinated abomination (…) Sounds like modern web development.

Joker_vD 5y ago
Or shell scripting, legacy or modern, doesn't matter.

krossitalk 5y ago

This starts out strong and then trails off without a real conclusion or opinion on glue systems. A bad build system or lack of maintenance can happen to any project.

xwowsersx 5y ago
Agreed. I'm kind of surprised to see this post so high up on the front page here.

shireboy 5y ago

Not sure if this is exactly what the article refers to, but at one of my customers I helped create an app literally called Glue. It has things like a task scheduler for scheduling http posts to developers apps, or monitoring email boxes and posting to web hooks when email comes in. It _does_ need maintenance but I think it beats putting that logic in each app.

nicktorba 5y ago
There seem to be a lot of components like this that need to be developed. For example, there are plenty of tools to create ML model endpoints (send data, receive prediction) but many times, those models are not customer/user facing. They will be hit by some other program/system (often on a regularly timed schedule). One solution is to embed the model in the application that needs the predictions, but this can get hairy when it comes to updates/maintanence/etc. Over the next few years, we will see some generalized "model invocation" components that integrate with common data sources for these use cases. You can see an inkling of this with the seldon-core kafka integration: https://docs.seldon.io/projects/seldon-core/en/stable/streaming/kafka.html https://docs.seldon.io/projects/seldon-core/en/stable/stream...

sigmonsays 5y ago

is kubernetes considered a glue system?

bovermyer 5y ago
Not really, in my opinion. It's a platform. I think a glue system is meant to connect disparate pieces that otherwise could not interact with each other.
chubot 5y ago
I'd definitely say so. Combined with Docker, its job is to "tame" / coordinate / connect extremely heterogeneous software, much of which wasn't designed to run on clusters. I think of shell as the language for glue (and we need a better one). and I explored this relationship in a followup to a popular post [1]: http://www.oilshell.org/blog/2021/07/cloud-review.html http://www.oilshell.org/blog/2021/07/cloud-review.html [1] https://news.ycombinator.com/item?id=27903720 https://news.ycombinator.com/item?id=27903720
dehrmann 5y ago
It's large and designed to the point that no, it isn't, but interestingly, it usually replaces a lot of glue systems.
coding123 5y ago
Seems like people think so, but I would say no, it's just a platform that runs VMs. It doesn't really care about the data running through services other than making sure it makes it to the destination. That being said, it's quickly becoming easier to plug and play traditional message busses or queuing systems by deploying a container to it. So it helps simplify the addition of some kind of glue system.

PaulHoule 5y ago

The article makes the interesting claim that Clojure, Elixir and similar languages are good for "glue" but doesn't make any effort to develop the argument. For a long-time we've heard the opinion that complex systems are best written in a combination of a "systems" language (say C++ or Java) and a "scripting" language (say Tcl, Lua or Clojure.) That's how video games are written, for instance. Clojure and Elixir have the particular angle of being suitable for concurrency, which more conventional scripting languages (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various global interpreter locks.

Lammy 5y ago
> which more conventional scripting languages (e.g. Python, Ruby, PHP, ...) fall down at thanks to the various global interpreter locks For what it's worth, each instance of the new "Ractor" concurrency primitive in Ruby 3 has its own interpreter lock: https://docs.ruby-lang.org/en/master/doc/ractor_md.html https://docs.ruby-lang.org/en/master/doc/ractor_md.html
- ptx 5y ago
  Something similar is on the way for Python with subinterpreters: https://pythondev.readthedocs.io/subinterpreters.html https://pythondev.readthedocs.io/subinterpreters.html
marcinzm 5y ago
>global interpreter locks. Does this really matter for a glue language in a distributed system? Async for non-blocking IO and then just run multiple processes. Allows for things like killing a process and letting the OS cleanup. For example, in Java you're not supposed to just kill a thread.
- PaulHoule 5y ago
  It's an interesting question. I've written a lot of "glue" in Python using asyncio, websockets, AQMP and similar things for my IoT system at home. The coding is fun and the performance seems excellent, but that's because the workload is low. If I was at the point where I needed more than one core worth of performance I wouldn't be so happy. One case where I am a little frustrated with Python now is a "batch job" that processes and resizes images. There are numerous half-baked ways to farm out the work to separate processes. These would be "good enough" but they are still half baked, and the frameworks that give a little structure to the process have one thing in common: they don't run on Windows. In a pure "glue" case there is also an Amdahl's law kind of situation: it's not crazy to have a 32-core or more system today and a single-threaded "glue" system that dispatches tasks will have a hard time keeping those cores busy if it is spending even 2% of the CPU effort that it takes to do the real work.

tonyhb 5y ago

Been thinking about glue systems a bunch - the platform I'm working on (Inngest [0]) solves this kind of problem. I think "glue systems" can be generalized by saying: systems that consolidate events, and react to events in real-time. Most everything that happens in your system is an event, and things need to react to this. For example: - A glue system will almost certainly plumb stripe events info a billing system, plus other systems for things like failed payments. This system shouldn't go down (HA). It should be able to react to changes easily - as biz ops change frequently. It should be auditable, easy to debug, and easy to deploy. The first pass at this in startups is usually... just doing stuff in the API request, or building webhook endpoints that manage this for you within the API. Then, you might add Sidekiq, or SQS/Lambda. Then, you might build out event-driven architecture using SQS/Kafka. It's kind of a pain, and still it's not easy to grok, debug, or architect. Inngest handles all of this for you, though. It consolidates events from every system (internal and external, via APIs, webhooks, oauth, etc.) and then allows you to run DAG-based workflows in real-time whenever events are received. With full logging, debugging (step-over debugging), retries, user-auditing, changelogs, version control, etc. I think that event-driven systems and glue-based systems overall haven't had much love in the developer UX, but I'm hopeful we can change that. If you're interested in using us or working with us on building the platform, ping me :) [0] https://www.inngest.com https://www.inngest.com

tut-urut-utut 5y ago
So many words to just say ESB. I know it's not fancy and falling out of fashion, but what you describe here is what different ESB/MIddleware systems have been doing for ages.
- tonyhb 5y ago
  Yeah, agree, there's a large sector of overlap. Basically that, but modern, and built for engineers with a polished experience.
lukeramsden 5y ago
Inngest looks very cool - is there any chance of this being an "open-source SaaS" / open-core type thing like, for example, TimescaleDB or Temporal? Would love to contribute to something like this, but not as paid employment.

zaphar 5y ago

I don't understand what he means with the build systems comment? What does that have to do with Glue systems? I could speculate that he wishes the build systems accounted for all the supposedly siloed dependent systems in some way but it's not clear what exactly his complaint here is.

pphysch 5y ago

The top two comments as of now, paraphrasing: 1) "Complex systems should be designed like video games" 2) "Complex systems should be designed around a huge event bus SaaS" Video games/engines are architected around responsiveness (low latency) & content creation. Sending every little event to a remote SaaS for processing is sure to add tremendous latency. To be fair, many video games due make heavy use of event processing/message passing. But the architecture of a video game event system & SaaS event bus are quite different for performance, security, reliability reasons. Complex distributed systems are, well, complex, and can't be reduced to a simple formula or architecture. Better to spend time accurately understanding the problem that is being solved.

jusonchan81 5y ago

We have been using Netflix Conductor and it has a ton of features that helps with glueing systems.