9 ms·
A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang b
by pphysch 4y ago
A bit disappointed by the architecture -- it's a Django stack with MySQL, Redis, RabbitMQ, and Celery -- for what is effectively AlertManager (a single golang binary) with a nicer web frontend + Grafana integration + etc.
I'm curious why/if this architecture was chosen. I get that it started as a standalone product (Amixr), but in the current state it is hard to rationalize deploying this next to Grafana in my current containerless setting.
- lazyant 4y agoCurious as to what architecture you would have preferred or why this pretty standard stack (that can be deployed to k8s) is not giving you.
- chrisandchris 4y agoNot OP, but one may interpret your response as "I don't understand why you prefer a single binary over this architecture that requires 6 different services and prefers k8s". IMHO, OP just stated that one could solve this with less dependencies and have the same (if not a better) result.
- pphysch 4y agoYes, thank you. I would be surprised if this same product couldn't be delivered with just Python(Django) + SQLite + Redis (assuming writing everything in Go is unrealistic). Spinning up a venv and launching a local Redis instance is significantly more reasonable than having to configure MySQL, RabbitMQ, and Celery.
- lazyant 4y agoI missed that interpretation :( IMHO a fat binary written from scratch would have been a way worse choice than to use a standard stack, both in terms of bugs and time, let alone Open Source contributions or any scalability. In terms of number of services, what do you get rid of that produce a better result? maybe RMQ and use a worse queue?, celery and write your own task manager or use another dependency?
- theptip 4y agoFor a simple low-scale app you can often do without Redis and Celery/RMQ if you just push everything into Postgres. Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.
- gjulianm 4y agoI bet quite a lot, probably at least 10-50 per second without doing anything special for performance, i.e. multiple queries per alert, calling different APIs, things like that. I don't know of many places that are dealing with alerts measured in "per second" as a unit. Not to mention that having multiple components doesn't mean it's "scalable" by default, it could happen that some part of the pipeline doesn't like multiple instances of something.
- picozeta 4y agoHow does a message queue work via Postgres? Many people (including me) use Redis to run background jobs.
- infogulch 4y agolmgtfy https://www.crunchydata.com/blog/message-queuing-using-native-postgresql https://www.crunchydata.com/blog/message-queuing-using-nativ...
- minusf 4y agohttps://github.com/procrastinate-org/procrastinate https://github.com/procrastinate-org/procrastinate https://github.com/gavinwahl/django-postgres-queue https://github.com/gavinwahl/django-postgres-queue
- theptip 4y agoHere's the option I'm familiar with (siblings have others too): https://github.com/malthe/pq https://github.com/malthe/pq Doesn't have all the plumbing you'd want, there is a wrapper (https://github.com/bretth/django-pq/ https://github.com/bretth/django-pq/) that seems to give you an entrypoint command more like `celery worker ...` but I've not investigated it closely.
- gjulianm 4y agoInstallation in a regular system without Kubernetes? Right now I can install Grafana, Prometheus and Alertmanager in a regular Linux system using distribution packages, and just worry about those programs themselves. If I want to install OnCall, I need not only OnCall plus four other non-trivial dependencies that will still need configuration, management and troubleshooting. All for something that is going to deal with far less load than any of Grafana/Prometheus/Alertmanager. I honestly do not understand it.
- lazyant 4y agoyou can install this stack without kubernetes no? I don't see anything k8s-specific
- gjulianm 4y agoThe problem still stands of adding dependencies, extra complexity and configuration. I’m usually happy about Grafana/Prometheus deployments because the base installation is fairly simple and self-contained, but this looks like a bit of a mess.
- heavyset_go 4y agoYes, there is nothing Kubernetes specific here, and this can be deployed using whatever container orchestration system you want.
- pphysch 4y agoAny of the following: Python(Django)+Redis+[SQLite] Python(Django)+Postgres [Compiled Go binary]+[SQLite] SQLite barely even counts as an architectural dependency TBH :)
- vhold 4y agoAlertManager is one component of a more complicated infrastructure. https://prometheus.io/docs/introduction/overview/#architecture https://prometheus.io/docs/introduction/overview/#architectu... https://kubernetes.io/docs/concepts/overview/components/ https://kubernetes.io/docs/concepts/overview/components/
- pphysch 4y agoOnCall also does nothing unless you have something external firing alerts for you. They both fill similar niches in a larger monitoring system; this does not excuse OnCall having a drastically more complex internal architecture.
- Too 4y agoThat picture shows the complete monitoring stack with lots of optional components (pushgw, service discovery, Grafana). When comparing to OnCall you need OnCall AND still the rest of that Prometheus picture. Compare with this picture where everything in the leftmost Alert Detection box is what you see in the Prometheus architecture. https://grafana.com/docs/oncall/latest/getting-started/ https://grafana.com/docs/oncall/latest/getting-started/
- skullone 4y agoThat seems like a perfectly reasonable architecture. If only all of us could work on battle tested components like those during our job!
- contravariant 4y agoFor something that is supposed to add some more features to the basic email/HTTP message alert like grafana generates, I do wonder what extra features require an additional 2 databases, a message queue and a separate task queue.
- skullone 4y agoprobably keeps history, state, escalation flow, etc?
- MarquesMa 4y agoThis. I find open source projects written in Go or Rust are usually more pleasant to work with than Java, Django or Rails, etc. They have less clunky dependencies, are less resource-hungry, and can ship with single executables which make people's life much easier. Just think about Gitea vs GitLab.
- matsemann 4y agoNot sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment. As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes. Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.
- FridgeSeal 4y agoPython application deployments are all fun and games until suddenly the documentation starts unironically suggesting that you should “write your configuration as a Python script” that should get mounted to some random specific directory within the app as if that could ever be a sane and rational idea.
- Volundr 4y ago> Not sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment. Maybe I'm behind the times, but I can't figure out what you mean here. As far as I know 'java -jar' or servlets are still the most common ways of running a Java app. Are you talking graal and native image?
- matsemann 4y agoFor deploying your own stuff, most people do as before, yes. But even then, it's at least still only a single jar file, containing all dependencies. Not like a typical python project where they ask you to run some command to fetch dependencies and you have to pray it will work on your system. But using jlink for java, one can package everything to a smaller runtime distributed together with the application. So then I feel it will be not much different than a Go executable. > The generated JRE with your sample application does not have any other dependencies... > You can distribute your application bundled with the custom runtime in custom-runtime. It includes your application. From the guide here https://access.redhat.com/documentation/en-us/openjdk/11/html-single/using_jlink_to_customize_java_runtime_environment/index https://access.redhat.com/documentation/en-us/openjdk/11/htm...
- motakuk 4y agoI agree that multi-component architecture is harder to deploy. We did our best and prepared tooling to make deployment an easy thing. Helm (https://github.com/grafana/oncall/tree/dev/helm/oncall https://github.com/grafana/oncall/tree/dev/helm/oncall), docker-composes for hobby and dev environments. Besides deployment, there are two main priorities for OnCall architecture: 1) It should be as "default" as possible. No fancy tech, no hacking around 2) It should deliver notifications no matter what. We chose the most "boring" (no offense Django community, that's a great quality for a framework) stack we know well: Django, Rabbit, Celery, MySQL, Redis. It's mature, reliable, and allows us to build a message bus-based pipeline with reliable and predictable migrations. It's important for such a tool to be based on message bus because it should have no single point of failure. If worker will die, the other will pick up the task and deliver alert. If Slack will go down, you won't loose your data. It will continue delivering to other destinations and will deliver to Slack once it's up. The architecture you see in the repo was live for 3+ years now. We were able to perform a few hundreds of data migrations without downtimes, had no major downtimes or data loss. So I'm pretty happy with this choice.
- Deritio 4y agoHearing your message bus assumption sounds like one of the most ridiculous claims I heard. Sorry but why is rabbitmq really necessary?
- gen220 4y agoI think your decisions were reasonable, as is the opinion of the person you're responding to. To be fair, even in its current form, it should be possible to operate this system with sqlite (i.e. no db server) and in-process celery workers (i.e. no rabbit MQ) if configured correctly, assuming they're not using MySQL-specific features in the app. Using a message bus, a persistent data store behind a SQL interface, and a caching layer are all good design choices. I think the OP's concern is less with your particular implementations, and more with the principle of preventing operators from bringing their own preferred implementation of those interfaces to the table. They mentioned that it makes sense because you were a standalone product, so stack portability was less of a concern. But as FOSS, you're opening yourself up to different standards on portability. It requires some work on the maintainer to make the application tolerant to different fulfillments of the same interfaces. But it's good work. It usually results in cleaner separation of concerns between application logic and caching/message bus/persistence logic, for one. It also allows your app to serve a wider audience: for example, those who are locked-in to using Postgres/Kafka/Memcached.
- minusf 4y agonot gonna argue that a single binary is the ultimate deploy solution but running a django app is not that difficult (although i am biased cause i do that for a living). i love django projects but mysql, celery and rabbitmq -- no thanks.
- pphysch 4y agoDon't get me wrong, I love Django and think its a great framework for writing internal tools like this. Redis gets a pass too since Django has native support for it in 4.0+. It's really the (IMHO unnecessary) combo of MySQL+RabbitMQ+Celery that turns me off. Redis itself has had solid support for building reliable distributed task streaming for nearly 4 years (Redis ConsumerGroups introduced in 2018).
- mkl95 4y ago> Django stack with MySQL, Redis, RabbitMQ, and Celery MySQL is a weird if not slightly disturbing choice. Other than that it's a boring, battle-tested stack that is relatively easy to scale. I agree that Go is nicer, but I'm biased by several years of dealing with horrific Flask / Django projects.
- airstrike 4y ago> several years of dealing with horrific Flask / Django projects. I think you misspelled "beautiful"
- goodpoint 4y agoThat's very bad. 99% of organizations don't have a volume of alerts that justifies any of MySQL, Redis and RabbitMQ. Complexity comes at a steep price when something critical (e.g. OnCall) breaks and you have to debug it in a hurry. Shoving everything in a container and closing the lid does not help.
- alex_dev 4y agoOne of the most frustrating aspects of being a software engineer is dealing with others that love to over-engineer. Unfortunately, they make enough noise that complex solutions are necessary that it gets managers scared about taking any easier, simpler solutions.
- deleted 4y ago[deleted]
- heavyset_go 4y agoThat's a tried and true stack, and a very good one for maintaining sane levels of reliability, consistency, durability etc. Resource wise, at least with Celery, RabbitMQ and Django, they're also pretty lean. It even ships in containers along with Docker Compose files and Helm charts, which would suit the deployment use cases of 99% of users. I understand that you're not using containers, but I don't think that's a limitation that many are inflicting upon themselves as of late, and if pressed, installing Docker Compose takes about 5 minutes and you don't have to think about it again.
- pphysch 4y ago> Docker Compose takes about 5 minutes and you don't have to think about it again Except when you need to pin and repin versions to comply with a security policy, which may be why you aren't even running containers in the first place