9 ms·

Real-time notifications from systemd to Slack

70 points by mirceasoaica 10y ago

deleted 10y ago
[deleted]
mfukar 10y ago
If vomiting logs on slack is a tool anybody is considering to manage lifetime and detect crashes, no wonder reliability is a strange word to messaging wannabes. This thing has nothing to do with systems administration, notifications, status updates, or crashing.
nodesocket 10y ago
Interesting service scaledrone. How are you different than pusher, pubnub, or hydna.com?
- raresp 10y ago
  At a very first view I can say it's cheaper. And this is important for startups and small/medium businesses. They also provide examples for a lot of languages, I guess I'm going to try their service. Good job, Scaledrone!
  - SEMW 10y ago
    Shameless plug: if you want something almost as cheap but without having to drop the ability to subscribe from all client libs (not just js), queryable history, presence, connection state recovery, webhooks, stats, firehose to queues, etc, then have a look at https://ably.io https://ably.io. (disclaimer: I work there)
lorenzhs 10y ago
Not to be overly nitpicky here but it's spelled "systemd", not "SystemD". From its website: Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd. It's that simple. https://www.freedesktop.org/wiki/Software/systemd/ https://www.freedesktop.org/wiki/Software/systemd/
- maccard 10y ago
  if you're not being overly nitpicky, what are you being? if you'd clicked on the article you would have seen that the mistake was only in the HN submission title, and not in the title.
  - Raidok 10y ago
    Both of them are fixed now.
- tannhaeuser 10y ago
  Insisting on Unix naming conventions is odd when SystemD tramples over more important Unix system concepts left and right.
orf 10y ago
What about using sentry and getting better stack traces with local variables as well as slack notifications? Stack traces are fine and all, but without locals it's often hard to track down the issue.
Xylakant 10y ago
For a previous setup we built a small script that attaches to dbus and just listens to the notifications and pushes interesting events to our logging service. Once you understand how dbus works, that's actually pretty neat and catches crashes as well as intentional status changes.
Curious42 10y ago
Is Systemd a nohup equivalent? If it is, then what's the benefit of using one over the other?
- fnord123 10y ago
  nohup just runs a process that will ignore SIGHUP. systemd can also run processes that ignore SIGHUP. But systemd does a lot of things that nohup doesn't do. Please don't attempt to use nohup as a daemon management system for anything but the noddiest of tasks. If you're going to use anything, you're probably best choosing from this list: https://en.wikipedia.org/wiki/Operating_system_service_management https://en.wikipedia.org/wiki/Operating_system_service_manag...
  - JdeBP 10y ago
    ... which misses out quite a lot, from s6 through perp to initng.
- koolba 10y ago
  While nohup is used to daemonize a single process, systemd does that and a ton of other stuff. It's a replacement init system that includes everything from daemonization, log handling, auto restarts, and by this time next year will probably make coffee for you as well.
  - bandrami 10y ago
    This is what I've never understood about systemd. Somebody will ask "doesn't inetd already do that?" or "doesn't nohup already do that?" or "doesn't runit already do that?" and the answer is always "yes, but systemd does a bunch of other things something else already did too". shrug Its setup is way, way too brittle for me, but I guess the QR codes are kind of neat.
    - deathanatos 10y ago
      My requirements (for server side daemons) are approximately: start a daemon as a child, and monitor it; if it dies, restart it. Ideally, give me a command-line interface to the managing daemon to gracefully start/stop the child. nohup doesn't do this, making it simply not the right tool for the job. I've never used runit, but it looks closer.
hendzen 10y ago
This will also notify if you intentionally shut down the service. If you want to only see failures, you can use an OnFailure directive.
arjie 10y ago
Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?
- deleted 10y ago
  [deleted]
- hornetblack 10y ago
  systemd has --user option, which makes it a service manager for a particular user, if that's what you mean. I used it for some time to run an emacs daemon. (until I found that it closed between logouts and changed to using the system manager isntead)
  - deleted 10y ago
    [deleted]
  - tophercyll 10y ago
    You can make user level systemd instances run at startup and stick around regardless of login/logout. loginctl enable-linger <username> This enables some nice deployment strategies that don't require root.
    - hornetblack 10y ago
      Thanks. I figured this was possible. I just never got around to looking at loginctl's options.
- geocar 10y ago
  Yes. I used to use runit/daemontools/inittab. My favourite thing about systemd is that it is increasingly available, and while it has it's faults, it has instances (macros you can use to kickoff a fleet of services easily) and pretty good isolation features. It also has a "systemctl-over-ssh" feature which is quite nice, and which allows you to use an .ssh/authorized_keys file instead of sudo to allow access to certain administrative tasks.
praveshjain 10y ago
I don't understand. How is it better than using HTTP POST to post to the slack channel yourself?
- jon-wood 10y ago
  It doesn't require changing the underlying systems that are being managed, giving it a lot more flexibility. So long as the service can log to STDOUT it can be integrated with Slack.
  - praveshjain 10y ago
    Oh I get it. Thanks.
qznc 10y ago
I'm currently building something similar for matrix instead of slack. My idea was to clone the mail(1) interface, but tee(1) is also a nice idea! In the process, I also discovered sendxmpp, which provides mail(1) for XMPP. It does not support encryption and is written in Perl, so I am building my own.
skarap 10y ago
What if slack is down/unreachable? Won't it keep the service from starting?
- lima 10y ago
  If it makes slacktee fail with an error code, yes, it would prevent startup. The fix is a - sign: ExecStartPre=-/usr/bin/foo Would still mark the service as "failed" but wouldn't prevent it from starting.
lima 10y ago
Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.
- notheguyouthink 10y ago
  What would you suggest? We're in need of an alerting platform and making use of Slack, not because it's flawless or anything, but because we're already paying for it and it's convenient for the mass junk. Ie, not strictly mission critical alerts. What medium might be better than Slack? (to be clear, i'm asking, not saying Slack is good for this)
  - amyjess 10y ago
    Zabbix. It's open-source and pretty flexible. I maintain my company's Zabbix installation, and I've come to appreciate it (and 3.x finally has a non-ugly UI!).
    - lima 10y ago
      We use Nagios + CheckMK, but I believe neither this nor Zabbix solve the alerting part. Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat. Not complicated at all, just 100 lines of Python code that contain the business logic. Anything that relies on a single medium is unsuitable for anything but unimportant alerts. What if Slack goes down for 2 hour? Unlikely, but definitely possible. This ensures that every alert is explicitly acknowledged by someone, and that unimportant alerts are quickly forgotten without wondering whether someone handled them or not. We have different applications sending alerts, not just Nagios (because Nagios sucks at processing events as opposed to states), and it would quickly become unmanageable without some sort of middleware.
      - 10y ago

70 points by mirceasoaica 10y ago

deleted 10y ago

[deleted]

If vomiting logs on slack is a tool anybody is considering to manage lifetime and detect crashes, no wonder reliability is a strange word to messaging wannabes. This thing has nothing to do with systems administration, notifications, status updates, or crashing.

nodesocket 10y ago

Interesting service scaledrone. How are you different than pusher, pubnub, or hydna.com?

raresp 10y ago
At a very first view I can say it's cheaper. And this is important for startups and small/medium businesses. They also provide examples for a lot of languages, I guess I'm going to try their service. Good job, Scaledrone!
- SEMW 10y ago
  Shameless plug: if you want something almost as cheap but without having to drop the ability to subscribe from all client libs (not just js), queryable history, presence, connection state recovery, webhooks, stats, firehose to queues, etc, then have a look at https://ably.io https://ably.io. (disclaimer: I work there)

lorenzhs 10y ago

Not to be overly nitpicky here but it's spelled "systemd", not "SystemD". From its website: Yes, it is written systemd, not system D or System D, or even SystemD. And it isn't system d either. Why? Because it's a system daemon, and under Unix/Linux those are in lower case, and get suffixed with a lower case d. And since systemd manages the system, it's called systemd. It's that simple. https://www.freedesktop.org/wiki/Software/systemd/ https://www.freedesktop.org/wiki/Software/systemd/

maccard 10y ago
if you're not being overly nitpicky, what are you being? if you'd clicked on the article you would have seen that the mistake was only in the HN submission title, and not in the title.
- Raidok 10y ago
  Both of them are fixed now.
tannhaeuser 10y ago
Insisting on Unix naming conventions is odd when SystemD tramples over more important Unix system concepts left and right.

orf 10y ago

What about using sentry and getting better stack traces with local variables as well as slack notifications? Stack traces are fine and all, but without locals it's often hard to track down the issue.

Xylakant 10y ago

For a previous setup we built a small script that attaches to dbus and just listens to the notifications and pushes interesting events to our logging service. Once you understand how dbus works, that's actually pretty neat and catches crashes as well as intentional status changes.

Curious42 10y ago

Is Systemd a nohup equivalent? If it is, then what's the benefit of using one over the other?

fnord123 10y ago
nohup just runs a process that will ignore SIGHUP. systemd can also run processes that ignore SIGHUP. But systemd does a lot of things that nohup doesn't do. Please don't attempt to use nohup as a daemon management system for anything but the noddiest of tasks. If you're going to use anything, you're probably best choosing from this list: https://en.wikipedia.org/wiki/Operating_system_service_management https://en.wikipedia.org/wiki/Operating_system_service_manag...
- JdeBP 10y ago
  ... which misses out quite a lot, from s6 through perp to initng.
koolba 10y ago
While nohup is used to daemonize a single process, systemd does that and a ton of other stuff. It's a replacement init system that includes everything from daemonization, log handling, auto restarts, and by this time next year will probably make coffee for you as well.
- bandrami 10y ago
  This is what I've never understood about systemd. Somebody will ask "doesn't inetd already do that?" or "doesn't nohup already do that?" or "doesn't runit already do that?" and the answer is always "yes, but systemd does a bunch of other things something else already did too". shrug Its setup is way, way too brittle for me, but I guess the QR codes are kind of neat.
  - deathanatos 10y ago
    My requirements (for server side daemons) are approximately: start a daemon as a child, and monitor it; if it dies, restart it. Ideally, give me a command-line interface to the managing daemon to gracefully start/stop the child. nohup doesn't do this, making it simply not the right tool for the job. I've never used runit, but it looks closer.

hendzen 10y ago

This will also notify if you intentionally shut down the service. If you want to only see failures, you can use an OnFailure directive.

arjie 10y ago

Can anyone who's used systemd share what it's like to have it for process management? We use monit, and it's all right, but I occasionally wonder. Does anyone run it also as a separate user to manage certain applications? So that you could have certain people log on and operate some things and others operate other things?

deleted 10y ago
[deleted]
hornetblack 10y ago
systemd has --user option, which makes it a service manager for a particular user, if that's what you mean. I used it for some time to run an emacs daemon. (until I found that it closed between logouts and changed to using the system manager isntead)
- deleted 10y ago
  [deleted]
- tophercyll 10y ago
  You can make user level systemd instances run at startup and stick around regardless of login/logout. loginctl enable-linger <username> This enables some nice deployment strategies that don't require root.
  - hornetblack 10y ago
    Thanks. I figured this was possible. I just never got around to looking at loginctl's options.
geocar 10y ago
Yes. I used to use runit/daemontools/inittab. My favourite thing about systemd is that it is increasingly available, and while it has it's faults, it has instances (macros you can use to kickoff a fleet of services easily) and pretty good isolation features. It also has a "systemctl-over-ssh" feature which is quite nice, and which allows you to use an .ssh/authorized_keys file instead of sudo to allow access to certain administrative tasks.

praveshjain 10y ago

I don't understand. How is it better than using HTTP POST to post to the slack channel yourself?

jon-wood 10y ago
It doesn't require changing the underlying systems that are being managed, giving it a lot more flexibility. So long as the service can log to STDOUT it can be integrated with Slack.
- praveshjain 10y ago
  Oh I get it. Thanks.

qznc 10y ago

I'm currently building something similar for matrix instead of slack. My idea was to clone the mail(1) interface, but tee(1) is also a nice idea! In the process, I also discovered sendxmpp, which provides mail(1) for XMPP. It does not support encryption and is written in Perl, so I am building my own.

skarap 10y ago

What if slack is down/unreachable? Won't it keep the service from starting?

lima 10y ago
If it makes slacktee fail with an error code, yes, it would prevent startup. The fix is a - sign: ExecStartPre=-/usr/bin/foo Would still mark the service as "failed" but wouldn't prevent it from starting.

lima 10y ago

Why is everyone okay with relying on Slack for sensitive communication or even business critical stuff like monitoring? It lacks everything a good alerting system has (acknowledgements, fine-grained notifications...). Even those addresses in the stacktraces could be sensitive in other situations since they contain ASLR offsets.

notheguyouthink 10y ago
What would you suggest? We're in need of an alerting platform and making use of Slack, not because it's flawless or anything, but because we're already paying for it and it's convenient for the mass junk. Ie, not strictly mission critical alerts. What medium might be better than Slack? (to be clear, i'm asking, not saying Slack is good for this)
- amyjess 10y ago
  Zabbix. It's open-source and pretty flexible. I maintain my company's Zabbix installation, and I've come to appreciate it (and 3.x finally has a non-ugly UI!).
  - lima 10y ago
    We use Nagios + CheckMK, but I believe neither this nor Zabbix solve the alerting part. Our solution is a custom notification broker that decides whom to alert and then waits for an acknowledgement. It uses different backends including our company chat. Not complicated at all, just 100 lines of Python code that contain the business logic. Anything that relies on a single medium is unsuitable for anything but unimportant alerts. What if Slack goes down for 2 hour? Unlikely, but definitely possible. This ensures that every alert is explicitly acknowledged by someone, and that unimportant alerts are quickly forgotten without wondering whether someone handled them or not. We have different applications sending alerts, not just Nagios (because Nagios sucks at processing events as opposed to states), and it would quickly become unmanageable without some sort of middleware.
    - 10y ago