4 ms·

I think another thing we might want to learn about is how to sound the alarm when the break glass is used. Is there an easy way of doing that with SSH? Running

I think another thing we might want to learn about is how to sound the alarm when the break glass is used. Is there an easy way of doing that with SSH? Running a command to page the ops/security team when a server receives a login attempt with an emergency credential?

nix23 6y ago
Maybe monitor the Emergency Machine itself? If it boot's up, emergency credentials are probably used? But really good point, and i love the analogy to 'break glass'
withinboredom 6y ago
You can physically put the (yubikey) device in a vault that will physically sound an alarm when opened. It could also have a battery-powered arduino inside the box (with SIM breakout) that texted the devops team when opened.
- chrisweekly 6y ago
  No idea why you were downvoted; it seems like a reasonable idea to me. (Also, IMHO downvoting a good-faith comment like yours is a lazy alternative to posting a substantive response.)
  - BillinghamJ 6y ago
    Overcomplex technical solution to a simple problem. Besides which, if you really want to go full-on with technically clever solutions, keep in mind you could ensure no cellular service prior to opening. But then we're just getting into the realms of silly situations.
    - dcow 6y ago
      Would you attempt to use the key if you knew the CTO, SRE and OPS teams were paged as soon as the safe was accessed?
lormayna 6y ago
You can use pam-hooks module to execute scripts at login/logout.
andylynch 6y ago
There are tools like Powerbroker which do this, and also privileged access management more generally - popular at banks and the like. Also SSH (the company)
aidos 6y ago
You can add a script at ~/.ssh/rc that’s run on each login. You’d need to be careful to make sure it couldn’t be changed if you were relying on it for notifications.
- gnufx 6y ago
  How do you record the key used from that (assuming that's what's required)?
gnufx 6y ago
I don't know what it looks like for a certificated system, but syslog records the private key used for login in a fairly vanilla Debian. If you worry about things like that and aren't looking at physical access (as suggested elsewhere), you presumably have remote syslog and audit which you can check.
tashian 6y ago
Hey there, I wrote this post. It's a great question. One benefit of using certificates for emergency access is that SSHD logging can be configured to show a lot more detail about the certificate that was used. With public keys, there isn't anything to show. But with certificates you have a key ID, serial number, principals, CA fingerprint, etc. So, that log is a good hook for sounding the alarm. A more advanced version of this would allow you to record a reason for using the emergency access key when the connection is made (or when sudo is used).
- jlgaddis 6y ago
  > With public keys, there isn't anything to show. There's, at minimum, client IP address, username, and the key fingerprint -- which has always been good enough for me. There might be even more details available but I'm not sitting in front of a computer to check.
rsync 6y ago
"I think another thing we might want to learn about is how to sound the alarm when the break glass is used. Is there an easy way of doing that with SSH?" Yes - quite simple and old-fashioned, actually ... I have this line in the SSH users' .login file: /usr/local/sbin/sms 4153331111 4158882222 "USER LOGIN TO XXX - $DATE" >& /dev/null ... where the 'sms' command, above, is a shell script I wrote to call twilio messaging with the curl command. A very simple example of that would be: curl -X POST -d "Body=$msg" -d "From=$from" -d "To=$to" "https://api.twilio.com/2010-04-01/Accounts/$accountsid/Messages" -u "$accountsid:$authtoken" ... and this works like a charm. Alternatively, you could rick-roll your on-call sysadmin: /usr/local/bin/curl -XPOST https://api.twilio.com/2010-04-01/Accounts/$accountsid/Calls.json --data-urlencode "To=$number" --data-urlencode "From=$callerid" --data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" -u $accountsid:$authtoken (the voice.xml demo is, in fact, Rick Astley)
- rnotaro 6y ago
  Direct link of the audio that the XML will play: https://demo.twilio.com/docs/classic.mp3 https://demo.twilio.com/docs/classic.mp3
- jlgaddis 6y ago
  What prevents a user from disabling or bypassing the command execution (if one desired to do so)?
8organicbits 6y ago
Login shell for emergency accounts could be a script that "sounds the alarm" and then drops to a bash shell. Edit: ooo @rsync just gave another good approach
jlgaddis 6y ago
Generating alerts from syslog messages is something that we've been doing for decades.

nix23 6y ago

Maybe monitor the Emergency Machine itself? If it boot's up, emergency credentials are probably used? But really good point, and i love the analogy to 'break glass'

withinboredom 6y ago

You can physically put the (yubikey) device in a vault that will physically sound an alarm when opened. It could also have a battery-powered arduino inside the box (with SIM breakout) that texted the devops team when opened.

chrisweekly 6y ago
No idea why you were downvoted; it seems like a reasonable idea to me. (Also, IMHO downvoting a good-faith comment like yours is a lazy alternative to posting a substantive response.)
- BillinghamJ 6y ago
  Overcomplex technical solution to a simple problem. Besides which, if you really want to go full-on with technically clever solutions, keep in mind you could ensure no cellular service prior to opening. But then we're just getting into the realms of silly situations.
  - dcow 6y ago
    Would you attempt to use the key if you knew the CTO, SRE and OPS teams were paged as soon as the safe was accessed?

lormayna 6y ago

You can use pam-hooks module to execute scripts at login/logout.

andylynch 6y ago

There are tools like Powerbroker which do this, and also privileged access management more generally - popular at banks and the like. Also SSH (the company)

aidos 6y ago

You can add a script at ~/.ssh/rc that’s run on each login. You’d need to be careful to make sure it couldn’t be changed if you were relying on it for notifications.

gnufx 6y ago
How do you record the key used from that (assuming that's what's required)?

gnufx 6y ago

I don't know what it looks like for a certificated system, but syslog records the private key used for login in a fairly vanilla Debian. If you worry about things like that and aren't looking at physical access (as suggested elsewhere), you presumably have remote syslog and audit which you can check.

tashian 6y ago

Hey there, I wrote this post. It's a great question. One benefit of using certificates for emergency access is that SSHD logging can be configured to show a lot more detail about the certificate that was used. With public keys, there isn't anything to show. But with certificates you have a key ID, serial number, principals, CA fingerprint, etc. So, that log is a good hook for sounding the alarm. A more advanced version of this would allow you to record a reason for using the emergency access key when the connection is made (or when sudo is used).

jlgaddis 6y ago
> With public keys, there isn't anything to show. There's, at minimum, client IP address, username, and the key fingerprint -- which has always been good enough for me. There might be even more details available but I'm not sitting in front of a computer to check.

rsync 6y ago

"I think another thing we might want to learn about is how to sound the alarm when the break glass is used. Is there an easy way of doing that with SSH?" Yes - quite simple and old-fashioned, actually ... I have this line in the SSH users' .login file: /usr/local/sbin/sms 4153331111 4158882222 "USER LOGIN TO XXX - $DATE" >& /dev/null ... where the 'sms' command, above, is a shell script I wrote to call twilio messaging with the curl command. A very simple example of that would be: curl -X POST -d "Body=$msg" -d "From=$from" -d "To=$to" "https://api.twilio.com/2010-04-01/Accounts/$accountsid/Messages" -u "$accountsid:$authtoken" ... and this works like a charm. Alternatively, you could rick-roll your on-call sysadmin: /usr/local/bin/curl -XPOST https://api.twilio.com/2010-04-01/Accounts/$accountsid/Calls.json --data-urlencode "To=$number" --data-urlencode "From=$callerid" --data-urlencode "Url=http://demo.twilio.com/docs/voice.xml" -u $accountsid:$authtoken (the voice.xml demo is, in fact, Rick Astley)

rnotaro 6y ago
Direct link of the audio that the XML will play: https://demo.twilio.com/docs/classic.mp3 https://demo.twilio.com/docs/classic.mp3
jlgaddis 6y ago
What prevents a user from disabling or bypassing the command execution (if one desired to do so)?

8organicbits 6y ago

Login shell for emergency accounts could be a script that "sounds the alarm" and then drops to a bash shell. Edit: ooo @rsync just gave another good approach

jlgaddis 6y ago

Generating alerts from syslog messages is something that we've been doing for decades.