5 ms·

recon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issu

recon and observer_cli are the tools I reach out first to debug any issues in production. In any other language, I usually think about how to reproduce the issue locally. With Elixir, I just get into a remote shell in the affected machine and live debug the issue, and there are cases where we applied hotfix by using eval right there from the shell. The idea of the remote shell itself is alien to most languages.

busterarm 4y ago
And unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.
- jon-wood 4y ago
  This sort of thing doesn’t have to be a compliance breach, but you will likely need some way of ensuring there’s a second person in the loop, typically that would take the form of having someone in a separate production infrastructure team actually driving a while you talk them through what needs to happen.
  - busterarm 4y ago
    Yes and with the added benefit of having to explain that control to your rotating bunch of compliance people every single year. I'm not criticizing the methodology as much as the useless performative nature of compliance work.
    - d4mi3n 4y ago
      Compliance is performative until it isn't. If you've ever been party to a breach, the role of compliance and an audit trail to the security narrative becomes _very_ important. Consider: 1. We had a breach. A factor in this was insufficient oversight on a process that granted privileged access to customer data. We fixed the problem, promise that your data is safe, and don't believe this will happen again. 2. We had a breach. A factor in this was due to a gap in an existing control around customer data that had a problem we had not anticipated. These were the people involved. This is exactly how this problem occurred. This is the data that was exposed. This is documentation of our response to this incident. This is our existing policy around how we handle data and how we respond to breaches. Customers, partners, regulators, and law enforcement respond a lot better when you can demonstrate good intent and at least imply that you have some kind of process. Of the two scenarios I outlined, the latter provides those assurances. Compliance isn't the only way to do this, but it's often the easiest.
  - freedomben 4y ago
    You also need a way of recording what changed. when committing to git it's easy, but live eval-ing into prod sounds harder.
- mattbaker 4y ago
  Still wildly useful debugging things locally too!
- throwawaymaths 4y ago
  It is possible to certify Erlang and elixir as fips-140 compliant.
- andrewflnr 4y ago
  Is it any different than any other kind of direct prod access? I mean, you have to have controls for accessing prod, so it seems like you could use similar ones for REPL access, but that's logic and I know that only has a tenuous relationship with compliance...

busterarm 4y ago

And unfortunately the kind of thing that compliance flags as a big no-no once you've got any kind of filing or privacy requirements.

jon-wood 4y ago
This sort of thing doesn’t have to be a compliance breach, but you will likely need some way of ensuring there’s a second person in the loop, typically that would take the form of having someone in a separate production infrastructure team actually driving a while you talk them through what needs to happen.
- busterarm 4y ago
  Yes and with the added benefit of having to explain that control to your rotating bunch of compliance people every single year. I'm not criticizing the methodology as much as the useless performative nature of compliance work.
  - d4mi3n 4y ago
    Compliance is performative until it isn't. If you've ever been party to a breach, the role of compliance and an audit trail to the security narrative becomes _very_ important. Consider: 1. We had a breach. A factor in this was insufficient oversight on a process that granted privileged access to customer data. We fixed the problem, promise that your data is safe, and don't believe this will happen again. 2. We had a breach. A factor in this was due to a gap in an existing control around customer data that had a problem we had not anticipated. These were the people involved. This is exactly how this problem occurred. This is the data that was exposed. This is documentation of our response to this incident. This is our existing policy around how we handle data and how we respond to breaches. Customers, partners, regulators, and law enforcement respond a lot better when you can demonstrate good intent and at least imply that you have some kind of process. Of the two scenarios I outlined, the latter provides those assurances. Compliance isn't the only way to do this, but it's often the easiest.
- freedomben 4y ago
  You also need a way of recording what changed. when committing to git it's easy, but live eval-ing into prod sounds harder.
mattbaker 4y ago
Still wildly useful debugging things locally too!
throwawaymaths 4y ago
It is possible to certify Erlang and elixir as fips-140 compliant.
andrewflnr 4y ago
Is it any different than any other kind of direct prod access? I mean, you have to have controls for accessing prod, so it seems like you could use similar ones for REPL access, but that's logic and I know that only has a tenuous relationship with compliance...