5 ms·

I don't like being the voice of the purist in this, but this seems like a bandaid on a bullet wound. For most of the cases where this would seem to be useful t

by bigdubs 11y ago

I don't like being the voice of the purist in this, but this seems like a bandaid on a bullet wound. For most of the cases where this would seem to be useful there is probably a failure upstream of that usage that should really be fixed.

doriftoshoes 11y ago
Giant disclaimer: I work at StackStorm...but I also have an extensive Ops background. This is really the next step in runbook automation. It gives users a way to express procedures and operational patterns in code (the workflow definitions are in yaml). With any sort of automated remediation there is always the concern of "painting over the mold" but at the same time you don't want to get stuck doing a large number of manual steps when you could be focusing your energy on tracking down the root cause of the issue and resolving that. The more important aspect to me personally is the easy of version controlling these ops patterns. Store the workflow definitions in source control and it is easy to diff the changes in your procedure.
- mst 11y ago
  It seems YAML is the new S-expressions for people trying to pretend they didn't actually invent a programming language. This is, I guess, less annoying than executable XML.
  - doriftoshoes 11y ago
    YAML is definitely the hotness right now but it works. No need to invent a full blown language for something like this. Way easier than trying to write json.
    - mst 11y ago
      > Way easier than trying to write json. Which is why ingy and I invented http://p3rl.org/JSONY http://p3rl.org/JSONY for config files.
    - eropple 11y ago
      Or you can use Ruby or Python and Perl and have a scripting language that looks and acts as a scripting language is generally expected to look and act. (And with Ruby in particular it's very easy to provide a flexible and terse DSL that provides significant benefits on its own.) This "invent your own worse programming language" fad is disappointing.
      - mst 11y ago
        For plain config, I mostly use JSONY now. For executable declarationish things, I like Tcl (usually embedded in perl so I can use Moo(se) for OO rather than the inferior crap in other languages). Hashicorp's HCL is an interesting middle ground.
        eropple 11y ago
        I can respect Tcl. I'm not super familiar with it aside from hacking on eggdrop, but it's in the same ballpark. For configuration, I just source a shell script and grab env vars wherever I can. It's the most portable (between applications, not necessarily between platforms) option available to me and plays nicely with stuff like jails. At a glance, I don't quite understand the value prop of JSONY over YAML, which (as you see used in something like Rails) allows for a good bit of flexibility with references to more tersely communicate intent. I think HCL is one of the worse decisions made by a somewhat influential software company in quite some time. It's harder to use than YAML (which is funny because the stated reason for its existence is "YAML is harder", and it makes me really, really curious what sort of users they're polling) while being less expressive and they're very careful to not really care about anybody's interop "because you can just use JSON instead", except that, as trying to work with Terraform amply proved, no, you can't use JSON, because that path isn't tested because nobody cares and unit tests are hard. =( HCL is a regular problem in my life. I needed to vent a little.
        crdoconnor 11y ago
        >for configuration, I just source a shell script and grab env vars wherever I can. That can cause you headaches when you want to store lists or associative data in your config.
    - zobzu 11y ago
      not sure how yaml is easier than json also not sure how yaml is easier than python or what not
      - lnkmails 11y ago
        StackStorm developer here! Just to clarify, YAML is one way to write the workflow spec. Just like how YAML is way to write docker compose file. We picked YAML over JSON because it is less verbose and allows one to add comments which is pretty useful if someone is looking at a remediation workflow early in the AM. We do use YAQL expressions for some variable manipulation but that is not a YAML spec. A workflow is a collection of actions and actions can be written in almost any language (as a script) or python (first class actions). So you can think YAML workflow as a declarative spec rather than a programming language. Maybe that helps?
- crdoconnor 11y ago
  Some nice readable example YAML definitions would be good to show on the README.
mst 11y ago
> For most of the cases where this would seem to be useful there is probably a failure upstream of that usage that should really be fixed. Of course. But in the mean time, it would be nice to keep production up.
ewindisch 11y ago
Monitoring exists because software is not perfect and will never be perfect. Software raises errors because it doesn't know what to when those events occur. Operators usually know what needs to be done for their specific environment, even if the software they're running doesn't. If we could eliminate the bullet wounds we wouldn't EVER need logs. As long as we have logs, we need some way to process them and react to their events. For many organizations, for decades, this has been to have an alerts system and an active operations team fighting fires. Those teams maintain knowledge-bases to track institutional knowledge of how to manually react to these events. Solutions such as Stackstorm and IBM ZAware seem to be created to allow that institutional knowledge to be automated. I've also seen (and built) proof-of-concepts using Bayesian filters as part of such systems. It's been a long-time coming and I'm happy solutions are evolving to address this need. Finally, I think that bandaids such as this may be, might be precisely what is needed in some cases. Assuming it's even a legal or technical possibility, the cost of the "right fix" may far exceed the harm of leaving a bug unfixed. Sometimes the right fix has a large time-cost, which automation can help bridge the gap for operations teams while a permanent fix is developed (or more hardware is acquired, etc).
ascendantlogic 11y ago
And how do you universally fix the bullet wounds? How do you stop every upstream issue from ever occurring so middle-of-the-night remediation doesn't have to happen? Of course everyone would love the actually solve every problem but real life is messier than that. For every issue that needs to be fixed, there's usually one or more reasons why they can't be fixed the "right way" right this instant: Management bullshit, prioritization, $$$, etc etc.

by bigdubs 11y ago

doriftoshoes 11y ago

Giant disclaimer: I work at StackStorm...but I also have an extensive Ops background. This is really the next step in runbook automation. It gives users a way to express procedures and operational patterns in code (the workflow definitions are in yaml). With any sort of automated remediation there is always the concern of "painting over the mold" but at the same time you don't want to get stuck doing a large number of manual steps when you could be focusing your energy on tracking down the root cause of the issue and resolving that. The more important aspect to me personally is the easy of version controlling these ops patterns. Store the workflow definitions in source control and it is easy to diff the changes in your procedure.

mst 11y ago
It seems YAML is the new S-expressions for people trying to pretend they didn't actually invent a programming language. This is, I guess, less annoying than executable XML.
- doriftoshoes 11y ago
  YAML is definitely the hotness right now but it works. No need to invent a full blown language for something like this. Way easier than trying to write json.
  - mst 11y ago
    > Way easier than trying to write json. Which is why ingy and I invented http://p3rl.org/JSONY http://p3rl.org/JSONY for config files.
  - eropple 11y ago
    Or you can use Ruby or Python and Perl and have a scripting language that looks and acts as a scripting language is generally expected to look and act. (And with Ruby in particular it's very easy to provide a flexible and terse DSL that provides significant benefits on its own.) This "invent your own worse programming language" fad is disappointing.
    - mst 11y ago
      For plain config, I mostly use JSONY now. For executable declarationish things, I like Tcl (usually embedded in perl so I can use Moo(se) for OO rather than the inferior crap in other languages). Hashicorp's HCL is an interesting middle ground.
      - eropple 11y ago
        I can respect Tcl. I'm not super familiar with it aside from hacking on eggdrop, but it's in the same ballpark. For configuration, I just source a shell script and grab env vars wherever I can. It's the most portable (between applications, not necessarily between platforms) option available to me and plays nicely with stuff like jails. At a glance, I don't quite understand the value prop of JSONY over YAML, which (as you see used in something like Rails) allows for a good bit of flexibility with references to more tersely communicate intent. I think HCL is one of the worse decisions made by a somewhat influential software company in quite some time. It's harder to use than YAML (which is funny because the stated reason for its existence is "YAML is harder", and it makes me really, really curious what sort of users they're polling) while being less expressive and they're very careful to not really care about anybody's interop "because you can just use JSON instead", except that, as trying to work with Terraform amply proved, no, you can't use JSON, because that path isn't tested because nobody cares and unit tests are hard. =( HCL is a regular problem in my life. I needed to vent a little.
        crdoconnor 11y ago
        >for configuration, I just source a shell script and grab env vars wherever I can. That can cause you headaches when you want to store lists or associative data in your config.
  - zobzu 11y ago
    not sure how yaml is easier than json also not sure how yaml is easier than python or what not
    - lnkmails 11y ago
      StackStorm developer here! Just to clarify, YAML is one way to write the workflow spec. Just like how YAML is way to write docker compose file. We picked YAML over JSON because it is less verbose and allows one to add comments which is pretty useful if someone is looking at a remediation workflow early in the AM. We do use YAQL expressions for some variable manipulation but that is not a YAML spec. A workflow is a collection of actions and actions can be written in almost any language (as a script) or python (first class actions). So you can think YAML workflow as a declarative spec rather than a programming language. Maybe that helps?
crdoconnor 11y ago
Some nice readable example YAML definitions would be good to show on the README.

mst 11y ago

> For most of the cases where this would seem to be useful there is probably a failure upstream of that usage that should really be fixed. Of course. But in the mean time, it would be nice to keep production up.

ewindisch 11y ago

Monitoring exists because software is not perfect and will never be perfect. Software raises errors because it doesn't know what to when those events occur. Operators usually know what needs to be done for their specific environment, even if the software they're running doesn't. If we could eliminate the bullet wounds we wouldn't EVER need logs. As long as we have logs, we need some way to process them and react to their events. For many organizations, for decades, this has been to have an alerts system and an active operations team fighting fires. Those teams maintain knowledge-bases to track institutional knowledge of how to manually react to these events. Solutions such as Stackstorm and IBM ZAware seem to be created to allow that institutional knowledge to be automated. I've also seen (and built) proof-of-concepts using Bayesian filters as part of such systems. It's been a long-time coming and I'm happy solutions are evolving to address this need. Finally, I think that bandaids such as this may be, might be precisely what is needed in some cases. Assuming it's even a legal or technical possibility, the cost of the "right fix" may far exceed the harm of leaving a bug unfixed. Sometimes the right fix has a large time-cost, which automation can help bridge the gap for operations teams while a permanent fix is developed (or more hardware is acquired, etc).

ascendantlogic 11y ago

And how do you universally fix the bullet wounds? How do you stop every upstream issue from ever occurring so middle-of-the-night remediation doesn't have to happen? Of course everyone would love the actually solve every problem but real life is messier than that. For every issue that needs to be fixed, there's usually one or more reasons why they can't be fixed the "right way" right this instant: Management bullshit, prioritization, $$$, etc etc.