45 ms·

Also I always thought notebooks would be a great devops tool, kind of like a super command line that has easily observable steps grouped in chunks and graphical

by rb808 9y ago

Also I always thought notebooks would be a great devops tool, kind of like a super command line that has easily observable steps grouped in chunks and graphical feedback. No one else seems to think so though so maybe I'm wrong.

jacquesm 9y ago
That makes good sense, in case it helps your confidence. It's a middle ground between literate programming and a traditional IDE and there is nothing like that aimed at the ops space where it would actually be quite valuable.
- nickbarnwell 9y ago
  emacs' org-mode / org-babel is well suited for it. This blog post [1] has some fantastic examples. The problem, of course, is that it's so closely tied to emacs. 1: http://howardism.org/Technical/Emacs/literate-devops.html http://howardism.org/Technical/Emacs/literate-devops.html
  - craigching 9y ago
    Nice, I just posted the video from that link above ;) Great minds think alike!
    - nickbarnwell 9y ago
      Howard's blog and github are an amazing resource; I'm a huge fan of his demo-it package as well. Highly recommend checking it out if you ever give technical presentations or talks.
existencebox 9y ago
As disclaimer/context, am a dev on the Azure hosted Jupyter Notebooks product, You're not wrong! (Or at the least, it's a topic that has come across our ears before, and is something I certainly agree with) Obviously I probably shouldn't go off spouting all the pipe dreams I have in this space, but given that I got my start doing Ops work and tried to keep an eye for things I might have liked back then, I can assure you there you're not alone. I always saw the similarity foremost as a direct upgrade to the "runbooks"/"firefighting/deploy checklists" that crop up all too often.
- alexeldeib 9y ago
  Another Azure dev checking in :) Have you seen Application Insights Workbooks [0]? Basically you can have interactive notebooks and run analytics queries against your telemetry, generate charts, add text cells, etc. It's picking up usage for investigating outages, e.g., have a Workbook with a query that looks at your dependency calls and determine what service is failing + produce a visualization. Workbooks don't actually execute any external actions, though. It's solely an analysis tool. Runbooks skew the other direction, they are for executing scripts (more or less). Jupyter/python seems to fit in a nice gap where this could be bridged, especially with the level of existing python support from azure sdk + cli. PS: a dev from Workbooks has seen Azure Notebooks, and was curious a while back about how he could integrate the functionality [1] [0]: https://docs.microsoft.com/en-us/azure/application-insights/app-insights-usage-workbooks https://docs.microsoft.com/en-us/azure/application-insights/... [1]: http://blog.my-is300.com/2017/06/what-i-work-on-application-insights-workbooks/ http://blog.my-is300.com/2017/06/what-i-work-on-application-...
  - gardnerjr 9y ago
    wow, a link to my blog (that second link) made hackernews? that's exciting! Anyway, yeah, workbooks in appinsights is almost like notebooks for non-programmers? kinda? you string together markdown, parameters, and analytics queries (and very soon metrics across more of azure) into reports. But the parameters stuff lets you do more interactive things to hide/show sections now. i really need to do a new blog post about all the new stuff that's in there that wasn't last june! i've prototyped some stuff to export an AI workbook to an azure/jupyter notebook, as there's some support for querying analytics already from a python package. there just hasn't been enough demand for it so far (not as much as we expected, anyway?)
craigching 9y ago
How about Emacs + org mode? https://www.youtube.com/watch?v=dljNabciEGg https://www.youtube.com/watch?v=dljNabciEGg I thought this video was pretty cool personally!
- cup-of-tea 9y ago
  Very cool! Watching someone else using emacs is always a great learning experience.
amirathi 9y ago
I built a platform exactly for this. Check it out here: https://www.nurtch.com/ https://www.nurtch.com/
- sytse 9y ago
  Wow, it is awesome how you managed to integrate runbooks, Jupyter, and monitoring, well done and great video!
  - amirathi 9y ago
    Thanks :) Just saw that you are CEO of GitLab. Good job making your runbooks public [1]. I converted one of your runbook into an executable notebook to convey my point. Check the before/after screenshot here: https://blog.amirathi.com/2018/03/27/codify-infra-runbooks-with-jupyter-style-notebook/ https://blog.amirathi.com/2018/03/27/codify-infra-runbooks-w... You should consider using Nurtch :) [1]https://gitlab.com/gitlab-com/runbooks https://gitlab.com/gitlab-com/runbooks
wohlergehen 9y ago
Native Jupyter has the magic %%bash command, so a lot of this should just be possible, i.e. you start a cell with that, and it will invoke bash to execute it. More generally, there is %%script <x>, which executes the cell using <x>. 0: http://ipython.readthedocs.io/en/stable/interactive/magics.html http://ipython.readthedocs.io/en/stable/interactive/magics.h...
lmeyerov 9y ago
Yep! We've been experimenting with a variant for security & fraud investigations at Graphistry. Our original experiments with just notebooks resulted in a few people on super early adopter teams getting a kick out of it, but in critical ~operational settings, having to deal with code was... not great. Mostly looked like siloed personal use. BUT. The core concept is great. We made a form more focused on interactive investigation, so you can jump from an alert in a dashboard into a rich & interactive visual session with pre-wrangled data: https://youtu.be/B3ZZWx9WUEk?t=1m32s https://youtu.be/B3ZZWx9WUEk?t=1m32s . Depending on what you see, can easily pull in more data, or refine what's there. And agreed, I think it'd be fun to try in big devops/netops scenarios!

by rb808 9y ago

jacquesm 9y ago

That makes good sense, in case it helps your confidence. It's a middle ground between literate programming and a traditional IDE and there is nothing like that aimed at the ops space where it would actually be quite valuable.

nickbarnwell 9y ago
emacs' org-mode / org-babel is well suited for it. This blog post [1] has some fantastic examples. The problem, of course, is that it's so closely tied to emacs. 1: http://howardism.org/Technical/Emacs/literate-devops.html http://howardism.org/Technical/Emacs/literate-devops.html
- craigching 9y ago
  Nice, I just posted the video from that link above ;) Great minds think alike!
  - nickbarnwell 9y ago
    Howard's blog and github are an amazing resource; I'm a huge fan of his demo-it package as well. Highly recommend checking it out if you ever give technical presentations or talks.

existencebox 9y ago

As disclaimer/context, am a dev on the Azure hosted Jupyter Notebooks product, You're not wrong! (Or at the least, it's a topic that has come across our ears before, and is something I certainly agree with) Obviously I probably shouldn't go off spouting all the pipe dreams I have in this space, but given that I got my start doing Ops work and tried to keep an eye for things I might have liked back then, I can assure you there you're not alone. I always saw the similarity foremost as a direct upgrade to the "runbooks"/"firefighting/deploy checklists" that crop up all too often.

alexeldeib 9y ago
Another Azure dev checking in :) Have you seen Application Insights Workbooks [0]? Basically you can have interactive notebooks and run analytics queries against your telemetry, generate charts, add text cells, etc. It's picking up usage for investigating outages, e.g., have a Workbook with a query that looks at your dependency calls and determine what service is failing + produce a visualization. Workbooks don't actually execute any external actions, though. It's solely an analysis tool. Runbooks skew the other direction, they are for executing scripts (more or less). Jupyter/python seems to fit in a nice gap where this could be bridged, especially with the level of existing python support from azure sdk + cli. PS: a dev from Workbooks has seen Azure Notebooks, and was curious a while back about how he could integrate the functionality [1] [0]: https://docs.microsoft.com/en-us/azure/application-insights/app-insights-usage-workbooks https://docs.microsoft.com/en-us/azure/application-insights/... [1]: http://blog.my-is300.com/2017/06/what-i-work-on-application-insights-workbooks/ http://blog.my-is300.com/2017/06/what-i-work-on-application-...
- gardnerjr 9y ago
  wow, a link to my blog (that second link) made hackernews? that's exciting! Anyway, yeah, workbooks in appinsights is almost like notebooks for non-programmers? kinda? you string together markdown, parameters, and analytics queries (and very soon metrics across more of azure) into reports. But the parameters stuff lets you do more interactive things to hide/show sections now. i really need to do a new blog post about all the new stuff that's in there that wasn't last june! i've prototyped some stuff to export an AI workbook to an azure/jupyter notebook, as there's some support for querying analytics already from a python package. there just hasn't been enough demand for it so far (not as much as we expected, anyway?)

craigching 9y ago

How about Emacs + org mode? https://www.youtube.com/watch?v=dljNabciEGg https://www.youtube.com/watch?v=dljNabciEGg I thought this video was pretty cool personally!

cup-of-tea 9y ago
Very cool! Watching someone else using emacs is always a great learning experience.

amirathi 9y ago

I built a platform exactly for this. Check it out here: https://www.nurtch.com/ https://www.nurtch.com/

sytse 9y ago
Wow, it is awesome how you managed to integrate runbooks, Jupyter, and monitoring, well done and great video!
- amirathi 9y ago
  Thanks :) Just saw that you are CEO of GitLab. Good job making your runbooks public [1]. I converted one of your runbook into an executable notebook to convey my point. Check the before/after screenshot here: https://blog.amirathi.com/2018/03/27/codify-infra-runbooks-with-jupyter-style-notebook/ https://blog.amirathi.com/2018/03/27/codify-infra-runbooks-w... You should consider using Nurtch :) [1]https://gitlab.com/gitlab-com/runbooks https://gitlab.com/gitlab-com/runbooks

wohlergehen 9y ago

Native Jupyter has the magic %%bash command, so a lot of this should just be possible, i.e. you start a cell with that, and it will invoke bash to execute it. More generally, there is %%script <x>, which executes the cell using <x>. 0: http://ipython.readthedocs.io/en/stable/interactive/magics.html http://ipython.readthedocs.io/en/stable/interactive/magics.h...

lmeyerov 9y ago

Yep! We've been experimenting with a variant for security & fraud investigations at Graphistry. Our original experiments with just notebooks resulted in a few people on super early adopter teams getting a kick out of it, but in critical ~operational settings, having to deal with code was... not great. Mostly looked like siloed personal use. BUT. The core concept is great. We made a form more focused on interactive investigation, so you can jump from an alert in a dashboard into a rich & interactive visual session with pre-wrangled data: https://youtu.be/B3ZZWx9WUEk?t=1m32s https://youtu.be/B3ZZWx9WUEk?t=1m32s . Depending on what you see, can easily pull in more data, or refine what's there. And agreed, I think it'd be fun to try in big devops/netops scenarios!