9 ms·
Literate Devops with Emacs Org-Mode (2014)
- eadmund 9y agoOrg-mode & Magit really are the secret superpowers of modern emacs. And notmuch (full-text search of one's email, at least one order of magnitude faster than Gmail's!). And elfeed. And gnus. And dired. And pretty much everything else in emacs …
- s4vi0r 9y agoIs there some way to tell org exactly how I want it to spit out HTML? I was going to do my blog entirely with org-mode+org-publish, but it spits out a mess of HTML and also a small bit of js for some reason, so I'm just writing html directly myself now. I don't really find writing html myself all that annoying or anything, but it'd be nice if I could get org to generate really clean/simple HTML for me that I could then write a small script to parse and wrap in divs I've defined for styling.
- enitihas 9y agoNot the ideal solution, but you can always use pandoc to convert org to html, or even markdown and configure the html generation from there.
- nine_k 9y agoBoth html and markdown miss various org-specific features, like code fragments, drawers (imagine that one could want them to be able top open and close on click), task states, etc. A general-purpose markup language like XML would be an ideal intermediate export format; XSLT can basically emit anything.
- philsnow 9y agoFor drawers you could use https://developer.mozilla.org/en-US/docs/Web/HTML/Element/details https://developer.mozilla.org/en-US/docs/Web/HTML/Element/de... but yeah in general I agree
- deleted 9y ago[deleted]
- btschaegg 9y agoI've been looking for a similar solution - albeit not for HTML - but haven't found anything convincing so far. My train of thought was: It'd be great to have a a canonical XML format for org files to export to, since after that, you could essentially write XSLT transforms for almost any target format (even back to org-files to go full circle). In my case, that e.g. would be odt-files with branding for the organization I'm taking minutes for. After a bit of googling, I also found an email[1] on the mailing list asking for more or less exactly that, but things don't seem to have gotten far. I've been thinking about adapting the existing HTML export for such an XML format, but I really haven't had the time for it. :-/ [1]: https://lists.gnu.org/archive/html/emacs-orgmode/2009-08/msg00189.html https://lists.gnu.org/archive/html/emacs-orgmode/2009-08/msg...
- icebraining 9y agoPandoc supports org-mode, so you should be able to use it to convert to DocBook, which you can then use with XLST (pandoc also supports HTML directly, by the way).
- btschaegg 9y agoIf I remember correctly, I had a look at the pandoc option, but got to the conclusion that this way, I would throw away useful 'syntactical' information (for example, I use todo-flags, tags and timestamps to track open tasks). If you use org-mode without the need to extract such specific information (and are only interested in generating a document), pandoc might of course be a good way to go. Edit: ...or maybe there even is a hack to identify and extract things like timestamps anyway. I didn't look into it that much.
- omaranto 9y agoI doubt XSLT is any easier to write than a new Org export backend, but that's probably a matter of personal preference.
- celeritascelery 9y agoI know you can define CSS for how you want your export to look.
- taeric 9y agoFor the most part, just styling what it does isn't that tough. Similarly, you can tell it to spit out raw html for some fun. http://taeric.github.io/cube-permutations-1.html http://taeric.github.io/cube-permutations-1.html shows some silliness I did recently that includes styling up portions of the page. Do you want to do something much more complicated/simple than this?
- deleted 9y ago[deleted]
- deleted 9y ago[deleted]
- funkaster 9y agoNot exactly what you're asking for but I got great results with org-page[0]. It might be worth taking a look. [0]: https://github.com/kelvinh/org-page https://github.com/kelvinh/org-page
- kqr 9y agoAre you kidding!? This is Emacs! You can tell it exactly how to do everything. Just advise/redefine the relevant functions. ;) The documentation of ox and publish does not cover everything, but when in doubt, just ask Emacs what code it runs when you do the thing you want to modify. On my site (https://two-wrongs.com/ https://two-wrongs.com/) there is very little left of the original HTML. I'll see tomorrow if I can dig up the Lisp code I wrote for it.
- kaushalmodi 9y agoYou website design is awesome! Do you have the site source public? If allowed, I'd be interested in converting that to a theme for Hugo (https://gohugo.io https://gohugo.io), and possibly demoing that on how to use that with my package ox-hugo.
- omaranto 9y agoI doubt there's anything actually wrong with the HTML that Org produces that would keep you from putting it in a blog, but you can certainly configure things to your liking. At the very least you can easily modify the HTML preamble and postamble, the javascript script tags and the CSS in style tags that Org's HTML export produces. All of those are just strings stored in variables that you can customize: org-html-preamble, org-html-postamble, org-html-scripts and org-html-style-default. There's more stuff you can configure by just reassigning some variable, see the file ox-html.el. If tweaking those is not enough for you, you can write your own HTML export. You wouldn't have to parse Org syntax: you'd receive the Org markup already parsed into a nested list structure. If you don't use many Org features, writing an Org backend that supports only those probably isn't that much work. And you'd have ox-html.el to base your own backend on.
- abhirag 9y agoCustom css is supported for html export, from the docs at (https://orgmode.org/manual/CSS-support.html https://orgmode.org/manual/CSS-support.html) -- The HTML export back-end includes a compact default style in each exported HTML file. To override the default style with another style, use these keywords in the Org file. They will replace the global defaults the HTML exporter uses. #+HTML_HEAD: <link rel="stylesheet" type="text/css" href="style1.css" /> #+HTML_HEAD_EXTRA: <link rel="alternate stylesheet" type="text/css" href="style2.css" /> To just turn off the default style, customize org-html-head-include-default-style variable, or use this option line in the Org file. #+OPTIONS: html-style:nil
- deleted 9y ago[deleted]
- gkya 9y agoI've been blogging with org-publish for a while, but I switched to a Ruby script instead [1]. I'd not recommend using org-publish for this, because it's slow, and it's hard to add new posts. It's nice for static projects (say like a bunch of PDFs or some documentation maybe), but it does not lend itself to a more dynamic thing like a blog (dynamic in the sense that you want to add new pages rather frequently). Applying a custom HTML template is difficult, it means you need to either implement many org-export functions or use HTML_HEAD settings and a stylesheet to tame org's output. I say this as a happy everyday user of Org mode, both for word processing and any sort of notes, and my entire agenda and todo-lists and stuff. The JS is for interacting with tables, if I'm not wrong. [1] https://www.gkayaalp.com/blog/20180221_siteremake.html https://www.gkayaalp.com/blog/20180221_siteremake.html
- kaushalmodi 9y agoI am assuming your site remake is still in progress? The fonts on your web site show up in super-huge size at the moment. You can see the font size on other sites like this one for comparison: [1]. [1]: https://i.imgur.com/um3yTdU.png https://i.imgur.com/um3yTdU.png
- gkya 9y agoThanks a lot! I haven't been able to reproduce that in any computers and browsers I have access to. Is that a Retina screen maybe?
- kaushalmodi 9y agoI don't use Mac, so not sure what resolution qualifies as "Retina", but my Dell monitor resolution is 2560 x 1440.
- BeetleB 9y agoThere are, but I never explored them. I use Pelican, and although it doesn't support org files, it's easy to write a plugin to add your own formats. Behind the scene, I think I use pandoc to convert from org to rst, which Pelican does support.
- kaushalmodi 9y agoYes, this came up on /r/emacs recently, see[1] (you can control if you want the HTML preamble and/or postamble). [1]: https://www.reddit.com/r/emacs/comments/809xwx/how_to_get_plain_boring_html_export_out_of_orgmode/duv5p6z/ https://www.reddit.com/r/emacs/comments/809xwx/how_to_get_pl...
- kaushalmodi 9y agoTooting my own horn, if you love living in Org mode, and are willing to try out Hugo (https://gohugo.io https://gohugo.io), I have an Org exporter package ox-hugo that exports to Markdown + required front-matter for Hugo. The package is even more magic if you choose to use a single Org file to export to multiple posts/pages. Few examples: 1. ox-hugo documentation site from one file[1], source[2] 2. My blog[3] 3. ox-hugo test site (300+ test pages and counting)[4] [1]: https://ox-hugo.scripter.co/ https://ox-hugo.scripter.co/ [2]: https://github.com/kaushalmodi/ox-hugo/raw/master/doc/ox-hugo-manual.org https://github.com/kaushalmodi/ox-hugo/raw/master/doc/ox-hug... [3]: https://scripter.co/ https://scripter.co/ [4]: https://ox-hugo.scripter.co/test/ https://ox-hugo.scripter.co/test/
- hammerandtongs 9y agoWeird... it's as if text was really important in computing and having a tool at the center that excelled at processing text put you in more fluid control of your computing environment.
- Cacti 9y agoIve tried so many times to use gnus over the years, without luck. Any suggestions? Or links?
- nextos 9y agoIMHO Notmuch is easier and much simpler for most email workflows.
- ams6110 9y agoI agree. I have used gnus for email in the past, but when I discovered notmuch I moved to it immediately. Gnus can work if you want to treat your email like a News Group. Not a bad approach for email lists for example, where you are mostly a reader and occasionally a contributor, and you rarely need to find an old message that you've already read. But it's not quite the right model for the way most people deal with the rest of their email which is one-to-one or small group threads. Notmuch is like the best of the newsgroup and email models. It has a great threaded view, tagging and filtering are easy, but it still makes everything very easy to find when you need it. Also in combination with w3m it does a really good job at rendering HTML emails in plain text. Even those with tables and other layout tend to be rendered very accurately within the constraints of a fixed font plain text view.
- _emacsomancer_ 9y agoI'd recommend mu4e [https://www.djcbsoftware.nl/code/mu/mu4e.html https://www.djcbsoftware.nl/code/mu/mu4e.html] for email. I used it for a number of years; switched to gnus for a year, and then ended up back on mu4e.
- quotemstr 9y agoOrg-mode has been nice the few times I've tried it, but without decent mobile support or multi-master syncing, it's hard to adopt it as my main productivity tool. I've tried a few mobile org-mode clients and been universally disappointed. I'm increasingly convinced that the right path forward is to give Emacs an Android port (it's not that hard) and let the mobile org just be the mainline org, perhaps with a layer of android.view.View widgets (orchestrated from lisp) as an interface.
- calpaterson 9y agoI use dropbox for multi-master syncing. Seems to work ok. Emacs even has some file locking stuff that seems to help prevent me from clobbering changes I've made on another machine
- akkartik 9y agoYou mean you're modifying files stored in Dropbox from your phone? What sort of editor setup do you use?
- calpaterson 9y agoSorry no - I mean between desktop machines. There are a few different android apps for the phone (I think they are Dropbox based) though I don't use any.
- secstate 9y agoCheck out orgzly: http://www.orgzly.com/ http://www.orgzly.com/
- Spivak 9y agoAs a user of Orgzly it's fine for basic notes and task management but you hit the limits of what it can do fairly early. It's a small partial Java implementation of the Org file format.
- clhodapp 9y agoI know it's being worked on but every time I see posts about how great magit is, it pains be because my experience is that it continues to be ungodly slow on MacOS. Other MacOS/Emacs folks, is my experience considered atypical at this point?
- weaksauce 9y agoHow slow is ungodly slow to you? It's pretty slow on Firefox's code base but that repo is huge. For most things it's not lightning fast but definitely not ungodly slow.
- qdot76367 9y agoOh, yeah, gecko is not a fair comparison at all for any git utility. Even on my beefy linux box, m-c takes an age to load under magit. Even doing CLI-based interactive rebases takes a while. Yay 300k+ commits.
- weaksauce 9y agoIs there a tool for git that only downloads the last say 1000 or 2000 commits or is a full history tree fundamental to git working correctly?
- yorwba 9y agoIt's the --depth option for git clone/fetch: https://www.git-scm.com/docs/git-clone#git-clone---depthltdepthgt https://www.git-scm.com/docs/git-clone#git-clone---depthltde...
- Cacti 9y agoYes it is atypical. Have you tried looking through the github issues on the subject? There seem to be a number of possibilities, depending on your environment.
- bennofs 9y agoAre you using stock magit? I think a few of the GitHub integration plugins for example have performance issues, so it may not be Magit's fault but some of the other extension you're using.
- brandonmenc 9y agoIs there a good side-by-side video of someone using magit vs the command line? I keep reading the documentation for magit, and I can't really imagine it giving me a huge advantage over GitX plus good command line completion settings.
- MaikuMori 9y agoI don't have side by side comparison, but I'm adept at command line and I still use Magit even when I have command line or other IDE available. Magit is my go to tool, I only use command line to quickly switch branches sometimes or for non-standard things. Things I love about Magit: - CHUNKING .. staging chunks or unstaging chunks is so easy (my favorite feature) - General overview of information - diffs, rebases, etc - It's really fast when you learn the key chords Other thing I use for Git: - Command line (if I'm in command line and need to change a branch, might as well do it there) - Tig (Log is fine in Magit too, just old habbits)
- confounded 9y agoIt’s mainly that things happen in a couple of key presses instead of a few typed words. For example to stage things you can just tap ‘s’ while running your cursor down a list of changed lines/files. To commit, it’s just ‘c c’, and you’re in an editor, in a markdown-like mode, with hard wrapping lines at a good default. Amending previous commits is also very easy, and there are some quality of life abstractions involving branches which I’ve found very useful. But, ultimately, it’s all the same stuff. If you have a good system with Gitx at the moment (and not much of an interest in Emacs), I wouldn’t worry about it.
- mtzet 9y agoI'll second the chunking feature. Magit is very good at staging and unstaging files and PARTS of a file. I don't know any good way of doing this on the command line.
- peterbecich 9y ago"ag" and "ag.el" have been of great utility to me. Together they replace "find-grep," and exclude files in a relevant .gitignore from the search. https://github.com/Wilfred/ag.el https://github.com/Wilfred/ag.el Read find-grep exclusion patterns from a .findignore file: https://emacs.stackexchange.com/a/19038/6682 https://emacs.stackexchange.com/a/19038/6682
- ryukafalz 9y agoSeconded. ag is amazing on its own, but really shines in emacs since all the results have hyperlinks to their files/line numbers.
- matt_wulfeck 9y agoThis is a cool use of emacs, but containers have made the process of iterative development/system creation so much easier. If you have the deployment story down, it’s even easier. And the configs are much more portable.
- deleted 9y ago[deleted]
- nine_k 9y agoThe process demonstrated can be adapted to a wide variety of environments, containers or not. It's mostly about making a literate program, executable on the spot, sort of like a Jupyter notebook, only for shell commands (or nearly anything, frankly).
- jle17 9y agoFWIW, Jupyter notebooks also have a kernel to support shell commands.
- nimbius 9y agomuch of whats covered in org mode can be handled by vim's folding option. if you write ansible playbooks with coworkers who handle automation as a stream of consciousness out of a single file, folding can save you some sanity. http://vim.wikia.com/wiki/Folding http://vim.wikia.com/wiki/Folding
- nine_k 9y agoSince we're off-topic anyway: have you tried Spacemacs? It's all the ingrained vim keybindings on top of a... more powerful system.
- ZenoArrow 9y agoPersonally, if I was going to give Vim a try again I'd use Oni: https://github.com/onivim/oni https://github.com/onivim/oni
- jlarocco 9y agofolding is a convenience feature of org-mode, and really not what it's all about. I'm sure most of the individual features are available in vim and other editors, but it's tying everything together in a neat package that's the main draw of org-mode.
- preek 9y agoThe ability to fold is defintively important to org-mode, but that's not really the point. Folding is a core-feature of Emacs (hs-minor-mode: https://www.gnu.org/software/emacs/manual/html_node/emacs/Hideshow.html https://www.gnu.org/software/emacs/manual/html_node/emacs/Hi...) and not inherent to org-mode. org-mode is a major-mode and covers a little more than that^^
- quiq 9y agoDo vim's folds give you a full literate programming environment with support for executing code, saving results in vars, WEB syntax, etc? From what I'm familiar with, vim's folding is a lot closer to outline-mode[1], which org relies on for folding. The folding is a very small part of org-mode's functionality. [1] https://www.gnu.org/software/emacs/manual/html_node/emacs/Outline-Mode.html https://www.gnu.org/software/emacs/manual/html_node/emacs/Ou...
- mjmein 9y agoOrg-mode and babel sound like good ideas, but in practice I've found it very finicky to use. My main problem with it is that, unlike with executing commands in the shell, with org you don't get any progress, and emacs seems to hang while running the command. For example, running apt update tends to freeze emacs without any progress reporting for a few minutes. I'm hoping that I'm doing something wrong and that someone can chime in with a suggestion, because I really like org-mode, and I really like the idea of doing devops using it.
- Spivak 9y agoIt's not 'freezing' Emacs so to speak, source block execution is synchronous. Check out: https://github.com/astahlman/ob-async https://github.com/astahlman/ob-async if you want to change that.
- mjmein 9y agoThanks. I've tried that in the past. The problem is still that it doesn't give any feedback on progress. I still struggle to see how people actually use this with devops on a day to day basis. I think that some type of integration with tmux would work well - have a running tmux session, and then just queue commands from emacs to that session, instead of running directly through emacs.
- gkya 9y agoIf you want to see progress, it should be easy to add a process filter somewhere (I bet there already is a hook for that, but not that hard to do even if there isn't one) and make it display sth. on a secondary buffer. That's the thing with Emacs: if you don't have it, it's very easy to add it.
- ZenoArrow 9y agoSo "Literate DevOps" is writing comments in your shell scripts and running the commands line by line? I agree it's a good idea. I do similar things when writing PowerShell scripts (can run them line by line using PowerShell ISE). EDIT: If you're going to downvote me, it'd be better to explain why.
- elviejo 9y agoI didn't downvote but I'll try to explain. Is the fact that you got the idea exactly backwards: is not about putting comments on your scripts. Is about writing an article, intended to be read by humans, that is also executable. Imagine it this way: Suppose you are reading a blog post on how to install, configure vagrant and install drupal on it. The article is pretty clear an well written and you enjoy it. And you also know that is correct because every single one of the instructions shown was executed while you were reading the article and every output was given back to you. That is literate programming, is closer to a jupyter notebook than to 'comments on scripts'.
- ZenoArrow 9y ago> "That is literate programming, is closer to a jupyter notebook than to 'comments on scripts'." Surely that's a matter of perspective? Imagine you have a text document containing one command and one description of that command. Is this a literate programming document or not? Furthermore, does the mode change depending on which tool is used to open the source file? For example, if I open a Bash script in a text editor then it's a document, and if I open it in Bash then it's a script, but if I could open it in a tool that allows me to both read and execute the script at my leisure does it then become a literate programming document? Aside from this, to use your Jupyter notebook example, is it necessary for the results of execution to be captured in-line in order to follow the literate programming methodology?
- jonnycomputer 9y agoAnswering questions one doesn't like is better than voting them down.