6 ms·

Show HN: Jq-Like Tool for Markdown

325 points by yshavit 2y ago

There have been a few times I wanted the ability to select some text out of a Markdown doc. For example, a GitHub CI check to ensure that PRs / issues / etc are properly formatted. This can be done to some extent with regex, but those expressions are brittle and hard to read or edit later. mdq uses a familiar pipe syntax to navigate the Markdown in a structured way. It's in 0.x because I don't want to fully commit to the syntax being stable, in case real-world testing shows that the syntax needs tweaking. But I think the project is in a pretty good spot overall, and would be interested in feedback!

lanstin 2y ago
Ironically one of the reasons markdown (and other text based file formats) were popular because you could use regular find/grep to analyze it, and version control to manage it.
- cdbattags 2y ago
  Definitely, but it's neat nonetheless because more and more things are "structured Markdown" these days. Extremely useful for AI reasoning and outputs.
  - llm_trw 2y ago
    Man if we only had some type of markdown meant for machines to understand, that was specifically designed to handle arbitrary information nesting and tagging our lives would be so much better now. We could have called it something like extended markdown language or something and use a wicked acronym like eXMaLa for it. Shame no such technology exists and never did.
- monsieurbanana 2y ago
  > because you could use regular find/grep to analyze it They were meant to be analyzable in some ways. Count lines, extract headers, maybe sed-replace some words. But being able to operate/analyze over multiline strings was never a strong point of unix tools.
- zahlman 2y ago
  I don't think anyone ever really expected to see widespread use of regexes to alter the structure of a Markdown document. Honestly, while something like "look for numbers and surround them with double-asterisks to put them in boldface" is feasible enough (and might even work!), I can't imagine that a lot of people would do that sort of thing very often (or want to) anyway. If a document is supposed to have structure - even something as simple as nested lists of paragraphs - it doesn't seem realistic to expect regular text manipulation tools to do a whole lot with them. Something like "remove the second paragraph of the third entry in the fourth bullet-point list" is well beyond any sane use of any regex dialect that might be powerful enough. (Keeping in mind that traditional regexes can't balance brackets; presumably they can't properly track indentation levels either.) See also: TOML - generally quite human-editable, but still very much structured with potentially arbitrary nesting.
  - kmstout 2y ago
    > (Keeping in mind that traditional regexes can't balance brackets; presumably they can't properly track indentation levels either.) You're right: Regular expressions are equivalent to finite state machines[1], which lack the infinite memory needed to handle arbitrarily nested structures [2]. If there is a depth limit, however, it is possible (but painful) to craft a regex to describe the situation. For example, suppose you have a language where angle brackets serve as grouping symbols, like parentheses usually do elsewhere [3]. Ignoring other characters, you could verify balanced brackets up to one nesting level with /^(<>)*$/ and two levels with /^(<(<[^<>]*>|[^<>])*>)*$/ Don't do this when you have better options. --- [1] https://reindeereffect.github.io/2018/06/24/index.html https://reindeereffect.github.io/2018/06/24/index.html [2] As do any machines I can afford, but my money can buy a pretty good illusion. [3] < and > are not among the typical regex metacharacters, so they make for an easier discussion.
  - lanstin 2y ago
    I think that prospect (of programmatically structurally editing markdown files) would have made everyone burst out in laughter in 2000; if you want to programmatically alter stuff, put it into sexp's or some other syntax. SGML. Apparently human readable but really a tricky format leads to this sort of thing: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...
twinkjock 2y ago
Thanks for sharing this Yuval! Thanks as well for using permissive licenses so I can use this at work.
- imglorp 2y ago
  Curious, which license can't you use at work for a simple shell tool? Considering you're not linking against it, even GPL3 should be okay, right?
  - twinkjock 2y ago
    Most big employers consider GPL unusable and will only allow Apache, MIT, BSD, and other permissive patent-free licenses.
    - ta8903 2y ago
      So they don't let you use grep? Sounds like hell.
    - imglorp 2y ago
      Yeah, caution with GPL is understandable if you're shipping it or making it a part of a hosted service. But internal use is not the intention of GPL, so a blanket ban like that would put that shop at a disadvantage, like the other comment said, that basically rules out all kinds of things.
unglaublich 2y ago
My flow is to go through the Pandoc JSON AST and then use Jq. This works for other input formats, too.
- yshavit 2y ago
  I'm curious how ergonomic you find that? I did look at the pandoc JSON initially, and found it fairly awkward to work with. It's a great interchange format, but doesn't seem optimized for either human interaction or scripting. (It's definitely possible to use it for scripting, it just felt cumbersome to me, personally.)
- saghm 2y ago
  I've never had a need for parsing markdown like this, bit I have to wonder, would it make to go through HTML instead, given that it's what markdown is designed to compile to? At that point, I'd assume there's any number of existing XML tools that work work, and my (maybe naive) assumption is that typical markdown documents would be relatively flat compared to how deeply nested "native" HTML/XML often gets, so it doesn't seem like most queries would require particularly complex XPath to be able to specify.
  - MathMonkeyMan 2y ago
    I did this for a tool that checks relative links in markdown files, e.g. readmes in a repo. markdown -> xhtml -> sxml -> logic (racket)
nodesocket 2y ago
How is it parsing? Just normal string and regex matching or transforming markdown to an intermediate structured language?
- yshavit 2y ago
  For the markdown, I'm using https://github.com/wooorm/markdown-rs https://github.com/wooorm/markdown-rs, which is a formal parser that produces an AST. For the query language, I have a very simple hand-rolled parser.
broodbucket 2y ago
I think you'd benefit of having some more real-world-ish examples in the README, as someone who doesn't intuit what I'd want to use this for.
- iJohnDoe 2y ago
  Agreed. At least 5 examples of output shown being used against a standard markdown document.
- yshavit 2y ago
  That's a great idea, thanks! I'll do that tomorrow or so. As a preview, two specific cases I've seen: 1) In PRs, some companies like to have semi-structured metadata, like a link to a related ticket under a heading "Ticket". In mdq, you could find that using `# Ticket | [](^https://issues.acme.com/ https://issues.acme.com/)` 2) Many projects ask people who submit bugs to check off whether they've searched for existing bugs. `- [x] I've looked in the bug tracker for existing bugs`
Hamzahkm 2y ago
[flagged]
verdverm 2y ago
> GitHub PRs are Markdown documents, and some organizations have specific templates with checklists for all reviewers to complete. Enforcing these often requires ugly regexes that are a pain to write and worse to debug This is because GitHub is not building the features we need, instead they are putting their energy towards the AI land grab. Bitbucket, by contrast, has a feature where you can block PRs using a checkbox list outside of the description box. There are better ways to solve this first example from OP readme. Cool project, I write mainly MDX these days, would be cool to see support for that dialect
- yshavit 2y ago
  The Markdown parsing library I'm using supports MDX, so it shouldn't be too difficult to come up with syntax for those components. I haven't done that yet, but mostly because I didn't want to go down that path until I knew there was interest and had a concrete use case or two to inform the query syntax. If you want to open an enhancement request issue, I'm happy to take a look (PRs also welcome, but not required). If you're not on GitHub, let me know and we can figure out some other way to get the request tracked. Thanks for taking a look at the project!
  - verdverm 2y ago
    I don't write rust and already have an MDX toolbox that fits my needs. Browser, GH, and IDE search / TOC are good enough for me. I'm currently in a phase of trying to shed tools and added complexity, rather than add them
    - yshavit 2y ago
      Fair enough!
- codelion 2y ago
  it's a shame when core feature development seems to lag. i've also been working w/ MDX lately & agree that support would be a great addition.
- 2y ago
dleeftink 2y ago
Kind of aligned with this is MarkdownDB, providing an SQLite backend to your Markdown files [0]. Cool to see this, I feel the structure of .md files is not always equally respected or regarded as a data serialisation target. [0]: https://markdowndb.com/ https://markdowndb.com/
aqueueaqueue 2y ago
Why not MD -> json, then use jq! That would be half a static site generator there!
moonshotideas 2y ago
Love this! One persons opinion - I’d change it to mq - less chars are always better for command
kbd 2y ago
Cool thanks for sharing! I'll have to check this out. I've wanted something similar. After trying a bunch of the usual ones, the only "notes system" I've stuck with is just a directory of markdown files that's automatically committed to git on any change using watchexec. I've wanted to add a little smarts to it so I could use it to track tasks (eg. sort, prune completed, forward uncomplete tasks over to the next day's journal, collect tasks from "projects", etc.) so I started writing some Rust code using markdown-rs. Then, to round-trip markdown with changes, only the javascript version of the library currently supports serializing github flavored markdown. So then I actually dumped the markdown ast to json from rust and picked it up in js to serialize it for a proof of concept. That's about as far as I got so far. But while markdown-rs saves position information, it doesn't save source token information (like, * and - are both list items) so you can't reliably round-trip. FWIW, the other thing I was hoping to do was treat markdown documents as trees (based on headings) use an xpath kind of language to pull out sections. Anyway, will check out your code, thanks for posting.
threecheese 2y ago
Interesting; one thing you may have learned researching existing tools and libraries: many of them serialize markdown to html before running structured extraction/manipulation - even stuff like converting to pdf. The core assumption here is that Markdown was/is designed to be serializeable to html - this is why a markdown document/AST is mostly not a tree structure, for tree-ish elements such as sub-sections. Instead, it is flat, an array of elements in order of appearance in the document. Apparently this most closely matches the structure of html, at both the block and inline levels. Only Lists and Blockquotes (afair) support nesting. Ex: h1 -> paragraph -> h2 -> paragraph is not nested, it is an array of four ordered elements. Anyway, you might throw a task at Cursor or Copilot to see how an equivalent implementation using html fares against your test suite, you may be able to develop more quickly.
dcreater 2y ago
What purpose does this serve that grep doesn't?
foo42 2y ago
This is one of those moments where you come across a tool _just_ at the right moment. I have a task for which this will be perfect
spiffyk 2y ago
Thanks for sharing! No immediate use-case for me right now, but good to know something like this exists. I wanted to point out little nitpicks for the documented shell invocations: cat example.md | mdq '# usage' This can be changed into a stdin file redirect to avoid invoking an extra `cat` process (see Useless use of cat [1]): mdq '# usage' < example.md In a similar fashion, you can avoid an extra `echo` process here: echo "$ISSUE_TEXT" | mdq -q '- [x] I have searched for existing issues' by changing to this: mdq -q '- [x] I have searched for existing issues' <<< "$ISSUE_TEXT" [1]: https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat
- mzl 2y ago
  From the linked web-page A cat written with UUOC might still be preferred for readability reasons, as reading a piped stream left-to-right might be easier to conceptualize.[14] Also, one wrong use of the redirection symbol > instead of < (often adjacent on keyboards) may permanently delete the content of a file, in other words clobbering, and one way to avoid this is to use cat with pipes. I personally think the cat-style is easier to read since it only uses commands and pipes, with no need to keep track of redirection directions.
zerkten 2y ago
Thanks! I have to grapple with some markdown across multiple repos and this'll be a helpful tool in the toolchest.
linklater12 2y ago
congrats on your tool, will check it out. I have a side question on markdown: cursor messes up markdown generation quite often for me. I think its responses are always in markdown with sections for code and asking it to generate markdown breaks it. So the question: any ideas on how to have cursor generate markdown?
pokstad 2y ago
Please don’t reimplement JQ. That problem is already solved. Instead, just provide a tool that can convert your target syntax into JSON, then it can be piped to JQ for querying.
infogulch 2y ago
I've always wanted a "literate programming" / jupyter-style notebook based on markdown. Maybe this could help make something like that possible.
- arminiusreturns 2y ago
  Emacs org-mode and babel are what I use instead. Very powerful alternative to Jupyter imho, but of course not for everyone. I have used commit hooks so that non-emacs users could push updates that still get executed to avoid the emacs tie in though, the biggest issue with it I've found.
frankfrank13 2y ago
I worked on a project converting word docs to markdown so they could more easily be ingested into an LLM, one issue was that context windows used to be very short, so we would basically split on `\n#` to get sections, but this turns into a whole thing where you have to make guesses about which header level is appropriate to split at, and then you turn each section into a separate chunk in FAISS. Anyways we ended up using HTML instead of MD but theres so much tooling for traversing HTML and not MD. This would have been helpful for that
breck 2y ago
[dead]

325 points by yshavit 2y ago

lanstin 2y ago

Ironically one of the reasons markdown (and other text based file formats) were popular because you could use regular find/grep to analyze it, and version control to manage it.

cdbattags 2y ago
Definitely, but it's neat nonetheless because more and more things are "structured Markdown" these days. Extremely useful for AI reasoning and outputs.
- llm_trw 2y ago
  Man if we only had some type of markdown meant for machines to understand, that was specifically designed to handle arbitrary information nesting and tagging our lives would be so much better now. We could have called it something like extended markdown language or something and use a wicked acronym like eXMaLa for it. Shame no such technology exists and never did.
monsieurbanana 2y ago
> because you could use regular find/grep to analyze it They were meant to be analyzable in some ways. Count lines, extract headers, maybe sed-replace some words. But being able to operate/analyze over multiline strings was never a strong point of unix tools.
zahlman 2y ago
I don't think anyone ever really expected to see widespread use of regexes to alter the structure of a Markdown document. Honestly, while something like "look for numbers and surround them with double-asterisks to put them in boldface" is feasible enough (and might even work!), I can't imagine that a lot of people would do that sort of thing very often (or want to) anyway. If a document is supposed to have structure - even something as simple as nested lists of paragraphs - it doesn't seem realistic to expect regular text manipulation tools to do a whole lot with them. Something like "remove the second paragraph of the third entry in the fourth bullet-point list" is well beyond any sane use of any regex dialect that might be powerful enough. (Keeping in mind that traditional regexes can't balance brackets; presumably they can't properly track indentation levels either.) See also: TOML - generally quite human-editable, but still very much structured with potentially arbitrary nesting.
- kmstout 2y ago
  > (Keeping in mind that traditional regexes can't balance brackets; presumably they can't properly track indentation levels either.) You're right: Regular expressions are equivalent to finite state machines[1], which lack the infinite memory needed to handle arbitrarily nested structures [2]. If there is a depth limit, however, it is possible (but painful) to craft a regex to describe the situation. For example, suppose you have a language where angle brackets serve as grouping symbols, like parentheses usually do elsewhere [3]. Ignoring other characters, you could verify balanced brackets up to one nesting level with /^(<>)*$/ and two levels with /^(<(<[^<>]*>|[^<>])*>)*$/ Don't do this when you have better options. --- [1] https://reindeereffect.github.io/2018/06/24/index.html https://reindeereffect.github.io/2018/06/24/index.html [2] As do any machines I can afford, but my money can buy a pretty good illusion. [3] < and > are not among the typical regex metacharacters, so they make for an easier discussion.
- lanstin 2y ago
  I think that prospect (of programmatically structurally editing markdown files) would have made everyone burst out in laughter in 2000; if you want to programmatically alter stuff, put it into sexp's or some other syntax. SGML. Apparently human readable but really a tricky format leads to this sort of thing: https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-from-hell https://ruudvanasseldonk.com/2023/01/11/the-yaml-document-fr...

twinkjock 2y ago

Thanks for sharing this Yuval! Thanks as well for using permissive licenses so I can use this at work.

imglorp 2y ago
Curious, which license can't you use at work for a simple shell tool? Considering you're not linking against it, even GPL3 should be okay, right?
- twinkjock 2y ago
  Most big employers consider GPL unusable and will only allow Apache, MIT, BSD, and other permissive patent-free licenses.
  - ta8903 2y ago
    So they don't let you use grep? Sounds like hell.
  - imglorp 2y ago
    Yeah, caution with GPL is understandable if you're shipping it or making it a part of a hosted service. But internal use is not the intention of GPL, so a blanket ban like that would put that shop at a disadvantage, like the other comment said, that basically rules out all kinds of things.

unglaublich 2y ago

My flow is to go through the Pandoc JSON AST and then use Jq. This works for other input formats, too.

yshavit 2y ago
I'm curious how ergonomic you find that? I did look at the pandoc JSON initially, and found it fairly awkward to work with. It's a great interchange format, but doesn't seem optimized for either human interaction or scripting. (It's definitely possible to use it for scripting, it just felt cumbersome to me, personally.)
saghm 2y ago
I've never had a need for parsing markdown like this, bit I have to wonder, would it make to go through HTML instead, given that it's what markdown is designed to compile to? At that point, I'd assume there's any number of existing XML tools that work work, and my (maybe naive) assumption is that typical markdown documents would be relatively flat compared to how deeply nested "native" HTML/XML often gets, so it doesn't seem like most queries would require particularly complex XPath to be able to specify.
- MathMonkeyMan 2y ago
  I did this for a tool that checks relative links in markdown files, e.g. readmes in a repo. markdown -> xhtml -> sxml -> logic (racket)

nodesocket 2y ago

How is it parsing? Just normal string and regex matching or transforming markdown to an intermediate structured language?

yshavit 2y ago
For the markdown, I'm using https://github.com/wooorm/markdown-rs https://github.com/wooorm/markdown-rs, which is a formal parser that produces an AST. For the query language, I have a very simple hand-rolled parser.

broodbucket 2y ago

I think you'd benefit of having some more real-world-ish examples in the README, as someone who doesn't intuit what I'd want to use this for.

iJohnDoe 2y ago
Agreed. At least 5 examples of output shown being used against a standard markdown document.
yshavit 2y ago
That's a great idea, thanks! I'll do that tomorrow or so. As a preview, two specific cases I've seen: 1) In PRs, some companies like to have semi-structured metadata, like a link to a related ticket under a heading "Ticket". In mdq, you could find that using `# Ticket | [](^https://issues.acme.com/ https://issues.acme.com/)` 2) Many projects ask people who submit bugs to check off whether they've searched for existing bugs. `- [x] I've looked in the bug tracker for existing bugs`

Hamzahkm 2y ago

[flagged]

verdverm 2y ago

> GitHub PRs are Markdown documents, and some organizations have specific templates with checklists for all reviewers to complete. Enforcing these often requires ugly regexes that are a pain to write and worse to debug This is because GitHub is not building the features we need, instead they are putting their energy towards the AI land grab. Bitbucket, by contrast, has a feature where you can block PRs using a checkbox list outside of the description box. There are better ways to solve this first example from OP readme. Cool project, I write mainly MDX these days, would be cool to see support for that dialect

yshavit 2y ago
The Markdown parsing library I'm using supports MDX, so it shouldn't be too difficult to come up with syntax for those components. I haven't done that yet, but mostly because I didn't want to go down that path until I knew there was interest and had a concrete use case or two to inform the query syntax. If you want to open an enhancement request issue, I'm happy to take a look (PRs also welcome, but not required). If you're not on GitHub, let me know and we can figure out some other way to get the request tracked. Thanks for taking a look at the project!
- verdverm 2y ago
  I don't write rust and already have an MDX toolbox that fits my needs. Browser, GH, and IDE search / TOC are good enough for me. I'm currently in a phase of trying to shed tools and added complexity, rather than add them
  - yshavit 2y ago
    Fair enough!
codelion 2y ago
it's a shame when core feature development seems to lag. i've also been working w/ MDX lately & agree that support would be a great addition.
2y ago

dleeftink 2y ago

Kind of aligned with this is MarkdownDB, providing an SQLite backend to your Markdown files [0]. Cool to see this, I feel the structure of .md files is not always equally respected or regarded as a data serialisation target. [0]: https://markdowndb.com/ https://markdowndb.com/

aqueueaqueue 2y ago

Why not MD -> json, then use jq! That would be half a static site generator there!

moonshotideas 2y ago

Love this! One persons opinion - I’d change it to mq - less chars are always better for command

kbd 2y ago

Cool thanks for sharing! I'll have to check this out. I've wanted something similar. After trying a bunch of the usual ones, the only "notes system" I've stuck with is just a directory of markdown files that's automatically committed to git on any change using watchexec. I've wanted to add a little smarts to it so I could use it to track tasks (eg. sort, prune completed, forward uncomplete tasks over to the next day's journal, collect tasks from "projects", etc.) so I started writing some Rust code using markdown-rs. Then, to round-trip markdown with changes, only the javascript version of the library currently supports serializing github flavored markdown. So then I actually dumped the markdown ast to json from rust and picked it up in js to serialize it for a proof of concept. That's about as far as I got so far. But while markdown-rs saves position information, it doesn't save source token information (like, * and - are both list items) so you can't reliably round-trip. FWIW, the other thing I was hoping to do was treat markdown documents as trees (based on headings) use an xpath kind of language to pull out sections. Anyway, will check out your code, thanks for posting.

threecheese 2y ago

Interesting; one thing you may have learned researching existing tools and libraries: many of them serialize markdown to html before running structured extraction/manipulation - even stuff like converting to pdf. The core assumption here is that Markdown was/is designed to be serializeable to html - this is why a markdown document/AST is mostly not a tree structure, for tree-ish elements such as sub-sections. Instead, it is flat, an array of elements in order of appearance in the document. Apparently this most closely matches the structure of html, at both the block and inline levels. Only Lists and Blockquotes (afair) support nesting. Ex: h1 -> paragraph -> h2 -> paragraph is not nested, it is an array of four ordered elements. Anyway, you might throw a task at Cursor or Copilot to see how an equivalent implementation using html fares against your test suite, you may be able to develop more quickly.

dcreater 2y ago

What purpose does this serve that grep doesn't?

foo42 2y ago

This is one of those moments where you come across a tool _just_ at the right moment. I have a task for which this will be perfect

spiffyk 2y ago

Thanks for sharing! No immediate use-case for me right now, but good to know something like this exists. I wanted to point out little nitpicks for the documented shell invocations: cat example.md | mdq '# usage' This can be changed into a stdin file redirect to avoid invoking an extra `cat` process (see Useless use of cat [1]): mdq '# usage' < example.md In a similar fashion, you can avoid an extra `echo` process here: echo "$ISSUE_TEXT" | mdq -q '- [x] I have searched for existing issues' by changing to this: mdq -q '- [x] I have searched for existing issues' <<< "$ISSUE_TEXT" [1]: https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat

mzl 2y ago
From the linked web-page A cat written with UUOC might still be preferred for readability reasons, as reading a piped stream left-to-right might be easier to conceptualize.[14] Also, one wrong use of the redirection symbol > instead of < (often adjacent on keyboards) may permanently delete the content of a file, in other words clobbering, and one way to avoid this is to use cat with pipes. I personally think the cat-style is easier to read since it only uses commands and pipes, with no need to keep track of redirection directions.

zerkten 2y ago

Thanks! I have to grapple with some markdown across multiple repos and this'll be a helpful tool in the toolchest.

linklater12 2y ago

congrats on your tool, will check it out. I have a side question on markdown: cursor messes up markdown generation quite often for me. I think its responses are always in markdown with sections for code and asking it to generate markdown breaks it. So the question: any ideas on how to have cursor generate markdown?

pokstad 2y ago

Please don’t reimplement JQ. That problem is already solved. Instead, just provide a tool that can convert your target syntax into JSON, then it can be piped to JQ for querying.

infogulch 2y ago

I've always wanted a "literate programming" / jupyter-style notebook based on markdown. Maybe this could help make something like that possible.

arminiusreturns 2y ago
Emacs org-mode and babel are what I use instead. Very powerful alternative to Jupyter imho, but of course not for everyone. I have used commit hooks so that non-emacs users could push updates that still get executed to avoid the emacs tie in though, the biggest issue with it I've found.

frankfrank13 2y ago

I worked on a project converting word docs to markdown so they could more easily be ingested into an LLM, one issue was that context windows used to be very short, so we would basically split on `\n#` to get sections, but this turns into a whole thing where you have to make guesses about which header level is appropriate to split at, and then you turn each section into a separate chunk in FAISS. Anyways we ended up using HTML instead of MD but theres so much tooling for traversing HTML and not MD. This would have been helpful for that

breck 2y ago

[dead]