22 ms·
Wow, I wasn't expecting my email to Jeff to end up as a front-page blog post! The point here is that Markdown doesn't have a spec, nor do any of its variants t
by dgreensp 14y ago
Wow, I wasn't expecting my email to Jeff to end up as a front-page blog post!
The point here is that Markdown doesn't have a spec, nor do any of its variants to my knowledge, so I was proposing to come up with some Markdown-like language that does have a spec. Under discussion here is the more ambitious (but also appealing) plan of writing an official spec for Markdown, the same way JavaScript got a spec in the form of ECMAScript that we now identify with JavaScript itself.
A spec is a long, tedious, human-readable document that explains the behavior of a system in unambiguous terms. Specs are important because they allow us to reason about a language like Markdown without reference to any particular implementation, and they allow people to write implementations (Markdown processors) independently that behave identically. The Markdown Syntax Documentation is not a spec (it's highly ambiguous), nor is any implementation (not human-readable; some behaviors are probably accidental or incidental and difficult to port perfectly). The hard part of writing a spec is codifying the details in English, and secondarily making decisions about what should happen in otherwise ambiguous or undefined cases.
My motivation for working on a Markdown spec is first and foremost avoiding "bit rot" of content, which happens when we write content against one Markdown implementation and then later process it with another. We don't have this concern with HTML, JSON, or JavaScript, or at least we know what bounds to stay within to write code that will work on any implementation. This is achieved through specs, even if only implementers ever read them.
I would love pointers to Markdown processors that are implemented in a more principled way than the original code, for example using standard-looking lexing and parsing passes, but that still handle nested blockquotes and bullet lists together with hard-wrapped paragraphs.
- halostatue 14y agohttps://github.com/jgm/peg-markdown https://github.com/jgm/peg-markdown and https://github.com/fletcher/peg-multimarkdown https://github.com/fletcher/peg-multimarkdown I haven't looked at the implementations of these, but they are most certainly grammar-based, not regex-based.
- eslaught 14y agoYou should look at Pandoc[1]. It's a Markdown-to-everything converter, and though I'm not super familiar with the code I believe it's well written. One of my favorite tools for writing. [1]: http://johnmacfarlane.net/pandoc/ http://johnmacfarlane.net/pandoc/
- kenko 14y agoPandoc is fantastic, the parsing code is clear, and its extensions to markdown are well thought out. It's not just a markdown->everything converter, though. My install understands these: Input formats: native, json, markdown, markdown+lhs, rst, rst+lhs, textile, html, latex, latex+lhs Output formats: native, json, html, html5, html+lhs, html5+lhs, s5, slidy, dzslides, docbook, opendocument, latex, latex+lhs, beamer, beamer+lhs, context, texinfo, man, markdown, markdown+lhs, plain, rst, rst+lhs, mediawiki, textile, rtf, org, asciidoc, odt, docx, epub
- intractable 14y ago+1 for Pandoc. It's awesome. Also, it needs to be mentioned that Pandoc supports its own set of (IMO) sensible extensions / defaults to Gruber markdown [1]. [1] http://johnmacfarlane.net/pandoc/README.html#pandocs-markdown http://johnmacfarlane.net/pandoc/README.html#pandocs-markdow...
- sandGorgon 14y agokramdown[1] is a ruby based Markdown parser. AFAIK it is P-E-G based and has some nifty support for tables, etc. [1] http://kramdown.rubyforge.org/ http://kramdown.rubyforge.org/
- Loic 14y agoYou also need to add that the parser of Pandoc is not regular expression based. It is a very robust parser, parsing the document in an internal abstract tree you can easily modify. You can also adapt the parser to your needs with simple extensions. You can then get very nice outputs in HTML or PDF[1]. As it is a nice command line tool, it is easy to generate everything with a simple Makefile too. [1]: http://notes.ceondo.com/mongrel2-zmq-paas/ http://notes.ceondo.com/mongrel2-zmq-paas/
- evolve2k 14y agoYes I came here also to add a +1 to ensuring that whatever standards discussions would include Pandoc and it's awesome implementation and converter. From the link, scroll down to 'Pandoc’s Markdown' about a third of the way down the page for all the details. http://johnmacfarlane.net/pandoc/README.html http://johnmacfarlane.net/pandoc/README.html
- someone13 14y agoAnother good parser is python-markdown2 [1], which has a very extensive set of test cases, and the code is very readable and well-commented. [1] https://github.com/trentm/python-markdown2 https://github.com/trentm/python-markdown2
- csaba161 14y agojust checked this, not very good
- fiddlosopher 14y ago[lunamark](https://github.com/jgm/lunamark/tree/master/lunamark https://github.com/jgm/lunamark/tree/master/lunamark) is another PEG-based implementation.
- greggman 14y agoSpecs are important but I'd argue a conformance test is equally or even more important. With a conformance test people can test their implementations and when ambiguities arrise new tests can be added or old tests fixed. Without, conformance tests different interpretations of a spec lead to divergent behavior. Once that behavior is out there long enough it becomes difficult to fix as people are depended on it and/or its quirks.
- mistercow 14y agoI'm confused. How are you supposed to have a conformance test without first having a spec? To what would the implementations be conforming, if not a spec?
- greggman 14y agoSorry, What I meant to really say is specs are not sufficient. Sure specs are important. But even with specs, without comprehensive conformance tests implementations will diverge. I work on the WebGL spec and tests based off the OpenGL ES spec and tests. Even though the OpenGL ES spec is long and detailed, every edge case for which there is no test is broken or different in one driver or another.
- ZeroGravitas 14y agoIt's entirely possible to build a conformance test suite around an existing 'black box' of code and accept that whatever it does, even stuff that might be considered a bug, is what any compliant implementation should also do. (You could also fix some of the more egregious bugs, and then expect implementations to follow your new behaviour)
- jamesrom 14y agoIn an ideal world, the spec would be the test. A spec is just a human readable test. A test is just a machine readable spec.
- th 14y ago> Without, conformance tests different interpretations of a spec lead to divergent behavior. I think you mean either: > Without conformance tests incorrect interpretations of a spec lead to divergent behavior Or > Without conformance tests different interpretations of an ambiguous spec lead to divergent behavior. Conformance tests should test specific behavior defined in the spec. Tests will never (or rarely) prove spec conformance but spec conformance should always prove test conformance.
- sandGorgon 14y agoOne of the requests I want to make is to make Markdown suitable for a ebook toolchain. This means some of the abilities from Tex, like including one MD file in another. Including images, etc. A lot of these would not impact Markdown as such, but would be a massive boon for those doing self-publishing. I do understand that image alignment is not a light topic, but it is an opportunity to define a text/image layout standard for reflowable content. I dont know if Tex does that well enough.
- Helianthus 14y agoI don't know if Reddit's implementation of Markdown is principled, but as a syntax it's pretty ubiquitous so you might have the traction to make it standard.
- aleemb 14y agoI hope some more thought goes into the usability as well. I find the syntax for links confuses a lot of users for whom I have enabled mark down editing. Wiki style [[link text]] works great if you assume links have no space or maybe something similar. Similarly the syntax for images has also always bothered me.
- vidarh 14y agoI have no problems remembering the Markdown link format, but I never remember the expected order in what you describe as "Wiki style" (which is really a style employed by some wiki software - there are plenty of wiki software that uses other syntaxes, including Markdown)
- buro9 14y agoI agree with all of this. I'd also like to note that some parts of Markdown from the user perspective are non-intuitive and clumsy. Such as links and images (inline). Markdown works so well because it is intuitive and appeals to those who once saw old word processors. They don't have to worry about syntax, and can just enter their text into a textarea (free from JavaScript WYSIWYG interference and the inherent troubles of running that on old and new mobile phones, their playstation, web browsers, etc)... and it just works. Yet some parts of markdown are simply not intuitive. Links and images are two places where I see in usability testing that the end user will constantly refer to help documentation to figure out how to do it. Beyond getting the code consistent, maintainable, and testable I'd love to see the language itself solve some of the papercuts that trouble the lay end user. Realising that what I was planning to do for my project (discussion forums, tumblr for forums) was to create an alternative to markdown that would resolve some of these user issues as well as parser issues, I had already decided that I would not call it markdown and that I would educate my users in something new that hopefully solves their and my needs and would remain very stable due to a well-thought out and documented design in the first place. If what you're proposing is in this vein then consider my hat thrown in, what help I can give I will... take me to your git repository.
- alexchamberlain 14y agoWhat is your problem with the link syntax? I think it's one of the best bits.
- jacobr 14y agoYou have to remember 1) which comes first, url or link text? 2) is it () or []?
- alexchamberlain 14y agoI'm just wondering whether a better solution to this is a polite warning in the UI... Did you mean []() instead of ()[]?
- bitcartel 14y agoWhy are people still interested in Markdown - what problem is it solving and for whom? It's 2012 and we have HTML 5 compliant WYSIWYG text editors. We no longer have to write plain text littered with special codes, for the purpose of running through a parser, to produce HTML which looks nice on a web page. Maybe it made sense a decade ago when web forms had terrible editors, but not anymore. I think Joe Internet writing blog posts and forum replies would agree with me. For developers, a README file in plain text looks great everywhere, and avoids any Github vs Bitbucket vs Assembla display issues. If you need to write structured documentation for a system, there's probably already a designated markup language you're supposed to use, so Markdown doesn't help there.
- krsunny 14y agoBecause in 2012 there are still lots of plain textareas, like the one I'm writing this reply in.
- keithpeter 14y ago"Why are people still interested in Markdown - what problem is it solving and for whom?" I understand the question and wonder why it attracted down-votes. The executive summary of my answer: light markup is used in places other than text areas on Web sites. Many of us use Markdown to keep formatted text in, and, ultimately, to generate valid html markup from. Markdown is not just for textareas in Web pages. Whole books have been marked up in markdown. Markdown files are just text files with 'explicit' markup within them and so will always be accessible/editable/processable in the future. Markdown is 'flexible' in the sense that it allows alternative markup for the same output, and is 'light' in the sense that a limited range of styling and block formats are available. Compare that with LaTeX which shifts with each tex-live release, and which means that markup written some years ago may not format without hand editing on a modern LaTeX release. There is the reference implementation available from Gruber's site, and, as others have mentioned, pandoc implements markdown in a way that can be extended. I'm dubious of the need for an 'official' specification; I rather like the scrappy streetwise nature of Markdown.
- JadeNB 14y ago
- arnarbi 14y agoI do not agree that a spec has to be written in English. Like most natural languages, it is very easy to be ambiguous and to underspecify. Resolving this can make English just as human un-readable as anything else. Specs like this should be written as executable reference implementations in a well defined programming language. This can very well be human readable, and should be done without regard for efficient execution. It's less ambiguous, amenable to automated conformance testing, and is easier to evolve than a natural language document.
- chc 14y agoIf that's the spec, how do we know which aspects of the executable are really part of the specification and which are implementation details? For (a somewhat extreme for the sake of illustration) example, if the spec is written for Ruby 1.8, do C implementations have to intentionally slow down to match its execution speed?
- Too 14y agoUse of well established standards. For example BNF or railroad diagrams can cover a majority of a language specificaton without using English.
- chc 14y agoI think you're unintentionally moving the goalposts here. The idea I was questioning wasn't "Specs should be written in something other than English," it was "Specs should take the form of an executable reference implementation." AFAIK railroad diagrams are not executable. I'm also not sure how BNF is supposed to specify things like "the map function applies a function passed in to each argument in a list".
- arnarbi 14y agoOf course not. Such a spec would state what are its inputs and outputs, either as natural language descriptions or as types (say, if the reference language supports it). Everything outside that (e.g. timing) would be fair game to optimize. Moreover, as I hinted at, a spec should not be optimized for speed or efficiency, but rather readability and unambiguity.
- ExpiredLink 14y agoIMO, for the spec the most important point is to add mandatory versioning information on the first line of markdown scripts, e.g. _markdown_version_ Rockdown_1.0 This would allow processor implementers to support more than one markdown version (for a transition period or in general).
- jdbernard 14y agoUgh, no. This defeats the simplicity of Markdown. I do not want to have to remember some obscure implementation versioning string whenever I want to start a new document or comment. Maybe as an option, sure, but I will probably never use it.
- draegtun 14y ago>I would love pointers to Markdown processors that are implemented in a more principled way than the original code, for example using standard-looking lexing and parsing passes... Have a look at Markdent - An event-based Markdown parser toolkit. https://metacpan.org/module/Markdent https://metacpan.org/module/Markdent
- gordonguthrie 14y agoI wrote a manual parser with look-aheads in Erlang (making extensive use of Erlang's pattern matching. https://github.com/hypernumbers/erlmarkdown https://github.com/hypernumbers/erlmarkdown No regexps.... except for url's and emails :(
- jamii 14y ago> I would love pointers to Markdown processors that are implemented in a more principled way than the original code This is almost what you want: http://news.ycombinator.com/item?id=555153 http://news.ycombinator.com/item?id=555153 EDIT: wrong link :(
- uvtc 14y agoI'm not so sure what's needed here is a spec. What people really want is for all Markdown implementations to be basically the same so they don't have to learn any implementation-specific ideosyncrasies to switch from one to another. The problem though, as I see it, tends to go away under certain circumstances. And the circumstances, I think, are these: * If you've got a strong implementation, robust, actively-maintained, and runs fast, * is easy to obtain, install, and use, * is well tested and well documented, and * has just the right blend of sensible additions to the syntax (for example, tables, def lists, LaTeX math, etc.) --- done tastefully, then folks will just use that, model their own implementations after that, and just overall start considering that to be the standard. I think this has been slowly and steadily happening with Pandoc. And, aside from all that, two additional "killer features" that Pandoc seems to have over other implementations: 1. it can convert to/from other doc markup formats, thus making it easy to just convert your existing docs to pandoc-markdown and then use that as your master source format to generate other formats you might need; and 2. with its carefully-chosen set of additional features, it has been slowly proving itself capable of being a replacement for raw LaTeX for certain types of longer technical documents. My understanding is that there's even some features in the works (for the next release) for converting between markdown dialects --- which would make it even easier to convert markdown files of various flavors into plain standard pandoc-markdown. So, if you're looking for a standard, I'd suggest that it's for the most part already here. :)