8 ms·

Hope is an alternative to markup languages

49 points by gjvc 6y ago

gjvc 6y ago
repository at https://github.com/poef/hope/ https://github.com/poef/hope/
bovermyer 6y ago
Since my project doesn't use this, you could say... it has no Hope.
abnry 6y ago
Interesting idea. I've never thought of separating the markup and the text into two different files. To me, it suggests a more machine friendly way of keeping track of markup. Directly writing sounds very painful.
- gjvc 6y ago
  Yes, this is looking in the direction of a content API for documents, and is not meant to be human-generated.
abathur 6y ago
My mind's been picking at related ideas from different angles for at least a decade now, I guess. I'm not sure how I feel about this one, but one thing I like about separation approaches in general is that it's easy to imagine how N different annotations can all exist for the same underlying document, so you could use something along these lines to share/exchange/discover/crowd-source annotations. That said, my general yearning isn't to kick all markup out of my documents--but to kick presentational markup out. Semantic markup can add a lot of affordances to your documents, and if I'm already committing to working with something that requires a toolchain--I suspect I'd rather just generate plaintext or presentational markups from there?
yarnover 6y ago
This reminds me of how Nisus (the word processor program) styled text. The document, a Classic-Mac-OS file, contained plain text in the data fork and styling information in the resource fork. If I recall correctly!
- Mikhail_Edoshin 6y ago
  Yes, this was a rather good way to deal with it. Even if the content wasn't rich text, it still may have things like current position, folded sections and so on. It totally pays to keep this stuff separate from the main content as the latter stays readable by any other program that supports text. In general the whole business of file type plus creator type felt much better than the single file type of now. E.g. in desktop publishing Illustrator EPS were very different from Photoshop EPS and it was very convenient to be able to tell them apart.
porpoise 6y ago
One of the my favorite apps, Scapple (from the same company that produces Scrivener), stores their data using just this approach (plaintext + character ranges), but in XML. It does make the data file much more readable, and it makes it possible to grep for text without having to worry about regex to skip through tags, but IMO the convenience is outweighed by the fact that there is no way to do mass search and replace (that is, using external tools) without totally messing up the formatting. IMO the best approach is inline makeup but for the text editor to have the ability to hide tags. An example is PowerGrep's docx editor. You can open an .docx file and it will be editable entirely in plain text, but when you save it, all the formatting is preserved. In some way existing text editors (e.g. Sublime) are already capable of doing this via some kind of fold by regex functionality
- hansvm 6y ago
  I've toyed around off and on with FUSE drivers for various purposes, and hiding information like tags and then preserving those tags on save is something you can delegate to your filesystem. The advantage is that tag-hiding now works automagically with ripgrep, your favorite editors, and any other dev tool. You _are_ limited in that you can only detect edits during interactions with the filesystem, which is strictly less information than you'd have in an editor tracking every keystroke, but a good diffing algorithm still mostly gets the job done.
O_H_E 6y ago
> On the left you can edit the content and below it the markup. On the right is the rendered output I am not sure if I can see anything on the right/left. this is what I see on chromium/firefox https://i.imgur.com/quySrXT.png https://i.imgur.com/quySrXT.png. No rendered veiw!
- Poefke 6y ago
  Original author here, hope is an old experiment and I hadn't updated it to avoid mixed http/https content. This solved now in the online demo.
lisper 6y ago
The problem with this is that if you want to make any changes to the plain text you have to edit the markup in unison if you want the result to look coherent. So this gives up one of the principal benefits of markup languages: you can edit them with a simple text editor in a straightforward manner.
- qsort 6y ago
  To be fair the author acknowledges this. To make an analogy, this seems to me more like the equivalent of ASM than the equivalent of C, the only reasonable way to make this work is to have a high-level language that "compiles down" to Hope.
  - lisper 6y ago
    Sure, but now you've come full-circle: what is that high-level language going to look like, if not a traditional markup language? And if Hope is intended to be object code rather than source code, what's the point of making it a text format at all?
    - qsort 6y ago
      If I have to be honest I don't see much from this either, but a potential upside is structural separation of text and presentation in a way that's different from HTML/CSS, in this case the content would be pure ASCII text as opposed to a tree structure/object model. I'm not sure I agree with this premise either, the hierarchical structure seems inherently semantical to me, but the argument of the OP is it isn't the case. > what's the point of making it a text format at all? The text part is ASCII, so it make sense to represent it as such. The "presentation" part could be binary.
      - Poefke 6y ago
        The only reason I used a text format is because I wanted to keep the code as simple as possible. Only introducing extra complexity when I hit a wall somewhere. I don't know if the current format will scale to more complex use cases, but I really like the fact that you can read HTML and reason about it, so I kept to that in Hope as well. As for the text part, that is simple. I like to be able to manipulate text as text. There are many many tools to work with plain text. I can use regular expressions to search through text, get ranges of the result and apply an annotation to those ranges. You lose a lot of that in any other format.
githubalphapapa 6y ago
Buffers and strings with range-indexed metadata is also how Emacs's text properties work. For example, this shows the print syntax for strings with properties: (concat (propertize "foo" 'face '(:weight bold)) " " (propertize "bar" 'link "http://bar")) ;;=> #("foo bar" 0 3 (face (:weight bold)) 4 7 (link "http://bar"))
gioele 6y ago
The technical term for this kind of markup is "stand-off markup" or "stand-off annotation". There is wealth of research done on stand-off markup since the SGML times, with a renaissance in 2005-2010. Historically, the main drawback (as highlighted by other comments) is that stand-off needs a way to refer to the content, i.e. a pointing mechanism. Pointers in a pointing mechanism can be either a) hard to write but easy to maintain or b) simple to write but prone to breaking after a change (and thus requiring support from authoring tools). Nobody has found a viable ergonomic balance yet.
- Poefke 6y ago
  Agreed, you cannot edit one part without the other. But there are many situations where this is not a drawback. Nobody is writing Word documents in a text editor for example. But I'm looking at a way in which you may be able to add annotations to a document that isn't under your control. If the document is either guaranteed not to change, you won't have a problem. Documents hosted on IPFS or a versioned URL to a Hyper document fit this bill. If you want annotations to automatically change when a remote document changes, you need to be able to 'follow' the changes. Between when you originally 'linked' to the document and now, what has changed in the document. Which parts are deleted, which parts are added. If you know that, you can update the offsets in your annotations. So I'm looking into making a versioned document format to allow just that.
OJFord 6y ago
But why? Why is a plaintext unformatted file with just words and no markup any improvement? They're both editable, readable in a raw form, and probably not how you want them to actually be consumed. So why not just keep it in one file?
- slightwinder 6y ago
  Markup und plaintext are seperate entities by nature, and by seperating them physically, tools which don't understand the markup would work better with the pure plaintext.
catlifeonmars 6y ago
What problem does this solve?
- Poefke 6y ago
  Hi, original author here. When I started out I wasn't trying to solve a problem. I was exploring a concept I was curious about. But in the end, we (me and a colleague) ended up building it into our WYSIWYG HTML editor as an alternative to the browsers execCommand editing API. When editing rich text, an API that is build around ranges for everything is a perfect fit. The editor (https://simplyedit.io/ https://simplyedit.io/) converts a contenteditable area's content to hope, we apply some markup (annotations) and convert the result back to HTML. I wrote the Hope-HTML conversion routine once, and all the range based manipulations are now one-liners instead of treewalking horror shows.
valenterry 6y ago
This is 6 years old with no progress...
- Poefke 6y ago
  Yup, it was an experiment. There is a follow up, called cobalt: https://github.com/poef/cobalt/ https://github.com/poef/cobalt/ But its not meant as production ready code, instead I'm exploring the concept and what it means for editing, rendering, etc.
Mikhail_Edoshin 6y ago
XML is a serialization format. It's trivial to mechanically separate the markup from the content when reading and to merge them back when writing. (It should even be the preferred approach for many usage scenarios as it's rather efficient.) The description implies content and markup are physically separate; it can be done, but why? Although there is at least one upside: the same text can have multiple different markups attached (e.g. syntax and revision control or syntax and extra highlighting).
- simonh 6y ago
  Why is you want revision control of both the text and the syntax markup? This seems incredibly brittle.
  - Mikhail_Edoshin 6y ago
    No, I meant that the same text may have two or three kinds of associated markup runs: syntax, revision control, and highlighting so that we can nicely publish a snippet.
    - simonh 6y ago
      Right, so revision control is one of the types of markup of the text. But what happens when you want revision control of some of the other types of markup as well as for the text. Or when you want highlighting of different revisions of the text.
      - Mikhail_Edoshin 6y ago
        Revision control is not exactly markup. It is metadata and so is the syntax information. It's derived from text or from something else (comparison of two text by VCS), but not added there manually. Markup is normally metadata added manually, although a markup language can be used for both manual and automatic metadata. So we have a source and some associated metadata that is crafted manually. I don't see this to differ much from other cases when we need to manually sync two distinct files (e.g. C source and header files), so I don't quite understand the issue, I guess.
bloak 6y ago
Where do you draw the line between "content" and "mark-up"? * Word break nearly always gets treated as content, denoted by the space character. * Line break can be either: '\n' or <br>. * Paragraph break is usually treated as mark-up: <p>. Is there a good reason for that, or is it just historical accident?
martyalain 6y ago
Reminds me of John McCarthy's lament at the W3C's choice of SGML as the basis for HTML : « An environment where the markup, styling and scripting is all s-expression based would be nice. » Could this project be the beginning of an answer : http://lambdaway.free.fr/lambdaspeech/ http://lambdaway.free.fr/lambdaspeech/ Alain Marty
Mikhail_Edoshin 6y ago
One thing that Hope doesn't seem to support are contentless elements. E.g. some drawing may look so: 100:100:svg:svg 100-100:svg:g /* group */ 100-100:svg:rect 100-100:svg:rect These elements have no text content, so their extent is 0 and offset is same. Yet they also have hierarchical relationships and you need to store extra data to express those. E.g. in this example you can guess that 'g' and 'rect's are descendants of 'svg' and the first 'rect' is a child of the 'g' but only because you have some preconceived ideas about the meaning of these elements. There's not enough data to tell if the second 'rect' is a part of 'g' or not. You need to store at least the level in the hierarchy. If you stored the data in a database (i.e. without implied order) you'd also need the relative position of the element, but here you have the order built-in.
- Poefke 6y ago
  Original author here, this is exactly right. Hierarchy and contentless annotations don't mix well. This is the main disadvantage when compared to xml/html and the like. On the other hand, using annotations with offset pointers to the main text make things like overlapping annotations and multiple sets of annotations possible, which are difficult to pull of in xml/html.
Poefke 6y ago
Hi, original author here, I just found out my old project featured here and love the thoughtful responses. If you are interested, I've been working on a new version of and on, called cobalt (https://github.com/poef/cobalt/ https://github.com/poef/cobalt/). Hope was the proof of concept, cobalt is a more rigorous implementation. Specifically the range algebra is complete there. It's still a hobby research project. There's a lot I've learned building and using it, about the things that HTML is good at, and what it isn't good at. What I'm trying these days is to see if I can get a WYSIWYG editor with a user experience that comes close to the user experience of writing markdown. This isn't easy and its interesting to explore what makes that so.

49 points by gjvc 6y ago

gjvc 6y ago

repository at https://github.com/poef/hope/ https://github.com/poef/hope/

bovermyer 6y ago

Since my project doesn't use this, you could say... it has no Hope.

abnry 6y ago

Interesting idea. I've never thought of separating the markup and the text into two different files. To me, it suggests a more machine friendly way of keeping track of markup. Directly writing sounds very painful.

gjvc 6y ago
Yes, this is looking in the direction of a content API for documents, and is not meant to be human-generated.

abathur 6y ago

My mind's been picking at related ideas from different angles for at least a decade now, I guess. I'm not sure how I feel about this one, but one thing I like about separation approaches in general is that it's easy to imagine how N different annotations can all exist for the same underlying document, so you could use something along these lines to share/exchange/discover/crowd-source annotations. That said, my general yearning isn't to kick all markup out of my documents--but to kick presentational markup out. Semantic markup can add a lot of affordances to your documents, and if I'm already committing to working with something that requires a toolchain--I suspect I'd rather just generate plaintext or presentational markups from there?

yarnover 6y ago

This reminds me of how Nisus (the word processor program) styled text. The document, a Classic-Mac-OS file, contained plain text in the data fork and styling information in the resource fork. If I recall correctly!

Mikhail_Edoshin 6y ago
Yes, this was a rather good way to deal with it. Even if the content wasn't rich text, it still may have things like current position, folded sections and so on. It totally pays to keep this stuff separate from the main content as the latter stays readable by any other program that supports text. In general the whole business of file type plus creator type felt much better than the single file type of now. E.g. in desktop publishing Illustrator EPS were very different from Photoshop EPS and it was very convenient to be able to tell them apart.

porpoise 6y ago

One of the my favorite apps, Scapple (from the same company that produces Scrivener), stores their data using just this approach (plaintext + character ranges), but in XML. It does make the data file much more readable, and it makes it possible to grep for text without having to worry about regex to skip through tags, but IMO the convenience is outweighed by the fact that there is no way to do mass search and replace (that is, using external tools) without totally messing up the formatting. IMO the best approach is inline makeup but for the text editor to have the ability to hide tags. An example is PowerGrep's docx editor. You can open an .docx file and it will be editable entirely in plain text, but when you save it, all the formatting is preserved. In some way existing text editors (e.g. Sublime) are already capable of doing this via some kind of fold by regex functionality

hansvm 6y ago
I've toyed around off and on with FUSE drivers for various purposes, and hiding information like tags and then preserving those tags on save is something you can delegate to your filesystem. The advantage is that tag-hiding now works automagically with ripgrep, your favorite editors, and any other dev tool. You _are_ limited in that you can only detect edits during interactions with the filesystem, which is strictly less information than you'd have in an editor tracking every keystroke, but a good diffing algorithm still mostly gets the job done.

O_H_E 6y ago

> On the left you can edit the content and below it the markup. On the right is the rendered output I am not sure if I can see anything on the right/left. this is what I see on chromium/firefox https://i.imgur.com/quySrXT.png https://i.imgur.com/quySrXT.png. No rendered veiw!

Poefke 6y ago
Original author here, hope is an old experiment and I hadn't updated it to avoid mixed http/https content. This solved now in the online demo.

lisper 6y ago

The problem with this is that if you want to make any changes to the plain text you have to edit the markup in unison if you want the result to look coherent. So this gives up one of the principal benefits of markup languages: you can edit them with a simple text editor in a straightforward manner.

qsort 6y ago
To be fair the author acknowledges this. To make an analogy, this seems to me more like the equivalent of ASM than the equivalent of C, the only reasonable way to make this work is to have a high-level language that "compiles down" to Hope.
- lisper 6y ago
  Sure, but now you've come full-circle: what is that high-level language going to look like, if not a traditional markup language? And if Hope is intended to be object code rather than source code, what's the point of making it a text format at all?
  - qsort 6y ago
    If I have to be honest I don't see much from this either, but a potential upside is structural separation of text and presentation in a way that's different from HTML/CSS, in this case the content would be pure ASCII text as opposed to a tree structure/object model. I'm not sure I agree with this premise either, the hierarchical structure seems inherently semantical to me, but the argument of the OP is it isn't the case. > what's the point of making it a text format at all? The text part is ASCII, so it make sense to represent it as such. The "presentation" part could be binary.
    - Poefke 6y ago
      The only reason I used a text format is because I wanted to keep the code as simple as possible. Only introducing extra complexity when I hit a wall somewhere. I don't know if the current format will scale to more complex use cases, but I really like the fact that you can read HTML and reason about it, so I kept to that in Hope as well. As for the text part, that is simple. I like to be able to manipulate text as text. There are many many tools to work with plain text. I can use regular expressions to search through text, get ranges of the result and apply an annotation to those ranges. You lose a lot of that in any other format.

githubalphapapa 6y ago

Buffers and strings with range-indexed metadata is also how Emacs's text properties work. For example, this shows the print syntax for strings with properties: (concat (propertize "foo" 'face '(:weight bold)) " " (propertize "bar" 'link "http://bar")) ;;=> #("foo bar" 0 3 (face (:weight bold)) 4 7 (link "http://bar"))

gioele 6y ago

The technical term for this kind of markup is "stand-off markup" or "stand-off annotation". There is wealth of research done on stand-off markup since the SGML times, with a renaissance in 2005-2010. Historically, the main drawback (as highlighted by other comments) is that stand-off needs a way to refer to the content, i.e. a pointing mechanism. Pointers in a pointing mechanism can be either a) hard to write but easy to maintain or b) simple to write but prone to breaking after a change (and thus requiring support from authoring tools). Nobody has found a viable ergonomic balance yet.

Poefke 6y ago
Agreed, you cannot edit one part without the other. But there are many situations where this is not a drawback. Nobody is writing Word documents in a text editor for example. But I'm looking at a way in which you may be able to add annotations to a document that isn't under your control. If the document is either guaranteed not to change, you won't have a problem. Documents hosted on IPFS or a versioned URL to a Hyper document fit this bill. If you want annotations to automatically change when a remote document changes, you need to be able to 'follow' the changes. Between when you originally 'linked' to the document and now, what has changed in the document. Which parts are deleted, which parts are added. If you know that, you can update the offsets in your annotations. So I'm looking into making a versioned document format to allow just that.

OJFord 6y ago

But why? Why is a plaintext unformatted file with just words and no markup any improvement? They're both editable, readable in a raw form, and probably not how you want them to actually be consumed. So why not just keep it in one file?

slightwinder 6y ago
Markup und plaintext are seperate entities by nature, and by seperating them physically, tools which don't understand the markup would work better with the pure plaintext.

catlifeonmars 6y ago

What problem does this solve?

Poefke 6y ago
Hi, original author here. When I started out I wasn't trying to solve a problem. I was exploring a concept I was curious about. But in the end, we (me and a colleague) ended up building it into our WYSIWYG HTML editor as an alternative to the browsers execCommand editing API. When editing rich text, an API that is build around ranges for everything is a perfect fit. The editor (https://simplyedit.io/ https://simplyedit.io/) converts a contenteditable area's content to hope, we apply some markup (annotations) and convert the result back to HTML. I wrote the Hope-HTML conversion routine once, and all the range based manipulations are now one-liners instead of treewalking horror shows.

valenterry 6y ago

This is 6 years old with no progress...

Poefke 6y ago
Yup, it was an experiment. There is a follow up, called cobalt: https://github.com/poef/cobalt/ https://github.com/poef/cobalt/ But its not meant as production ready code, instead I'm exploring the concept and what it means for editing, rendering, etc.

Mikhail_Edoshin 6y ago

XML is a serialization format. It's trivial to mechanically separate the markup from the content when reading and to merge them back when writing. (It should even be the preferred approach for many usage scenarios as it's rather efficient.) The description implies content and markup are physically separate; it can be done, but why? Although there is at least one upside: the same text can have multiple different markups attached (e.g. syntax and revision control or syntax and extra highlighting).

simonh 6y ago
Why is you want revision control of both the text and the syntax markup? This seems incredibly brittle.
- Mikhail_Edoshin 6y ago
  No, I meant that the same text may have two or three kinds of associated markup runs: syntax, revision control, and highlighting so that we can nicely publish a snippet.
  - simonh 6y ago
    Right, so revision control is one of the types of markup of the text. But what happens when you want revision control of some of the other types of markup as well as for the text. Or when you want highlighting of different revisions of the text.
    - Mikhail_Edoshin 6y ago
      Revision control is not exactly markup. It is metadata and so is the syntax information. It's derived from text or from something else (comparison of two text by VCS), but not added there manually. Markup is normally metadata added manually, although a markup language can be used for both manual and automatic metadata. So we have a source and some associated metadata that is crafted manually. I don't see this to differ much from other cases when we need to manually sync two distinct files (e.g. C source and header files), so I don't quite understand the issue, I guess.

bloak 6y ago

Where do you draw the line between "content" and "mark-up"? * Word break nearly always gets treated as content, denoted by the space character. * Line break can be either: '\n' or <br>. * Paragraph break is usually treated as mark-up: <p>. Is there a good reason for that, or is it just historical accident?

martyalain 6y ago

Reminds me of John McCarthy's lament at the W3C's choice of SGML as the basis for HTML : « An environment where the markup, styling and scripting is all s-expression based would be nice. » Could this project be the beginning of an answer : http://lambdaway.free.fr/lambdaspeech/ http://lambdaway.free.fr/lambdaspeech/ Alain Marty

Mikhail_Edoshin 6y ago

One thing that Hope doesn't seem to support are contentless elements. E.g. some drawing may look so: 100:100:svg:svg 100-100:svg:g /* group */ 100-100:svg:rect 100-100:svg:rect These elements have no text content, so their extent is 0 and offset is same. Yet they also have hierarchical relationships and you need to store extra data to express those. E.g. in this example you can guess that 'g' and 'rect's are descendants of 'svg' and the first 'rect' is a child of the 'g' but only because you have some preconceived ideas about the meaning of these elements. There's not enough data to tell if the second 'rect' is a part of 'g' or not. You need to store at least the level in the hierarchy. If you stored the data in a database (i.e. without implied order) you'd also need the relative position of the element, but here you have the order built-in.

Poefke 6y ago
Original author here, this is exactly right. Hierarchy and contentless annotations don't mix well. This is the main disadvantage when compared to xml/html and the like. On the other hand, using annotations with offset pointers to the main text make things like overlapping annotations and multiple sets of annotations possible, which are difficult to pull of in xml/html.

Poefke 6y ago

Hi, original author here, I just found out my old project featured here and love the thoughtful responses. If you are interested, I've been working on a new version of and on, called cobalt (https://github.com/poef/cobalt/ https://github.com/poef/cobalt/). Hope was the proof of concept, cobalt is a more rigorous implementation. Specifically the range algebra is complete there. It's still a hobby research project. There's a lot I've learned building and using it, about the things that HTML is good at, and what it isn't good at. What I'm trying these days is to see if I can get a WYSIWYG editor with a user experience that comes close to the user experience of writing markdown. This isn't easy and its interesting to explore what makes that so.