16 ms·

Rewriting LaTeX in Pure Rust

248 points by xvilka 6y ago

xvilka 6y ago
The port was done with the help of c2rust and the following refactoring. See more information at https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459
ibraheemdev 6y ago
Wow, this looks like a pretty impressive rewrite!
est31 6y ago
I really love Rust (believe it's the best systems programming language out there), so I've checked out tectonic a while ago. In many ways it represents precisely the direction I think Tex based typesetting should go Sadly I can't use it though as it doesn't support biber which would be needed for Unicode author names: https://github.com/tectonic-typesetting/tectonic/issues/35 https://github.com/tectonic-typesetting/tectonic/issues/35
- xcambar 6y ago
  I'm no expert to anything typesetting systems, but I've noticed this on the homepage: > Thanks to the power of XeTeX, Tectonic can use modern OpenType fonts and is fully Unicode-enabled. ...which seems to oppose your statement and the linked issue. Can you explain, please?
  - vnorilo 6y ago
    Biber is a reference manager, so orthogonal to XeTeX or general unicode support.
  - est31 6y ago
    That refers to including Unicode in the document directly, which is great that it works, but for references, bibtex is part of the pipeline. You create a .bib file containing info about the references, and then do commands like \cite{} to cite to one of them. If you have Unicode in any of your references (author name, paper name, etc), you have a problem. The solution is to replace bibtex with biblatex and biber, but that requires biber support which currently doesn't exist :(. Example error that bibtex users can run into: https://superuser.com/questions/60432/unicode-characters-in-bibtex https://superuser.com/questions/60432/unicode-characters-in-...
    - xcambar 6y ago
      Cristal clear. Thanks!
nynx 6y ago
Getting LaTex running through wasm would be great. Tools like this, which just interact mostly with the filesystem, are a great example of what non-web wasm can do by simplifying the distribution process.
- mkl 6y ago
  You're in luck (if a year behind :-) https://news.ycombinator.com/item?id=21710105 https://news.ycombinator.com/item?id=21710105
danyim 6y ago
Why link to a fork instead of the main project? https://github.com/tectonic-typesetting/tectonic https://github.com/tectonic-typesetting/tectonic
- Thomashuet 6y ago
  The reason is that the main project is written in C whereas this fork uses Rust instead as explained in https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459
  - dwohnitmok 6y ago
    How much of that is auto-generated and pre-existing C code though? My dusty recollection from a while ago is that most of Tectonic's development is in Rust. It looks like the fork is trying to get rid of all the remaining C code?
    - hojjat12000 6y ago
      Here is the percentages of different languages in the original tectonic repo: C: 80.5%, TeX: 7.8%, Rust 7.5%, C++ 4.0%, Other 0.2% And here is the forked version: Rust 85.2%, TeX 7.3%, C 6.2%, C++ 1.2%, Shell 0.1% So, looks like %80 more rust in this one (auto-generated and refactored)
  - deleted 6y ago
    [deleted]
- scoopertrooper 6y ago
  That fork is quite curious, it seems like they've really gone out and created quite an active little fork, but haven't spent the time to explain what is different and interesting about their fork in the readme.
  - infogulch 6y ago
    The about summarizes it: > Experimental Oxidization of Tectonic the TeX/LaTeX engine.
Evidlo 6y ago
Currently using this to build my Master's thesis. It's nice because it detects all the packages used in the tex source and downloads them automatically.
- mkl 6y ago
  What do you need that isn't there already with TeXLive? I haven't downloaded a package in many years, despite many reinstalls and a lot of varied Latex use.
  - diarrhea 6y ago
    You are not having any issues because TeXLive is everything. The `full` scheme on `install-sh` or `texlive-full` on Debian takes up around 7 GB last time I looked, 2 to 3 of which are PDFs for package documentation. That is, by a long shot, the largest install a lot of people have on their system. Nowadays, no biggie, but systems like Miktex allow to install-on-demand and of course, it allows you to shrink your installation significantly. Full TexLive contains a number of packages average "Western hemisphere scientific authors", arguably the largest LaTeX demographic, will never need: Eastern and other languages, music, humanities.
    - SiempreViernes 6y ago
      You don't play very many new games do you?
      - diarrhea 6y ago
        Being on HN and talking about TeXLive, I was thinking in terms of Linux, for which those 90GB blockbuster games don't exist (?). So I did a mental jump there, because to me TeXLive is the "native" Linux TeX distribution, and Miktex the "native" Windows distribution.
        mkl 6y ago
        TeXLive has to me been the obvious choice on Windows for years. Miktex always seemed to be fussy and missing things the few times I tried to use it.
wd40-can 6y ago
Really cool to see all those rewrites of existing stuff in Rust. Does anyone know if the Rust community has ever done anything original, though?
- coldtea 6y ago
  >Does anyone know if the Rust community has ever done anything original, though? Yes, they made an original, better, programming language.
- anand-bala 6y ago
  > Does anyone know if the Rust community has ever done anything original, though? Depends on what you call "original". Is the Servo layout engine _not_ original because layout engines already exist? :) Moving away from the rhetorical question, I think there are several reasons to rewrite something in Rust, the first and foremost being the huge improvement in memory safety/management. The other reasons are typically related to speed: C wasn't built with concurrency in mind, and trying to develop multi-threaded C applications that run in a cross-platform manner is not straightforward. On the other hand, Rust has the following: 1) Memory safety during concurrency [1]. 2) Multi-threading in the standard library [1]. 3) An amazing package ecosystem revolving around "crates". And there are several build specifically to make concurrency safe and easy to implement. [2] [1] https://doc.rust-lang.org/stable/book/ch16-00-concurrency.html https://doc.rust-lang.org/stable/book/ch16-00-concurrency.ht... [2] https://crates.io/categories/concurrency https://crates.io/categories/concurrency
  - jbergens 6y ago
    Rust seems to be a lot safer than C or C++ but I don't really understand why a documentation system must be written in a low level language. Why not use C#, Java, Go or something similar? Most users can't really need extreme performance.
    - jabl 6y ago
      I'm quite sure anybody who has written substantial documents with *TeX disagrees that "extreme performance" isn't needed.
hvasilev 6y ago
I think that Rust is a perfect open source language. For the longest time the open source community has shown that it has an issue with innovation, however is great at recreating existing ideas and products. I think that this strength can be combined somehow with the "rewrite in Rust" strength and get great synergy.
- saagarjha 6y ago
  > For the longest time the open source community has shown that it has an issue with innovation How so? A bunch of my favorite software has no real closed-source equivalent, sometime precisely because it’s open source.
  - hvasilev 6y ago
    The vast majority of open source software is a clone of a closed-source alternative. I currently having issues with thinking of a widely adopted, consumer-oriented software that has no closed-source alternative. What is your favorite software that has no real closed-source equivalent?
    - saagarjha 6y ago
      QEMU, maybe?
      - hvasilev 6y ago
        how is this widely adopted, think in the scale of Linux or Gimp.
        saagarjha 6y ago
        In addition to being useful in its own right, QEMU powers or has powered a number of virtualization projects, including VirtualBox's software-based virtualization, parts of Xen, Android Studio's emulator, …
whereistimbo 6y ago
But what is the license? Checking up the repo, I see the license is MIT, but I'm not quite satisfied with it. I mean does LaTeX permits rewrites like this?
- racingmars 6y ago
  The project appears to be based on XeTeX, which is covered by the MIT license. I don't know the full history of XeTeX, but if it was indeed a re-write of TeX/LaTeX from the ground up (and not based on the TeX source code), then the original TeX/LaTeX license wouldn't necessarily apply. But if it borrowed actual source code, templates, etc., from other projects under different licenses, then it probably gets more complicated.
mkl 6y ago
Previous discussion: 1 year ago: https://news.ycombinator.com/item?id=21172964 https://news.ycombinator.com/item?id=21172964 3.5 years ago: https://news.ycombinator.com/item?id=14450448 https://news.ycombinator.com/item?id=14450448 The project seems pretty active, though, so some of that may now be irrelevant.
- priyanshuraj 6y ago
  Why it may be irrelevant that the project is active? Not sure I understood the point behind your last sentence. Can you explain please?
  - sundarurfriend 6y ago
    Please do not downvote questions like this (the comment is faded out as I'm writing this). It's a genuine question from a new user, possibly a non-native English speaker, trying to understand someone's comment.
    - priyanshuraj 6y ago
      Thank you for the support. Yes it was a genuine question. I am not native English speaker. I was trying to understand the comment.
  - DerSaidin 6y ago
    I think the intent is: the previous discussions were some time ago and may be out of date, so those links may not be relevant.
  - Leherenn 6y ago
    Probably more outdated than irrelevant.
  - mkl 6y ago
    "That" refers to the previous discussion just mentioned, and "now" confirms it by contrasting with 1 or 3.5 years ago. The point is that it is possible some of the criticisms and problems discussed back then could have been fixed since then (though I don't know, hence "may").
shp0ngle 6y ago
I wonder Does this still depend, even indirectly, at the WEB files written by Knuth? It has the strange license that says "Copying of this file is authorized only if (1) you are D. E. Knuth, or if % (2) you make absolutely no changes to your copy." http://mirror.las.iastate.edu/tex-archive/systems/knuth/dist/tex/tex.web http://mirror.las.iastate.edu/tex-archive/systems/knuth/dist...
- deleted 6y ago
  [deleted]
- smichel17 6y ago
  …Those version numbers… I love it! % Version 0 was released in September 1982 after it passed a variety of tests. % Version 1 was released in November 1983 after thorough testing. % Version 1.1 fixed ``disappearing font identifiers'' et alia (July 1984). % Version 1.2 allowed `0' in response to an error, et alia (October 1984). % Version 1.3 made memory allocation more flexible and local (November 1984). % Version 1.4 fixed accents right after line breaks, et alia (April 1985). % Version 1.5 fixed \the\toks after other expansion in \edefs (August 1985). % Version 2.0 (almost identical to 1.5) corresponds to "Volume B" (April 1986). % Version 2.1 corrected anomalies in discretionary breaks (January 1987). % Version 2.2 corrected "(Please type...)" with null \endlinechar (April 1987). % Version 2.3 avoided incomplete page in premature termination (August 1987). % Version 2.4 fixed \noaligned rules in indented displays (August 1987). % Version 2.5 saved cur_order when expanding tokens (September 1987). % Version 2.6 added 10sp slop when shipping leaders (November 1987). % Version 2.7 improved rounding of negative-width characters (November 1987). % Version 2.8 fixed weird bug if no \patterns are used (December 1987). % Version 2.9 made \csname\endcsname's "relax" local (December 1987). % Version 2.91 fixed \outer\def\a0{}\a\a bug (April 1988). % Version 2.92 fixed \patterns, also file names with complex macros (May 1988). % Version 2.93 fixed negative halving in allocator when mem_min<0 (June 1988). % Version 2.94 kept open_log_file from calling fatal_error (November 1988). % Version 2.95 solved that problem a better way (December 1988). % Version 2.96 corrected bug in "Infinite shrinkage" recovery (January 1989). % Version 2.97 corrected blunder in creating 2.95 (February 1989). % Version 2.98 omitted save_for_after at outer level (March 1989). % Version 2.99 caught $$\begingroup\halign..$$ (June 1989). % Version 2.991 caught .5\ifdim.6... (June 1989). % Version 2.992 introduced major changes for 8-bit extensions (September 1989). % Version 2.993 fixed a save_stack synchronization bug et alia (December 1989). % Version 3.0 fixed unusual displays; was more \output robust (March 1990). % Version 3.1 fixed nullfont, disabled \write{\the\prevgraf} (September 1990). % Version 3.14 fixed unprintable font names and corrected typos (March 1991). % Version 3.141 more of same; reconstituted ligatures better (March 1992). % Version 3.1415 preserved nonexplicit kerns, tidied up (February 1993). % Version 3.14159 allowed fontmemsize to change; bulletproofing (March 1995). % Version 3.141592 fixed \xleaders, glueset, weird alignments (December 2002). % Version 3.1415926 was a general cleanup with minor fixes (February 2008). % Version 3.14159265 was similar (January 2014).
alexpovel 6y ago
This looks amazing and like everything I always wanted. Sadly, I think basing off XeTeX and not LuaTeX is a mistake. Certainly renders it unusable for me. Having Lua integration is just great. Also, `lualatex` does not have some of the limitations of `xelatex` (memory limitations, `contours` package, ...), but I guess this XeTeX reimplementation can work on removing those implementations, so that only lack of Lua integration remains. Also, like another person said, not having biber breaks my workflow as well, which specifically tries to leverage the "latest and greatest" of what LaTeX has to offer [0]: `pdflatex` is obsolete, so `lualatex` it is. `nomencl`, `makeindex` etc. are obsolete, so `glossaries-extra` it is. `bibtex` is obsolete, so `biber` it is. Throw in `latexmk` for automatic compilation (which the tool presented here does too, which is a biggie! [1]) and CI/CD and you have a 1970s tool in 2020s attire. Lua rounds off the picture. Among other things, this given Unicode-native (gasp) code/documents, and great automation capabilities (`latexmk`, CI/CD, Lua). I think a modern TeX engine reimplementation should support all of the above, which are arguably the best modern options there are. [0]: https://collaborating.tuhh.de/alex/latex-git-cookbook https://collaborating.tuhh.de/alex/latex-git-cookbook [1]: I wonder if the logs are available though? aux, blg etc. are important for debugging and shouldn't be dropped outright.
- chriszhang 6y ago
  > Also, `lualatex` does not have some of the limitations of `xelatex` (memory limitations, `contours` package, ...) Are there some more details of the memory limitations you can share with us?
  - alexpovel 6y ago
    LuaLaTeX allocates memory as-needed, see section 3.4.1 in the manual [0] (and comments/answers in this thread [1]). Base TeX has an arbitrary, by modern standards low memory limit, leading to a whole class of errors plaguing unsuspecting users [2], and spawning entire extensions to deal with these limitations [3]. This is simply an artefact of times past and has no technical relevance nowadays. LuaTeX allows dynamic allocation, with the available system RAM as the upper limit (so effectively, no limitations in everyday usage). Now, I could not find a mention of memory handling in the XeTeX reference manual [4]. People are using tricks like `tikzexternalize` with xelatex [5, 6]. Especially the first point makes me think XeLaTeX inherits base TeX memory handling/limits, but I cannot confirm this. I just know that all my problems disappeared when switching from XeLaTeX to LuaLaTeX. Lastly, see here [7] for a comprehensive (albeit somewhat anecdotal) list of advantages of LuaTeX over XeTeX. Of that list, `microtype` is another significant functionality I rely on. [0]: http://www.tug.org/texlive//devsrc/Master/texmf-dist/doc/context/documents/general/manuals/luatex.pdf http://www.tug.org/texlive//devsrc/Master/texmf-dist/doc/con... [1]: https://tex.stackexchange.com/q/7953/ https://tex.stackexchange.com/q/7953/ [2]: https://tex.stackexchange.com/search?q=tex+capacity+exceeded https://tex.stackexchange.com/search?q=tex+capacity+exceeded [3]: https://tex.stackexchange.com/a/482560/ https://tex.stackexchange.com/a/482560/ [4]: http://mirrors.ctan.org/info/xetexref/xetex-reference.pdf http://mirrors.ctan.org/info/xetexref/xetex-reference.pdf [5]: https://tex.stackexchange.com/q/438131/ https://tex.stackexchange.com/q/438131/ [6]: https://tex.stackexchange.com/q/334250/ https://tex.stackexchange.com/q/334250/ [7]: https://tex.stackexchange.com/q/126206/ https://tex.stackexchange.com/q/126206/
ageofwant 6y ago
This is great. After all these years the venerable and super useful TeX/LaTeX meme has once again managed to propagate into a new and exiting ecosystem. This would be good for the insufferable Rust community as well, I hope. Perhaps as the language they invested their identity in, actually becomes useful and venerated, they will calm the fuck down and start becoming adult members of the useful tool community. Nice one rustcucks, next do Python.
- deleted 6y ago
  [deleted]
jfk13 6y ago
Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and grew from there. While it generally works really well, it's not a solid piece of engineering on which to build the future. The resulting code doesn't benefit from being Rust, except for the added buzzword-ness. It's simply a translation of the C code, littered with unsafe blocks, as it has not been architected to work with Rust's ownership semantics, borrow checker, etc. What XeTeX (or LuaTeX, though I don't find the Lua integration important -- but that depends heavily on your use cases) needs is a rewrite that preserves backward compatibility for documents, while re-architecting the engine using a modern language such as Rust. Simply wrapping the 1980s-era code in a "skin" of Rust syntax brings little value.
- grey-area 6y ago
  The advantage of this approach (as opposed to a rewrite), is that you reproduce all the bugs and misfeatures that people in the wild depend on. You can then add a test suite and start refactoring and gradually move to a codebase you're happy with, while breaking few or no users on the way. Another approach is to split it into blocks and replace parts of the system, but whether that is feasible depends on the software and how modular it is. Finally, you could try a clean-room rewrite, but that could take years without visible results and is hard to find the motivation for if existing software works ok.
  - cryptos 6y ago
    But the advantage of avoiding C specific bugs is shallow, because the Rust code contains lots of unsafe blocks. Bugfixes made to the original source are not automatically ported to the Rust code base. From my point of view the chosen porting "strategy" doesn't make much sense. It is more of a toy project to see what's possible. What would really make sense would be starting with an extensive test suite and trying to build a properly architected Rust implementation according to it.
sega_sai 6y ago
I'm not sure rewriting Latex in XYZ is a good idea. The whole thing is a mess and a pain to use (and I use it multiple times a week). Sure the documents often look good, but when they don't or there is an error, fixing it just becomes a nightmare. I wish somebody would design a modern typesetting system that would accept say latex equations but without 50 years of cruft and levels of macroses upon macroses.
- Veen 6y ago
  Matthew Butterick is working on the Quad document processor, which is intended to "modernize the good ideas in LaTeX," but I'm unsure what its equation typesetting is like. It's written in Racket. https://docs.racket-lang.org/quad/ https://docs.racket-lang.org/quad/
  - earthscienceman 6y ago
    The final note in the documentation on this software is a quote very fitting of a project re-thinking digital typesetting: “A way of doing something original is by trying something so painstaking that nobody else has ever bothered with it.” — Brian Eno
- peteretep 6y ago
  Sile appears to still be under active development: https://github.com/sile-typesetter/sile https://github.com/sile-typesetter/sile
- marmaduke 6y ago
  I don’t like Word but it does accept Latex math and is easy to use.
  - gspr 6y ago
    Word does not use plain text input, and its output looks objectively terrible compared to LaTeX. These two things make it completely inadequate as a competitor.
smarx007 6y ago
Will the project offer a matching program to https://en.wikipedia.org/wiki/Knuth_reward_check https://en.wikipedia.org/wiki/Knuth_reward_check? :)
marktangotango 6y ago
This “x implemented in y” is a curious thing we are seeing with growing frequency nowadays (imo). On the one hand, good for these folks, to be rewriting something old in a new way (be it language, library, or framework). On the other hand, could this be an indicator that we’ve reached a certain level where it is easier to rehash the ideas of the past than create new work or invent new ideas or applications? Maybe someone with a more classical education can comment on how this manifests in the arts?
- geodel 6y ago
  Yeah, roughly something like doing cover versions of older songs but with modern, state of the art music arrangement. From what I know people do seem to love that.
- colejohnson66 6y ago
  What’s next is obviously JS or even WASM to typeset on the client side! /s
  - hojjat12000 6y ago
    That should be fairly easy in Rust.
- secondcoming 6y ago
  The best way to learn a language is to port an existing tool to that language. Whether the rewrite is better depends on how you measure 'better', but what's certain is that someone gets to put 'I know Rust' on their CV.
hatmatrix 6y ago
Could this unify LaTeX and MathJax? I feel like any rewrite of LaTeX that could address portability would be a huge plus.
- yza 6y ago
  Would you care to elaborate on the point of your suggested integration? Are you (seriously) proposing the inclusion of a javascript backend into *TeX distribution?
paultopia 6y ago
Does this mean that at some point maybe it'll be possible to just drop it into webassembly?
todd8 6y ago
The title is a bit misleading. LaTeX is a macro package written using the TeX macro system, so I doubt that LaTeX itself is being rewritten in Rust. TeX, created by Donald Knuth, is now over 40 years old. It has a number of remarkable characteristics: * TeX is essentially bug free. It can be a challenging program to use, but in my experience I was always consoled by the knowledge that if something wasn't working it was my fault (in contrast to my use of other document processing systems and word processors). See [1]. * The TeX program is extraordinarily well documented. Knuth has an entire book which is the literate program comprises TeX. See [2]. * TeX was originally written in a version of Pascal, the source code was written using a literate style and processed using tools the Knuth wrote (Tangle and Weave). This was later machine translated into C. * TeX hasn't changed. It reached version 3 in 1989 and Knuth essentially froze it at this point. One of its great values is that documents written in TeX will always produce the same output, whether written last century, this century, or next century. We will always be able to see what the original author wanted. * TeX is unusually capable. Knuth is obviously a perfectionist. Unhappy with the state of digital typography when he wrote TeX he designed an entire toolchain for creating fonts call MetaFont. I used it to design my company's first unique logo back in the 90's. (I only needed five characters.) See [3]. Knuth did this before Postscipt was invented. * TeX is public domain. Before open source was a thing, Knuth put TeX in the public domain. * TeX is the Lingua Franca used by Mathematicians, Physicists, Computer Scientists and others for books, academic papers, and even presentations; nothing else comes close. * TeX has a number of widely used forks, notably XeTeX and LuaTeX. XeTeX added Unicode support and LuaTeX is an attempt to make TeX and it's companion LaTeX more easily programmable. See [4] and [5]. * TeX has a powerful macro system. This is how packages are added to TeX. The most widely used such package is LaTeX itself. LaTeX is not written in C or Pascal, it is written as a complex set of macros. * Tex's macro system is great for its intended purpose, but not so great for how it is currently used. Knuth originally used the macros to define layouts of books and papers. Things like line spacing font choice for page numbers and such can be easily encapsulated as TeX macros. TeX's macro system is powerful enough that even complex packages like LaTeX could be written in it. Today, there are packages like TikZ/Pgf that are very sophisticated and complex programs written entirely in TeX macros. TikZ/Pgf is a powerful graphics, drawing, and even animation system. It's manual is 1271 pages long. See [6] Where does this leave the originally linked project called Tectonic? It's not obvious from the github repository, but the user web site is a bit more informative, see [7]. This "rewrite" seems to rely on XeTeX and provides some command line operations that streamline the invocation of XeTeX. I don't understand its value proposition. For me, what's really needed is a better language than TeX's macro system for writing extensions. A few years ago I was writing TeX code (macros) to format spine labels for the books in my personal library. I used the number of pages in the book to adjust the size and layout of the call number and 2D barcode printed on the label. All of this was quite challenging in TeX's macro system, so I wrote a simple Python program that generated the LaTeX output that resulted in a page of labels that were easy to print. Maybe I could have done it in LuaTeX using Lua, but at the time LuaTeX was so poorly documented (how ironic) that I couldn't figure out how to do it. The adoption of some programming language that could guarantee backward compatibility indefinitely and a software package written in it that was compatible with LaTeX2e or LaTeX3 and replaced them. This language should not be TeX's macro system. The right language, perhaps something like ML or Scheme, would make programmed typesetting so much easier. [1] https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.4380190702 https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.43801907..., behind a paywall but copies are easily available. [2] https://www-cs-faculty.stanford.edu/~knuth/abcde.html https://www-cs-faculty.stanford.edu/~knuth/abcde.html [3] https://en.wikipedia.org/wiki/Metafont https://en.wikipedia.org/wiki/Metafont [4] https://en.wikipedia.org/wiki/XeTeX https://en.wikipedia.org/wiki/XeTeX [5] https://en.wikipedia.org/wiki/LuaTeX https://en.wikipedia.org/wiki/LuaTeX [6] http://mirrors.ibiblio.org/CTAN/graphics/pgf/base/doc/pgfmanual.pdf http://mirrors.ibiblio.org/CTAN/graphics/pgf/base/doc/pgfman... [7] https://tectonic-typesetting.github.io/en-US/ https://tectonic-typesetting.github.io/en-US/ See Also: * Wikipedia entry for TeX: https://en.wikipedia.org/wiki/TeX https://en.wikipedia.org/wiki/TeX * Wikipedia entry for MetaFont: https://en.wikipedia.org/wiki/Metafont https://en.wikipedia.org/wiki/Metafont * Wikipedia entry for LaTeX: https://en.wikipedia.org/wiki/LaTeX https://en.wikipedia.org/wiki/LaTeX
mproud 6y ago
Is the rendering speed better, or probably not?
qkhhly 6y ago
\insteadof re-write \latex, it \might be \more \interesting to \change this \backslash \filled \syntax.
svat 6y ago
While the name “Rewriting LaTeX in Pure Rust” is very exciting, there are two problems: • “LaTeX” is a set of macros (ab)using the macro/text-expansion feature of an underlying TeX engine (LuaTeX, XeTeX, pdTeX, original Knuth TeX, or this one), to provide things like cross-references, section numbering, etc. When you as a typical user use “LaTeX”, you're actually using these macros, along with a huge variety of macro “packages” written by many authors, that together add up to orders of magnitude more lines of code than the TeX engine itself (about 25000 lines originally: https://tex.stackexchange.com/a/505664 https://tex.stackexchange.com/a/505664). Most of the incompatibilities and error messages a typical user encounters when using LaTeX are caused not by the TeX engine but by these packages or LaTeX, which this project doesn't touch. • This project comes about as: •— tex.web, Knuth's original source code, written in his literate-programming system WEB (a preprocessor on top of Pascal), •— extended to xetex.web (XeTeX), to provide Unicode support etc, •— automatically tangled to Pascal code (macros expanded and constants replaced), •— automatically translated to C code, •— now automatically translated to Rust code, by wrapping the C code in `unsafe` Rust blocks. So it has gone through multiple rounds of machine translation. Actually, I'm impressed that this project has made many improvements in undoing and restoring some of the WEB macros, since the last time this was mentioned here. (Compare a random comparison from 1 year ago: https://gist.github.com/627399d0150e66d211a264bc05b33beb https://gist.github.com/627399d0150e66d211a264bc05b33beb with the same comparison today: https://gist.github.com/71e47f2276f9c6c030efe0e3357ef3bf https://gist.github.com/71e47f2276f9c6c030efe0e3357ef3bf) — two of the four differences I had pointed out in my comment then (https://news.ycombinator.com/item?id=21177367 https://news.ycombinator.com/item?id=21177367) are gone, and the only ones to mention are: • The comments from the original are gone. • Something like “cur_if:=subtype(p);” becomes “ cur_if = MEM[p].b16.s0 as i16;” So there's still a long way to go. But even with all that, what would be more useful IMO are (in order of difficulty): • Translate directly from WEB (xetex.web) to Rust. The original uses only a subset of Pascal so IMO it should be possible to write an automatic translator that preserves more of the original. Something along these lines has been done for translating WEB to CWEB by Martin Ruckert (see https://w3-o.cs.hm.edu/users/ruckert/public_html/web2w/index.html https://w3-o.cs.hm.edu/users/ruckert/public_html/web2w/index... and http://tug.org/TUGboat/Contents/listauthor.html#Ruckert,Martin http://tug.org/TUGboat/Contents/listauthor.html#Ruckert,Mart... and his demo from TUG 2020). Alternatively, one could give up on XeTeX and start with the LuaTeX source code (in C). • Understand how the TeX program is written/works, and write a more idiomatic translation in Rust: something along these lines is in progress as a one-person hobby project by Emily Eisenberg (one of the original developers of KaTeX): see https://github.com/xymostech/XymosTeX https://github.com/xymostech/XymosTeX This one is most exciting to me, as I also have (not very useful so far) grand plans to understand TeX and to make it more understandable for everybody (https://shreevatsa.net/tex/program https://shreevatsa.net/tex/program). • As most of the code a user deals with is in the macro/LaTeX layer, modify TeX macro expansion to make it more “debuggable” (https://tex.stackexchange.com/a/384881 https://tex.stackexchange.com/a/384881, https://cstheory.stackexchange.com/a/40282 https://cstheory.stackexchange.com/a/40282): show full stack traces beyond what one gets with \errorcontextlines=\maxdimen (so file names and line numbers, by tracking the provenance of token lists), show arguments scanned (“passed in”) and expected/found types thereof, cleanly separate the macro expansion layer from the typesetting (line-breaking / page-breaking) layer, maybe even (JIT?) “compile” some of the lower-level or frequently-used macros — the eventual goal would be for the user to completely understand what is going on when they compile their LaTeX document, so that (1) they are less surprised by errors and know what to do, (2) they may be induced (if they're programmers) to do less in the (unsuitable) macro layer of the TeX engine and do more at the appropriate layer: either do it in a preprocessor that runs on the .tex file, or do it at the typesetting layer via something like LuaTeX's hooks (pre_linebreak_filter etc). Anyway, this project could still end up getting there, so let's see! Interacting with the TeX community (the TeX StackExchange, mailing lists, write an article for TUGboat, etc) might also be useful.

248 points by xvilka 6y ago

The port was done with the help of c2rust and the following refactoring. See more information at https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459

ibraheemdev 6y ago

Wow, this looks like a pretty impressive rewrite!

est31 6y ago

I really love Rust (believe it's the best systems programming language out there), so I've checked out tectonic a while ago. In many ways it represents precisely the direction I think Tex based typesetting should go Sadly I can't use it though as it doesn't support biber which would be needed for Unicode author names: https://github.com/tectonic-typesetting/tectonic/issues/35 https://github.com/tectonic-typesetting/tectonic/issues/35

xcambar 6y ago
I'm no expert to anything typesetting systems, but I've noticed this on the homepage: > Thanks to the power of XeTeX, Tectonic can use modern OpenType fonts and is fully Unicode-enabled. ...which seems to oppose your statement and the linked issue. Can you explain, please?
- vnorilo 6y ago
  Biber is a reference manager, so orthogonal to XeTeX or general unicode support.
- est31 6y ago
  That refers to including Unicode in the document directly, which is great that it works, but for references, bibtex is part of the pipeline. You create a .bib file containing info about the references, and then do commands like \cite{} to cite to one of them. If you have Unicode in any of your references (author name, paper name, etc), you have a problem. The solution is to replace bibtex with biblatex and biber, but that requires biber support which currently doesn't exist :(. Example error that bibtex users can run into: https://superuser.com/questions/60432/unicode-characters-in-bibtex https://superuser.com/questions/60432/unicode-characters-in-...
  - xcambar 6y ago
    Cristal clear. Thanks!

nynx 6y ago

Getting LaTex running through wasm would be great. Tools like this, which just interact mostly with the filesystem, are a great example of what non-web wasm can do by simplifying the distribution process.

mkl 6y ago
You're in luck (if a year behind :-) https://news.ycombinator.com/item?id=21710105 https://news.ycombinator.com/item?id=21710105

danyim 6y ago

Why link to a fork instead of the main project? https://github.com/tectonic-typesetting/tectonic https://github.com/tectonic-typesetting/tectonic

Thomashuet 6y ago
The reason is that the main project is written in C whereas this fork uses Rust instead as explained in https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459
- dwohnitmok 6y ago
  How much of that is auto-generated and pre-existing C code though? My dusty recollection from a while ago is that most of Tectonic's development is in Rust. It looks like the fork is trying to get rid of all the remaining C code?
  - hojjat12000 6y ago
    Here is the percentages of different languages in the original tectonic repo: C: 80.5%, TeX: 7.8%, Rust 7.5%, C++ 4.0%, Other 0.2% And here is the forked version: Rust 85.2%, TeX 7.3%, C 6.2%, C++ 1.2%, Shell 0.1% So, looks like %80 more rust in this one (auto-generated and refactored)
- deleted 6y ago
  [deleted]
scoopertrooper 6y ago
That fork is quite curious, it seems like they've really gone out and created quite an active little fork, but haven't spent the time to explain what is different and interesting about their fork in the readme.
- infogulch 6y ago
  The about summarizes it: > Experimental Oxidization of Tectonic the TeX/LaTeX engine.

Evidlo 6y ago

Currently using this to build my Master's thesis. It's nice because it detects all the packages used in the tex source and downloads them automatically.

mkl 6y ago
What do you need that isn't there already with TeXLive? I haven't downloaded a package in many years, despite many reinstalls and a lot of varied Latex use.
- diarrhea 6y ago
  You are not having any issues because TeXLive is everything. The `full` scheme on `install-sh` or `texlive-full` on Debian takes up around 7 GB last time I looked, 2 to 3 of which are PDFs for package documentation. That is, by a long shot, the largest install a lot of people have on their system. Nowadays, no biggie, but systems like Miktex allow to install-on-demand and of course, it allows you to shrink your installation significantly. Full TexLive contains a number of packages average "Western hemisphere scientific authors", arguably the largest LaTeX demographic, will never need: Eastern and other languages, music, humanities.
  - SiempreViernes 6y ago
    You don't play very many new games do you?
    - diarrhea 6y ago
      Being on HN and talking about TeXLive, I was thinking in terms of Linux, for which those 90GB blockbuster games don't exist (?). So I did a mental jump there, because to me TeXLive is the "native" Linux TeX distribution, and Miktex the "native" Windows distribution.
      - mkl 6y ago
        TeXLive has to me been the obvious choice on Windows for years. Miktex always seemed to be fussy and missing things the few times I tried to use it.

wd40-can 6y ago

Really cool to see all those rewrites of existing stuff in Rust. Does anyone know if the Rust community has ever done anything original, though?

coldtea 6y ago
>Does anyone know if the Rust community has ever done anything original, though? Yes, they made an original, better, programming language.
anand-bala 6y ago
> Does anyone know if the Rust community has ever done anything original, though? Depends on what you call "original". Is the Servo layout engine _not_ original because layout engines already exist? :) Moving away from the rhetorical question, I think there are several reasons to rewrite something in Rust, the first and foremost being the huge improvement in memory safety/management. The other reasons are typically related to speed: C wasn't built with concurrency in mind, and trying to develop multi-threaded C applications that run in a cross-platform manner is not straightforward. On the other hand, Rust has the following: 1) Memory safety during concurrency [1]. 2) Multi-threading in the standard library [1]. 3) An amazing package ecosystem revolving around "crates". And there are several build specifically to make concurrency safe and easy to implement. [2] [1] https://doc.rust-lang.org/stable/book/ch16-00-concurrency.html https://doc.rust-lang.org/stable/book/ch16-00-concurrency.ht... [2] https://crates.io/categories/concurrency https://crates.io/categories/concurrency
- jbergens 6y ago
  Rust seems to be a lot safer than C or C++ but I don't really understand why a documentation system must be written in a low level language. Why not use C#, Java, Go or something similar? Most users can't really need extreme performance.
  - jabl 6y ago
    I'm quite sure anybody who has written substantial documents with *TeX disagrees that "extreme performance" isn't needed.

hvasilev 6y ago

I think that Rust is a perfect open source language. For the longest time the open source community has shown that it has an issue with innovation, however is great at recreating existing ideas and products. I think that this strength can be combined somehow with the "rewrite in Rust" strength and get great synergy.

saagarjha 6y ago
> For the longest time the open source community has shown that it has an issue with innovation How so? A bunch of my favorite software has no real closed-source equivalent, sometime precisely because it’s open source.
- hvasilev 6y ago
  The vast majority of open source software is a clone of a closed-source alternative. I currently having issues with thinking of a widely adopted, consumer-oriented software that has no closed-source alternative. What is your favorite software that has no real closed-source equivalent?
  - saagarjha 6y ago
    QEMU, maybe?
    - hvasilev 6y ago
      how is this widely adopted, think in the scale of Linux or Gimp.
      - saagarjha 6y ago
        In addition to being useful in its own right, QEMU powers or has powered a number of virtualization projects, including VirtualBox's software-based virtualization, parts of Xen, Android Studio's emulator, …

whereistimbo 6y ago

But what is the license? Checking up the repo, I see the license is MIT, but I'm not quite satisfied with it. I mean does LaTeX permits rewrites like this?

racingmars 6y ago
The project appears to be based on XeTeX, which is covered by the MIT license. I don't know the full history of XeTeX, but if it was indeed a re-write of TeX/LaTeX from the ground up (and not based on the TeX source code), then the original TeX/LaTeX license wouldn't necessarily apply. But if it borrowed actual source code, templates, etc., from other projects under different licenses, then it probably gets more complicated.

mkl 6y ago

Previous discussion: 1 year ago: https://news.ycombinator.com/item?id=21172964 https://news.ycombinator.com/item?id=21172964 3.5 years ago: https://news.ycombinator.com/item?id=14450448 https://news.ycombinator.com/item?id=14450448 The project seems pretty active, though, so some of that may now be irrelevant.

priyanshuraj 6y ago
Why it may be irrelevant that the project is active? Not sure I understood the point behind your last sentence. Can you explain please?
- sundarurfriend 6y ago
  Please do not downvote questions like this (the comment is faded out as I'm writing this). It's a genuine question from a new user, possibly a non-native English speaker, trying to understand someone's comment.
  - priyanshuraj 6y ago
    Thank you for the support. Yes it was a genuine question. I am not native English speaker. I was trying to understand the comment.
- DerSaidin 6y ago
  I think the intent is: the previous discussions were some time ago and may be out of date, so those links may not be relevant.
- Leherenn 6y ago
  Probably more outdated than irrelevant.
- mkl 6y ago
  "That" refers to the previous discussion just mentioned, and "now" confirms it by contrasting with 1 or 3.5 years ago. The point is that it is possible some of the criticisms and problems discussed back then could have been fixed since then (though I don't know, hence "may").

shp0ngle 6y ago

I wonder Does this still depend, even indirectly, at the WEB files written by Knuth? It has the strange license that says "Copying of this file is authorized only if (1) you are D. E. Knuth, or if % (2) you make absolutely no changes to your copy." http://mirror.las.iastate.edu/tex-archive/systems/knuth/dist/tex/tex.web http://mirror.las.iastate.edu/tex-archive/systems/knuth/dist...

deleted 6y ago
[deleted]
smichel17 6y ago
…Those version numbers… I love it! % Version 0 was released in September 1982 after it passed a variety of tests. % Version 1 was released in November 1983 after thorough testing. % Version 1.1 fixed ``disappearing font identifiers'' et alia (July 1984). % Version 1.2 allowed `0' in response to an error, et alia (October 1984). % Version 1.3 made memory allocation more flexible and local (November 1984). % Version 1.4 fixed accents right after line breaks, et alia (April 1985). % Version 1.5 fixed \the\toks after other expansion in \edefs (August 1985). % Version 2.0 (almost identical to 1.5) corresponds to "Volume B" (April 1986). % Version 2.1 corrected anomalies in discretionary breaks (January 1987). % Version 2.2 corrected "(Please type...)" with null \endlinechar (April 1987). % Version 2.3 avoided incomplete page in premature termination (August 1987). % Version 2.4 fixed \noaligned rules in indented displays (August 1987). % Version 2.5 saved cur_order when expanding tokens (September 1987). % Version 2.6 added 10sp slop when shipping leaders (November 1987). % Version 2.7 improved rounding of negative-width characters (November 1987). % Version 2.8 fixed weird bug if no \patterns are used (December 1987). % Version 2.9 made \csname\endcsname's "relax" local (December 1987). % Version 2.91 fixed \outer\def\a0{}\a\a bug (April 1988). % Version 2.92 fixed \patterns, also file names with complex macros (May 1988). % Version 2.93 fixed negative halving in allocator when mem_min<0 (June 1988). % Version 2.94 kept open_log_file from calling fatal_error (November 1988). % Version 2.95 solved that problem a better way (December 1988). % Version 2.96 corrected bug in "Infinite shrinkage" recovery (January 1989). % Version 2.97 corrected blunder in creating 2.95 (February 1989). % Version 2.98 omitted save_for_after at outer level (March 1989). % Version 2.99 caught $$\begingroup\halign..$$ (June 1989). % Version 2.991 caught .5\ifdim.6... (June 1989). % Version 2.992 introduced major changes for 8-bit extensions (September 1989). % Version 2.993 fixed a save_stack synchronization bug et alia (December 1989). % Version 3.0 fixed unusual displays; was more \output robust (March 1990). % Version 3.1 fixed nullfont, disabled \write{\the\prevgraf} (September 1990). % Version 3.14 fixed unprintable font names and corrected typos (March 1991). % Version 3.141 more of same; reconstituted ligatures better (March 1992). % Version 3.1415 preserved nonexplicit kerns, tidied up (February 1993). % Version 3.14159 allowed fontmemsize to change; bulletproofing (March 1995). % Version 3.141592 fixed \xleaders, glueset, weird alignments (December 2002). % Version 3.1415926 was a general cleanup with minor fixes (February 2008). % Version 3.14159265 was similar (January 2014).

alexpovel 6y ago

This looks amazing and like everything I always wanted. Sadly, I think basing off XeTeX and not LuaTeX is a mistake. Certainly renders it unusable for me. Having Lua integration is just great. Also, `lualatex` does not have some of the limitations of `xelatex` (memory limitations, `contours` package, ...), but I guess this XeTeX reimplementation can work on removing those implementations, so that only lack of Lua integration remains. Also, like another person said, not having biber breaks my workflow as well, which specifically tries to leverage the "latest and greatest" of what LaTeX has to offer [0]: `pdflatex` is obsolete, so `lualatex` it is. `nomencl`, `makeindex` etc. are obsolete, so `glossaries-extra` it is. `bibtex` is obsolete, so `biber` it is. Throw in `latexmk` for automatic compilation (which the tool presented here does too, which is a biggie! [1]) and CI/CD and you have a 1970s tool in 2020s attire. Lua rounds off the picture. Among other things, this given Unicode-native (gasp) code/documents, and great automation capabilities (`latexmk`, CI/CD, Lua). I think a modern TeX engine reimplementation should support all of the above, which are arguably the best modern options there are. [0]: https://collaborating.tuhh.de/alex/latex-git-cookbook https://collaborating.tuhh.de/alex/latex-git-cookbook [1]: I wonder if the logs are available though? aux, blg etc. are important for debugging and shouldn't be dropped outright.

chriszhang 6y ago
> Also, `lualatex` does not have some of the limitations of `xelatex` (memory limitations, `contours` package, ...) Are there some more details of the memory limitations you can share with us?
- alexpovel 6y ago
  LuaLaTeX allocates memory as-needed, see section 3.4.1 in the manual [0] (and comments/answers in this thread [1]). Base TeX has an arbitrary, by modern standards low memory limit, leading to a whole class of errors plaguing unsuspecting users [2], and spawning entire extensions to deal with these limitations [3]. This is simply an artefact of times past and has no technical relevance nowadays. LuaTeX allows dynamic allocation, with the available system RAM as the upper limit (so effectively, no limitations in everyday usage). Now, I could not find a mention of memory handling in the XeTeX reference manual [4]. People are using tricks like `tikzexternalize` with xelatex [5, 6]. Especially the first point makes me think XeLaTeX inherits base TeX memory handling/limits, but I cannot confirm this. I just know that all my problems disappeared when switching from XeLaTeX to LuaLaTeX. Lastly, see here [7] for a comprehensive (albeit somewhat anecdotal) list of advantages of LuaTeX over XeTeX. Of that list, `microtype` is another significant functionality I rely on. [0]: http://www.tug.org/texlive//devsrc/Master/texmf-dist/doc/context/documents/general/manuals/luatex.pdf http://www.tug.org/texlive//devsrc/Master/texmf-dist/doc/con... [1]: https://tex.stackexchange.com/q/7953/ https://tex.stackexchange.com/q/7953/ [2]: https://tex.stackexchange.com/search?q=tex+capacity+exceeded https://tex.stackexchange.com/search?q=tex+capacity+exceeded [3]: https://tex.stackexchange.com/a/482560/ https://tex.stackexchange.com/a/482560/ [4]: http://mirrors.ctan.org/info/xetexref/xetex-reference.pdf http://mirrors.ctan.org/info/xetexref/xetex-reference.pdf [5]: https://tex.stackexchange.com/q/438131/ https://tex.stackexchange.com/q/438131/ [6]: https://tex.stackexchange.com/q/334250/ https://tex.stackexchange.com/q/334250/ [7]: https://tex.stackexchange.com/q/126206/ https://tex.stackexchange.com/q/126206/

ageofwant 6y ago

This is great. After all these years the venerable and super useful TeX/LaTeX meme has once again managed to propagate into a new and exiting ecosystem. This would be good for the insufferable Rust community as well, I hope. Perhaps as the language they invested their identity in, actually becomes useful and venerated, they will calm the fuck down and start becoming adult members of the useful tool community. Nice one rustcucks, next do Python.

deleted 6y ago
[deleted]

jfk13 6y ago

Personally, I think this approach -- which as far as the engine is concerned looks like a largely-automated "translation" of xetex from C/C++ to Rust -- is misguided. The xetex code is a byzantine tangle of disparate pieces that evolved over the course of a number of years and several changes of direction; it originally started as a personal tool to address one individual's use case for a "Unicode-capable TeX", and grew from there. While it generally works really well, it's not a solid piece of engineering on which to build the future. The resulting code doesn't benefit from being Rust, except for the added buzzword-ness. It's simply a translation of the C code, littered with unsafe blocks, as it has not been architected to work with Rust's ownership semantics, borrow checker, etc. What XeTeX (or LuaTeX, though I don't find the Lua integration important -- but that depends heavily on your use cases) needs is a rewrite that preserves backward compatibility for documents, while re-architecting the engine using a modern language such as Rust. Simply wrapping the 1980s-era code in a "skin" of Rust syntax brings little value.

grey-area 6y ago
The advantage of this approach (as opposed to a rewrite), is that you reproduce all the bugs and misfeatures that people in the wild depend on. You can then add a test suite and start refactoring and gradually move to a codebase you're happy with, while breaking few or no users on the way. Another approach is to split it into blocks and replace parts of the system, but whether that is feasible depends on the software and how modular it is. Finally, you could try a clean-room rewrite, but that could take years without visible results and is hard to find the motivation for if existing software works ok.
- cryptos 6y ago
  But the advantage of avoiding C specific bugs is shallow, because the Rust code contains lots of unsafe blocks. Bugfixes made to the original source are not automatically ported to the Rust code base. From my point of view the chosen porting "strategy" doesn't make much sense. It is more of a toy project to see what's possible. What would really make sense would be starting with an extensive test suite and trying to build a properly architected Rust implementation according to it.

sega_sai 6y ago

I'm not sure rewriting Latex in XYZ is a good idea. The whole thing is a mess and a pain to use (and I use it multiple times a week). Sure the documents often look good, but when they don't or there is an error, fixing it just becomes a nightmare. I wish somebody would design a modern typesetting system that would accept say latex equations but without 50 years of cruft and levels of macroses upon macroses.

Veen 6y ago
Matthew Butterick is working on the Quad document processor, which is intended to "modernize the good ideas in LaTeX," but I'm unsure what its equation typesetting is like. It's written in Racket. https://docs.racket-lang.org/quad/ https://docs.racket-lang.org/quad/
- earthscienceman 6y ago
  The final note in the documentation on this software is a quote very fitting of a project re-thinking digital typesetting: “A way of doing something original is by trying something so painstaking that nobody else has ever bothered with it.” — Brian Eno
peteretep 6y ago
Sile appears to still be under active development: https://github.com/sile-typesetter/sile https://github.com/sile-typesetter/sile
marmaduke 6y ago
I don’t like Word but it does accept Latex math and is easy to use.
- gspr 6y ago
  Word does not use plain text input, and its output looks objectively terrible compared to LaTeX. These two things make it completely inadequate as a competitor.

smarx007 6y ago

Will the project offer a matching program to https://en.wikipedia.org/wiki/Knuth_reward_check https://en.wikipedia.org/wiki/Knuth_reward_check? :)

marktangotango 6y ago

This “x implemented in y” is a curious thing we are seeing with growing frequency nowadays (imo). On the one hand, good for these folks, to be rewriting something old in a new way (be it language, library, or framework). On the other hand, could this be an indicator that we’ve reached a certain level where it is easier to rehash the ideas of the past than create new work or invent new ideas or applications? Maybe someone with a more classical education can comment on how this manifests in the arts?

geodel 6y ago
Yeah, roughly something like doing cover versions of older songs but with modern, state of the art music arrangement. From what I know people do seem to love that.
colejohnson66 6y ago
What’s next is obviously JS or even WASM to typeset on the client side! /s
- hojjat12000 6y ago
  That should be fairly easy in Rust.
secondcoming 6y ago
The best way to learn a language is to port an existing tool to that language. Whether the rewrite is better depends on how you measure 'better', but what's certain is that someone gets to put 'I know Rust' on their CV.

hatmatrix 6y ago

Could this unify LaTeX and MathJax? I feel like any rewrite of LaTeX that could address portability would be a huge plus.

yza 6y ago
Would you care to elaborate on the point of your suggested integration? Are you (seriously) proposing the inclusion of a javascript backend into *TeX distribution?

paultopia 6y ago

Does this mean that at some point maybe it'll be possible to just drop it into webassembly?

todd8 6y ago

The title is a bit misleading. LaTeX is a macro package written using the TeX macro system, so I doubt that LaTeX itself is being rewritten in Rust. TeX, created by Donald Knuth, is now over 40 years old. It has a number of remarkable characteristics: * TeX is essentially bug free. It can be a challenging program to use, but in my experience I was always consoled by the knowledge that if something wasn't working it was my fault (in contrast to my use of other document processing systems and word processors). See [1]. * The TeX program is extraordinarily well documented. Knuth has an entire book which is the literate program comprises TeX. See [2]. * TeX was originally written in a version of Pascal, the source code was written using a literate style and processed using tools the Knuth wrote (Tangle and Weave). This was later machine translated into C. * TeX hasn't changed. It reached version 3 in 1989 and Knuth essentially froze it at this point. One of its great values is that documents written in TeX will always produce the same output, whether written last century, this century, or next century. We will always be able to see what the original author wanted. * TeX is unusually capable. Knuth is obviously a perfectionist. Unhappy with the state of digital typography when he wrote TeX he designed an entire toolchain for creating fonts call MetaFont. I used it to design my company's first unique logo back in the 90's. (I only needed five characters.) See [3]. Knuth did this before Postscipt was invented. * TeX is public domain. Before open source was a thing, Knuth put TeX in the public domain. * TeX is the Lingua Franca used by Mathematicians, Physicists, Computer Scientists and others for books, academic papers, and even presentations; nothing else comes close. * TeX has a number of widely used forks, notably XeTeX and LuaTeX. XeTeX added Unicode support and LuaTeX is an attempt to make TeX and it's companion LaTeX more easily programmable. See [4] and [5]. * TeX has a powerful macro system. This is how packages are added to TeX. The most widely used such package is LaTeX itself. LaTeX is not written in C or Pascal, it is written as a complex set of macros. * Tex's macro system is great for its intended purpose, but not so great for how it is currently used. Knuth originally used the macros to define layouts of books and papers. Things like line spacing font choice for page numbers and such can be easily encapsulated as TeX macros. TeX's macro system is powerful enough that even complex packages like LaTeX could be written in it. Today, there are packages like TikZ/Pgf that are very sophisticated and complex programs written entirely in TeX macros. TikZ/Pgf is a powerful graphics, drawing, and even animation system. It's manual is 1271 pages long. See [6] Where does this leave the originally linked project called Tectonic? It's not obvious from the github repository, but the user web site is a bit more informative, see [7]. This "rewrite" seems to rely on XeTeX and provides some command line operations that streamline the invocation of XeTeX. I don't understand its value proposition. For me, what's really needed is a better language than TeX's macro system for writing extensions. A few years ago I was writing TeX code (macros) to format spine labels for the books in my personal library. I used the number of pages in the book to adjust the size and layout of the call number and 2D barcode printed on the label. All of this was quite challenging in TeX's macro system, so I wrote a simple Python program that generated the LaTeX output that resulted in a page of labels that were easy to print. Maybe I could have done it in LuaTeX using Lua, but at the time LuaTeX was so poorly documented (how ironic) that I couldn't figure out how to do it. The adoption of some programming language that could guarantee backward compatibility indefinitely and a software package written in it that was compatible with LaTeX2e or LaTeX3 and replaced them. This language should not be TeX's macro system. The right language, perhaps something like ML or Scheme, would make programmed typesetting so much easier. [1] https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.4380190702 https://onlinelibrary.wiley.com/doi/abs/10.1002/spe.43801907..., behind a paywall but copies are easily available. [2] https://www-cs-faculty.stanford.edu/~knuth/abcde.html https://www-cs-faculty.stanford.edu/~knuth/abcde.html [3] https://en.wikipedia.org/wiki/Metafont https://en.wikipedia.org/wiki/Metafont [4] https://en.wikipedia.org/wiki/XeTeX https://en.wikipedia.org/wiki/XeTeX [5] https://en.wikipedia.org/wiki/LuaTeX https://en.wikipedia.org/wiki/LuaTeX [6] http://mirrors.ibiblio.org/CTAN/graphics/pgf/base/doc/pgfmanual.pdf http://mirrors.ibiblio.org/CTAN/graphics/pgf/base/doc/pgfman... [7] https://tectonic-typesetting.github.io/en-US/ https://tectonic-typesetting.github.io/en-US/ See Also: * Wikipedia entry for TeX: https://en.wikipedia.org/wiki/TeX https://en.wikipedia.org/wiki/TeX * Wikipedia entry for MetaFont: https://en.wikipedia.org/wiki/Metafont https://en.wikipedia.org/wiki/Metafont * Wikipedia entry for LaTeX: https://en.wikipedia.org/wiki/LaTeX https://en.wikipedia.org/wiki/LaTeX

mproud 6y ago

Is the rendering speed better, or probably not?

qkhhly 6y ago

\insteadof re-write \latex, it \might be \more \interesting to \change this \backslash \filled \syntax.

svat 6y ago

While the name “Rewriting LaTeX in Pure Rust” is very exciting, there are two problems: • “LaTeX” is a set of macros (ab)using the macro/text-expansion feature of an underlying TeX engine (LuaTeX, XeTeX, pdTeX, original Knuth TeX, or this one), to provide things like cross-references, section numbering, etc. When you as a typical user use “LaTeX”, you're actually using these macros, along with a huge variety of macro “packages” written by many authors, that together add up to orders of magnitude more lines of code than the TeX engine itself (about 25000 lines originally: https://tex.stackexchange.com/a/505664 https://tex.stackexchange.com/a/505664). Most of the incompatibilities and error messages a typical user encounters when using LaTeX are caused not by the TeX engine but by these packages or LaTeX, which this project doesn't touch. • This project comes about as: •— tex.web, Knuth's original source code, written in his literate-programming system WEB (a preprocessor on top of Pascal), •— extended to xetex.web (XeTeX), to provide Unicode support etc, •— automatically tangled to Pascal code (macros expanded and constants replaced), •— automatically translated to C code, •— now automatically translated to Rust code, by wrapping the C code in `unsafe` Rust blocks. So it has gone through multiple rounds of machine translation. Actually, I'm impressed that this project has made many improvements in undoing and restoring some of the WEB macros, since the last time this was mentioned here. (Compare a random comparison from 1 year ago: https://gist.github.com/627399d0150e66d211a264bc05b33beb https://gist.github.com/627399d0150e66d211a264bc05b33beb with the same comparison today: https://gist.github.com/71e47f2276f9c6c030efe0e3357ef3bf https://gist.github.com/71e47f2276f9c6c030efe0e3357ef3bf) — two of the four differences I had pointed out in my comment then (https://news.ycombinator.com/item?id=21177367 https://news.ycombinator.com/item?id=21177367) are gone, and the only ones to mention are: • The comments from the original are gone. • Something like “cur_if:=subtype(p);” becomes “ cur_if = MEM[p].b16.s0 as i16;” So there's still a long way to go. But even with all that, what would be more useful IMO are (in order of difficulty): • Translate directly from WEB (xetex.web) to Rust. The original uses only a subset of Pascal so IMO it should be possible to write an automatic translator that preserves more of the original. Something along these lines has been done for translating WEB to CWEB by Martin Ruckert (see https://w3-o.cs.hm.edu/users/ruckert/public_html/web2w/index.html https://w3-o.cs.hm.edu/users/ruckert/public_html/web2w/index... and http://tug.org/TUGboat/Contents/listauthor.html#Ruckert,Martin http://tug.org/TUGboat/Contents/listauthor.html#Ruckert,Mart... and his demo from TUG 2020). Alternatively, one could give up on XeTeX and start with the LuaTeX source code (in C). • Understand how the TeX program is written/works, and write a more idiomatic translation in Rust: something along these lines is in progress as a one-person hobby project by Emily Eisenberg (one of the original developers of KaTeX): see https://github.com/xymostech/XymosTeX https://github.com/xymostech/XymosTeX This one is most exciting to me, as I also have (not very useful so far) grand plans to understand TeX and to make it more understandable for everybody (https://shreevatsa.net/tex/program https://shreevatsa.net/tex/program). • As most of the code a user deals with is in the macro/LaTeX layer, modify TeX macro expansion to make it more “debuggable” (https://tex.stackexchange.com/a/384881 https://tex.stackexchange.com/a/384881, https://cstheory.stackexchange.com/a/40282 https://cstheory.stackexchange.com/a/40282): show full stack traces beyond what one gets with \errorcontextlines=\maxdimen (so file names and line numbers, by tracking the provenance of token lists), show arguments scanned (“passed in”) and expected/found types thereof, cleanly separate the macro expansion layer from the typesetting (line-breaking / page-breaking) layer, maybe even (JIT?) “compile” some of the lower-level or frequently-used macros — the eventual goal would be for the user to completely understand what is going on when they compile their LaTeX document, so that (1) they are less surprised by errors and know what to do, (2) they may be induced (if they're programmers) to do less in the (unsuitable) macro layer of the TeX engine and do more at the appropriate layer: either do it in a preprocessor that runs on the .tex file, or do it at the typesetting layer via something like LuaTeX's hooks (pre_linebreak_filter etc). Anyway, this project could still end up getting there, so let's see! Interacting with the TeX community (the TeX StackExchange, mailing lists, write an article for TUGboat, etc) might also be useful.