5 ms·

I think completing the rewrite[1] of Tectonic[2] (Rust-powered XeTeX) has bigger potential and it's already working in many cases. [1] https://github.com/crlf0

by xvilka 5y ago

I think completing the rewrite[1] of Tectonic[2] (Rust-powered XeTeX) has bigger potential and it's already working in many cases. [1] https://github.com/crlf0710/tectonic https://github.com/crlf0710/tectonic [2] https://tectonic-typesetting.github.io/en-US/ https://tectonic-typesetting.github.io/en-US/

dhosek 5y ago
My goal with finl is not to re-create TeX but to reimagine it. The macro-oriented programming language is a big pain point, as are the whole concept of category codes. In many instances, LaTeX is still hemmed in by the constraints of ca 1980 computing technology. If you look through tex.web, you'll see things like weird string concatenations that were done in order to save a handful of bytes of memory. It was a radical change to TeX in the 80s to move to 32-bit addressing for the main memory array from 16-bit addressing. And Unicode support is occasionally weird, particularly in the incompatibilities between how ^^ literals are managed in pdfTeX vs XeTeX/LuaTeX (in the former, ^^D7 is a byte, in the latter ^^D7 is a code point and turns into a two-byte UTF sequence).
- xvilka 5y ago
  I understand your motivation. The thing is, there were many attempts to reimagine TeX, and none of them succeeded. Not because there were bad ideas, far from that. It's a huge project requiring a significant development team.
  - tavianator 5y ago
    Or one (1) Donald Knuth
  - guavaNinja 5y ago
    Plus a huge number of TeX packages have been made and used. You can't just ask people to leave all of that and learn a new language.
    - dhosek 5y ago
      The idea is that a LaTeX document and a finl document would look nearly identical between \begin{document} and \end{document}. As for packages, they fall into two categories: * document classes. Creating document classes is too difficult with LaTeX and a new document class with finl will be a relatively trivial thing to create, * extensions to LaTeX to remedy deficiencies. Some, like babel/polyglossia and the six million improvements on tabular would be things baked into the core format of finl. The most notable third-party ecosystem that would be translated would be the whole TikZ environment. I'm aiming to be able to allow TikZ illustrations to be cut and pasted into finl and to replicate at least the most common extensions. The other big one is the whole beamer ecosystem. The finl equivalent will keep similar if not identical syntax but will be superior in that it will be able to allow non-PDF backends for output (so, for example, one could run finl-ppt mypresentation.finl and get a PowerPoint presentation with all the capabilities that PowerPoint gives for things like animations and transitions that are difficult or impossible to achieve in a PDF-based presentation.
- lahcim8 5y ago
  Recently I have been also thinking about TeX's internals, also because I feel the pain points you describe in at https://www.finl.xyz/ https://www.finl.xyz/. I have too thought about experimenting with TeX rewrites/alternatives and particularly in Rust. I don't know if you are aware, but finl has been also submitted to Lobsters (https://lobste.rs/s/udu5oe/finl_is_not_latex_reinventing_latex_for https://lobste.rs/s/udu5oe/finl_is_not_latex_reinventing_lat...). One of the comments mentions SILE (https://sile-typesetter.org/ https://sile-typesetter.org/). I looked into it (https://www.youtube.com/watch?v=5BIP_N9qQm4 https://www.youtube.com/watch?v=5BIP_N9qQm4 was nice introduction, although I don't know how up to date it is). It seems to me that SILE replaces many pain points in TeX, while being somewhat close to LuaTeX (and TeX in general), due to use of similiar components (Lua, alogorithms, hyphenation patters, libraries). It even supports "LaTeX" like input syntax, but frankly I don't think that is the part of LaTeX I would keep :). Can you share your opinion on SILE?
  - xvilka 5y ago
    There is also Patoline[1][2], but it looks abandoned now. [1] https://patoline.github.io/ https://patoline.github.io/ [2] https://github.com/patoline/patoline https://github.com/patoline/patoline
    - claudionaoto 5y ago
      There is also SATySFi[1] (another typesetting system in Ocaml), it is active but mainly in japanese. [1] https://github.com/gfngfn/SATySFi https://github.com/gfngfn/SATySFi
  - jojohohanon 5y ago
    The examples on sile’s own webpage are hardly inspiring. Is it better than markdown, as laid out by my browser and the free pluggin, and saved to pdf? Not from those examples.
boromi 5y ago
What does [1] do over [2] ? I looked at the readme of [1] but it's just a fork of [2], so I'm not sure why [2] needs a rewrite? Is [1] just a rewrite of [2] in Rust?
- xvilka 5y ago
  Tectonic[2] is the Rust wrapper around C and C++ code of XeTeX. The fork rewrote everything, also aims to split it into separate crates that might be reused elsewhere, or to use some external crate instead of the own reimplementation.
nsajko 5y ago
Why would Tectonic have big potential? Why is it noteworthy at all? What benefits does it provide compared to ConTeXt?
- lahcim8 5y ago
  While I think that Tectonic can have benefits for those coming to "TeX", ConTeXt is by all means superior to all other TeXs and even Tectonic. While at the core there is always a TeX engine (original TeX, pdfTeX, XeTeX, LuaTeX) that provides typesetting primitives, almost everyone uses a macro package on top of that (plain TeX, LaTeX, ConTeXt) and there usually are many extending packages that are either targeted for one format or more of them (e.g. TikZ works in all three, while e.g. beamer is LaTeX specific). For many documents there can also be other external programs involved, e.g. BibTeX for bibliographies. The idea of LaTeX is that it is a core that provides basic facilities and programmer interface. A lot of essential functionality is then implemented by external packages. Hence, one doesn't usually get away by only installing engine (e.g. XeTeX) and format (LaTeX) to get e.g. XeLaTeX, but needs many other packages, like fontspec (for macro interface of XeTeX's OpenType support). That is why people install distributions, like TeX Live or MikTeX, which provide a way for installing all engines, formats and packages (and other software like BibTeX and specialized TeX editors). Because of the single pass nature of TeX, to get things like forward references in documents right, one needs to process the document multiple times, there are scripts like "latexmk", which can do this automatically (it can even run BibTeX and other external software, the right number of times and in the right order). In ConTeXt the situation is different. It is not a "minimal" format like LaTeX, but instead has all of the interesting functionality built in and nicely thought out. Also it has its own "distribution" that contains everything needed - the engine, format and even some fonts. While even ConTeXt has to be run multiple times to get references right, normally user doesn't even know about it, because "context" is a symlink to a script that does this automatically. Apart from that ConTeXt is vastly superior to any other TeX (both in engine and format). Tectonic solves many pain points of XeLaTeX specifically: - no need to install a distribution, all necessary files are downloaded when needed (from standard TeX Live), - runs LaTeX / external programs the right number of times, - nicer command line interface, - AFAIK can be somewhat used programmatically, But these problems are not present in ConTeXt. And even when comparing Tectonic to full TeX distrbutions, it obviously misses other engines / formats. All in all: comparing Tectonic and ConTeXt is not straightforward. Both make it easy for users to compile (different kind of) TeX documents. My opinion: If you can use ConTeXt, do it. If you have to use LaTeX and can use modern variant of it, perhaps Tectonic is an easy way that lets you get away from knowing too much about all the internals of TeX, that you don't want to know anyways. (Installing "LaTeX" (and the size of it) is one of the biggest problems I see people have with it, so this easy way of getting started is not to be neglected.)
lahcim8 5y ago
I too have recently been interested in TeX and Rust. Apart from Tectonic (wrapper around XeTeX and dvipdfmx) and the attempt to rewrite it in Rust (your first link), I also found [1], an attempt to rewrite TeX itself in Rust. I also understand that you are a supporter of the Tectonic in Rust effort, so hopefully you can fill me in on the current progress. At first I though that just rewriting XeTeX/dvipdfmx in Rust just for sake of being written in Rust was foolish, because of TeX's untypical memory model. Also because of the manual translation using c2rust (also on C code generated from Pascal code that uses a lot of macros). But after lookin at it now it seems that the rewrite progressed and the result are parts looking very Rusty, which is nice. I too had ideas about TeX in Rust, but I think that starting with LuaTeX would be much more beneficial. Apart from LuaTeX having obvious support of scripting in Lua and being very extensible, XeTeX has other disadvantages. See for example [2], which still in my opinion misses many internal differences, where LuaTeX is much superior. Do you have any tips on how to join the Tectonic / TeX in Rust community? Where can I potentially discuss my foolish TeX/Rust ideas, being very new to Rust? [1] https://github.com/crlf0710/tex-rs https://github.com/crlf0710/tex-rs [2] https://tex.stackexchange.com/a/593217/125126 https://tex.stackexchange.com/a/593217/125126
- xvilka 5y ago
  They have a forum but now discuss the switch to GitHub Discussions[1]. The issue about oxidizing everything is here[2]. If you want to help the project there is a number of issues[3] that can be done, in particular syncronization with the Tectonic mainline and XeTeX code update[4] [1] https://github.com/tectonic-typesetting/tectonic/issues/766 https://github.com/tectonic-typesetting/tectonic/issues/766 [2] https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459 [3] https://github.com/crlf0710/tectonic/issues https://github.com/crlf0710/tectonic/issues [4] https://github.com/crlf0710/tectonic/issues/299 https://github.com/crlf0710/tectonic/issues/299
  - lahcim8 5y ago
    Thank you for the pointers. I will try to get involved, unfortunately I won't have much time for it in the near future.

by xvilka 5y ago

dhosek 5y ago

My goal with finl is not to re-create TeX but to reimagine it. The macro-oriented programming language is a big pain point, as are the whole concept of category codes. In many instances, LaTeX is still hemmed in by the constraints of ca 1980 computing technology. If you look through tex.web, you'll see things like weird string concatenations that were done in order to save a handful of bytes of memory. It was a radical change to TeX in the 80s to move to 32-bit addressing for the main memory array from 16-bit addressing. And Unicode support is occasionally weird, particularly in the incompatibilities between how ^^ literals are managed in pdfTeX vs XeTeX/LuaTeX (in the former, ^^D7 is a byte, in the latter ^^D7 is a code point and turns into a two-byte UTF sequence).

xvilka 5y ago
I understand your motivation. The thing is, there were many attempts to reimagine TeX, and none of them succeeded. Not because there were bad ideas, far from that. It's a huge project requiring a significant development team.
- tavianator 5y ago
  Or one (1) Donald Knuth
- guavaNinja 5y ago
  Plus a huge number of TeX packages have been made and used. You can't just ask people to leave all of that and learn a new language.
  - dhosek 5y ago
    The idea is that a LaTeX document and a finl document would look nearly identical between \begin{document} and \end{document}. As for packages, they fall into two categories: * document classes. Creating document classes is too difficult with LaTeX and a new document class with finl will be a relatively trivial thing to create, * extensions to LaTeX to remedy deficiencies. Some, like babel/polyglossia and the six million improvements on tabular would be things baked into the core format of finl. The most notable third-party ecosystem that would be translated would be the whole TikZ environment. I'm aiming to be able to allow TikZ illustrations to be cut and pasted into finl and to replicate at least the most common extensions. The other big one is the whole beamer ecosystem. The finl equivalent will keep similar if not identical syntax but will be superior in that it will be able to allow non-PDF backends for output (so, for example, one could run finl-ppt mypresentation.finl and get a PowerPoint presentation with all the capabilities that PowerPoint gives for things like animations and transitions that are difficult or impossible to achieve in a PDF-based presentation.
lahcim8 5y ago
Recently I have been also thinking about TeX's internals, also because I feel the pain points you describe in at https://www.finl.xyz/ https://www.finl.xyz/. I have too thought about experimenting with TeX rewrites/alternatives and particularly in Rust. I don't know if you are aware, but finl has been also submitted to Lobsters (https://lobste.rs/s/udu5oe/finl_is_not_latex_reinventing_latex_for https://lobste.rs/s/udu5oe/finl_is_not_latex_reinventing_lat...). One of the comments mentions SILE (https://sile-typesetter.org/ https://sile-typesetter.org/). I looked into it (https://www.youtube.com/watch?v=5BIP_N9qQm4 https://www.youtube.com/watch?v=5BIP_N9qQm4 was nice introduction, although I don't know how up to date it is). It seems to me that SILE replaces many pain points in TeX, while being somewhat close to LuaTeX (and TeX in general), due to use of similiar components (Lua, alogorithms, hyphenation patters, libraries). It even supports "LaTeX" like input syntax, but frankly I don't think that is the part of LaTeX I would keep :). Can you share your opinion on SILE?
- xvilka 5y ago
  There is also Patoline[1][2], but it looks abandoned now. [1] https://patoline.github.io/ https://patoline.github.io/ [2] https://github.com/patoline/patoline https://github.com/patoline/patoline
  - claudionaoto 5y ago
    There is also SATySFi[1] (another typesetting system in Ocaml), it is active but mainly in japanese. [1] https://github.com/gfngfn/SATySFi https://github.com/gfngfn/SATySFi
- jojohohanon 5y ago
  The examples on sile’s own webpage are hardly inspiring. Is it better than markdown, as laid out by my browser and the free pluggin, and saved to pdf? Not from those examples.

boromi 5y ago

What does [1] do over [2] ? I looked at the readme of [1] but it's just a fork of [2], so I'm not sure why [2] needs a rewrite? Is [1] just a rewrite of [2] in Rust?

xvilka 5y ago
Tectonic[2] is the Rust wrapper around C and C++ code of XeTeX. The fork rewrote everything, also aims to split it into separate crates that might be reused elsewhere, or to use some external crate instead of the own reimplementation.

nsajko 5y ago

Why would Tectonic have big potential? Why is it noteworthy at all? What benefits does it provide compared to ConTeXt?

lahcim8 5y ago
While I think that Tectonic can have benefits for those coming to "TeX", ConTeXt is by all means superior to all other TeXs and even Tectonic. While at the core there is always a TeX engine (original TeX, pdfTeX, XeTeX, LuaTeX) that provides typesetting primitives, almost everyone uses a macro package on top of that (plain TeX, LaTeX, ConTeXt) and there usually are many extending packages that are either targeted for one format or more of them (e.g. TikZ works in all three, while e.g. beamer is LaTeX specific). For many documents there can also be other external programs involved, e.g. BibTeX for bibliographies. The idea of LaTeX is that it is a core that provides basic facilities and programmer interface. A lot of essential functionality is then implemented by external packages. Hence, one doesn't usually get away by only installing engine (e.g. XeTeX) and format (LaTeX) to get e.g. XeLaTeX, but needs many other packages, like fontspec (for macro interface of XeTeX's OpenType support). That is why people install distributions, like TeX Live or MikTeX, which provide a way for installing all engines, formats and packages (and other software like BibTeX and specialized TeX editors). Because of the single pass nature of TeX, to get things like forward references in documents right, one needs to process the document multiple times, there are scripts like "latexmk", which can do this automatically (it can even run BibTeX and other external software, the right number of times and in the right order). In ConTeXt the situation is different. It is not a "minimal" format like LaTeX, but instead has all of the interesting functionality built in and nicely thought out. Also it has its own "distribution" that contains everything needed - the engine, format and even some fonts. While even ConTeXt has to be run multiple times to get references right, normally user doesn't even know about it, because "context" is a symlink to a script that does this automatically. Apart from that ConTeXt is vastly superior to any other TeX (both in engine and format). Tectonic solves many pain points of XeLaTeX specifically: - no need to install a distribution, all necessary files are downloaded when needed (from standard TeX Live), - runs LaTeX / external programs the right number of times, - nicer command line interface, - AFAIK can be somewhat used programmatically, But these problems are not present in ConTeXt. And even when comparing Tectonic to full TeX distrbutions, it obviously misses other engines / formats. All in all: comparing Tectonic and ConTeXt is not straightforward. Both make it easy for users to compile (different kind of) TeX documents. My opinion: If you can use ConTeXt, do it. If you have to use LaTeX and can use modern variant of it, perhaps Tectonic is an easy way that lets you get away from knowing too much about all the internals of TeX, that you don't want to know anyways. (Installing "LaTeX" (and the size of it) is one of the biggest problems I see people have with it, so this easy way of getting started is not to be neglected.)

lahcim8 5y ago

I too have recently been interested in TeX and Rust. Apart from Tectonic (wrapper around XeTeX and dvipdfmx) and the attempt to rewrite it in Rust (your first link), I also found [1], an attempt to rewrite TeX itself in Rust. I also understand that you are a supporter of the Tectonic in Rust effort, so hopefully you can fill me in on the current progress. At first I though that just rewriting XeTeX/dvipdfmx in Rust just for sake of being written in Rust was foolish, because of TeX's untypical memory model. Also because of the manual translation using c2rust (also on C code generated from Pascal code that uses a lot of macros). But after lookin at it now it seems that the rewrite progressed and the result are parts looking very Rusty, which is nice. I too had ideas about TeX in Rust, but I think that starting with LuaTeX would be much more beneficial. Apart from LuaTeX having obvious support of scripting in Lua and being very extensible, XeTeX has other disadvantages. See for example [2], which still in my opinion misses many internal differences, where LuaTeX is much superior. Do you have any tips on how to join the Tectonic / TeX in Rust community? Where can I potentially discuss my foolish TeX/Rust ideas, being very new to Rust? [1] https://github.com/crlf0710/tex-rs https://github.com/crlf0710/tex-rs [2] https://tex.stackexchange.com/a/593217/125126 https://tex.stackexchange.com/a/593217/125126

xvilka 5y ago
They have a forum but now discuss the switch to GitHub Discussions[1]. The issue about oxidizing everything is here[2]. If you want to help the project there is a number of issues[3] that can be done, in particular syncronization with the Tectonic mainline and XeTeX code update[4] [1] https://github.com/tectonic-typesetting/tectonic/issues/766 https://github.com/tectonic-typesetting/tectonic/issues/766 [2] https://github.com/tectonic-typesetting/tectonic/issues/459 https://github.com/tectonic-typesetting/tectonic/issues/459 [3] https://github.com/crlf0710/tectonic/issues https://github.com/crlf0710/tectonic/issues [4] https://github.com/crlf0710/tectonic/issues/299 https://github.com/crlf0710/tectonic/issues/299
- lahcim8 5y ago
  Thank you for the pointers. I will try to get involved, unfortunately I won't have much time for it in the near future.