4 ms·

I absolutely agree, but would go even further. When you say Xetex "not a solid piece of engineering on which to build the future", I think the same thing also

I absolutely agree, but would go even further. When you say Xetex "not a solid piece of engineering on which to build the future", I think the same thing also applies to the original Tex engine written by Knuth. By modern software engineering standards, the original Tex implementation is a nightmare. It's enormously difficult to extend or add new features, and this has resulted in (1) comparatively few extensions being made and (2) when such extensions have been made (e.g. Xetex), they are very difficult technically.

taeric 6y ago
Oddly, I disagree with this take. TeX stands as a ridiculously stable codebase. Something that is not valued by near anyone in industry today. Such that the things we think make good engineering are standing on empirically weak arguments. Now, aesthetically I absolutely agree. It is an ugly language by most standards. But, if folks tried more extension and less porting to a new language, I'd wager they could get far. Instead, we seem to mainly get attempts at trying an extension by first establishing a new base language. Every time.
- returningfory2 6y ago
  > But, if folks tried more extension and less porting to a new language, I'd wager they could get far. This is what happens - pdfTex, Xetex, Luatex: these are all extensions of the core Tex implementation. The problem is that making these extensions is extraordinarily difficult given the software architecture of Tex. And, notoriously, sharing improvements between extensions is also very very difficult. The end result is that we have compatibility few improvements and complete stagnation is some aspects in the typesetting space (for example, no alternative/improvement to Tex's pagination algorithm). The problem is that Tex is architected as a monolithic application which you can't easily plug extra stuff into. All of the extensions to Tex have worked by forking the source code entirely, which I think is not a great model.
svat 6y ago
I agree, but one also can't blame Knuth: he wrote the program the way he knew best (he's a machine-code programmer at heart), under the constraints at the time (portability at various academic sites circa 1980 practically dictated Pascal, then Pascal's limitations required a preprocessor like WEB, etc). In fact, the earlier (TeX78) implementation in the SAIL language was written less monolithically, as a bunch of separate modules. He also did his best to make the implementation and source code understandable, publishing the program in print as an extensively documented/commented book (another reason for WEB), gave a workshop of 12 lectures about the implementation of the program, even had a semester-long course at Stanford with that book (program source code) as textbook (with exercises and exam problems). He also wrote TeX with hooks and some of its core functionality written as extensions using those hooks, hoping it would show others how to extend it. He has multiple times expressed surprise that more people didn't write their own versions of TeX. “Rewriting a typesetting system is fairly easy.” He seems to have overestimated the ability of others to read his code. If anything, I think a lesson from the TeX situation is that one's work can be too good: if he had simply published the algorithms at a high level (only the Knuth-Plass line-breaking algorithm was published as an independent paper) then maybe others would have implemented/combined them in interesting ways, but by publishing the entire source code and offering rewards for bugs etc, TeX got a (deserved) reputation as a very high quality stable and bug-free codebase and everyone wanted to use literally TeX itself. What's worse is that for a few years after it was created, TeX was possibly more widely available and more portable (what with its TRIP test and all that) than any single programming language (one had a much higher chance of TeX macros working consistently everywhere TeX was used, than code written in say C or Pascal): so it must have seemed natural to write large things like LaTeX entirely in TeX macros. As “TeX macros” wasn't designed or intended as a full-fledged programming language, we can see the effects today.
- JadeNB 6y ago
  > As “TeX macros” wasn't designed or intended as a full-fledged programming language, we can see the effects today. Making them Turing complete was a conscious decision, though a reluctant one: > Guy Steele began lobbying for more capabilities early on, and I [Knuth] put many such things into the second version of TEX, TEX82, because of his urging. http://maps.aanhet.net/maps/pdf/16_15.pdf http://maps.aanhet.net/maps/pdf/16_15.pdf

taeric 6y ago

Oddly, I disagree with this take. TeX stands as a ridiculously stable codebase. Something that is not valued by near anyone in industry today. Such that the things we think make good engineering are standing on empirically weak arguments. Now, aesthetically I absolutely agree. It is an ugly language by most standards. But, if folks tried more extension and less porting to a new language, I'd wager they could get far. Instead, we seem to mainly get attempts at trying an extension by first establishing a new base language. Every time.

returningfory2 6y ago
> But, if folks tried more extension and less porting to a new language, I'd wager they could get far. This is what happens - pdfTex, Xetex, Luatex: these are all extensions of the core Tex implementation. The problem is that making these extensions is extraordinarily difficult given the software architecture of Tex. And, notoriously, sharing improvements between extensions is also very very difficult. The end result is that we have compatibility few improvements and complete stagnation is some aspects in the typesetting space (for example, no alternative/improvement to Tex's pagination algorithm). The problem is that Tex is architected as a monolithic application which you can't easily plug extra stuff into. All of the extensions to Tex have worked by forking the source code entirely, which I think is not a great model.

svat 6y ago

I agree, but one also can't blame Knuth: he wrote the program the way he knew best (he's a machine-code programmer at heart), under the constraints at the time (portability at various academic sites circa 1980 practically dictated Pascal, then Pascal's limitations required a preprocessor like WEB, etc). In fact, the earlier (TeX78) implementation in the SAIL language was written less monolithically, as a bunch of separate modules. He also did his best to make the implementation and source code understandable, publishing the program in print as an extensively documented/commented book (another reason for WEB), gave a workshop of 12 lectures about the implementation of the program, even had a semester-long course at Stanford with that book (program source code) as textbook (with exercises and exam problems). He also wrote TeX with hooks and some of its core functionality written as extensions using those hooks, hoping it would show others how to extend it. He has multiple times expressed surprise that more people didn't write their own versions of TeX. “Rewriting a typesetting system is fairly easy.” He seems to have overestimated the ability of others to read his code. If anything, I think a lesson from the TeX situation is that one's work can be too good: if he had simply published the algorithms at a high level (only the Knuth-Plass line-breaking algorithm was published as an independent paper) then maybe others would have implemented/combined them in interesting ways, but by publishing the entire source code and offering rewards for bugs etc, TeX got a (deserved) reputation as a very high quality stable and bug-free codebase and everyone wanted to use literally TeX itself. What's worse is that for a few years after it was created, TeX was possibly more widely available and more portable (what with its TRIP test and all that) than any single programming language (one had a much higher chance of TeX macros working consistently everywhere TeX was used, than code written in say C or Pascal): so it must have seemed natural to write large things like LaTeX entirely in TeX macros. As “TeX macros” wasn't designed or intended as a full-fledged programming language, we can see the effects today.

JadeNB 6y ago
> As “TeX macros” wasn't designed or intended as a full-fledged programming language, we can see the effects today. Making them Turing complete was a conscious decision, though a reluctant one: > Guy Steele began lobbying for more capabilities early on, and I [Knuth] put many such things into the second version of TEX, TEX82, because of his urging. http://maps.aanhet.net/maps/pdf/16_15.pdf http://maps.aanhet.net/maps/pdf/16_15.pdf