13 ms·

I've said it before and I'll say it again: the single most cost effective way to speed up math and physics research by a philanthropist would be taking a few mi

I've said it before and I'll say it again: the single most cost effective way to speed up math and physics research by a philanthropist would be taking a few million dollars and hiring a team of developers to bring LaTeX into the 21st century: Take the 100 most popular packages and fold them into the main code base (so I don't need to install conflicting packages just to get three columns!). Create a non-buggy desktop IDE/wordprocessor that works seamlessly using a traditional interface or a LyX-like interface. Merge/buy/replace Overleaf with an online service that syncs reliably with the desktop IDE. And, most importantly and difficultly, create a standard way for generating beautiful web pages (including mobile) from the same LaTeX++ files that can be used to generate beautiful PDFs. The annual NSF physics budget is roughly a quarter billion dollars. This only needs to make NSF-funded physicists 1% more productive over a decade to justify spending $20M, and the entire world would benefit. EDIT: Lots of commenters worry that this will end up like various closed-source for-profit tools like Google Docs. But for a closer analogy they should look at some of the excellent free open-source science software being produced with philanthropic funding such as Zotero. Physicists will only adopt it if it makes their lives easier (and maybe not even then...). EDIT 2: There are some pretty strong parallels between the commenters suggesting this can already be achieved with homegrown tools and the infamous HN Dropbox comment.

generationP 6y ago
Yes for IDE. Yes for HTML+ output (though what exactly the "+" entails is a tricky thing; HTML+CSS won't suffice). Not sure about merging the 100 most popular packages. Most likely their conflicts cannot be resolved in a backwards-compatible way, and then you have the standard "101 standards" outcome and old tex files start rotting.
- jessriedel 6y ago
  Yea, I really meant merging in the functionality of the most popular packages, and keeping as much of the syntax as possible without compatibility issues. I agree with you about those risks.
angrais 6y ago
You can use VSCode as LaTeX IDE and git to sync to overleaf... Which costs slightly less than $20M
- arachnid92 6y ago
  My thoughts exactly. I'm currently working on a paper together with an older professor who barely manages to use Overleaf, and this is how I make it work. I keep my copy of the paper on my personal remote git repo. On my local workspace, I keep a clone of that repo with an additional branch tracking the Overleaf version. Then it's just a matter of merging branches whenever any of us make changes. She gets to enjoy the convenience of using Overleaf and I get to keep using VSCode and all my extensions without hassle.
AlexandrB 6y ago
Considering what "modernizing" other parts of the tech stack has meant, I would expect a 21st century LaTeX to be hundreds of megabytes, require running a buggy, slow electron app, and phone home to a variety of SaaS providers when you use it. Edit: Someone beat me to it by suggesting VSCode as an IDE.
- rimunroe 6y ago
  Hundreds of megabytes would be a considerable improvement for me. I think my last installation of LaTeX and the few packages I depended on was around two gigabytes.
  - shuckles 6y ago
    I think you are confusing RAM with storage.
    - rimunroe 6y ago
      I suppose they could have meant that, but I think it's pretty reasonable to interpret that as having meant size on disk
      - shuckles 6y ago
        You’re right. The original post was not clear.
  - ajmurmann 6y ago
    And it's already slow and needs to run multiple times to generate a table of contents and bibtex Edit: phone autocomplete typo
    - godelski 6y ago
      Generally I use a Makefile that looks like this ``` SRC :=$(wildcard .tex .bib) Paper.pdf: $(SRC) pdflatex Paper bibtex Paper.aux pdflatex Paper pdflatex Paper short: $(SRC) pdflatex Paper clean: rm -f .glo .log .dvi .gls .toc .aux .ist .out .glg .pdf .bbl .blg .lof .brf *.sta .PHONY: clean ``` I hate that you have to run things multiple times to get references right. Really it should just be `pdflatex Paper` and you should be done. Every once in awhile I just get references that don't want to stick. Same with images. [ht!] and floatbarriers only go so far. As much as tikz is a pain (and beautiful), I'd definitely invest more time into making image placement and references easier for the average user. EDIT: code has a lot of wildcards before every extension but I'm not sure how to escape those on HN
contravariant 6y ago
Seems easier to take org or markdown and add stuff to it until it's on par with LaTeX.
- jessriedel 6y ago
  Markdown is not even 10% of the way toward a full document typesetting system, so you might as well throw it away and create a new codebase. The only thing you're really suggesting then is making it backward-compatible with the markdown syntax.
  - contravariant 6y ago
    Is markdown anything other than syntax? But yes quite a big part on (mostly) pagination is missing, which will need to be added. I still think this is easier than merging the commonly used LaTeX packages into one code base and then building a proper IDE and a method to output to HTML. Much better to start with markdown, extend KaTeX to support whatever you still need for math notation, and then work on making pandoc output to PDF (or equivalent) natively. On of the bigger problems is bridging the gap between unpaginated HTML and paginated PDF. However if you start from the LaTeX side you'll need to deal with decades of technological debt as well.
zitterbewegung 6y ago
This is going to be really unpopular but you could allow people to write their papers in Microsoft word . Either that or the other answer would be to say that’s why you have grad students .
- simion314 6y ago
  It is like asking developers to write html pages and code in Word.
- privong 6y ago
  > This is going to be really unpopular but you could allow people to write their papers in Microsoft word . Many journals do allow Word submissions. Nature, for example[0] [0] "we strongly encourage you to incorporate the manuscript text and figures into a single PDF or Microsoft Word file" https://www.nature.com/nature/for-authors/initial-submission https://www.nature.com/nature/for-authors/initial-submission
- rat9988 6y ago
  Word with a template is the easiest way I've found to write scientific papers.
- henrikeh 6y ago
  Word is the preferred format in many fields. In my field, Word is preferred for conference submissions, because the tight page limits (2-3 pages) is easier to work with in a WYSIWYG editor.
- MaxBarraclough 6y ago
  Proprietary payware that doesn't support Linux and doesn't work with Git? For current users of LaTeX, it's dead in the water. > other answer would be to say that’s why you have grad students . Of which discipline?
  - zitterbewegung 6y ago
    Any of them. Professors generally chase grants to fund grad students who need tasks to get a PhD.
inimino 6y ago
Knuth wrote TeX as a labor of love. The surest way to kill everything good about it would be a few million dollars, and a team, and to turn it into fucking Google Docs for science. Ask Knuth to name one person who has the clearest vision for the future of TeX, and give that person a stipend to work on it for the next ten years, or for life, and you might actually do some good.
- nerdponx 6y ago
  Leave TeX alone, just build better tools on top of it.
  - derefr 6y ago
    I mean, that's what they were proposing; LaTeX is "on top of TeX", and LaTeX is what they were suggesting should be fixed.
    - nerdponx 6y ago
      Yes, but I was responding to a comment about TeX.
  - jfk13 6y ago
    While it's possible to build a lot of things on top, maybe it'd be worth making some minor updates to the core TeX engine itself, to bring it into the world of Unicode and OpenType fonts... Oh, there are a couple of such candidates, actually, built with very different philosophies: there's XeTeX, intended to graft Unicode and OpenType support onto TeX with as little disruption as possible, such that it "just works" with minimal effort; and there's LuaTeX, which "deconstructs" TeX to give you an infinitely-flexible typesetting toolkit that can do just about anything, if you can figure out how to wield it.
    - nerdponx 6y ago
      Good points.
- 6y ago
felixfbecker 6y ago
Imo papers of the future should be authored in semantic HTML, which can be rendered to PDF, but most importantly be reflowed and read on any digital device, also by people with disabilities, and indexed/searched. The capabilities are all there - MathML, CSS for print layout, <figure>, SVG, highlight.js, LaTeX.css, ...
- isatty 6y ago
  I disagree (or maybe because I don't fully understand your comment). In my (limited, I'm not a FE dev) experience, you layout an HTML page by hand but I do not want that with a research paper. I want to write my paper and let LaTeX lay it out for me. Of course I'll have to wrestle with an unruly table or figure now and again but for the most part, the layout and typesetting of the document is handled for me.
- _bxg1 6y ago
  It's the right platform, but it's the wrong authoring experience for people who might not be able to code at all, much less understand the nuances of CSS and how it interacts with the DOM. People using LaTeX don't want to code something, they want to write something and just have requirements that go beyond the typical use-case for "writing something". A layer is needed on top. Maybe that could take the form of a single JS and/or CSS file that gets dropped into an HTML page and then the author can create their documents with nothing but special tag attributes/classes. But likely it would need to be even more abstracted than that. Here's a good attempt that's being made. It's an extended version of Markdown that covers charts, equations, etc.: https://casual-effects.com/markdeep/ https://casual-effects.com/markdeep/
- josefx 6y ago
  I wrote some documentation in html and wanted to export it as pdf. The main blocker: an index with page numbers. css has them, but support for them is mostly non existant.
thorwasdfasdf 6y ago
I'd say, make sure it's backwards compatible with the old way of doing it because there's a lot of scientists who have been using it for a looong time and don't like change. You have to remember, for a scientist, a tool is just a tool, not all of them are going to invest a lot of time in learning a whole new process.
- gibolt 6y ago
  The old way doesn't have to change. It still exists. Make a wrapper GUI that makes it far more approachable. Syncing and other funtions could be handled by a library that works with either
CJefferson 6y ago
The "Website" point brings up an important point. LaTeX is suprisingly terrible at producing accessible documents. There are various systems and packages which make heroic efforts to make accessible output, but you have to use them from the start, as they work like you suggest -- they reimplement (or at least heavily modify) various LaTeX packages to make their outputs accessible. I've had a blind PhD student and the standard of using LaTeX to produce PDFs for academic papers has been the single biggest block in his progress.
- marvy 6y ago
  I assumed that if ctrl-F works, then screen readers also work. Can you share a bit of information about what tends to go wrong, and/or how document authors can fix it?
  - j-pb 6y ago
    Almost no documents properly support ctrl-F. You'll get at best single word search, but you won't find coherent sentences. There are arbitrary line-breaks caused by various styling and layout packages (most columns are broken). There are even publishers that purposefully put garbage into the PDF's text layer as a braindead form of "piracy protection". PDF and TeX are broken beyond repair in terms of accessibility. The former because it allows the text and image to be split, and very few people care about text as long as it looks alright. And the latter because it's entire document representation model is based around computation. Knuth purposefully made it Turing complete in a misguided attempt at timelessness (no need for a replacement if it's infinitely extensible right?), which means there is no proper way to view a TeX document as "just text data". I've worked in a library digitising documents, and whenever we encountered PDFs we'd just OCR the damn thing. We need to throw the thing out, swallow the bitter pill of sub optimal kerning, and replace it with HTML asap.
    - solinent 6y ago
      > PDF and TeX are broken beyond repair in terms of accessibility. All that needs to be done is to have a accessible text section of the PDF--then Tex can just include the source of the document minus the Tex commands into the PDF, and PDF readers can have the screen reader work. The text could also just be stored per page, if the blind require to communicate where they found a particular item to a sighted person.
bfirsh 6y ago
The important/difficult problem of generating beautiful web pages is most of the way there already: https://github.com/arxiv-vanity/engrafo https://github.com/arxiv-vanity/engrafo There are some features in LaTeXML, which powers Engrafo, for adding some "LaTeX++" features, like embedding JS, etc. Still could do with a lot of work, though.
setgree 6y ago
> the single most cost effective way to speed up math and physics research by a philanthropist would be taking a few million dollars and hiring a team of developers to bring LaTeX into the 21st century This might be a niche opinion, but I think Rmarkdown & Jupyter get us a non-negligible portion of the way there. You can use LaTeX chunks when you need equations, markdown syntax when you need formatting (IIRC this is a pain with LaTeX), and executed code chunks to (e.g.) create and insert a graph. (One advantage of Rmarkdown is that you can have as many languages in a single document as knitr can handle, each its own chunk [0]). Neither is all the way there yet, but, as Yihui Xie puts it [1]: > I have a dream that one day all students and researchers will forget what "formatting a paper" even means. I have a dream that one day journals and grad schools no longer have style guides. I have a dream that one day no missing $ is inserted, and \hbox is never overfull. [0]: https://bookdown.org/yihui/rmarkdown/language-engines.html https://bookdown.org/yihui/rmarkdown/language-engines.html [1] https://twitter.com/xieyihui/status/1022873179532996609 https://twitter.com/xieyihui/status/1022873179532996609
- j-pb 6y ago
  bUt MuH TypESeTtInG! I think you're absolutely right. But people seem to value a specific flavour of aesthetics over easy to understand and reproducible research. I guess that's maybe partly caused by the publication pressure, once you start to pump out papers for pumping out papers sake, there is little incentive to go beyond the superficial.
  - derefr 6y ago
    I have a dream that one day we'll get a language for typesetting that isn't a markup language. Rather, this language would be designed to come after writing, in the same way that flagship Desktop Publishing software (InDesign, Publisher) make the assumption that layout happens after writing. (Sure, you can write inside the layout software, but you're not supposed to. They're assuming e.g. a journalist–editor–layout–typesetter pipeline.) You'd have a "layout file" (written in the typesetting DSL) that has separate "non-typeset rich-text" files linked to it via file/URI references (just like Desktop Publishing apps do.) The person doing layout would never modify the linked source files; if they wanted to e.g. change certain character sequences into filligrees, that'd be something stored entirely as an annotation in the layout file, not as markup in the source file. In that world, scientists just write the "non-typeset rich text" (for which Rmarkdown / Jupyter would be a great fit); and then it's the job of science journals to produce the "layout + typesetting annotations." And hey, then journals would actually have a reason to exist! They would be to science as physical magazines are to journalism at this point: a place to find the articles you can already read online, but typeset much better.
fizixer 6y ago
No need (edit: except maybe for collaborative editing). Use TeXLive and install the updated copy every year (it has annual releases, and you simply delete the old install directory). If you're not on linux, spin a Ubuntu VM, and install TeXLive. (Strictly speaking you don't even need GUI on the machine where pdf gets compiled. I add 'scp output.pdf mydesktop:.' to my Makefile so the file is transferred to my desktop everytime I make. On the desktop the pdf is already open on the big screen and if your pdf reader is good, it'll update the opened pdf everytime the file on harddisk gets updated) I never use overleaf, and I had no problem finished up my thesis in record time. LaTeX wasn't even an issue. edit: Overleaf will be helpful for collaborative editing though. I personally would use github/gitlab instead if I ever have to do collaboration.
ergl 6y ago
Writing the paper is the easy part, after spending months researching past literature, coming up with models and hypothesis, and performing experiments. I really doubt Latex is the bottleneck for doing research.
- j-pb 6y ago
  You'd be surprised. I've spend hours debugging other peoples references, bibliographies and journal dictated styles. Also if you think that all of academia has MONTHS to work on papers, with our current publish or perish culture, you're wayyyyyy off.
  - ergl 6y ago
    I admit I've been lucky with deadlines. The shortest span has been 3 weeks, but that was one time where most of the work was already done, and only then we looked for suitable conferences to submit the paper.
- jessriedel 6y ago
  Nothing in my comment suggests a bottleneck. It's just a matter of making people a few percentage point more efficient at a very low price.
  - jpab 6y ago
    You referred to it as "the single most cost effective way to speed up math and physics research". I don't think it's so surprising that people are interpreting that to mean you think LaTeX is a bottleneck in the research process. I guess you intended emphasis on the "cost effective" part of the statement, but not everyone will read it like that (I didn't).
    - jessriedel 6y ago
      I mean, it wasn't merely emphasis. Likewise, "NBA" is not mere emphasis in the statement "Muggsy Bogues is the shortest person in NBA history".
truckerbill 6y ago
Using a scripting language to format text always seemed insane to me. People love LaTeX but it's nowhere near as intuitive as writing formulas down on a piece of paper. Having two panes, one for the output seems also very hacky and unintuitive. Instant feedback in one view, typing at the speed of thought should be the goal, and if it means a new paradigm I'm all for it. Imagine it was something you could actually program with; those gross inline math functions would suddenly be so much more parsable.
- BlueTemplar 6y ago
  LuaLaTeX ?
- abdullahkhalids 6y ago
  Remember that scene in the matrix when he says "I don't even see the code anymore". After a short while (about a couple of hundred pages of typing latex math for me), the code is mostly as intuitive as the visual formulas. At least for not too long equations. [1] https://www.youtube.com/watch?v=MvEXkd3O2ow https://www.youtube.com/watch?v=MvEXkd3O2ow
- jessriedel 6y ago
  LyX exists. It is used only by a small minority of physicists even though it's been around for 25 years.
stared 6y ago
Oh, it is a lot of effort. Just pouring a lot of money and hoping that we can get through warts (without losing the best parts) of LaTeX is unreasonable. And in fact, there are projects: - https://www.mathjax.org/ https://www.mathjax.org/, and its much faster (and less feature-complete) variant https://katex.org/ https://katex.org/ - http://www.luatex.org/ http://www.luatex.org/ - I guess this is the closest to what you mean - https://rstudio.github.io/distill/ https://rstudio.github.io/distill/ (using Knitr; it can generate HTML or PDFs) etc, etc. Even now, other libraries use BibTeX, which would be the easiest to replace in some more standardized format (JSON or YAML, so it is easy to load from web, directly).
Thrymr 6y ago
How about a modern TeX formatting layer to replace LaTeX?
andi999 6y ago
Actually the monolithic Tex has been done. Its called context. (Every couple of years I try for a few hours to install it under windows but then give up). Sometimes I manage to install it and then give up because of the introductory texts. https://en.wikipedia.org/wiki/ConTeXt https://en.wikipedia.org/wiki/ConTeXt (but I still believe there is gold at the bottom of this) Edit: woooah. I see there is now an installer for context, since april 2019! https://wiki.contextgarden.net/Installation https://wiki.contextgarden.net/Installation here we go again....
LeifCarrotson 6y ago
I used LyX to write a pretty significant paper back in 1.6, a decade later Lyx is now on 2.3; I just installed it and opened my old research paper. Obviously, I haven't done a thorough test, but from 5 minutes poking around there's a huge improvement in appearance, documentation, and usability! There's a similar story for the electronics design software package KiCAD - I played with that a bit in school as well, when it was more like a group of separate software packages duct-taped together with some import/export scripts. The enthusiasts weren't bothered by and seemed to actually enjoy the lack of polish. Recently, though, CERN has injected a bunch of money, programming time, and real-world use engineer's time into the project and it really shows. I definitely agree that the NSF (or any major LaTeX-using organization) injecting a few million dollars into LyX would be an incredible boon to human productivity. I do feel that it would be better for an organization with internal consumers of the software to undertake this effort than for an outside team to just start with a blank canvas. The latter too frequently ends up with the primary users being developers, where bug reports or feature requests will be responded to with "Just compile with -D FEATURE_ENABLE" or "For the syntax of that menu item, read the block comment and source code of the foo.cpp/bar() method" - basically, where TeX is right now. Most users of most software will install it with the Windows .exe, a few will use the .dmg/.deb/.rpm or their package manager: a vanishingly small subset are compiling it from source. And if you expect that your users should be able to read C++, your user base is going to very small.
pnw_hazor 6y ago
Interesting. When I did some internships at Los Alamos (LANL) my lab neighbor was a theoretical physicist. He used Word and referred to LaTex as the Fortran of word processors.
narrationbox 6y ago
I have an alternative view: The fastest way to speed up research would be to fund an open source Wolfram software suite with feature parity. For a lot of hard tech you need good multiphysics and CFD simulations, both of which suffer from a severe lack of open source tooling. Most of the proprietary engineering software can be cloned rapidly once a solid foundation is built. Even though Julia and Python are making progress, in terms of ergonomics and productivity it is still hard to beat the offerings by Wolfram. (If somebody is looking for a project for Mozilla's "fixing the internet" incubator, maybe give this a shot.)
- abdullahkhalids 6y ago
  Python ecosystem is not even close. I recently switched from Mathematica to Python, and I am going mad from juggling half a dozen packages with overlapping features. Really wish somebody would make a meta package consisting of numpy, scipy, sympy, sage that removes conflicts and overlaps, and renames all functions with the same naming scheme.
johannes1234321 6y ago
Luckily Knuth himself has this on his radar as this "earthshaking anouncment" by him shows: https://youtu.be/eKaI78K_rgA https://youtu.be/eKaI78K_rgA (Won't spoil it by giving more details)
MperorM 6y ago
I wish I could upvote this a thousand times and then some. Already in the 80's Jeff Bezos refused to use latex due to its absurd difficulty to use. I can't believe it's been 40 years and there still isn't a better tool to write papers with. It took me less than an hour to create a Google Docs template that looks roughly as good as a standard Latex document. The reason Latex looks beautiful is due to a set of wisely chosen defaults, nothing more. Yet the focus of Google Docs clearly isn't scientific papers. I don't imagine it would take more than a few passionate employees at Google to expand Docs to a powerhouse of scientific writing. Pictures of my template can be found below: https://imgur.com/a/cC0e0gG https://imgur.com/a/cC0e0gG (and a blogpost about my Latex induced misery) http://mathiasbonde.com/?p=72 http://mathiasbonde.com/?p=72
- milquetoastaf 6y ago
  That's a great template - mind sharing?
- mNovak 6y ago
  I agree beautiful defaults help LaTeX, all the more so because I always use someone else's well defined template. But, my main issues with Docs/Word is not just "look", rather: - smart figure/caption/table handling (no orphaned captions, no figures hanging out in the middle of the page) - easy, label-based cross referencing (not selecting each from a popup interface) - text-based equation input All that being said, I still have an elaborate Word template I use for proposal writing. Because often I have to pass it off to others (who may not know TeX) to contribute, and sometimes I need to quickly bastardize my nice formatting to squeeze into hard page limits. ShareLatex / Overleaf is very good, but I'd love to see slightly more ease of use as OP suggests. I'd like to see a switchable WYSIWYG / code editor, so laypeople can make simple formatting changes visually without needing to know the right commands, and it propagates correctly into the underlying LaTeX.
  - MperorM 6y ago
    Totally agree with your points. Latex remains the best tool for many tasks. But none of the things you described, I think latex does particularly well either. Better than Google docs and the likes, sure, but I think it's possible to do better. I truly think Google Docs could be so much more than it currently is. They for example built a brilliant speech-to-text feature that is only available on desktop. Mobile is the one place I could imagine using such a feature. Imagine writing your documents with speech while hiking deep in the woods. Feeling the fresh air with a view of a beautiful brook as Google transcribes your thoughts. That's how I want to write! Why on earth put so much effort into such a brilliant feature, and then have it only be available to users already sitting in front of a keyboard!? Apologies for the tangent, it was where my mind wanted me to go!
epicureanideal 6y ago
Don't most journals accept MS Word submissions now? Doesn't Word provide all the formatting needed? I also prefer Latex but I wonder if we're talking about solving a problem nobody has?
est31 6y ago
You should check out Tectonic. https://tectonic-typesetting.github.io https://tectonic-typesetting.github.io It fixes some of the issues that LaTex has.
blackrock 6y ago
This sounds like an opportunity for someone to create a python module to generate scientific documents. You’ll get all the benefits of python, with the added benefits of generating formatted text. Call it pyTeX. LOL..
teleforce 6y ago
I am surprised nobody here mentioned TeXmacs. It's nothing to do with TeX or Emacs, but it's inspired by them. Basically the concept is structural graph based document editing software similar to Tioga/Cedar by Xerox back in 1980s but unfortunately the research and improvement for better tools for editing document seems to stagnate a lot after 90s. I think the idea of having an easy to use WYSIWYG editor for scientific document editing need to be popular now to significantly increase scientist productivity. Remember that web was invented to increase scientist productivity and the rest is history. On a related note, personally I think Overleaf or Google Docs is missing the point or the pain points. Even though both of them enabled offline editing but they are really a cloud first applications. The better approach is to have something like TeXmacs (native desktop first application) and make a seamless integration with synchronization and versioning to the cloud. With a proper use of synchronization and versioning technology like rsync and Git, together with seamless connectivity tools and protocols for examples Wireguard VPN, WebDAV or even the latest SMB over QUIC approach, I really think this is very feasible. The CONCEPT is similar to the useful and successful Watcom product or now SQL Anywhere database editing application.

generationP 6y ago

Yes for IDE. Yes for HTML+ output (though what exactly the "+" entails is a tricky thing; HTML+CSS won't suffice). Not sure about merging the 100 most popular packages. Most likely their conflicts cannot be resolved in a backwards-compatible way, and then you have the standard "101 standards" outcome and old tex files start rotting.

jessriedel 6y ago
Yea, I really meant merging in the functionality of the most popular packages, and keeping as much of the syntax as possible without compatibility issues. I agree with you about those risks.

angrais 6y ago

You can use VSCode as LaTeX IDE and git to sync to overleaf... Which costs slightly less than $20M

arachnid92 6y ago
My thoughts exactly. I'm currently working on a paper together with an older professor who barely manages to use Overleaf, and this is how I make it work. I keep my copy of the paper on my personal remote git repo. On my local workspace, I keep a clone of that repo with an additional branch tracking the Overleaf version. Then it's just a matter of merging branches whenever any of us make changes. She gets to enjoy the convenience of using Overleaf and I get to keep using VSCode and all my extensions without hassle.

AlexandrB 6y ago

Considering what "modernizing" other parts of the tech stack has meant, I would expect a 21st century LaTeX to be hundreds of megabytes, require running a buggy, slow electron app, and phone home to a variety of SaaS providers when you use it. Edit: Someone beat me to it by suggesting VSCode as an IDE.

rimunroe 6y ago
Hundreds of megabytes would be a considerable improvement for me. I think my last installation of LaTeX and the few packages I depended on was around two gigabytes.
- shuckles 6y ago
  I think you are confusing RAM with storage.
  - rimunroe 6y ago
    I suppose they could have meant that, but I think it's pretty reasonable to interpret that as having meant size on disk
    - shuckles 6y ago
      You’re right. The original post was not clear.
- ajmurmann 6y ago
  And it's already slow and needs to run multiple times to generate a table of contents and bibtex Edit: phone autocomplete typo
  - godelski 6y ago
    Generally I use a Makefile that looks like this ``` SRC :=$(wildcard .tex .bib) Paper.pdf: $(SRC) pdflatex Paper bibtex Paper.aux pdflatex Paper pdflatex Paper short: $(SRC) pdflatex Paper clean: rm -f .glo .log .dvi .gls .toc .aux .ist .out .glg .pdf .bbl .blg .lof .brf *.sta .PHONY: clean ``` I hate that you have to run things multiple times to get references right. Really it should just be `pdflatex Paper` and you should be done. Every once in awhile I just get references that don't want to stick. Same with images. [ht!] and floatbarriers only go so far. As much as tikz is a pain (and beautiful), I'd definitely invest more time into making image placement and references easier for the average user. EDIT: code has a lot of wildcards before every extension but I'm not sure how to escape those on HN

contravariant 6y ago

Seems easier to take org or markdown and add stuff to it until it's on par with LaTeX.

jessriedel 6y ago
Markdown is not even 10% of the way toward a full document typesetting system, so you might as well throw it away and create a new codebase. The only thing you're really suggesting then is making it backward-compatible with the markdown syntax.
- contravariant 6y ago
  Is markdown anything other than syntax? But yes quite a big part on (mostly) pagination is missing, which will need to be added. I still think this is easier than merging the commonly used LaTeX packages into one code base and then building a proper IDE and a method to output to HTML. Much better to start with markdown, extend KaTeX to support whatever you still need for math notation, and then work on making pandoc output to PDF (or equivalent) natively. On of the bigger problems is bridging the gap between unpaginated HTML and paginated PDF. However if you start from the LaTeX side you'll need to deal with decades of technological debt as well.

zitterbewegung 6y ago

This is going to be really unpopular but you could allow people to write their papers in Microsoft word . Either that or the other answer would be to say that’s why you have grad students .

simion314 6y ago
It is like asking developers to write html pages and code in Word.
privong 6y ago
> This is going to be really unpopular but you could allow people to write their papers in Microsoft word . Many journals do allow Word submissions. Nature, for example[0] [0] "we strongly encourage you to incorporate the manuscript text and figures into a single PDF or Microsoft Word file" https://www.nature.com/nature/for-authors/initial-submission https://www.nature.com/nature/for-authors/initial-submission
rat9988 6y ago
Word with a template is the easiest way I've found to write scientific papers.
henrikeh 6y ago
Word is the preferred format in many fields. In my field, Word is preferred for conference submissions, because the tight page limits (2-3 pages) is easier to work with in a WYSIWYG editor.
MaxBarraclough 6y ago
Proprietary payware that doesn't support Linux and doesn't work with Git? For current users of LaTeX, it's dead in the water. > other answer would be to say that’s why you have grad students . Of which discipline?
- zitterbewegung 6y ago
  Any of them. Professors generally chase grants to fund grad students who need tasks to get a PhD.

inimino 6y ago

Knuth wrote TeX as a labor of love. The surest way to kill everything good about it would be a few million dollars, and a team, and to turn it into fucking Google Docs for science. Ask Knuth to name one person who has the clearest vision for the future of TeX, and give that person a stipend to work on it for the next ten years, or for life, and you might actually do some good.

nerdponx 6y ago
Leave TeX alone, just build better tools on top of it.
- derefr 6y ago
  I mean, that's what they were proposing; LaTeX is "on top of TeX", and LaTeX is what they were suggesting should be fixed.
  - nerdponx 6y ago
    Yes, but I was responding to a comment about TeX.
- jfk13 6y ago
  While it's possible to build a lot of things on top, maybe it'd be worth making some minor updates to the core TeX engine itself, to bring it into the world of Unicode and OpenType fonts... Oh, there are a couple of such candidates, actually, built with very different philosophies: there's XeTeX, intended to graft Unicode and OpenType support onto TeX with as little disruption as possible, such that it "just works" with minimal effort; and there's LuaTeX, which "deconstructs" TeX to give you an infinitely-flexible typesetting toolkit that can do just about anything, if you can figure out how to wield it.
  - nerdponx 6y ago
    Good points.
6y ago

felixfbecker 6y ago

Imo papers of the future should be authored in semantic HTML, which can be rendered to PDF, but most importantly be reflowed and read on any digital device, also by people with disabilities, and indexed/searched. The capabilities are all there - MathML, CSS for print layout, <figure>, SVG, highlight.js, LaTeX.css, ...

isatty 6y ago
I disagree (or maybe because I don't fully understand your comment). In my (limited, I'm not a FE dev) experience, you layout an HTML page by hand but I do not want that with a research paper. I want to write my paper and let LaTeX lay it out for me. Of course I'll have to wrestle with an unruly table or figure now and again but for the most part, the layout and typesetting of the document is handled for me.
_bxg1 6y ago
It's the right platform, but it's the wrong authoring experience for people who might not be able to code at all, much less understand the nuances of CSS and how it interacts with the DOM. People using LaTeX don't want to code something, they want to write something and just have requirements that go beyond the typical use-case for "writing something". A layer is needed on top. Maybe that could take the form of a single JS and/or CSS file that gets dropped into an HTML page and then the author can create their documents with nothing but special tag attributes/classes. But likely it would need to be even more abstracted than that. Here's a good attempt that's being made. It's an extended version of Markdown that covers charts, equations, etc.: https://casual-effects.com/markdeep/ https://casual-effects.com/markdeep/
josefx 6y ago
I wrote some documentation in html and wanted to export it as pdf. The main blocker: an index with page numbers. css has them, but support for them is mostly non existant.

thorwasdfasdf 6y ago

I'd say, make sure it's backwards compatible with the old way of doing it because there's a lot of scientists who have been using it for a looong time and don't like change. You have to remember, for a scientist, a tool is just a tool, not all of them are going to invest a lot of time in learning a whole new process.

gibolt 6y ago
The old way doesn't have to change. It still exists. Make a wrapper GUI that makes it far more approachable. Syncing and other funtions could be handled by a library that works with either

CJefferson 6y ago

The "Website" point brings up an important point. LaTeX is suprisingly terrible at producing accessible documents. There are various systems and packages which make heroic efforts to make accessible output, but you have to use them from the start, as they work like you suggest -- they reimplement (or at least heavily modify) various LaTeX packages to make their outputs accessible. I've had a blind PhD student and the standard of using LaTeX to produce PDFs for academic papers has been the single biggest block in his progress.

marvy 6y ago
I assumed that if ctrl-F works, then screen readers also work. Can you share a bit of information about what tends to go wrong, and/or how document authors can fix it?
- j-pb 6y ago
  Almost no documents properly support ctrl-F. You'll get at best single word search, but you won't find coherent sentences. There are arbitrary line-breaks caused by various styling and layout packages (most columns are broken). There are even publishers that purposefully put garbage into the PDF's text layer as a braindead form of "piracy protection". PDF and TeX are broken beyond repair in terms of accessibility. The former because it allows the text and image to be split, and very few people care about text as long as it looks alright. And the latter because it's entire document representation model is based around computation. Knuth purposefully made it Turing complete in a misguided attempt at timelessness (no need for a replacement if it's infinitely extensible right?), which means there is no proper way to view a TeX document as "just text data". I've worked in a library digitising documents, and whenever we encountered PDFs we'd just OCR the damn thing. We need to throw the thing out, swallow the bitter pill of sub optimal kerning, and replace it with HTML asap.
  - solinent 6y ago
    > PDF and TeX are broken beyond repair in terms of accessibility. All that needs to be done is to have a accessible text section of the PDF--then Tex can just include the source of the document minus the Tex commands into the PDF, and PDF readers can have the screen reader work. The text could also just be stored per page, if the blind require to communicate where they found a particular item to a sighted person.

bfirsh 6y ago

The important/difficult problem of generating beautiful web pages is most of the way there already: https://github.com/arxiv-vanity/engrafo https://github.com/arxiv-vanity/engrafo There are some features in LaTeXML, which powers Engrafo, for adding some "LaTeX++" features, like embedding JS, etc. Still could do with a lot of work, though.

setgree 6y ago

> the single most cost effective way to speed up math and physics research by a philanthropist would be taking a few million dollars and hiring a team of developers to bring LaTeX into the 21st century This might be a niche opinion, but I think Rmarkdown & Jupyter get us a non-negligible portion of the way there. You can use LaTeX chunks when you need equations, markdown syntax when you need formatting (IIRC this is a pain with LaTeX), and executed code chunks to (e.g.) create and insert a graph. (One advantage of Rmarkdown is that you can have as many languages in a single document as knitr can handle, each its own chunk [0]). Neither is all the way there yet, but, as Yihui Xie puts it [1]: > I have a dream that one day all students and researchers will forget what "formatting a paper" even means. I have a dream that one day journals and grad schools no longer have style guides. I have a dream that one day no missing $ is inserted, and \hbox is never overfull. [0]: https://bookdown.org/yihui/rmarkdown/language-engines.html https://bookdown.org/yihui/rmarkdown/language-engines.html [1] https://twitter.com/xieyihui/status/1022873179532996609 https://twitter.com/xieyihui/status/1022873179532996609

j-pb 6y ago
bUt MuH TypESeTtInG! I think you're absolutely right. But people seem to value a specific flavour of aesthetics over easy to understand and reproducible research. I guess that's maybe partly caused by the publication pressure, once you start to pump out papers for pumping out papers sake, there is little incentive to go beyond the superficial.
- derefr 6y ago
  I have a dream that one day we'll get a language for typesetting that isn't a markup language. Rather, this language would be designed to come after writing, in the same way that flagship Desktop Publishing software (InDesign, Publisher) make the assumption that layout happens after writing. (Sure, you can write inside the layout software, but you're not supposed to. They're assuming e.g. a journalist–editor–layout–typesetter pipeline.) You'd have a "layout file" (written in the typesetting DSL) that has separate "non-typeset rich-text" files linked to it via file/URI references (just like Desktop Publishing apps do.) The person doing layout would never modify the linked source files; if they wanted to e.g. change certain character sequences into filligrees, that'd be something stored entirely as an annotation in the layout file, not as markup in the source file. In that world, scientists just write the "non-typeset rich text" (for which Rmarkdown / Jupyter would be a great fit); and then it's the job of science journals to produce the "layout + typesetting annotations." And hey, then journals would actually have a reason to exist! They would be to science as physical magazines are to journalism at this point: a place to find the articles you can already read online, but typeset much better.

fizixer 6y ago

No need (edit: except maybe for collaborative editing). Use TeXLive and install the updated copy every year (it has annual releases, and you simply delete the old install directory). If you're not on linux, spin a Ubuntu VM, and install TeXLive. (Strictly speaking you don't even need GUI on the machine where pdf gets compiled. I add 'scp output.pdf mydesktop:.' to my Makefile so the file is transferred to my desktop everytime I make. On the desktop the pdf is already open on the big screen and if your pdf reader is good, it'll update the opened pdf everytime the file on harddisk gets updated) I never use overleaf, and I had no problem finished up my thesis in record time. LaTeX wasn't even an issue. edit: Overleaf will be helpful for collaborative editing though. I personally would use github/gitlab instead if I ever have to do collaboration.

ergl 6y ago

Writing the paper is the easy part, after spending months researching past literature, coming up with models and hypothesis, and performing experiments. I really doubt Latex is the bottleneck for doing research.

j-pb 6y ago
You'd be surprised. I've spend hours debugging other peoples references, bibliographies and journal dictated styles. Also if you think that all of academia has MONTHS to work on papers, with our current publish or perish culture, you're wayyyyyy off.
- ergl 6y ago
  I admit I've been lucky with deadlines. The shortest span has been 3 weeks, but that was one time where most of the work was already done, and only then we looked for suitable conferences to submit the paper.
jessriedel 6y ago
Nothing in my comment suggests a bottleneck. It's just a matter of making people a few percentage point more efficient at a very low price.
- jpab 6y ago
  You referred to it as "the single most cost effective way to speed up math and physics research". I don't think it's so surprising that people are interpreting that to mean you think LaTeX is a bottleneck in the research process. I guess you intended emphasis on the "cost effective" part of the statement, but not everyone will read it like that (I didn't).
  - jessriedel 6y ago
    I mean, it wasn't merely emphasis. Likewise, "NBA" is not mere emphasis in the statement "Muggsy Bogues is the shortest person in NBA history".

truckerbill 6y ago

Using a scripting language to format text always seemed insane to me. People love LaTeX but it's nowhere near as intuitive as writing formulas down on a piece of paper. Having two panes, one for the output seems also very hacky and unintuitive. Instant feedback in one view, typing at the speed of thought should be the goal, and if it means a new paradigm I'm all for it. Imagine it was something you could actually program with; those gross inline math functions would suddenly be so much more parsable.

BlueTemplar 6y ago
LuaLaTeX ?
abdullahkhalids 6y ago
Remember that scene in the matrix when he says "I don't even see the code anymore". After a short while (about a couple of hundred pages of typing latex math for me), the code is mostly as intuitive as the visual formulas. At least for not too long equations. [1] https://www.youtube.com/watch?v=MvEXkd3O2ow https://www.youtube.com/watch?v=MvEXkd3O2ow
jessriedel 6y ago
LyX exists. It is used only by a small minority of physicists even though it's been around for 25 years.

stared 6y ago

Oh, it is a lot of effort. Just pouring a lot of money and hoping that we can get through warts (without losing the best parts) of LaTeX is unreasonable. And in fact, there are projects: - https://www.mathjax.org/ https://www.mathjax.org/, and its much faster (and less feature-complete) variant https://katex.org/ https://katex.org/ - http://www.luatex.org/ http://www.luatex.org/ - I guess this is the closest to what you mean - https://rstudio.github.io/distill/ https://rstudio.github.io/distill/ (using Knitr; it can generate HTML or PDFs) etc, etc. Even now, other libraries use BibTeX, which would be the easiest to replace in some more standardized format (JSON or YAML, so it is easy to load from web, directly).

Thrymr 6y ago

How about a modern TeX formatting layer to replace LaTeX?

andi999 6y ago

Actually the monolithic Tex has been done. Its called context. (Every couple of years I try for a few hours to install it under windows but then give up). Sometimes I manage to install it and then give up because of the introductory texts. https://en.wikipedia.org/wiki/ConTeXt https://en.wikipedia.org/wiki/ConTeXt (but I still believe there is gold at the bottom of this) Edit: woooah. I see there is now an installer for context, since april 2019! https://wiki.contextgarden.net/Installation https://wiki.contextgarden.net/Installation here we go again....

LeifCarrotson 6y ago

I used LyX to write a pretty significant paper back in 1.6, a decade later Lyx is now on 2.3; I just installed it and opened my old research paper. Obviously, I haven't done a thorough test, but from 5 minutes poking around there's a huge improvement in appearance, documentation, and usability! There's a similar story for the electronics design software package KiCAD - I played with that a bit in school as well, when it was more like a group of separate software packages duct-taped together with some import/export scripts. The enthusiasts weren't bothered by and seemed to actually enjoy the lack of polish. Recently, though, CERN has injected a bunch of money, programming time, and real-world use engineer's time into the project and it really shows. I definitely agree that the NSF (or any major LaTeX-using organization) injecting a few million dollars into LyX would be an incredible boon to human productivity. I do feel that it would be better for an organization with internal consumers of the software to undertake this effort than for an outside team to just start with a blank canvas. The latter too frequently ends up with the primary users being developers, where bug reports or feature requests will be responded to with "Just compile with -D FEATURE_ENABLE" or "For the syntax of that menu item, read the block comment and source code of the foo.cpp/bar() method" - basically, where TeX is right now. Most users of most software will install it with the Windows .exe, a few will use the .dmg/.deb/.rpm or their package manager: a vanishingly small subset are compiling it from source. And if you expect that your users should be able to read C++, your user base is going to very small.

pnw_hazor 6y ago

Interesting. When I did some internships at Los Alamos (LANL) my lab neighbor was a theoretical physicist. He used Word and referred to LaTex as the Fortran of word processors.

narrationbox 6y ago

I have an alternative view: The fastest way to speed up research would be to fund an open source Wolfram software suite with feature parity. For a lot of hard tech you need good multiphysics and CFD simulations, both of which suffer from a severe lack of open source tooling. Most of the proprietary engineering software can be cloned rapidly once a solid foundation is built. Even though Julia and Python are making progress, in terms of ergonomics and productivity it is still hard to beat the offerings by Wolfram. (If somebody is looking for a project for Mozilla's "fixing the internet" incubator, maybe give this a shot.)

abdullahkhalids 6y ago
Python ecosystem is not even close. I recently switched from Mathematica to Python, and I am going mad from juggling half a dozen packages with overlapping features. Really wish somebody would make a meta package consisting of numpy, scipy, sympy, sage that removes conflicts and overlaps, and renames all functions with the same naming scheme.

johannes1234321 6y ago

Luckily Knuth himself has this on his radar as this "earthshaking anouncment" by him shows: https://youtu.be/eKaI78K_rgA https://youtu.be/eKaI78K_rgA (Won't spoil it by giving more details)

MperorM 6y ago

I wish I could upvote this a thousand times and then some. Already in the 80's Jeff Bezos refused to use latex due to its absurd difficulty to use. I can't believe it's been 40 years and there still isn't a better tool to write papers with. It took me less than an hour to create a Google Docs template that looks roughly as good as a standard Latex document. The reason Latex looks beautiful is due to a set of wisely chosen defaults, nothing more. Yet the focus of Google Docs clearly isn't scientific papers. I don't imagine it would take more than a few passionate employees at Google to expand Docs to a powerhouse of scientific writing. Pictures of my template can be found below: https://imgur.com/a/cC0e0gG https://imgur.com/a/cC0e0gG (and a blogpost about my Latex induced misery) http://mathiasbonde.com/?p=72 http://mathiasbonde.com/?p=72

milquetoastaf 6y ago
That's a great template - mind sharing?
mNovak 6y ago
I agree beautiful defaults help LaTeX, all the more so because I always use someone else's well defined template. But, my main issues with Docs/Word is not just "look", rather: - smart figure/caption/table handling (no orphaned captions, no figures hanging out in the middle of the page) - easy, label-based cross referencing (not selecting each from a popup interface) - text-based equation input All that being said, I still have an elaborate Word template I use for proposal writing. Because often I have to pass it off to others (who may not know TeX) to contribute, and sometimes I need to quickly bastardize my nice formatting to squeeze into hard page limits. ShareLatex / Overleaf is very good, but I'd love to see slightly more ease of use as OP suggests. I'd like to see a switchable WYSIWYG / code editor, so laypeople can make simple formatting changes visually without needing to know the right commands, and it propagates correctly into the underlying LaTeX.
- MperorM 6y ago
  Totally agree with your points. Latex remains the best tool for many tasks. But none of the things you described, I think latex does particularly well either. Better than Google docs and the likes, sure, but I think it's possible to do better. I truly think Google Docs could be so much more than it currently is. They for example built a brilliant speech-to-text feature that is only available on desktop. Mobile is the one place I could imagine using such a feature. Imagine writing your documents with speech while hiking deep in the woods. Feeling the fresh air with a view of a beautiful brook as Google transcribes your thoughts. That's how I want to write! Why on earth put so much effort into such a brilliant feature, and then have it only be available to users already sitting in front of a keyboard!? Apologies for the tangent, it was where my mind wanted me to go!

epicureanideal 6y ago

Don't most journals accept MS Word submissions now? Doesn't Word provide all the formatting needed? I also prefer Latex but I wonder if we're talking about solving a problem nobody has?

est31 6y ago

You should check out Tectonic. https://tectonic-typesetting.github.io https://tectonic-typesetting.github.io It fixes some of the issues that LaTex has.

blackrock 6y ago

This sounds like an opportunity for someone to create a python module to generate scientific documents. You’ll get all the benefits of python, with the added benefits of generating formatted text. Call it pyTeX. LOL..

teleforce 6y ago

I am surprised nobody here mentioned TeXmacs. It's nothing to do with TeX or Emacs, but it's inspired by them. Basically the concept is structural graph based document editing software similar to Tioga/Cedar by Xerox back in 1980s but unfortunately the research and improvement for better tools for editing document seems to stagnate a lot after 90s. I think the idea of having an easy to use WYSIWYG editor for scientific document editing need to be popular now to significantly increase scientist productivity. Remember that web was invented to increase scientist productivity and the rest is history. On a related note, personally I think Overleaf or Google Docs is missing the point or the pain points. Even though both of them enabled offline editing but they are really a cloud first applications. The better approach is to have something like TeXmacs (native desktop first application) and make a seamless integration with synchronization and versioning to the cloud. With a proper use of synchronization and versioning technology like rsync and Git, together with seamless connectivity tools and protocols for examples Wireguard VPN, WebDAV or even the latest SMB over QUIC approach, I really think this is very feasible. The CONCEPT is similar to the useful and successful Watcom product or now SQL Anywhere database editing application.