19 ms·

Weird Lexical Syntax

433 points by jart 2y ago

llm_trw 2y ago
I've done a fair bit of forth and I've not seen c" used. The usual string printing operator is ." .
- mananaysiempre 2y ago
  Counted (“Pascal”) strings are rare nowadays so C" is not often used. Its addr len equivalent is S" and that one is fairly common in string manipulation code.
- kragen 2y ago
  Right, c" is for when you want to pass a literal string to some other word, not print it. But I agree that it's not very common, because you normally use s" for that, which leaves the address and length on the stack, while c" leaves just an address on the stack, pointing to a one-byte count field followed by the bytes. I think adding c" in Forth-83 (and renaming " to s") was a mistake, and it would have been better to deprecate the standard words that expect or produce such counted strings, other than count itself. See https://forth-standard.org/standard/alpha https://forth-standard.org/standard/alpha, https://forth-standard.org/standard/core/Cq https://forth-standard.org/standard/core/Cq, https://forth-standard.org/standard/core/COUNT https://forth-standard.org/standard/core/COUNT, and https://forth-standard.org/standard/core/Sq https://forth-standard.org/standard/core/Sq. You can easily add new string and comment syntaxes to Forth, though. For example, you can add BCPL-style // comments to end of line with this line of code in, I believe, all standard Forths, though I've only tested it in GForth: : // 10 word drop ; immediate Getting it to work in block files requires more work but is still only a few lines of code. The standard word \ does this, and see \ decompiles the GForth implementation as : \ blk @ IF >in @ c/l / 1+ c/l * >in ! EXIT THEN source >in ! drop ; immediate This kind of thing was commonly done for text editor commands, for example; you might define i as a word that reads text until the end of the line and inserts it at the current position in the editor, rather than discarding it like my // above. Among other things, the screen editor in F83 does exactly that. So, as with Perl, PostScript, TeX, m4, and Lisps that support readmacros, you can't lex Forth without executing it.
skrebbel 2y ago
This was a delightful read, thanks!
croisillon 2y ago
Glad to see confirmed that PHP is the most non weird programming language ;)
- deleted 2y ago
  [deleted]
- rererereferred 2y ago
  I recently learned php's heredoc can have space before it and it will remove those spaces from the lines in the string: $a = <<<EOL This is not indented but this has 4 spaces of indentation EOL; But the spaces have to match, if any line has less spaces than the EOL it gives an error.
- alganet 2y ago
  There are two types of languages: the ones full of quirks and the ones no one uses.
skitter 2y ago
Another syntax oddity (not mentioned here) that breaks most highlighters: In Java, unicode escapes can be anywhere, not just in strings. For example, the following is a valid class: class Foo\u007b} and this assert will not trigger: assert // String literals can have unicode escapes like \u000A! "Hello World".equals("\u00E4");
- ivanjermakov 2y ago
  I have never seen this in Java! Is there any use cases where it could be useful?
  - susam 2y ago
    I don't know about usefulness but it does let us write identifiers using Unicode characters. For example: public class Foo { public static void main(String[] args) { double \u03c0 = 3.14159265; System.out.println("\u03c0 = " + \u03c0); } } Output: $ javac Foo.java && java Foo π = 3.14159265 Of course, nowadays we can simply write this with any decent editor: public class Foo { public static void main(String[] args) { double π = 3.14159265; System.out.println("π = " + π); } } Support for Unicode escape sequences is a result of how the Java Language Specification (JLS) defines InputCharacter. Quoting from Section 3.4 of JLS <https://docs.oracle.com/javase/specs/jls/se23/jls23.pdf https://docs.oracle.com/javase/specs/jls/se23/jls23.pdf>: InputCharacter: UnicodeInputCharacter but not CR or LF UnicodeInputCharacter is defined as the following in section 3.3: UnicodeInputCharacter: UnicodeEscape RawInputCharacter UnicodeEscape: \ UnicodeMarker HexDigit HexDigit HexDigit HexDigit UnicodeMarker: u {u} HexDigit: (one of) 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F RawInputCharacter: any Unicode character As a result the lexical analyser honours Unicode escape sequences absolutely anywhere in the program text. For example, this is a valid Java program: public class Bar { public static void \u006d\u0061\u0069\u006e(String[] args) { System.out.println("hello, world"); } } Here is the output: $ javac Bar.java && java Bar hello, world However, this is an incorrect Java program: public class Baz { // This comment contains \u6d. public static void main(String[] args) { System.out.println("hello, world"); } } Here is the error: $ javac Baz.java Baz.java:2: error: illegal unicode escape // This comment contains \u6d. ^ 1 error Yes, this is an error even if the illegal Unicode escape sequence occurs in a comment!
mcphage 2y ago
At one point there was an open source project to formally specify Ruby, but I don’t know if it’s still alive: https://github.com/ruby/spec https://github.com/ruby/spec Hmm, it seems to be alive, but based more on behavior than syntax.
keybored 2y ago
Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff I wonder if makes-sense-to-computer-scientist is a good goal. They invented the problem of using a uniform delimiter for strings and then had to solve their own problem. Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. What do C-like programming languages use single quotes for? To quote characters. Why do you need to quote characters? I’ve never seen a literal character which needed an "end character" marker. Raw strings would still be useful but you wouldn’t need raw strings just to do a very basic thing like make a string which has typewriter quotes in it. Of course this was for C-like languages. Don’t even get me started on shell and related languages where basically everything is a string and you have to make a single-quote/double-quote battle plan before doing anything slightly nested. [1] https://lore.kernel.org/git/vpq3808p40o.fsf@anie.imag.fr/ https://lore.kernel.org/git/vpq3808p40o.fsf@anie.imag.fr/ [2] Notwithstanding us Europeans that use a dead-key keyboard layout where you have to type twice to get one measly backtick (not that I use those)
- deleted 2y ago
  [deleted]
- pwdisswordfishz 2y ago
  > The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#` https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---cleanupltmodegt https://git-scm.com/docs/git-commit#Documentation/git-commit...
  - keybored 2y ago
    See "commit cleanup". There’s surprising layers to this. That the reporter in that thread says that git-commit will “happily” accept `#` in commit messages is half-true: it will accept it if you don’t edit the message since the `default` cleanup (that you linked to) will not remove comments if the message is given through things like `-m` and not an editing session. So `git commit -m'#something' is fine. But then try to do rebase and cherry-pick and whatever else later, maybe get a merge commit message with a commented "conflicted" files. Well it can get confusing.
yen223 2y ago
select'select'select is a perfectly valid SQL query, at least for Postgres. Languages' approach to whitespace between tokens is all over the place
notsylver 2y ago
As soon as I saw this was part of llamafile I was hoping that it would be used to limit LLM output to always be "valid" code as soon as it saw the backticks, but I suppose most LLMs don't have problems with that anyway. And I'm not sure you'd want something like that automatically forcing valid code anyway
- dilap 2y ago
  llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over so in theory you could notice "```python" or whatever and then start restricting to valid python code. (in least in theory, not sure how feasible/possible it would be in practice w/ their grammar format.) for code i'm not sure how useful it would be since likely any model that is giving you working code wouldn't be struggling w/ syntax errors anyway? but i have had success experimentally using the feature to drive fiction content for a game from a smaller llm to be in a very specific format.
  - notsylver 2y ago
    yeah, ive used llama.cpp grammars before, which is why i was thinking about it. i just think it'd be cool for llamafile to do basically that, but with included defaults so you could eg, require JSON output. it could be cool for prototyping or something. but i dont think that would be too useful anyway, most of the time i think you would want to restrict it to a specific schema, so i can only see it being useful for something like a tiny local LLM for code completion, but that would just encourage valid-looking but incorrect code. i think i just like the idea of restricting LLM output, it has a lot of interesting use cases
    - dilap 2y ago
      gotchya. i do think that is a cool idea actually -- LLMs tiny enough to do useful things with formally structured output but not big enough to nail the structure ~100% is probably not an empty set.
pwdisswordfishz 2y ago
> Of all the languages, I've saved the best for last, which is Ruby. Now here's a language whose syntax evades all attempts at understanding. TeX with its arbitrarily reprogrammable lexer: how adorable
- fanf2 2y ago
  Lisp reader macros allow you to program its lexer too.
  - skydhash 2y ago
    You can basically define a new language with a few lines of code in Racket.
pansa2 2y ago
> TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings. So to highlight a string, one must count curly brackets and maintain a stack of parser states. Presumably this is also true in Python - IIRC the brace-delimited fields within f-strings may contain arbitrary expressions. More generally, this must mean that the lexical grammar of those languages isn't regular. "Maintaining a stack" isn't part of a finite-state machine for a regular grammar - instead we're in the realm of pushdown automata and context-free grammars. Is it even possible to support generalized string interpolation within a strictly regular lexical grammar?
- deleted 2y ago
  [deleted]
- aphantastic 2y ago
  > Is it even possible to support generalized string interpolation within a strictly regular lexical grammar? Almost certainly not, a fun exercise is to attempt to devise a Pumping tactic for your proposed language. If it doesn’t exist, it’s not regular. https://en.m.wikipedia.org/wiki/Pumping_lemma_for_regular_languages https://en.m.wikipedia.org/wiki/Pumping_lemma_for_regular_la...
- fanf2 2y ago
  Complicated interpolation can be lexed as a regular language if you treat strings as three separate lexical things, eg in JavaScript template literals there are, `stuff${ }stuff${ }stuff` so the ${ and } are extra closing and opening string delimiters, leaving the nesting to be handled by the parser. You need a lexer hack so that the lexer does not treat } as the start of a string literal, except when the parser is inside an interpolation but all nested {} have been closed.
irdc 2y ago
I’d be interested to see a re-usable implementation of joe's[0] syntax highlighting.[1] The format is powerful enough to allow for the proper highlighting of Python f-strings.[2] 0. https://joe-editor.sf.net/ https://joe-editor.sf.net/ 1. https://github.com/cmur2/joe-syntax/blob/joe-4.4/misc/HowItWorks.md https://github.com/cmur2/joe-syntax/blob/joe-4.4/misc/HowItW... 2. https://gist.github.com/irdc/6188f11b1e699d615ce2520f03f1d0da https://gist.github.com/irdc/6188f11b1e699d615ce2520f03f1d0d...
- pama 2y ago
  Interestingly, python f-strings changed their syntax at version 3.12, so highlighting should depend on the version.
  - irdc 2y ago
    It’s just that nesting them arbitrarily is now allowed, right? That shouldn’t matter much for a mere syntax highlighter then. And one could even argue that code that relies on this too much is not really for human consumption.
    - pansa2 2y ago
      Also, you can now use the same quote character that encloses an f-string within the {} expressions. That could make them harder to tokenize, because it makes it harder to recognise the end of the string.
- akira2501 2y ago
  I've actually made several lexers and parsers based on the joe DFA style of parsing. The state and transition syntax was something that I always understood much more easily than the standard tools. The downside is your rulesets tend to get more verbose and are a little bit harder to structure than they might ideally be in other languages more suited towards the purpose, but I actually think that's an advantage, as it's much easier to reason about every production rule when looking at the code.
rererereferred 2y ago
In the C# multiquoted strings, how does it know this: Console.WriteLine(""""""); Console.WriteLine(""""""); Are 2 triplequoted empty strings and not one "\nConsole.WriteLine(" sixtuplequoted string?
- ygra 2y ago
  The former, I'd say. https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/strings/#raw-string-literals https://learn.microsoft.com/en-us/dotnet/csharp/programming-... For a multi-line string the quotes have to be on their own line.
- Joker_vD 2y ago
  If the opening quotes are followed by anything that is not a whitespace before the next new-line (or EOF), then it's a single-line string. I imagine implementing those things took several iterations :)
- yen223 2y ago
  It's a syntax error! Unterminated raw string literal. https://replit.com/@Wei-YenYen/DistantAdmirableCareware#main.cs https://replit.com/@Wei-YenYen/DistantAdmirableCareware#main...
  - Joker_vD 2y ago
    Ah, so there is no backtracking in lexer for this case. Makes sense.
ygra 2y ago
As for C#'s triple-quoted strings, they actually came from Java before and C# ended up adopting the same or almost the same semantics. Including stripping leading whitespace.
pdw 2y ago
Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $(echo "$x+$y" | bc)" Make :) echo "$(x) plus $(y) equals $(shell echo "$x+$y" | bc)" > Tcl Tcl is funny because comments are only recognized in code, and since it's a homoiconic, it's very hard to distinguish code and data. { } are just funny string delimiters. E.g.: xyzzy {#hello world} Is xyzzy a command that takes a code block or a string? There's no way to tell. (Yes, that means that the Tcl tokenizer/parser cannot discard comments: only at evaluation time it's possible to tell if something is a comment or not.) > SQL PostgreSQL has the very convenient dollar-quoted strings: https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-DOLLAR-QUOTING https://www.postgresql.org/docs/current/sql-syntax-lexical.h... E.g. these are equivalent: 'Dianne''s horse' $$Dianne's horse$$ $SomeTag$Dianne's horse$SomeTag$
- deleted 2y ago
  [deleted]
- autarch 2y ago
  Perl lets you do this too: my $foo = 5; my $bar = 'x'; my $quux = "I have $foo $bar\'s: @{[$bar x $foo]}"; print "$quux\n"; This prints out: I have 5 x's: xxxxx The "@{[...]}" syntax is abusing Perl's ability to interpolate an _array_ as well as a scalar. The inner "[...]" creates an array reference and the outer "@{...}" dereferences it. For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference.
  - Izkata 2y ago
    > For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference. ...because it's an array value? Aside from how the languages handle references, how is that part any different from, for example, this in python: >>> [5 * 'x'] ['xxxxx'] You can put (almost) anything there, as long as it's an expression that evaluates to a value. The resulting value is what goes into the array.
__MatrixMan__ 2y ago
This was a fun read, but it left me a bit more sympathetic to the lisp perspective, which (if I've understood it) is that syntax, being not an especially important part of a language, is more of a hurdle than a help, and should be as simple and uniform as possible so we can focus on other things. Which is sort of ironic because learning how to do structural editing on lisps has absolutely been more hurdle than help so far, but I'm sure it'll pay off eventually.
- mqus 2y ago
  Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the semantics (which might be undiscovered until runtime).
  - drewr 2y ago
    I don't understand your distinction between syntax and semantics. If the semantics are complex, wouldn't that mean the syntax is thus complex?
    - SuperCuber 2y ago
      lisp's syntax is simple - its just parenthesis to define a list, first element of a list is executed as a function. but for example a language like C has many different syntaxes for different operations, like function declaration or variable or array syntax, or if/switch-case etc etc. so to know C syntax you need to learn all these different ways to do different things, but in lisp you just need to know how to match parenthesis. But of course you still want to declare variables, or have if/else and switch case. So you instead need to learn the builtin macros (what GP means by semantics) and their "syntax" that is technically not part of the language's syntax but actually is since you still need all those operations enough that they are included in the standard library and defining your own is frowned upon.
kazinator 2y ago
I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested in certain ways. Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trailing material from a region. Matches and regions can declare that they are contained. In that case they are not active unless they occur in a containing region. Contained matches declare which regions contain them. Regions declare which other regions they contain. That's the basic semantic architecture; there are bells and whistles in the system due to situations that arise. I don't think even Justine could develop that in an interview, other than as an overnight take home.
- kazinator 2y ago
  Here is an example of something hard to handle: TXR language with embedded TXR Lisp. This is the "genman" script which takes the raw output of a manpage to HTML converter, and massages it to form the HTML version of the TXR manual: https://www.kylheku.com/cgit/txr/tree/genman.txr https://www.kylheku.com/cgit/txr/tree/genman.txr Everything that is white (not colored) is literal template material. Lisp code is embedded in directives, like @(do ...). In this scheme, TXR keywords appear purple, TXR Lisp ones green. They can be the same; see the (and ...) in line 149, versus numerous occurrences of @(and). Quasistrings contain nested syntax: see 130 where `<a href ..> ... </a>` contains an embedded (if ...). That could itself contain a quasistring with more embedded code. TXR's txr.vim" and tl.vim* syntax definition files are both generated by this: https://www.kylheku.com/cgit/txr/tree/genvim.txr https://www.kylheku.com/cgit/txr/tree/genvim.txr
- saghm 2y ago
  Naively, I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then iterate over the tokens and update the color of a token based on the type of node (and maybe just tracking a diff to avoid needing to recolor things that haven't changed). I'm guessing that if this isn't done, it's for efficiency reasons (e.g. due to requiring parsing the whole file to highlight rather than just the part currently visible on the screen)?
susam 2y ago
> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment. And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program: (* (* Nested (**) *) comment *) val _ = print "hello, world\n" Here is the output: $ sml < hello.sml Standard ML of New Jersey (64-bit) v110.99.5 [built: Thu Mar 14 17:56:03 2024] - = hello, world $ mlton hello.sml && ./hello hello, world Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested comments were never part of the language.
- dahart 2y ago
  There are 3 things I find funny about that comment: ML didn’t have single-line comments, so same level of surprising limitation. I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. And what bearing does the comment syntax have on the expressiveness of a language? I would argue absolutely none at all, by definition. :P
  - susam 2y ago
    > ML didn’t have single-line comments, so same level of surprising limitation. It is not quite clear to me why the lack of single-line comments is such a surprising limitation. After all, a single-line block comment can easily serve as a substitute. However, there is no straightforward workaround for the lack of nested block comments. > I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. I was thinking of Fortran in this context. For instance, Fortran 77 lacked function pointers and offered a limited set of control flow structures, along with cumbersome support for recursion. I know Fortran, with its native support for multidimensional arrays, excelled in numerical and scientific computing but C quickly became the preferred language for general purpose computing. While very few today would consider C a pinnacle of expressiveness, when I was learning C, the landscape of mainstream programming languages was much more restricted. In fact, the preface to the first edition of K&R notes the following: "In our experience, C has proven to be a pleasant, expressive and versatile language for a wide variety of programs." C, Pascal, etc. stood out as some of the few mainstream programming languages that offered a reasonable level of expressiveness. Of course, Lisp was exceptionally expressive in its own right, but it wasn't always the best fit for certain applications or environments. > And what bearing does the comment syntax have on the expressiveness of a language? Nothing at all. I agree. The expressiveness of C comes from its grammar, which the language parser handles. Support for nested comments, in the context of C, is a concern for the lexer, so indeed one does not directly influence the other. However, it is still curious that a language with such a sophisticated grammar and parser could not allocate a bit of its complexity budget to support nested comments in its lexer. This is a trivial matter, I know, but I still couldn't help but wonder about it.
lupire 2y ago
> You'll notice its hash function only needs to consider a single character in in a string. That's what makes it perfect, Is that a joke? https://en.m.wikipedia.org/wiki/Perfect_hash_function https://en.m.wikipedia.org/wiki/Perfect_hash_function
- jaen 2y ago
  No. Taking the value of a single character is a correct perfect hash function, assuming there exists a position for the input string set where all characters differ.
- anitil 2y ago
  It just so happens that a single character was all that's needed in that case. In the general case it wouldn't be
playingalong 2y ago
Nice read. I guess the article could be called Falsehoods Programmers Assume of Programming Language Syntaxes.
TomatoCo 2y ago
I think my favorite C trigraph was something like do_action() ??!??! handle_error() It almost looks like special error handling syntax but still remains satisfying once you realize it's an || logical-or statement and it's using short circuiting rules to execute handle error if the action returns a non-zero value.
- wslh 2y ago
  Did you choose the legacy C trigraphs over || for aesthetic purposes?
  - wslh 2y ago
    Could you review my comment on HN? Please educate me if there is something I haven’t understood, rather than downvoting my question.
    - samatman 2y ago
      The grandparent post is specifically about trigraphs. Saying something about trigraphs was the end-in-itself, trigraphs were chosen to illustrate something about trigraphs. So your question made no sense. Hope that helps.
      - Izkata 2y ago
        Maybe the confusion was the other way, more like "why is that funny/interesting?" An attempt to answer that: In English, mixing ?! at the end of a question is a way of indicating bewilderment. Like "What was that?!"
      - wslh 2y ago
        My question was precisely about why the user like trigraphs over using just || on this case. It is a very clear question and makes all the sense.
        teo_zero 2y ago
IshKebab 2y ago
I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here. Is this a "no third party dependencies" thing?
- jart 2y ago
  I don't want to require everyone who builds llamafile from source need to install rust. I don't even require that people install the gperf command, since I can build gperf as a 700kb actually portable executable and vendor it in the repo. Tree sitter I'd imagine does a really great highly precise job with the languages it supports. However it appears to support fewer of them than I am currently. I'm taking a breadth first approach to syntax highlighting, due to the enormity of languages LLMs understand.
  - IshKebab 2y ago
    I think the Rust component of tree-sitter-highlight is actually pretty small (Tree Sitter generates C for the actual parser). But fair enough - fewer dependencies is always nice, especially in C++ (which doesn't have a modern package manager) and in ML where an enormous janky Python installation is apparently a perfectly normal thing to require.
    - mdaniel 2y ago
      I somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth 1: https://docs.conan.io/2/introduction.html https://docs.conan.io/2/introduction.html
      - IshKebab 2y ago
        No, if anything vcpkg is "the C++ package manager", but it's nowhere near pervasive and easy-to-use enough to come close to even Pip. It's leagues away from Cargo, Go, and other actually good PL package managers.
jim_lawless 2y ago
Forth has a default syntax, but Forth code can execute during the compilation process allowing it to accept/compile custom syntaxes.
SonOfLilit 2y ago
Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 https://perlmonks.org/?node_id=663393 C++ templates have a similar issue, I think.
- fanf2 2y ago
  I think possibly the most hilariously complicated instance of this is in perl’s tokenizer, toke.c (which starts with a Tolkien quote, 'It all comes from here, the stench and the peril.' — Frodo). There’s a function called intuit_more which works out if $var[stuff] inside a regex is a variable interpolation followed by a character class, or an array element interpolation. Its result can depend on whether something in the stuff has been declared as a variable or not. But even if you ignore the undecidability, the rest is still ridiculously complicated. https://github.com/Perl/perl5/blob/blead/toke.c#L4502 https://github.com/Perl/perl5/blob/blead/toke.c#L4502
  - ufo 2y ago
    Wow. I wonder how that function came to be in the first place. Surely it couldn't have started out that complicated?
- _cbsz 2y ago
  > C++ templates have a similar issue TIL! I went and dug up a citation: https://blog.reverberate.org/2013/08/parsing-c-is-literally-undecidable.html https://blog.reverberate.org/2013/08/parsing-c-is-literally-...
- layer8 2y ago
  How could a search for a Collatz counterexample possibly terminate? ;)
- chubot 2y ago
  Yup, bash and GNU Make have the same issue as Perl does, and I mention the C++ issue here too: Parsing Bash is Undecidable - https://www.oilshell.org/blog/2016/10/20.html https://www.oilshell.org/blog/2016/10/20.html I remember a talk from Larry Wall on Perl 6 (now Raku), where he says this type of thing is a mistake. Raku can be statically parsed, as far as I know.
petesergeant 2y ago
> Perl also has this goofy convention for writing man pages in your source code The world corpus of software would be much better documented if everywhere else had stolen this from Perl. Inline POD is great.
- kragen 2y ago
  Perl and Python stole it from Emacs Lisp, though Perl took it further. I'm not sure where Java stole it from, but nowadays Doxygen is pretty common for C code. Unfortunately this results in people thinking that Javadoc and Doxygen are substitutes for actual documentation like the Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequate for a reference manual.
  - mdaniel 2y ago
    > Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequate for a reference manual. Well, they're not doing themselves any favors by just willy nilly mixing C with "user-facing" defuns <https://emba.gnu.org/emacs/emacs/-/blob/ed1d691184df4b50da6b8e1a207e9ccd88aa9ffb/src/process.c#L1003-1004 https://emba.gnu.org/emacs/emacs/-/blob/ed1d691184df4b50da6b...>. I was curious if they could benefit from "literate programming" since OrgMode is the bee's knees but not with that style coding they can't
    - kragen 2y ago
      I didn't mean that specifically the Emacs source code was not organized in the right way for a reference manual. I meant that C and Java source code in general isn't. And C++, which is actually where people use Doxygen more. The Python standard library manual is also exemplary, and also necessarily organized differently from the source code.
      - mdaniel 2y ago
        > The Python standard library manual is also exemplary Maybe parts of it are, but as a concrete example https://docs.python.org/3/library/re.html#re.match https://docs.python.org/3/library/re.html#re.match is just some YOLO about what, specifically, is the first argument to re.match: string, or compiled expression? Well, it's both! Huzzah! I guess they get points for consistency because the first argument to re.compile is also "both" But, any idea what type re.compile returns? cause https://docs.python.org/3/library/re.html#re.compile https://docs.python.org/3/library/re.html#re.compile is all "don't you worry about it" versus its re.match friend who goes out of their way to state that it is an re.Match object Would it have been so hard to actually state it, versus requiring someone to invoke type() to get <class 're.Pattern'>?
metadat 2y ago
> The languages I decided to support are Ada, Assembly, BASIC, C, C#, C++, COBOL, CSS, D, FORTH, FORTRAN, Go, Haskell, HTML, Java, JavaScript, Julia, JSON, Kotlin, ld, LISP, Lua, m4, Make, Markdown, MATLAB, Pascal, Perl, PHP, Python, R, Ruby, Rust, Scala, Shell, SQL, Swift, Tcl, TeX, TXT, TypeScript, and Zig. A few (admittedly silly) questions about the list: 1. Why no Erlang, Elixir, or Crystal? Erlang appears to be just at the author's boundary at #47 on the TIOBE index. https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ 2. What is "Shell"? Sh, Bash, Zsh, Windows Cmd, PowerShell..? 3. Perl but no Awk? Curious why, because Awk is a similar but comparatively trivial language. Widely used, too. To be fair, Awk, Erlang, and Elixir rank low on popularity. Yet m4, Tcl, TeX, and Zig aren't registered in the top 50 at all. What's the methodology / criteria? Only things the author is already familiar with? Still a fun article.
- Yasuraka 2y ago
  Tiobes's index is quite literally worthless, especially with regards to its stated purpose, let alone as a general point of orientation. I'd wish that purple would stop lending it any credibility.
- Kwpolska 2y ago
  "Shell" in the context of a syntax highlighting language picker almost always means a Unixy shell, most likely something along the lines of Bash.
dakiol 2y ago
Wouldn’t be possible to let the LLM do the highlighting? Instead of returning code in plain text, it could return code within html with the appropriate tags. Maybe it’s harder than it sounds… but if it’s just for highlighting the code the LLM returns, I wouldn’t mind the highlighting not being 100% accurate.
- trashburger 2y ago
  Would be much slower and eat up precious context window.
layer8 2y ago
The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack https://en.wikipedia.org/wiki/Lexer_hack Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions. More generally, however, I find it useful to highlight (say) types differently from variables or functions, which in some (most?) popular languages requires full parsing and symbol table information. Some IDEs therefore implement two levels of syntax highlighting, a basic one that only requires lexical information, and an extended one that kicks in when full grammar and type information becomes available.
- legobmw99 2y ago
  I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked
  - layer8 2y ago
    Given all the things that were new to the author in the article, I wouldn’t be shocked at all. There’s just a huge number of things to know, or to have come across.
    - jraph 2y ago
      Justine is proficient in C, she is the author of a libc (cosmopolitan) among other things, like Actually Portable Executables [1]. I would expect her to know C quite well, and that's probably an understatement. [1] https://justine.lol/ape.html https://justine.lol/ape.html
  - quietbritishjim 2y ago
    I think you're thinking of something different to the issue in the parent comment. The most vexing parse is, as the name suggests, a problem at the parsing stage rather than the earlier lexing phase. Unlike the referenced lexing problem, it does't require any hack for compilers to deal with it. That's because it's not really a problem for the compiler; it's humans that find it surprising.
murkt 2y ago
Author hasn’t tried to highlight TeX. Which is good for their mental health, I suppose, as it’s generally impossible to fully highlight TeX without interpreting it. Even parsing is not enough, as it’s possible to redefine what each character does. You can make it do things like “and now K means { and C means }”. Yes, you can find papers on arXiv that use this god-forsaken feature.
- jart 2y ago
  I wrote https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafile/highlight_tex.cpp https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... and it does a reasonable job highlighting without breaking for all the .tex files I could find on my hard drive. My goal is to hopefully cover 99.9% of real world usage, since that'll likely cover everything an LLM might output. Esoteric syntax also usually isn't a problem, so long as it doesn't cause strings and comments to extend forever, eclipsing the rest of the source code in a file.
  - murkt 2y ago
    Yes, when goal isn’t to support 100% of all the weird stuff, then it’s orders of magnitude easier!
- nathell 2y ago
  Same with Common Lisp (you can redefine the read table), although that’s likely abused less often on arXiv.
- bobbylarrybobby 2y ago
  I couldn't believe it when I learned that \makeatletter does not “make (something) at a letter (character)” but rather “treats the '@' character as a letter when parsing”.
xonix 2y ago
No AWK?
sundarurfriend 2y ago
The final line number count is missing Julia. Based on the file in the repo, it would be at the bottom of the first column: between ld and R. Among the niceties listed here, the one I'd wish for Julia to have would be C#'s "However many quotes you put on the lefthand side, that's what'll be used to terminate the string at the other end". Documentation that talks about quoting would be so much easier to read (in source form) with something like that.
- sundarurfriend 2y ago
  One nicety that Julia does have that I didn't know about (or had forgotten) is nested multi-line comments. #= this one has a #= nested comment =# inside of it and that works fine! =#
nusaru 2y ago
> Ruby is the union of all earlier languages, and it's not even formally documented. It's documented, but you need $250 to spare: https://www.iso.org/standard/59579.html https://www.iso.org/standard/59579.html
- mdaniel 2y ago
  Well, according to (ahem) a copy that I found, it only goes up to MRI 1.9 and goes out of its way to say "welp, the world is changing, so we're just going to punt until Ruby stabilizes" which is damn cheating for a standard IMHO Also, while doing some digging I found there actually are a number of the standards that are legitimately publicly available https://standards.iso.org/ittf/PubliclyAvailableStandards/index.html https://standards.iso.org/ittf/PubliclyAvailableStandards/in...
- vidarh 2y ago
  ISO Ruby is a tiny, dated subset of Ruby. I doubt you'll find much Ruby that conforms to it. The Ruby everyone uses is much better defined by RubySpec etc. via test cases, but that's not complete either.
tomcam 2y ago
> If you ever want to confuse your coworkers, then one great way to abuse this syntax is by replacing the heredoc marker with an empty string Maybe I am in favor of the death penalty after all
petters 2y ago
> I'm not sure who wants to be able to syntax highlight C at 35 MB per second, but I am now able to do so Fast, but tcc *compiles* C to binary code at 29 MB/s on a really old computer: https://bellard.org/tcc/#speed https://bellard.org/tcc/#speed Should be possible to go much faster but probably not needed
- BiteCode_dev 2y ago
  Justine vs Bellard, that's a nice setup.
transfire 2y ago
Impressive work! I am surprised Smalltalk and Prolog are in there though.
fahrnfahrnfahrn 2y ago
While developing the syntax for a programming language in the early 80s, I discovered that allowing spaces in identifiers was unambiguous, e.g., upper left corner = scale factor * old upper left corner.
KolmogorovComp 2y ago
The article doesn’t address why they chose not use treesiter and instead roll their own syntax highlight system. Edit: same question with answer here https://news.ycombinator.com/item?id=42026554 https://news.ycombinator.com/item?id=42026554

433 points by jart 2y ago

llm_trw 2y ago

I've done a fair bit of forth and I've not seen c" used. The usual string printing operator is ." .

mananaysiempre 2y ago
Counted (“Pascal”) strings are rare nowadays so C" is not often used. Its addr len equivalent is S" and that one is fairly common in string manipulation code.
kragen 2y ago
Right, c" is for when you want to pass a literal string to some other word, not print it. But I agree that it's not very common, because you normally use s" for that, which leaves the address and length on the stack, while c" leaves just an address on the stack, pointing to a one-byte count field followed by the bytes. I think adding c" in Forth-83 (and renaming " to s") was a mistake, and it would have been better to deprecate the standard words that expect or produce such counted strings, other than count itself. See https://forth-standard.org/standard/alpha https://forth-standard.org/standard/alpha, https://forth-standard.org/standard/core/Cq https://forth-standard.org/standard/core/Cq, https://forth-standard.org/standard/core/COUNT https://forth-standard.org/standard/core/COUNT, and https://forth-standard.org/standard/core/Sq https://forth-standard.org/standard/core/Sq. You can easily add new string and comment syntaxes to Forth, though. For example, you can add BCPL-style // comments to end of line with this line of code in, I believe, all standard Forths, though I've only tested it in GForth: : // 10 word drop ; immediate Getting it to work in block files requires more work but is still only a few lines of code. The standard word \ does this, and see \ decompiles the GForth implementation as : \ blk @ IF >in @ c/l / 1+ c/l * >in ! EXIT THEN source >in ! drop ; immediate This kind of thing was commonly done for text editor commands, for example; you might define i as a word that reads text until the end of the line and inserts it at the current position in the editor, rather than discarding it like my // above. Among other things, the screen editor in F83 does exactly that. So, as with Perl, PostScript, TeX, m4, and Lisps that support readmacros, you can't lex Forth without executing it.

skrebbel 2y ago

This was a delightful read, thanks!

croisillon 2y ago

Glad to see confirmed that PHP is the most non weird programming language ;)

deleted 2y ago
[deleted]
rererereferred 2y ago
I recently learned php's heredoc can have space before it and it will remove those spaces from the lines in the string: $a = <<<EOL This is not indented but this has 4 spaces of indentation EOL; But the spaces have to match, if any line has less spaces than the EOL it gives an error.
alganet 2y ago
There are two types of languages: the ones full of quirks and the ones no one uses.

skitter 2y ago

Another syntax oddity (not mentioned here) that breaks most highlighters: In Java, unicode escapes can be anywhere, not just in strings. For example, the following is a valid class: class Foo\u007b} and this assert will not trigger: assert // String literals can have unicode escapes like \u000A! "Hello World".equals("\u00E4");

ivanjermakov 2y ago
I have never seen this in Java! Is there any use cases where it could be useful?
- susam 2y ago
  I don't know about usefulness but it does let us write identifiers using Unicode characters. For example: public class Foo { public static void main(String[] args) { double \u03c0 = 3.14159265; System.out.println("\u03c0 = " + \u03c0); } } Output: $ javac Foo.java && java Foo π = 3.14159265 Of course, nowadays we can simply write this with any decent editor: public class Foo { public static void main(String[] args) { double π = 3.14159265; System.out.println("π = " + π); } } Support for Unicode escape sequences is a result of how the Java Language Specification (JLS) defines InputCharacter. Quoting from Section 3.4 of JLS <https://docs.oracle.com/javase/specs/jls/se23/jls23.pdf https://docs.oracle.com/javase/specs/jls/se23/jls23.pdf>: InputCharacter: UnicodeInputCharacter but not CR or LF UnicodeInputCharacter is defined as the following in section 3.3: UnicodeInputCharacter: UnicodeEscape RawInputCharacter UnicodeEscape: \ UnicodeMarker HexDigit HexDigit HexDigit HexDigit UnicodeMarker: u {u} HexDigit: (one of) 0 1 2 3 4 5 6 7 8 9 a b c d e f A B C D E F RawInputCharacter: any Unicode character As a result the lexical analyser honours Unicode escape sequences absolutely anywhere in the program text. For example, this is a valid Java program: public class Bar { public static void \u006d\u0061\u0069\u006e(String[] args) { System.out.println("hello, world"); } } Here is the output: $ javac Bar.java && java Bar hello, world However, this is an incorrect Java program: public class Baz { // This comment contains \u6d. public static void main(String[] args) { System.out.println("hello, world"); } } Here is the error: $ javac Baz.java Baz.java:2: error: illegal unicode escape // This comment contains \u6d. ^ 1 error Yes, this is an error even if the illegal Unicode escape sequence occurs in a comment!

mcphage 2y ago

At one point there was an open source project to formally specify Ruby, but I don’t know if it’s still alive: https://github.com/ruby/spec https://github.com/ruby/spec Hmm, it seems to be alive, but based more on behavior than syntax.

keybored 2y ago

Meanwhile NeoVim doesn’t syntax highlight my commit message properly if I have messed with "commit cleanup" enough. The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#`. One suggestion was to allow backslash escapes in commit messages since that makes sense to a computer scientist.[1] But looking at all of this lexical stuff I wonder if makes-sense-to-computer-scientist is a good goal. They invented the problem of using a uniform delimiter for strings and then had to solve their own problem. Maybe it was hard to use backtick in the 70’s and 80’s, but today[2] you could use backtick to start a string and a single quote to end it. What do C-like programming languages use single quotes for? To quote characters. Why do you need to quote characters? I’ve never seen a literal character which needed an "end character" marker. Raw strings would still be useful but you wouldn’t need raw strings just to do a very basic thing like make a string which has typewriter quotes in it. Of course this was for C-like languages. Don’t even get me started on shell and related languages where basically everything is a string and you have to make a single-quote/double-quote battle plan before doing anything slightly nested. [1] https://lore.kernel.org/git/vpq3808p40o.fsf@anie.imag.fr/ https://lore.kernel.org/git/vpq3808p40o.fsf@anie.imag.fr/ [2] Notwithstanding us Europeans that use a dead-key keyboard layout where you have to type twice to get one measly backtick (not that I use those)

deleted 2y ago
[deleted]
pwdisswordfishz 2y ago
> The comment character in Git commit messages can be a problem when you insist on prepending your commits with some "id" and the id starts with `#` https://git-scm.com/docs/git-commit#Documentation/git-commit.txt---cleanupltmodegt https://git-scm.com/docs/git-commit#Documentation/git-commit...
- keybored 2y ago
  See "commit cleanup". There’s surprising layers to this. That the reporter in that thread says that git-commit will “happily” accept `#` in commit messages is half-true: it will accept it if you don’t edit the message since the `default` cleanup (that you linked to) will not remove comments if the message is given through things like `-m` and not an editing session. So `git commit -m'#something' is fine. But then try to do rebase and cherry-pick and whatever else later, maybe get a merge commit message with a commented "conflicted" files. Well it can get confusing.

yen223 2y ago

select'select'select is a perfectly valid SQL query, at least for Postgres. Languages' approach to whitespace between tokens is all over the place

notsylver 2y ago

As soon as I saw this was part of llamafile I was hoping that it would be used to limit LLM output to always be "valid" code as soon as it saw the backticks, but I suppose most LLMs don't have problems with that anyway. And I'm not sure you'd want something like that automatically forcing valid code anyway

dilap 2y ago
llama.cpp does support something like this -- you can give it a grammar which restricts the set of available next tokens that are sampled over so in theory you could notice "```python" or whatever and then start restricting to valid python code. (in least in theory, not sure how feasible/possible it would be in practice w/ their grammar format.) for code i'm not sure how useful it would be since likely any model that is giving you working code wouldn't be struggling w/ syntax errors anyway? but i have had success experimentally using the feature to drive fiction content for a game from a smaller llm to be in a very specific format.
- notsylver 2y ago
  yeah, ive used llama.cpp grammars before, which is why i was thinking about it. i just think it'd be cool for llamafile to do basically that, but with included defaults so you could eg, require JSON output. it could be cool for prototyping or something. but i dont think that would be too useful anyway, most of the time i think you would want to restrict it to a specific schema, so i can only see it being useful for something like a tiny local LLM for code completion, but that would just encourage valid-looking but incorrect code. i think i just like the idea of restricting LLM output, it has a lot of interesting use cases
  - dilap 2y ago
    gotchya. i do think that is a cool idea actually -- LLMs tiny enough to do useful things with formally structured output but not big enough to nail the structure ~100% is probably not an empty set.

pwdisswordfishz 2y ago

> Of all the languages, I've saved the best for last, which is Ruby. Now here's a language whose syntax evades all attempts at understanding. TeX with its arbitrarily reprogrammable lexer: how adorable

fanf2 2y ago
Lisp reader macros allow you to program its lexer too.
- skydhash 2y ago
  You can basically define a new language with a few lines of code in Racket.

pansa2 2y ago

> TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings. So to highlight a string, one must count curly brackets and maintain a stack of parser states. Presumably this is also true in Python - IIRC the brace-delimited fields within f-strings may contain arbitrary expressions. More generally, this must mean that the lexical grammar of those languages isn't regular. "Maintaining a stack" isn't part of a finite-state machine for a regular grammar - instead we're in the realm of pushdown automata and context-free grammars. Is it even possible to support generalized string interpolation within a strictly regular lexical grammar?

deleted 2y ago
[deleted]
aphantastic 2y ago
> Is it even possible to support generalized string interpolation within a strictly regular lexical grammar? Almost certainly not, a fun exercise is to attempt to devise a Pumping tactic for your proposed language. If it doesn’t exist, it’s not regular. https://en.m.wikipedia.org/wiki/Pumping_lemma_for_regular_languages https://en.m.wikipedia.org/wiki/Pumping_lemma_for_regular_la...
fanf2 2y ago
Complicated interpolation can be lexed as a regular language if you treat strings as three separate lexical things, eg in JavaScript template literals there are, `stuff${ }stuff${ }stuff` so the ${ and } are extra closing and opening string delimiters, leaving the nesting to be handled by the parser. You need a lexer hack so that the lexer does not treat } as the start of a string literal, except when the parser is inside an interpolation but all nested {} have been closed.

irdc 2y ago

I’d be interested to see a re-usable implementation of joe's[0] syntax highlighting.[1] The format is powerful enough to allow for the proper highlighting of Python f-strings.[2] 0. https://joe-editor.sf.net/ https://joe-editor.sf.net/ 1. https://github.com/cmur2/joe-syntax/blob/joe-4.4/misc/HowItWorks.md https://github.com/cmur2/joe-syntax/blob/joe-4.4/misc/HowItW... 2. https://gist.github.com/irdc/6188f11b1e699d615ce2520f03f1d0da https://gist.github.com/irdc/6188f11b1e699d615ce2520f03f1d0d...

pama 2y ago
Interestingly, python f-strings changed their syntax at version 3.12, so highlighting should depend on the version.
- irdc 2y ago
  It’s just that nesting them arbitrarily is now allowed, right? That shouldn’t matter much for a mere syntax highlighter then. And one could even argue that code that relies on this too much is not really for human consumption.
  - pansa2 2y ago
    Also, you can now use the same quote character that encloses an f-string within the {} expressions. That could make them harder to tokenize, because it makes it harder to recognise the end of the string.
akira2501 2y ago
I've actually made several lexers and parsers based on the joe DFA style of parsing. The state and transition syntax was something that I always understood much more easily than the standard tools. The downside is your rulesets tend to get more verbose and are a little bit harder to structure than they might ideally be in other languages more suited towards the purpose, but I actually think that's an advantage, as it's much easier to reason about every production rule when looking at the code.

rererereferred 2y ago

In the C# multiquoted strings, how does it know this: Console.WriteLine(""""""); Console.WriteLine(""""""); Are 2 triplequoted empty strings and not one "\nConsole.WriteLine(" sixtuplequoted string?

ygra 2y ago
The former, I'd say. https://learn.microsoft.com/en-us/dotnet/csharp/programming-guide/strings/#raw-string-literals https://learn.microsoft.com/en-us/dotnet/csharp/programming-... For a multi-line string the quotes have to be on their own line.
Joker_vD 2y ago
If the opening quotes are followed by anything that is not a whitespace before the next new-line (or EOF), then it's a single-line string. I imagine implementing those things took several iterations :)
yen223 2y ago
It's a syntax error! Unterminated raw string literal. https://replit.com/@Wei-YenYen/DistantAdmirableCareware#main.cs https://replit.com/@Wei-YenYen/DistantAdmirableCareware#main...
- Joker_vD 2y ago
  Ah, so there is no backtracking in lexer for this case. Makes sense.

ygra 2y ago

As for C#'s triple-quoted strings, they actually came from Java before and C# ended up adopting the same or almost the same semantics. Including stripping leading whitespace.

pdw 2y ago

Some random things that the author seem to have missed: > but TypeScript, Swift, Kotlin, and Scala take string interpolation to the furthest extreme of encouraging actual code being embedded inside strings Many more languages support that: C# $"{x} plus {y} equals {x + y}" Python f"{x} plus {y} equals {x + y}" JavaScript `${x} plus ${y} equals ${x + y}` Ruby "#{x} plus #{y} equals #{x + y}" Shell "$x plus $y equals $(echo "$x+$y" | bc)" Make :) echo "$(x) plus $(y) equals $(shell echo "$x+$y" | bc)" > Tcl Tcl is funny because comments are only recognized in code, and since it's a homoiconic, it's very hard to distinguish code and data. { } are just funny string delimiters. E.g.: xyzzy {#hello world} Is xyzzy a command that takes a code block or a string? There's no way to tell. (Yes, that means that the Tcl tokenizer/parser cannot discard comments: only at evaluation time it's possible to tell if something is a comment or not.) > SQL PostgreSQL has the very convenient dollar-quoted strings: https://www.postgresql.org/docs/current/sql-syntax-lexical.html#SQL-SYNTAX-DOLLAR-QUOTING https://www.postgresql.org/docs/current/sql-syntax-lexical.h... E.g. these are equivalent: 'Dianne''s horse' $$Dianne's horse$$ $SomeTag$Dianne's horse$SomeTag$

deleted 2y ago
[deleted]
autarch 2y ago
Perl lets you do this too: my $foo = 5; my $bar = 'x'; my $quux = "I have $foo $bar\'s: @{[$bar x $foo]}"; print "$quux\n"; This prints out: I have 5 x's: xxxxx The "@{[...]}" syntax is abusing Perl's ability to interpolate an _array_ as well as a scalar. The inner "[...]" creates an array reference and the outer "@{...}" dereferences it. For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference.
- Izkata 2y ago
  > For reasons I don't remember, the Perl interpreter allows arbitrary code in the inner "[...]" expression that creates the array reference. ...because it's an array value? Aside from how the languages handle references, how is that part any different from, for example, this in python: >>> [5 * 'x'] ['xxxxx'] You can put (almost) anything there, as long as it's an expression that evaluates to a value. The resulting value is what goes into the array.

__MatrixMan__ 2y ago

This was a fun read, but it left me a bit more sympathetic to the lisp perspective, which (if I've understood it) is that syntax, being not an especially important part of a language, is more of a hurdle than a help, and should be as simple and uniform as possible so we can focus on other things. Which is sort of ironic because learning how to do structural editing on lisps has absolutely been more hurdle than help so far, but I'm sure it'll pay off eventually.

mqus 2y ago
Having a simple syntax might be fine for computers but syntax is mainly designed to be read and written by humans. Having a simple one like lisp then just makes syntactic discussions a semantic problem, just shifting the layers. And I think an complex syntax is far easier to read and write than a simple syntax with complex semantics. You also get a faster feedback loop in case the syntax of your code is wrong vs the semantics (which might be undiscovered until runtime).
- drewr 2y ago
  I don't understand your distinction between syntax and semantics. If the semantics are complex, wouldn't that mean the syntax is thus complex?
  - SuperCuber 2y ago
    lisp's syntax is simple - its just parenthesis to define a list, first element of a list is executed as a function. but for example a language like C has many different syntaxes for different operations, like function declaration or variable or array syntax, or if/switch-case etc etc. so to know C syntax you need to learn all these different ways to do different things, but in lisp you just need to know how to match parenthesis. But of course you still want to declare variables, or have if/else and switch case. So you instead need to learn the builtin macros (what GP means by semantics) and their "syntax" that is technically not part of the language's syntax but actually is since you still need all those operations enough that they are included in the standard library and defining your own is frowned upon.

kazinator 2y ago

I don't think it's easy to write a good syntax coloring engine like the one in Vim. Syntax coloring has to handle context: different rules for material nested in certain ways. Vim's syntax higlighter lets you declare two kinds of items: matches and regions. Matches are simpler lexical rules, whereas regions have separate expressions for matching the start and end and middle. There are ways to exclude leading and trailing material from a region. Matches and regions can declare that they are contained. In that case they are not active unless they occur in a containing region. Contained matches declare which regions contain them. Regions declare which other regions they contain. That's the basic semantic architecture; there are bells and whistles in the system due to situations that arise. I don't think even Justine could develop that in an interview, other than as an overnight take home.

kazinator 2y ago
Here is an example of something hard to handle: TXR language with embedded TXR Lisp. This is the "genman" script which takes the raw output of a manpage to HTML converter, and massages it to form the HTML version of the TXR manual: https://www.kylheku.com/cgit/txr/tree/genman.txr https://www.kylheku.com/cgit/txr/tree/genman.txr Everything that is white (not colored) is literal template material. Lisp code is embedded in directives, like @(do ...). In this scheme, TXR keywords appear purple, TXR Lisp ones green. They can be the same; see the (and ...) in line 149, versus numerous occurrences of @(and). Quasistrings contain nested syntax: see 130 where `<a href ..> ... </a>` contains an embedded (if ...). That could itself contain a quasistring with more embedded code. TXR's txr.vim" and tl.vim* syntax definition files are both generated by this: https://www.kylheku.com/cgit/txr/tree/genvim.txr https://www.kylheku.com/cgit/txr/tree/genvim.txr
saghm 2y ago
Naively, I would have assumed that the "correct" way to write a syntax highlighter would be to parse into an AST and then iterate over the tokens and update the color of a token based on the type of node (and maybe just tracking a diff to avoid needing to recolor things that haven't changed). I'm guessing that if this isn't done, it's for efficiency reasons (e.g. due to requiring parsing the whole file to highlight rather than just the part currently visible on the screen)?

susam 2y ago

> Every C programmers (sic) knows you can't embed a multi-line comment in a multi-line comment. And every Standard ML programmer might find this to be a surprising limitation. The following is a valid Standard ML program: (* (* Nested (**) *) comment *) val _ = print "hello, world\n" Here is the output: $ sml < hello.sml Standard ML of New Jersey (64-bit) v110.99.5 [built: Thu Mar 14 17:56:03 2024] - = hello, world $ mlton hello.sml && ./hello hello, world Given how C was considered one of the "expressive" languages when it arrived, it's curious that nested comments were never part of the language.

dahart 2y ago
There are 3 things I find funny about that comment: ML didn’t have single-line comments, so same level of surprising limitation. I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. And what bearing does the comment syntax have on the expressiveness of a language? I would argue absolutely none at all, by definition. :P
- susam 2y ago
  > ML didn’t have single-line comments, so same level of surprising limitation. It is not quite clear to me why the lack of single-line comments is such a surprising limitation. After all, a single-line block comment can easily serve as a substitute. However, there is no straightforward workaround for the lack of nested block comments. > I’ve never heard someone refer to C as “expressive”, but maybe it was in 1972 when compared to assembly. I was thinking of Fortran in this context. For instance, Fortran 77 lacked function pointers and offered a limited set of control flow structures, along with cumbersome support for recursion. I know Fortran, with its native support for multidimensional arrays, excelled in numerical and scientific computing but C quickly became the preferred language for general purpose computing. While very few today would consider C a pinnacle of expressiveness, when I was learning C, the landscape of mainstream programming languages was much more restricted. In fact, the preface to the first edition of K&R notes the following: "In our experience, C has proven to be a pleasant, expressive and versatile language for a wide variety of programs." C, Pascal, etc. stood out as some of the few mainstream programming languages that offered a reasonable level of expressiveness. Of course, Lisp was exceptionally expressive in its own right, but it wasn't always the best fit for certain applications or environments. > And what bearing does the comment syntax have on the expressiveness of a language? Nothing at all. I agree. The expressiveness of C comes from its grammar, which the language parser handles. Support for nested comments, in the context of C, is a concern for the lexer, so indeed one does not directly influence the other. However, it is still curious that a language with such a sophisticated grammar and parser could not allocate a bit of its complexity budget to support nested comments in its lexer. This is a trivial matter, I know, but I still couldn't help but wonder about it.

lupire 2y ago

> You'll notice its hash function only needs to consider a single character in in a string. That's what makes it perfect, Is that a joke? https://en.m.wikipedia.org/wiki/Perfect_hash_function https://en.m.wikipedia.org/wiki/Perfect_hash_function

jaen 2y ago
No. Taking the value of a single character is a correct perfect hash function, assuming there exists a position for the input string set where all characters differ.
anitil 2y ago
It just so happens that a single character was all that's needed in that case. In the general case it wouldn't be

playingalong 2y ago

Nice read. I guess the article could be called Falsehoods Programmers Assume of Programming Language Syntaxes.

TomatoCo 2y ago

I think my favorite C trigraph was something like do_action() ??!??! handle_error() It almost looks like special error handling syntax but still remains satisfying once you realize it's an || logical-or statement and it's using short circuiting rules to execute handle error if the action returns a non-zero value.

wslh 2y ago
Did you choose the legacy C trigraphs over || for aesthetic purposes?
- wslh 2y ago
  Could you review my comment on HN? Please educate me if there is something I haven’t understood, rather than downvoting my question.
  - samatman 2y ago
    The grandparent post is specifically about trigraphs. Saying something about trigraphs was the end-in-itself, trigraphs were chosen to illustrate something about trigraphs. So your question made no sense. Hope that helps.
    - Izkata 2y ago
      Maybe the confusion was the other way, more like "why is that funny/interesting?" An attempt to answer that: In English, mixing ?! at the end of a question is a way of indicating bewilderment. Like "What was that?!"
    - wslh 2y ago
      My question was precisely about why the user like trigraphs over using just || on this case. It is a very clear question and makes all the sense.
      - teo_zero 2y ago

IshKebab 2y ago

I don't understand why you wouldn't use Tree Sitter's syntax highlighting for this. I mean it's not going to be as fast but that clearly isn't an issue here. Is this a "no third party dependencies" thing?

jart 2y ago
I don't want to require everyone who builds llamafile from source need to install rust. I don't even require that people install the gperf command, since I can build gperf as a 700kb actually portable executable and vendor it in the repo. Tree sitter I'd imagine does a really great highly precise job with the languages it supports. However it appears to support fewer of them than I am currently. I'm taking a breadth first approach to syntax highlighting, due to the enormity of languages LLMs understand.
- IshKebab 2y ago
  I think the Rust component of tree-sitter-highlight is actually pretty small (Tree Sitter generates C for the actual parser). But fair enough - fewer dependencies is always nice, especially in C++ (which doesn't have a modern package manager) and in ML where an enormous janky Python installation is apparently a perfectly normal thing to require.
  - mdaniel 2y ago
    I somehow thought Conan[1] was the C++ package manager; it's at least partially supported by GitLab, for what that's worth 1: https://docs.conan.io/2/introduction.html https://docs.conan.io/2/introduction.html
    - IshKebab 2y ago
      No, if anything vcpkg is "the C++ package manager", but it's nowhere near pervasive and easy-to-use enough to come close to even Pip. It's leagues away from Cargo, Go, and other actually good PL package managers.

jim_lawless 2y ago

Forth has a default syntax, but Forth code can execute during the compilation process allowing it to accept/compile custom syntaxes.

SonOfLilit 2y ago

Justine gets very close to the hairiest parsing issue in any language without encountering it: Perl's syntax is undecidable, because the difference between treating some characters as a comment or as a regex can depend on the type of a variable that is only determined e.g. based on whether a search for a Collatz counterexample terminates, or just, you know, user input. https://perlmonks.org/?node_id=663393 https://perlmonks.org/?node_id=663393 C++ templates have a similar issue, I think.

fanf2 2y ago
I think possibly the most hilariously complicated instance of this is in perl’s tokenizer, toke.c (which starts with a Tolkien quote, 'It all comes from here, the stench and the peril.' — Frodo). There’s a function called intuit_more which works out if $var[stuff] inside a regex is a variable interpolation followed by a character class, or an array element interpolation. Its result can depend on whether something in the stuff has been declared as a variable or not. But even if you ignore the undecidability, the rest is still ridiculously complicated. https://github.com/Perl/perl5/blob/blead/toke.c#L4502 https://github.com/Perl/perl5/blob/blead/toke.c#L4502
- ufo 2y ago
  Wow. I wonder how that function came to be in the first place. Surely it couldn't have started out that complicated?
_cbsz 2y ago
> C++ templates have a similar issue TIL! I went and dug up a citation: https://blog.reverberate.org/2013/08/parsing-c-is-literally-undecidable.html https://blog.reverberate.org/2013/08/parsing-c-is-literally-...
layer8 2y ago
How could a search for a Collatz counterexample possibly terminate? ;)
chubot 2y ago
Yup, bash and GNU Make have the same issue as Perl does, and I mention the C++ issue here too: Parsing Bash is Undecidable - https://www.oilshell.org/blog/2016/10/20.html https://www.oilshell.org/blog/2016/10/20.html I remember a talk from Larry Wall on Perl 6 (now Raku), where he says this type of thing is a mistake. Raku can be statically parsed, as far as I know.

petesergeant 2y ago

> Perl also has this goofy convention for writing man pages in your source code The world corpus of software would be much better documented if everywhere else had stolen this from Perl. Inline POD is great.

kragen 2y ago
Perl and Python stole it from Emacs Lisp, though Perl took it further. I'm not sure where Java stole it from, but nowadays Doxygen is pretty common for C code. Unfortunately this results in people thinking that Javadoc and Doxygen are substitutes for actual documentation like the Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequate for a reference manual.
- mdaniel 2y ago
  > Emacs Lisp Reference Manual, which cannot be generated from docstrings, because the organization of the source code is hopelessly inadequate for a reference manual. Well, they're not doing themselves any favors by just willy nilly mixing C with "user-facing" defuns <https://emba.gnu.org/emacs/emacs/-/blob/ed1d691184df4b50da6b8e1a207e9ccd88aa9ffb/src/process.c#L1003-1004 https://emba.gnu.org/emacs/emacs/-/blob/ed1d691184df4b50da6b...>. I was curious if they could benefit from "literate programming" since OrgMode is the bee's knees but not with that style coding they can't
  - kragen 2y ago
    I didn't mean that specifically the Emacs source code was not organized in the right way for a reference manual. I meant that C and Java source code in general isn't. And C++, which is actually where people use Doxygen more. The Python standard library manual is also exemplary, and also necessarily organized differently from the source code.
    - mdaniel 2y ago
      > The Python standard library manual is also exemplary Maybe parts of it are, but as a concrete example https://docs.python.org/3/library/re.html#re.match https://docs.python.org/3/library/re.html#re.match is just some YOLO about what, specifically, is the first argument to re.match: string, or compiled expression? Well, it's both! Huzzah! I guess they get points for consistency because the first argument to re.compile is also "both" But, any idea what type re.compile returns? cause https://docs.python.org/3/library/re.html#re.compile https://docs.python.org/3/library/re.html#re.compile is all "don't you worry about it" versus its re.match friend who goes out of their way to state that it is an re.Match object Would it have been so hard to actually state it, versus requiring someone to invoke type() to get <class 're.Pattern'>?

metadat 2y ago

> The languages I decided to support are Ada, Assembly, BASIC, C, C#, C++, COBOL, CSS, D, FORTH, FORTRAN, Go, Haskell, HTML, Java, JavaScript, Julia, JSON, Kotlin, ld, LISP, Lua, m4, Make, Markdown, MATLAB, Pascal, Perl, PHP, Python, R, Ruby, Rust, Scala, Shell, SQL, Swift, Tcl, TeX, TXT, TypeScript, and Zig. A few (admittedly silly) questions about the list: 1. Why no Erlang, Elixir, or Crystal? Erlang appears to be just at the author's boundary at #47 on the TIOBE index. https://www.tiobe.com/tiobe-index/ https://www.tiobe.com/tiobe-index/ 2. What is "Shell"? Sh, Bash, Zsh, Windows Cmd, PowerShell..? 3. Perl but no Awk? Curious why, because Awk is a similar but comparatively trivial language. Widely used, too. To be fair, Awk, Erlang, and Elixir rank low on popularity. Yet m4, Tcl, TeX, and Zig aren't registered in the top 50 at all. What's the methodology / criteria? Only things the author is already familiar with? Still a fun article.

Yasuraka 2y ago
Tiobes's index is quite literally worthless, especially with regards to its stated purpose, let alone as a general point of orientation. I'd wish that purple would stop lending it any credibility.
Kwpolska 2y ago
"Shell" in the context of a syntax highlighting language picker almost always means a Unixy shell, most likely something along the lines of Bash.

dakiol 2y ago

Wouldn’t be possible to let the LLM do the highlighting? Instead of returning code in plain text, it could return code within html with the appropriate tags. Maybe it’s harder than it sounds… but if it’s just for highlighting the code the LLM returns, I wouldn’t mind the highlighting not being 100% accurate.

trashburger 2y ago
Would be much slower and eat up precious context window.

layer8 2y ago

The author may have missed that lexing C is actually context-sensitive, i.e. you need a symbol table: https://en.wikipedia.org/wiki/Lexer_hack https://en.wikipedia.org/wiki/Lexer_hack Of course, for syntax highlighting this is only relevant if you want to highlight the multiplication operator differently from the dereferencing operator, or declarations differently from expressions. More generally, however, I find it useful to highlight (say) types differently from variables or functions, which in some (most?) popular languages requires full parsing and symbol table information. Some IDEs therefore implement two levels of syntax highlighting, a basic one that only requires lexical information, and an extended one that kicks in when full grammar and type information becomes available.

legobmw99 2y ago
I’d be shocked if jart didn’t know this, but it seems unlikely that an LLM would generate one of these most vexing parses, unless explicitly asked
- layer8 2y ago
  Given all the things that were new to the author in the article, I wouldn’t be shocked at all. There’s just a huge number of things to know, or to have come across.
  - jraph 2y ago
    Justine is proficient in C, she is the author of a libc (cosmopolitan) among other things, like Actually Portable Executables [1]. I would expect her to know C quite well, and that's probably an understatement. [1] https://justine.lol/ape.html https://justine.lol/ape.html
- quietbritishjim 2y ago
  I think you're thinking of something different to the issue in the parent comment. The most vexing parse is, as the name suggests, a problem at the parsing stage rather than the earlier lexing phase. Unlike the referenced lexing problem, it does't require any hack for compilers to deal with it. That's because it's not really a problem for the compiler; it's humans that find it surprising.

murkt 2y ago

Author hasn’t tried to highlight TeX. Which is good for their mental health, I suppose, as it’s generally impossible to fully highlight TeX without interpreting it. Even parsing is not enough, as it’s possible to redefine what each character does. You can make it do things like “and now K means { and C means }”. Yes, you can find papers on arXiv that use this god-forsaken feature.

jart 2y ago
I wrote https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafile/highlight_tex.cpp https://github.com/Mozilla-Ocho/llamafile/blob/main/llamafil... and it does a reasonable job highlighting without breaking for all the .tex files I could find on my hard drive. My goal is to hopefully cover 99.9% of real world usage, since that'll likely cover everything an LLM might output. Esoteric syntax also usually isn't a problem, so long as it doesn't cause strings and comments to extend forever, eclipsing the rest of the source code in a file.
- murkt 2y ago
  Yes, when goal isn’t to support 100% of all the weird stuff, then it’s orders of magnitude easier!
nathell 2y ago
Same with Common Lisp (you can redefine the read table), although that’s likely abused less often on arXiv.
bobbylarrybobby 2y ago
I couldn't believe it when I learned that \makeatletter does not “make (something) at a letter (character)” but rather “treats the '@' character as a letter when parsing”.

xonix 2y ago

No AWK?

sundarurfriend 2y ago

The final line number count is missing Julia. Based on the file in the repo, it would be at the bottom of the first column: between ld and R. Among the niceties listed here, the one I'd wish for Julia to have would be C#'s "However many quotes you put on the lefthand side, that's what'll be used to terminate the string at the other end". Documentation that talks about quoting would be so much easier to read (in source form) with something like that.

sundarurfriend 2y ago
One nicety that Julia does have that I didn't know about (or had forgotten) is nested multi-line comments. #= this one has a #= nested comment =# inside of it and that works fine! =#

nusaru 2y ago

> Ruby is the union of all earlier languages, and it's not even formally documented. It's documented, but you need $250 to spare: https://www.iso.org/standard/59579.html https://www.iso.org/standard/59579.html

mdaniel 2y ago
Well, according to (ahem) a copy that I found, it only goes up to MRI 1.9 and goes out of its way to say "welp, the world is changing, so we're just going to punt until Ruby stabilizes" which is damn cheating for a standard IMHO Also, while doing some digging I found there actually are a number of the standards that are legitimately publicly available https://standards.iso.org/ittf/PubliclyAvailableStandards/index.html https://standards.iso.org/ittf/PubliclyAvailableStandards/in...
vidarh 2y ago
ISO Ruby is a tiny, dated subset of Ruby. I doubt you'll find much Ruby that conforms to it. The Ruby everyone uses is much better defined by RubySpec etc. via test cases, but that's not complete either.

tomcam 2y ago

> If you ever want to confuse your coworkers, then one great way to abuse this syntax is by replacing the heredoc marker with an empty string Maybe I am in favor of the death penalty after all

petters 2y ago

> I'm not sure who wants to be able to syntax highlight C at 35 MB per second, but I am now able to do so Fast, but tcc *compiles* C to binary code at 29 MB/s on a really old computer: https://bellard.org/tcc/#speed https://bellard.org/tcc/#speed Should be possible to go much faster but probably not needed

BiteCode_dev 2y ago
Justine vs Bellard, that's a nice setup.

transfire 2y ago

Impressive work! I am surprised Smalltalk and Prolog are in there though.

fahrnfahrnfahrn 2y ago

While developing the syntax for a programming language in the early 80s, I discovered that allowing spaces in identifiers was unambiguous, e.g., upper left corner = scale factor * old upper left corner.

KolmogorovComp 2y ago

The article doesn’t address why they chose not use treesiter and instead roll their own syntax highlight system. Edit: same question with answer here https://news.ycombinator.com/item?id=42026554 https://news.ycombinator.com/item?id=42026554