12 ms·

The hardest program I've ever written (2015)

261 points by graderjs 5y ago

amelius 5y ago
Does it find a true optimum, or just some approximation?
- algon33 5y ago
  Approximation
- munificent 5y ago
  As long as it doesn't hit a built-in hard limit for search space exploration (which is in practice only encountered on pathological generated code), it will find the optimally scored set of line breaks.
CodesInChaos 5y ago
As a programmer I prefer formatters that don't introduce those heuristic line-breaks based on line length. I'm still hoping that Rust will eventually get such a formatter. Unfortunately the people responsible for rustfmt seem to have a strong preference for the "ignore line-breaks the user inserted" approach.
- mbrock 5y ago
  Zig's formatter has no notion of line length. If you want an argument list to be stacked vertically, you insert a trailing comma after the last argument, otherwise the formatter will put it all on a single line. I was a bit bothered by this at first but I came to really like it.
- jynelson 5y ago
  It's not the maintainers - rustfmt has an official style guide it's not allowed to break without an RFC: https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/guide.md https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide...
- umvi 5y ago
  Agreed. Formatters are supposed to remove all cognitive burden related to formatting. But formatters like Black (for python) will do line-length based formatting which reintroduces the cognitive burden again ("oh crap, my variable names are too long, better shorten them so this line stops getting broken up"). I like gofmt better for this reason. It doesn't break up your lines based on some arbitrary line length.
  - sixstringtheory 5y ago
    I usually set the line length limit to somewhere around 80 so I can have several columns visible at once without wrapping or truncation. So far my magic number is three columns: three code files, or one/two code files with a terminal and/or browser window thrown in the mix. Or any of these columns can be split into two vertically stacked boxes for a total of six things. I’m also a (nonultrawide) single screen coder (after many forays into the multi screen world) which has undoubtedly guided my preference.
  - munificent 5y ago
    Ultimately code is a visual medium (for most users). It is a data format consumed primarily by human eyeballs, so there is no escaping the reality that things like identifier length, line length, wrapping, etc. matter. Any other strategy is like trying to design a chair without thinking about butts. You may come up with some sort of elegant Bauhaus mathematically perfect work of art, but no one will want to sit in it.
- boardwaalk 5y ago
  This grinds me gears. Especially when you have multiple similar lines and one is a character longer and the formatter breaks that line. It’s like, I could scan the code easier and understand it better without you doing that, thank you! A similar thing is match statements — some arms using braces vs statements. There’s also a bunch of heuristics in rustfmt that are complex to the point that I literally couldn’t format the code the way it does without building some sort of decision tree annotated with uneven column limits (e.g. “70% of the column limit”). If I can’t and therefore wouldn’t format the code the way the formatter does, there’s an issue. Formatters are for consistency, I think, and sometimes they work against that.
tgv 5y ago
Heuristics? That sounds like a job for machine learning, and I'm not being frivolous. I think that it is doable, and when it gets it wrong, the consequences are almost nil. It would at least make a decent graduation project. Let's not think of the spin-off: FaaS.
- iudqnolq 5y ago
  Part of the complexity is if I format my code and commit it, and then you checkout and make a change, we don't want your formatter to have different opinions on how to format my code. For this you need (partial) stability at least across minor versions, which is harder with a less-explainable algorithm.
- munificent 5y ago
  There is research into using ML for automated formatting. Personally, I'm not a fan. The heuristics are relatively simple and when hand-authored can be explained. Throwing ML at it discards explainability and risks really weird formatting decisions on edge cases for relatively little upside. My experience is that people prefer formatting that is: 1. Unsurprising. 2. Nice looking. 3. Simple. In roughly that order. Using ML might increase 2 but at the expense of 1 and certainly 3.
- klyrs 5y ago
  Why replace a domain-specific heuristic with domain-agnostic machine learning? And where do you get high quality training data?
wheelerof4te 5y ago
So many formatters popping up. If the old-timers could do without them, I wonder about the usefulness of them. The best code formatter is you.
- junga 5y ago
  But you are not me and here we are.
- lordpankake 5y ago
  Old-timers did without memory safety too, which is why I'm switching our company's stack over to good ol' C
  - wheelerof4te 5y ago
    Some of the older programming languages had memory safety. C is not the only old programming language.
- tomjen3 5y ago
  Do you also wonder about the usefulness of git?
  - wheelerof4te 5y ago
    Sometimes, yes. But I never worked in a big professional team before, where git's features shine. And git != code formatter.
    - scq 5y ago
      In a team is also where code formatters shine, IMO. The biggest advantages (to me at least) are that they almost entirely eliminate formatting from code review, and the consistent style makes it easy to read and edit code written by different people.
    - unfocussed_mike 5y ago
      Preaching to the choir here I suspect, but if other lone developers are reading this: Git's features shine just as brightly, IMO, if you work for yourself. Granted a good number of them are often not very relevant to a lone developer, but if you work for or by yourself on any projects over the long term, you should build git (or some other distributed source control, but probably git) into the way you work. Not to excess, by any means. I use a very small subset of git's features and in practice many of my simpler projects don't even use branches, because it's not in the nature of the changes I am making to those projects or the time management I do. But for example you should consider git an essential step between dev and live -- using it to deploy -- and you should look at how you could use it to facilitate staging and testing. Combined with a changelog and relentless use of comments and notes, git helps "structured forgetting", which as a freelancer is pretty crucial; sometimes you work frantically on a thing for a month, get paid and then it comes back to you years later. > And git != code formatter. No, obviously, but the needs of the former are supported by the benefits of the latter. That said, I use one for golang but not, in general, for PHP. I should find a code-formatter I can bend to my will for PHP, but after 17 years of increasingly complex lone PHP development, I know what I need from my own formatting requirements in order to manage projects.
mjevans 5y ago
The Dart formatter sounds really advanced, and reflects potential complexity of the language. I think I'd still prefer to see a formatter attempt to preserve any formatting which is already 'good enough' to pass as an output threshold. Code isn't just a recipe for a computer to do something, it's a language for explaining to other programmers what that thing is and what's important to the structure of accomplishing it. The choice of where to place a break can matter for cognition and can be almost as important as the printed characters for organizing thoughts.
- munificent 5y ago
  > reflects potential complexity of the language. The language is fairly complex syntactically and that definitely adds some cost to formatting. But I think much of the complexity comes from two things: 1. A lot of idiomatic Dart uses function literals in a block-like way, as in: test(() { expect(1 + 2, 3); }); 2. But Dart doesn't actually having trailing block argument syntax like Smalltalk, Ruby, and Kotlin. So the formatter has to look at the closures passed to an argument list and decide which ones look better using block formatting, versus regular argument list formatting like: someFunction( () { expect(1 + 2, 3); }); Also, at the time I first wrote dartfmt, there was a lot of very nicely hand-formatted code in the wild that used different subtle layout choices to make different argument lists look nice. In order to persuade people to adopt the formatter at all, it had to be sophisticated enough to figure out many of those patterns and apply them automatically. It's not as good as a human (mainly because it doesn't have semantic context) but it had to be pretty close or people wouldn't have tried it. Now that it's well established, I think it would probably be possible to simplify how it formats while still making users happy. Possibly happier because the results would be a little easier to predict. > Code isn't just a recipe for a computer to do something, it's a language for explaining to other programmers what that thing is and what's important to the structure of accomplishing it. An automated formatter will never be as good as carefully crafted artisanal formatting. In particular, automated formatters don't know what stuff means. A good human might choose to line break a function call like so: setColor(red: 123, green: 54, blue: 26, alpha: 45); Because they know that "RGB" is a single coherent concept and alpha is less closely related. An automated formatter doesn't (and probably shouldn't) have that domain knowledge. But the value proposition of automated formatting is not just "how nice is the resulting code to read". You have to look at the total value proposition of completely yielding formatting to a tool versus allowing human control over it. When it's completely automated: 1. You can run it on generated code that contains absolutely no whitespace and still get nice output. 2. Humans can do large-scale refactorings, format, and get output that is consistent with the existing state of the codebase without having to understand any local style preferences. 3. Humans never have to spend time deciding how to format. Further, they don't even have to spend time deciding if they should format. 4. When reading a random codebase, it is likely to be formatted in a style you are used to even if you have zero communication with that team. This is particularly important in open source. 5. The code looks familiar to you wherever you encounter it: IDEs, plain text editors, code review tools, blog posts, StackOverflow answers. As opposed to letting everyone pick their own style and relying on users to apply their preferred style locally, it's just always in a familiar style. 6. Like any automation, the tool doesn't make mistakes. Even very careful humans hand-formatting make more mistakes than they realize. (I know because I've looked at their code). Those mistakes can be distracting for readers. 7. It gets people out of the mindset of being nitpicky about style. It encourages them to stay focused on the structure and naming of their code, which is what really matters. 8. It eliminates style arguments in code reviews. Those take time and, worse, cause disharmony, for next to no benefit. I think it's very worth excepting some small loss of overall formatting quality to get those in return.
oneeyedpigeon 5y ago
I have a particular fondness for well-formatted html that I can read via 'view source' - in contrast to the div-overloaded soup that I usually encounter. Periodically, I toy with automated formatting for html until I remember that it's essentially impossible - two html sources can have a single space character difference and simultaneously produce the same output and different output, depending on an external CSS file. This kind of stuff is tricky.
- deleted 5y ago
  [deleted]
- ameliaquining 5y ago
  I'm confused, don't the browser dev tools do exactly this? That seems better than sending pretty-printed HTML on the wire, which is a bunch of unnecessary bytes that users have to pay for.
  - oneeyedpigeon 5y ago
    The browser dev tools display something very different to the source of the page - it's a live view of the document, for a start.
    - ameliaquining 5y ago
      That's addressed easily enough: disable JavaScript, so that the document can't change.
      - oneeyedpigeon 5y ago
        Unless I haven't discovered the method, it's also a LOT harder to search the DOM for a given string using the dev tools than it is via source.
        ameliaquining 5y ago
        In Chrome, you just Ctrl+F on the Elements tab and then start typing.
knorker 5y ago
Shouldn't this be marked 2015?
checker659 5y ago
This is the second or third article on the difficulty of line breaking I've read on HN just this week. Why aren't there any good *exhaustive* tome on the art of text editors / line breaking / text shaping / text rendering / text on the GPU etc. I'd pay good money.
- xarope 5y ago
  is this /s, in reference to latex/knuth and the precursor to yak shaving?
- bombcar 5y ago
  There’s not a single reference that I know of, at least not covering all aspects. An interesting place to start may be this PDF and it’s references: https://mirror.math.princeton.edu/pub/CTAN/info/memdesign/memdesign.pdf https://mirror.math.princeton.edu/pub/CTAN/info/memdesign/me... TeX famously does line breaking in a perhaps decent way - but it and text shaping become more of an art than a definite list of rules to follow.
jll29 5y ago
(2015)
r34 5y ago
That's something that I've been thinking about yesterday, while writing my code in PHPStorm. I thought how much easier those modern tools make programmer life, how intuitive they are and how hard it must have been to get to the current state of art. Thanks for that, creators!
dorianmariefr 5y ago
Related for JavaScript, Ruby, HTML and many more https://prettier.io https://prettier.io And the creator of prettier/plugin-ruby worked on a pure-Ruby implementation https://github.com/ruby-syntax-tree/syntax_tree https://github.com/ruby-syntax-tree/syntax_tree
MeteorMarc 5y ago
Maybe a code formatter should be just brutally simple and predictable. Fearing the look of long, complicated statements, coders will shorten their statements and just do one thing per statement.
- criddell 5y ago
  It also encourages shorter and often less descriptive names. That's not necessarily a good thing.
  - eyelidlessness 5y ago
    This really depends a lot on the language syntax, naming of built ins, and common idioms. Like Java is notorious for its long lines because it’s so verbose and long naming is the common convention. Lisps are notoriously far off to the other extreme. There’s quite a lot of room between those extremes, occupied by languages like Python and Ruby. And then there’s languages like JavaScript and especially TypeScript which span most of the range depending on preference.
axydlbaaxr 5y ago
So wise to share the failures and pitfalls, along with the successes.
Shadonototra 5y ago
language formatter/server with semantic analysis are the hardest thing ever
egberts1 5y ago
Sheesh. That is hard. And that does NOT pales in comparison to me polishing the “tiny” Bash regex for removal of inline comment (as denoted by hash, semicolon or double-slash) in the INI-format (version 1.4 2009) file … while … and while permitting those same inline comment characters in quoted string to be allowed in (along with its sub-sequential string up to its ending pair of matching single/double quote symbol. I have a working regex (passed by many regex online testers) but in bash yet, NO! To graderjs of HN, the Author of Dart code for matter, you got mad respect from me.
lesam 5y ago
It's interesting that the article specifically mentions the go formatter, but fails to notice that the go formatter sidesteps this problem entirely by not setting a line length constraint: https://news.ycombinator.com/item?id=16434566 https://news.ycombinator.com/item?id=16434566
- xphx 5y ago
  > If every statement fit within the column limit of the page, yup. It’s a piece of cake. (I think that’s what gofmt does.) But our formatter also keeps your code within the line length limit.
734129837261 5y ago
The day job interviews for programmers ask "write me a language formatter, you have 3 hours" I'll probably end up in jail. Those things are way beyond my skillset and I'm glad smarter people than me exist. If you're one of those people: thank you. I love you.
- viginti_tres 5y ago
  Why jail? Cause you'll lose your mind and do harmful things? Asking for a friend
  - andai 5y ago
    The result will be so bad, it will be considered a violation of the Geneva Convention.
svalorzen 5y ago
I'm not sure I understand why dynamic programming wouldn't work (and the author explicitly mentioned Knuth). Tex's main job is literally doing line breaks, which is the exact same problem being tackled here. I would expect a similar approach (progressively build a graph of the most promising breaking points) to be effective. Why wouldn't it be the case here?
- mirekrusin 5y ago
  Yes, strange, it looks like that's his solution plus some adhoc logic. At the same time he's more knowledgeable than I am so dunno.
- erichocean 5y ago
  > I would expect a similar approach (progressively build a graph of the most promising breaking points) to be effective. Why wouldn't it be the case here? That's…how he does it.
- pantsforbirds 5y ago
  There are some nice clarifications to the problems he ran into with his DP implementation with the little skulls at the bottom of the article.
- svat 5y ago
  As someone very familiar with the Knuth–Plass line-breaking algorithm (https://tex.stackexchange.com/a/423578/48 https://tex.stackexchange.com/a/423578/48), an important difference I see here is that for paragraphs (the domain of TeX), there is no "state" that needs to be preserved across lines: if you know that your paragraph is going to choose a certain break-point, then you can pretty much typeset the "before" and "after" parts independently, each optimally. (With one exception: there is a penalty for hyphens being on successive lines, so we need to track whether the previous line was hyphenated.) This is the "optimal substructures" property that makes it so amenable to dynamic programming. With the code formatter, to format the part after a certain character, you need to keep track of the indentation depth of all the expressions that have not yet terminated at this point — because you presumably want parallel expressions to be formatted with the same indentation depth, for closing parentheses to match their corresponding opening parentheses, etc. For example, in this example: experimental = document.querySelectorAll('link').any((link) => link.attributes['rel'] == 'import' && link.attributes['href'] == POLYMER_EXPERIMENTAL_HTML); and, say (I'm making this up): experimental = document.querySelectorAll('link').any( (link) => link.attributes['rel'] == 'import' && link.attributes['href'] == POLYMER_EXPERIMENTAL_HTML); — knowing that there's a break after the `&&` is not enough; you also need to know the indentation of the previous expressions, to decide how you're going to format the part after the `&&`. This is what the author alludes to in the post: > A line break changes the indentation of the remainder of the statement, which in turn affects which other line breaks are needed. Sorry, Knuth. No dynamic programming this time. […] For most of the time, the formatter did use dynamic programming and memoization. […] It worked fairly well, but was a nightmare to debug. > It was highly recursive, and ensuring that the keys to the memoization table were precise enough to not cause bugs but not so precise that the cache lookups always fail was a very delicate balancing act. Over time, the amount of data needed to uniquely identify the state of a subproblem grew, including things like the entire expression nesting stack at a point in the line, and the memoization table performed worse and worse. In TeX, paragraphs have each line of the same width (simple case) or can have a \parshape (in general), but these are "global" constraints that don't depend on what breaks you choose.
deleted 5y ago
[deleted]
raverbashing 5y ago
Whenever I read something like this I wonder that current languages (even the higher level ones) are poor at expressing higher-level concepts like that in a practical way and capturing that complexity (in an easily manageable) form One of the hardest parts of programming is understanding what's happening from reading code. And if you abstract too much "the traditional way" then it is just even harder to understand.
coliveira 5y ago
I don't understand why people have this fetich for automatic formatters. If you really want this, you should be using old style FORTRAN or something similar. The good thing about modern languages is that you don't depend on the location of code in the page for it to work. If you start worrying too much about exact formatting, you throw away this big advantage. I really prefer code in the location where I put it, not where are machine thinks it is best. And if you think that formatting is a problem to understand the code, let's get real: this is the smallest of the problems. There are tons of other things that make code complicated to read, like variable and function names, the particular style of your code, how you split it into classes and files, the algorithm you're using, and so many other, more important things. I can guarantee you that if a piece of code is well written, you can understand it independent of where you put braces or the number of spaces you're using.
- derefr 5y ago
  I take it that you don't work on a large corporate code-base / don't have to code-review other people's code? Auto-formatting (esp. when used as a pre-commit hook) means that changes people make to the style are ignored/reverted (and/or, that places where people introduce a different style in new code, are auto-formatted back into the existing style immediately, rather than that needing to be an additional commit later on.) Thus, no spurious diff lines from formatting. Thus, not having to wade through a bunch of "noise" diff-lines, to get to the "signal" of semantic changes at code-review time. Also, having auto-formatting on both your main branch + development branches, makes merge/rebase conflicts less likely to happen. (Which basically boils down to "fewer noise diff-lines" again.) In other words: auto-formatting makes code more machine-legible to syntax-blind parsers; which in turn allows tooling like diff(1) to be more helpful. (Yes, we could just have language-syntax-aware semantic-level diff/merge/etc. tools. Not sure why nobody ever made these. I bet this is one of those things where Lisp users have had it for ages but using their own parallel world of abstractions that doesn't exist in C/POSIX.)
  - coliveira 5y ago
    This has nothing to do with formatting. When you create a change to a code base you should be submitting only the lines that are new/changed. If someone is submitting purely formatting changes, he/she's just wrong and you should reject that during review.
jb3689 5y ago
I’ve been the primary maintainer for a vim plug-in for a number of years. Dealing with indentation expectations of a modern, complex programming language without an AST parser (you need to be able to deal with code that doesn’t compile) is one of the hardest problems I’ve had to work on. Dealing with string and comment detection, dealing with the constant influx of new features, keeping it performant and maintainable in a dedicated scripting language with bespoke debugging tools. The best approach I’ve come up with is to be ruthless about testing and use strictly TDD for everything Take a look at how complex the code is for something like vim-ruby to get a feel for what I’m talking about
LudwigNagasena 5y ago
Long statements is one of the reason I dislike “fluent interfaces”. To me long statements feel like a problem of bad language design. And a super smart formatter feels like a crutch when what you really want is a leg.
thedatamonger 5y ago
Bravo! Well written and informative! And as someone who's obsessed with Dart at the moment timely! Thanks!
dang 5y ago
Related: The Hardest Program I've Ever Written – How a code formatter works (2015) - https://news.ycombinator.com/item?id=22706242 https://news.ycombinator.com/item?id=22706242 - March 2020 (125 comments) The Hardest Program I've Ever Written (2015) - https://news.ycombinator.com/item?id=17271963 https://news.ycombinator.com/item?id=17271963 - June 2018 (76 comments) The Hardest Program I've Ever Written (2015) - https://news.ycombinator.com/item?id=15063193 https://news.ycombinator.com/item?id=15063193 - Aug 2017 (48 comments) The Hardest Program I've Ever Written - https://news.ycombinator.com/item?id=10195091 https://news.ycombinator.com/item?id=10195091 - Sept 2015 (76 comments)
paxys 5y ago
If I ever have to write a code formatter, it will strictly enforce one line per statement and disallow artificial line breaks. Devs who end up writing 5000-character function chains better have a wide monitor.
cosmiccatnap 5y ago

261 points by graderjs 5y ago

amelius 5y ago

Does it find a true optimum, or just some approximation?

algon33 5y ago
Approximation
munificent 5y ago
As long as it doesn't hit a built-in hard limit for search space exploration (which is in practice only encountered on pathological generated code), it will find the optimally scored set of line breaks.

CodesInChaos 5y ago

As a programmer I prefer formatters that don't introduce those heuristic line-breaks based on line length. I'm still hoping that Rust will eventually get such a formatter. Unfortunately the people responsible for rustfmt seem to have a strong preference for the "ignore line-breaks the user inserted" approach.

mbrock 5y ago
Zig's formatter has no notion of line length. If you want an argument list to be stacked vertically, you insert a trailing comma after the last argument, otherwise the formatter will put it all on a single line. I was a bit bothered by this at first but I came to really like it.
jynelson 5y ago
It's not the maintainers - rustfmt has an official style guide it's not allowed to break without an RFC: https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/guide.md https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide...
umvi 5y ago
Agreed. Formatters are supposed to remove all cognitive burden related to formatting. But formatters like Black (for python) will do line-length based formatting which reintroduces the cognitive burden again ("oh crap, my variable names are too long, better shorten them so this line stops getting broken up"). I like gofmt better for this reason. It doesn't break up your lines based on some arbitrary line length.
- sixstringtheory 5y ago
  I usually set the line length limit to somewhere around 80 so I can have several columns visible at once without wrapping or truncation. So far my magic number is three columns: three code files, or one/two code files with a terminal and/or browser window thrown in the mix. Or any of these columns can be split into two vertically stacked boxes for a total of six things. I’m also a (nonultrawide) single screen coder (after many forays into the multi screen world) which has undoubtedly guided my preference.
- munificent 5y ago
  Ultimately code is a visual medium (for most users). It is a data format consumed primarily by human eyeballs, so there is no escaping the reality that things like identifier length, line length, wrapping, etc. matter. Any other strategy is like trying to design a chair without thinking about butts. You may come up with some sort of elegant Bauhaus mathematically perfect work of art, but no one will want to sit in it.
boardwaalk 5y ago
This grinds me gears. Especially when you have multiple similar lines and one is a character longer and the formatter breaks that line. It’s like, I could scan the code easier and understand it better without you doing that, thank you! A similar thing is match statements — some arms using braces vs statements. There’s also a bunch of heuristics in rustfmt that are complex to the point that I literally couldn’t format the code the way it does without building some sort of decision tree annotated with uneven column limits (e.g. “70% of the column limit”). If I can’t and therefore wouldn’t format the code the way the formatter does, there’s an issue. Formatters are for consistency, I think, and sometimes they work against that.

tgv 5y ago

Heuristics? That sounds like a job for machine learning, and I'm not being frivolous. I think that it is doable, and when it gets it wrong, the consequences are almost nil. It would at least make a decent graduation project. Let's not think of the spin-off: FaaS.

iudqnolq 5y ago
Part of the complexity is if I format my code and commit it, and then you checkout and make a change, we don't want your formatter to have different opinions on how to format my code. For this you need (partial) stability at least across minor versions, which is harder with a less-explainable algorithm.
munificent 5y ago
There is research into using ML for automated formatting. Personally, I'm not a fan. The heuristics are relatively simple and when hand-authored can be explained. Throwing ML at it discards explainability and risks really weird formatting decisions on edge cases for relatively little upside. My experience is that people prefer formatting that is: 1. Unsurprising. 2. Nice looking. 3. Simple. In roughly that order. Using ML might increase 2 but at the expense of 1 and certainly 3.
klyrs 5y ago
Why replace a domain-specific heuristic with domain-agnostic machine learning? And where do you get high quality training data?

wheelerof4te 5y ago

So many formatters popping up. If the old-timers could do without them, I wonder about the usefulness of them. The best code formatter is you.

junga 5y ago
But you are not me and here we are.
lordpankake 5y ago
Old-timers did without memory safety too, which is why I'm switching our company's stack over to good ol' C
- wheelerof4te 5y ago
  Some of the older programming languages had memory safety. C is not the only old programming language.
tomjen3 5y ago
Do you also wonder about the usefulness of git?
- wheelerof4te 5y ago
  Sometimes, yes. But I never worked in a big professional team before, where git's features shine. And git != code formatter.
  - scq 5y ago
    In a team is also where code formatters shine, IMO. The biggest advantages (to me at least) are that they almost entirely eliminate formatting from code review, and the consistent style makes it easy to read and edit code written by different people.
  - unfocussed_mike 5y ago
    Preaching to the choir here I suspect, but if other lone developers are reading this: Git's features shine just as brightly, IMO, if you work for yourself. Granted a good number of them are often not very relevant to a lone developer, but if you work for or by yourself on any projects over the long term, you should build git (or some other distributed source control, but probably git) into the way you work. Not to excess, by any means. I use a very small subset of git's features and in practice many of my simpler projects don't even use branches, because it's not in the nature of the changes I am making to those projects or the time management I do. But for example you should consider git an essential step between dev and live -- using it to deploy -- and you should look at how you could use it to facilitate staging and testing. Combined with a changelog and relentless use of comments and notes, git helps "structured forgetting", which as a freelancer is pretty crucial; sometimes you work frantically on a thing for a month, get paid and then it comes back to you years later. > And git != code formatter. No, obviously, but the needs of the former are supported by the benefits of the latter. That said, I use one for golang but not, in general, for PHP. I should find a code-formatter I can bend to my will for PHP, but after 17 years of increasingly complex lone PHP development, I know what I need from my own formatting requirements in order to manage projects.

mjevans 5y ago

The Dart formatter sounds really advanced, and reflects potential complexity of the language. I think I'd still prefer to see a formatter attempt to preserve any formatting which is already 'good enough' to pass as an output threshold. Code isn't just a recipe for a computer to do something, it's a language for explaining to other programmers what that thing is and what's important to the structure of accomplishing it. The choice of where to place a break can matter for cognition and can be almost as important as the printed characters for organizing thoughts.

munificent 5y ago
> reflects potential complexity of the language. The language is fairly complex syntactically and that definitely adds some cost to formatting. But I think much of the complexity comes from two things: 1. A lot of idiomatic Dart uses function literals in a block-like way, as in: test(() { expect(1 + 2, 3); }); 2. But Dart doesn't actually having trailing block argument syntax like Smalltalk, Ruby, and Kotlin. So the formatter has to look at the closures passed to an argument list and decide which ones look better using block formatting, versus regular argument list formatting like: someFunction( () { expect(1 + 2, 3); }); Also, at the time I first wrote dartfmt, there was a lot of very nicely hand-formatted code in the wild that used different subtle layout choices to make different argument lists look nice. In order to persuade people to adopt the formatter at all, it had to be sophisticated enough to figure out many of those patterns and apply them automatically. It's not as good as a human (mainly because it doesn't have semantic context) but it had to be pretty close or people wouldn't have tried it. Now that it's well established, I think it would probably be possible to simplify how it formats while still making users happy. Possibly happier because the results would be a little easier to predict. > Code isn't just a recipe for a computer to do something, it's a language for explaining to other programmers what that thing is and what's important to the structure of accomplishing it. An automated formatter will never be as good as carefully crafted artisanal formatting. In particular, automated formatters don't know what stuff means. A good human might choose to line break a function call like so: setColor(red: 123, green: 54, blue: 26, alpha: 45); Because they know that "RGB" is a single coherent concept and alpha is less closely related. An automated formatter doesn't (and probably shouldn't) have that domain knowledge. But the value proposition of automated formatting is not just "how nice is the resulting code to read". You have to look at the total value proposition of completely yielding formatting to a tool versus allowing human control over it. When it's completely automated: 1. You can run it on generated code that contains absolutely no whitespace and still get nice output. 2. Humans can do large-scale refactorings, format, and get output that is consistent with the existing state of the codebase without having to understand any local style preferences. 3. Humans never have to spend time deciding how to format. Further, they don't even have to spend time deciding if they should format. 4. When reading a random codebase, it is likely to be formatted in a style you are used to even if you have zero communication with that team. This is particularly important in open source. 5. The code looks familiar to you wherever you encounter it: IDEs, plain text editors, code review tools, blog posts, StackOverflow answers. As opposed to letting everyone pick their own style and relying on users to apply their preferred style locally, it's just always in a familiar style. 6. Like any automation, the tool doesn't make mistakes. Even very careful humans hand-formatting make more mistakes than they realize. (I know because I've looked at their code). Those mistakes can be distracting for readers. 7. It gets people out of the mindset of being nitpicky about style. It encourages them to stay focused on the structure and naming of their code, which is what really matters. 8. It eliminates style arguments in code reviews. Those take time and, worse, cause disharmony, for next to no benefit. I think it's very worth excepting some small loss of overall formatting quality to get those in return.

oneeyedpigeon 5y ago

I have a particular fondness for well-formatted html that I can read via 'view source' - in contrast to the div-overloaded soup that I usually encounter. Periodically, I toy with automated formatting for html until I remember that it's essentially impossible - two html sources can have a single space character difference and simultaneously produce the same output and different output, depending on an external CSS file. This kind of stuff is tricky.

deleted 5y ago
[deleted]
ameliaquining 5y ago
I'm confused, don't the browser dev tools do exactly this? That seems better than sending pretty-printed HTML on the wire, which is a bunch of unnecessary bytes that users have to pay for.
- oneeyedpigeon 5y ago
  The browser dev tools display something very different to the source of the page - it's a live view of the document, for a start.
  - ameliaquining 5y ago
    That's addressed easily enough: disable JavaScript, so that the document can't change.
    - oneeyedpigeon 5y ago
      Unless I haven't discovered the method, it's also a LOT harder to search the DOM for a given string using the dev tools than it is via source.
      - ameliaquining 5y ago
        In Chrome, you just Ctrl+F on the Elements tab and then start typing.

knorker 5y ago

Shouldn't this be marked 2015?

checker659 5y ago

This is the second or third article on the difficulty of line breaking I've read on HN just this week. Why aren't there any good *exhaustive* tome on the art of text editors / line breaking / text shaping / text rendering / text on the GPU etc. I'd pay good money.

xarope 5y ago
is this /s, in reference to latex/knuth and the precursor to yak shaving?
bombcar 5y ago
There’s not a single reference that I know of, at least not covering all aspects. An interesting place to start may be this PDF and it’s references: https://mirror.math.princeton.edu/pub/CTAN/info/memdesign/memdesign.pdf https://mirror.math.princeton.edu/pub/CTAN/info/memdesign/me... TeX famously does line breaking in a perhaps decent way - but it and text shaping become more of an art than a definite list of rules to follow.

jll29 5y ago

(2015)

r34 5y ago

That's something that I've been thinking about yesterday, while writing my code in PHPStorm. I thought how much easier those modern tools make programmer life, how intuitive they are and how hard it must have been to get to the current state of art. Thanks for that, creators!

dorianmariefr 5y ago

Related for JavaScript, Ruby, HTML and many more https://prettier.io https://prettier.io And the creator of prettier/plugin-ruby worked on a pure-Ruby implementation https://github.com/ruby-syntax-tree/syntax_tree https://github.com/ruby-syntax-tree/syntax_tree

MeteorMarc 5y ago

Maybe a code formatter should be just brutally simple and predictable. Fearing the look of long, complicated statements, coders will shorten their statements and just do one thing per statement.

criddell 5y ago
It also encourages shorter and often less descriptive names. That's not necessarily a good thing.
- eyelidlessness 5y ago
  This really depends a lot on the language syntax, naming of built ins, and common idioms. Like Java is notorious for its long lines because it’s so verbose and long naming is the common convention. Lisps are notoriously far off to the other extreme. There’s quite a lot of room between those extremes, occupied by languages like Python and Ruby. And then there’s languages like JavaScript and especially TypeScript which span most of the range depending on preference.

axydlbaaxr 5y ago

So wise to share the failures and pitfalls, along with the successes.

Shadonototra 5y ago

language formatter/server with semantic analysis are the hardest thing ever

egberts1 5y ago

Sheesh. That is hard. And that does NOT pales in comparison to me polishing the “tiny” Bash regex for removal of inline comment (as denoted by hash, semicolon or double-slash) in the INI-format (version 1.4 2009) file … while … and while permitting those same inline comment characters in quoted string to be allowed in (along with its sub-sequential string up to its ending pair of matching single/double quote symbol. I have a working regex (passed by many regex online testers) but in bash yet, NO! To graderjs of HN, the Author of Dart code for matter, you got mad respect from me.

lesam 5y ago

It's interesting that the article specifically mentions the go formatter, but fails to notice that the go formatter sidesteps this problem entirely by not setting a line length constraint: https://news.ycombinator.com/item?id=16434566 https://news.ycombinator.com/item?id=16434566

xphx 5y ago
> If every statement fit within the column limit of the page, yup. It’s a piece of cake. (I think that’s what gofmt does.) But our formatter also keeps your code within the line length limit.

734129837261 5y ago

The day job interviews for programmers ask "write me a language formatter, you have 3 hours" I'll probably end up in jail. Those things are way beyond my skillset and I'm glad smarter people than me exist. If you're one of those people: thank you. I love you.

viginti_tres 5y ago
Why jail? Cause you'll lose your mind and do harmful things? Asking for a friend
- andai 5y ago
  The result will be so bad, it will be considered a violation of the Geneva Convention.

svalorzen 5y ago

I'm not sure I understand why dynamic programming wouldn't work (and the author explicitly mentioned Knuth). Tex's main job is literally doing line breaks, which is the exact same problem being tackled here. I would expect a similar approach (progressively build a graph of the most promising breaking points) to be effective. Why wouldn't it be the case here?

mirekrusin 5y ago
Yes, strange, it looks like that's his solution plus some adhoc logic. At the same time he's more knowledgeable than I am so dunno.
erichocean 5y ago
> I would expect a similar approach (progressively build a graph of the most promising breaking points) to be effective. Why wouldn't it be the case here? That's…how he does it.
pantsforbirds 5y ago
There are some nice clarifications to the problems he ran into with his DP implementation with the little skulls at the bottom of the article.
svat 5y ago
As someone very familiar with the Knuth–Plass line-breaking algorithm (https://tex.stackexchange.com/a/423578/48 https://tex.stackexchange.com/a/423578/48), an important difference I see here is that for paragraphs (the domain of TeX), there is no "state" that needs to be preserved across lines: if you know that your paragraph is going to choose a certain break-point, then you can pretty much typeset the "before" and "after" parts independently, each optimally. (With one exception: there is a penalty for hyphens being on successive lines, so we need to track whether the previous line was hyphenated.) This is the "optimal substructures" property that makes it so amenable to dynamic programming. With the code formatter, to format the part after a certain character, you need to keep track of the indentation depth of all the expressions that have not yet terminated at this point — because you presumably want parallel expressions to be formatted with the same indentation depth, for closing parentheses to match their corresponding opening parentheses, etc. For example, in this example: experimental = document.querySelectorAll('link').any((link) => link.attributes['rel'] == 'import' && link.attributes['href'] == POLYMER_EXPERIMENTAL_HTML); and, say (I'm making this up): experimental = document.querySelectorAll('link').any( (link) => link.attributes['rel'] == 'import' && link.attributes['href'] == POLYMER_EXPERIMENTAL_HTML); — knowing that there's a break after the `&&` is not enough; you also need to know the indentation of the previous expressions, to decide how you're going to format the part after the `&&`. This is what the author alludes to in the post: > A line break changes the indentation of the remainder of the statement, which in turn affects which other line breaks are needed. Sorry, Knuth. No dynamic programming this time. […] For most of the time, the formatter did use dynamic programming and memoization. […] It worked fairly well, but was a nightmare to debug. > It was highly recursive, and ensuring that the keys to the memoization table were precise enough to not cause bugs but not so precise that the cache lookups always fail was a very delicate balancing act. Over time, the amount of data needed to uniquely identify the state of a subproblem grew, including things like the entire expression nesting stack at a point in the line, and the memoization table performed worse and worse. In TeX, paragraphs have each line of the same width (simple case) or can have a \parshape (in general), but these are "global" constraints that don't depend on what breaks you choose.

deleted 5y ago

[deleted]

raverbashing 5y ago

Whenever I read something like this I wonder that current languages (even the higher level ones) are poor at expressing higher-level concepts like that in a practical way and capturing that complexity (in an easily manageable) form One of the hardest parts of programming is understanding what's happening from reading code. And if you abstract too much "the traditional way" then it is just even harder to understand.

coliveira 5y ago

I don't understand why people have this fetich for automatic formatters. If you really want this, you should be using old style FORTRAN or something similar. The good thing about modern languages is that you don't depend on the location of code in the page for it to work. If you start worrying too much about exact formatting, you throw away this big advantage. I really prefer code in the location where I put it, not where are machine thinks it is best. And if you think that formatting is a problem to understand the code, let's get real: this is the smallest of the problems. There are tons of other things that make code complicated to read, like variable and function names, the particular style of your code, how you split it into classes and files, the algorithm you're using, and so many other, more important things. I can guarantee you that if a piece of code is well written, you can understand it independent of where you put braces or the number of spaces you're using.

derefr 5y ago
I take it that you don't work on a large corporate code-base / don't have to code-review other people's code? Auto-formatting (esp. when used as a pre-commit hook) means that changes people make to the style are ignored/reverted (and/or, that places where people introduce a different style in new code, are auto-formatted back into the existing style immediately, rather than that needing to be an additional commit later on.) Thus, no spurious diff lines from formatting. Thus, not having to wade through a bunch of "noise" diff-lines, to get to the "signal" of semantic changes at code-review time. Also, having auto-formatting on both your main branch + development branches, makes merge/rebase conflicts less likely to happen. (Which basically boils down to "fewer noise diff-lines" again.) In other words: auto-formatting makes code more machine-legible to syntax-blind parsers; which in turn allows tooling like diff(1) to be more helpful. (Yes, we could just have language-syntax-aware semantic-level diff/merge/etc. tools. Not sure why nobody ever made these. I bet this is one of those things where Lisp users have had it for ages but using their own parallel world of abstractions that doesn't exist in C/POSIX.)
- coliveira 5y ago
  This has nothing to do with formatting. When you create a change to a code base you should be submitting only the lines that are new/changed. If someone is submitting purely formatting changes, he/she's just wrong and you should reject that during review.

jb3689 5y ago

I’ve been the primary maintainer for a vim plug-in for a number of years. Dealing with indentation expectations of a modern, complex programming language without an AST parser (you need to be able to deal with code that doesn’t compile) is one of the hardest problems I’ve had to work on. Dealing with string and comment detection, dealing with the constant influx of new features, keeping it performant and maintainable in a dedicated scripting language with bespoke debugging tools. The best approach I’ve come up with is to be ruthless about testing and use strictly TDD for everything Take a look at how complex the code is for something like vim-ruby to get a feel for what I’m talking about

LudwigNagasena 5y ago

Long statements is one of the reason I dislike “fluent interfaces”. To me long statements feel like a problem of bad language design. And a super smart formatter feels like a crutch when what you really want is a leg.

thedatamonger 5y ago

Bravo! Well written and informative! And as someone who's obsessed with Dart at the moment timely! Thanks!

dang 5y ago

Related: The Hardest Program I've Ever Written – How a code formatter works (2015) - https://news.ycombinator.com/item?id=22706242 https://news.ycombinator.com/item?id=22706242 - March 2020 (125 comments) The Hardest Program I've Ever Written (2015) - https://news.ycombinator.com/item?id=17271963 https://news.ycombinator.com/item?id=17271963 - June 2018 (76 comments) The Hardest Program I've Ever Written (2015) - https://news.ycombinator.com/item?id=15063193 https://news.ycombinator.com/item?id=15063193 - Aug 2017 (48 comments) The Hardest Program I've Ever Written - https://news.ycombinator.com/item?id=10195091 https://news.ycombinator.com/item?id=10195091 - Sept 2015 (76 comments)

paxys 5y ago

If I ever have to write a code formatter, it will strictly enforce one line per statement and disallow artificial line breaks. Devs who end up writing 5000-character function chains better have a wide monitor.

cosmiccatnap 5y ago