7 ms·

My takeaway from the article - the success of their language model illustrates what a huge fraction of our code is boilerplate. Yes, it's helpful that the syst

My takeaway from the article - the success of their language model illustrates what a huge fraction of our code is boilerplate. Yes, it's helpful that the system shows bugs, but it does this, not through careful analysis of the control flow or subtle type analysis, but by "probability of each token appearing given the previous tokens". If such a large proportion of our code follows common patterns, are we not wasting huge amounts of time writing and testing the same functions across thousands or millions of pieces of code? If we (almost) always follow a certain pattern, should not that pattern be embedded in a library or language, so vastly reducing the opportunity for errors or bugs?

alenmilk 4y ago
Yes, this is true as long as the method or function doesn't contain if's changing the behaviour depending on the data. In other words if the problem is so well defined that you can create a method that solves the problem and it doesn't need to take into account x variations of the problem then it is fine. This is the copy-paste versus creating a function discussion. Problem is the x variations of the problem and you need the code to do different things depending on the variation, we usually break the modularity of the function instead of separating the generic and non-generic parts. Hence the ifs in the function. From what I have seen people are unable to do this in their own codebase properly so I don't think it will happen globally. But on the other hand libraries are kind of the answer to the problem and as problems get well defined, one starts using libraries. Raising the level of abstraction is a continuous process.
pwinnski 4y ago
We are not compression algorithms! If we were, we could replace the most common block of boilerplate code with token 'A', the second-most block of boilerplate code with token 'B,' and so on, writing programs in very few bytes. God have mercy on anyone trying to debug such a program, though. Any language with no boilerplate at all is a black box of incomprehensibility. Java has, I think, more boilerplate than average, while some other languages have less boilerplate than average. IDEs can help with some of this, which is why I finally stopped writing all code in vim.
- AnimaLibera 4y ago
  Allowing to compress code this much is the goal of golfing languages (such as 05AB1E (or osabie) or Pyth (not Python)). The code golf stack exchange forum contains a lot of programming challenges where the goal is to write the shorest program (in bytes) that does what the challenge asks, and some answers are truly impressive, with somewhat non-trivial algorithms being implemented in as few as 4 bytes (in extreme cases). Granted, these are programming challenges and not production code to be deployed, and some golfing languages are designed for a specific kind of task or algorithm that may let us think that the algorithm was actually pre-implemented in the language (and sometimes it is kinda true), but still, worth taking a look at it.
lmm 4y ago
We don't talk in maximally compressed strings; a bit of redundancy in grammar helps make sure people can understand each other. The fact that spellcheckers are possible doesn't mean our language is too sparse and wasteful. A lot of programming languages have a lot of room for improvement though.
- WalterBright 4y ago
  A programming language with no redundancy in it means the compiler cannot detect any errors - because every sequence of characters forms a valid program. The skill is in selecting the optimal amount and form of redundancy. For example, the typical ; statement terminator is redundant. People often ask, since it is redundant, why not remove it? People have tried that, and found out that the redundancy makes for far better error detection.
  - lmm 4y ago
    On that specific example, not my experience at all. (The general point is of course valid, though I think it follows very directly from what I said)
  - pharmakom 4y ago
    Semicolon is a strange example when so many successful languages don’t have them. I think a better example is CoffeeScript, where almost any string is a valid program but probably not the one you wanted.
    - WalterBright 4y ago
      > Semicolon is a strange example when so many successful languages don’t have them. They usually have another statement terminator instead, like a newline.
  - amelius 4y ago
    No, with redundancy your compiler can catch a certain class of errors (call them "avoidable") that doesn't exist in case of no redundancy. Of course, in both cases you still have the unavoidable errors.
danjc 4y ago
Effort in a codebase is unevenly distributed. The 90% of the code that looks like boilerplate probably represents 20% of the effort.
- drpixie 4y ago
  You're probably right about the effort - but I expect that such "boilerplate" code contains (or leads to) much more than 20% of the bugs. This 90% of code is not genuine word-for-word boilerplate (copy/pasted from a known good source). This code is typically constructed fresh each time; or worse, copied from somewhere similar and quickly tweaked for names/types! (I do it, and I see it done all the time.) I expect that the remaining 10% non-boilerplate code, taking 80% of the effort, is much more carefully considered, and less likely to contain those clumsy/forgetful off-by-one or buffer-overflow bugs.
  - fiddlerwoaroof 4y ago
    Yeah, one of the interesting results in empirical studies of defect rates is that defect rate is influenced by lines of code more than other factors like “static types”. Similarly, analyses of defects have discovered that they tend to occur at the end of repetitive sequences of code, because the developer has sort of switched into autopilot mode. I think the obvious conclusion here (and my experience bears this out to some extent) is that languages and libraries that force boilerplate on you produce buggier code than languages and libraries that abstract the boilerplate away.
  - kbenson 4y ago
    There's a reason many languages have large overflowing repositories of modules (or there are well known libraries) that can be downloaded and used that provide boilerplate solutions for many things. Most people don't like writing that boilerplate once they know how to do it and have done it a few times, and would rather just call a function do_that_thing_need_done_on(input1, input2). If it can't be factored out like that and is actual language boilerplate beyond a few lines, that's a failure of the language. If these AI models are suggesting the code that could be called in a library/module instead of the the code to actually include and call a well known and trusted library or module, I'm not sure that's progress. At least when someone notices a bug or better way to do it and updates that module or library, consumers of that module can update and benefit from it, or at a minimum see that there were bugs in the version they're running they might want to address at some point.
nradov 4y ago
Yes and I raised the same concern when GitHub Copilot was released. If our code contains so little entropy that an AI can reliably predict the next sequence of tokens then that is evidence we are operating at too low a level of abstraction. Such tools can certainly be helpful in working with today's popular languages, but what would a future language that allows for abstracting away all that boilerplate look like? Since this is HN I'm sure someone will say that the answer is Lisp. But can we do better?
- tgv 4y ago
  Gzipped Lisp would be an improvement in that reasoning. I’m sure you agree that’s not a very desirable way to write code.
  - pharmakom 4y ago
    The number of bytes isn’t really significant. It’s the conceptual distance between what we want to achieve and the concrete steps in the code. Lisp may be an improvement.
    - tgv 4y ago
      But now you're already heading into much vaguer territory. Readability is also important. Very important, I would say. That requires easily identifiable markers for loops, conditions, functions, etc., something Lisp lacks. This might be a place where keyword coloring could be useful, but then we're relying on external help. Another issue is consistency. Take C, Javascript, or Go. Many loops are of the form for (var i = 0; i < n; i++) { ... } You could argue that "for i < n" provides the same information, but then you'd have to find a way to start the loop at a different offset, use a different end condition, or different "increment".
      - pharmakom 4y ago
        > But now you're already heading into much vaguer territory. Yes programming language design is a social science (imo)
ludovicianul 4y ago
This is exactly my view, especially with the web apps. If you take a distributed system, the majority of components/microservices will have more than 50% commonality in behaviour. Therefore you do mostly the same things when you start a new one. Even if code itself might be harder to generate, as even a CRUD app might have specific behaviour, testing it is definitely the same, especially when doing negative scenarios, boundary testing, CRUD operations, etc. I wrote a tool specifically for this purpose, targeted at REST APIs, aiming to automate this repetitive work and let you focus on the tests which are specific to the context.
andai 4y ago
>probability of each token appearing given the previous tokens Sounds like tokenizer -> Markov chain? Surely something trained on a TPU is more sophisticated than something we could have done in the middle of the 20th century?
IanCal 4y ago
Written language has this too. A basic lookup table of frequencies can tell you that jkkj is a typo in an English word. "Nobody else is really writing code that has the fragment in you just wrote" can find syntax errors. Better language models can find more subtle relationships. At some point the variations mean that more abstractions don't really help.
pharmakom 4y ago
I agree. And yet most developers claim JS, Python, Java etc are totally sufficient.
- wikfwikf 4y ago
  Perl is a wonderful, innovative language which failed because it tried to remove intratextual redundancy in the way you are suggesting. A string of length N is vastly more likely to be a valid Perl program than a valid Python program. Ultimately this meant that Perl programs, while easier to type, were much harder to read, and extremely easy to misinterpret.
  - ilyt 4y ago
    There is also something to be said about nudging developer on the "right" way to do stuff. Perl is not only hard to read because there are many shortcuts that might look like line noise for the inexperienced (hell, Rust have a bunch of those too), it's because there is a bunch of the ways to do anything. Like take humble array Perl> @a $VAR1 = 1; $VAR2 = 2; $VAR3 = 3; $VAR4 = 4; $VAR5 = 5; Perl> @a + 2 $VAR1 = 7; Why adding a number to array results in number ? Because array in scalar context returns its length. It leads to some very compact code if (@a > 3) {print "big array"} but, uh, what you do if you want to see length of string ? well length($string). Will that also work for arrays ? Nope, if you want to force scalar context you're supposed to do scalar(@array). So how to add 2 arrays ? @c = (@a, @b) # obviously. But wait, what we really typed after variable expansion is ((1,2),(2,3)) and in other languages irb(main):001:0> a = [1,2] => [1, 2] irb(main):002:0> b=[2,3] => [2, 3] irb(main):003:0> c = [a,b] => [[1, 2], [2, 3]] >>> a = [1,2] >>> b = [2,3] >>> c = [a,b] >>> c [[1, 2], [2, 3]] that's exactly what we get. Confusing ? Sure. But it saves few characters!
constantcrying 4y ago
It really is the exact opposite approach to static analysis, which tries to see what the code really does andhow that leads to bugs. I have had (quite expensive) static analysis tools detect genuine bugs, e.g. a somewhat subtle overflow. What it can never detect though is correct code that misses the intention of the programmer. E.g. whether some mathematical function is accurate. The language models try, by statistical means, to derive what should be there. Given enough data they will start to have sone (statistical) grasp on the intention. I am not entirely sure about the boilerplate though. Often you need some minor variation of an already existing pattern. Trying to unify those slightly divergent patterns into one schema can very easily lead to very hard to understand code. Another thing is thar boilerplate is fairly easy to write and to test, because it is familiar, reducing the actual effort which goes into it. Sometimes it is just better to not reuse code.
- kneebonian 4y ago
  Honestly this is what I have loved about Kotlin. There seems like there is now just a certain amount of boilerplate in every Java file and Kotlin just chose to bake that all into the language, the other instance where we see this happening is with the Lombok library in Java. Although personally I hate annotations.
- yccs27 4y ago
  +1 on the last paragraph: Predictability means the code follows a pattern, not that it is boilerplate. Some amount of predictable code is necessary just to spell out what the code does, so that even someone unfamiliar with the pattern can simply read and understand it.
- ajuc 4y ago
  When you go too far reducing boilerplate you get to a point where configuration becomes the code and the actual code becomes black box that people barely understand. And then they replicate what's already in the black box because they aren't sure it's there, and you do the same things over and over in every layer. And in some layers you do it one way and in others - other way. And then requirements change and you change the code and it still works the old way SOME of the time, but you only discover that in production, because the test case you used when you developed is handled on the layer where you changed it correctly. And then if it's buried deep enough somebody will add another layer and fix the cases that were found - there. And that's how the disgusting legacy code happens. KISS, please. Unnecessary abstraction is the root of almost all problems in programming.
GuB-42 4y ago
I don't think so. Sure, most code is boilerplate, except for that one thing, and that one thing can be anywhere. For example, let's say you want to write a function that returns the checksum of a bunch of data. That's a very common thing to do, there are plenty of libraries that do that, and I have seen the CRC32 lookup table in many places, sometimes I am the one who put it there. Now, why rewrite such a function? - Ignore some part of the message - Use different constants - Fetch data in a special way (i.e. not a file or memory buffer) - Have some kind of a progress meter - The library you may want to use is not available (can be for technical, legal or policy reasons) - Some in-loop operation is needed (ex: byte swapping) - Have a specific termination condition (ex: end-of-message marker) - And many others, including combinations of the above If you ignore all these points and only see the generic checksum function, yes, it is boilerplate and can be factorized. But these special cases are the reason why it may not be the case, and the reason why there are so many coding jobs. It is also the reason why we don't have real (Lv5) self driving cars yet, why there are pilots in the cockpit, why MS Office and the like have so many "useless" features, why so many attempts to make software cleaner and simpler fail, etc...
- julian37 4y ago
  That's hardly a convincing example. All of these points can be solved elegantly with a stream abstraction, which can be cheap or free given a sufficiently advanced language and compiler. As for legal or policy reasons, those still aren't reasons to write boilerplate code. Your reimplementation can be tight and reuse other abstractions or include their own.
  - GuB-42 4y ago
    A stream abstraction is a solution so some (not all) of these problems, and indeed some libraries use them, but a stream abstraction that is powerful enough to solve most of these problems may result in more complex code than just rewriting the checksum algorithm from scratch. And there is a limit on how compilers can optimize, especially considering that checksum calculation may be critical to performance. In reality, few people need to write their own checksumming function, but sometimes, it is the best thing to do. And it is just an example, there are many other instance where an off the shelf solution is not appropriate because of some detail: string manipulation, parsing, data structures (especially the "intrusive" kind), etc... And since you are probably going to have several of these in your project, it will result in a lot of boilerplate. If it was so generic not to require boilerplate, it probably has been developed already and you would be working on something else. Abstractions are almost invariably more complex, slower, more error-prone and generally worse than the direct equivalent. They are, however, reusable, that's the entire point. So one person goes through the pain of writing a nice library, and it makes life a little easier for the thousands of people who use it, generally, that's a win. But if you write an abstraction for a single use case, it is generally worse than boilerplate.
WithinReason 4y ago
I think you misunderstand machine learning. "probability of each token appearing given the previous tokens" is how humans write code too: We write code based on what we want to do and what we have written before. "what I want to do" was captured in the comments added.

alenmilk 4y ago

Yes, this is true as long as the method or function doesn't contain if's changing the behaviour depending on the data. In other words if the problem is so well defined that you can create a method that solves the problem and it doesn't need to take into account x variations of the problem then it is fine. This is the copy-paste versus creating a function discussion. Problem is the x variations of the problem and you need the code to do different things depending on the variation, we usually break the modularity of the function instead of separating the generic and non-generic parts. Hence the ifs in the function. From what I have seen people are unable to do this in their own codebase properly so I don't think it will happen globally. But on the other hand libraries are kind of the answer to the problem and as problems get well defined, one starts using libraries. Raising the level of abstraction is a continuous process.

pwinnski 4y ago

We are not compression algorithms! If we were, we could replace the most common block of boilerplate code with token 'A', the second-most block of boilerplate code with token 'B,' and so on, writing programs in very few bytes. God have mercy on anyone trying to debug such a program, though. Any language with no boilerplate at all is a black box of incomprehensibility. Java has, I think, more boilerplate than average, while some other languages have less boilerplate than average. IDEs can help with some of this, which is why I finally stopped writing all code in vim.

AnimaLibera 4y ago
Allowing to compress code this much is the goal of golfing languages (such as 05AB1E (or osabie) or Pyth (not Python)). The code golf stack exchange forum contains a lot of programming challenges where the goal is to write the shorest program (in bytes) that does what the challenge asks, and some answers are truly impressive, with somewhat non-trivial algorithms being implemented in as few as 4 bytes (in extreme cases). Granted, these are programming challenges and not production code to be deployed, and some golfing languages are designed for a specific kind of task or algorithm that may let us think that the algorithm was actually pre-implemented in the language (and sometimes it is kinda true), but still, worth taking a look at it.

lmm 4y ago

We don't talk in maximally compressed strings; a bit of redundancy in grammar helps make sure people can understand each other. The fact that spellcheckers are possible doesn't mean our language is too sparse and wasteful. A lot of programming languages have a lot of room for improvement though.

WalterBright 4y ago
A programming language with no redundancy in it means the compiler cannot detect any errors - because every sequence of characters forms a valid program. The skill is in selecting the optimal amount and form of redundancy. For example, the typical ; statement terminator is redundant. People often ask, since it is redundant, why not remove it? People have tried that, and found out that the redundancy makes for far better error detection.
- lmm 4y ago
  On that specific example, not my experience at all. (The general point is of course valid, though I think it follows very directly from what I said)
- pharmakom 4y ago
  Semicolon is a strange example when so many successful languages don’t have them. I think a better example is CoffeeScript, where almost any string is a valid program but probably not the one you wanted.
  - WalterBright 4y ago
    > Semicolon is a strange example when so many successful languages don’t have them. They usually have another statement terminator instead, like a newline.
- amelius 4y ago
  No, with redundancy your compiler can catch a certain class of errors (call them "avoidable") that doesn't exist in case of no redundancy. Of course, in both cases you still have the unavoidable errors.

danjc 4y ago

Effort in a codebase is unevenly distributed. The 90% of the code that looks like boilerplate probably represents 20% of the effort.

drpixie 4y ago
You're probably right about the effort - but I expect that such "boilerplate" code contains (or leads to) much more than 20% of the bugs. This 90% of code is not genuine word-for-word boilerplate (copy/pasted from a known good source). This code is typically constructed fresh each time; or worse, copied from somewhere similar and quickly tweaked for names/types! (I do it, and I see it done all the time.) I expect that the remaining 10% non-boilerplate code, taking 80% of the effort, is much more carefully considered, and less likely to contain those clumsy/forgetful off-by-one or buffer-overflow bugs.
- fiddlerwoaroof 4y ago
  Yeah, one of the interesting results in empirical studies of defect rates is that defect rate is influenced by lines of code more than other factors like “static types”. Similarly, analyses of defects have discovered that they tend to occur at the end of repetitive sequences of code, because the developer has sort of switched into autopilot mode. I think the obvious conclusion here (and my experience bears this out to some extent) is that languages and libraries that force boilerplate on you produce buggier code than languages and libraries that abstract the boilerplate away.
- kbenson 4y ago
  There's a reason many languages have large overflowing repositories of modules (or there are well known libraries) that can be downloaded and used that provide boilerplate solutions for many things. Most people don't like writing that boilerplate once they know how to do it and have done it a few times, and would rather just call a function do_that_thing_need_done_on(input1, input2). If it can't be factored out like that and is actual language boilerplate beyond a few lines, that's a failure of the language. If these AI models are suggesting the code that could be called in a library/module instead of the the code to actually include and call a well known and trusted library or module, I'm not sure that's progress. At least when someone notices a bug or better way to do it and updates that module or library, consumers of that module can update and benefit from it, or at a minimum see that there were bugs in the version they're running they might want to address at some point.

nradov 4y ago

Yes and I raised the same concern when GitHub Copilot was released. If our code contains so little entropy that an AI can reliably predict the next sequence of tokens then that is evidence we are operating at too low a level of abstraction. Such tools can certainly be helpful in working with today's popular languages, but what would a future language that allows for abstracting away all that boilerplate look like? Since this is HN I'm sure someone will say that the answer is Lisp. But can we do better?

tgv 4y ago
Gzipped Lisp would be an improvement in that reasoning. I’m sure you agree that’s not a very desirable way to write code.
- pharmakom 4y ago
  The number of bytes isn’t really significant. It’s the conceptual distance between what we want to achieve and the concrete steps in the code. Lisp may be an improvement.
  - tgv 4y ago
    But now you're already heading into much vaguer territory. Readability is also important. Very important, I would say. That requires easily identifiable markers for loops, conditions, functions, etc., something Lisp lacks. This might be a place where keyword coloring could be useful, but then we're relying on external help. Another issue is consistency. Take C, Javascript, or Go. Many loops are of the form for (var i = 0; i < n; i++) { ... } You could argue that "for i < n" provides the same information, but then you'd have to find a way to start the loop at a different offset, use a different end condition, or different "increment".
    - pharmakom 4y ago
      > But now you're already heading into much vaguer territory. Yes programming language design is a social science (imo)

ludovicianul 4y ago

This is exactly my view, especially with the web apps. If you take a distributed system, the majority of components/microservices will have more than 50% commonality in behaviour. Therefore you do mostly the same things when you start a new one. Even if code itself might be harder to generate, as even a CRUD app might have specific behaviour, testing it is definitely the same, especially when doing negative scenarios, boundary testing, CRUD operations, etc. I wrote a tool specifically for this purpose, targeted at REST APIs, aiming to automate this repetitive work and let you focus on the tests which are specific to the context.

andai 4y ago

>probability of each token appearing given the previous tokens Sounds like tokenizer -> Markov chain? Surely something trained on a TPU is more sophisticated than something we could have done in the middle of the 20th century?

IanCal 4y ago

Written language has this too. A basic lookup table of frequencies can tell you that jkkj is a typo in an English word. "Nobody else is really writing code that has the fragment in you just wrote" can find syntax errors. Better language models can find more subtle relationships. At some point the variations mean that more abstractions don't really help.

pharmakom 4y ago

I agree. And yet most developers claim JS, Python, Java etc are totally sufficient.

wikfwikf 4y ago
Perl is a wonderful, innovative language which failed because it tried to remove intratextual redundancy in the way you are suggesting. A string of length N is vastly more likely to be a valid Perl program than a valid Python program. Ultimately this meant that Perl programs, while easier to type, were much harder to read, and extremely easy to misinterpret.
- ilyt 4y ago
  There is also something to be said about nudging developer on the "right" way to do stuff. Perl is not only hard to read because there are many shortcuts that might look like line noise for the inexperienced (hell, Rust have a bunch of those too), it's because there is a bunch of the ways to do anything. Like take humble array Perl> @a $VAR1 = 1; $VAR2 = 2; $VAR3 = 3; $VAR4 = 4; $VAR5 = 5; Perl> @a + 2 $VAR1 = 7; Why adding a number to array results in number ? Because array in scalar context returns its length. It leads to some very compact code if (@a > 3) {print "big array"} but, uh, what you do if you want to see length of string ? well length($string). Will that also work for arrays ? Nope, if you want to force scalar context you're supposed to do scalar(@array). So how to add 2 arrays ? @c = (@a, @b) # obviously. But wait, what we really typed after variable expansion is ((1,2),(2,3)) and in other languages irb(main):001:0> a = [1,2] => [1, 2] irb(main):002:0> b=[2,3] => [2, 3] irb(main):003:0> c = [a,b] => [[1, 2], [2, 3]] >>> a = [1,2] >>> b = [2,3] >>> c = [a,b] >>> c [[1, 2], [2, 3]] that's exactly what we get. Confusing ? Sure. But it saves few characters!

constantcrying 4y ago

It really is the exact opposite approach to static analysis, which tries to see what the code really does andhow that leads to bugs. I have had (quite expensive) static analysis tools detect genuine bugs, e.g. a somewhat subtle overflow. What it can never detect though is correct code that misses the intention of the programmer. E.g. whether some mathematical function is accurate. The language models try, by statistical means, to derive what should be there. Given enough data they will start to have sone (statistical) grasp on the intention. I am not entirely sure about the boilerplate though. Often you need some minor variation of an already existing pattern. Trying to unify those slightly divergent patterns into one schema can very easily lead to very hard to understand code. Another thing is thar boilerplate is fairly easy to write and to test, because it is familiar, reducing the actual effort which goes into it. Sometimes it is just better to not reuse code.

kneebonian 4y ago
Honestly this is what I have loved about Kotlin. There seems like there is now just a certain amount of boilerplate in every Java file and Kotlin just chose to bake that all into the language, the other instance where we see this happening is with the Lombok library in Java. Although personally I hate annotations.
yccs27 4y ago
+1 on the last paragraph: Predictability means the code follows a pattern, not that it is boilerplate. Some amount of predictable code is necessary just to spell out what the code does, so that even someone unfamiliar with the pattern can simply read and understand it.
ajuc 4y ago
When you go too far reducing boilerplate you get to a point where configuration becomes the code and the actual code becomes black box that people barely understand. And then they replicate what's already in the black box because they aren't sure it's there, and you do the same things over and over in every layer. And in some layers you do it one way and in others - other way. And then requirements change and you change the code and it still works the old way SOME of the time, but you only discover that in production, because the test case you used when you developed is handled on the layer where you changed it correctly. And then if it's buried deep enough somebody will add another layer and fix the cases that were found - there. And that's how the disgusting legacy code happens. KISS, please. Unnecessary abstraction is the root of almost all problems in programming.

GuB-42 4y ago

I don't think so. Sure, most code is boilerplate, except for that one thing, and that one thing can be anywhere. For example, let's say you want to write a function that returns the checksum of a bunch of data. That's a very common thing to do, there are plenty of libraries that do that, and I have seen the CRC32 lookup table in many places, sometimes I am the one who put it there. Now, why rewrite such a function? - Ignore some part of the message - Use different constants - Fetch data in a special way (i.e. not a file or memory buffer) - Have some kind of a progress meter - The library you may want to use is not available (can be for technical, legal or policy reasons) - Some in-loop operation is needed (ex: byte swapping) - Have a specific termination condition (ex: end-of-message marker) - And many others, including combinations of the above If you ignore all these points and only see the generic checksum function, yes, it is boilerplate and can be factorized. But these special cases are the reason why it may not be the case, and the reason why there are so many coding jobs. It is also the reason why we don't have real (Lv5) self driving cars yet, why there are pilots in the cockpit, why MS Office and the like have so many "useless" features, why so many attempts to make software cleaner and simpler fail, etc...

julian37 4y ago
That's hardly a convincing example. All of these points can be solved elegantly with a stream abstraction, which can be cheap or free given a sufficiently advanced language and compiler. As for legal or policy reasons, those still aren't reasons to write boilerplate code. Your reimplementation can be tight and reuse other abstractions or include their own.
- GuB-42 4y ago
  A stream abstraction is a solution so some (not all) of these problems, and indeed some libraries use them, but a stream abstraction that is powerful enough to solve most of these problems may result in more complex code than just rewriting the checksum algorithm from scratch. And there is a limit on how compilers can optimize, especially considering that checksum calculation may be critical to performance. In reality, few people need to write their own checksumming function, but sometimes, it is the best thing to do. And it is just an example, there are many other instance where an off the shelf solution is not appropriate because of some detail: string manipulation, parsing, data structures (especially the "intrusive" kind), etc... And since you are probably going to have several of these in your project, it will result in a lot of boilerplate. If it was so generic not to require boilerplate, it probably has been developed already and you would be working on something else. Abstractions are almost invariably more complex, slower, more error-prone and generally worse than the direct equivalent. They are, however, reusable, that's the entire point. So one person goes through the pain of writing a nice library, and it makes life a little easier for the thousands of people who use it, generally, that's a win. But if you write an abstraction for a single use case, it is generally worse than boilerplate.

WithinReason 4y ago

I think you misunderstand machine learning. "probability of each token appearing given the previous tokens" is how humans write code too: We write code based on what we want to do and what we have written before. "what I want to do" was captured in the comments added.