3 ms·
sometimes people writing stack overflow answers stop to think, or test a piece of code they are going to suggest as a solution, or even go back and edit words t
by kragen 3y ago
sometimes people writing stack overflow answers stop to think, or test a piece of code they are going to suggest as a solution, or even go back and edit words they've already written
this is also true of books, sometimes, and even blog posts
consequently, a model that models the processes of stopping to think or testing code, or considers whether a given word might be revised after more thought, will do better at predicting the next word of stack overflow answers or books or blog posts than one that only models humans' reactive free association processes
therefore andrew gelman is mistaken about the limitations of that loss function
that logic of course doesn't imply anything in particular about transformers or any other ann architecture, but we can, i hope, observe that currently deployed software is already doing such modeling to some extent