3 ms·

> Another thing that's kind of scary (and makes it worrying if this is used for Copilot) is the second prompt to make the text uppercase results in code that is

> Another thing that's kind of scary (and makes it worrying if this is used for Copilot) is the second prompt to make the text uppercase results in code that is superficially correct, but is very semantically wrong - innerHTML.toUpperCase() is dangerous because it not only makes the content uppercase, it also modifies the attributes on the HTML elements inside. This definitely broke the vote button, which uses inline JS which is case sensitive. It also destroys any attached event handler since the elements are basically deleted then re-created. This is actually an issue I have with all these Transformer-based code generators - they have no inherent constraints on safe and correct code and often seem to generate superficially correct but bad and potentially even dangerous code. I remember that the first Copilot showcase also included stuff like that (not to mention that it sometimes generates GPL'd code). All the model does is a very complex form of association learning. It may "understand" the relationship between English and various programming languages, but you cannot code in any constraints about optimization, security, licensing etc. There is so much bad code out there on the internet and this model may have seen a lot of it. It's also no coincidence that most demos shown so far are very high level dynamic languages like Javascript and Python.

tectonic 5y ago
Completely agree. It currently tends to write unsafe, error-prone code. The next step is to figure out how to rein it in, either with new techniques or rejection sampling from a large set of possible outputs.
smitop 5y ago
With some prompt engineering, you can get Codex to produce better results. In these examples I wrote up to `makeUpper`, Codex wrote the rest (with temperature = 0): // JavaScript one-liner to make the text of element with ID athing uppercase const makerUpper = function(id) { document.getElementById(id).innerHTML = document.getElementById(id).innerHTML.toUpperCase(); }; vs // JavaScript one-liner to make the text of element with ID athing uppercase while following all security best practices const makerUppercase = function(id) { const element = document.getElementById(id); element.textContent = element.textContent.toUpperCase(); };
- mediumdeviation 5y ago
  The second result is more semantically correct, but it will not function if called on tr.athing because tr.athing contains HTML elements that will be deleted when you replace the text. It is still much safer than innerHTML which will silently corrupt attributes. It's also interesting you need to prompt Codex for security best practices (and a bit questionable if it even "knows" anything about best practices) I guess part of it is that a one-liner is impossible. Here's what I would write given the prompt const makeUppercase = (id) => { const element = document.getElementById(id); if (element == null) return; const makeChildNodeUpper = (node) => { if (node.nodeType === Node.TEXT_NODE) { node.nodeValue = node.nodeValue.toUpperCase(); } else { node.childNodes.forEach(makeChildNodeUpper); } } makeChildNodeUpper(element); }
  - IdiocyInAction 5y ago
    > It's also interesting you need to prompt Codex for security best practices Well, that's one of the central lessons of ML - garbage in, garbage out. There is a lot of garbage code out there and no easy a priori way to distinguish garbage code from good code.
  - chrismorgan 5y ago
    Here’s a pretty good one-liner to make the text uppercase: document.getElementById(id).style.textTransform = 'uppercase';
    - parhamn 5y ago
      That was in their first comment.

tectonic 5y ago

Completely agree. It currently tends to write unsafe, error-prone code. The next step is to figure out how to rein it in, either with new techniques or rejection sampling from a large set of possible outputs.

smitop 5y ago

With some prompt engineering, you can get Codex to produce better results. In these examples I wrote up to `makeUpper`, Codex wrote the rest (with temperature = 0): // JavaScript one-liner to make the text of element with ID athing uppercase const makerUpper = function(id) { document.getElementById(id).innerHTML = document.getElementById(id).innerHTML.toUpperCase(); }; vs // JavaScript one-liner to make the text of element with ID athing uppercase while following all security best practices const makerUppercase = function(id) { const element = document.getElementById(id); element.textContent = element.textContent.toUpperCase(); };

mediumdeviation 5y ago
The second result is more semantically correct, but it will not function if called on tr.athing because tr.athing contains HTML elements that will be deleted when you replace the text. It is still much safer than innerHTML which will silently corrupt attributes. It's also interesting you need to prompt Codex for security best practices (and a bit questionable if it even "knows" anything about best practices) I guess part of it is that a one-liner is impossible. Here's what I would write given the prompt const makeUppercase = (id) => { const element = document.getElementById(id); if (element == null) return; const makeChildNodeUpper = (node) => { if (node.nodeType === Node.TEXT_NODE) { node.nodeValue = node.nodeValue.toUpperCase(); } else { node.childNodes.forEach(makeChildNodeUpper); } } makeChildNodeUpper(element); }
- IdiocyInAction 5y ago
  > It's also interesting you need to prompt Codex for security best practices Well, that's one of the central lessons of ML - garbage in, garbage out. There is a lot of garbage code out there and no easy a priori way to distinguish garbage code from good code.
- chrismorgan 5y ago
  Here’s a pretty good one-liner to make the text uppercase: document.getElementById(id).style.textTransform = 'uppercase';
  - parhamn 5y ago
    That was in their first comment.