6 ms·

Not really, code even in high level languages is always lower level than English just for computer nonsense reasons. Example: "read a CSV file and add a column

by qcnguy 1y ago

Not really, code even in high level languages is always lower level than English just for computer nonsense reasons. Example: "read a CSV file and add a column containing the multiple of the price and quantity columns". That's about 20 words. Show me the programming language that can express that entire feature in 20 words. Even very English-like languages like Python or Kotlin might just about do it, if you're working in something else like C++ then no. In practice, this spec will expand to changes to your dependency lists (and therefore you must know what library is used for CSV parsing in your language, the AI knows this stuff better than you), then there's some file handling, error handling if the file doesn't exist, maybe some UI like flags or other configuration, working out what the column names are, writing the loop, saving it back out, writing unit tests. Any reasonable programmer will produce a very similar PR given this spec but the diff will be much larger than the spec.

cess11 1y ago
Iverson languages could do that quite succinctly.
sarchertech 1y ago
>"read a CSV file and add a column containing the multiple of the price and quantity columns" This is an underspecification if you want to reliably repeatably produce similar code. The biggest difference is that some developers will read the whole CSV into memory before doing the computations. In practice the difference between those implementation is huge. Another big difference is how you represent the price field. If you parse them as floats and the quantity is big enough, you'll end up with errors. Even if quantity is small, you'll have to deal with rounding in your new column. You didn't even specific the name of the new column, so the name is going to be different every time you run the LLM. What happens if you run this on a file the program has already been ran on? And these are just a few of the reasonable ways of fitting that spec but producing wildly different programs. Making a spec that has a good chance of producing a reasonably similar program each time looks more like: “Read input.csv (UTF-8, comma-delimited, header row). Read it line by line, do not load the entire file into memory. Parse the price and quantity columns as numbers, stripping currency symbols and thousands separators; interpret decimals using a dot (.). Treat blanks as null and leave the result null for such rows. Compute per-row line_total = round(Decimal(price) * Decimal(quantity), 2). Append line_total as the last column (name the column "Total") without reordering existing columns, and write to output.csv, preserving quoting and delimiter. Do not overwrite existing columns. Do not evaluate or emit spreadsheet formulas.” And even then you couldn't just check this in and expect the same code to be generated each time, you'd need a large test suite--just to constraint the LLM. And even then the LLM would still occasionally find ways to generate code that passes the tests but does thing you don't want it to.
- qcnguy 1y ago
  But why would I want to reliably produce similar code? The underspec is deliberate. Maybe I don't care about the name of the column as long as it's reasonable. How to represent prices: same. This is computer nonsense. There's one right way to do it, the LLM knows that way, it should do it. How to do it scalably: same. If the file is named the agent can just look at its size to decide on the best implementation. Your alternative spec is too detailed and has many details that can be easily inferred by the AI, like defaulting to UTF-8 and comma delimited. This is my point. There are many possible implementations in code, some better and some worse, and we shouldn't need to spell out all that detail in English when so much of it is just about implementation quality.
  - sarchertech 1y ago
    >But why would I want to reliably produce similar code? If you're doing a one short CSV then an LLM or a custom program is the wrong way to do it. Any spreadsheet editor can do this task instantly with 4 symbols. Assuming you want a repeatable process you need to define that repeatable process with enough specificity to make it repeatable and reliable. You can do this in a formal language created for this or you can do invent your own English like specification language. You can create a very loose specification and let someone else, a programmer or an LLM define the reliable, repeatable process for you. If you go with a junior programmer or an LLM though, you have to verify that the process they designed is actually reliable and repeatable. Many times it won't be and you'll need to make changes. It's easier to write a few lines of python than to go through that process--unless you don't already know how to program, in which case you can't verify the output anyway. That's not to say that I don't see beneficial use cases for AI, this just isn't one of them. >This is my point. There are many possible implementations in code, some better and some worse, and we shouldn't need to spell out all that detail in English when so much of it is just about implementation quality. If you don't actually care about implementation quality or correctness, sure. You should, and LLMs can not reliably pick the correct implementation details. They aren't even close to being able to do that. The only people who are able to produce working software with LLMs are either writing very very detailed specifications. To the point where they aren't operating at a much higher level than Python. Btw I had a Claude Sonnet 4 agent try your prompt. It produced a 90 line python file in 7 minutes that reads the entire file into memory, performs floating point multiplication, doesn't correctly display the money values, and would crash if the price column ever had any currency symbols.
foobarbecue 1y ago
> Show me the programming language that can express that entire feature in 20 words. In python: import pandas mycsv = pandas.read_csv("/path/to/input.csv") mycsv['total_cost'] = mycsv.price*mycsv.quantity Not only is this shorter, but it contains all of the critical information that you left out of your english prompt: where is the csv? what are the input columns named? what are output columns named? what do you want to do with the output? I also find it easier to read than your english prompt.
- miyoji 1y ago
  > `mycsv = pandas.read_csv("/path/to/input.csv")` You have to count the words in the functions you call to get the correct length of the implementation, which in this case is far far more than 20 words. read_csv has more than 20 arguments, you can't even write the function definition in under 20 words. Otherwise, I can run every program by importing one function (or an object with a single method, or what have you) and just running that function. That is obviously a stupid way to count.
  - JambalayaJimbo 1y ago
    You actually have to count the number of bytes in the generated machine code to get the real count
    - foobarbecue 1y ago
      Ok but how much physical space do those bytes take up? Need to measure them.
  - foobarbecue 1y ago
    I really can't tell if this is meant as a joke. Anyway, I just wrote what I, personally, would type in a normal work day to accomplish this coding task.
    - miyoji 1y ago
      It isn't a joke, you need the Kolmogorov complexity of the code that implements the feature, which has nothing to do with the fact that you're using someone else's solution. You may not have to think about all the code needed to parse a CSV, but someone did and that's a cost of the feature, whether you want to think about it or not. Again, if someone else writes a 100,000 line function for you, and they wrap it in a "do_the_thing()" method, you calling it is still calling a 100,000 line function, the computer still has to run those lines and if something goes wrong, SOMEONE has to go digging in it. Ignoring the costs you don't pay is ridiculous.
datadrivenangel 1y ago
Full code with prebuilt libraries/packages/components will be the winning setup.

by qcnguy 1y ago

cess11 1y ago

Iverson languages could do that quite succinctly.

sarchertech 1y ago

>"read a CSV file and add a column containing the multiple of the price and quantity columns" This is an underspecification if you want to reliably repeatably produce similar code. The biggest difference is that some developers will read the whole CSV into memory before doing the computations. In practice the difference between those implementation is huge. Another big difference is how you represent the price field. If you parse them as floats and the quantity is big enough, you'll end up with errors. Even if quantity is small, you'll have to deal with rounding in your new column. You didn't even specific the name of the new column, so the name is going to be different every time you run the LLM. What happens if you run this on a file the program has already been ran on? And these are just a few of the reasonable ways of fitting that spec but producing wildly different programs. Making a spec that has a good chance of producing a reasonably similar program each time looks more like: “Read input.csv (UTF-8, comma-delimited, header row). Read it line by line, do not load the entire file into memory. Parse the price and quantity columns as numbers, stripping currency symbols and thousands separators; interpret decimals using a dot (.). Treat blanks as null and leave the result null for such rows. Compute per-row line_total = round(Decimal(price) * Decimal(quantity), 2). Append line_total as the last column (name the column "Total") without reordering existing columns, and write to output.csv, preserving quoting and delimiter. Do not overwrite existing columns. Do not evaluate or emit spreadsheet formulas.” And even then you couldn't just check this in and expect the same code to be generated each time, you'd need a large test suite--just to constraint the LLM. And even then the LLM would still occasionally find ways to generate code that passes the tests but does thing you don't want it to.

qcnguy 1y ago
But why would I want to reliably produce similar code? The underspec is deliberate. Maybe I don't care about the name of the column as long as it's reasonable. How to represent prices: same. This is computer nonsense. There's one right way to do it, the LLM knows that way, it should do it. How to do it scalably: same. If the file is named the agent can just look at its size to decide on the best implementation. Your alternative spec is too detailed and has many details that can be easily inferred by the AI, like defaulting to UTF-8 and comma delimited. This is my point. There are many possible implementations in code, some better and some worse, and we shouldn't need to spell out all that detail in English when so much of it is just about implementation quality.
- sarchertech 1y ago
  >But why would I want to reliably produce similar code? If you're doing a one short CSV then an LLM or a custom program is the wrong way to do it. Any spreadsheet editor can do this task instantly with 4 symbols. Assuming you want a repeatable process you need to define that repeatable process with enough specificity to make it repeatable and reliable. You can do this in a formal language created for this or you can do invent your own English like specification language. You can create a very loose specification and let someone else, a programmer or an LLM define the reliable, repeatable process for you. If you go with a junior programmer or an LLM though, you have to verify that the process they designed is actually reliable and repeatable. Many times it won't be and you'll need to make changes. It's easier to write a few lines of python than to go through that process--unless you don't already know how to program, in which case you can't verify the output anyway. That's not to say that I don't see beneficial use cases for AI, this just isn't one of them. >This is my point. There are many possible implementations in code, some better and some worse, and we shouldn't need to spell out all that detail in English when so much of it is just about implementation quality. If you don't actually care about implementation quality or correctness, sure. You should, and LLMs can not reliably pick the correct implementation details. They aren't even close to being able to do that. The only people who are able to produce working software with LLMs are either writing very very detailed specifications. To the point where they aren't operating at a much higher level than Python. Btw I had a Claude Sonnet 4 agent try your prompt. It produced a 90 line python file in 7 minutes that reads the entire file into memory, performs floating point multiplication, doesn't correctly display the money values, and would crash if the price column ever had any currency symbols.

foobarbecue 1y ago

> Show me the programming language that can express that entire feature in 20 words. In python: import pandas mycsv = pandas.read_csv("/path/to/input.csv") mycsv['total_cost'] = mycsv.price*mycsv.quantity Not only is this shorter, but it contains all of the critical information that you left out of your english prompt: where is the csv? what are the input columns named? what are output columns named? what do you want to do with the output? I also find it easier to read than your english prompt.

miyoji 1y ago
> `mycsv = pandas.read_csv("/path/to/input.csv")` You have to count the words in the functions you call to get the correct length of the implementation, which in this case is far far more than 20 words. read_csv has more than 20 arguments, you can't even write the function definition in under 20 words. Otherwise, I can run every program by importing one function (or an object with a single method, or what have you) and just running that function. That is obviously a stupid way to count.
- JambalayaJimbo 1y ago
  You actually have to count the number of bytes in the generated machine code to get the real count
  - foobarbecue 1y ago
    Ok but how much physical space do those bytes take up? Need to measure them.
- foobarbecue 1y ago
  I really can't tell if this is meant as a joke. Anyway, I just wrote what I, personally, would type in a normal work day to accomplish this coding task.
  - miyoji 1y ago
    It isn't a joke, you need the Kolmogorov complexity of the code that implements the feature, which has nothing to do with the fact that you're using someone else's solution. You may not have to think about all the code needed to parse a CSV, but someone did and that's a cost of the feature, whether you want to think about it or not. Again, if someone else writes a 100,000 line function for you, and they wrap it in a "do_the_thing()" method, you calling it is still calling a 100,000 line function, the computer still has to run those lines and if something goes wrong, SOMEONE has to go digging in it. Ignoring the costs you don't pay is ridiculous.

datadrivenangel 1y ago

Full code with prebuilt libraries/packages/components will be the winning setup.