3 ms·

I'd really like to see how this does with nontrivial examples. I'm happy to be proven wrong, but I bet the bash one isn't that effective for real world sysadmin

by 4oo4 4y ago

I'd really like to see how this does with nontrivial examples. I'm happy to be proven wrong, but I bet the bash one isn't that effective for real world sysadmin tasks, since most bash things also require knowledge of the commands in-between the actual bash "glue", and usually requires regexes and pipelines to get the exact answer you want. Pretty much every time I look up a bash command (probably from the same sources this model trains on) I have to tweak it to do what I need. For example, here are some things I pulled from `.bash_history`, and what I think I was meaning to do. # Get the IP address of running lxc container ${CONTAINER_NAME} CONTAINER_NAME=some-container; lxc-ls -f | grep ${CONTAINER_NAME} | awk '{print $5}' | sed -z 's|-.*\s||g;' # Find the latest git tag version, excluding release candidates and development builds git describe --tags `git rev-list --tags --exclude=*-rc* --max-count=100` | grep -vP '(\-rc|dev)' | head -1 # get the database name from parameters.yml grep 'database_name:' parameters.yml | sed 's|.*:||g; s|\s||g;' I bet #3 would would be feasible, but I'd be extremely surprised if it could get close on the first two. PowerShell has the advantage of a really consistent standard library, syntax, objects, and has the .NET type system behind it, so any answer is probably already in the format you want or requires minimal manipulation. I can see it working much better there, and that's also probably why the example video only shows PowerShell too. Granted, I am probably the opposite of the target audience. All that being said, would you really trust running some random commands you don't understand enough to write, because some remote server told you to? It should tell you the commands first rather than just executing them.

not_a_bot_ryan 4y ago
You should try those! You can easily create your own few examples of complex behavior (like the ones you already have), add them to a prompt file, load that prompt and try some other hard queries. I've been quite surprised at what it can handle with the right few examples
delta_p_delta_x 4y ago
> It should tell you the commands first rather than just executing them. From the videos, I believe you have to press Enter to actually execute the suggested command. > why the example video only shows PowerShell too The readme did say that it was inspired by a similar project targetting zsh[0], so I don't see why over time the model couldn't improve to suggest what you want. [0]: https://github.com/tom-doerr/zsh_codex https://github.com/tom-doerr/zsh_codex
- 4oo4 4y ago
  > From the videos, I believe you have to press Enter to actually execute the suggested command. Cool, that's good to hear if that's the case. > The readme did say that it was inspired by a similar project targetting zsh[0], so I don't see why over time the model couldn't improve to suggest what you want. Interesting, I know very little about all the cool stuff that zsh can do compared with bash, I suppose if it works well for zsh it could probably work for bash then, since zsh is more complex? Personally I can't see myself ever using these kinds of tools, even if it fit my usecase. In part because I like to figure things out for myself, and also because I don't like outsourcing decisions to unaccountable ML models. I think these are cool ideas, but I think in the long run they will cause more problems than they solve, just like no-code development platforms. IMO all ML models need to be able to explain themselves before we can trust them with anything important.
  - not_a_bot_ryan 4y ago
    Disclaimer - I worked on this Codex CLI prototype. You definitely do have to press enter to execute the command - as with Copilot, you - the developer - can, and should, retain control. I also like figuring things out myself, but find myself using this tool to help me do so. If I'm using kubectl to run some esoteric command, I'd rather not leave the shell to figure it out - my usage pattern is usually to ask once or twice, and then use that knowledge to handle it on my own the next time
    - 4oo4 4y ago
      That is a really good use case I didn't think about, since I'm so used to having to leave the terminal to look stuff up in a browser when I can't find something with a manpage or `--help`/`Get-Help`. I'll admit I'm probably being overly skeptical (as I am with a lot of ML things in general), but regardless I think this is really cool, and you guys did a great job on it! I should probably actually try it instead of trying to second guess it. Is there a way to have it just echo the output instead of having the input ready for you to press enter? Only asking since I'm a clumsy person at the keyboard sometimes :)

by 4oo4 4y ago

not_a_bot_ryan 4y ago

You should try those! You can easily create your own few examples of complex behavior (like the ones you already have), add them to a prompt file, load that prompt and try some other hard queries. I've been quite surprised at what it can handle with the right few examples

delta_p_delta_x 4y ago

> It should tell you the commands first rather than just executing them. From the videos, I believe you have to press Enter to actually execute the suggested command. > why the example video only shows PowerShell too The readme did say that it was inspired by a similar project targetting zsh[0], so I don't see why over time the model couldn't improve to suggest what you want. [0]: https://github.com/tom-doerr/zsh_codex https://github.com/tom-doerr/zsh_codex

4oo4 4y ago
> From the videos, I believe you have to press Enter to actually execute the suggested command. Cool, that's good to hear if that's the case. > The readme did say that it was inspired by a similar project targetting zsh[0], so I don't see why over time the model couldn't improve to suggest what you want. Interesting, I know very little about all the cool stuff that zsh can do compared with bash, I suppose if it works well for zsh it could probably work for bash then, since zsh is more complex? Personally I can't see myself ever using these kinds of tools, even if it fit my usecase. In part because I like to figure things out for myself, and also because I don't like outsourcing decisions to unaccountable ML models. I think these are cool ideas, but I think in the long run they will cause more problems than they solve, just like no-code development platforms. IMO all ML models need to be able to explain themselves before we can trust them with anything important.
- not_a_bot_ryan 4y ago
  Disclaimer - I worked on this Codex CLI prototype. You definitely do have to press enter to execute the command - as with Copilot, you - the developer - can, and should, retain control. I also like figuring things out myself, but find myself using this tool to help me do so. If I'm using kubectl to run some esoteric command, I'd rather not leave the shell to figure it out - my usage pattern is usually to ask once or twice, and then use that knowledge to handle it on my own the next time
  - 4oo4 4y ago
    That is a really good use case I didn't think about, since I'm so used to having to leave the terminal to look stuff up in a browser when I can't find something with a manpage or `--help`/`Get-Help`. I'll admit I'm probably being overly skeptical (as I am with a lot of ML things in general), but regardless I think this is really cool, and you guys did a great job on it! I should probably actually try it instead of trying to second guess it. Is there a way to have it just echo the output instead of having the input ready for you to press enter? Only asking since I'm a clumsy person at the keyboard sometimes :)