6 ms·

If the results were stable, reproducable and somehow memoizable, this would be actually insanely useful. Perhaps it could modify a cache file somewhere with the

If the results were stable, reproducable and somehow memoizable, this would be actually insanely useful. Perhaps it could modify a cache file somewhere with the generated Python code, and that could be committed.

wging 4y ago
Per the README (and the source code), requests to GPT are cached. I know that doesn't solve the stability problem, since the cache is keyed on the exact prompt among other parameters, but it's something at least. The source code shows that it's using pickling to store OpenAI responses on the local filesystem. See here: https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L50-L56 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt... and here: https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L76-L87 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...
- ianbicking 4y ago
  It really should be saving them to JSON instead of Pickle, but I gave up trying to figure out how to properly rehydrate the openai Completion objects. If it was JSON then it wouldn't be crazy to add it to your repository. (I guess Pickle is stable enough in practice, but it would offend my personal sensibilities to check them in.)
  - HyperSane 4y ago
    What is wrong with Pickle?
    - andreareina 4y ago
      Arbitrary code execution is a big one.
      - HyperSane 4y ago
        I'm aware, but it is not an issue if you only use it with your own data. You can even cryptographically sign it.
    - ianbicking 4y ago
      Besides security it's pretty opaque, and unnecessarily so. The data all comes over JSON so there's no non-JSONable types involved. It would be nice to be able to run tools like cost summarizing over the JSON without having to load up any other code.
etaioinshrdlu 4y ago
The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the upgrade, they might just put out a new model id param...)
- varunkmohan 4y ago
  It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere. This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.
  - xyzzy123 4y ago
    Not promising determinism gives them a lot more implementation wiggle room to update / change things. My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.
  - swyx 4y ago
    Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s=46&t=L75UJ1fe78VCiBPUK6PddA https://twitter.com/taliaringer/status/1511411984398487564?s...
    - varunkmohan 4y ago
      Interesting, I guess we implemented this op properly :) It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.
- nasir 4y ago
  normally you can choose the model version so that's not a concern.
- ilaksh 4y ago
  They seem pretty consistent with temperature 0. Maybe not 100% but very close.
keuriGPT 4y ago
I'm no python expert but it looks like this does happen in memory: https://github.com/ianb/infinite-ai-array/blob/main/iaia/magicmodule.py#L23 https://github.com/ianb/infinite-ai-array/blob/main/iaia/mag... I can't imagine it's too hard to serialize the `existing` python dict so subsequent runs are deterministic
- wging 4y ago
  It is also caching requests on-disk. See for instance https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L76-L87 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt... The problem* is that if a prompt changes slightly then it won't be a cache hit. * okay, one problem.
samwillis 4y ago
Looks like Dill [0] would be a good option for serialising the generated code. The built in Python Pickle doesn't support pickling functions properly, but Dill extends it to enable this sort of functionality. 0: https://pypi.org/project/dill/ https://pypi.org/project/dill/
recuter 4y ago
In what manner do you reckon would it be insanely useful?

wging 4y ago

Per the README (and the source code), requests to GPT are cached. I know that doesn't solve the stability problem, since the cache is keyed on the exact prompt among other parameters, but it's something at least. The source code shows that it's using pickling to store OpenAI responses on the local filesystem. See here: https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L50-L56 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt... and here: https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L76-L87 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt...

ianbicking 4y ago
It really should be saving them to JSON instead of Pickle, but I gave up trying to figure out how to properly rehydrate the openai Completion objects. If it was JSON then it wouldn't be crazy to add it to your repository. (I guess Pickle is stable enough in practice, but it would offend my personal sensibilities to check them in.)
- HyperSane 4y ago
  What is wrong with Pickle?
  - andreareina 4y ago
    Arbitrary code execution is a big one.
    - HyperSane 4y ago
      I'm aware, but it is not an issue if you only use it with your own data. You can even cryptographically sign it.
  - ianbicking 4y ago
    Besides security it's pretty opaque, and unnecessarily so. The data all comes over JSON so there's no non-JSONable types involved. It would be nice to be able to run tools like cost summarizing over the JSON without having to load up any other code.

etaioinshrdlu 4y ago

The backend OpenAI APIs are not deterministic even with temperature 0, and they might upgrade the model weights/tuning behind your back too? (Not sure about the upgrade, they might just put out a new model id param...)

varunkmohan 4y ago
It's very strange that they don't guarantee determinism. We trained a model at Codeium that does have determinism at inference time - it's not super hard if you don't just use random seeds everywhere. This makes it a lot harder to rely on OpenAI's API if you are doing any sort of caching / evaluation workloads.
- xyzzy123 4y ago
  Not promising determinism gives them a lot more implementation wiggle room to update / change things. My guess is that promising it right now is not a priority. It would be a great "pro" or "enterprise" type feature tho.
- swyx 4y ago
  Pytorch/CUDA is apparently sometimes nondeterministic https://twitter.com/taliaringer/status/1511411984398487564?s=46&t=L75UJ1fe78VCiBPUK6PddA https://twitter.com/taliaringer/status/1511411984398487564?s...
  - varunkmohan 4y ago
    Interesting, I guess we implemented this op properly :) It is strange that cub does not guarantee determinism here - it's not too hard an op to implement in CUDA.
nasir 4y ago
normally you can choose the model version so that's not a concern.
ilaksh 4y ago
They seem pretty consistent with temperature 0. Maybe not 100% but very close.

keuriGPT 4y ago

I'm no python expert but it looks like this does happen in memory: https://github.com/ianb/infinite-ai-array/blob/main/iaia/magicmodule.py#L23 https://github.com/ianb/infinite-ai-array/blob/main/iaia/mag... I can't imagine it's too hard to serialize the `existing` python dict so subsequent runs are deterministic

wging 4y ago
It is also caching requests on-disk. See for instance https://github.com/ianb/infinite-ai-array/blob/main/iaia/gptclient.py#L76-L87 https://github.com/ianb/infinite-ai-array/blob/main/iaia/gpt... The problem* is that if a prompt changes slightly then it won't be a cache hit. * okay, one problem.

samwillis 4y ago

Looks like Dill [0] would be a good option for serialising the generated code. The built in Python Pickle doesn't support pickling functions properly, but Dill extends it to enable this sort of functionality. 0: https://pypi.org/project/dill/ https://pypi.org/project/dill/

recuter 4y ago

In what manner do you reckon would it be insanely useful?