3 ms·

I'm worried about the version hell of relying on HuggingFace's transformers. In the model card : config.json [1] and generation_config.json there are strings "

I'm worried about the version hell of relying on HuggingFace's transformers. In the model card : config.json [1] and generation_config.json there are strings "transformers_version": "4.28.1" and it refers to some common architecture "GPTNeoXForCausalLM" . Which if I follow the string soft link correctly means, it refers to this file [2] with a long history of commits including some recents [3]. I don't understand how to get out of the version hell. I can install and pin a version of the transformer library, but sooner or later if I chain different foundation models I'll have to do this for each different model card that has its own version. And then those transformer library version are probably dependent on some pytorch versions. I don't understand how you can't reliably track a bug, or a change in behavior, or guarantee that the model that you spend millions of dollar training doesn't rust, is reproducible or become unusable due to this mess. (And also the weights are de-serialized pickled python dictionary which is from a safety point of view equivalent to running untrusted binaries and very worrying on its own.) [1]https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b/blob/main/config.json https://huggingface.co/stabilityai/stablelm-tuned-alpha-7b/b... [2]https://github.com/huggingface/transformers/blob/v4.28.1/src/transformers/models/gpt_neox/modeling_gpt_neox.py https://github.com/huggingface/transformers/blob/v4.28.1/src... [3]https://github.com/huggingface/transformers/commits/v4.28.1/src/transformers/models/gpt_neox/modeling_gpt_neox.py https://github.com/huggingface/transformers/commits/v4.28.1/...

jxy 4y ago
In terms of pickled files, we may ask them to release the models in safetensors. That would be a big improvement. There's not much we can do about dependencies on pytorch or other python libraries. Perhaps people can make more independent implementations. The redundancy in implementations would help.
- substation13 4y ago
  I'm surprised that some of the cutting edge AI work is not being done in pure Rust. Why even have a binding layer?
  - ElFitz 4y ago
    Python seems to have won over the Data / ML world. So I’d wager they use what they and their intended audience know.
    - tough 4y ago
      Someone implemented llama on Go recently, https://github.com/gotzmann/llama.go https://github.com/gotzmann/llama.go Wouldn't discard a rust implementation of some LLM architecture at some point
      - ElFitz 4y ago
        There’s also ggml, which is quite impressive considering they get whisper and other models to run on a phone.
      - Philpax 4y ago
        Hi! I'm a maintainer of https://github.com/rustformers/llama-rs https://github.com/rustformers/llama-rs. We're planning to expand our model support soon.
  - MacsHeadroom 4y ago
    https://github.com/rustformers/llama-rs https://github.com/rustformers/llama-rs
- sdenton4 4y ago
  The pickles drive me crazy. Tensorflow saved models are a great way to solve the problem... Save the computation graph and weights, and drop all the crusty code dependencies. I think ONNX models are similar. I expect there should be a Jax equivalent at some point, as Jax is basically perfectly designed for this (everything is expressed in lax operations, which allows changing implementations for cpu/gpu/tpu freely... So just save the list of lax ops).
lhl 4y ago
I've been diving in lately and while it's not efficient, the only way to do manage is to create a new conda/mamba environment, or a custom Docker image for all the conflicting packages. For safety and speed, you should prefer the safetensor format: https://huggingface.co/docs/safetensors/speed https://huggingface.co/docs/safetensors/speed If you know what you are doing you can do your own conversions: https://github.com/huggingface/safetensors https://github.com/huggingface/safetensors or for safety, https://huggingface.co/spaces/diffusers/convert https://huggingface.co/spaces/diffusers/convert
brucethemoose2 4y ago
> And then those transformer library version are probably dependent on some pytorch versions. They are not, and I dont think the model even cares about the transformers version. I run git transformers/diffusers and PyTorch 2.1 in all sorts of old repos, and if it doesnt immediately work, usually theres just small changes to APIs here and there that make scripts unhappy, and that you can manually fix.

jxy 4y ago

In terms of pickled files, we may ask them to release the models in safetensors. That would be a big improvement. There's not much we can do about dependencies on pytorch or other python libraries. Perhaps people can make more independent implementations. The redundancy in implementations would help.

substation13 4y ago
I'm surprised that some of the cutting edge AI work is not being done in pure Rust. Why even have a binding layer?
- ElFitz 4y ago
  Python seems to have won over the Data / ML world. So I’d wager they use what they and their intended audience know.
  - tough 4y ago
    Someone implemented llama on Go recently, https://github.com/gotzmann/llama.go https://github.com/gotzmann/llama.go Wouldn't discard a rust implementation of some LLM architecture at some point
    - ElFitz 4y ago
      There’s also ggml, which is quite impressive considering they get whisper and other models to run on a phone.
    - Philpax 4y ago
      Hi! I'm a maintainer of https://github.com/rustformers/llama-rs https://github.com/rustformers/llama-rs. We're planning to expand our model support soon.
- MacsHeadroom 4y ago
  https://github.com/rustformers/llama-rs https://github.com/rustformers/llama-rs
sdenton4 4y ago
The pickles drive me crazy. Tensorflow saved models are a great way to solve the problem... Save the computation graph and weights, and drop all the crusty code dependencies. I think ONNX models are similar. I expect there should be a Jax equivalent at some point, as Jax is basically perfectly designed for this (everything is expressed in lax operations, which allows changing implementations for cpu/gpu/tpu freely... So just save the list of lax ops).

lhl 4y ago

I've been diving in lately and while it's not efficient, the only way to do manage is to create a new conda/mamba environment, or a custom Docker image for all the conflicting packages. For safety and speed, you should prefer the safetensor format: https://huggingface.co/docs/safetensors/speed https://huggingface.co/docs/safetensors/speed If you know what you are doing you can do your own conversions: https://github.com/huggingface/safetensors https://github.com/huggingface/safetensors or for safety, https://huggingface.co/spaces/diffusers/convert https://huggingface.co/spaces/diffusers/convert

brucethemoose2 4y ago

> And then those transformer library version are probably dependent on some pytorch versions. They are not, and I dont think the model even cares about the transformers version. I run git transformers/diffusers and PyTorch 2.1 in all sorts of old repos, and if it doesnt immediately work, usually theres just small changes to APIs here and there that make scripts unhappy, and that you can manually fix.