3 ms·

Im not an expert, and I don't have nvidia, but I assume you need to setup CUDA and install the CUDA pytorch stuff? Most docs Ive read on setting up finetuners

Im not an expert, and I don't have nvidia, but I assume you need to setup CUDA and install the CUDA pytorch stuff? Most docs Ive read on setting up finetuners and inference require some extra stuff. Taking some LORA fine tuners, they include instructions like this: conda create -n llm-finetuner python=3.10 conda activate llm-finetuner conda install -y cuda -c nvidia/label/cuda-11.7.0 conda install -y pytorch=2 pytorch-cuda=11.7 -c pytorch When I experimented with Stable Diffusion and ROCM (amd card), i had to do similar but with pythorch-rocm. and when I was doing a CPU only, did `pytorch-cpu`. So maybe your attempt didn't use the GPUs at all, because 12 mins is about what I had on a CPU for inference on other models of similar size.

zamnos 3y ago
The error message implies that the compiled default libraries on the M1 don't support the model format, even though it works fine in Paperspace. The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Traceback (most recent call last): File "/Users/fragmede/projects/llm/dolly/foo.py", line 5, in <module> instruct_pipeline = pipeline( ^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 776, in pipeline framework, model = infer_framework_load_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/base.py", line 271, in infer_framework_load_model raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.") ValueError: Could not load model databricks/dolly-v2-12b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.gpt_neox.modeling_gpt_neox.GPTNeoXForCausalLM'>).
- Daegalus 3y ago
  I was referring to his TIL post about setting it up on paperspace, not about apple hardware.
  - zamnos 3y ago
    ah, apologies, i misread your comment and was more excited to share since I was able to try on my system.
    - Daegalus 3y ago
      No worries, it happens. I will admit the way I answered wasn't clear that I was referring to the linked page and not the question in the post. All good.

zamnos 3y ago

The error message implies that the compiled default libraries on the M1 don't support the model format, even though it works fine in Paperspace. The argument `trust_remote_code` is to be used with Auto classes. It has no effect here and is ignored. Traceback (most recent call last): File "/Users/fragmede/projects/llm/dolly/foo.py", line 5, in <module> instruct_pipeline = pipeline( ^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/__init__.py", line 776, in pipeline framework, model = infer_framework_load_model( ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/transformers/pipelines/base.py", line 271, in infer_framework_load_model raise ValueError(f"Could not load model {model} with any of the following classes: {class_tuple}.") ValueError: Could not load model databricks/dolly-v2-12b with any of the following classes: (<class 'transformers.models.auto.modeling_auto.AutoModelForCausalLM'>, <class 'transformers.models.gpt_neox.modeling_gpt_neox.GPTNeoXForCausalLM'>).

Daegalus 3y ago
I was referring to his TIL post about setting it up on paperspace, not about apple hardware.
- zamnos 3y ago
  ah, apologies, i misread your comment and was more excited to share since I was able to try on my system.
  - Daegalus 3y ago
    No worries, it happens. I will admit the way I answered wasn't clear that I was referring to the linked page and not the question in the post. All good.