3 ms·
Almost there. I've got everything installed and want to run your inference script. Model loading fails, TypeError: Got unsupported ScalarType BFloat16 File
by dekhn 4y ago
Almost there. I've got everything installed and want to run your inference script. Model loading fails, TypeError: Got unsupported ScalarType BFloat16
File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/utils/offload.py", line 25, in offload_weight
array = weight.numpy()
I verified my numpy knows nothing of bfloat16:
>>> import torch
>>> t=torch.arange(0,6).resize(2,3).to(dtype=torch.bfloat16)
/home/dek/miniconda3/lib/python3.9/site-packages/torch/_tensor.py:586: UserWarning: non-inplace resize is deprecated
warnings.warn("non-inplace resize is deprecated")
>>> t
tensor([[0., 1., 2.],
[3., 4., 5.]], dtype=torch.bfloat16)
>>> t.numpy()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Got unsupported ScalarType BFloat16
Seems par for the course that I'd be able to get the world's most advanced NLP model downloaded over gigabit to my home supercomputer only to be stymied by a data type.
- dekhn 4y agoComment three: after hacking my way around that, the script ran for another few hours doing disk IE (writing the offload) such that my directory is now 1.3T (330GB .git/lfs/objects, 330GB sharded weights, and 607G of "offload". Then it failed again: File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/big_modeling.py", line 188, in dispatch_model main_device = [d for d in device_map.values() if d not in ["cpu", "disk"] [0] IndexError: list index out of range Now I'm curious just how long it will take to repro this error (IE, running again, with the offload files aalready written). It's also puzzling since I set the device_map to 'auto'. Again, all par for the course and stuff I expected (having worked in HPC/ML/science for 3 decades, you get used to research codes).
- moyix 4y agoHuh. That one looks like it wasn't able to place any part of the model on the GPU. I know there is currently an issue where if the first layer of the model is too big to fit on the GPU it will put all layers on the CPU/disk (rather than trying to see if a later layer would fit). But the 3080Ti has 12GB so I'd be surprised if that's happening here?
- deleted 4y ago[deleted]
- zholer 4y agoI see the same error as well. Looking at the underlying accelerate code, it looks like if GPU is not an option then CPU and disc are completely rejected and the above error is thrown.
- moyix 4y agoJust to double-check, are you using the git versions of transformers and accelerate? I believe the BFloat16 support is pretty recent (22 days ago) and may not be in conda/pip: https://github.com/huggingface/accelerate/commit/f13c59f91e20beed3e8017de6e6818666e4e46b7 https://github.com/huggingface/accelerate/commit/f13c59f91e2...