This is pretty decent, but a bit slow on my M2 Pro. Runs better on CPU, which is strange. Still, here's a quick guide to getting it to work on Metal: --re

This is pretty decent, but a bit slow on my M2 Pro. Runs better on CPU, which is strange. Still, here's a quick guide to getting it to work on Metal: --requirements.txt additions-- torchvision==0.18.0 accelerate==0.30.1 --gpu_utils.py patch-- def select_device(min_memory = 2048): logger = logging.getLogger(__name__) if torch.backends.mps.is_available(): device = torch.device('mps') return device could probably do with support for device_map for multiple backends... Edit: it also seems tho hallucinate/become increasingly unreliable with longer sentences.