3 ms·

I couldn't get the downloadable binary to work, or the binary I compiled myself: ./SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-

• deleted 7mo ago
  [deleted]
• aegis_camera 7mo ago
  git clone https://github.com/SharpAI/SwiftLM https://github.com/SharpAI/SwiftLM # no --recursive needed cd SwiftLM swift build -c release ### Please let me know if this fix the issue: # Copy metallib next to the binary (one-time step) cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/
• deleted 7mo ago
  [deleted]
• simonw 7mo ago
  Claude Code helped me figure out this recipe (inspired by a similar workaround in the CI scripts): git clone --recursive https://github.com/SharpAI/SwiftLM.git cd SwiftLM swift build -c release # Trick to copy in that missing mlx.metallib file uv run --with mlx-metal python -c " import importlib.metadata, pathlib, shutil d = importlib.metadata.distribution('mlx-metal') metallib = pathlib.Path(d._path).parent / 'mlx/lib/mlx.metallib' shutil.copy(metallib, '.build/release/') print(f'Copied {metallib} -> .build/release/mlx.metallib') # Now start the server (downloads 69GB Qwen model) .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 But the server crashed when I tried to run a prompt through it: freed pointer was not the last allocation
  • aegis_camera 7mo ago
    the Python mlx-metal trick is actually what's crashing it. The mlx.metallib from pip is a different version of MLX than what your Swift binary was built against. It gets past the startup error but then corrupts the GPU memory allocator at inference time → freed pointer was not the last allocation. Use the version-matched metallib that's already in the repo: cp LocalPackages/mlx-swift/Source/Cmlx/mlx/mlx/backend/metal/kernels/default.metallib \ .build/release/ .build/release/SwiftLM \ --model mlx-community/Qwen3.5-122B-A10B-4bit \ --stream-experts \ --port 5413 This is the exact metallib that was compiled alongside the Swift code — no version mismatch. Future pre-built releases will bundle it automatically.