2 ms·
The project doesn't just use 2-bit - that was one of the formats they tried, but when that didn't give good tool calls they switched to 4-bit.
by simonw 8mo ago
The project doesn't just use 2-bit - that was one of the formats they tried, but when that didn't give good tool calls they switched to 4-bit.
- tarruda 8mo agoIn my case it the 2.46BPW has been working flawless for tool calling, so I don't think 2-bit was the culprit for JSON failing. They did reduce the number of experts, so maybe that was it?