11 ms·
You can run on cpu and regular ram, but gpu is quite a bit faster. You need about a gig of RAM/nvram per billion parameters (plus some headroom for a context w
by more_corn 3y ago
You can run on cpu and regular ram, but gpu is quite a bit faster.
You need about a gig of RAM/nvram per billion parameters (plus some headroom for a context window). Lower precision doesn’t really affect quality.
When Ethereum flipped from proof of work to proof of stake, a lot of used high-end cards hit the market.
4 of them in a cheap server would do the trick. Would be a great business model for some cheap colo to stand up a crap-ton of those and rent while servers to everyone here.
In the meantime if you’re interested in a cheap server as described above, post in this thread.