3 ms·
RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.
by akg_67 1mo ago
RAM 32GB, tokens are less than 20, may be about 12-15. Main issue is context window, I have set to 64k but practically beyond 32k, too much swap and slow down.