3 ms·
Downloaded it's very responsive and the ability to edit contexts is really useful. I was curious as how you are able to calculate the number of prompt tokens in
by Boojum333 3y ago
Downloaded it's very responsive and the ability to edit contexts is really useful. I was curious as how you are able to calculate the number of prompt tokens in real time. Did you implement a method for Byte Pair Encoding yourself and how did you manage to match it with OpenAI's implementation for ChatGPT?