eyegor

12 ms·

91.
by eyegor 3y ago
The lazy way is to use text-generation-webui, use an exllamav2 conversion of your model, and turn down context length until it fits (and tick the 8 bit cache option). If you go over your vram it will cut your speed substantially. Like 60&#x
92.
by eyegor 3y ago
Any 7b can run well (~50 tok/s) on an 8gb gpu if you tune the context size. 13b can sometimes run well but typically you'll end up with a tiny context window or slow inference. For cpu, I wouldn't recommend going above 1.3b u
93.
by eyegor 3y ago
Comcast may be complete garbage to work with, but you can always get a support contact. For any issue. Google is opposed to support.
94.
by eyegor 3y ago
For anyone looking to try this out on consumer grade hardware here's a q4 version [0]. From initial testing it's subjectively a bit behind deepseek-coder-instruct [1] at the same size and quantization. Deepseeks model is near magi
95.
by eyegor 3y ago
I'd defer to the llama.cpp or exllamav2 repos for quantizing help, I just use their guides myself. For practical use, I'd say searching for the quantized model on huggingface.co is the way to go. Chances are high that someone else
96.
by eyegor 3y ago
I use either llama.cpp or exllamav2 as the backend, but speed is mostly dependent on how many layers fit in your gpu. I'm not sure which gpu I was thinking of for that 50-100 tok/s, on my 2070 I can only get around 15 tok/s o
97.
by eyegor 3y ago
7b can run fine on consumer hardware if you quantize to 4bit. I get around 50-100 token/s on 7b q4 models on older 8gb gpus I have lying around, which is plenty for chat. I don't know of any good resource for "consumer hardwa
98.
by eyegor 3y ago
There are several web apis that Mozilla refuses to support, for example webhid or webusb, which all chromium browsers have had for years at this point. Same for little css features and some bits in the webaudio and webrtc implementations. M
99.
by eyegor 3y ago
This happens periodically, where google services act funny with a Firefox user agent. Just use edge or chromes user agent string, and Google sites will work again.
100.
by eyegor 3y ago
Hmm, I just assumed people knew about it. Submitted https://news.ycombinator.com/item?id=38495176 Self hosted 6.7b model is phenomenal in my experience, I actually prefer it to chatgpt a lot of the time. Similar quality cod
101.
Deepseek Coder instruct – 6.7B model beats gpt3.5-turbo in coding(github.com)
6 points by eyegor 3y ago|1 comments
102.
by eyegor 3y ago
Quantized 6.7b - https://huggingface.co/TheBloke/deepseek-coder-6.7B-instruct... 1.3b - https://huggingface.co/TheBloke/deepseek-coder-1.3b-instruct... In my experience, these models are at least
103.
by eyegor 3y ago
In my experience, the deepseek-coder-instruct family is at least as good as gpt-3.5-turbo on python. Even the 1.3b models are very good (and run okay on cpu), although you should use larger if you have the vram. There are even larger models
104.
by eyegor 3y ago
It was fixed in 2019 though? I don't understand your point. https://github.com/jqlang/jq/issues/1380
105.
by eyegor 3y ago
Dropbox seems to still work fine in my experience.
106.
by eyegor 3y ago
Considering how poorly it seems to support cuda as a backend [0], I wouldn't hold my breath about non intel vendor support (amd cpu or gpu). If you want to try anyway, here's the benchmark suite [1] and I couldn't find sample
107.
by eyegor 3y ago
My understanding is they reduced the number of ensembles feeding gpt4 so they could support more customers. I want to say they cut it from 16 to 8. Take that with a grain of salt, that comes through the rumor telephone. Are you prompting it
108.
by eyegor 3y ago
https://katex.org/ may work?
109.
by eyegor 3y ago
Yeah all the pieces are already there, but I was trying to make something like a player replacer that would let the user load arbitrary glsl shaders to use. The idea being to provide usable upscaling or filters for weak connections or old v
110.
by eyegor 3y ago
Kind of unrelated, but how difficult do you think it would be to hack support for glsl shaders in a browser? I tried to look into it once, but got a bit lost in the media source side of things. My idea was to try to add glsl shaders as post
111.
by eyegor 3y ago
Wow, faststream works great for normal web players. Doesn't seem to work on any youtube videos when using the ff extension in the store though. Gets stuck loading forever. Edit: fixed, works well in every test case
112.
by eyegor 3y ago
How do you work with so much python code with no type annotations? I get it for smaller projects but isn't half this stuff meant to be library code?
113.
by eyegor 3y ago
See also https://github.com/cupy/cupy https://github.com/inducer/pyopencl
114.
by eyegor 3y ago
This is true, but no one in the industrial world is using macs. You even address this in your first paragraph. No IT group in their right mind is going to add a 3rd environment to manage (apple). Beyond being a headache to have multiple ope
115.
by eyegor 3y ago
Most of the time an easy optimization is to pad fields of your struct to a 32 bit boundary. Almost any compiler will do this for you (look up "struct alignment / padding"). If the compiler is going to do this anyway, might as
116.
by eyegor 3y ago
You should watch this: https://youtu.be/cD7YFUYLpDc?si=rUm6IR3IKXyzcaDB to better understand why high sample rates are a waste of time, instead of just reading about nyquist. "accurately preserving a 6kHz sine wave&quo
117.
by eyegor 3y ago
Intel has been running oneapi for several years, and as a long time user I assure you it's horrible to deal with. It's the only software I've dealt with that breaks other software. Every year or two I'd have to fully rei
118.
by eyegor 3y ago
If that's true, you should probably update the docs. Everything I could find implied dotnet, jvm, python were still unsupported. For example, the roadmap section of the readme mentions most of these as future targets but nothing even m
119.
by eyegor 3y ago
Haproxy is great in the end but it is pretty awful to work with. Once you've toiled in the mines of acl commands, going back to nginx/caddy is a breath of fresh air. Unless recent versions have completely changed the game, using h
120.
by eyegor 3y ago
The m1/m2 chips are generally stupid effecient compared to Intel chips (or even amd/arm/etc)... Are you sure the power draw is comparable? Apple is quite well known for kneecapping hardware with terrible thermal solutions and

More ›