tarruda

23 ms·

331.
by tarruda 3y ago
I'm curious as to what qualifies as "taking code". If you copy some code and change variable names, do some minor refactoring or reformatting, is that considered the same code?
332.
by tarruda 3y ago
If the model has big enough context window, yes. If the 100k word book is good enough for you is another story. GPT-4 turbo has a 128k token context, which might be good enough for your book. You might also use another strategy: write a sum
333.
by tarruda 3y ago
> How does LXD compare to Docker? They have different use cases. Docker runs single applications/daemons. LXD runs a whole Linux system by spawning an init process which takes care of spawning and managing other processes that compo
334.
by tarruda 3y ago
LXD is separate from libvirt, I don't think one affects the other. The only thing common between LXD and libvirt is that both use qemu to run virtual machines.
335.
by tarruda 3y ago
> Mistral-embed, our embedding endpoint, serves an embedding model with a 1024 embedding dimension. Our embedding model has been designed with retrieval capabilities in mind. It achieves a retrieval score of 55.26 on MTEB. Is there any i
336.
by tarruda 3y ago
Disclaimer: I'm a ML newbie, so this might be all incorrect. My intuition is that there are 8 7b models trained on knowledge domains. For example, one of those 7b models might be good at coding, while another one might be good at story
337.
by tarruda 3y ago
Correction, it should have the performance of a single 12B parameter model: "Concretely, Mixtral has 45B total parameters but only uses 12B parameters per token. It, therefore, processes input and generates output at the same speed and
338.
by tarruda 3y ago
> so that it will be possible to do the inference in low ram doing something similar to "paging". I was also hoping for that, but my initial research suggests that you need to fit everything into VRAM (or RAM when doing llama.c
339.
by tarruda 3y ago
I wonder if Linus will publicly scold the developer responsible for this, like the famous "Mauro, STFU!" e-mail
340.
by tarruda 3y ago
Tried Orca 2 first reasoning challenge and it responded correctly (though not in a "chat" way) > John and Mark are in a room with a ball, a basket and a box. John puts the ball in the box, then leaves for work. While John is aw
341.
by tarruda 3y ago
If this quantization method works with smaller models, it would enable running up to 33B models with only 12GB VRAM. Especially important for democratizing access to Mistral MoE new model.
342.
by tarruda 3y ago
> I can run it on my Macbook Air at 12tkps, can't wait to try this on my desktop. That seems kinda low, are you using Metal GPU acceleration with llama.cpp? I don't have a macbook, but saw some of the llama.cpp benchmarks that
343.
by tarruda 3y ago
No idea TBH, we'll have to wait and see. Some say it might be possible to efficiently swap the expert weights if you can fit everything in RAM: https://x.com/brandnarb/status/1733163321036075368?s=20
344.
by tarruda 3y ago
I only have a RTX 3070 with 8GB VRam. It can run quantized 7B models well, but this is 8 x 7B. Maybe an RTX 3090 with 24GB VRAM can do it.
345.
by tarruda 3y ago
I'm curious about their business model.
346.
by tarruda 3y ago
> Presumably, the same expert would frequently be selected for a number of tokens in a row In other words, assuming you ask a coding question and there's a coding expert in the mix, it would answer it completely.
347.
by tarruda 3y ago
I will be super happy if this is true. Even if you can't fit all of them in the VRAM, you could load everything in tmpfs, which at least removes disk I/O penalty.
348.
by tarruda 3y ago
Theoretically it could fit into a single 24GB GPU if 4-bit quantized. Exllama v2 has even more efficient quantization algorithm, and was able to fit 70B models in 24GB gpu, but only with 2048 tokens of context.
349.
by tarruda 3y ago
Damn, I was hoping it was still a single 7B model that I would be able to run on my GPU
350.
by tarruda 3y ago
Still 7B, but now with 32k context. Looking forward to see how it compares with the previous one, and what the community does with it.
351.
by tarruda 3y ago
Not familiar with `Clevis + Tang`, but the way I would solve is my implementing an IPC mechanism where an external process can provide the encryption passphrase. This would allow syncthing to start at boot, but untrusted devices would start
352.
by tarruda 3y ago
My biggest problem with how this feature is implemented is that the encryption passphrase is stored in plain text in the syncthing config file. I wish there was a way to lock the passphrase in the session keyring, with syncthing only sendin
353.
by tarruda 3y ago
I understood that point. The raspberry pi would still serve as an always on endpoint regardless of the untrusted device encryption feature. In my case I don't use this feature on the pi, but I do use it on my phone which has 256gb stor
354.
by tarruda 3y ago
A cheap and nice self hosted alternative way to have an "always on" sync endpoint is with a raspberry pi 4 with SSD (don't try this with an SD card). My current setup is running a wireguard server, syncthing and DuckDNS on th
355.
by tarruda 3y ago
An alternative (which I use) is GPU passthrough, which works on laptops that have a physical HDMI output to the Nvidia card. I'm only a causal gamer, but never had any compatibility by simple running all games in a windows VM. This set
356.
by tarruda 3y ago
> Replication is with rsync and borg, not snapshots. I backup my files with borg, but I still use snapshots during the Borg backup to ensure files are not modified during the process
357.
by tarruda 3y ago
I see. I can't give any anecdotal evidence on ChatGPT/Gemini/Bard, but I've been running small LLMs locally over the past few months and have amazing experience with these two models: - https://huggingface.co&
358.
by tarruda 3y ago
Good one. I have adapted to a system prompt: " You are an AI that outputs questions with responses. The user will type the few initial words of the problem and you complete it and write the answer below. " This allows to just type
359.
by tarruda 3y ago
I get what you mean, but what would such "qualitative evaluation" look like?
360.
by tarruda 3y ago
A couple of tests I've done in their live demo. - Implement the quicksort algorithm in python. I like this test because usually LLMs do the first time using list comprehensions, then I ask to avoid allocations. - Write the game "s

More ›