lhl

31 ms·

1. 301.▲
   by lhl 3y ago

   This is the most well-maintained list of commercially usable open LLMs: https://github.com/eugeneyan/open-llms MPT, OpenLLaMA, and Falcon are probably the most generally useful. For code, Replit Code (specifically repl
2. 302.▲
   by lhl 3y ago

   I've found https://gpt4all.io/ to be the fastest way to get started. I've also started moving my notes to https://llm-tracker.info/ which should help make it easier for people getting started: ht
3. 303.▲
   by lhl 3y ago

   In terms of speed, we're talking about 140t/s for 7B models, and 40t/s for 33B models on a 3090/4090 now.[1] (1 token ~= 0.75 word) It's quite zippy. llama.cpp performs close on Nvidia GPUs now (but they don't
4. 304.▲
   by lhl 3y ago

   Related: * Fruit Fly Brain Observatory https://www.fruitflybrain.org/#/posts/explore_ffbo which lets you visualize and explore datasets * Neurokernel http://neurokernel.github.io/ - open source so
5. 305.▲
   by lhl 3y ago

   While this might be somewhat applicable for Hofstadter or Tegmark, I think it doesn't applies quite the same way to Hinton. Remember, until this Spring, he's been at Google Brain and has been supervising grad students until recent
6. 306.▲
   by lhl 3y ago

   No, the Steam Deck would have never upgraded to Dragon Crest (2022 on those leaked roadmaps). In case you didn't realize, the Steam Deck wasn't released until Feb 25 2022, and only sold 1M units in 2022 (vs 3M projected in 2023).
7. 307.▲
   by lhl 3y ago

   I think it's too easy to fall into the anecdote trap, and I like the comments with study data, like the Pew Research and Line Research (sequential surveys, even better)... I also think adoption will depend on "industry." Chat
8. 308.▲
   by lhl 3y ago

   I don't think a next-gen APU for a Steam Deck 2 is going to be that big a problem. The Steam Deck's APU is officially designated: "AMD Custom APU 0405"[1], basically a mix and match of AMD's same-generation parts. W
9. 309.▲
   by lhl 3y ago

   For YouTube, I'd recommend setting up HW decoding. Depending on your distro and GPU, it can be a pain to set up, but it's much more efficient: https://wiki.archlinux.org/title/Hardware_video_acceleration Pers
10. 310.▲
    by lhl 3y ago

    Here's a good writeup that goes into more depth than most of the READMEs on fine-tuning: https://erichartford.com/uncensored-models
11. 311.▲
    by lhl 3y ago

    Here are some high level answers: "7B" refers to the number of parameters or weights for a model. For a specific model, the versions with more parameters take more compute power to train and perform better. A foundational model is
12. 312.▲
    by lhl 3y ago

    There's also OpenLLaMA, which has a 13B version as well and is a straight drop-in (except for code generation due to multiple space tokenization: https://github.com/openlm-research/open_llama#update-0615202... ).
13. 313.▲
    by lhl 3y ago

    A team at Microsoft Research asked the same question and just published a paper about part of that at least: TinyStories: How Small Can Language Models Be and Still Speak Coherent English? https://arxiv.org/abs/2305.077
14. 314.▲
    by lhl 3y ago

    The 13th-gen Framework gets just under 12h on Windows 11 https://www.youtube.com/watch?v=tuw-YpbFkkM - for Linux, battery life will be up to how your distro does power management. The Ubuntu guide I linked has a read time o
15. 315.▲
    by lhl 3y ago

    I think you're way overestimating the power usage of modern displays. The Framework display (BOE095F) backlight based on my personal testing uses ~2.1W at 100% (440 nits) (this is as expected, you can look at datasheets on similar disp
16. 316.▲
    by lhl 3y ago

    One additional note is that it's possible to make Magnesium alloys much more resistant to burning (adding Calcium and Yttrium for example): https://www.ait.ac.at/en/research-topics/casting-technologie... Rela
17. 317.▲
    by lhl 3y ago

    In case you haven't seen, in April, Framework published an official Optimizing Ubuntu Battery Life guide that is quite useful: https://knowledgebase.frame.work/en_us/optimizing-ubuntu-bat... If that doesn't h
18. 318.▲
    by lhl 3y ago

    Forgot to mention that Fabrice Bellard has run lm-eval on a range of functional tests (lambada, hellaswag, winogrande, piqa, coqa) for q4, q8 (and unquantized on smaller) models: https://bellard.org/ts_server/ Losses a
19. 319.▲
    by lhl 3y ago

    > how everyone is in this mad quantization rush but nobody's putting up benchmarks to show that it works (tinybox is resolutely supporting non quantized LLaMA) I don't think this is true. llama.cpp has historically been very co
20. 320.▲
    by lhl 3y ago

    That's not true/a hallucination, you can check the /moderations endpoint in your browser's console. I am running into the exact same truncation issue, but the moderation return is: """ { "flagged&
21. 321.▲
    by lhl 3y ago

    Actually it's HF's leaderboard that's bugged. Falcon is only on top since their MMLU scores are bugged across all LLaMA models: https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
22. 322.▲
    by lhl 3y ago

    For Falcon specifically, this is easy, it's embedded here: https://huggingface.co/blog/falcon#demo or you can access the demo here: https://huggingface.co/spaces/HuggingFaceH4/falcon-chat
23. 323.▲
    by lhl 3y ago

    I agree that AMD should be dedicating an engineer to making sure all the top popular ML/AI projects can run on their hardware, but maybe they also need to spin up a wiki listing compatibility... AMD GPU acceleration via CLBlast was mer
24. 324.▲
    by lhl 3y ago

    GPTQ for LLaMAs w ROCm works with https://github.com/turboderp/exllama/ but Falcon inferencing is a different beast.
25. 325.▲
    by lhl 3y ago

    I saw there was an answer already in your issue, although you plan on doing a lot of inferencing on your GPU, I'd highly recommend you consider dual-booting into Linux. It turns out exllama merged ROCm support last week and more than 2
26. 326.▲
    by lhl 3y ago

    Not a 30 series, but on my 4090 I'm getting 32.0 tokens/s on a 13b q4_0 model (uses about 10GiB of VRAM) w/ full context (`-ngl 99 -n 2048 --ignore-eos` to force all layers on GPU memory and to do a full 2048 context). As a p
27. 327.▲
    by lhl 3y ago

    llama.cpp can be run with a speedup for AMD GPUs when compiled with `LLAMA_CLBLAST=1` and there is also a HIPified fork [1] being worked on by a community contributor. The other week I was poking on how hard it would be to get an AMD card r
28. 328.▲
    by lhl 3y ago

    While I think Apple throwing their hat in the ring with a super-high end device is great for the market and should light a fire under Meta (and everyone else's butt), I think that the idea that Apple has a super-advanced technology lea
29. 329.▲
    by lhl 3y ago

    Based on your followup post, I think you're confused. Binocular overlap doesn't magically give you double the perceived resolution. If it worked that way, everyone would have a way higher PPD, but it doesn't and they don&#x27
30. 330.▲
    by lhl 3y ago

    At 90-100 degrees, a 4K/eye display will likely end up being around 35-40 PPD (pixels per degree). This is about equivalent to what a FHD display would like like (27" monitor at 60cm: https://qasimk.io/screen-ppd&#