3 ms·

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequ

I've spent the past couple days packaging an LLM playground environment as a Nix expression. it's been pure hell. also nice to see you again, superkuh. I frequented your IRC channel about a decade ago.

siraben 4y ago
Have you been successful in getting the LLM playground up with Nix?
- sterlind 4y ago
  yes, almost! I used poetry2nix and grafted a bunch of overrides to fix the torch-2.0 build, and I just got cuda working with it. I'm testing triton now. I'll submit my PR to poetry2nix so watch that space if you want it.
superkuh 4y ago
Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen. Also, hi? I don't recall you by that nick but the internet is a small place sometimes.
- sterlind 4y ago
  oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.) specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the learning curve. did our channel survive the demise of freenode? I was andares, I think I used to be annoying but I've gotten better.
  - superkuh 4y ago
    Ah. Hi! Yes. We still exist in the same place but on libera now.
  - anotherhue 4y ago
    Care to share some of your progress? I have similar (stronger?) feelings regarding Dockerfile's big-ball-of-state nonsense. (The irony of holding this opinion while dealing with pre-trained AI models is not lost)
    - sterlind 4y ago
      I finished my work on poetry2nix and submitted a PR which works perfectly (at least with preferWheels=true.) now I have a wonderful live environment with torch, triton, transformers, etc. Docker builds are fast and lightweight since I use buildLayeredImage. it is, truly, the promised land my forefathers prophesized.

siraben 4y ago

Have you been successful in getting the LLM playground up with Nix?

sterlind 4y ago
yes, almost! I used poetry2nix and grafted a bunch of overrides to fix the torch-2.0 build, and I just got cuda working with it. I'm testing triton now. I'll submit my PR to poetry2nix so watch that space if you want it.

superkuh 4y ago

Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen. Also, hi? I don't recall you by that nick but the internet is a small place sometimes.

sterlind 4y ago
oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.) specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the learning curve. did our channel survive the demise of freenode? I was andares, I think I used to be annoying but I've gotten better.
- superkuh 4y ago
  Ah. Hi! Yes. We still exist in the same place but on libera now.
- anotherhue 4y ago
  Care to share some of your progress? I have similar (stronger?) feelings regarding Dockerfile's big-ball-of-state nonsense. (The irony of holding this opinion while dealing with pre-trained AI models is not lost)
  - sterlind 4y ago
    I finished my work on poetry2nix and submitted a PR which works perfectly (at least with preferWheels=true.) now I have a wonderful live environment with torch, triton, transformers, etc. Docker builds are fast and lightweight since I use buildLayeredImage. it is, truly, the promised land my forefathers prophesized.