ianpurton

8 ms·

61.
by ianpurton 3y ago
So, what would be the hardware setup for this? Can it run on 1 GPU and swap between experts.
62.
by ianpurton 3y ago
https://bionic-gpt.com/blog/you-dont-need-a-vector-database/
63.
by ianpurton 3y ago
Then put that in a docker container for people like me who are too impatient to read 10 step documents.
64.
by ianpurton 3y ago
Sort of. I'm working on https://bionic-gpt.com/ There are a lot of moving parts. So for example we had someone trying to use the software and it didn't really work for their use case. https://github.com
65.
by ianpurton 3y ago
Rust on nails. https://rust-on-nails.com/
66.
by ianpurton 3y ago
Apparently there's a hole in the wall and the police are looking into it.
67.
by ianpurton 3y ago
The problem with nuclear is that their is so much FUD around it that it becomes not just expensive but very difficult to build. Mainly due to new regulations meaning projects have to re-engineer themselves before they are even complete. Lea
68.
by ianpurton 3y ago
The first thing they should work on is their landing page. Because for sure I'm their target market but I've got no idea what it does. I spent 10 seconds trying to figure it out then I was gone.
69.
by ianpurton 3y ago
The typography plugin which comes out of tailwind gives a clean readable experience in my opinion. https://tailwindcss.com/docs/typography-plugin
70.
by ianpurton 3y ago
Running LLama 2 70b a Mac Studio gets 13 tokens per second. 2 x 4090 is more like 38 tokens per second. But it's harder than that. As the 2 x 4090 is using https://llm.mlc.ai/ which is highly optimized so perhaps the M
71.
by ianpurton 3y ago
I've just finished writing an article on hardware required for LLM's in production https://bionic-gpt.com/blog/llm-hardware/ Would be interesting to see how this performed running Llama 2 7b.
72.
Hardware Requirements for LLM's in Production(bionic-gpt.com)
1 points by ianpurton 3y ago|0 comments
73.
by ianpurton 3y ago
We're working on an open source chat gpt that runs on kubernetes. https://github.com/bionic-gpt/bionic-gpt And you're right many companies are looking at running LLMs in house.
74.
by ianpurton 3y ago
SOC2 is a process, i.e. become compliant then get an audit. So become compliant and put the label on your website. Don't get an audit. With any customers when asked always mention that you have followed all the procedures and are waiti
75.
You don't need a Vector Database(bionic-gpt.com)
3 points by ianpurton 3y ago|0 comments
76.
by ianpurton 3y ago
40,000 a year for a support contract. https://bionic-gpt.com/services/
77.
by ianpurton 3y ago
Rust supports more software engineering best practices than Go. Some people say the learning curve for Rust is steep. But, in a way you're learning how to write safer code using the latest in language techniques.
78.
by ianpurton 3y ago
As an architect working on LLM applications I have these criteria for a database. - Full SQL support - Has good tooling around migrations (i.e. dbmate) - Good support for running in Kubernetes or in the cloud - Well understood by operations
79.
by ianpurton 3y ago
> my co-founder wants to pocket these speaking engagements and not run them through our business. I can't imagine this would generate a lot of income for the CEO. I've no idea but I guess speaking engagements are in the low tho
80.
by ianpurton 3y ago
I think the first Rust ML framework to gain massive traction will be the one that supports quantisation out of the box. At the moment there's no Rust ML library that does what llama.cpp or ggml can do.
81.
by ianpurton 3y ago
This will run on a PC with 16GB of ram (possible less) It's a 7b model quantized. docker run -it --rm ghcr.io/purton-tech/mpt-7b-chat
82.
by ianpurton 3y ago
I'm currently writing up a deployment architecture for LLM's and the API question is answered here https://fine-tuna.com/docs/choosing-a-model/model/ Basically you can get a Docker container that wi
83.
by ianpurton 3y ago
Really nice.
84.
by ianpurton 3y ago
I would take the d out of the -itd, otherwise I get no logs and can't see that it downloads pytorch_model before giving me access. Otherwise, great job.
85.
by ianpurton 3y ago
From you home page, I couldn't figure out exactly what you do. Can you parse PDF into embeddings for a vector database?
86.
by ianpurton 3y ago
Did you mean user interface or API? Anyway for UI you could look at chainlit, for API some of the models are already getting wrapped up in an open ai compatible rest interface. See https://github.com/go-skynet/LocalAI
87.
by ianpurton 3y ago
> Meanwhile, they'll happily pay Deloitte or Accenture 3x as much as what I charge, and then take the misguided ramblings of a junior-sold-as-senior at face value, not knowing the difference. This is so true. It's also very dif
88.
by ianpurton 3y ago
Web components are already built into the browser. Lit is like a wrapper around those.
89.
by ianpurton 3y ago
Salsa classes.
90.
by ianpurton 3y ago
Kubernetes means you don't have to learn each clouds way of doing a deployment. You just learn the k8s way then use that with Google, Azure or whatever. So your skillset is reusable.

More ›