philipkiely

7 ms·

31.
by philipkiely 2y ago
Yeah so MoE doesn't really come into play for production serving -- once you are batching your requests you hit every expert at a large enough batch size so you have to think about running the models as a whole. There are two ways we c
32.
by philipkiely 2y ago
They're using Llama.cpp which is an amazing tool for local inference but doesn't match fast inference frameworks like TensorRT-LLM/SGLang for production speeds and throughputs on Hopper GPUs. The Unsloth quantizations are rea
33.
by philipkiely 2y ago
Just spent a few minutes this morning spinning up a H100 model server and trying an FP8 quantized version (including kv cache quantization) to fit it on 2 H100s -- speed and quality looking promising. I'm excited to see if the better i
34.
by philipkiely 2y ago
Congrats Dex! Excited to see what people build with this + tools like Stripe's new agent payments SDK (issuing a payment seems like a great place to ask permission).
35.
by philipkiely 2y ago
Baseten | Tech Leads, Software Engineers, and more | SF, NY, Remote | Full-time 2 years and 8 months ago, I joined Baseten after seeing them in a "Who is Hiring" post on HN. Doing so was one of the three best decisions I've e
36.
How to build function calling and JSON mode for open-source and fine-tuned LLMs(baseten.co)
1 points by philipkiely 2y ago|0 comments
37.
by philipkiely 2y ago
This is a complete shameless plug but I just published some documentation on automatically building Whisper inference engines with TensorRT-LLM which has the batch inference that you're looking for: https://docs.baseten.co&#
38.
How to double tokens per second for Llama 3 with Medusa(baseten.co)
2 points by philipkiely 2y ago|0 comments
39.
by philipkiely 2y ago
Baseten | SF, NY, or REMOTE | Engineering | www.baseten.co Baseten provides fast, scalable inference for AI/ML models. We're at Series B and growing fast with great customers like Descript, Bland, and Patreon. Cool tech, unbeatabl
40.
by philipkiely 2y ago
I've been working with this for a couple of weeks ... there are real challenges in traditional setups going from prompt in -> prediction out to building the actual backends that blend business logic and inference for multiple models
41.
FP8: Efficient model inference with 8-bit floating point numbers(baseten.co)
2 points by philipkiely 3y ago|0 comments
42.
by philipkiely 3y ago
Does anyone know if this change is going to affect top-level domains operated by Google, for example .dev?
43.
Three techniques to adapt LLMs for any use case(baseten.co)
1 points by philipkiely 3y ago|0 comments
44.
by philipkiely 4y ago
Baseten | Software Engineers | Full-time | REMOTE (optional offices in SF, NYC) I joined Baseten over a year ago after seeing a comment in "Who is hiring." Best decision of my career. Baseten is a platform for deploying and servin
45.
Serving four million Riffusion requests in two days(baseten.co)
5 points by philipkiely 4y ago|0 comments
46.
by philipkiely 4y ago
Happy Thanksgiving to everyone on HN today! For young people like me who are new to the tech industry, this year has been a realization that tech is anything other than "a rollercoaster that only goes up." But I am so thankful for
47.
Show HN: Free Stable Diffusion 2.0 hosted interface(app.baseten.co)
25 points by philipkiely 4y ago|2 comments
48.
Try it yourself: Speech to text with Whisper(app.baseten.co)
5 points by philipkiely 4y ago|0 comments
49.
Deploying Stable Diffusion in Production Using Truss(baseten.co)
3 points by philipkiely 4y ago|0 comments
50.
Hosted Stable Diffusion Demo(app.baseten.co)
7 points by philipkiely 4y ago|0 comments
51.
by philipkiely 4y ago
Truss contributor here MLFlow is awesome; it's powerful and flexible and extends through multiple valuable portions of the ML cycle. Truss aims to be simpler to use in both straightforward and complex cases, requiring fewer lines of co
52.
Show HN: Truss – Serve any ML model without boilerplate code(github.com)
68 points by philipkiely 4y ago|9 comments
53.
Code generation interactive demo (Salesforce Codegen mono 2B)(app.baseten.co)
2 points by philipkiely 4y ago|0 comments
54.
by philipkiely 4y ago
There are no technical limitations in terms of scaling or permissions preventing you from building something customer-facing on Baseten! Personally, I'm very excited for indie hacker types to get their hands on Baseten and start shippi
55.
by philipkiely 4y ago
I always love transparency like this from authors, thanks for sharing! This line resonated way too much with me: "Initially, I thought five months might be enough for me to write it, but the scope quickly grew, and the progress was slo
56.
Show HN: Baseten – Build ML-powered applications(baseten.co)
112 points by philipkiely 4y ago|11 comments
57.
How BaseTen is using “docs as code”(blog.baseten.co)
5 points by philipkiely 5y ago|0 comments
58.
by philipkiely 5y ago
Baseten | REMOTE (US, Canada, Europe, and more), SF US | Full-time | https://baseten.co A personal note: I joined Baseten just over a month ago after seeing a post in January's "Who is Hiring" on HN, and I am very
59.
by philipkiely 5y ago
This is a great opportunity to support your favorite content creators in tech, if there is a blog or YouTube channel that you find yourself turning to regularly, see if they sell any information products, premium subscriptions, or training.
60.
by philipkiely 5y ago
In 2020, I launched a book that did really well for a few days, including on this platform. Despite extremely inconsistent marketing efforts, throughout 2021 residual sales netted me an average of over 500 dollars per month. It is wonderful

More ›