edunteman

8 ms·

61.
by edunteman 3y ago
here's an awesome post on the landscape https://hamel.dev/blog/posts/prompt/
62.
by edunteman 3y ago
answered in different thread. tldr: not that different for now. we're likely to do some serverside optimizations, esp. given our gpu inference history.
63.
by edunteman 3y ago
yeah I had a moment working with fructose where I realized "oh this is more like functional programming than I expected"
64.
by edunteman 3y ago
eventually, but priority goes toward finding an abstraction that feels right. We're very likely to break this package API, still v0. Sticking with openai till we have more confidence in the foundation being correct.
65.
by edunteman 3y ago
Yes. TGI is Huggingface's version of LLVM (some nuance, of course). LLVM also launched grammar support recently too, so we'll be looking into it.
66.
by edunteman 3y ago
Currently, quite comparable and obviously Instructor is more mature and feature rich. They're going the "patch the openAI client" approach which makes code written still use openAI SDK patterns which is pretty smart. Jason se
67.
by edunteman 3y ago
currently don't have pydantic support yet, but we're not too opinionated on that. I know it's seemed to emerge as a standard, and I imagine useful in the context of running fructose in a FastAPI handler, but we led with datac
68.
by edunteman 3y ago
Still learning about the landscape so can't give informed opinions. LMQL is a new one for me, will check it out. What we're mostly going for is composability vs abstraction. What's the smallest nugget of lift we can do for yo
69.
by edunteman 3y ago
Many of our early users have said this as well. I don't want this to turn into an abstraction monstrosity: the more unadulterated the prompt, the better. We're looking to outlines as inspiration for doing this logic as part of the
70.
by edunteman 3y ago
It's not 100% yet. Route to that: 1. Clientside, retry strategy on failed parse. Not yet implemented, we throw an exception on parse fail right now, but soon to be implemented. Not ideal because of token burn and latency, but the best
71.
by edunteman 3y ago
Seems like a great feature (and honestly allows us to do smarter things for strictly structured generation). I'm curious, what's your main motivation for local llms vs hosted APIs?
72.
by edunteman 3y ago
Will and Steve! Great work shipping this, very excited for what you build next
73.
Show HN: Fructose – LLM calls as strongly typed functions(github.com)
218 points by edunteman 3y ago|99 comments
74.
by edunteman 3y ago
Given this is a hosted API rather than arbitrary hosting, why choose the word "serverless"? Do you plan to offer arbitrary hosting in the future? (bias: am Banana CEO)
75.
by edunteman 3y ago
Hey! Banana founder here. Explicit webhook support coming out soon, though one could always add an http POST request to their webhook endpoint at the end of their handler to send the data that way rather than awaiting the results from the c
76.
by edunteman 3y ago
yeah, we're optimizing the infra for realtime inference, though people definitely still do run training on us, with a weights upload implemented at the end of your handler.
77.
by edunteman 3y ago
Thanks for the +1! Small note here: our billing is changing within the next month, to up-front payments that apply as a credit balance to your account. It still won't have minimums and you'll have the option to set up auto-refill
78.
by edunteman 3y ago
https://docs.banana.dev/banana-docs/core-concepts/billing You're only billed for active replica time. Call comes in, we start a replica, it handles the request, it waits around for a 10s (configurable) idle t
79.
by edunteman 3y ago
Hey! Would love to have you try https://banana.dev (bias: I'm one of the founders). We run A100s for you and scale 0->1->n->0 on demand, so you only pay for what you use. I'm at erik@banana.dev if you want any
80.
by edunteman 3y ago
Can only publicly answer one of these: - reusability of workloads: yes, introducing the community templates feature ( https://banana.dev/templates ) for common models has dramatically cut back on storage requirements and tran
81.
by edunteman 3y ago
Neat! Bit too busy now, but we'll hopefully put some technical blogs over time to explain these things. We don't do any predictive scaling yet; only when a call hits the queue do we scale replicas. Replicas cold boot (pod scheduli
82.
by edunteman 3y ago
Also hard disagree (as one of the Banana founders). Many users on our platform spend less than $10 a month on A100 GPUs, while building whole startups. Compared to the alternative of minimum $1k monthly for an always-on A100.
83.
by edunteman 3y ago
Hey! Banana cofounder here. Firstly, thank you so much for trying our service, we'll do our best to meet performance expectations and win you back! Re: #1 and #2, cold boots are the most vital thing for us to solve, because it fixes #1
84.
by edunteman 4y ago
which file formats does this work with?
85.
by edunteman 4y ago
Blackberry, the most successful fruit company of our time
86.
by edunteman 5y ago
this gives me energy
87.
by edunteman 5y ago
I've been using this product for a month now, in production for my startup. It's been helpful with my secrets but my favorite use case is using it instead for endpoint storage for service discovery across my dev, staging, and prod
88.
by edunteman 6y ago
Hey HN! My name's Erik, and I took OpenAI's CLIP zero-shot image classifier https://openai.com/blog/clip/ and API-ified it. It takes in a list of images (local .jpg or .png, or URL) and a list of natural
89.
by edunteman 6y ago
Hey HN! My name's Erik, and I took OpenAI's CLIP zero-shot image classifier https://openai.com/blog/clip/ and API-ified it. It takes in a list of images (local .jpg or .png, or URL) and a list of natural
90.
by edunteman 6y ago
http://gph.is/2cpbE6Y

More ›