3 ms·
We use GPT-4 pretty heavily in a Typescript project, but have noticed lag from the TS versions of popular libraries (OpenAI’s npm lib, Langchain TS, etc.). Thi
by jumploops 3y ago
We use GPT-4 pretty heavily in a Typescript project, but have noticed lag from the TS versions of popular libraries (OpenAI’s npm lib, Langchain TS, etc.).
This framework is exciting to see. Even though Python is the “language of AI” most foundational models just sit behind an HTTP endpoint, making the web (and thus JS/TS) a perfect fit, as you’ve called out.
It’d be neat to see a caching layer (maybe similar API to evals?) that can be a drop-in for production workflows where the responses are somewhat deterministic.
- nichochar 3y agoGlad to hear this, indeed we think there's opportunity for some more cutting edge tooling in the TS ecosystem. We absolutely want to add a caching layer. Actually, we think middleware is where a lot of the value of the framework will come: it enables a whole bunch of features, e.g. sending errors to datasets for labeling, caching, user throttling, analytics, A/B tests, ... We're likely going to build the serving module next which will cover this.
- jumploops 3y agoVery cool. To be completely candid, we just hit OpenAI directly, no 3rd party libs involved at the moment (just fetch). We're open to trying more TS-focused libraries, but definitely more hesitant after our initial experiences with other libs. The less magic the better (no hidden prompts, etc.).