3 ms·
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
by akshay_akula 1mo ago
Open source and no markup is the right default for a gateway. The caching question above is the one I would want answered before swapping models though.
- SilenN 1mo agoAns: we rarely switch, often times it's just a "switch to using this model for your agent"
- akshay_akula 1mo agoYou guys should look into ngrok ai gateway. We have some small models running on local hardware that we tried to use but it was too painful. Just wanted to not waste all the compute hit one endpoint and be done but as a team.