nirga

8 ms·

31.
by nirga 3y ago
I wouldn't call it misleading marketing - it is what it is, similar to what you can get today from tools like Langsmith, etc - Observability for the LLM part of your system, but using your existing tools. You can further extend that to
32.
by nirga 3y ago
You don't need to create an account in 2 services - you can just connect the SDK directly to Grafana - https://traceloop.com/docs/integration/grafana Happy to assist if needed over slack - https://
33.
by nirga 3y ago
Hey Marc :wave: Would love to see you integrate and adopt this as soon as it makes sense to you. OpenTelemetry is a great and mature piece of technology and we should all be aligning around it now, while it’s still easy to do so.
34.
by nirga 3y ago
The open source is free for all ofc. Our platform provides capabilities for monitoring and detecting hallucinations hence cost more.
35.
by nirga 3y ago
Same ticket that gets you to install something like Sentry - you wanna see what's happening in production and get alerted when things go wrong
36.
by nirga 3y ago
Thanks for the issues - I'll fix it! :sweat_smile: Reg. Grafana and others - it's simple, just set the env vars - https://www.traceloop.com/docs/openllmetry/integrations/intr...
37.
by nirga 3y ago
Super easy - you can just use the standalone instrumentations directly - https://www.traceloop.com/docs/openllmetry/tracing/without-s...
38.
by nirga 3y ago
Yes, Traceloop is kind of a Sentry for LLMs
39.
by nirga 3y ago
That's interesting! tbh you're not the first engineer I heard saying that!
40.
by nirga 3y ago
You mean that enterprises would prefer on-prem, right? But if I'm an engineer working on a large enterprise, what's the difference between a SaaS providing an on-prem option, and an OSS? It's not like I can just git clone a r
41.
Ask HN: How much does open source play a role when choosing a software?
3 points by nirga 3y ago|6 comments
42.
by nirga 3y ago
It’s hard to build a community around an open source if you’re not on GitHub imo. And if you’re not building an open source, than iiuc GitHub can’t use your code for copilot training so the main reason for not choosing GitHub according to t
43.
Mistral AI releases GPT-4 like MoE model as a torrent(twitter.com)
5 points by nirga 3y ago|0 comments
44.
Kubectl Powered by GPT(github.com)
1 points by nirga 3y ago|0 comments
45.
by nirga 3y ago
It amazes me that in 2023 this is not a solved problem by design of the language. Why go doesn’t adapt the “optional” notion of other languages so that if you have a variable you either know it is not null or know that you must check fo
46.
Evaluating LLM quality with the ROUGE metric(traceloop.com)
1 points by nirga 3y ago|0 comments
47.
by nirga 3y ago
It should work, but LangChain has many quirks so it can depend on which syntax you're using. Ping us on slack and we'll assist - https://join.slack.com/t/traceloopcommunity/shared_invite/zt...
48.
by nirga 3y ago
Yes, since we're using vanilla OpenTelemetry, you can set your exporter to whatever you want, including OpenTelemetry Protocol File Exporter. But I'd still use some sort of a dashboard, like Jaeger or one of the open source observ
49.
by nirga 3y ago
Yes, ofc. The LLM instrumentations are just like all other instrumentations.
50.
by nirga 3y ago
Sure, would love to! I'll ping you.
51.
by nirga 3y ago
Note that these products aren't the same, even though they all fall under the category of observability - similarly to how you'd use Datadog AND Sentry, although both can be called "observability platforms". I do think v
52.
by nirga 3y ago
Thanks! Related to what you're saying, I was actually expecting some reactions from devs who'd ask "why is it a separate repo and not part of opentelemetry from day 1?". And for that my answer would be that I think havin
53.
by nirga 3y ago
Wow, where do I start? The APIs are not that similar, but we're trying to use the same set of semantic conventions for everyone so for example you'll always get the model version, or the temperature in the same attribute. Which ki
54.
by nirga 3y ago
Yes, only traces for now. We do want to send out metrics for prompt length, token usage, etc. like you mentioned. Hopefully will be available soon (and we welcome contributions :) )
55.
by nirga 3y ago
2 differences: 1. You don't have instrumentations for libraries like OpenAI, LangChain, etc. so you need to manually open spans 2. As you said, there are no semantic conventions for logging things like prompts and chains. What we did i
56.
by nirga 3y ago
but it's open! :)
57.
Show HN: OpenLLMetry – OpenTelemetry-based observability for LLMs(github.com)
154 points by nirga 3y ago|31 comments
58.
Ask HN: TypeScript or Python for LLM Apps?
2 points by nirga 3y ago|0 comments
59.
by nirga 4y ago
Yes, it's exactly the same setup. I think OpenTelemetry gives a whole world of possibilities you can build on top which are much more than just observability.
60.
by nirga 4y ago
We're creating the backend equivalent of what cypress is doing. So in cypress you're generating FE tests where you click buttons and assert on FE components. Our framework can do that for checking BE flows. Let's say a user r

More ›