4 ms·
Open-source tools for LLM monitoring and observability
- AllenNLP Interpret: A library for interpreting and visualizing LLM predictions, assisting in model explanation and debugging. It works for any model of your choice.
- LangKit: An open-source toolkit for monitoring Large Language Models (LLMs). Features include assessing text quality and relevance, hallucinations check, sentiment and toxicity analysis.
- BERTViz: Specifically designed for visualizing and interpreting BERT-based LLMs. Helps visualize attention in NLP Models (BERT, GPT2, BART, etc.).
- SHAP (SHapley Additive exPlanations): A game theoretic approach to explain the output of any machine learning model. Allows users to use models from the transformers library by HuggingFace.
- AI Fairness 360: An extensible open-source toolkit can help you examine, report, and mitigate discrimination and bias in machine learning models throughout the AI application lifecycle
- Prometheus: An open-source monitoring toolkit for collecting and querying metrics from LLMs in real time.
- Grafana: Integrates with tools like Prometheus and Elasticsearch to provide visualization and analysis of LLM metrics and logs.
Learn more about LLM monitoring and observability at https://www.turingpost.com/p/monitoring
- retrovrv 3y agohttps://portkey.ai/ https://portkey.ai/ - is also a nice addition to the list
- Joschkabraun 3y agohttps://docs.parea.ai/observability/logging_and_tracing https://docs.parea.ai/observability/logging_and_tracing isn't open-source for LLM monitoring but the evaluation metrics to assess LLM app quality are: https://github.com/parea-ai/parea-sdk-py https://github.com/parea-ai/parea-sdk-py
- pranabgohain 3y agoI would add https://www.kloudmate.com https://www.kloudmate.com to the list, for collecting metrics, logs and traces via OpenTelemetry.
- mso96 3y agoalso we are working on the Next-Generation Observability Co-pilot for Developers https://www.infrastack.ai/ https://www.infrastack.ai/
- samstave 3y agoI dont need early access, but I am super curious, can you tell me more about what youre building, I can infer and I like the images that pop into my head, but I'd like to hear more. (infers-structure as a service)
- yagizdegirmenci 3y agoHelicone is also pretty decent: https://helicone.ai https://helicone.ai