lbeurerkellner

7 ms·

61.
by lbeurerkellner 2y ago
Typically these logs are available, but hard to read or just dumped in some JSON file. However, they have been efforts like https://explorer.invariantlabs.ai/benchmarks/ that try to make agents more transparent in that
62.
Understanding Agentic Intelligence: Inspecting runtime traces of actual agents(explorer.invariantlabs.ai)
1 points by lbeurerkellner 2y ago|0 comments
63.
by lbeurerkellner 2y ago
This is so awesome.
64.
Show HN: A CTF challenge that let's you attack an LLM-based agent
5 points by lbeurerkellner 2y ago|0 comments
65.
by lbeurerkellner 2y ago
Automatically rendered link previews also play nicely into this.
66.
Invariant: A security scanner for AI agents(github.com)
1 points by lbeurerkellner 2y ago|0 comments
67.
by lbeurerkellner 2y ago
A similar setting is explored in this running CTF challenge: https://invariantlabs.ai/ctf-challenge-24 Basically, LLM apps that post to link-enabled chat feeds are all vulnerable. What is even worse, if you consider link pr
68.
Invariant CTF 24: Fool an AI Agent to Extract the Secret Password(invariantlabs.ai)
1 points by lbeurerkellner 2y ago|0 comments
69.
by lbeurerkellner 2y ago
Avoiding these kind of leaks is one of the core motivations behind the Invariant analyzer for LLM applications: https://github.com/invariantlabs-ai/invariant Essentially a context-aware security monitor for LLMs.
70.
by lbeurerkellner 2y ago
There is a pull request to add a security monitor that makes sure it does not do anything unreasonable: https://github.com/OpenDevin/OpenDevin/pull/3058
71.
Invariant CTF: Fool an AI Agent to Extract the Secret Password(invariantlabs.ai)
4 points by lbeurerkellner 2y ago|0 comments
72.
LVE Project: A Repository of Language Model Vulnerabilities and Exposures(github.com)
2 points by lbeurerkellner 2y ago|0 comments
73.
AI Software Should Be More Like Plain Old Software(sigarch.org)
1 points by lbeurerkellner 2y ago|0 comments
74.
by lbeurerkellner 3y ago
there is an LVE for this: https://lve-project.org/reliability/repetition/openai--gpt-3...
75.
by lbeurerkellner 3y ago
This works until it doesn’t: https://lve-project.org/blog/how-effective-are-llm-safety-fi...
76.
LMQL Developer Survey(lmql.ai)
2 points by lbeurerkellner 3y ago|0 comments
77.
Show HN: LVE – The First Open Repository of LLM Vulnerabilities and Exposures(lve-project.org)
4 points by lbeurerkellner 3y ago|0 comments
78.
Fuel To The Fire: How effective are LLM-based safety filters for AI systems?(lve-project.org)
3 points by lbeurerkellner 3y ago|1 comments
79.
LMQL 0.7 brings Procedural Prompt Programming(lmql.ai)
1 points by lbeurerkellner 3y ago|0 comments
80.
by lbeurerkellner 3y ago
LMQL author here. You may want to try the preview 0.7b release of LMQL, chat is not in the stable 0.0.6.* series. Lots of things are coming in LMQL also, so don't worry :)
81.
LMQL – A language for typed, constraint-guided and efficient LLM programming(docs.lmql.ai)
3 points by lbeurerkellner 3y ago|0 comments
82.
by lbeurerkellner 3y ago
Thank you for bringing up LMQL. We have active branches with regex, parsers and types (structured and other types) which will also soon be upstreamed, improving typed LLM use beyond the current template-based approach we support.
83.
by lbeurerkellner 3y ago
LLM calls are just function calls, so most functional composition is already afforded by any general-purpose language out there. If you need fancy stuff, use something like Python‘s functools. Working on https://github.com/e
84.
Show HN: LMQL becomes simpler and adds llama.cpp(lmql.ai)
6 points by lbeurerkellner 3y ago|0 comments
85.
by lbeurerkellner 3y ago
I think one should not underestimate the impact on downstream performance the output format can have. From a modelling perspective it is unclear whether asking/fine-tuning the model to generate JSON (or YAML) output is really lossless
86.
by lbeurerkellner 3y ago
It's interesting to think about this form of computation (LLM + function call) in terms of circuitry. It is still unclear to me however, if the sequential form of reasoning imposed by a sequence of chat messages is the right model here
87.
by lbeurerkellner 3y ago
If anything this removes a major roadblock for libraries/languages that want to employ LLM calls as a primitive, no? Although, I fear the vendor lock-in intensifies here, also given how restrictive and specific the Chat API. Either way
88.
by lbeurerkellner 3y ago
LMQL dev here. A bit of an unusual way of getting highlighted on here, I must say. I am sorry to hear that you have issues installing LMQL. We are working very hard on LMQL, to make it as seamless and accessible as possible (as you have see
89.
by lbeurerkellner 3y ago
It's still a bit hacky in the current PyPi version of LMQL, but you can also use it from the command line, just like `python -c`: echo "Who are you?" | lmql run "argmax '\"Q:{await input()} A:[RESULT]'
90.
by lbeurerkellner 3y ago
Yes, you are correct. LMQL has its own interpreter and runtime, that executes and translates your query programs into multiple LLM calls, required to satisfy your overall workload. It is inherently multi-part and automates several consecuti

More ›