4 ms·
Show HN: Detect when an LLM silently changes behavior for the same prompt
- catarina_eng 8mo agoSame prompt. Same code. Different output weeks later. Was it the model or my code? Built a small tool to answer this: python examples/model_drift_detector.py If the provider changed behavior between runs: STATUS=CHANGED rc=2 INTERPRETATION=Same request produced a different response → The change came from the model, not your code. Works offline. No SaaS. Request + response hashes stored in a tamper-evident bundle. pip install aelitium
- chrisjj 8mo ago> This is AI provider accountability This is very far short of AI provider accountability. accountability noun an obligation or willingness to accept responsibility or to account for one's actions
- catarina_eng 8mo agoFair point — "accountability" implies enforcement mechanisms that aren't here. More accurate: tamper-evident logging. You capture a request/response pair with a hash. Later you can verify the pair hasn't changed. What anyone does with that evidence is a separate question. The README mentions this under the Trust Boundary section — I'll make the distinction clearer in the top section.
- chrisjj 8mo agoI agree. Good luck!