4 ms·
The pact drift problem is real and underrated. Most teams don't notice it until a customer complains about something that was working fine six months ago — by t
by matrixgard 7mo ago
The pact drift problem is real and underrated. Most teams don't notice it until a customer complains about something that was working fine six months ago — by then the audit trail is gone and you're debugging vibes, not code.
The decay-by-time approach you're describing is interesting but I'd push back slightly on fixed 7-day windows. Behavior drift is much more correlated with model updates and prompt changes than calendar time — an agent that hasn't changed in 90 days is probably more trustworthy than one that got a prompt tweak last week. Versioning pact scores to model and prompt hash might give you a sharper signal than wall-clock expiry.
What's your current plan for detecting when a model provider silently updates the underlying weights? That's the scenario that breaks pact scores without any change on your side.