3 ms·

I was curious to understand how Copilot implements its harness, and also how I was exhausting my quota so quickly. End up going down a rabbit hole of intercepti

I was curious to understand how Copilot implements its harness, and also how I was exhausting my quota so quickly. End up going down a rabbit hole of intercepting its network traffic with mitmproxy. A few interesting things I found along the way: - watched model/capability discovery and routing happen in real time - looked at what gets injected into context and sent with ghost completions - found that recent edits can pull in context from files other than the one you're currently editing (including infamous .env) - found the SQLite session store behind Chronicle, including previous prompts/responses - watched the model query that history through tool calls I then went through the VS Code source to reconcile some of what I was seeing on the wire with the actual implementation. Overall some interesting lessons around how their harness is implemented.

jiehong 2mo ago
How do you actually cleanly solve that .env issue? Anything cross platform and coding agent agnostic? I suppose that .env file should be removed, but then things aren’t easy: no native multiplatform secret manager, or the std lib of the language doesn’t offer an API over the native secret store, etc. Or a "secret injection proxy" for some cases could work I guess.
- sandos 2mo ago
  Data retention clauses? I dont see how you can ever really trust an LLM anyway to follow instructions.
- mehackernewsacc 2mo ago
  Do you feel that something like https://secretspec.dev/ https://secretspec.dev/ addresses these points?
- ElectricalUnion 2mo ago
  Infisical, or Bitwarden Secret Manager? Those two look like perfectly reasonable if the llm is just careless (but still, nothing prevents the LLM from intentionally cat'ing /proc/self/environ or from running /usr/bin/env or set or similar)
- adityazero 2mo ago
  [flagged]
- theozero 2mo ago
  https://varlock.dev https://varlock.dev (free, open source) can pull secrets from many places, and has a credential broker (proxy) to inject placeholders, then replace with real secrets at the network boundary. There are a few other tools like this, but ours seems to be the most flexible so far.

jiehong 2mo ago

How do you actually cleanly solve that .env issue? Anything cross platform and coding agent agnostic? I suppose that .env file should be removed, but then things aren’t easy: no native multiplatform secret manager, or the std lib of the language doesn’t offer an API over the native secret store, etc. Or a "secret injection proxy" for some cases could work I guess.

sandos 2mo ago
Data retention clauses? I dont see how you can ever really trust an LLM anyway to follow instructions.
mehackernewsacc 2mo ago
Do you feel that something like https://secretspec.dev/ https://secretspec.dev/ addresses these points?
ElectricalUnion 2mo ago
Infisical, or Bitwarden Secret Manager? Those two look like perfectly reasonable if the llm is just careless (but still, nothing prevents the LLM from intentionally cat'ing /proc/self/environ or from running /usr/bin/env or set or similar)
adityazero 2mo ago
[flagged]
theozero 2mo ago
https://varlock.dev https://varlock.dev (free, open source) can pull secrets from many places, and has a credential broker (proxy) to inject placeholders, then replace with real secrets at the network boundary. There are a few other tools like this, but ours seems to be the most flexible so far.