3 ms·

If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontri

If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontrivial number of errors/codebases.

LunaSea 3y ago
This would definitely not work in most bug cases just based on the fact that context size is too short and models too dumb. If you have a "can't access key 'foo' of undefined" error, adding a check if the object is undefined or not is not a fix. It's simply silencing the error. The bug might be way, way down in the stack, maybe even in the backend or somewhere else in the client that made the backend store invalid data my mistake. I highly doubt that the current breed of models are capable of this.
- zachthewf 3y ago
  Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It is certainly not enough for every case, but it is enough (and the models are smart enough) to be useful.
  - LunaSea 3y ago
    > Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It really isn't. I just did a test with a random TypeScript file with 400 lines of code which translates to almost exactly 3000 tokens in the GPT-4 token counter app. This means that we are talking about being capable to analyse a maximum of 43 small / medium sized files. Each analysis will also cost $1.28 without any additional intermediary costs.
ciex 3y ago
They have this as an experimental feature (which is opt-in and I won't try it)
- zachthewf 3y ago
  I didn't know that. Maybe it's not good enough so they figured they had to train on customer data to ship something high quality, hence this change.
Lariscus 3y ago
Just what I need, pull requests with a bunch of AI generated drivel. I really don't want to spend my days figuring out if the PR that just got assigned to me contains an actual issue or some AI hallucination.
- zachthewf 3y ago
  If it's attached to a Sentry error, there's clearly some issue which produces the exception.

If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontrivial number of errors/codebases.

LunaSea 3y ago

This would definitely not work in most bug cases just based on the fact that context size is too short and models too dumb. If you have a "can't access key 'foo' of undefined" error, adding a check if the object is undefined or not is not a fix. It's simply silencing the error. The bug might be way, way down in the stack, maybe even in the backend or somewhere else in the client that made the backend store invalid data my mistake. I highly doubt that the current breed of models are capable of this.

zachthewf 3y ago
Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It is certainly not enough for every case, but it is enough (and the models are smart enough) to be useful.
- LunaSea 3y ago
  > Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It really isn't. I just did a test with a random TypeScript file with 400 lines of code which translates to almost exactly 3000 tokens in the GPT-4 token counter app. This means that we are talking about being capable to analyse a maximum of 43 small / medium sized files. Each analysis will also cost $1.28 without any additional intermediary costs.

ciex 3y ago

They have this as an experimental feature (which is opt-in and I won't try it)

zachthewf 3y ago
I didn't know that. Maybe it's not good enough so they figured they had to train on customer data to ship something high quality, hence this change.

Lariscus 3y ago

Just what I need, pull requests with a bunch of AI generated drivel. I really don't want to spend my days figuring out if the PR that just got assigned to me contains an actual issue or some AI hallucination.

zachthewf 3y ago
If it's attached to a Sentry error, there's clearly some issue which produces the exception.