3 ms·

What would they benefit from this? Why do they have anything to do with AI at all?

eigenket 3y ago
Maybe they want to sell their user's data to people training AIs rather than training them themselves?
buildfocus 3y ago
I can imagine there lots of practical ways they can use AI - guessing the cause & solution of issues (this seems to be an off by one error, or you might want to check for null values on this line), automatically finding & linking you to related code for new errors (this failing connection was probably created over here), and detecting which errors are important and which are noise that you can ignore (the biggest problem with Sentry in my experience). In each case, training on their existing data will help a lot with this. All of that would be future product development, but I suspect this addition is for exactly that reason: so they can begin to prepare the ground for later development, not because they're going to immediately roll this out tomorrow.
zachthewf 3y ago
If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontrivial number of errors/codebases.
- LunaSea 3y ago
  This would definitely not work in most bug cases just based on the fact that context size is too short and models too dumb. If you have a "can't access key 'foo' of undefined" error, adding a check if the object is undefined or not is not a fix. It's simply silencing the error. The bug might be way, way down in the stack, maybe even in the backend or somewhere else in the client that made the backend store invalid data my mistake. I highly doubt that the current breed of models are capable of this.
  - zachthewf 3y ago
    Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It is certainly not enough for every case, but it is enough (and the models are smart enough) to be useful.
    - LunaSea 3y ago
      > Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It really isn't. I just did a test with a random TypeScript file with 400 lines of code which translates to almost exactly 3000 tokens in the GPT-4 token counter app. This means that we are talking about being capable to analyse a maximum of 43 small / medium sized files. Each analysis will also cost $1.28 without any additional intermediary costs.
- ciex 3y ago
  They have this as an experimental feature (which is opt-in and I won't try it)
  - zachthewf 3y ago
    I didn't know that. Maybe it's not good enough so they figured they had to train on customer data to ship something high quality, hence this change.
- Lariscus 3y ago
  Just what I need, pull requests with a bunch of AI generated drivel. I really don't want to spend my days figuring out if the PR that just got assigned to me contains an actual issue or some AI hallucination.
  - zachthewf 3y ago
    If it's attached to a Sentry error, there's clearly some issue which produces the exception.
happytiger 3y ago
They want the data for AI but not just for their products. They want it as a business asset. It helps boost their valuation and makes them attractive to potential acquirers, but they also can sell de-identified data outside the company per their own terms. That just means they don’t sell PII for less than two customers in a data identifiable group. See the terms: > Sentry may use Usage Data and Service Data for analytics and product development (including to train or improve AI Features and generate Outputs). Sentry will not disclose Usage Data or Outputs externally, unless they have been de-identified so that they do not individually identify Customer, its Users or any other person and are aggregated with data across Sentry’s other customers.
- the_mitsuhiko 3y ago
  > They want the data for AI but not just for their products. They want it as a business asset. That is incorrect, the data is exclusively used to provide services to customers. The blog post is quite specific about the intended use.
  - happytiger 3y ago
    That is NOT what their own terms and conditions, which I’m quoting in the comment you are replying to actually says: > Sentry will not disclose Usage Data or Outputs externally, unless they have been de-identified so that they do not individually identify Customer, its Users or any other person and are aggregated with data across Sentry’s other customers. You can write whatever you want in a blog post. It’s the legal terms and what’s in the contract that matters and presumably shows intent. In this case they are clearly planning to show or sell de-identified data as it’s in their terms and conditions explicitly. What is being said in the blog and what the company is giving themselves a license to do are not the same.
    - the_mitsuhiko 3y ago
      We put an update into the blog post that addresses this: https://blog.sentry.io/ai-privacy-and-terms-of-service-updates/ https://blog.sentry.io/ai-privacy-and-terms-of-service-updat...
JimDabell 3y ago
At the moment they have a feature where you can click on a button on a crash report and it will use generative AI to explain the problem / suggest a fix. Sometimes it works, sometimes it doesn’t. They also have features tied into version control where you can tag a commit to close a crash report etc. So I would expect them to be able to train on the data this gives them to suggest more accurate fixes down to the source code level.
the_mitsuhiko 3y ago
> What would they benefit from this? Today one of the largest challenge with Sentry at scale is that every error looks the same. If statements only get you that far. One of the ways in which we want to improve this is to use issue interaction and the event's content to predict the severity of an issue. The second case is a version of suggested fix that uses your code to suggest actual fixes to your code. In this case the data never crosses your organization's boundary and is private to yourself.

Maybe they want to sell their user's data to people training AIs rather than training them themselves?

I can imagine there lots of practical ways they can use AI - guessing the cause & solution of issues (this seems to be an off by one error, or you might want to check for null values on this line), automatically finding & linking you to related code for new errors (this failing connection was probably created over here), and detecting which errors are important and which are noise that you can ignore (the biggest problem with Sentry in my experience). In each case, training on their existing data will help a lot with this. All of that would be future product development, but I suspect this addition is for exactly that reason: so they can begin to prepare the ground for later development, not because they're going to immediately roll this out tomorrow.

zachthewf 3y ago

If I were running product at Sentry I'd be working on an agent that opened pull requests to fix your Sentry errors. This would work today with GPT-4 on a nontrivial number of errors/codebases.

LunaSea 3y ago
This would definitely not work in most bug cases just based on the fact that context size is too short and models too dumb. If you have a "can't access key 'foo' of undefined" error, adding a check if the object is undefined or not is not a fix. It's simply silencing the error. The bug might be way, way down in the stack, maybe even in the backend or somewhere else in the client that made the backend store invalid data my mistake. I highly doubt that the current breed of models are capable of this.
- zachthewf 3y ago
  Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It is certainly not enough for every case, but it is enough (and the models are smart enough) to be useful.
  - LunaSea 3y ago
    > Today the largest context window for GPT-4 is 128k tokens. That's around 300 pages of English text. It's a lot of code! It really isn't. I just did a test with a random TypeScript file with 400 lines of code which translates to almost exactly 3000 tokens in the GPT-4 token counter app. This means that we are talking about being capable to analyse a maximum of 43 small / medium sized files. Each analysis will also cost $1.28 without any additional intermediary costs.
ciex 3y ago
They have this as an experimental feature (which is opt-in and I won't try it)
- zachthewf 3y ago
  I didn't know that. Maybe it's not good enough so they figured they had to train on customer data to ship something high quality, hence this change.
Lariscus 3y ago
Just what I need, pull requests with a bunch of AI generated drivel. I really don't want to spend my days figuring out if the PR that just got assigned to me contains an actual issue or some AI hallucination.
- zachthewf 3y ago
  If it's attached to a Sentry error, there's clearly some issue which produces the exception.

happytiger 3y ago

They want the data for AI but not just for their products. They want it as a business asset. It helps boost their valuation and makes them attractive to potential acquirers, but they also can sell de-identified data outside the company per their own terms. That just means they don’t sell PII for less than two customers in a data identifiable group. See the terms: > Sentry may use Usage Data and Service Data for analytics and product development (including to train or improve AI Features and generate Outputs). Sentry will not disclose Usage Data or Outputs externally, unless they have been de-identified so that they do not individually identify Customer, its Users or any other person and are aggregated with data across Sentry’s other customers.

the_mitsuhiko 3y ago
> They want the data for AI but not just for their products. They want it as a business asset. That is incorrect, the data is exclusively used to provide services to customers. The blog post is quite specific about the intended use.
- happytiger 3y ago
  That is NOT what their own terms and conditions, which I’m quoting in the comment you are replying to actually says: > Sentry will not disclose Usage Data or Outputs externally, unless they have been de-identified so that they do not individually identify Customer, its Users or any other person and are aggregated with data across Sentry’s other customers. You can write whatever you want in a blog post. It’s the legal terms and what’s in the contract that matters and presumably shows intent. In this case they are clearly planning to show or sell de-identified data as it’s in their terms and conditions explicitly. What is being said in the blog and what the company is giving themselves a license to do are not the same.
  - the_mitsuhiko 3y ago
    We put an update into the blog post that addresses this: https://blog.sentry.io/ai-privacy-and-terms-of-service-updates/ https://blog.sentry.io/ai-privacy-and-terms-of-service-updat...

JimDabell 3y ago

At the moment they have a feature where you can click on a button on a crash report and it will use generative AI to explain the problem / suggest a fix. Sometimes it works, sometimes it doesn’t. They also have features tied into version control where you can tag a commit to close a crash report etc. So I would expect them to be able to train on the data this gives them to suggest more accurate fixes down to the source code level.

the_mitsuhiko 3y ago

> What would they benefit from this? Today one of the largest challenge with Sentry at scale is that every error looks the same. If statements only get you that far. One of the ways in which we want to improve this is to use issue interaction and the event's content to predict the severity of an issue. The second case is a version of suggested fix that uses your code to suggest actual fixes to your code. In this case the data never crosses your organization's boundary and is private to yourself.