11 ms·

How Sentry Receives 20B Events per Month While Preparing to Handle Twice That

164 points by ryangoldman 9y ago

mclarke 9y ago
seriously though, that gif of Matt is excellent
- buryat 9y ago
  yeah, 9MiB for a gif is excellent
  - raresp 9y ago
    have you heard about Lazy Image Load?
    - mattrobenolt 9y ago
      Why would you want to lazy load it?
DmitryOlshansky 9y ago
Hate these X millions / billions per month or year or a centuary. Let’s do the math. So it’s 20B/30 = 660 m / day 660 / 24 ~= 25 m / hour 25000k / 3.6k sec = 8000 events/sec Now peaks got to be larger then these measly 8000 events. Even say peak is 50k/second. Dang an Nginx can serve a web page faster then that on a single modern notebook. CPU won’t even saturate to do that. Or if you were to write them to a consumer grade spinning drive disk at 100mb/s you’d get 50k/s if each event is 2kbytes in size. As usual I’m not impressed.
- zeeg 9y ago
  We actually do a lot more than 20B (not sure what the number is actually pulled from), but it's also important to note that even 8000 events/sec isn't free. - 40k per event (average, it gets into 2mb territory for some) - streaming processing of everything on the fly - storage/querying of data
  - DmitryOlshansky 9y ago
    I hear you, at the same time you have embarassingly parallel problem that can be easily sharded per customer. (Do I understand your service correctly?) Processing 8k * 40k is a feat but I’d think you have more then a few dozens of nodes. Then it gets into ~100s ev/s per node territory. Quite manageable even for e.g. complex search engines to ingress. Of course, it all depends on where your bottlenecks are.
    - the_mitsuhiko 9y ago
      > I hear you, at the same time you have embarassingly parallel problem that can be easily sharded per customer. (Do I understand your service correctly?) The actual event ingestion from an external consumer to our queue is indeed very boring. What makes Sentry internally complex is how incredibly complicated those events can get. For instance before we can make sense of most iOS (as wlel as other native) or JavaScript events we need to go through a fairly complex event processing system. A lot of thought went into making this process very efficient. Also obviously even if a problem like ours can be easily solved in parallel, you don't just want to add more servers you also want to run a reasonably lean operation. Instead of just throwing more machines at the issue we want to make sure we internally do not do completely crazy things but optimize our processes.
      - le-mark 9y ago
        This is an interesting line of discussion. How do you know you are optimally utilizing the hardware you do have? For example, is each machine running at near 100% cpu? Is the network card saturated? Is the database the bottleneck? How about the search indexing operation? What metrics are you tracking and how are you tracking them? If you're at liberty to discuss, it would be highly enlightening!
        the_mitsuhiko 9y ago
        > How do you know you are optimally utilizing the hardware you do have? That's probably one better to answer for actual ops people. However what we generally do is looking at our metrics to see where we see room for improvement. Two obvious cases we identified and resolved through improving our approaches have been JavaScript sourcemap processing and debug symbol handling. There we knew we were slow based on how far p95 timing was from the average and median. We looked specifically at what problematic event types looked like and what we can do to handle them better. We now use just better approaches to dealing with this data, multiple layers of more intelligent caches and that cut down out event processing time and with it the amount of actual hardware we need. So the answer at least as far as event processing goes: for some cases we spent time on we were CPU bound and knew we can do better and did. I wrote a bit about what we did on our blog where we moved sourcemap processing from Python to Rust. The latests big wins in the processing department have been a rewrite of our dSYM handling where we now use a custom library to preprocess DWARF files and various symbol tables into a more compact format so we can do much faster lookups for precisely the data we are about. > Is the network card saturated? In other cases that has been an issue, but that's less of a problem on the actual workers which are relevant to that number we are discussing here. In particular as far as the load balancers goes there is a ton more data coming in but not all will make it to the system. A non uncommon case is someone deploys our code into a mobile app SDK and then stops paying. The apps will still spam us, but we want to block this off before it hits the expensive parts of the system etc. > Is the database the bottleneck? When is a database not a concern ;) What we are tracking is a ton, how we are tracking is a lot of datadog.
- Kiro 9y ago
  How much is a lot for you? 1 billion/s?
  - DmitryOlshansky 9y ago
    At 8k/s given a bunch of hardware a typical cloud service will throw at it it can be down to ~100ev/s per node. Yes I’d like at least 1k/s per node. That’s something interesting.
- StavrosK 9y ago
  > Hate these X millions / billions per month or year or a centuary. I hear you, I hate misleading examples too. > Dang an Nginx can serve a web page faster then that on a single modern notebook. "What? Your moon-rocket can only fit three people? My couch can do six. I am not impressed."
  - DmitryOlshansky 9y ago
    I just gave some examples of what can be done at similar speed on dirt cheap hardware. I my view it helps to understand scale. More real world example an API at my work logs routinely more then 20M+ events per day, all indexed for NRT search. It’s just a byproduct of the actual work though. It’s considered business as usual, nobody counts how it adds up half a billion per month. I assume they index it in some search engine as well. Yeah the scale is x20 bigger, but it’s their _primary_ task.
    - the_mitsuhiko 9y ago
      > More real world example an API at my work logs routinely more then 20M+ events per day, all indexed for NRT search. It’s just a byproduct of the actual work though. But do your events require you to process gigabytes of debug symbol data per event? Or through megabytes of JSON sourcemap data to unobfuscate stacktrace where some operations are not even possible in linear time? Don't forget a Sentry event is not a simple data blob we store. It requires very expensive and complex processing.
      - deleted 9y ago
        [deleted]
      - DmitryOlshansky 9y ago
        Our events are a mess of text that is processed with some convoluted parser to build several cross-reference indexes. At least it allows to piece together events from the same request, etc. But the point is that it’s at best 1% of all the work being done, and it’s not the key problem solved by the system. Parsing a stack frame would be actually in the same ballpark. Yes, processing it would involve maybe a ~100 index lookups to match stack frames to the source etc, but see below. Gigabytes of debug symbols? Such as fitting all in a 1/10 of RAM of a typical server you mean? As a hash table, or heck even a trie so you could do range query as well. Same about a source map. It’s not THAT hard to build a largish lookup table entirely in RAM. Especially if its the MAIN POINT of your business. I’m curious about that non-linear problem with JSON source map. Could you elaborate? Also you don’t need to cross-reference symbols/events across your customers. The lookup is thus limited to a shard of that client which I’d dare say won’t contain gigabytes of debug symbols.
- aidos 9y ago
  I am impressed. Sentry is a brilliant bit of code available for everyone to download and use. It just works and gets better all the time. And really, in terms of load - at that scale it's all pretty impressive. The payloads are big. They contain the full stack trace along with most the local vars.
- ubercore 9y ago
  I mean, serving a web page is easy. Processing incoming events and everything else that Sentry does is not trivial. You're not comparing likes to likes.
  - notyourday 9y ago
    If that's the case why NH littered with these kinds of issues? https://news.ycombinator.com/item?id=15584315 https://news.ycombinator.com/item?id=15584315
    - ubercore 9y ago
      I mean serving a web page, from nginx, which is what the person was talking about. Not a database-driven web application.
  - DmitryOlshansky 9y ago
    Yes, but they don’t run the whole thing on a single laptop do they? Roughly speaking you take what a laptop can do and a multiply it by 1000x to estimate what a cloud service could easily do. (It’s just an estimate, the reality is not working like that, I KNOW) Do not take it as Nginx and webpages example, it’s indeed a poor analogy. I can’t remember the right number for e.g. ElasticSearch fuzzy search across 10m documents of out hand so I’d pick the simpler less relevant number. Actually fuzzy full text search across 10m documents is at least in the range of 100s/sec on a laptop. Meaning that they have resources to do at least e.g. 100k-500k/s full text searches across gigabytes of symbols. I bet their queries are simpler then that though (not fuzzy nor full text). It’s debug symbols after all not a natural text.
- shallot_router 9y ago
  Serving a static web page 8-50k times per second and doing some really complex processing 8-50k times per second are worlds apart.
- ktamura 9y ago
  At Treasure Data, we ingest about 1-2 million events per second. Using your #, this translates to ~6 trillion events per month. If you (or anyone on this thread) is interested in learning more, please feel free to ping me.
tyingq 9y ago
20B/month is interesting in the sense that Sentry should only be getting a transaction when there's an error in some third party application. Makes you wonder if a couple of their clients paid for this error monitoring service, then quit watching it. That's a lot of errors.
- comboy 9y ago
  If you deploy some non-edge-case bug then you can easily generate thousands of events before you even receive the notification - assuming your service is any popular, but even if it's not it can be e.g. some stuck background job. Btw, it's on average 7.5k events per second. Not that drastic but of course peak, not average is what matters.
  - tyingq 9y ago
    I get the idea, but I would think the right use of the product is to fix the root causes, or suppress errors that aren't actionable. I was basing this off of somewhere on the site that mentioned roughly 200 companies using it. Perhaps it's grown. Would be interesting to know the total per month after they are deduplicated.
    - zeeg 9y ago
      Not sure where you'd see 200 companies (we've had more than that for most of Sentry's lifetime). Not every team lines up to a company, but napkin math I'd say we have 10k - 20k companies using our cloud service. That's also excluding all of the open source installations, which is a pretty large number.
  - the_mitsuhiko 9y ago
    disclaimer: Armin from Sentry here I'm not super happy that this number is in the title because I find this number myself very irrelevant to our operations (not even sure this number is correct, it sounds too small). What's more challenging to us is the complexity of individual events and the bursting of it. A lot of interesting engineering went into that.
synthc 9y ago
too bad the dashboard is slow as molasses
- zeeg 9y ago
  Could you reach out to support with your account details? There's a cost to the data volume, but generally queries should respond very quickly.
austinpray 9y ago
I had the pleasure of hanging out with the Sentry folks in SF for a couple days last week. Honestly a top-notch group. It was eye-opening to see what is possible when you have an organization like theirs where you have a lot of talent and pride in your product across the board.
debacle 9y ago
How does this compare to other competitors in the RUM space, including open source tools?
- the_mitsuhiko 9y ago
  I'm not sure what rum is (other than an alcoholic beverage) but as far as I know Sentry is the only actively maintained Open Source tool in its space (which is crash reporting). Might be wrong on that though.
  - rkwz 9y ago
    Real User Monitoring https://en.wikipedia.org/wiki/Real_user_monitoring https://en.wikipedia.org/wiki/Real_user_monitoring
dabernathy89 9y ago
We started using paid Sentry recently (previously were on an old self-hosted version). It's mostly been great, but boy is search clunky. Very difficult to find anything that doesn't automatically appear on the first page of recent events.
- zeeg 9y ago
  It's definitely on our short list of problems to solve, but unfortunately the solution takes a bit more than traditional feature development.
  - Waterluvian 9y ago
    This statement has me very curious. Are you able to elaborate at all on what you mean? Like is this more of an underpinning architecture issue that can't simply be addressed by scheduling dev time?
    - korzun 9y ago
      Data collection and organization at that scale is extremely complex.
    - zeeg 9y ago
      The storage mechanism isn't fast/flexible enough to do anything really great. It's fairly easy for us to do precise matches (with some caveats), but given everything is cached at the database level we can't easily compute results based on your search query. Effectively what we do is: SELECT * from groups where ID IN ( SELECT group_id FROM index WHERE key = :searchKey AND value =:searchValue UNION [...] ) (This is a simplification but it should give you an idea of how the queries are built) Because of the way the model works, its very hard to do certain things like exclusion queries, and more importantly all of the results you're seeing are still cached. The biggest pain point here is if you're searching for e.g. "ConnectionError environment:production", you really don't want to see anything related to non-production. We're solving that problem immediately, but its just the tip of the iceberg. Next year we're kicking off a large project to overhaul the key infrastructure which powers the stream/search functionality, with some pretty big ambitions. Other services in the space generally use something like Elastic search, which can provide some of this out of the box. We've always been built on SQL/Redis, and given that Elastic has its own set of problems we've decided that it's likely best for us to move to a columnar store format that doesn't cache results (e.g. counts), but rather computes them in real-time much like Scuba.
baristaGeek 9y ago
We're using it and it 'magically' detects errors in CI that no human would've had detected.
Yokohiii 9y ago
tl;dr we are hiring
CyberDildonics 9y ago
Is 7,700 per second really that impressive?

164 points by ryangoldman 9y ago

mclarke 9y ago

seriously though, that gif of Matt is excellent

buryat 9y ago
yeah, 9MiB for a gif is excellent
- raresp 9y ago
  have you heard about Lazy Image Load?
  - mattrobenolt 9y ago
    Why would you want to lazy load it?

DmitryOlshansky 9y ago

Hate these X millions / billions per month or year or a centuary. Let’s do the math. So it’s 20B/30 = 660 m / day 660 / 24 ~= 25 m / hour 25000k / 3.6k sec = 8000 events/sec Now peaks got to be larger then these measly 8000 events. Even say peak is 50k/second. Dang an Nginx can serve a web page faster then that on a single modern notebook. CPU won’t even saturate to do that. Or if you were to write them to a consumer grade spinning drive disk at 100mb/s you’d get 50k/s if each event is 2kbytes in size. As usual I’m not impressed.

zeeg 9y ago
We actually do a lot more than 20B (not sure what the number is actually pulled from), but it's also important to note that even 8000 events/sec isn't free. - 40k per event (average, it gets into 2mb territory for some) - streaming processing of everything on the fly - storage/querying of data
- DmitryOlshansky 9y ago
  I hear you, at the same time you have embarassingly parallel problem that can be easily sharded per customer. (Do I understand your service correctly?) Processing 8k * 40k is a feat but I’d think you have more then a few dozens of nodes. Then it gets into ~100s ev/s per node territory. Quite manageable even for e.g. complex search engines to ingress. Of course, it all depends on where your bottlenecks are.
  - the_mitsuhiko 9y ago
    > I hear you, at the same time you have embarassingly parallel problem that can be easily sharded per customer. (Do I understand your service correctly?) The actual event ingestion from an external consumer to our queue is indeed very boring. What makes Sentry internally complex is how incredibly complicated those events can get. For instance before we can make sense of most iOS (as wlel as other native) or JavaScript events we need to go through a fairly complex event processing system. A lot of thought went into making this process very efficient. Also obviously even if a problem like ours can be easily solved in parallel, you don't just want to add more servers you also want to run a reasonably lean operation. Instead of just throwing more machines at the issue we want to make sure we internally do not do completely crazy things but optimize our processes.
    - le-mark 9y ago
      This is an interesting line of discussion. How do you know you are optimally utilizing the hardware you do have? For example, is each machine running at near 100% cpu? Is the network card saturated? Is the database the bottleneck? How about the search indexing operation? What metrics are you tracking and how are you tracking them? If you're at liberty to discuss, it would be highly enlightening!
      - the_mitsuhiko 9y ago
        > How do you know you are optimally utilizing the hardware you do have? That's probably one better to answer for actual ops people. However what we generally do is looking at our metrics to see where we see room for improvement. Two obvious cases we identified and resolved through improving our approaches have been JavaScript sourcemap processing and debug symbol handling. There we knew we were slow based on how far p95 timing was from the average and median. We looked specifically at what problematic event types looked like and what we can do to handle them better. We now use just better approaches to dealing with this data, multiple layers of more intelligent caches and that cut down out event processing time and with it the amount of actual hardware we need. So the answer at least as far as event processing goes: for some cases we spent time on we were CPU bound and knew we can do better and did. I wrote a bit about what we did on our blog where we moved sourcemap processing from Python to Rust. The latests big wins in the processing department have been a rewrite of our dSYM handling where we now use a custom library to preprocess DWARF files and various symbol tables into a more compact format so we can do much faster lookups for precisely the data we are about. > Is the network card saturated? In other cases that has been an issue, but that's less of a problem on the actual workers which are relevant to that number we are discussing here. In particular as far as the load balancers goes there is a ton more data coming in but not all will make it to the system. A non uncommon case is someone deploys our code into a mobile app SDK and then stops paying. The apps will still spam us, but we want to block this off before it hits the expensive parts of the system etc. > Is the database the bottleneck? When is a database not a concern ;) What we are tracking is a ton, how we are tracking is a lot of datadog.
Kiro 9y ago
How much is a lot for you? 1 billion/s?
- DmitryOlshansky 9y ago
  At 8k/s given a bunch of hardware a typical cloud service will throw at it it can be down to ~100ev/s per node. Yes I’d like at least 1k/s per node. That’s something interesting.
StavrosK 9y ago
> Hate these X millions / billions per month or year or a centuary. I hear you, I hate misleading examples too. > Dang an Nginx can serve a web page faster then that on a single modern notebook. "What? Your moon-rocket can only fit three people? My couch can do six. I am not impressed."
- DmitryOlshansky 9y ago
  I just gave some examples of what can be done at similar speed on dirt cheap hardware. I my view it helps to understand scale. More real world example an API at my work logs routinely more then 20M+ events per day, all indexed for NRT search. It’s just a byproduct of the actual work though. It’s considered business as usual, nobody counts how it adds up half a billion per month. I assume they index it in some search engine as well. Yeah the scale is x20 bigger, but it’s their _primary_ task.
  - the_mitsuhiko 9y ago
    > More real world example an API at my work logs routinely more then 20M+ events per day, all indexed for NRT search. It’s just a byproduct of the actual work though. But do your events require you to process gigabytes of debug symbol data per event? Or through megabytes of JSON sourcemap data to unobfuscate stacktrace where some operations are not even possible in linear time? Don't forget a Sentry event is not a simple data blob we store. It requires very expensive and complex processing.
    - deleted 9y ago
      [deleted]
    - DmitryOlshansky 9y ago
      Our events are a mess of text that is processed with some convoluted parser to build several cross-reference indexes. At least it allows to piece together events from the same request, etc. But the point is that it’s at best 1% of all the work being done, and it’s not the key problem solved by the system. Parsing a stack frame would be actually in the same ballpark. Yes, processing it would involve maybe a ~100 index lookups to match stack frames to the source etc, but see below. Gigabytes of debug symbols? Such as fitting all in a 1/10 of RAM of a typical server you mean? As a hash table, or heck even a trie so you could do range query as well. Same about a source map. It’s not THAT hard to build a largish lookup table entirely in RAM. Especially if its the MAIN POINT of your business. I’m curious about that non-linear problem with JSON source map. Could you elaborate? Also you don’t need to cross-reference symbols/events across your customers. The lookup is thus limited to a shard of that client which I’d dare say won’t contain gigabytes of debug symbols.
aidos 9y ago
I am impressed. Sentry is a brilliant bit of code available for everyone to download and use. It just works and gets better all the time. And really, in terms of load - at that scale it's all pretty impressive. The payloads are big. They contain the full stack trace along with most the local vars.
ubercore 9y ago
I mean, serving a web page is easy. Processing incoming events and everything else that Sentry does is not trivial. You're not comparing likes to likes.
- notyourday 9y ago
  If that's the case why NH littered with these kinds of issues? https://news.ycombinator.com/item?id=15584315 https://news.ycombinator.com/item?id=15584315
  - ubercore 9y ago
    I mean serving a web page, from nginx, which is what the person was talking about. Not a database-driven web application.
- DmitryOlshansky 9y ago
  Yes, but they don’t run the whole thing on a single laptop do they? Roughly speaking you take what a laptop can do and a multiply it by 1000x to estimate what a cloud service could easily do. (It’s just an estimate, the reality is not working like that, I KNOW) Do not take it as Nginx and webpages example, it’s indeed a poor analogy. I can’t remember the right number for e.g. ElasticSearch fuzzy search across 10m documents of out hand so I’d pick the simpler less relevant number. Actually fuzzy full text search across 10m documents is at least in the range of 100s/sec on a laptop. Meaning that they have resources to do at least e.g. 100k-500k/s full text searches across gigabytes of symbols. I bet their queries are simpler then that though (not fuzzy nor full text). It’s debug symbols after all not a natural text.
shallot_router 9y ago
Serving a static web page 8-50k times per second and doing some really complex processing 8-50k times per second are worlds apart.
ktamura 9y ago
At Treasure Data, we ingest about 1-2 million events per second. Using your #, this translates to ~6 trillion events per month. If you (or anyone on this thread) is interested in learning more, please feel free to ping me.

tyingq 9y ago

20B/month is interesting in the sense that Sentry should only be getting a transaction when there's an error in some third party application. Makes you wonder if a couple of their clients paid for this error monitoring service, then quit watching it. That's a lot of errors.

comboy 9y ago
If you deploy some non-edge-case bug then you can easily generate thousands of events before you even receive the notification - assuming your service is any popular, but even if it's not it can be e.g. some stuck background job. Btw, it's on average 7.5k events per second. Not that drastic but of course peak, not average is what matters.
- tyingq 9y ago
  I get the idea, but I would think the right use of the product is to fix the root causes, or suppress errors that aren't actionable. I was basing this off of somewhere on the site that mentioned roughly 200 companies using it. Perhaps it's grown. Would be interesting to know the total per month after they are deduplicated.
  - zeeg 9y ago
    Not sure where you'd see 200 companies (we've had more than that for most of Sentry's lifetime). Not every team lines up to a company, but napkin math I'd say we have 10k - 20k companies using our cloud service. That's also excluding all of the open source installations, which is a pretty large number.
- the_mitsuhiko 9y ago
  disclaimer: Armin from Sentry here I'm not super happy that this number is in the title because I find this number myself very irrelevant to our operations (not even sure this number is correct, it sounds too small). What's more challenging to us is the complexity of individual events and the bursting of it. A lot of interesting engineering went into that.

synthc 9y ago

too bad the dashboard is slow as molasses

zeeg 9y ago
Could you reach out to support with your account details? There's a cost to the data volume, but generally queries should respond very quickly.

austinpray 9y ago

I had the pleasure of hanging out with the Sentry folks in SF for a couple days last week. Honestly a top-notch group. It was eye-opening to see what is possible when you have an organization like theirs where you have a lot of talent and pride in your product across the board.

debacle 9y ago

How does this compare to other competitors in the RUM space, including open source tools?

the_mitsuhiko 9y ago
I'm not sure what rum is (other than an alcoholic beverage) but as far as I know Sentry is the only actively maintained Open Source tool in its space (which is crash reporting). Might be wrong on that though.
- rkwz 9y ago
  Real User Monitoring https://en.wikipedia.org/wiki/Real_user_monitoring https://en.wikipedia.org/wiki/Real_user_monitoring

dabernathy89 9y ago

We started using paid Sentry recently (previously were on an old self-hosted version). It's mostly been great, but boy is search clunky. Very difficult to find anything that doesn't automatically appear on the first page of recent events.

zeeg 9y ago
It's definitely on our short list of problems to solve, but unfortunately the solution takes a bit more than traditional feature development.
- Waterluvian 9y ago
  This statement has me very curious. Are you able to elaborate at all on what you mean? Like is this more of an underpinning architecture issue that can't simply be addressed by scheduling dev time?
  - korzun 9y ago
    Data collection and organization at that scale is extremely complex.
  - zeeg 9y ago
    The storage mechanism isn't fast/flexible enough to do anything really great. It's fairly easy for us to do precise matches (with some caveats), but given everything is cached at the database level we can't easily compute results based on your search query. Effectively what we do is: SELECT * from groups where ID IN ( SELECT group_id FROM index WHERE key = :searchKey AND value =:searchValue UNION [...] ) (This is a simplification but it should give you an idea of how the queries are built) Because of the way the model works, its very hard to do certain things like exclusion queries, and more importantly all of the results you're seeing are still cached. The biggest pain point here is if you're searching for e.g. "ConnectionError environment:production", you really don't want to see anything related to non-production. We're solving that problem immediately, but its just the tip of the iceberg. Next year we're kicking off a large project to overhaul the key infrastructure which powers the stream/search functionality, with some pretty big ambitions. Other services in the space generally use something like Elastic search, which can provide some of this out of the box. We've always been built on SQL/Redis, and given that Elastic has its own set of problems we've decided that it's likely best for us to move to a columnar store format that doesn't cache results (e.g. counts), but rather computes them in real-time much like Scuba.

baristaGeek 9y ago

We're using it and it 'magically' detects errors in CI that no human would've had detected.

Yokohiii 9y ago

tl;dr we are hiring

CyberDildonics 9y ago

Is 7,700 per second really that impressive?