5 ms·
We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it pr
by Weryj 1y ago
We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.
The overhead at low volume is pretty high, but in the higher volumes (25M transactions/24h) it's a massive cost saving for us.
Edit:
There were just some initial headaches with needing to increase kafka partitions and add replications to the transaction processors, otherwise we didn't quite leverage the available compute and the backpressure would fill Redis up until OOM.
- vanschelven 1y agoYeah I fully get how that's a volume where going self-hosted Sentry makes perfect sense at the bottom line and including any upkeep you might have. Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day. [0] https://www.bugsink.com/scalable-and-reliable/ https://www.bugsink.com/scalable-and-reliable/
- selcuka 1y ago> Bugsink's also quite scalable[0], but I wouldn't recommend it a 25M/day. Well, your homepage disagrees with this statement: > Bugsink can deal with millions of events per day on dirt cheap hardware
- kelnos 1y agoTo me, "millions" usually means less than 10M. 25M falls into "tens of millions" to me. But it's a very fuzzy way of quantifying something, and open to various interpretations.
- selcuka 1y agoFair point, but if it can cope with millions on "dirt cheap hardware" it logically follows that it can do 25M on more expensive hardware.
- pebble 1y agoSame here with the community maintained Helm chart. Not the easiest thing but quite reasonable for almost two years now. This is for 50M transactions per month and we're seeing massive cost savings compared to SaaS at this volume as well. For those interested in only errors, the self-hosted version recently introduced errors-only mode which should cut down on the containers.
- lnenad 1y ago> It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands. I think this is a repeated question but... are you considering the cost of the people managing the deployment, security oversight, dealing with downtime etc?
- bayindirh 1y agoIf you can keep the people doing all the things, they become cheaper over time. Because as your system settles and people become more competent, both downtime and effort required to mend these problems reduce dramatically, and you can give more responsibilities to the same people without overloading them. Disclosure: I'm a sysadmin.
- eptcyka 1y agoI wonder what is your managers take on this, given your incentives here.
- bayindirh 1y agoHonestly asking, what my incentives are looking like from there?
- eptcyka 1y agoYou are incentivised to argue that it is good to keep employing sysadmins for self hosting, because that will keep you employed. You have a monetary incentive, thus you are a bit biased, in my opinion.
- bayindirh 1y agoI think I didn't elaborate my point enough, so there's a misunderstanding. What I said is true for places where they already have sysadmins for various tasks. For the job I do (it's easy to find), you have to employ system administrations to begin with. So, at least for my job, working the way I described in my original comment is the modus operandi for the job itself. If the company you're working in doesn't prefer self-hosting things, and doesn't need system administrators for anything, you might be true, but having a couple of capable sysadmins on board both enables self-hosting and allows this initiative to grow without much extra cost, because it gets cheaper as the sysadmins learn and understand what they're doing, so they can handle more things with the same/less effort. See, system administrators are lazy people. They'd rather solve problems for once and for all and play PacMan in their spare time.
- tgv 1y agoAm I reading correctly that your software generates 25 million error messages per day?
- xmodem 1y agoSentry does a lot more than tracking errors - presumably most of those transactions are 'breadcrumb'-style events.
- pebble 1y agoThat is more likely performance traces or session replays.
- Weryj 1y agoNope, 25M transactions. In Sentry a transaction is more like an OTEL-trace. Errors are much lower ;)
- almd 1y agosorry if this is a silly question, the quick google search didn’t give me clues. Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response?
- Volundr 1y ago> Transactions like full user flows start to finish, or 1 transaction = 1 post/get and 1 response? For most applications we are talking closer to 1 transportation 1 web request. Distributed tracing across microservices is possible, the level of extra effort required depends on your stack. But that's also the out of the box, plug and play stuff. With lower level APIs you define your own transactions, when they start and end, which is needed for tracing applications where there isn't a built in framework integration (e.x not a web application).
- Weryj 1y agoI do have one major complaint though, in dotnet, the tracing/errors are always captured regardless of the sampling rate. So you end up with a lot more memory usage on high throughput/low memory services with no way to lower it. There's a ticket now open to stop this, but it's still in progress.
- brungarc 1y agoAvoiding allocations when a transaction isn’t sampled should be pretty trivial. Only gotcha is that you’d still trace id propagation to tie errors together regardless of transactions. But tracing and transactions got decoupled a while ago so shouldn’t be a big problem. I’ll leave a comment on ticket pointing to this post
- parthdesai 1y agoIt's open source, you guys could always create a PR to fix it. That's the power of open source!
- no_wizard 1y agothere's no guarantee it will get merged though, even if a PR is created. Forking has down sides that can't be hand waved away too, especially for a service like this.
- 5Qn8mNbc2FNCiVV 1y agoIt's just a client library, what's the alternative, put a proxy in front that drops 90% of the events
- zeeg 1y agoAny chance you can link me to the ticket? Feel free to email - david at sentry
- Weryj 1y agoIt looks like there was some motion a week ago. https://github.com/getsentry/sentry-dotnet/issues/3636#event-17195720737 https://github.com/getsentry/sentry-dotnet/issues/3636#event...
- iJohnDoe 1y ago> a high-end server. 96c, 512gb. It ends up only costing around $300 a month Wow, that's really cheap. I'm seriously overpaying for my cloud provider and need to try Hetzner. I always assumed Hetzner was only European based.
- mdaniel 1y agoJust be forewarned it doesn't seem to offer one iota of IAM, so whether or not one is "overpaying" for a cloud provider depends on what you're getting from them. If you mean "rent a machine," then likely. If you mean "have the machines heal themselves instead of Pagerduty waking me up" then reasonable people can differ about where that money is going FWIW, https://lowendbox.com/ https://lowendbox.com/ is good fun for the former set of things, too