8 ms·
Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
by carlosdp 6y ago
Gotta say though, having rolled grafana and prometheus and such on my own plenty of times before, if you are a startup and can afford Datadog, use Datadog.
- Hates_ 6y agoI'd be interested to know a bit of detail as we're looking into Grafana/Promethus a bit.
- dharmab 6y agoPrometheus is a very needy child in terms of data volume and hardware resources. Running it is at least one engineers' full time job- if you're a startup, you can outsource monitoring for a tiny fraction of the price, then move to Prometheus later if you are successful.
- edoceo 6y agoMy company's Prometheus setup was super easy, one $10/mo box. About 1 week of fiddling all the exporters and configs but now it just runs and has for months. You can be small with Prometheus and grow into needing an FTE for it - w/o having the migration hurdle of moving out-source to in-source
- lima 6y agoPlease elaborate - how is it one engineer's full time job? We run Prometheus in production and this hasn't been our experience at all. A single machine can easily handle hundreds of thousands of time series, performance is good, and maintaining the alerting rules is a shared responsibility for the entire team (as it should be).
- gen220 6y agoI think the parent's complaint is a function of how your engineering org "uses" prometheus. If you use it as a store for all time series data generated by your business, and you want to have indefinite or very-long-term storage, managing prometheus does become a challenge. (hence m3, chronosphere, endless other companies and tech built to scale the backend of prometheus). IMO, this is a misuse of the technology, but a lot of unicorn startups have invested a lot of engineering resources into using it this way. And a lot of new companies are using it this way; hence the "one engineer's FT job".
- dharmab 6y agoI'd agree; I'm at a large corp that has a need to store our data for a very long term. If we were using Prometheus as an ephemeral/short term TSDB to drive alerting only, it would be really easy.
- user5994461 6y agoFor reference: A single machine can push 5k metrics so you're saying a single prometheus instance can easily serve 20s of hosts. lol
- sevagh 6y agoFalse in my experience. Full-time job? After the initial learning curve, a simple 2x redundant Prometheus poller setup on can last for a long time. Ours lasted for 30,000,000 timeseries until encountering performance issues. After that, we needed some more effort to scale out horizontally with Thanos, but again, once it's set up, it maintains itself.
- ciguy 6y agoThis is good advice, however you will also want to make sure you have a plan to get off datadog when you grow. Datadog is one of the easiest to use and most comprehensive out of the box. But it gets really expensive as you begin to scale up and add servers, cloud accounts, services etc.... At a certain scale, rolling your own monitoring and alerting becomes cost effective again as Datadog begins to charge an arm and a leg. I've seen Datadog bills that could easily pay for 2 full time engineers.
- lima 6y agoWe migrated customers off DataDog for this exact reason. Rolling a custom Prometheus / Grafana / Alertmanager setup is not hard at all, more powerful, and it's much easier to do it right from the start.
- user5994461 6y ago> I've seen Datadog bills that could easily pay for 2 full time engineers. Which means the companies are running thousands of hosts. So they definitely need both datadog and a full team of sysadmin/devops/SRE to handle that infrastructure.
- carlosdp 6y agoOn the contrary, the bigger you get, the more you need Datadog to scale. Once you grow big enough, you can make volume deals with them, you don't pay list price. I'd much rather pay them than have a single engineer have to spend any time managing infrastructure that isn't core to our product, especially since Datadog will always do it better.
- ciguy 6y agoI respectfully strongly disagree. At a certain scale when you can afford to have a full-time engineer working on it, open source tooling will give you a better, cheaper and more flexible solution since it's easy to customize. Datadog is good for the mainstream cases but it's not all that flexible for unusual or edge cases.
- Axsuul 6y agoAs a counterpoint: I have been rolling my own Grafana/Prometheus for many years for my startup and it's been pretty trivial
- risyachka 6y agoSame here. DataDog would cost me at least 10x.
- yellow_mixer 6y agoHow are you running it? On AWS? What’s your infra look like.
- Axsuul 6y agoGCP with Docker Swarm, scheduled on the same node (n2-standard-4). Let me know if any other questions
- carlosdp 6y agoDoes "scheduled on the same node" imply you are running everything on one node?
- Axsuul 6y agoNope, just that Prometheus and Grafana are running together on the same node. My entire infrastructure is multi-node however.
- yellow_mixer 6y agoHow much data are you sending and ingesting?
- Axsuul 6y agoI don't have any hard numbers but quite a bit. Everyday we process millions of background jobs, thousands of database queries per second, etc. and stats are collected and sent to Prometheus for all those.
- dijit 6y agoI just costed a datadog deployment (based on your comment) and it would cost me my yearly salary every month. No thanks. :/
- rattray 6y agoCan you say something vague about your deployment scale?
- dijit 6y ago2,500~ compute instances. Looking at the pricing page the cost $15/instance so $37,500 Middle-of-the-road developer salary is like $35k in most of Europe, outside of the capitals. Although it does say: "Volume discounts available (500+ hosts/mo). Contact us." at the bottom, so I guess 500 is a lot.
- user5994461 6y ago2500 instances could be millions per months in AWS costs. The smallest instances with some disks and bandwidth fees can push 100k a month. Spending a fraction of that to monitor that sort of infrastructure is absolutely justified. I can tell you from experience that datadog gives discount even for 100+ hosts, I don't know what they can do for 2500, but if it were me I wouldn't accept anything less than 50% off. Honestly you need to forgot about your salary, it's irrelevant when it comes to running a company. Imagine a driver in a shipping company deciding to deliver on a scooter rather than a truck because the truck is worth more than his yearly salary.
- dijit 6y agoIt's not millions, but it's the many multiples of hundreds of thousands. (and it's mainly GCP/bare metal). I guess the point I am driving at here is that there's such a thing as "business critical costs" (IE: can we ship our product or not) which is the majority of infra costs we have today, and then there's "optimisation costs". Usually when we discuss things like optimisation costs its along the lines of: "Will this product save us enough time to justify it's expense". Often, sadly, the answer is no. Terraform Enterprise is an example of a time where we said: Yes. -- because the API allows us to deploy CI/CD jobs which provision little versions of our infrastructure, saving us many man-days of time in provisioning and testing every year. As eluded to in the sibling thread, there's almost no way that we can save 3 or more peoples worth of time every year, we're 3 people right now and we have metrics collection, log tracing and alerting already. -- so it's a hard sell to the business types.
- jakozaur 6y agoDataDog per host pricing can be very expensive. Metrics are provided by many platforms. If you need logs too, you may look at Sumo Logic which got way cheaper metrics in typical use case. Disclaimer: I work at Sumo Logic.
- bpodgursky 6y agoLike any SaaS dev tool, when at scale, you negotiate and pay a fraction of the list price. It's meaningless to look at the price of Datadog@5 hosts -- at 500 or 5000, you're paying a completely detached number from the website list price, likely a small fraction.
- sciurus 6y agoBased on my experience at two companies with thousands of hosts on Datadog, this is not true. You'll only be able to negotiate a small discount. You'll also experience their habit of launching new features, waiting a while for customers to adopt them, then starting to charge extra for them. Don't get me wrong, Datadog has great products. But they're also great at extracting money from their customers.
- freehunter 6y agoWhich blows away the reason a lot of people/teams/companies like SaaS. Because there is no negotiation, no sales requisitions, no long lead time while they come up with a quote. You see the price you pay the price you get the service, same as anyone else.
- cloudpundit 6y agoEvery SaaS provider offers discounts based on length of commitment and volume of spend. Most will say look at AWS, but even there you have list prices and private/bulk pricing. Throw in EDPs, negotiated credits, savings plans, reservations, etc and you're nowhere near list prices.
- carlosdp 6y agoIt's expensive for good reason, it's the best out there.
- scaryclam 6y agoThis is not my experience at all. We've got it setup and monitoring all sorts of things. After the initial setup, the only reason we've needed to touch it is when we've introduced new things and wanted to update the config. In fact, data dog was far more of a pain, and far less useful than prometheus has been.