5 ms·
One thing to pay close attention to when making a build v.s. buy decision is the impact that billing models will have on your usage of a tool. Take logging for
by simonw 6y ago
One thing to pay close attention to when making a build v.s. buy decision is the impact that billing models will have on your usage of a tool.
Take logging for example. If you buy a log aggregation platform like Splunk Cloud or Loggly the pricing is likely based on the quantity of data you ingest per day.
This can set up a weird incentive. If you are already close to the limit of your plan, you'll find that engineers are discouraged from logging new things.
This can have a subtle effect on your culture. Engineers who don't want to get into a budgeting conversation will end up avoiding using key tools, and this can cost you a lot of money in terms of invisible lost productivity.
Tools that charge per-head have a similar problem: if your analytics tool charges per head, your junior engineers won't have access to this. This means you won't build a culture where engineers use analytics to help make decisions.
This is a very tricky dynamic. On the one hand it's clearly completely crazy to invest in building your own logging or analytics solutions - you should be spending engineering effort solving the problems that are unique to your company!
But on the other hand, there are significant, hard-to-measure hidden costs of vendors with billing mechanisms that affect your culture in negative ways.
I don't have a solution to this. It's just something I've encountered that makes the "build v.s. buy" decision a lot more subtle than it can first appear.
- tixocloud 6y agoVery interesting perspective. Ran into the same issues where Salesforce was charging us by the number of columns we had that ended up forcing the team to overload a column with a JSON object so we could get it to work without overpaying. Integration and maintenance turned out it be a downright nightmare. Would be interested to hear various pricing models to understand the pros/cons for each especially in the logging/analytics space. Does per storage, per event, per user make sense in certain cases?
- nitrogen 6y agoFor things that are directly generating revenue, it makes sense to have per-X pricing because it aligns your vendors' interests with your own. In that case, you want to incentivize the vendor to maximize your usage, because that usage is making you money.
- beckingz 6y agoI have lived through this same thing with Salesforce. A brutal experience.
- supernovae 6y agoI hate to break it to ya, but if you do cheats like this when you use reliable off the shelf COTS solutions, imagine the shortcuts they're going to do if/when they try it do this themselves. Salesforce isn't cheap, but its not exactly expensive either. However, with this short sighted shortcut you did, you certainly cost your company big. That data is useless. If you upgrade, transition, migrate you now have to spend employee time fixing what was a fixed cost. Human time is expensive. But what is more expensive is putting artificial costs where they don't need to be.
- tixocloud 6y agoCompletely agree. Btw, the abov example wasn’t my company. The CEO just didn’t want to spend. He was also a huge proponent of building everything in-house so we ended up tons of customized software that nobody knew how it ran when people started to leave. The company’s growth eventually stalled and was acquired. Shame because we had some brilliant engineers.
- telesilla 6y agoWe've often gone down the route of 'modify an open source project' for this reason. For the price of a developer, we get the product in perpetuity and don't have these exact limitations you specified of headcount or worrying about usage. For almost everything out there, you can find a version of it that's been made available with source code. Granted, unless you are essentially a software company this is probably difficult to maintain, given the additional level of expertise required. Another option we've found works is buy, to just get started, and invest moving forward in our own project. Getting something off the ground quickly tends to help us understand better the requirements.
- WrtCdEvrydy 6y agoHonestly, I'm fan of the open source self hosted (especially if only some of the enterprise features are paid). We set it up, configure it, evaluate it and buy the enterprise license (it's usually a per-box license).
- gshulegaard 6y agoI love this comment because it articulates a dynamic that I think is often overlooked much better than I could. I might even go a step farther and suggest that anything that is core to your business flywheel should be built and not bought (in general). Mostly because you don't want pricing to act as a disincentive to exercising the flywheel.
- pseudalopex 6y agoThen time is the disincentive.
- gshulegaard 6y agoAh, but if it is core to your business flywheel that is precisely where you should be spending the most time.
- dmlittle 6y agoEven for built systems the cost of ownership and maintenance will grow as the system grows. Take the logging example above and say you're using the Elastic Stack. Things are going pretty well and usage is increasing. Over time your system grows and logging is now present in all of your applications. You've seen your logging rate go from 100 logs/s, to 1,000 logs/s, to 10,000 logs/s and now are encroaching the 100,000 logs/s rate. Over time the amount it cost to host your ES cluster has surely gone up. You probably went from a single node cluster when developing the solution, to a multi-node cluster, to now a hot-warm-cold cluster with several more nodes than you originally had. You also have a dedicated team making sure your in-house logging solution is working and stable. The cost of ownership and maintenance of this system is real. Not only do you need to pay for the huge clusters you'll likely have a dedicated team supporting it. Most people I've talked to in the industry don't like maintaining their in-house logging solutions and would gladly pay for one if it wasn't for the fact that it was too expensive to switch then. Because you need specialized people that know this logging solution it's also hard to hire for this role when the person currently maintaining this system eventually leaves (because he doesn't want to do this anymore). At the end of the day it's cheaper to pay for a logging solution even if it's in the hundreds of thousands of dollars a year than it is to build your own.
- jokethrowaway 6y agoMemories of developers not logging stuff on datadog not to go over limit. We ended up running our own ELK stack and it was much better.
- KenanSulayman 6y agoELK is killing it. Especially with their apm server.
- yagi_baba 6y agoI've lived that life with Datadog. I was the dev who was the most into logging so I got voluntold to be in charge of the Datadog rollout. Told everyone it was not a good idea and presented cost estimates to why, we're better off just keeping our own logs and using one of many open source projects that let you pull in logs and analyze them. But all the cool kids were on Datadog, so onto it we went. What killed us was the combo of the desire to really aggressive log on the staging-master and release-candidate environments (five envs total) and keep the logs for more than 7 days so the QA team could compare the logs from the current release candidate to the previous release candidates. Between the volume of having all the services set at the info level and the desire to keep it more than 7 days, Datadog was >30k a month. Which at least finally got me permission to set up an open source log manager and switch everyone over to that. Once the initial panic over the Datadog bill died down.
- dmlittle 6y agoI haven't used DD logs personally but from what I've been told if you send the majority of your logs straight to archive to an S3 bucket and only rehydrate the logs when you need to access them their pricing is pretty reasonable. Because you're storing the logs in your S3 bucket you can keep them however long you want and rehydrate them when you need them (say 30 days later). The bill will still be in the low thousands but paying 30-40k per year for a logging solution is probably worth the cost if it means you don't have to maintain it. Having logs and metrics (and the APM if you're using it) all in one place is _really_ nice.
- cinquemb 6y agoYou have to survive as company long enough where paying 30k/month sounds reasonable for logging… esp if your revenue is nowhere near it, and are not VC funded… and you will always have to maintain your integrations at the very least.
- dmlittle 6y agoIf you're using log rehydration with DD it's hard to hit the $30k/mo fee. Before I left my last company we were evaluating DD logging and it came out to ~$5-6k/mo for about 25B logs per month. When you start you'll likely pay a few hundred dollars per month tops. Your personal time alone is probably worth more than that.
- mch82 6y agoSo true! If the tool can’t be used by everyone on your team, then it is unlikely to be used. I’ve experienced this many times.
- novok 6y agoYou can still run into that budgeting conversation when you build, but this time it's with an internal team.
- pembrook 6y ago> I don't have a solution to this. You basically just articulated the solution. Spend the money to give more of your team access to the analytics tool, using the argument made above. It will still be infinitely cheaper (by like 100X) than building anything. And the more you spend with a vendor, the more they’ll be willing to completely customize their product for you.
- simonw 6y agoThat's an easy answer if you control the budget. If you're an engineer in a larger software organization, you probably don't have that spending authority. Which means you have to spend time convincing the people with that authority to spend that money. That's enough of a friction point that many engineers won't bother - which is why I talk about the invisible cultural damage that this problem causes. You and I know that it will be cheaper. The challenge is getting organizational buy-in.
- pembrook 6y agoCertainly a problem, but the time you’d need to spend convincing higher ups is still trivial compared to the time you’d need to spend to build, maintain, and update the tool.
- carlmr 6y ago>the time you’d need to spend convincing higher ups is still trivial From this sentence alone I'm not sure how much time you spent working for a large company. It's not just that you have to convince your manager, depending on how much is to be spent they have to convince theirs. Often creating very long decision paths. And it's not only the time you spent but sometimes also the delay incurred that messes with this approach. Often enough even with buy in you get the answer that it can be planned for next year's budget. One year delay later, you might get the approval, but it may be too late. And if the company is large enough often they've already built a subpar internal solution that everyone is convinced you should use. Since it's "already working".
- 6y ago
- acdha 6y agoI’ve seen this repeatedly over my career. At the large organizations I’ve worked at (.edu, .gov), unless you have a large general tools budget it’s surprisingly hard to beat the open-source tools - simply running a procurement takes months and might require arguments with non-technical staff who don’t understand why certain options are not truly equivalent even if the salesperson swears they are. This is how you end up spending 7+ figures on enterprise tools while the technical staff are all using Prometheus, Grafana, and ELK because they actually work and are available to the entire group. On the “clearly crazy to invest in building your own” front I think it’s useful to add the nuance about how much of that time is purely related to the product and how much is the tuning, integration, understanding the data, etc. which every product requires. I think we’re prone to underestimate the latter and have something like Splunk save less time and cost more than expected because the commodity part it optimized for wasn’t as much of the whole as anticipated.
- simonw 6y agoI posted an expanded version of this comment on my blog, with a suggestion that maybe there are some internal process improvements that could help address the problem: https://simonwillison.net/2020/Dec/13/build-vs-buy/ https://simonwillison.net/2020/Dec/13/build-vs-buy/ "Thinking about this more, I realize that this isn’t a technology problem: it’s a process and culture problem. So there should be a process and cultural solution. One thing that might work would be to explicitly consider this issue in the vendor selection conversations, then document it once the new tool has been implemented. A company-wide document listing these tools, with clear guidance as to when it’s appropriate to increase capacity/spend and a documented owner (individual or team) plus contact details could really help overcome the standard engineer’s resistance to having conversations about budget."
- Viliam1234 6y agoOnce we moved a database to cloud, because we were told it would be cheaper, since we would only pay for the space we actually used. It also meant that whenever the data size increased, however slightly, the invoice also increased, and managers started investigating whether this increase of data size was truly necessary. Sometimes even when then data size didn't change, managers kept asking whether it could be further decreased. Making the billing proportional to X means that X will become a political topic. Choose your X carefully!
- simonw 6y ago"Making the billing proportional to X means that X will become a political topic. Choose your X carefully!" That's such a great way to put it.
- giovannibonetti 6y ago> This can have a subtle effect on your culture. Engineers who don't want to get into a budgeting conversation will end up avoiding using key tools, and this can cost you a lot of money in terms of invisible lost productivity. For exactly this reason I wanted to build a developer screening tool from scratch [1]. In all the companies I worked for, I never got the budget to pay for one of the available tools in the market (most have "contact us pricing"). The result is that most teams I've met spend a lot of time manually evaluating coding challenges. This is a waste of time both for the company and the candidate. A custom screening tool allows the company to calibrate it according to the perceived average candidate level, and avoid those algorithm questions that are useless for most small to medium companies. In my previous company it was used in a few hiring processes and it worked fine - both for back-end and front-end development. It is not open source yet because your need some basic Docker and Rails skills to use it - even though the screening test can be in any language(s) you want (that runs on Docker). If it matches your experience and you want to try it out, please contact me. [1] https://github.com/skilltest/skilltest https://github.com/skilltest/skilltest