9 ms·
When costs are nonlinear, keep it small
- abeppu 6y agoThe author uses "nonlinear" but means "superlinear", because I guess sublinear cost behaviors don't cause the same kind of dramatic problems, and so we can get away with not thinking about them most of the time. I think it says something about the ways that software systems give rise to a particular flavor of problems, that allows us to (relatively effectively) ignore sublinear cost behavior, whereas in other areas those would be really important dynamics.
- mediaman 6y agoWhen would sublinear costs exist? Sublinear costs seem like something that you'd be best off delaying to fix, maybe forever, since their impact lessens the longer you wait. Maybe this would apply when a major change will happen that would obviate the need for a fix: a planned building teardown causes a needed roof repair on the old building to have sublinear costs, or introducing a new subsystem that eliminates the old subsystem that had the outstanding repair. Or aesthetics: a minor marring of a building facade matters when it's pristine, but if you wait longer, the more other minor marrings appear, the less that first mar individually matters to the value of the building.
- Skunkleton 6y agoBatching is an example of a sublinear cost. The more loads of laundry that I do at one time, the lower the total cost to do laundry. Unfortunately in this case, the smell in my closet goes superlinear, so it is rarely worth it ;)
- Geee 6y agoBuying more groceries at a time is another obvious batching that one can do. So is cooking in bigger batches. Fixed cost in household chores is mostly about time, so you should try to batch them to save time. Buying groceries monthly vs. daily saves you 30 hours in a month, if it takes an hour for one trip.
- ouid 6y agofixed costs are sublinear.
- jbay808 6y agoExactly. If you need to make changes to the design of an injection molded part, you'd better make them all at once, because the cost of a new mold is $100k whether you make one change or seven.
- Retric 6y ago24/7 365 support is a less obvious sub linear cost. At 4am one person can be vast overkill, but you can’t get away with zero people. The same is true as you keep scaling support, having level 1 vs level 2 is really about filtering problems ever more efficiently and having enough work where that’s actually needed at 4AM on Christmas takes scale. Which is what economies of scale mean, just a huge range of sub linear costs all lumped together.
- rileymat2 6y agoI think that is just discontinuous, not sub linear.
- Retric 6y agoI think you’re misunderstanding the idea. Let’s use a simple model for N customers you need Pi people rounded up. As you increase by 10x the average cost per N customers keeps decreasing. So, f(N) = 4, f(10N) = 3.2, f(100N)= 3.15, f(1,000N)= 3.142 ... Except in the real world this is being optimized based on a forecast at every point in the day and every day of the year. You might not think of demand in those terms, but it’s a huge area of optimization at both large companies like Walmart all the way down to individual restaurants.
- faeyanpiraat 6y agoYou could also create combined call centers which service two industries which are similar enough, that the same support rep can serve them both, but have an opposite demand curve troughout the year. To see these opportunities you need data, but it is the most closely guarded secret of companies. In the short term it gives them an advantage, but in the long run society as a whole could benefit tremendously from open data sharing. Am I wrong about this?
- abeppu 6y agoYou don't always have the choice to delay. Sometimes the sublinear cost is associated with something you really need to do. Examples of sublinear costs: - When you pay lower per-unit price by ordering large lots. But you can't just wait a long time and be able to afford a giant order; your first small orders may help you generate revenue that let you buy the later big orders. - When your team is young perhaps you need 1 new laptop per new hire. A few years later, you need < 1 new laptop per new hire b/c some receive recycled laptops from departed employees. But you can't wait 3 years and then hire 100 people and 80 laptops. - You're growing some infrastructure which serves/covers some territory. At first, all new customers are in new territory; the ratio of new infrastructure to new customers is high. Later, some portion of new customers are covered by existing infrastructure, and that portion grows over time. But even if you could afford to build everything at once, you may not know where to build most of it until you have a bunch of customers. - I don't really know if this one is true, but a whole industry seems to believe that a burst of advertising effort all at once is more effective than a marginally greater volume of advertising spread over a longer period. But you can't do zero advertising for 5 years and then take over time square.
- im3w1l 6y agoIf you have a group of N potential customers that seek your services randomly and independently, then the standard deviation of customers is sqrt(N). This means you need less and less % extra capacity for handling randomly busy days the larger N is.
- johngalt 6y agoThe problem isn't that people make the wrong decisions about batch sizes or failure tolerance, but that they don't make a decision at all. The default state is a blind push towards both more changes and less pain. Not acknowledging that a tradeoff exists.
- faeyanpiraat 6y agoCould you expand on this a bit? How would I blindly push towards less pain in this case?
- seanwilson 6y ago> It’s like DevOps and CI/CD: more frequent deploys are safer. This happens because each deploy contains fewer changes. Is this generally true? Deploy as in going to production? I join a web app project that didn't have any tests and the code was brittle, where changing one part of the app could break a completely different part without you realising. It was much more efficient (and less stressful) to batch up lots of small changes and do one thorough manual testing session of the whole thing while we gradually got the codebase under control. It likely depends on how much automated testing you have, how bad it is if bugs go live and how quickly you need features live.
- jagged-chisel 6y agoIt’s true that deploying frequently reduce the surface of potential new problems per release. But to make it work flawlessly (not perfectly, but approaching) you do need some engineering rigor in the product.
- lostcolony 6y agoIt's generally true, yes. While manual testing changes the equation some (the testing effort is the same for each release; the more you can automate that, the greater gains), how easy to -fix- the issue becomes more complicated the more changes there are. 1 PR, you know what introduced the breakage. 30 far reaching ones, with interesting overlap between them? Good luck.
- seanwilson 6y ago> 1 PR, you know what introduced the breakage. 30? Good luck. To be clearer, I mean there would be say 30 pull requests merged into "develop" and they would only be merged into "master" and pushed live after detailed manual testing. This is versus merging each pull request directly into "master" and going live immediately, where you wouldn't have time to do detailed manual testing every merge. You would still have the git history in both approaches to debug.
- lostcolony 6y ago
- earthboundkid 6y agoI thought this would be about pricing. In pricing, there’s a real advantage to being small: prices are often artificially low at low volume to promote lock in. Then they gouge you in the middle. Then at the high end, they have to stop gouging you because you know that they know that you know you could go elsewhere for your volume. For example, AWS free tier is a great bargain. AWS is a terrible value once you’re paying the price of a bare metal box each year (or sooner!), but then at Netflix scale it balances out again because Amazon knows that Netflix could easily talk Microsoft into doing the port themselves if it would actually make financial sense. Most pricing is S-shaped, so it typically is best to be in the position of exploiting a loss leader or running at scale.
- minot 6y agoI still don’t understand how this business model works at all. Take cloudinary, for example. You can have your images transformed and hosted by cloudinary. So like you have a photo and let cloudinary examine the photo and serve different sizes, cropping to a human face as a circle for profile pictures for example. It is expensive. All you need is some idiot hitting refresh in a few tabs on the browser for a few hours on your website for you to run over your free tier. I’d imagine that anyone spending real money is always thinking of ways to get off of something like this?
- shash7 6y agoIt doesn't work like that. Cloudinary caches the transform so you only pay for the first time for the transform. Of course, you do have to pay for the bandwidth but by then the image would be cached somewhere on the network or on the browser anyways.
- josephg 6y agoI did some contracting work a few years ago, and one of our clients was a small entertainment news site. They used cloudinary for image hosting and were paying some insane amount for image hosting. I think their cloudinary bill was $8000/month or something. The client didn't even know that was expensive. They just saw it as a cost of doing business, alongside their cloudflare bill and so on. (And what they were paying us.) It took work over several weeks to convince them that that was a ripoff. I think their business people eventually called up cloudinary and negotiated a better rate, but by that point I was annoyed by the whole situation. The site used cloudinary's HTTP based API from the browser. I configured cloudflare to redirect any request hitting example.com/images/... to act as a caching proxy to cloudinary's actual servers. Unsurprisingly, that one configuration change made their cloudinary bill drop to about 1/20th what it was. When the client was billed we got a panicked phone call asking why it was so low, and if something was broken on their site. Anyway, tldr; lots of folks out there have no idea what a service like cloudinary should cost. Apparently more than enough to make cloudinary a profitable business.
- briantakita 6y agoI worked on a project that used Angular. Every major version of Angular holds back the version of it's dependencies, such as Typescript. When I came onboard this project, I came into a situation where the client simply did not upgrade their libraries & nodejs version for a few years. When it came time to catch up with the latest version of Angular, the experience was painful. Three major version updates later, (version 9.x to 10.x), the Angular upgrade became unbearable & I moved the project over to Svelte & a pnpm monorepo. Now all dependencies are up to date, the architecture is improved, & page size reduced. It took a few months to hammer out all of the edge cases due to the complexity of the app but release is imminent.
- oblio 6y agoThese "success stories" are so funny. "The old way was all over the place so instead of improving the existing thing, I completely rewrote everything from scratch and now everything is cool. Yay me/us!". Why couldn't you just upgrade the dependencies once then set up the same CI/CD you're presumably using for Svelte so that you can them upgrade versions easily?
- heavenlyblue 6y agoThey always forget about all of the bugs they have not found in their new app.
- gonzo41 6y agoI had a water pipe crack in the cement foundation of my house. Just and old pipe. The plumber drained and capped the pipe then ran a new pipe around my house in the ground. Next time it breaks it can be fixed. Rewrites aren't bad and usually quicker if you keep a lid on features. All your realy doing is writing down the business rules again.
- oblio 6y agoIs there an app, except for "Hello world", that's as simple as a pipe?
- bloomingeek 6y agoI've worked in facility maintenance since 1985. If maintenance isn't completed in a timely fashion the cost goes up, but also the risk of unexpected failure. In my experience, maintenance isn't authorized because of two main reasons:ignoring the problem or management being afraid to spend the money for the jobs. The money is in the budget, they just don't want to be seen as a "spender" to upper management.
- badhabit 6y agocron to the rescue
- m12k 6y agoIncidentally, that building maintenance graph showing why an ounce of prevention is worth a pound of cure is also useful for illustrating why poverty is such a drag on the economy - if you barely have the money to survive short term, then you don't have the money to invest into even slightly long term things like maintenance, health or education, which ends up costing you even more in the long run. It's an incredible waste of resources (and that's not even counting the human suffering) and it benefits nobody (except maybe the company granting you the emergency loan you use to repair your car when it finally breaks down, but you might never finish paying off).
- throwaway0a5e 6y agoEveryone repeats this trope yet when it comes to the lowest of the low margin businesses the only ones that seem to have any staying power are the ones that operate by putting things off until they can't and then going for the minimum viable fix.
- rutthenut 6y agoIf you invest money in developing (and testing) a new feature, then you don't get the benefit of that investment until it is actually deployed and made available for use. Until then, it is just burnt money with no return. Batching up many changes and features over longer and longer release periods brings so many issues, and definitely greater risks when actually deployed as the scale of change is larger. What the article perhaps doesn't pick up on so well is that poor or difficult deployment processes lead to this behaviour of batching and putting-off the deployments. Do them less often, have less periods of service impact. Is the logic. Which of course is one of the key points about DevOps, feedback loops, and CI/CD processes to simplify things. If you can make deployment easier, you can do them more often. Making the changes smaller can help ease deployments, so can be a virtuous circle. Though not if deployments are not reliable or repeatable.
- deleted 6y ago[deleted]
- 11thEarlOfMar 6y agoThere needs to be another dimension in this model. In building maintenance, there needs to be an inspection to ensure the maintenance was performed properly. For example, you may replace a cracked roof shingle as a do-it-yourself preventive maintenance activity, but not do it correctly and actually cause a leak that was not there when it was only cracked. In software, you need to have code reviews, unit and regression testing, or you can fix one minor bug only to introduce a catastrophic fail.