5 ms·
These perverse incentives run at the heart of almost all Developer Software as a Service tooling. Using someone else's hosted model incentivizes increasing toke
by erulabs 1y ago
These perverse incentives run at the heart of almost all Developer Software as a Service tooling. Using someone else's hosted model incentivizes increasing token usage, but it's nothing special about AI.
Consider Database-as-a-service companies: They're not incentivized to optimize on CPU usage, they charge per cpu. They're not incentivized to improve disk compression, they charge for disk-usage. There are several DB vendors who explicitly disable disk compression and happily charge for storage capacity.
When you run the software yourself, or the model yourself, the incentives aligned: use less power, use less memory, use less disk, etc.
- jiggawatts 1y agoMy favourite example of this is the recent trend towards “wide events” replacing logs and metrics… spearheaded and popularised by companies that charge by the gigabytes ingested.
- tptacek 1y agoCompanies that ingest logs generally rip their customers faces off with their pricing. At least oTel spans can be tail-sampled.
- jiggawatts 1y agoI worked out that it's cheaper to write logs to high-end Samsung SSDs and then throw them away every month than to retain them in the log analytics systems of some cloud services for the same period of time. Wait, no, sorry... that doesn't quite "paint the right picture". The "single use" SSDs are 75 times cheaper than storing the data in the cloud.
- tptacek 1y agoAs someone who works at a platform company that operates several (very) large log ingestion systems, if you're not indexing the logs usefully, having stored them on SSDs isn't doing much for you. It's just a weird comparison to make, is all I'm saying.
- jiggawatts 1y agoIndexing that is 75x the original uncompressed data volume? Because then I might accept the cost! Realistically all of these systems use some type of data compression such as Parquet files, so the data on disk is likely smaller than the ingested data.
- tptacek 1y agoMy point is that you're not really just paying for "storage".
- jiggawatts 1y agoWhat am I paying for? I worked out that the markup on CPU, network bandwidth, and storage for the default logging products from the major clouds is on the order of 25x to 500x. Okay, sure, there's some people that need to be paid, the back-end software may have some licensed components, etc, etc... But still, comparing this to any other cloud service, the gross profit margin is just ridiculous! It's the typical IT marketing trick of selling the commodity (VMs) at competitive prices, and then clawing back the profits via the "enterprise add-ons".
- tptacek 1y agoI mean: we faced the same buy/build issue, and we went "build", and while I wouldn't say it was our worst decision, it was not one of our best either. Certainly the problem doesn't simply boil down to NVME gigabyte costs! That said: we both agree, log ingestion services are extremely expensive.
- tmpz22 1y ago> When you run the software yourself, or the model yourself, the incentives aligned: use less power, use less memory, use less disk, etc. But my team's time is soooo valuable. It's sooo sooo sooo valuable. Oh and we can't afford to hire anyone else either. But our time its sooo valuable. We need these tools!
- alternatex 1y agoOpens PR with quadruple-nested for-loop running synchronous DB queries. - Premature optimization is the root of all evil, can't waste expensive dev hours on that..