6 ms·
I shudder to think how much Splunk costs at Twitter's scale.
by ceejayoz 5y ago
I shudder to think how much Splunk costs at Twitter's scale.
- CorpOverreach 5y agoI was thinking something along the same lines. The article said 42 TB per datacenter. How many datacenters is Twitter running?
- Daneel_ 5y agoThe article said three datacentres further down, so they're capturing a total of around 126TB/day, which is big but definitely a long way short of the biggest Splunk deploys (multi-PB afaik).
- jeffbee 5y agoThe whole architecture looks insanely expensive. Putting your debug logs through scribe, through Kafka, then into Splunk, and indexing them, has got to cost something like fifty dollars per gigabyte. I've always been a proponent over leaving logs where they were produced, not collecting them, and not indexing them, so an architecture like this I find just shocking.
- closeparen 5y agoCentralized logging is one of the many problems you sign up for once you opt for microservices on a cluster scheduler. Since service instances are ephemeral and cluster nodes only slightly less so, it doesn’t really work to leave the logs in place.
- mypalmike 5y agoWhen you have fleets of thousands of machines performing a given service, and thousands of services implementing your product, you simply must centralize and index logs if you want to have any chance of managing outages.
- jeffbee 5y agoExperience suggests otherwise.
- mypalmike 5y agoInteresting. What company have you worked for that runs hundreds of thousands of servers without centralized logging?
- RhodesianHunter 5y agoNot the person you replied to, but basically airlines, banks and other financial institutions, most things government or DOD are largely run this way.
- avl999 5y agoI worked on a tier 1 service at Amazon with over 1500 hosts in the fleet. Logs within the hour would be on the host and we would literally just grep through them, to grep through logs on a subset of the fleet or the entire fleet there was a simple utility that would ssh into multiple prod host and run grep in parallel. For logs older than an hour the logs would go gzipped into storage and the we zgrep'd though them. No centralized logging (unless you call logs being rotated off the host at end of the hour) and definitely no indexing. Never had any problems managing outages. Grep alone can go a long way in managing outages.
- Aeolun 5y agoIn enterprises of this size, it’s quite possible you get a request to debug something that happened a month ago. Your instance may be gone, so you need to debug going only by whatever was logged.
- dilyevsky 5y agoCan’t do that in cloud (i know twitter doesnt run in cloud) and autoscale - machine goes poof and so do the logs. One of the reasons why autoscaling to save $ is bs.
- jeffbee 5y agoThat’s simplistic. For one thing, at scale not every debug statement is a precious snowflake. Secondly, in an orchestration scheme like k8s there’s no reason why your container lifecycle can’t include a cleanup container that runs and either exports the logs or just sleeps for an hour, so ephemeral data lives a little longer.
- closeparen 5y agoExports your logs to where? I would call that place centralized logging.
- dilyevsky 5y agoI’m assuming neighboring nodes. Now you’re basically designing replicated (best effort), distributed index - a mini version of google search. This is a clever design but number of companies that can diy it with sufficient quality I can probably count on one hand
- jeffbee 5y agoFor the price of even a modest splunk enterprise setup you can hire half a dozen top notch backend/distributed systems engineers. That’s the opportunity cost of a setup like the one in the article.
- closeparen 5y agoI’ve heard this before and it sounds like Splunk’s margins are just way too high. Why doesn’t someone sell a logging platform that’s cheaper than writing your own at scale?
- johndfsgdgdfg 5y agoI am pretty sure Twitter gets a hufe discount. A company like Twitter is on Splunk platform itself a huge advertisement for more future traditional Fortune 500 companies.
- mulcahey 5y agoThis blog post read a little like an ad itself to be honest.
- jamjamjamjamjam 5y agoIt probably gave them a 7 figure discount
- jcims 5y agoWe log >>100TB/day in splunk and get all the discounts. It’s still ridiculously expensive. Many of the issues presented in the article ring very true. Splunk is pretty amazing for adhoc analysis/threat hunting. However, once you know what you’re looking for the value proposition drops precipitously.
- romantomjak 5y agoCan you give a ballpark figure? I can’t even begin to imagine how much that costs.
- jcims 5y agoAll in (software/hardware/people) ~$1-2k per terabyte of data indexed.
- halilduygulu 5y agowhat kind of use case can log 100tb? Do you actually query this data later?
- mdoms 5y agoThis is how you get organisations that pull in $3.7 billion in revenue but still can't turn a profit :)
- saagarjha 5y agoTwitter has been profitable for the last several quarters, FWIW.
- aiisjustanif 5y agoEgress filter before logging to Splunk people, hell if you can practice filtering better from the application itself. No need to say send all you debug logs to security, or all your info logs to a logging instance for site reliability.
- Aeolun 5y agoI’m fairly certain it’s more than $1,000,000/year
- ceejayoz 5y agoI'm fairly certain there's at least another zero.