10 ms·
I hated Splunk so much that I spent a couple days a few months ago writing a single 1200 line python script that does absolutely everything I need in terms of a
by eigenvalue 3y ago
I hated Splunk so much that I spent a couple days a few months ago writing a single 1200 line python script that does absolutely everything I need in terms of automatic log collection, ingestion, and analysis from a fleet of cloud instances. It pulls in all the log lines, enriches them with useful metadata like the IP address of the instance, the machine name, the log source, the datetime, etc. and stores it all in SQlite, which it then exposes to a very convenient web interface using Datasette.
I put it in a cronjob and it's infinitely better (at least for my purposes) than Splunk, which is just a total nightmare to use, and can be customized super easily and quickly. My coworkers all prefer it to Splunk as well. And oh yeah, it's totally free instead of costing my company thousands of dollars a year! If I owned CSCO stock I would sell it-- this deal shows incredibly bad judgment.
- bluedays 3y agoSounds like a startup
- tw04 3y agoFor how many data sources? The whole reason everyone goes to Splunk is that it scales, and scales incredibly well. Large enterprises can generate hundreds of terabytes to petabytes every day. Splunk has all sorts of issues, but to pretend as if you can replace them in any large shop with a 1200 line python script and SQLite is just being disingenuous. This acquisition falls right into Cisco's sweet spot, they aren't chasing shops that can dump all their security and infrastructure logging into a SQLite database and not have it tip over in an hour.
- eigenvalue 3y agoIt's around 6 data sources on ~25 machines, but it could be easily scaled to way more than that with a bit of work. And I mean less work than it takes to do even trivially simple things using the horrible Splunk API. There are many thousands of small companies using Splunk and getting totally ripped off for a very mediocre product with a rapacious and annoyingly aggressive salesforce.
- coalbin 3y agoThat is a tiny setup all things considered. You aren’t operating at a scale you’d need to consider a monitoring platform for.
- steveBK123 3y agoYou'd be surprised how many companies with infra that small have CTOs get consultant buzzword pilled into buying every SaaS under the sun nonetheless...
- mlhpdx 3y agoHow many servers does Stack overflow run on? It’s not a good measure of data volume or criticality. I think “expensive” here is basically relative to revenue/margin. Where margins are high, spending on Splunk (etc.) isn’t meaningful. Where margins are thin, it hurts. Basically, the arguments here seem to reflect the markets and business model folks are working under. Some pay, some can’t and some won’t - all valid.
- ilyt 3y agoBut you definitely want to, even if it simple ELK stack
- tw04 3y agoI think we're talking about very different levels of scale. Enterprises are generally feeding tens to hundreds of thousands of datapoints into Splunk depending on their size between servers, networking gear, endpoint devices, etc.
- ta1243 3y agoI have an order of magnitude more machines than you and would never in a million years consider splunk Right tool for the right job. Splunk is for mega-scale setups
- ignoramous 3y ago> it could be easily scaled to way more than that with a bit of work. I guess you'd appreciate the words easily and bit are doing a lot of heavy lifting there.
- baz00 3y agoSplunk does not scale to large data sources. It fucks out at a few TB and then you have to spend hours on the phone trying to work out which combination of licenses and sales reps you need to get going again. By which time you can just suck the damn log file and grep it on the box.
- teach 3y agoI'm gonna respectfully disagree that it fails "at a few TB". We send them 100s of terabytes a day.
- anonzzzies 3y agoBut, and this is not meant as criticism or insult as I have no idea how Splunk works, it is just based on other comments; do you know what license your company has with them? It appears that if you are paying them millions, it scales fine, otherwise, it does not?
- tekla 3y ago> I have no idea how Splunk works Cool > It appears that if you are paying them millions, it scales fine yes, if you pay someone for product and services, you get them. If you don't, you don't
- anonzzzies 3y agoYeah, because that is what I meant. A lot of services are useable without paying through the nose, this one apparently not, but thanks for the excellent input.
- baz00 3y agoIt's difficult to control data ingress so you end up in debt and on repayment plans. Which are expensive.
- anonzzzies 3y ago
- jbergens 3y agoI remember a client using Graylog. It was good for app logging and is available as open source.
- mongol 3y agoIt sounds like the difference between a car and a freight train.
- leoc 3y agoMy complaint is that this acquisition is going to add another 1-4 paragraphs of examinable marketing copy to the Cisco CCNP ENCOR textbook. I'll have to somehow remember not to confuse Splunk with Cisco Firepower NGIPS, which uses Snort. This is what happens when an industry starts to name its products after the sound effects from Peppa Pig.
- tekla 3y agoThis mostly sounds like a badly managed Splunk. If a 1200 line Python script is all you need to replace a Splunk instance, you weren't doing anything all that interesting or well in the first place. > useful metadata like the IP address of the instance, the machine name, the log source, the datetime, This should be tagged on every single log line already, and not something that you should be doing post-ingestion
- eigenvalue 3y agoThe logs included things like the systemd logs and stuff that I don’t have control over. You need to be able to enrich with arbitrary metadata for it to be generally useful. My point is more that a large portion of Splunk customers could do the same thing I did and be way better off. Obviously not their huge enterprise customers spending millions a year.
- runjake 3y agoWhy wouldn't you just use Graylog Free Edition? While it doesn't compete with Splunk, IMHO, it's much easier and much better than what 1,200 lines of Python could conjure up. Dashboarding and all. I love it and use it in a very large enterprise environment.
- asynchronous 3y agoThat “thousands of dollars per year” number seems quite a bit low for a Splunk license. Even for a small amount of data it’s more like thousands per month.
- spoonjim 3y agoI’m sure the Cisco CEO is quaking in his boots thinking about this cronjob
- geodel 3y agoWell today you are doing 100KB log processing, who knows, tomorrow you may end up doing 500KB log processing. It will be All Hands On on late night Friday to eliminate this existential threat.
- TheRealDunkirk 3y agoIt sounds like you reinvented the concept of a loghost with a database.
- moneywoes 3y agohave you released this anywhere
- eigenvalue 3y agoYes, just now: https://news.ycombinator.com/item?id=37600019 https://news.ycombinator.com/item?id=37600019
- ShrigmaMale 3y agolook at vector.dev and clickhouse. fast, has a language for extension, v easy to set up.
- evantbyrne 3y agoI used Vector in the Beaker Studio prototype back when it was designed to deploy directly to Ubuntu virtual machines. That was a couple years ago at this point, and it worked wonderfully!
- shandor 3y agoSounds awesome for your use case! …but this sounds so much like the legendary Dropbox release thread’s ”just use FTP, SVN, etc” that it made me smile :)
- Scarbutt 3y agoWell no, dropbox is aimed at non-technical oriented users. Sure, they have "enterprise" features for admins now but that's not how it started and in the end the product is vastly consumed by non technical users.
- eigenvalue 3y agoI hear you, but the difference is that Dropbox is actually good and reasonably priced. Splunk is horrible to use and costs 1,000x what it should, and they are super aggressive about harassing you about usage caps and threatening you constantly with huge price hikes. Dropbox has barely raised price over the years (until pretty recently at least) and has been rock solid and amazing.
- nemo44x 3y agoThere's quite a few log ingestion programs that can do all that for you. Did you have some type of specialized log that one of the various logging tools couldn't handle for some reason? It sounds like you recreated the ELK stack lol.
- magixx 3y agoIt's weird seeing no mention of Graylog anywhere here which is slightly different but I've found much easier to use in smaller setups. Unfortunately I have no idea what enterprise cost ends up looking like.
- eigenvalue 3y agoSince someone asked, I cleaned up my script and released it: https://news.ycombinator.com/item?id=37600019 https://news.ycombinator.com/item?id=37600019
- anonzzzies 3y agoGreat, finally someone who actually does that. So many examples here with people whining about their Dropbox thingy in 4 lines of Perl but never releasing anything for us to check out. Well done!
- prabhatsharma 3y agoWhy build in this age when too many open source solutions backed by opentelemetry standard are available. Use fluentbit/vector/otel-collector to capture data and send to some open source solution.
- eigenvalue 3y agoBecause I find all that stuff to be even more mental overhead to learn and work with, and super annoying to deploy and manage. It would literally take me longer to get one of those kinds of tools to work on my data the way I want it than it took me to make my own tool that does exactly what I want, exactly the way I want it, where it's incredibly trivial for me to add new kinds of logs or anything else. When you have a hugely complex, made by committee, enterprise-grade generic system/protocol like opentelemetry that does anything and everything, at any scale, it's always going to have huge amount of excess complexity when you are trying to do a specific simple thing well and quickly. It would be harder to figure out the config files for that stuff than it was to just make my own system.
- manicennui 3y agoThis comment is incredibly naive. Cisco isn't making acquisition decisions based on your happiness. Splunk's revenue is increasing every year and their losses decrease. It is an incredibly popular tool that complements their products and services well.
- ilyt 3y agoExpect entering splunk API key in next generation of their OSes for seamless monitoring
- manicennui 3y agoI don't know about their router/switch OSes in particular, but a lot of their products already have Splunk integration and they seem to have a couple of products built on top of Splunk.
- dingdong33 3y agoThis is most stupid comment I’ve ever read from here.
- phyzome 3y agoI used SumoLogic at my last job, which feels basically the same as Splunk. (Maybe not as fast? No idea on price.) There were times when it was easier to sync 45 GB of logs from S3 down to my laptop and run grep over them than it was to figure out the right arcane syntax and wait for the results. :-)
- deleted 3y ago[deleted]