3 ms·
Just get Splunk. I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year. The thing peop
by nl 14y ago
Just get Splunk.
I'm a pretty experienced Solr developer, and I've played with Elastic Search etc, and I've been using Splunk for about a year.
The thing people miss about Splunk unless they know it is how good the search interface is. For example, the search language roughly comparable to Lucene/Solr/Elastic Search, but also includes the ability to parse input files, and present results graphically. No open source solution integrates all that.
If you want to compete with Splunk (something I've thought about a few times) then you need to match that. I'd estimate 2 developer for a year to build out those features on top of Solr or ES.
- packetslave 14y agoYes, except Splunk gets very expensive, very quickly if you want more than the free tier gives you (features or indexing volume). 500mb/day is not all that much when you start shoving everything under the sun into it (and once you've used it, you'll want everything available to it).
- nl 14y agoWhat do you class as very expensive? We put multiple orders of magnitude more data than the free tier into Splunk, and it's still a lot cheaper than 2 developer-years. It is true, though, that if the licensing was cheaper we'd put even more data into it.
- heretohelp 14y agoDon't listen to this guy if you own more than a couple servers.
- nl 14y agoWhy do you say that? We have a couple of datacenters, so yes, we have more than a couple of servers.
- heretohelp 14y agoIn a situation where one has that much money to blow on something so limited, virtually anything would've sufficed. We did a trivial test of Splunk at my last company, it's extremely expensive and it's very easy to bump into its limitations. We were able to wreck the poor Splunk server with some rather sundry queries into a dataset that shouldn't have been that big of a deal. Issues that we took back to the company and didn't get any real answer on. Its popularity leads me to surmise that there is still a lot of money to be made in solving mundane problems. (Which is good news if you're a product-minded programmer)
- nl 14y agoWhat is extremely expensive for you? We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis. Without knowing details of exactly what you are doing it's difficult to comment on your problems with queries. It's true that something like Solr gives you more control over the indexing process, so you can optimize it more for specific queries. Splunk tends to rely more on saved searches (and the new search acceleration feature).
- heretohelp 14y ago>We find the overheads on storing & processing the data are much more than the cost of the license, on a per GB basis. What are you storing the data with...the etchings on wings of fairies? >Some blather about Splunk's "saved searches" We talked to the company, explored every avenue. Our volume of data simply overwhelmed it. (Data from three Apache servers. Lol.) I am 100% certain you know less than Splunk-The-Company, so our conversation is done here.
- nl 14y agoWhat are you storing the data with.. It's on a SAN. We'll probably migrate to local disks at some point. The pricing is typical SAN pricing[1]. * Our volume of data simply overwhelmed it. (Data from three Apache servers. Lol.)* Yeah, well we do a lot more data than that. [1] Take a look at the NetApp, Dell & EMC prices on http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-how-to-build-cheap-cloud-storage/ http://blog.backblaze.com/2009/09/01/petabytes-on-a-budget-h..., or look at http://serverfault.com/questions/76725/whats-the-nominal-cost-of-enterprise-storage-per-gb http://serverfault.com/questions/76725/whats-the-nominal-cos... and you'll be in the right price range.
- rdl 14y agoSplunk is absurdly priced for normal verbose syslogs for a bunch of hosts. You could preprocess or tune your logging to only send important stuff to Splunk to make up for this. It's cheap for application-specific logs where each line is relatively high value.
- zobzu 14y agoI think it defeats the purpose. Splunk is great but you need to pay for a license. What's missing is a free as in beer and as in freedom solution that is decent. Mostly because it means we can all commit fixes/updates/etc to it. Including people who can't pay for a product (but are willing to pay for support) such as communities.