6 ms·
You should try Scalyr (https://www.scalyr.com/ https://www.scalyr.com/). It's easier than juggling between six different tools. It was built by ex-Google DevOps
by gk1 13y ago
You should try Scalyr (https://www.scalyr.com/ https://www.scalyr.com/). It's easier than juggling between six different tools. It was built by ex-Google DevOps engineers for the same reason you made this Slideshare: the available tools suck.
(Full disclosure: I'm working with Scalyr, but you should still try it.)
- fotcorn 13y agoI don't think something like this should be hosted/SAAS. I don't want to send my Gigabytes of logfiles (sometimes containing confidential informations ...) over the internet to some unknown entity with probably questionable security. My small startup company already has more than 10 (virtual) servers which would cost us 500 dollars a month to monitor which is more than the servers itself cost.
- gk1 13y agoCompletely valid points. Here's how we're dealing with each: 1. With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. 2. We're exploring a different pricing structure right now that would address this scenario. If this is your only hesitation, I hope you check us out anyway and contact us about pricing.
- j05h 13y agoCustom parser won't guarantee that all of the data is dumped...and some of it you may even want in the system. Also -- lots of companies are seriously concerned about pushing their data externally. Making it a hard sell. However, as a long tail service, this looks great.
- blueskin_ 13y agoDoes the parser run on the client? If not, it defeats its own purpose.
- snewman 13y agoYes, on the client. See "Redaction" at scalyr.com/agent.
- nknighthb 13y ago> With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. This is both error prone and utterly defeats the purpose. Why would I pay a bunch of money for somebody else to manage my logs when I'd just have to keep them all anyway so I can get at the unredacted versions when there's a problem?
- snewman 13y ago[Scalyr founder here] We take security very seriously, but let me turn this around into a question: what would it take for you to trust an external service to manage your logs? Some of the things we're doing: 1. SSL everywhere (including internal traffic between our backend servers). 2. We add a tag to the raw representation of every string value, so that we can verify that data never leaks across accounts. (This has never detected a problem -- except in tests, because yes, we do test it.) 3. Implementing in a "safe" language (Java), to rule out low-level buffer management bugs. 4. As Greg noted, we make it trivial for you to redact sensitive data before it leaves your server. We are sometimes asked for an on-premises installable version of our service. We don't provide that because we're using economies of scale on the backend to completely change the log management experience: when you give us a query, every CPU and spindle in our entire cluster is briefly devoted to that query. This means that you aren't limited to graphing predefined metrics; you can do ad-hoc exploration of your entire log corpus in on the fly. E.g. display a histogram of response latencies for all requests for url XXX on server group YYY in the last 48 hours, and expect a near-instantaneous response.
- mrweasel 13y ago>what would it take for you to trust an external service to manage your logs? I think that's a really weird question that completely fails to address the concerns that some people might have. We do logging of sales, profit margins and stuff like that. You can't have access to that because: "You're not us". If you can read our data, then we're not going to use your service and to do anything useful the logs you really do need read access. Of cause you might have no reason to spy on our data, but the only safety is that you promise not to. We could seperate logs for different things, so webserver logs go to you, but email logs goes to an internal system, but then we would need two systems.
- wdewind 13y agoDo you ever email about this data or put it into spreadsheets on Google's servers?
- 13y ago
- blueskin_ 13y agoAgreed. Contracting out your monitoring seems to me to defeat the point of running your own infrastructure.
- caw 13y agoThere's a distinct lack of images on that website. I for one would like to see how I would diagnose problems without opening 5 tabs. A "demo" mode would be even more helpful.
- snewman 13y agoThanks for the feedback! Demoing a log management product is a bit tricky, because it's hard to come up with a representative data set that isn't sensitive. FWIW, we have put together a demo based on data scraped from the Github API -- it's not a typical server log data set, but it can be fun to explore: https://www.scalyr.com/login?prefillEmail=demo-account%40scalyr.com&prefillPassword=demodemo&originalUrl=https%3A%2F%2Fwww.scalyr.com%2FlogStart https://www.scalyr.com/login?prefillEmail=demo-account%40sca...
- ceejayoz 13y agohttps://www.scalyr.com/dash?page=Github-Statistics https://www.scalyr.com/dash?page=Github-Statistics is broken. "Operation not permitted (Read Configuration permission required)"
- snewman 13y agoOop -- thanks! We'll sort this out, might take a little while though.
- angersock 13y agoYeah, no, not what I want to hear from the people hosting my logfiles. There's your answer to "Why won't you host your server logs (which are usually key for troubleshooting flaming boxen)?"
- snewman 13y agoAs I probably should have clarified, this issue is specific to a single page in the demo, which we do not even link to at the moment (aside from a couple of older posts on our blog). Yes, it is embarrassing, and I apologize. However, if this issue had been standing between a customer and their data, we would have scrambled instantly. Everything in life is a tradeoff. If you entrust your logs to us, you run the risk that we have an outage or failure of some sort. On the other hand, internal systems can fail as well. We hope to serve people who prefer not to carry the responsibility of maintaining their own monitoring infrastructure, and/or are interested in the features and performance we provide.
- samstave 13y agoThe price is absolutely insane and a non-starter.
- snewman 13y agoWe've been hearing that loud and clear. :) FWIW, the price actually works quite well for a lot of people. On a GB-for-GB basis, we're actually much cheaper than other hosted log management solutions -- we work hard on backend efficiency and we pass that along. But yes, if you're using small virtual servers then the pricing model breaks down. We originally went with this model to provide more predictability; log volume is often more volatile than server count. We've heard enough complaints that we've decided to move to more volume-based pricing model, we're just working out the details.
- samstave 13y agoThanks for the reply... as an OPs guy there are a ton of layered problems with running a highy elastic infra on something like AWS: 1. Dynamic registration of ephemeral systems with a monitoring platform. 2. Security monitoring of same 3. Meaningful graphing When we are optimizing the purchase of hundreds upon hundreds of spot instances daily, where we are looking at grabbin hosts for just a couple cents an hour, the model of per host fees for things like StackDriver, CloudPassage and your service makes per-host pricing completely a no-go. I don't have a good idea how these should be priced; but I think its important for people to understand all the other costs associated with having a solid management platform for your environment that covers all the bases and doesn't require another round of funding! :)
- kylek 13y agoMaybe I'm missing something but this just looks like log analysis (akin to Splunk) and not actually server monitoring? (active health checks, notifications, snmp, etc) Pricing seems wonky too...is it cheating the licensing model to aggregate logs on a single syslog server and submit from a single agent?
- gk1 13y agoRegarding the pricing, we're in the process of revising the pricing structure so that it's based more on volume instead of how many servers you have. In that case, you wouldn't need to cheat. :)
- quicksilver03 13y agoThis is interesting for us as well... We currently have "only" 50 servers (mostly virtual) that need to be monitored, a per-server pricing would push the cost far too high regardless of the quality of the solution.
- gk1 13y agoMay I email you about this? The pricing structure will be changed soon to accommodate cases like yours--there are many of them--so if that's the only thing keeping you from trying Scalyr then I'd love to chat.
- quicksilver03 13y agoWe've already started using a competitor's product (Logentries) and we're happy with it, so we're not looking to Scalyr or other log management solutions at the moment. Thanks for listening to feedback though!
- snewman 13y agoLog analysis is the heart of the product, but we also gather system metrics, provide notifications (scalyr.com/helpalerts), and we recently rolled out a basic active-checks feature (scalyr.com/helpMonitors). There's lots more to be done; for instance, we don't have any SNMP support today. But the vision is to be a full-spectrum tool, and we're actively working toward that. As for the licensing model: we're going to move to per-GB pricing anyway, so no worries there. If you'd like something more concrete today, e-mail us at contact@scalyr.com and I'm sure we can sort out any pricing concerns.
- Sanddancer 13y agoSo what happens when my email server goes down? One of the huge advantages nagios has is that there are plugins that'll send SMSes, plugins that'll send me phone calls. Hell, people have written scripts that let them call to get nagios alerts [1]. One of the huge advantages of having something hosted in-house, and with nagios, is that it can be configured to a level of precision I can't see your tool coming close to achieving. Yes, Nagios' configuration is ugly and occasionally requires sacrifices to elder gods. At the same time, I've never found any sort of monitoring/alerting I've needed done that it can't handle. As much as your service looks cool for a specific subset of monitoring, it is still missing half the hooks as to why nagios is stubbornly sticking around. [1] http://www.googlux.com/callnagios.html http://www.googlux.com/callnagios.html