5 ms·

I don't think something like this should be hosted/SAAS. I don't want to send my Gigabytes of logfiles (sometimes containing confidential informations ...) over

by fotcorn 13y ago

I don't think something like this should be hosted/SAAS. I don't want to send my Gigabytes of logfiles (sometimes containing confidential informations ...) over the internet to some unknown entity with probably questionable security. My small startup company already has more than 10 (virtual) servers which would cost us 500 dollars a month to monitor which is more than the servers itself cost.

gk1 13y ago
Completely valid points. Here's how we're dealing with each: 1. With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. 2. We're exploring a different pricing structure right now that would address this scenario. If this is your only hesitation, I hope you check us out anyway and contact us about pricing.
- j05h 13y ago
  Custom parser won't guarantee that all of the data is dumped...and some of it you may even want in the system. Also -- lots of companies are seriously concerned about pushing their data externally. Making it a hard sell. However, as a long tail service, this looks great.
- blueskin_ 13y ago
  Does the parser run on the client? If not, it defeats its own purpose.
  - snewman 13y ago
    Yes, on the client. See "Redaction" at scalyr.com/agent.
- nknighthb 13y ago
  > With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. This is both error prone and utterly defeats the purpose. Why would I pay a bunch of money for somebody else to manage my logs when I'd just have to keep them all anyway so I can get at the unredacted versions when there's a problem?
snewman 13y ago
[Scalyr founder here] We take security very seriously, but let me turn this around into a question: what would it take for you to trust an external service to manage your logs? Some of the things we're doing: 1. SSL everywhere (including internal traffic between our backend servers). 2. We add a tag to the raw representation of every string value, so that we can verify that data never leaks across accounts. (This has never detected a problem -- except in tests, because yes, we do test it.) 3. Implementing in a "safe" language (Java), to rule out low-level buffer management bugs. 4. As Greg noted, we make it trivial for you to redact sensitive data before it leaves your server. We are sometimes asked for an on-premises installable version of our service. We don't provide that because we're using economies of scale on the backend to completely change the log management experience: when you give us a query, every CPU and spindle in our entire cluster is briefly devoted to that query. This means that you aren't limited to graphing predefined metrics; you can do ad-hoc exploration of your entire log corpus in on the fly. E.g. display a histogram of response latencies for all requests for url XXX on server group YYY in the last 48 hours, and expect a near-instantaneous response.
- mrweasel 13y ago
  >what would it take for you to trust an external service to manage your logs? I think that's a really weird question that completely fails to address the concerns that some people might have. We do logging of sales, profit margins and stuff like that. You can't have access to that because: "You're not us". If you can read our data, then we're not going to use your service and to do anything useful the logs you really do need read access. Of cause you might have no reason to spy on our data, but the only safety is that you promise not to. We could seperate logs for different things, so webserver logs go to you, but email logs goes to an internal system, but then we would need two systems.
  - wdewind 13y ago
    Do you ever email about this data or put it into spreadsheets on Google's servers?
    - 13y ago
blueskin_ 13y ago
Agreed. Contracting out your monitoring seems to me to defeat the point of running your own infrastructure.

by fotcorn 13y ago

gk1 13y ago

Completely valid points. Here's how we're dealing with each: 1. With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. 2. We're exploring a different pricing structure right now that would address this scenario. If this is your only hesitation, I hope you check us out anyway and contact us about pricing.

j05h 13y ago
Custom parser won't guarantee that all of the data is dumped...and some of it you may even want in the system. Also -- lots of companies are seriously concerned about pushing their data externally. Making it a hard sell. However, as a long tail service, this looks great.
blueskin_ 13y ago
Does the parser run on the client? If not, it defeats its own purpose.
- snewman 13y ago
  Yes, on the client. See "Redaction" at scalyr.com/agent.
nknighthb 13y ago
> With our custom parser you can replace or delete confidential information from your logs before they're stored on our servers. This is both error prone and utterly defeats the purpose. Why would I pay a bunch of money for somebody else to manage my logs when I'd just have to keep them all anyway so I can get at the unredacted versions when there's a problem?

snewman 13y ago

[Scalyr founder here] We take security very seriously, but let me turn this around into a question: what would it take for you to trust an external service to manage your logs? Some of the things we're doing: 1. SSL everywhere (including internal traffic between our backend servers). 2. We add a tag to the raw representation of every string value, so that we can verify that data never leaks across accounts. (This has never detected a problem -- except in tests, because yes, we do test it.) 3. Implementing in a "safe" language (Java), to rule out low-level buffer management bugs. 4. As Greg noted, we make it trivial for you to redact sensitive data before it leaves your server. We are sometimes asked for an on-premises installable version of our service. We don't provide that because we're using economies of scale on the backend to completely change the log management experience: when you give us a query, every CPU and spindle in our entire cluster is briefly devoted to that query. This means that you aren't limited to graphing predefined metrics; you can do ad-hoc exploration of your entire log corpus in on the fly. E.g. display a histogram of response latencies for all requests for url XXX on server group YYY in the last 48 hours, and expect a near-instantaneous response.

mrweasel 13y ago
>what would it take for you to trust an external service to manage your logs? I think that's a really weird question that completely fails to address the concerns that some people might have. We do logging of sales, profit margins and stuff like that. You can't have access to that because: "You're not us". If you can read our data, then we're not going to use your service and to do anything useful the logs you really do need read access. Of cause you might have no reason to spy on our data, but the only safety is that you promise not to. We could seperate logs for different things, so webserver logs go to you, but email logs goes to an internal system, but then we would need two systems.
- wdewind 13y ago
  Do you ever email about this data or put it into spreadsheets on Google's servers?
  - 13y ago

blueskin_ 13y ago

Agreed. Contracting out your monitoring seems to me to defeat the point of running your own infrastructure.