10 ms·

Distributed Logging Architecture in the Container Era

179 points by hkmurakami 10y ago

ivan_ah 10y ago
How does fluentd compare with logstash, the other logging swiss army knife?
- ktamura 10y ago
  There's been some comparisons in the past. http://logz.io/blog/fluentd-logstash/ http://logz.io/blog/fluentd-logstash/ https://www.pandastrike.com/posts/20150807-fluentd-vs-logstash https://www.pandastrike.com/posts/20150807-fluentd-vs-logsta... http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/
- windowsworkstoo 10y ago
  They are all much of a muchness....fwiw distros that ship with rsyslogd as the syslog daemon are in a good spot, since rsyslog can do sufficient routing and spooling to account for network interruption. One less thing to install. And it supports the latest structured syslog RFC. That said, I use nxlog basically everywhere because I run a heterogeneous environment and it works well on all OS's I use (and is fast and light on resources)
im_down_w_otp 10y ago
The first paragraph closes with a series a incorrect assertions contrasting containers with VMs, and "distributed logging architecture in the container era" is apparently nearly identical to the distributed logging architecture of the VM era in reality, but according to the article is a completely different thing and mindset altogether... again while looking basically identical.
- virtualpaul 10y ago
  VMs last much longer than containers, no? I can't remember the last time I spun up a VM that lived for less than a few seconds...
  - im_down_w_otp 10y ago
    Mine live for milliseconds, but I'm a weird case where that's intentional as a security construct. But that doesn't really have anything to do with the logging architecture. How long the thing lives doesn't matter when you're using aggregation, bypassing the local filesystem, and inspecting through a central portal. People have been doing this in VM environments for a lot longer than they've been using containers. Whole companies and services were born to facilitate this, and some have even died already, in the time before containers hit full stride in the hype cycle.
    - scaryclam 10y ago
      Yes, I was left scratching my head as to why containers play any significant role in the issue of distributed logging. If your choice of container runtime/vm/servers are creating an issue with logging then there's most likely an issue with the system architecture rather than actual logging. We've been building distributed systems (with containers and other approaches) for years and this is the first time I've heard of logging being a problem.
kozikow 10y ago
I recently just plugged python logger from https://github.com/GoogleCloudPlatform/gcloud-python#google-stackdriver-logging https://github.com/GoogleCloudPlatform/gcloud-python#google-... into my python codebase. Configuration took 15 minutes, I have nice dashboard showing recent logs, I can log either strings or jsons. I can analyze data later using BigQuery. It works from docker containers. I was contemplating trying out logstash or fluentd, but I don't see any major advantage over current simple solution. Can anyone more knowledgable than me help with explaining what I would get from logstash/fluentd?
- pfranz 10y ago
  Does any of your Python processes open a subprocess that then spawns another Python process as a child (and both gets logged)? If so, have you found any good way to associate the the parent and child logging messages? If you just write to stdout everything makes sense in the file top to bottom, but if you're logging json objects into a database I had problems; if your parent process echos stdout from the child you lose a lot of context about the child, you can also double-log child process messages (because the parent is also echoing them), and you can't easily associate the parent/child objects. I came up with a hacky solution, but I wasn't logging these events to a centralized server at the time. (I haven't used gcloud's dashboard, so I might be making wrong assumptions) To respond your question: it sounds like you have something that works fine for logging messages. Personally, I split it into categories 1) logs (serial events that needed context for any usefulness) and 2) events (metrics or exceptions). I wrote 1) to traditional log files and wrote 2) to an ElasticSearch or statsd database to log exceptions to. Ideally, I wanted to use the same mechanism for both and peel off relevant data into separate databases. A metrics database like Elasticsearch will let you query things like, "What modules give the most errors?" "Has this function been called more often this week than last?" "Is this process taking longer when using the newly released version compared to the old one?" etc.
  - derefr 10y ago
    Use the process group ID? (http://unix.stackexchange.com/a/139230 http://unix.stackexchange.com/a/139230)
rdli 10y ago
I'm surprised the article didn't mention anything about the nuances of actual logging in a distributed architecture. In particular, when a request starts to span multiple containers or microservices, you need a way to correlate all log data to a given request. You typically do this by propagating a trace ID through your logging system, and a vector clock-type thing for causal ordering. If you can't tell which log statements belong to which request and the order in which they occur, debugging is quite hard :-).
- ktamura 10y ago
  While the article does not touch on this explicitly, Docker's Log Driver for Fluentd supports tags, which can be used to pass on container name/ID and other metadata to wherever logs are sent. Since Docker Machine itself allows these metadata to be passed on via API ({{.Name}}, etc.) one can create a fairly robust system to track log lineage.
  - meddlepal 10y ago
    That approach requires a lot of post processing to make it useful. Further it's pretty hard (near impossible) to do causal ordering.
    - ktamura 10y ago
      I agree on the near-impossibility of causal ordering at scale. I don't see the need for a lot of "post processing". Adding filters in Fluentd is fairly simple: http://docs.fluentd.org/articles/filter-plugin-overview http://docs.fluentd.org/articles/filter-plugin-overview
  - rdli 10y ago
    How would passing on the container ID help in mapping multiple log messages to the same request? If a request is going through multiple containers, you'll get multiple container IDs. To solve this problem, you need a unique trace ID and vector clock, and this actually needs to propagated through your logging system, not through container tags.
voiper1 10y ago
This is going to sound dumb, but I started dumping all my logs into mysql so that I wouldn't also need to run an elastic search cluster. I then query by the error level, latency, status code to find items of interest. And everything is stored by some sort of request ID so that I can correlate multi-entry logs. It's not pretty -- there's no real-time alerts, but I'm not paying dollars per gb to AWS and I have no clue what google will end up charging... and I'm already running mysql. So there is that...
- packetized 10y ago
  I'm guessing your log write volume is relatively low.
- deleted 10y ago
  [deleted]
- bni 10y ago
  Congrats for choosing the solution with the least moving parts and rejecting premature optimization and over-engineering.
- brightball 10y ago
  It's a good solution depending on write volume. Watching the pricing on these logging services is always fairly shocking to me. I've done a similar thing with Mongo in the past with their capped collections.
bluecmd 10y ago
They say that without aggregation scale-out is impossible. That's simply not true. Using things like anycast and ECMP it's super easy as log destinations are usually stateless. Also, never had network contention because I logged too much unless something was really broken - and then only at that node. I'll continue sending events to my anycast address without any aggregation thank you very much.
- sagichmal 10y ago
  > They say that without aggregation scale-out is > impossible. That's simply not true. Using things like > anycast and ECMP it's super easy . . . Then you haven't worked at significant scale. That's fine! Just keep everything in context :)
  - bluecmd 10y ago
    A "probably" there would be nice. I don't want to have to defend myself, but I'm pretty confident about the scale I've worked on. I guess it boils down to what you call aggregation. Is having multiple stateless receivers behind anycast/ECMP aggregation writing to a distributed database aggregation? I'd argue not, but maybe this is where the difference in opinion lies.
r4um 10y ago
Inbuilt docker + fluentd driver + plain ES falls flat on its face under scale. Not to mention docker daemon keeps dying :(. Whats working for us now is local docker json logs -> heka -> kafka -> graylog -> ES. We even dockerized graylog to scale the processing up and down on demand. Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s
- edu115 10y ago
  Too bad that Heka has been officially deprecated, sounds like a similar use case to what we have.
- vardump 10y ago
  > Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s 250k messages/sec is pretty low. It'd be useful (not to mention entirely feasible) to be able to handle 5M/sec bursts on a single system and aggregate sustained 50M/s+ (10Gbps network). The system should be able to establish total order (serialization) locally and causal order over the network. I did some tests and found out hardware limits are somewhere between 20-50M messages per second (serialized) on current consumer grade X86. Practical implementation would of course be slower. Of course you need to go binary logging at that point, adhere to cache line boundaries, etc. mechanical sympathy. Maybe even do usermode networking at the aggregator server. Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. In my quick tests I found the string processing hit to be surprisingly high, 10-50x. C "sprintf" and C++ stringstream are atrociously slow. (Surprisingly, considering sprintf has a pretty complicated "bytecode" format specifier parsing loop, it was still significantly faster than stringstream implementation.) Timestamps are another huge performance issue. Typical system calls for high precision (a few microseconds or better) timestamp take several microseconds each - using one of those will alone drop performance to 100-500k range. It's of course much faster to use RDTSC, but then you have the issue that different CPU sockets have different offsets (sometimes large) and possibly also some frequency difference between them. Also older CPUs don't have invariant TSC, so their speed changes when CPU frequency changes. (Some word of warning about Windows QueryPerformanceCounter: When you test it on your development laptop, it appears fast, because it's using RDTSC behind the scenes. But when it runs on a NUMA server, it often changes behavior and becomes 20-100x slower, because Windows starts to use HPET instead. RDTSC takes maybe ~10 ns to execute and it's not a shared resource. HPET takes 1-2 microseconds to read and is a shared system resource, concurrent access from multiple cores will make it slower.)
sheraz 10y ago
I use graylog, and docker can ship logs using the GELF format. Furthermore, I've got my graylog setup running from a customized docker-compose.yml file, and this thing sings.
tveita 10y ago
The Docker logging model of line-by-line stdout/stderr doesn't seem very useful beyond debugging in the local console. I prefer configuring the application logger to ship logs directly to a proper log system when possible. E.g. Logback is perfectly happy shipping structured logs directly to Kafka or Elasticsearch, with no need to re-parse the formatted log output.
- deleted 10y ago
  [deleted]
- sagichmal 10y ago
  stdout/stderr is actually a perfectly lovely default contract between application and operational environment. It's available to all programs, it's sync by default, and the semantics are intuitive.
- jacques_chester 10y ago
  > The Docker logging model of line-by-line stdout/stderr doesn't seem very useful beyond debugging in the local console. It's a well-understood pattern that Heroku made more visible under "The 12 Factor App". stdout/stderr require no configuration in any language that I'm aware of. Why should the app care about how to wire up logging? That's a platform concern. On Cloud Foundry you get Loggregator, which frankly needs improvement, but for the most part you don't care about how to wire up logging. You print to stdout or stderr and the platform wicks that away to a firehose service for you. You can hook up kafka to spout, or elasticsearch, or anything else you like. Disclosure: I work at Pivotal, we donate the majority of engineering to Cloud Foundry.
SixSigma 10y ago
> But unlike syslogd, Fluentd was built from the ground up to unify log sources from microservices, so they can be used efficiently for production and analytics. syslogd(8) -r This option will enable the facility to receive message from the network using an internet domain socket with the syslog service (see services(5)). Why won't this work with dynamic DNS ?
lobster_johnson 10y ago
Fluentd is decent (though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go), but I'm still looking for a good solution to aggregate, inspect and search logs. In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. There was one (Loggly?) which provided a CLI tool for tailing, but it wasn't very impressive. (All the SaaS apps I have looked at do quite poorly when it comes to rendering live logs in a browser, too.) Any recommendations?
- yacn 10y ago
  We use Sumo Logic which has a Live-Tail feature [0], albeit I haven't used it myself yet. [0] https://www.sumologic.com/press/2016-01-21/sumo-logic-announces-availability-of-sumo-logic-live-tail/ https://www.sumologic.com/press/2016-01-21/sumo-logic-announ...
- nlh 10y ago
  Scalyr supports live tail (in both classic and new UI modes): http://www.scalyr.com http://www.scalyr.com (*disclaimer: I'm one of the co-founders)
  - lobster_johnson 10y ago
    Are the screenshots from the old or new UI? Looks a bit dated on first glance (sorry). However, the pricing looks very silly. 10GB/day is toy volumes, and only 30 days' retention, and this for $400/mo? That's not going to fly (sorry again).
    - nlh 10y ago
      No need to apologize :) Yes - screens are from old UI. New UI is currently in public beta and, as you'll see, dramatically improved: https://www.scalyr.com/product/new-ui/opt-in https://www.scalyr.com/product/new-ui/opt-in Re: pricing. A much bigger topic than there's room for here on HN, but in short - for a service that can ingest 1TB/day+ of your logs and gives you search times measured in milliseconds, you'll actually find our pricing is not only in line, but below a lot of other commercial providers (Splunk, Sumo Logic, etc.) And when you compare TCO with open source solutions (ELK, etc.) that require a significant amount of effort to scale, it's similarly competitive.

179 points by hkmurakami 10y ago

ivan_ah 10y ago

How does fluentd compare with logstash, the other logging swiss army knife?

ktamura 10y ago
There's been some comparisons in the past. http://logz.io/blog/fluentd-logstash/ http://logz.io/blog/fluentd-logstash/ https://www.pandastrike.com/posts/20150807-fluentd-vs-logstash https://www.pandastrike.com/posts/20150807-fluentd-vs-logsta... http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/ http://jasonwilder.com/blog/2013/11/19/fluentd-vs-logstash/
windowsworkstoo 10y ago
They are all much of a muchness....fwiw distros that ship with rsyslogd as the syslog daemon are in a good spot, since rsyslog can do sufficient routing and spooling to account for network interruption. One less thing to install. And it supports the latest structured syslog RFC. That said, I use nxlog basically everywhere because I run a heterogeneous environment and it works well on all OS's I use (and is fast and light on resources)

im_down_w_otp 10y ago

The first paragraph closes with a series a incorrect assertions contrasting containers with VMs, and "distributed logging architecture in the container era" is apparently nearly identical to the distributed logging architecture of the VM era in reality, but according to the article is a completely different thing and mindset altogether... again while looking basically identical.

virtualpaul 10y ago
VMs last much longer than containers, no? I can't remember the last time I spun up a VM that lived for less than a few seconds...
- im_down_w_otp 10y ago
  Mine live for milliseconds, but I'm a weird case where that's intentional as a security construct. But that doesn't really have anything to do with the logging architecture. How long the thing lives doesn't matter when you're using aggregation, bypassing the local filesystem, and inspecting through a central portal. People have been doing this in VM environments for a lot longer than they've been using containers. Whole companies and services were born to facilitate this, and some have even died already, in the time before containers hit full stride in the hype cycle.
  - scaryclam 10y ago
    Yes, I was left scratching my head as to why containers play any significant role in the issue of distributed logging. If your choice of container runtime/vm/servers are creating an issue with logging then there's most likely an issue with the system architecture rather than actual logging. We've been building distributed systems (with containers and other approaches) for years and this is the first time I've heard of logging being a problem.

kozikow 10y ago

I recently just plugged python logger from https://github.com/GoogleCloudPlatform/gcloud-python#google-stackdriver-logging https://github.com/GoogleCloudPlatform/gcloud-python#google-... into my python codebase. Configuration took 15 minutes, I have nice dashboard showing recent logs, I can log either strings or jsons. I can analyze data later using BigQuery. It works from docker containers. I was contemplating trying out logstash or fluentd, but I don't see any major advantage over current simple solution. Can anyone more knowledgable than me help with explaining what I would get from logstash/fluentd?

pfranz 10y ago
Does any of your Python processes open a subprocess that then spawns another Python process as a child (and both gets logged)? If so, have you found any good way to associate the the parent and child logging messages? If you just write to stdout everything makes sense in the file top to bottom, but if you're logging json objects into a database I had problems; if your parent process echos stdout from the child you lose a lot of context about the child, you can also double-log child process messages (because the parent is also echoing them), and you can't easily associate the parent/child objects. I came up with a hacky solution, but I wasn't logging these events to a centralized server at the time. (I haven't used gcloud's dashboard, so I might be making wrong assumptions) To respond your question: it sounds like you have something that works fine for logging messages. Personally, I split it into categories 1) logs (serial events that needed context for any usefulness) and 2) events (metrics or exceptions). I wrote 1) to traditional log files and wrote 2) to an ElasticSearch or statsd database to log exceptions to. Ideally, I wanted to use the same mechanism for both and peel off relevant data into separate databases. A metrics database like Elasticsearch will let you query things like, "What modules give the most errors?" "Has this function been called more often this week than last?" "Is this process taking longer when using the newly released version compared to the old one?" etc.
- derefr 10y ago
  Use the process group ID? (http://unix.stackexchange.com/a/139230 http://unix.stackexchange.com/a/139230)

rdli 10y ago

I'm surprised the article didn't mention anything about the nuances of actual logging in a distributed architecture. In particular, when a request starts to span multiple containers or microservices, you need a way to correlate all log data to a given request. You typically do this by propagating a trace ID through your logging system, and a vector clock-type thing for causal ordering. If you can't tell which log statements belong to which request and the order in which they occur, debugging is quite hard :-).

ktamura 10y ago
While the article does not touch on this explicitly, Docker's Log Driver for Fluentd supports tags, which can be used to pass on container name/ID and other metadata to wherever logs are sent. Since Docker Machine itself allows these metadata to be passed on via API ({{.Name}}, etc.) one can create a fairly robust system to track log lineage.
- meddlepal 10y ago
  That approach requires a lot of post processing to make it useful. Further it's pretty hard (near impossible) to do causal ordering.
  - ktamura 10y ago
    I agree on the near-impossibility of causal ordering at scale. I don't see the need for a lot of "post processing". Adding filters in Fluentd is fairly simple: http://docs.fluentd.org/articles/filter-plugin-overview http://docs.fluentd.org/articles/filter-plugin-overview
- rdli 10y ago
  How would passing on the container ID help in mapping multiple log messages to the same request? If a request is going through multiple containers, you'll get multiple container IDs. To solve this problem, you need a unique trace ID and vector clock, and this actually needs to propagated through your logging system, not through container tags.

voiper1 10y ago

This is going to sound dumb, but I started dumping all my logs into mysql so that I wouldn't also need to run an elastic search cluster. I then query by the error level, latency, status code to find items of interest. And everything is stored by some sort of request ID so that I can correlate multi-entry logs. It's not pretty -- there's no real-time alerts, but I'm not paying dollars per gb to AWS and I have no clue what google will end up charging... and I'm already running mysql. So there is that...

packetized 10y ago
I'm guessing your log write volume is relatively low.
deleted 10y ago
[deleted]
bni 10y ago
Congrats for choosing the solution with the least moving parts and rejecting premature optimization and over-engineering.
brightball 10y ago
It's a good solution depending on write volume. Watching the pricing on these logging services is always fairly shocking to me. I've done a similar thing with Mongo in the past with their capped collections.

bluecmd 10y ago

They say that without aggregation scale-out is impossible. That's simply not true. Using things like anycast and ECMP it's super easy as log destinations are usually stateless. Also, never had network contention because I logged too much unless something was really broken - and then only at that node. I'll continue sending events to my anycast address without any aggregation thank you very much.

sagichmal 10y ago
> They say that without aggregation scale-out is > impossible. That's simply not true. Using things like > anycast and ECMP it's super easy . . . Then you haven't worked at significant scale. That's fine! Just keep everything in context :)
- bluecmd 10y ago
  A "probably" there would be nice. I don't want to have to defend myself, but I'm pretty confident about the scale I've worked on. I guess it boils down to what you call aggregation. Is having multiple stateless receivers behind anycast/ECMP aggregation writing to a distributed database aggregation? I'd argue not, but maybe this is where the difference in opinion lies.

r4um 10y ago

Inbuilt docker + fluentd driver + plain ES falls flat on its face under scale. Not to mention docker daemon keeps dying :(. Whats working for us now is local docker json logs -> heka -> kafka -> graylog -> ES. We even dockerized graylog to scale the processing up and down on demand. Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s

edu115 10y ago
Too bad that Heka has been officially deprecated, sounds like a similar use case to what we have.
vardump 10y ago
> Does 250k messages/sec at peak easily, More details in this presentation https://www.youtube.com/watch?v=PB8dBnpaP8s https://www.youtube.com/watch?v=PB8dBnpaP8s 250k messages/sec is pretty low. It'd be useful (not to mention entirely feasible) to be able to handle 5M/sec bursts on a single system and aggregate sustained 50M/s+ (10Gbps network). The system should be able to establish total order (serialization) locally and causal order over the network. I did some tests and found out hardware limits are somewhere between 20-50M messages per second (serialized) on current consumer grade X86. Practical implementation would of course be slower. Of course you need to go binary logging at that point, adhere to cache line boundaries, etc. mechanical sympathy. Maybe even do usermode networking at the aggregator server. Binary logging is a must, because even something like string formatting is simply way too slow, by an order of magnitude. In my quick tests I found the string processing hit to be surprisingly high, 10-50x. C "sprintf" and C++ stringstream are atrociously slow. (Surprisingly, considering sprintf has a pretty complicated "bytecode" format specifier parsing loop, it was still significantly faster than stringstream implementation.) Timestamps are another huge performance issue. Typical system calls for high precision (a few microseconds or better) timestamp take several microseconds each - using one of those will alone drop performance to 100-500k range. It's of course much faster to use RDTSC, but then you have the issue that different CPU sockets have different offsets (sometimes large) and possibly also some frequency difference between them. Also older CPUs don't have invariant TSC, so their speed changes when CPU frequency changes. (Some word of warning about Windows QueryPerformanceCounter: When you test it on your development laptop, it appears fast, because it's using RDTSC behind the scenes. But when it runs on a NUMA server, it often changes behavior and becomes 20-100x slower, because Windows starts to use HPET instead. RDTSC takes maybe ~10 ns to execute and it's not a shared resource. HPET takes 1-2 microseconds to read and is a shared system resource, concurrent access from multiple cores will make it slower.)

sheraz 10y ago

I use graylog, and docker can ship logs using the GELF format. Furthermore, I've got my graylog setup running from a customized docker-compose.yml file, and this thing sings.

tveita 10y ago

The Docker logging model of line-by-line stdout/stderr doesn't seem very useful beyond debugging in the local console. I prefer configuring the application logger to ship logs directly to a proper log system when possible. E.g. Logback is perfectly happy shipping structured logs directly to Kafka or Elasticsearch, with no need to re-parse the formatted log output.

deleted 10y ago
[deleted]
sagichmal 10y ago
stdout/stderr is actually a perfectly lovely default contract between application and operational environment. It's available to all programs, it's sync by default, and the semantics are intuitive.
jacques_chester 10y ago
> The Docker logging model of line-by-line stdout/stderr doesn't seem very useful beyond debugging in the local console. It's a well-understood pattern that Heroku made more visible under "The 12 Factor App". stdout/stderr require no configuration in any language that I'm aware of. Why should the app care about how to wire up logging? That's a platform concern. On Cloud Foundry you get Loggregator, which frankly needs improvement, but for the most part you don't care about how to wire up logging. You print to stdout or stderr and the platform wicks that away to a firehose service for you. You can hook up kafka to spout, or elasticsearch, or anything else you like. Disclosure: I work at Pivotal, we donate the majority of engineering to Cloud Foundry.

SixSigma 10y ago

> But unlike syslogd, Fluentd was built from the ground up to unify log sources from microservices, so they can be used efficiently for production and analytics. syslogd(8) -r This option will enable the facility to receive message from the network using an internet domain socket with the syslog service (see services(5)). Why won't this work with dynamic DNS ?

lobster_johnson 10y ago

Fluentd is decent (though I rather wished Heka had taken off; it's much more flexible and in theory leaner and faster since it's Go), but I'm still looking for a good solution to aggregate, inspect and search logs. In particular, the ability to live-stream ("tail") logs seems to be a feature generally missing from logging aggregators. There was one (Loggly?) which provided a CLI tool for tailing, but it wasn't very impressive. (All the SaaS apps I have looked at do quite poorly when it comes to rendering live logs in a browser, too.) Any recommendations?

yacn 10y ago
We use Sumo Logic which has a Live-Tail feature [0], albeit I haven't used it myself yet. [0] https://www.sumologic.com/press/2016-01-21/sumo-logic-announces-availability-of-sumo-logic-live-tail/ https://www.sumologic.com/press/2016-01-21/sumo-logic-announ...
nlh 10y ago
Scalyr supports live tail (in both classic and new UI modes): http://www.scalyr.com http://www.scalyr.com (*disclaimer: I'm one of the co-founders)
- lobster_johnson 10y ago
  Are the screenshots from the old or new UI? Looks a bit dated on first glance (sorry). However, the pricing looks very silly. 10GB/day is toy volumes, and only 30 days' retention, and this for $400/mo? That's not going to fly (sorry again).
  - nlh 10y ago
    No need to apologize :) Yes - screens are from old UI. New UI is currently in public beta and, as you'll see, dramatically improved: https://www.scalyr.com/product/new-ui/opt-in https://www.scalyr.com/product/new-ui/opt-in Re: pricing. A much bigger topic than there's room for here on HN, but in short - for a service that can ingest 1TB/day+ of your logs and gives you search times measured in milliseconds, you'll actually find our pricing is not only in line, but below a lot of other commercial providers (Splunk, Sumo Logic, etc.) And when you compare TCO with open source solutions (ELK, etc.) that require a significant amount of effort to scale, it's similarly competitive.