5 ms·

Your marketing seems to emphasize storage heavily. Is that the biggest pain point in logging today? If so, my guess would be that retention period could be a bi

by tyre 11y ago

Your marketing seems to emphasize storage heavily. Is that the biggest pain point in logging today? If so, my guess would be that retention period could be a bigger differentiator than sheer gigabytes. What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch for example)? With Slack, it is integrations. If you could do something similar, like a bugsnag + new relic that aggregates all of our logs and notifies us when things happen (either individual incidents or aggregates like "nginx error log above 1 event/10s), you can have all my money. Another idea would be to standardize transactions across the stack, so I could trace a request from nginx through rails/phoenix, to postgresql, etc. Again, take my money. A better UI only helps me if I know when to look at it or what I'm looking for.

leeab 11y ago
Yup we will be doing some of the items you mentioned. But you gotta start somewhere and launch something. This is just our mvp. :)
- tyre 11y ago
  I hear you and congrats on the launch! To be clear, I'm not asking that you have all of this out of the gate. It's that I want to see the vision and get excited about where you're heading. Your early users are investing in your future and taking a bet on what you could become. Right now, I'm not sure what I should be dreaming about.
  - leeab 11y ago
    You're absolutely right...it's important to clarify the vision. Our vision is to do analytics in the future. Today it's aggregate/search/tail/alerting. I think the beauty of this is that our future is still somewhat unknown and we'll listen to our customers on where we ought to be headed.
    - tyre 11y ago
      For _you_ the unknown future may be exciting, but as a customer it isn't. I'm trusting you with a critical part of my infrastructure and betting that you're the future of logging. If I'm going to make that bet, I have to believe that you have a clear vision and that I'm on board with that. Obviously internally (as a fellow founder), things are always crazy and you adapt and grow. But your customers need more than the faith that you'll figure it out as you go.
acquihired 11y ago
You can do everything you are asking for with Striim. Www.striim.com
- leeab 11y ago
  I think we're targeting different audiences. We're only really for logging.
  - acquihired 11y ago
    Striim's target audience includes people looking for logging too. I was referring to the aggregation of the logs, including taking columns / data types from different log streams and combining them (in real time) to create new objects. You can then set alerts and build dashboards on the new objects. Additionally he mentioned standardizing the request across the stack. Since Striim can import the logs from all these types of services and combine them to create a new object that represents the request across the stack he could accomplish his second request as well. Combine that with the ability to ingest huge amounts of data and query in real time, it sounded like what he was looking for.
chrissnell 11y ago
Ding ding ding ding. It's not the log storage that we care about, it's what we can do with those logs. I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs. Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle. Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.
- leeab 11y ago
  The main reason we emphasized storage is because we built it with scaling that in mind, that's all. We've also built it with live tail in mind too. It's an okay differentiator since we believe we can handle the volume with our infrastructure. We also followed up with UI/UX which at the end of the day should make us a platform you can goto, get what you need and go back to work. Give us a try, even if you don't think you'll use the GBs. We have alerting, field parsing and live tail. Analytics is the only piece we don't have today, but we're working towards it.
  - chrissnell 11y ago
    I understand. For me, though, scaling is just a small piece of the puzzle and not really a differentiator. To give you an idea of our size, we run about ~500 instances in a private cloud hosted on about 45 dedicated servers at Rackspace. We use OpenStack for stateful services and data storage and we use Kubernetes/Docker/CoreOS for 12FA/microservices. We have many dozens of apps and even with all of that, we're still only pushing about 4-5 GB of logs each day. The number one most important problem we're trying to solve is the actual parsing of logs during ingestion. We need logs from commonly-used apps & frameworks to be automatically identified and parsed and we need an easy way to set up a custom parser for internal logs that doesn't require crazy hand-written regex. How does LogDNA handle parsing? What apps/frameworks are supported?

by tyre 11y ago

leeab 11y ago

Yup we will be doing some of the items you mentioned. But you gotta start somewhere and launch something. This is just our mvp. :)

tyre 11y ago
I hear you and congrats on the launch! To be clear, I'm not asking that you have all of this out of the gate. It's that I want to see the vision and get excited about where you're heading. Your early users are investing in your future and taking a bet on what you could become. Right now, I'm not sure what I should be dreaming about.
- leeab 11y ago
  You're absolutely right...it's important to clarify the vision. Our vision is to do analytics in the future. Today it's aggregate/search/tail/alerting. I think the beauty of this is that our future is still somewhat unknown and we'll listen to our customers on where we ought to be headed.
  - tyre 11y ago
    For _you_ the unknown future may be exciting, but as a customer it isn't. I'm trusting you with a critical part of my infrastructure and betting that you're the future of logging. If I'm going to make that bet, I have to believe that you have a clear vision and that I'm on board with that. Obviously internally (as a fellow founder), things are always crazy and you adapt and grow. But your customers need more than the faith that you'll figure it out as you go.

acquihired 11y ago

You can do everything you are asking for with Striim. Www.striim.com

leeab 11y ago
I think we're targeting different audiences. We're only really for logging.
- acquihired 11y ago
  Striim's target audience includes people looking for logging too. I was referring to the aggregation of the logs, including taking columns / data types from different log streams and combining them (in real time) to create new objects. You can then set alerts and build dashboards on the new objects. Additionally he mentioned standardizing the request across the stack. Since Striim can import the logs from all these types of services and combine them to create a new object that represents the request across the stack he could accomplish his second request as well. Combine that with the ability to ingest huge amounts of data and query in real time, it sounded like what he was looking for.

chrissnell 11y ago

Ding ding ding ding. It's not the log storage that we care about, it's what we can do with those logs. I've been through three logging vendors in the last two years (Loggly, LogEntries, now Sumo Logic) and I'm looking to move once again. The secret sauce is not the storage volume; our volume is relatively small: 4.3 GB/day. The secret sauce is the intelligent structuring of unstructured log data, analytics+alerting on the now-structured data, and live tail'ing of logs. Sumo Logic does an okay job at the parsing/querying/alerting/analytics piece but live tailing is a brand-new feature for them and it has limitations. Sumo Logic's pricing is also frustrating--they don't apply volume discounting until you're pushing something like 500 GB of logs each month. Again, we need affordable analytics; storage is a small piece of the puzzle. Honestly, our next move may be to an internal "HEK" stack: Heka, Elasticsearch, and Kibana. I've looked at the cost of doing this and it's actually more expensive when you consider what we'd spend to host a big ES cluster internally, but cost is not our #1 consideration, usability is.

leeab 11y ago
The main reason we emphasized storage is because we built it with scaling that in mind, that's all. We've also built it with live tail in mind too. It's an okay differentiator since we believe we can handle the volume with our infrastructure. We also followed up with UI/UX which at the end of the day should make us a platform you can goto, get what you need and go back to work. Give us a try, even if you don't think you'll use the GBs. We have alerting, field parsing and live tail. Analytics is the only piece we don't have today, but we're working towards it.
- chrissnell 11y ago
  I understand. For me, though, scaling is just a small piece of the puzzle and not really a differentiator. To give you an idea of our size, we run about ~500 instances in a private cloud hosted on about 45 dedicated servers at Rackspace. We use OpenStack for stateful services and data storage and we use Kubernetes/Docker/CoreOS for 12FA/microservices. We have many dozens of apps and even with all of that, we're still only pushing about 4-5 GB of logs each day. The number one most important problem we're trying to solve is the actual parsing of logs during ingestion. We need logs from commonly-used apps & frameworks to be automatically identified and parsed and we need an easy way to set up a custom parser for internal logs that doesn't require crazy hand-written regex. How does LogDNA handle parsing? What apps/frameworks are supported?