12 ms·
Show HN: LogDNA – Easy logging in the cloud
- leeab 11y agoHey everyone. I'm a co-founder / CTO of LogDNA. We were in Y Combinator's W15 batch working on a eCommerce marketing platform (PushMarket), only to realize we built a powerful logging system many of our friends wanted. We had our “slack moment” and decided to pivot. We were frustrated with the current logging landscape and built LogDNA around 3 philosophies: 1. More Storage - give away ample storage so you can log everything 2. Longer Search - faster and longer search retention 3. UI/UX - a much cleaner experience to interact with your log tails I’m happy to answer any questions you may have!
- jc_811 11y agoHi! Looks really great at first glance. Is there anything you guys do that differentiates yourself from other log management platforms (eg Kibana, Splunk, etc)
- leeab 11y agoMainly the 3 items we mentioned above. Technically, building a scalable infrastructure to ingest an order of magnitude of GBs more than other platforms was a bit challenging (and still is) but I think we're on our way to achieving that.
- ebouck 11y agoWe've been using the product for a few months now after have looked at most of the alternatives. The biggest differentiators for me have been: 1. Really straightforward, scriptable installation 2. Intuitive view of the log data 3. Very quick and easy to select a time frame 4. UI is oriented to cope with lots of server creation/destruction (very useful for auto-scaling environments and not well handled in some of the other products) We are very happy with the product - great job guys!
- leeab 11y agoMany thanks @ebouck!
- kazazes 11y agoAre there any client libraries for LogDNA or does it just stream /var/log/* and other logs to LogDNA? Also, you provide APTs and RPMs, but no sources. Is there any way I can build LogDNA and run it on Arch Linux?
- zeisss 11y agoI was also looking for documentation of how to forward data to them. Its probably only available after login.
- leeab 11y agoAhh yes documentation...we're also working on that as we speak. Sorry about that...we had to pick and choose what we could have for launch. But the instructions on logdna.com should be enough to get you started.
- leeab 11y agoClient libraries are coming soon! We decided on an agent mainly for ease of installation and so we can self-update the agent, so you don't have to manage it (especially for auto-scaling environments). We will be open sourcing our agent soon, we literally launched this week...some things we couldn't get to for launch. And yes, it currently streams /var/log/* by default and any other paths your specify to the agent.
- Gigablah 11y ago> We will be open sourcing our agent soon Great! I want to stick it in a Docker container.
- atmosx 11y agologstash/filebeat integration would make sense actually.
- tyre 11y agoYour marketing seems to emphasize storage heavily. Is that the biggest pain point in logging today? If so, my guess would be that retention period could be a bigger differentiator than sheer gigabytes. What's the killer use case or differentiator (vs. loggly+kibana+elasticsearch for example)? With Slack, it is integrations. If you could do something similar, like a bugsnag + new relic that aggregates all of our logs and notifies us when things happen (either individual incidents or aggregates like "nginx error log above 1 event/10s), you can have all my money. Another idea would be to standardize transactions across the stack, so I could trace a request from nginx through rails/phoenix, to postgresql, etc. Again, take my money. A better UI only helps me if I know when to look at it or what I'm looking for.
- leeab 11y agoYup we will be doing some of the items you mentioned. But you gotta start somewhere and launch something. This is just our mvp. :)
- tyre 11y agoI hear you and congrats on the launch! To be clear, I'm not asking that you have all of this out of the gate. It's that I want to see the vision and get excited about where you're heading. Your early users are investing in your future and taking a bet on what you could become. Right now, I'm not sure what I should be dreaming about.
- leeab 11y agoYou're absolutely right...it's important to clarify the vision. Our vision is to do analytics in the future. Today it's aggregate/search/tail/alerting. I think the beauty of this is that our future is still somewhat unknown and we'll listen to our customers on where we ought to be headed.
- tyre 11y agoFor _you_ the unknown future may be exciting, but as a customer it isn't. I'm trusting you with a critical part of my infrastructure and betting that you're the future of logging. If I'm going to make that bet, I have to believe that you have a clear vision and that I'm on board with that. Obviously internally (as a fellow founder), things are always crazy and you adapt and grow. But your customers need more than the faith that you'll figure it out as you go.
- maerF0x0 11y agoThe more storage thing seems disingenuous . Splunk's biggest plan is 3x yours. It seems that your real offer is just price per storage atm. More for the same money. infact only splunk light competes with you on price but on not storage GBs.
- leeab 11y agoI don't think we'll be targeting enterprise users like Splunk...especially since their featureset will take a long time to build out. We might be one day but not today. Yes, you are correct, more value for your money is our current offer. We want to build a great product and to do that, you have to hire great people, so we came up with a pricing that'll hopefully achieve that. But it's not set in stone and we'll adjust just like any other company.
- maerF0x0 11y ago> We want to build a great product and to do that, you have to hire great people, so we came up with a pricing that'll hopefully achieve that. None of that is customer value. Pricing must favor my needs over any of your company's long term goals.
- mbesto 11y agoHow do you differentiate against https://papertrailapp.com/ https://papertrailapp.com/ ?
- welder 11y ago100GB/month doesn't matter when you only store the logs for 2 days... Why does your free plan have such a short retention time? Is the 2 day retention time because you sync logs to the user's s3 bucket? If so then it would help conversion to say that next to the 2 days retention copy.
- leeab 11y agoWe had a specific audience in mind when we chose 2 days. We were targeting users who just needed to debug their apps, or a low traffic production app. For this, 2 days is plenty. The space is there mainly so they won't have to think about that 5GB/month may equal to 160mb/day or if a bug floods their logs and pushes them over their quota or dealing with overages. If you care about retention, chances are you're also building a serious product and you want to rely on LogDNA to be around. For that, we hope you'll be on a paid plan. We want to build the best product we can and by charging appropriately, we can afford better people.
- welder 11y agoThanks for the reply. Is it possible to sync logs to s3 for backup after then 2 day retention period?
- tudborg 11y agoDamn, this is _exactly_ what I just started building... ¯\_(ツ)_/¯ Oh well, It looks really nice!
- agopaul 11y agoIf I were you, I would put a little emphasis on the security of your service. Logs can definitely contain sensitive data and companies rely on you to store that data in a "proper" way. At least, as a possible customer, that is one of my biggest concern. But maybe I'm just too focused on security given that our clients ask us about it pretty much every day. So as a possible vendor (for us or other SaaS businesses) and being part of our chain of trust, I'd expect some words on how you handle security and how much it is important for you. That is lacking on the website if I'm correct. Anyway, congrats on the service! I think that there is room for LogDNA in the logging-warehousing market
- leeab 11y agoYes, thanks for the feedback. We'll be adding this and a FAQ section soon. We do encrypt everything in transit, end-to-end, just like most in our industry. We do not encrypt at rest at the moment but will be looking into it.
- illumin8 11y agoNot encrypting at rest seems like a huge oversight, given that your service most likely uses AWS S3 and it is dead simple to turn on encryption. I'm not sure anyone will trust you to store their log data without at least encrypting data at rest.
- toomuchtodo 11y agohttp://docs.aws.amazon.com/AmazonS3/latest/dev/UsingEncryption.html http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingEncrypti... It's literally setting a policy on a bucket and a flag on object PUTs if you're not managing the keys yourself.
- leeab 11y agoYes we do plan to be doing this for S3 archives.
- Rauchg 11y agoAwesome stuff leeab!
- leeab 11y agoThanks @Rauchg
- kordless 11y agoHappy to see the shell back in action somewhere. Well done!
- samstave 11y agoHIPAA Compliance?
- leeab 11y agoNot yet. We're still figuring our what's needed for this. It seems there's no auditing requirement at all.
- samstave 11y agoI want to help you figure this out. Email in profile.
- mateo411 11y agoYou must encrypt the logs at Rest. Also, if you are dealing with a production issue with HIPPA logs, you must keep them on the machine. You can't send anything in an email, send a stack trace over a messaging client, or put them into JIRA.
- dmmalam 11y agoI'd love to have all logs stored from day 1 (ie unlimited retention). Makes looking for bugs a million times easier, when you don't need to open up archived logs from S3. Might be an interesting technical challenge to the keep search fast with 100gbs of logs.
- welder 11y agoThey delete the logs after 2 days (free plan) and 60 days (most expensive plan available). The 100GB is only monthly transfer. Would be nice to have a cloud log service with unlimited retention that's searchable, but for now self-hosting is the best choice using logstash [1] for logging and statsd [2] for metrics. [1] https://github.com/elastic/logstash https://github.com/elastic/logstash [2] https://github.com/etsy/statsd https://github.com/etsy/statsd
- zobzu 11y agoHey it looks like your debian/main page install path does not sign packages or use any trusted channel.
- ldoroud 11y agoDNA is normally and naturally right-handed which means your logo is wrong!
- ghayes 11y agoQuick question, is https://heroku.logdna.com https://heroku.logdna.com down? Doesn't look like my logs are coming in and wondering if this is the HN hug of death? > curl: (7) Failed to connect to heroku.logdna.com port 443: Connection refused
- lobster_johnson 11y agoThe CLI looks interesting. Can I use it to pipe logs to you? That would be really useful for, say, running scripts and maintenance jobs on a bunch of boxes. That would allow me to collect data and inspect it later (or tail it live). Oh, and I see you provide an OS X package for the tool. I'd vastly prefer a Homebrew package over that.
- whalesalad 11y agoLogging definitely sucks. It all sucks. Graylog. Elasticsearch and Kibana. Don't even get me started on Logstash which is some of the worst software I've ever used. When you have a Docker/Kubernetes solution I will be (at another YC company) very interested in putting it through it's paces. Also I'd agree with others here that having a command line tail interface as well as a gui for browsing and savable queries would be bitchin.
- agopaul 11y agoCould you elaborate on Logstash? I was considering it just a few weeks ago for a project
- rtv 11y agoFinally a product I can make my work more efficient, thank you this is what I've been searching for, We will definitely hire you :)
- jorgecurio 11y agoWhat AWS stack will let you build something like this? I don't think I would trust a 3rd party with all of my log data for...I can't really see what the benefit of moving to cloud would be, a log file sits on a local disk or if you had to move it to the cloud you could use S3 or SQS even no?
- leeab 11y agoMainly so you can avoid tailing or grepping from 1 server at a time, or setting up an elaborate log shipping method of your own. We'll do that for you. :) Also, we just use AWS for EC2 mainly.
- tommoor 11y agoAnother benefit is if you have many services then viewing the logs in timestamped order can make debugging across services far easier. This is really quite difficult without a single service aggregating and indexing all your logs
- maerF0x0 11y agoIt gets especially useful when you start to get unique ids across "jobs". Imagine a single flow of work hits 10 services, but you can search for job_id=123 and get all the logs for the job . Now its easier to see what went wrong and where in the flow.
- nzoschke 11y agoNothing on AWS will give you the UX that LogDNA is delivering. And a good experience working with logs can be a huge productivity and visibility boost to your team. But the infrastructure components are certainly there on AWS. CloudWatch Logs will ingest tons of data at minimal cost. It has simple API for searching for events, but it's extremely slow to return results if you have lots of data in a LogGroup. It has a simple API to get the latest events but it has a 10s delay and its a bit tricky to get all the events from all the LogStreams in the right order. Kinesis will ingest tons of data at minimal cost, and let you stream it back in real time. Lambda can subscribe to all the CloudWatch Log or Kinesis Events for parsing, filtering, and forwarding. AWS ElasticSearch can ingest all the logs and give you a richer query language and visualization tools. I've been building all this into Convox, an open source AWS toolkit. Take a look at this CloudFormation template for an example of how to configure a CloudWatch Log Group and Lambda filter in your own AWS account: https://github.com/convox/rack/blob/master/api/dist/kernel.json#L249-L373 https://github.com/convox/rack/blob/master/api/dist/kernel.j...
- compumike 11y agoOur team at Pantelligent has been happily using the LogDNA beta for the last few months. The product is well designed and easy to use for many members across engineering, devops, and marketing roles. Very impressed so far, and this is only the beginning of where this product can go! Setup takes about 2 minutes -- give it a try today.
- chrisab 11y agoThanks for the kind words and it's awesome to have your team using our product. Your product feedback on how we can make this platform better has been golden!
- tyre 11y agoI downvoted you because this sounds too much like an infomercial. Be honest, share your story, support your YC brethren, but we're users, not VCs.
- paulspringett 11y agoNice! Have you got plans to support JSON / logstash formatted logs?
- leeab 11y agoWe actually already do this, but we don't show the results in the log viewer, but we do parse JSON and a whole bunch of various logstash patterns + some of our custom ones. You can even search on them today by doing: fieldname:keyword but it's just hard since you don't know what the fieldnames are...you can guess but we're exposing those soon!
- leef 11y agoWhy the severe restriction on users? $300/mo to go beyond five users eliminates any advantage on log volume.
- leeab 11y agoTo be honest, we weren't really sure what to pick. We just wanted to have something in place. It's not set in stone and we'll likely change things as we see different use cases.
- ohblahitsme 11y agoMaybe I'm missing something, but it looks like this has nothing to do with DNA? Why is it called LogDNA? Seems a little misleading. That aside, it looks very interesting!
- agentgt 11y agoI have few questions and comments. * Love the price * Can or will you add name value pairs? Logging for us is no longer just a single line of text. We have attached metadata. Kibana handles this nicely. * How good is your security? This is actually our biggest reason why would not use a service like this. Not that we put passwords or user sensitive data in logs we still have to keep them extremely secure due to compliance. I'm sure you have decent security but I would recommend emphasizing that and/or proving it.
- leeab 11y agoThanks for the feedback. Yes, just send us JSON in a line and we'll parse it. You can actually search on that today, we just haven't made a way for you to see the parsed fields yet (coming soon!) We're fully encrypted end-to-end in transit (but not at rest, currently). Yeah we need documentation mainly and couldn't get that in time for launch.
- zobzu 11y agoencryption at rest would be quite amazing.
- jaequery 11y agoAnyone know where this fits into servers requiring PCI-compliances?
- leeab 11y agoHaving gone through PCI compliance before, technically you cannot send us data that includes card numbers since we're not PCI compliant (yet). But if you don't log that type of data, there is no issue.
- nodesocket 11y agoLooks nice, but I am solid lover/supporter of Papertrail. I really love their simple/minimal approach. I think the missing piece of Papertrail is the ability to send JSON logs, and filter inside of a JSON documents.
- leeab 11y agoWe'll parse your JSON if you send it in one line. And you can search on it like this: fieldname:keyword I think we're pretty good on our UI/UX. Give us a try!
- nodesocket 11y agoSo we currently ship around 10GB a month, and it costs us $75 a month from Papertrail. Cost is really a non factor, wouldn't matter if is cost us $200. The ability to search and grep older documents is a bit slow on Papertrail though. Additionally, sending deeply nested JSON, and being able to search on props would be huge. Does LogDNA have a shipper from syslog? Do you have a node.js client lib? Finally can you talk about security? Is traffic encrypted over the wire? Do you encrypted disks at rest?
- leeab 11y agoWe parse JSON and you can search your own fields. We do not have syslog yet or code libraries. We have an agent that ships logs today via secure web socket. Everything is encrypted in transit, end-to-end. We haven't not researched into encryption at rest on EC2 yet.
- lobster_johnson 11y agoI'd like to use LogDNA, but we'd require syslog support, I think. We use Rsyslog to spool logs off to a central server, and we're not about to trust a third-party vendor to keep our data (ie., we want our own copy). Rsyslog can forward to any other syslog server and supports TLS. Edit: Rsyslog can pipe data to a command-line program via the "omprog" plugin, which might be an easier option for you.
- deleted 11y ago[deleted]
- phoenix24 11y agoI've been using scalyr.com for logging extensively, for last few months. How do you differentiate against them?
- leeab 11y agoOther than the post by their founder above, I'm not really familiar with them. But overall, the 3 points I mentioned in my opening comment is generally similar. But to really find out if you'll like us, you should just try installing us. We have a working live demo on https://logdna.com https://logdna.com
- chrissnell 11y agoI think your "Features" page could use some work. I don't consider syntax highlighting themes to be much of a feature. What I want to hear about is how you can take logs from all of these <dozens> of applications and intelligently parse them and deliver analytics and alerting on them.
- encoderer 11y agoCongrats on shipping! At Cronitor, we have a lot of users pushing events from PaperTrail for monitoring and alerts. I'd love to work with you on an easy integration at some point, though I do have an appreciation for how busy you are. We didn't add our first real integration -- PagerDuty in our case -- for a few months after launch.
- leeab 11y agoThanks! Send me an email at lee[]logdna.com
- cporios 11y agoScrolling down your website is very slow, weird and annoying. Also the scroll left to go back gesture isn't working.
- mbrock 11y agoIt's just astounding how web sites still insist on replacing the normal, perfectly fine scroll behavior with weird, buggy, slow, crazy, half-assed, dizzying, and totally unnecessary custom scroll implementations.
- thecatspaw 11y agoscrolling down worked fine for me, but going back by swiping didnt work
- melted 11y agoThis only solves a part of the problem. This is basically to collect and grep your stderr, useful for when you need to debug something, but that's about it. A far more interesting problem is to collect, ingest, and query structured (Proto, Thrift, etc) _application_ logs that people should also be writing.
- Swizec 11y agoQuestion: Why does a new startup/solution for this pop up every few months? Why is this space so difficult? I've definitely thought about building this myself, but haven't gotten around to it. It's just sending some text to a server, isn't it? What's so hard about it? Obviously I'm missing something. I'd love to know what :)
- leeab 11y agoHi there, I wasn't aware there were several startups every few months in logging. For us, it was an interesting space. Lots of changes happening. Yes you are somewhat right. It is sending text to a server. But then you send more text and more text and you need to search for all this text in a split second. If you've ever run your own ELK stack for example, you'll realize quickly that you're spending most of your time scaling elastic search to handle the data volume. There is a challenge to doing that.
- matzipan 11y agoIt's idiotic. You're not bringing in any major differentiation. Your idea will be implemented by your competitors in less than a week and you will have no competitive advantage and way less clients. You've been through YC? How the hell did you graduate? Because you just did mistake number 4: http://paulgraham.com/startupmistakes.html http://paulgraham.com/startupmistakes.html
- halayli 11y agoPG's writings aren't the bible. Situations and verticals vary. You still need to use your reasoning after all and judge yourself. Case in point, PG himself agreed to this business idea which in your opinion contradict himself.
- matzipan 11y agoTrue, PG's writings are not the bible just as his agreement with this business idea doesn't mean it's a good idea. I used my own judgement and concluded that this is a worthless idea, as competition is high and differentiation low. I referenced that article to give credit to the original author.
- jmtulloss 11y agoMy favorite part of logentries is the fact that you can use structured logs to build dashboards showing service health. My least favorite part of logentries is the UX around that whole thing. Any chance more features are coming in the structured logging area?
- midnightjasmine 11y agoAgreed on the LogEntries UX. I find it really frustrating.
- kevincox 11y agoFirst of all, your website is painful to use. It requires a ton of scrolling to see anything and as previously mentioned scrolling is painful. Secondly there is almost no detail so while your product sounds cool I can't see what it is like to use your product. Give me examples of creating queries or dashboards. There should definitely be a documentation section on your site! Also, a really cool feature I can't see anywhere else is reliable journald log uploading. You can dump as JSON which does most of the work but it is a pain implementing a reliable uploader with a HTTP API.
- leeab 11y agoHi there, We just built our website this week since we were heads down focused on product. Our documentation is coming soon. We just wanted to get a product into the hands of users and had to make tough decisions on what we needed. We could be perfect and launch 2 months from now or just launch now with what we have. But you're right, we will be plugging the holes quickly.
- matzipan 11y agoYet another startup solving yet another already solved problem with little to no differentiation. That's why we need to encourage more kids to do STEM and tackle hard problems, rather than encouraging "makers" and getting them all to be app builders. I have yet to live the day I will regain my faith in startups.
- leeab 11y agoI'd love to see what you have worked on. :)
- matzipan 11y agohttps://yourlogicalfallacyis.com/ad-hominem https://yourlogicalfallacyis.com/ad-hominem
- jessaustin 11y agoHaha, mischaracterizing valid arguments as "ad hominem" seems more common than logging startups!
- matzipan 11y agoHow is that?
- jessaustin 11y agohttps://hn.algolia.com/?query=ad%20hominem&sort=byDate&prefix&page=0&dateRange=all&type=comment https://hn.algolia.com/?query=ad%20hominem&sort=byDate&prefi... The current first comment: It's a straight up ad-hominem attack against Justin Keller, instead of criticising [sic] his writing it attacks his character. That's just as fallacious an invocation of ad hominem as your link was. Description of questionable behavior is not fallacious, in an article the purpose of which is to complain about questionable behavior. Likewise, it is entirely reasonable to expect withering unsupported criticism like yours to spring from some a personal position of some relevant experience.
- 11y ago
- koolba 11y agoecho "deb http://repo.logdna.com stable main" | sudo tee /etc/apt/sources.list.d/logdna.list sudo apt-get update sudo apt-get install -y --force-yes logdna-agent sudo logdna-agent -k 6a7b7c622290d1a49bbe5a94dc6828d Haven't tried this yet but the install instructions don't include adding a GPG key. Instead it has the option "--force-yes" which I believe skips that check. That would work for the initial install but wouldn't it complain down the road when you try to update the package?
- leeab 11y agoNo, sorry, I just didn't have time to set this up. We'll get this before the updates are made.
- circuitpeople 11y agoAWS SNS -> Lambda -> Firehose -> S3 | Redshift Done.
- rendambathu 11y agoAnything other than SNS? Within AWS or outside of it?
- guillegette 11y agoITT: when your "Show HN link" turns into your competitors list
- rgacote 11y agoCan you tell me how wide I have to open our Firewalls to get to your servers? (i.e., 1 or 2 IPs or an entire AWS range). We're in a locked-down compliance environment and opening all our servers to a wide range of external IP addresses is a problem. Would love to hear you have an aggregator/forwarding tool that I could run all my logs through to forward to your environment.
- doxcf434 11y agoI'd recommend making your pricing model simply per GB and then on a rate tier, don't play games with your customers. The quota system ends being a game, and simply ends up wasting management's time trying to ensure the company isn't either getting charged for overages or not using up the quote they have. I'd quite frankly not recommend companies like Sumologic solely based on the massive amounts of time they waste with these things.
- packetized 11y agoHow are you not hemorrhaging money by running this on top of AWS/RS/et al?
- packetized 11y agoFor what it's worth: I've recently built an internal logging platform that handles ~300GB/day on spare hardware with Heka/RMQ/ES/OTSDB/Kibana/Grafana. Running something similar from a SaaS logging provider would be (at bare minimum) a five digit monthly bill.
- askedrelic 11y agoLooks promising! I was able to get it installed and delivering logs in 5 mins. I really like the high volumes you offer; I really agree that volume should be cheaper. What Papertrail charges per GB seems really high IMO. That said, I'm just a hobbyist doing under 1GB month of traffic. I will try to stick with the free plan for as long as possible. If you could offer some nice features for $5-10/month, I would upgrade for that.
- tomkazarian 11y agoSet a canonical to https://logdna.com https://logdna.com to help with indexing/attribution issues of https://logdna.com/?ref=showhnshowhn https://logdna.com/?ref=showhnshowhn