9 ms·
How to talk to an open source project as a large scale or interesting user
- ibmthrowaway218 12y agoThis is one of the pitfalls of a public only support mailing list. I know I post using a gmail psuedonym to get support for various open source apps that I use for side projects. I'd be looking to provide an extra private/internal support email address made available to the largest customers.
- ciupicri 12y agoExcept that even the basic support [1] includes that: > Support via private (email) ticketing system, with guaranteed 8 hour response time, during European office hours (9AM – 6PM, or 3AM – noon EST). Includes resolution of 5 incidents. Does not include DNSSEC support. although CloudFlare should have probably bought the carrier grade support. [1]: https://www.powerdns.com/support.html https://www.powerdns.com/support.html
- otoburb 12y agoExactly correct. This implies that Cloudflare was using PowerDNS and (as was their right) chose not to pay for even basic support. I'm not sure what else PowerDNS could have done in this situation, and I can only imagine their teeth gnashing when they found they'd lost a lighthouse user. Typically, lighthouse customers almost always also pay for support. Upon reflection, their post-mortem blog post was extremely well handled and written.
- michaelt 12y agoWe in the computer business are well aware that some bug reports don't get fixed - such as because they require new features outside the scope of the project, or because fixing them is a huge hassle. A "carrier grade" support contract sounds expensive. If I'm going to my boss to ask for $x0,000 for a support contract because of a specific bug we're experiencing, I need to know the resolution won't be "WONTFIX"
- otoburb 12y agoThose are indeed common issues in our industry. However, in this specific case, PowerDNS specifically calls this out on their support section of their page: "Many (large) developments in PowerDNS have been funded by operators that wanted to expedite the authoring of the features they needed."[1] Also, carrier grade support does indeed sound expensive for a small shop. Might even be deemed expensive for a medium-large shop (since they don't list pricing details). However, for an outfit as large as CloudFlare, I find it hard pressed to believe they couldn't afford Basic Support considering that one of their engineers on this thread stated that DNS is a critical part of CloudFlare's core service offering. You don't ask for a support contract for a specific bug, but for a general class of support and current + future bugs. As part of the negotiation, your boss and organization can also leverage "lighthouse" status to influence PowerDNS to fix even minor bugs quickly as part of a negotiated Service Level Agreement ("SLA"). EDIT: To give CloudFlare the benefit of the doubt, it is possible that PowerDNS didn't make their support options clear to users before, and only updated their support pages after this post-mortem. [1] https://www.powerdns.com/enhancements.html https://www.powerdns.com/enhancements.html
- smackfu 12y agoAnd no prices on that page at all, so you need to contact them, and then you are dealing with a salesperson who sees CloudFare as a whale who needs to buy the $$$ carrier-grade support and get them a big commission.
- jfindley 12y agoThis is sadly such a common issue these days. Please give some idea of your pricing! Not providing any pricing information at all on your website is a massive disincentive to paying for your services. I don't want to have to deal with sales people who will continually spam me with email and phone calls if I ask for pricing, and will want to have a 20 minute call to "understand my requirements". By all means put an asterisk saying that the prices are guides only, or have large letters saying discounts may apply, or whatever, if you must. But not doing anything doesn't just aggravate your customers - it often means we go elsewhere or just use mailing lists for support.
- danielweber 12y agoIf the customer would rather build their own system than risk talking to someone on the phone where money might be discussed, I'm not sure that the open-source vendor is the problem.
- smackfu 12y agoThe contention was that Cloudfare could have just signed up for the cheap support plan if they wanted private problem reporting, and I don't think that would be as easy as was suggested.
- davidw 12y agoIt's not that hard to dig around and find someone you can email privately. For many open source projects, that can be kind of annoying, but if it's really, honestly, something that's kind of private, or if there's money involved, the developers are probably going to be more understanding.
- forgottenpass 12y agoThis circumstance would be amusing to me if it wasn't way too common. People that are employed in or close to software development are well aware that when communication with user/clients/whatever is complicated. Incomplete and misinformed interpretation of requirements are the poster child of difficulty to discuss software. Then a developer turns around and fails to communicate with the people that develop the software they use. For all the huffing and puffing I see people do about professionalism, very few of the "professional" mailing lists I'm subscribed to have adults that can say "I'm A at B, and am looking for a contact at C to email me off-list to discuss D."
- chris_wot 12y agoCurious to understand what the issue was.
- jgrahamc 12y agoI work on our DNS server, but I don't have a copy of the email that Matthew sent to the PowerDNS guys, but DNS is fundamental to what we do and so the ability to modify the DNS server was pretty important. I think we got to the point where we just said "there are N features that we want" and decided to build our own. The result is that we've been able to do all sorts of stuff that's CloudFlare-specific: bear in mind that we handle two different sorts of DNS queries: external ones (that might or might not serve a CloudFlare IP that is proxying for a customer site) and internal ones (that need access to public, non-CloudFlare IPs). We also recently introduced a CNAME-flattening feature: http://blog.cloudflare.com/introducing-cname-flattening-rfc-compliant-cnames-at-a-domains-root http://blog.cloudflare.com/introducing-cname-flattening-rfc-... There are other things we want to do to be fast in two ways: fast to respond (http://www.solvedns.com/dns-comparison/2014/04 http://www.solvedns.com/dns-comparison/2014/04) and fast to update (which means integration with our replicated, global settings databases). We also have all sorts of special things that we do to deal with attacks on our DNS infrastructure that include the DNS server talking directly to NICs for filtering and packet handling. We're also generating bpf bytecode directly for specific filtering activities. Bottom line was that DNS is a core technology for CloudFlare; it made sense to totally own it. I don't think this says anything bad about PowerDNS. Note: HTTP serving is a core technology. If nginx weren't so configurable (especially through OpenResty) we could easily have been having this discussion about a web server. But equally agentzh (who wrote OpenResty) works for us.
- lugg 12y ago> There were three main challenges. First, PowerDNS when under large in-bound DDoS loads would consume excessive resources and occasionally fall over. Second, and more insidious, because the version of PowerDNS we were using did not have abuse detection and rate limiting, we were increasingly seeing attempts at using CloudFlare authoritative DNS network to launch reflection attacks against others. And, third, it was very difficult to extend PowerDNS and add any sort of application logic which kept us from doing interesting things... like adding Easter Eggs. Source (was linked in article): http://blog.cloudflare.com/the-story-of-a-little-dns-easter-egg http://blog.cloudflare.com/the-story-of-a-little-dns-easter-...
- sambeau 12y agoOFF-TOPIC FOR ENGLISH-AS-SECOND-LANGUAGE: For HN'ers who are still learning the finer points of English this title shows one of the many weird inconsistencies of the language. Normally, when trying to say 'a thing' where the thing starts with a vowel we change the 'a' into 'an' so 'a axe' becomes 'an axe', 'a umbrella' becomes 'an umbrella'. However this is a notable exception: when pronouncing 'user' there is an unwritten 'y' e.g. it is said 'yoozer', thus we wouldn't normally add another consonant here so it should be 'a user' not 'an user'. If you say it out loud you can hear it (and hopefully feel the extra tongue work that, traditionally, we try to avoid). Another common exception is 'Hotel'. It takes 'an' rather than the expected 'a' as for some reason we drop the 'H'. For me it rolls better off the tongue with 'an' but this may purely be due to conditioning. 'Herb' is a difficult one as in the US it's pronounced 'erb' while in the UK it has a hard 'H'. Thus in the UK we use 'A herb' — I assume the US says 'an Herb' — correct me if I'm wrong. It's nonsense like this that makes me pity you brave fellows who attempt to master our pidgin of a language! Edit: change 'pigeon' to 'pidgin' :)
- krallja 12y ago> an herb Yep.
- bullsbarry 12y agoI wouldn't be so sure about the 'an' before hotel. That may be regional as well. I'm from the southern US and use 'a' there. However, 'an' would be used whenever the 'h' isn't voiced, which in my case would be words like honorable and honest.
- madeofpalk 12y ago> Another common exception is 'Hotel'. It takes 'an' rather than the expected 'a' as for some reason we drop the 'H'. For me it rolls better off the tongue with 'an' but this may purely be due to conditioning. I'm not sure about the rest of the world, but in Australia I don't think I've ever heard 'an hotel'. It's always 'I'm going to a hotel' Saying 'an Hotel' makes me pronounce 'Hotel' in a very uncomfortable way.
- 12y ago
- gilrain 12y agoTo complain that the only reason you gave someone bad support was because you didn't know who they were is pretty tone deaf. Would you want to dine somewhere that got panned by a restaurant critic and then wrote an essay saying, "Well, we'd have been a lot more careful with his food if we'd known he was an important critic!"?
- ahubert 12y agoI suggest actually _reading_ the post. It notes the tradeoff. It also notes we offer near infinite support for people asking politely and making their problems known. We also note that it isn't just a matter of scale.
- octo_t 12y agoYour analogy is flawed. This is like a restaurant critic saying the portions were far too small, but failing to reveal when they ordered that it wasn't just for one person, but actually a party of 12 people.
- gilrain 12y agoAh, thanks, that does help me understand the other perspective.
- jwheeler79 12y agohis analogy is 100% spot on actually in how i interpreted the article. 100%.
- markbnj 12y agoHmmm, I'd agree if the issues were all performance-related, or if the author had said "make sure to let us know if you are a large deployment." What the author actually said was "large or impressive" and that point was reinforced several times. I don't know how you measure "impressive" but it sounds a lot like "special" to me.
- octo_t 12y agoI imagine if you were using their product in an HFT setup (somehow) with only a few servers, but with sub millisecond requirements they might also be interested.
- roberthahn 12y agoWhat I'm seeing here is a customer support problem. And IMHO, PowerDNS isn't taking the correct steps to solve it. It's funny, because the answer is right in the article. If your customer: * is large and/or interesting * has a legal/security/marketing/ team that's worried about sharing information publicly * has an IT dept that doesn't know the whole company is depending on a single open source product * feels they have to use gmail accounts and made-up names to talk to you Then maybe a public forum really isn't the place to expect your customers to ask for help. I would suggest creating a partnership program for businesses like Cloudflare - swap tech support for publicity (which will then help drive adoption)
- gregd 12y agoI would agree. Giving away a good portion of your infrastructure in a public forum and asking for configuration help, IS just asking for trouble. The "large and interesting" deployment line was stated and reiterated several times as being "important" to the level of help that you'll receive from them. I fail to understand HOW this will garner more users to your service.
- ForHackernews 12y ago> I fail to understand HOW this will garner more users to your service. Through PR/publicity. "Used by Google, Amazon, Facebook, etc." makes an impressive case when you're choosing which project to use.
- gregd 12y agoWhat I was referring to was using this service if you're NOT Google, Amazon or Facebook. Nothing discourages me more from using a service if you differentiate the type of customer support you provide to a client based on their size or whether they have an "interesting" setup. In the grand scheme of things, Google, Amazon and Facebook are less normal, so if you're tailoring your support model around those types of clients, then it's not for me...
- 12y ago
- baudehlo 12y agoI have definitely had this issue with Haraka (a highly scalable SMTP server). On the one hand we have Craigslist as a user, who have been incredibly open and contributed heavily to the project, and on the other we have had people post cryptic bug reports that it didn't work for them, but clearly they were trying to use it at an interesting scale, and we would love to have helped. We pride ourselves on being one of the friendliest MTA communities out there, so to find people who give up and quit using it is always sad. Many times we've contributed custom code to companies for free to help them out, but we can't do that if you are silent. So speak up, companies, even if it's only in private messages.
- Mikushi 12y agoOn paper I agree, from the other side of the fence, here is what the process looks like: This looks good for what we want to do, let's test it out. Cryptic error. Google for 24h for a solution, can't find one. Ask on official mailing list / forum / bug tracker. Getting ignored as usual, or dismissed rudely. Never use that product ever again. I'm a CTO and I do that process more often that I would want to, sometimes I don't get that far because Install instruction sucks, and usually that's enough to deter me. A concrete example: Since The Guardian opened Scribe (their CMS WYSIWYG editor) I had it in the back of my mind for weeks because it looked really interesting and close to what I had been looking for. We were doing CMS upgrades recently, so I wanted to take the occasion to plug in Scribe. It took only 2h for me to put Scribe on my "never use ever again" list. It all look pretty neat at glance, quick install, nice plugin system, yadda yadda,... However, their one line install simply doesn't work, their is no list of dependencies anywhere, and simply no documentation. Logged an issue notifying them of the poor quality of their setup instructions. Got a reply 7 days later, which starts a discussion. But now it's too late, still have personal interest for Scribe, however my company has already moved on. I'm all for OpenSource, have some myself (though, more of a "here is what I did recently" type of guy), however if you want to pretend to be a serious library/framework/piece of software, that includes decent documentation with clear install instruction for most common distributions. Plus, dialogue is good, but most of the time I need it right now, or in a very short time frame, so more often that not I will simply drop back to another less interesting piece of library (or nasty one) that simply works, because that is what matters, it works when I need it. (Again with the Scribe example, no matter how much I dislike TinyMCE, I ended up using it, because one line of JavaScript is all it took for it to work).
- doxcf434 12y agoOften when a company want's to "help" it means make a sale regardless of how poor the technical implementation is, waste your time with SE level support, and power point presentations. When instead they should be focusing on the product first. If your user is reporting a scaling issue and your product is all about scale, that should be your first priority. Vendors also need to remember, your users change companies, a user could be at some small unknown company on day, and the next be in control of a large budget at a high profile company. Your reputation follows that user, and what they want to see is you're making a good product. If the smallest company is reporting an issue that would affect your customers, I'd want to see that you're addressing the issue with that company so I don't have to at my large/interesting company.
- dmourati 12y agoIt is impossible to comment on this specific example without seeing the actual email exchange. What I can say, is that a user chooses how to disclose their identity and the identity of their company making the best analysis of an imperfect situation. Having worked for largely smaller companies (not in the business of protecting the global internet), I've been relatively lax in posting who I am and where I work. This transparency has been helpful on more than a few occasions. If you are a "large and/or interesting" company, it would behoove you to disclose your identity in a safe and responsible way to projects (open source or otherwise) who represent any important part of your business. The trick is to do so in such a manner so as to derive the many benefits (increased responsiveness) without the many liabilities (spam, sales calls, etc). By remaining mum about who you are, the signal to noise ratio of your requests do not stand out from the crowd. This is unfortunate for both sides. It does, however, reinforce the notion that companies and projects should aspire to provide excellent support to all users. You never know when you have a "secret shopper" in your midst.
- philjackson 12y agoI read the first part of that quote as "White PowerDNS" and wondered if I was dreaming.
- pilif 12y agotangentially related: Thanks for once again pointing to PowerDNS. I've been using bind forever and I never really looked out for other alternatives ("it works - why should I bother learning something new?"). I had no idea what I was missing as from now on, I'll never again have to deal with forgotten serials or manually adding slave zones to the secondary DNS server. Everybody still running with bind, thinking it's "good enough", ask yourself how many times you were inconvenienced by either of the two issues and decide whether it might be worth looking at alternatives. The fact that PowerDNS apparently also is really performant doesn't matter for my mostly minimal use case, but it's good to know that if something great happens, I won't have to migrate again (not that migration is hard - powerdns comes with a helper tool to read your bind config and produce SQL for your zone files)
- dmourati 12y agoMay I recommend the excellent "Alternative DNS Servers" by JP Mens: http://www.amazon.com/Alternative-DNS-Servers-Deployment-Back-Ends/dp/0954452992/ref=sr_1_1?ie=UTF8&qid=1399390889&sr=8-1&keywords=dns+book+mens http://www.amazon.com/Alternative-DNS-Servers-Deployment-Bac...
- aktau 12y agoMay I recommend LuaDNS [1] for simple uses? Write your DNS config in lua and deploy it with git push, I like it a lot for all my side projects and my homepage. [1]: http://www.luadns.com/ http://www.luadns.com/
- EGreg 12y agoHow are DNS servers different from nameservers?
- jackgavigan 12y agoIt's interesting how the title "How to talk to an open source project as a large scale or interesting user" implies that the onus is on the user to approach the open source project in the correct manner. It might be more productive to think about "How an open source project should talk to a large scale or interesting user". I was deploying open source software in large corporations more than a decade ago[1] and I formed the impression that open source projects could do a LOT more to engage with and support their users. It sometimes felt like the developers just didn't care about users. Documentation was often sorely lacking[2] and "support" typically came in the form of mailing lists that were full of users trying to figure out how the software worked, and swapping advice and tips - often a case of the blind leading the blind. The end result was that corporate users would sometimes choose commercial software, despite the cost, because it came with decent support and/or had functionality that the open source alternative lacked. These days, I think that a product manager is an essential part of any open source project team. 1: e.g. http://mail-archives.apache.org/mod_mbox/tomcat-users/200009.mbox/%3COFF2F061DD.C2B7EB05-ON00256960.004C94C4%40srv.uk.deuba.com%3E http://mail-archives.apache.org/mod_mbox/tomcat-users/200009... 2: e.g. https://www.mail-archive.com/linux-raid@vger.rutgers.edu/msg00947.html https://www.mail-archive.com/linux-raid@vger.rutgers.edu/msg...
- SoftwareMaven 12y agoWhile I agree with you in general, after all, what's the point of building something if people can't use it, it misses the point of this article: if you have an interesting use case, particularly one that aligns very well with the goals of the project, you need to let the project know somehow. The developers are not psychic.
- x0x0 12y agoyour comments are, bluntly, massively entitled. One of the open source tradeoffs is that they often come with slightly more install/configuration work; in trade for usage freedom, the ability to modify the source, and of course, zero required cash outlay. I share ml code on cran because I wrote it and I'm happy to have other people use it; that doesn't come with a requirement for me to spend more than zero time helping them do so and when I do, it's because I happen to feel like it. Telling me you're doing something interesting (to me!) will often help interest me in helping you. The massive wave of demanding users that want hand holding typically don't even get a reply from me.
- alexdowad 12y agoNo sympathy. If your support for users is poor, then it is simply poor, and no excuses please. "We didn't know how big they were" doesn't cut the mustard.