6 ms·

>Being desperate we decided to seek support from the Internet and posted our findings to the dm-crypt mailing list When I see a company such as CloudFlare bein

>Being desperate we decided to seek support from the Internet and posted our findings to the dm-crypt mailing list When I see a company such as CloudFlare being so transparent about their difficulties, and trying to find an answer using their community members, it makes me love them even more. No ego, pure Professionalism

sneak 7y ago
Correspondingly, the response they received reflects just as strongly on the community itself.
- hyper_reality 7y ago
  Yep, the response they received was incredibly condescending. The follow-up from Cloudflare remained polite and added a lot more data, and was ignored. It's a shame because I've seen this condescending attitude quite frequently in the crypto open source community, and am not really sure how it arises. At least in this case it seems to have had the good outcome of motivating Cloudflare to dig in deeper and solve the problem by themselves.
  - dr_zoidberg 7y ago
    I'd say it comes from sitting at an ivory tower and giving 0 ducks about who you're talking to. In this case, the person/team asking was capable enough to go on their own, dig, test, change, and find a fix. It could've probably been easier for them if given proper directions. OTOH, perhaps those that answered from atop the tower had little idea of the mechanisms that the author(s?) dug out and changed. So double shame on them, for being condescending and not knowing. And it also falls on ourselves to be mindful of this behaviour, that can creep up on us without knowing. We sometimes think our time is super valuable and we don't have to spend it on some "newbie question" or this guy who doesn't understand. The past year I've been mentoring grad students in the lab I work at, and found myself once or twice going this route. I luckily caught it early, took a deep breath and gave them the time and explanations they needed. In the end I got a few nice surprises out of two amazing students, who were seeing a bit beyond what was evident.
    - belorn 7y ago
      I see many large project addressing this issue by not having a public list where you can talk directly to the developers. Instead there is user lists which public relation managers maintain and where the expected result from the original mail would either be no response or a polite and nicely written one about testing the user configuration options that they had already tested. That way the developer would not need to respond unless the user has shown enough proof of work to demonstrate to the public relation managers that the issue should be forwarded to a developer, in which case the answer the developer would reply with would be under the assumption that the person/team asking is capable enough to use the directions to dig, test, change, and create a patch which then later might be added to the project.
  - megous 7y ago
    Full response: https://www.spinics.net/lists/dm-crypt/msg07517.html https://www.spinics.net/lists/dm-crypt/msg07517.html From the PoV of the person who responded they didn't provide any relevant information that would indicate what platform they run, or what speed they expect, or why they think 800MiB/s seems slow to them. On many platforms this would be a pretty good result. At first look, it looks like they expected the speed of unencrypted storage, because that's what they tried to compare against. So the response seems reasonable at first glance to me. They got the answer to their main question. (which they omitted from their blog article)
    - gravitas 7y ago
      I disagree with your assessment. > If the numbers disturb you, then this is from lack of understanding on your side. This is arrogance on the part of the person replying that hand-waved away their problem "you just don't understand" when in fact, they (Cloudflare) do/did understand. They then went on to prove that it was due to queuing within the kernel, not the hardware as commented by this person in their flippant reply.
      - megous 7y ago
        Cloudflare did not understand at the time. Anyway, I'm not questioning that the reply was not very helpful, I just don't see it as unreasonable. I liked the technical parts of the CF writeup overall.
        marcinzm 7y ago
        >Cloudflare did not understand at the time And neither did the people who responded to them it seems.
        why_only_15 7y ago
        The next line after that is "You are probably unaware that encryption is a heavy-weight operation". If you're posting to the dm-crypt mailing list about encryption performance, you're probably very aware that encryption is a heavy-weight operation.
  - _jal 7y ago
    Honestly, read the original message and consider how you would have replied. They showed work in a vacuum - demonstrated that dm-crypt has costs over raw device access (I would hope so!) on some unknown hardware, and then asks "does this look right to you?" Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. People whine about technical mailing lists, I think because they don't get the context. Think of them as sort of like water coolers at an office that specializes in whatever the list is about. You get a short slice of expert attention in between doing whatever they actually have to get done. Throwing a bunch of data on the floor and saying "hey, is this expected?" is not going to work well. Seriously, what were they expecting?
    - rrss 7y ago
      > Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. It's entirely possible to say both these things in a much more constructive and less condescending tone than was used.
      - _jal 7y ago
        People are so very weirdly sensitive to these things when it is a big company that comes calling. Wonder why that is. Context matters. If you don't take the time to understand the context you're walking in to and don't follow local rules, don't be surprised if people are rude to you back. Not that I even think what they said was all that rude. Do you also think you can slide in to a gam3r chat and expect business etiquette?
        manigandham 7y ago
        I don't think a crypto mailing list is the same as "gam3r" subreddit, but it wasn't that rude overall. It was the tone that they "dont understand" when in fact Cloudflare does understand crypto and performance very well, and went so far as to dive into kernel code and submit patches that fixed the problem that others didnt even realize existed. Even so I agree this isn't worth such a big discussion.
- ezoe 7y ago
  And according to the cloudflare, the current dm-crypt implementation is a horrible bit rot which got no reviewing for 15 years.
- crest 7y ago
  Especially because my old Haswell-E workstation running FreeBSD has no problem maxing out four encrypted SATA SSDs (>500MB/s each) at the same time with AES-NI. There is no excuse for slow cipher implementations and the queuing sounds insane saving and restoring the SSE registers can't be expensive enough to justify all those context switches between kernel threads.
- pengaru 7y ago
  It's a public mailing list, I see zero upstream kernel commits from arno@* so it doesn't appear the response came from someone who actually knows and works with the dm-crypt code. I'm on a number of public mailing lists and there's often a participant who tends to be both available/communicative and callous in their communication style. My assumption is there's a filter effect going on here where some folks who have very poor social abilities wind up at their computer alone all the time and public mailing lists become part of their few remaining human interactions. What I'd take away from this particular dm-crypt interaction isn't that the community is assholes, but that the community is small and the mailing list poorly attended/inactive. In the past I've reported my own dm-crypt problems upstream and it took years to get a bisected regression reverted. Just getting relevant people to pay attention was a challenge.
peterwwillis 7y ago
You won't know about their ego or professionalism until you work with them. Posting on a mailing list and making a blog post about it is not proof of either of these, it's brand marketing. They're trumpeting their engineering talent to build good will/nerd rep so people will love their company, spend money there, and apply for jobs. (But what it does show is that they're good at marketing, because it's working)
- brnt 7y ago
  Sounds like they're also pretty good at making disk encryption faster.
- darkwater 7y ago
  It is also marketing for sure, but it is well done marketing. It's like having a high PageRank in Google because you actually write meaningful, useful, well-written blog posts which Google happens (happened) to value vs link factory blog posts.
- the_duke 7y ago
  Writing informative blog posts and publishing patches to the Linux kernel is the rare kind of marketing I can 100% support.

sneak 7y ago

Correspondingly, the response they received reflects just as strongly on the community itself.

hyper_reality 7y ago
Yep, the response they received was incredibly condescending. The follow-up from Cloudflare remained polite and added a lot more data, and was ignored. It's a shame because I've seen this condescending attitude quite frequently in the crypto open source community, and am not really sure how it arises. At least in this case it seems to have had the good outcome of motivating Cloudflare to dig in deeper and solve the problem by themselves.
- dr_zoidberg 7y ago
  I'd say it comes from sitting at an ivory tower and giving 0 ducks about who you're talking to. In this case, the person/team asking was capable enough to go on their own, dig, test, change, and find a fix. It could've probably been easier for them if given proper directions. OTOH, perhaps those that answered from atop the tower had little idea of the mechanisms that the author(s?) dug out and changed. So double shame on them, for being condescending and not knowing. And it also falls on ourselves to be mindful of this behaviour, that can creep up on us without knowing. We sometimes think our time is super valuable and we don't have to spend it on some "newbie question" or this guy who doesn't understand. The past year I've been mentoring grad students in the lab I work at, and found myself once or twice going this route. I luckily caught it early, took a deep breath and gave them the time and explanations they needed. In the end I got a few nice surprises out of two amazing students, who were seeing a bit beyond what was evident.
  - belorn 7y ago
    I see many large project addressing this issue by not having a public list where you can talk directly to the developers. Instead there is user lists which public relation managers maintain and where the expected result from the original mail would either be no response or a polite and nicely written one about testing the user configuration options that they had already tested. That way the developer would not need to respond unless the user has shown enough proof of work to demonstrate to the public relation managers that the issue should be forwarded to a developer, in which case the answer the developer would reply with would be under the assumption that the person/team asking is capable enough to use the directions to dig, test, change, and create a patch which then later might be added to the project.
- megous 7y ago
  Full response: https://www.spinics.net/lists/dm-crypt/msg07517.html https://www.spinics.net/lists/dm-crypt/msg07517.html From the PoV of the person who responded they didn't provide any relevant information that would indicate what platform they run, or what speed they expect, or why they think 800MiB/s seems slow to them. On many platforms this would be a pretty good result. At first look, it looks like they expected the speed of unencrypted storage, because that's what they tried to compare against. So the response seems reasonable at first glance to me. They got the answer to their main question. (which they omitted from their blog article)
  - gravitas 7y ago
    I disagree with your assessment. > If the numbers disturb you, then this is from lack of understanding on your side. This is arrogance on the part of the person replying that hand-waved away their problem "you just don't understand" when in fact, they (Cloudflare) do/did understand. They then went on to prove that it was due to queuing within the kernel, not the hardware as commented by this person in their flippant reply.
    - megous 7y ago
      Cloudflare did not understand at the time. Anyway, I'm not questioning that the reply was not very helpful, I just don't see it as unreasonable. I liked the technical parts of the CF writeup overall.
      - marcinzm 7y ago
        >Cloudflare did not understand at the time And neither did the people who responded to them it seems.
      - why_only_15 7y ago
        The next line after that is "You are probably unaware that encryption is a heavy-weight operation". If you're posting to the dm-crypt mailing list about encryption performance, you're probably very aware that encryption is a heavy-weight operation.
- _jal 7y ago
  Honestly, read the original message and consider how you would have replied. They showed work in a vacuum - demonstrated that dm-crypt has costs over raw device access (I would hope so!) on some unknown hardware, and then asks "does this look right to you?" Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. People whine about technical mailing lists, I think because they don't get the context. Think of them as sort of like water coolers at an office that specializes in whatever the list is about. You get a short slice of expert attention in between doing whatever they actually have to get done. Throwing a bunch of data on the floor and saying "hey, is this expected?" is not going to work well. Seriously, what were they expecting?
  - rrss 7y ago
    > Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. It's entirely possible to say both these things in a much more constructive and less condescending tone than was used.
    - _jal 7y ago
      People are so very weirdly sensitive to these things when it is a big company that comes calling. Wonder why that is. Context matters. If you don't take the time to understand the context you're walking in to and don't follow local rules, don't be surprised if people are rude to you back. Not that I even think what they said was all that rude. Do you also think you can slide in to a gam3r chat and expect business etiquette?
      - manigandham 7y ago
        I don't think a crypto mailing list is the same as "gam3r" subreddit, but it wasn't that rude overall. It was the tone that they "dont understand" when in fact Cloudflare does understand crypto and performance very well, and went so far as to dive into kernel code and submit patches that fixed the problem that others didnt even realize existed. Even so I agree this isn't worth such a big discussion.
ezoe 7y ago
And according to the cloudflare, the current dm-crypt implementation is a horrible bit rot which got no reviewing for 15 years.
crest 7y ago
Especially because my old Haswell-E workstation running FreeBSD has no problem maxing out four encrypted SATA SSDs (>500MB/s each) at the same time with AES-NI. There is no excuse for slow cipher implementations and the queuing sounds insane saving and restoring the SSE registers can't be expensive enough to justify all those context switches between kernel threads.
pengaru 7y ago
It's a public mailing list, I see zero upstream kernel commits from arno@* so it doesn't appear the response came from someone who actually knows and works with the dm-crypt code. I'm on a number of public mailing lists and there's often a participant who tends to be both available/communicative and callous in their communication style. My assumption is there's a filter effect going on here where some folks who have very poor social abilities wind up at their computer alone all the time and public mailing lists become part of their few remaining human interactions. What I'd take away from this particular dm-crypt interaction isn't that the community is assholes, but that the community is small and the mailing list poorly attended/inactive. In the past I've reported my own dm-crypt problems upstream and it took years to get a bisected regression reverted. Just getting relevant people to pay attention was a challenge.

peterwwillis 7y ago

You won't know about their ego or professionalism until you work with them. Posting on a mailing list and making a blog post about it is not proof of either of these, it's brand marketing. They're trumpeting their engineering talent to build good will/nerd rep so people will love their company, spend money there, and apply for jobs. (But what it does show is that they're good at marketing, because it's working)

brnt 7y ago
Sounds like they're also pretty good at making disk encryption faster.
darkwater 7y ago
It is also marketing for sure, but it is well done marketing. It's like having a high PageRank in Google because you actually write meaningful, useful, well-written blog posts which Google happens (happened) to value vs link factory blog posts.
the_duke 7y ago
Writing informative blog posts and publishing patches to the Linux kernel is the rare kind of marketing I can 100% support.