13 ms·

Speeding up Linux disk encryption

491 points by jgrahamc 7y ago

thereyougo 7y ago
>Being desperate we decided to seek support from the Internet and posted our findings to the dm-crypt mailing list When I see a company such as CloudFlare being so transparent about their difficulties, and trying to find an answer using their community members, it makes me love them even more. No ego, pure Professionalism
- sneak 7y ago
  Correspondingly, the response they received reflects just as strongly on the community itself.
  - hyper_reality 7y ago
    Yep, the response they received was incredibly condescending. The follow-up from Cloudflare remained polite and added a lot more data, and was ignored. It's a shame because I've seen this condescending attitude quite frequently in the crypto open source community, and am not really sure how it arises. At least in this case it seems to have had the good outcome of motivating Cloudflare to dig in deeper and solve the problem by themselves.
    - dr_zoidberg 7y ago
      I'd say it comes from sitting at an ivory tower and giving 0 ducks about who you're talking to. In this case, the person/team asking was capable enough to go on their own, dig, test, change, and find a fix. It could've probably been easier for them if given proper directions. OTOH, perhaps those that answered from atop the tower had little idea of the mechanisms that the author(s?) dug out and changed. So double shame on them, for being condescending and not knowing. And it also falls on ourselves to be mindful of this behaviour, that can creep up on us without knowing. We sometimes think our time is super valuable and we don't have to spend it on some "newbie question" or this guy who doesn't understand. The past year I've been mentoring grad students in the lab I work at, and found myself once or twice going this route. I luckily caught it early, took a deep breath and gave them the time and explanations they needed. In the end I got a few nice surprises out of two amazing students, who were seeing a bit beyond what was evident.
      - belorn 7y ago
        I see many large project addressing this issue by not having a public list where you can talk directly to the developers. Instead there is user lists which public relation managers maintain and where the expected result from the original mail would either be no response or a polite and nicely written one about testing the user configuration options that they had already tested. That way the developer would not need to respond unless the user has shown enough proof of work to demonstrate to the public relation managers that the issue should be forwarded to a developer, in which case the answer the developer would reply with would be under the assumption that the person/team asking is capable enough to use the directions to dig, test, change, and create a patch which then later might be added to the project.
    - megous 7y ago
      Full response: https://www.spinics.net/lists/dm-crypt/msg07517.html https://www.spinics.net/lists/dm-crypt/msg07517.html From the PoV of the person who responded they didn't provide any relevant information that would indicate what platform they run, or what speed they expect, or why they think 800MiB/s seems slow to them. On many platforms this would be a pretty good result. At first look, it looks like they expected the speed of unencrypted storage, because that's what they tried to compare against. So the response seems reasonable at first glance to me. They got the answer to their main question. (which they omitted from their blog article)
      - gravitas 7y ago
        I disagree with your assessment. > If the numbers disturb you, then this is from lack of understanding on your side. This is arrogance on the part of the person replying that hand-waved away their problem "you just don't understand" when in fact, they (Cloudflare) do/did understand. They then went on to prove that it was due to queuing within the kernel, not the hardware as commented by this person in their flippant reply.
        megous 7y ago
        Cloudflare did not understand at the time. Anyway, I'm not questioning that the reply was not very helpful, I just don't see it as unreasonable. I liked the technical parts of the CF writeup overall.
        marcinzm 7y ago
        >Cloudflare did not understand at the time And neither did the people who responded to them it seems.
        why_only_15 7y ago
        The next line after that is "You are probably unaware that encryption is a heavy-weight operation". If you're posting to the dm-crypt mailing list about encryption performance, you're probably very aware that encryption is a heavy-weight operation.
    - _jal 7y ago
      Honestly, read the original message and consider how you would have replied. They showed work in a vacuum - demonstrated that dm-crypt has costs over raw device access (I would hope so!) on some unknown hardware, and then asks "does this look right to you?" Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. People whine about technical mailing lists, I think because they don't get the context. Think of them as sort of like water coolers at an office that specializes in whatever the list is about. You get a short slice of expert attention in between doing whatever they actually have to get done. Throwing a bunch of data on the floor and saying "hey, is this expected?" is not going to work well. Seriously, what were they expecting?
      - rrss 7y ago
        > Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. It's entirely possible to say both these things in a much more constructive and less condescending tone than was used.
        _jal 7y ago
        People are so very weirdly sensitive to these things when it is a big company that comes calling. Wonder why that is. Context matters. If you don't take the time to understand the context you're walking in to and don't follow local rules, don't be surprised if people are rude to you back. Not that I even think what they said was all that rude. Do you also think you can slide in to a gam3r chat and expect business etiquette?
        manigandham 7y ago
        I don't think a crypto mailing list is the same as "gam3r" subreddit, but it wasn't that rude overall. It was the tone that they "dont understand" when in fact Cloudflare does understand crypto and performance very well, and went so far as to dive into kernel code and submit patches that fixed the problem that others didnt even realize existed. Even so I agree this isn't worth such a big discussion.
  - ezoe 7y ago
    And according to the cloudflare, the current dm-crypt implementation is a horrible bit rot which got no reviewing for 15 years.
  - crest 7y ago
    Especially because my old Haswell-E workstation running FreeBSD has no problem maxing out four encrypted SATA SSDs (>500MB/s each) at the same time with AES-NI. There is no excuse for slow cipher implementations and the queuing sounds insane saving and restoring the SSE registers can't be expensive enough to justify all those context switches between kernel threads.
  - pengaru 7y ago
    It's a public mailing list, I see zero upstream kernel commits from arno@* so it doesn't appear the response came from someone who actually knows and works with the dm-crypt code. I'm on a number of public mailing lists and there's often a participant who tends to be both available/communicative and callous in their communication style. My assumption is there's a filter effect going on here where some folks who have very poor social abilities wind up at their computer alone all the time and public mailing lists become part of their few remaining human interactions. What I'd take away from this particular dm-crypt interaction isn't that the community is assholes, but that the community is small and the mailing list poorly attended/inactive. In the past I've reported my own dm-crypt problems upstream and it took years to get a bisected regression reverted. Just getting relevant people to pay attention was a challenge.
- peterwwillis 7y ago
  You won't know about their ego or professionalism until you work with them. Posting on a mailing list and making a blog post about it is not proof of either of these, it's brand marketing. They're trumpeting their engineering talent to build good will/nerd rep so people will love their company, spend money there, and apply for jobs. (But what it does show is that they're good at marketing, because it's working)
  - brnt 7y ago
    Sounds like they're also pretty good at making disk encryption faster.
  - darkwater 7y ago
    It is also marketing for sure, but it is well done marketing. It's like having a high PageRank in Google because you actually write meaningful, useful, well-written blog posts which Google happens (happened) to value vs link factory blog posts.
  - the_duke 7y ago
    Writing informative blog posts and publishing patches to the Linux kernel is the rare kind of marketing I can 100% support.
beagle3 7y ago
Ages ago I benchmarked truecrypt overhead on my machine at the time (2006, I think?) and it was about 3%; I assumed that's a reasonable and still applicable number, also do dm-crypt and modern VeraCrypt. Guess I was get gradually more wrong through those years, according to the git archeology....
- singlow 7y ago
  Also, disk speed in 2006 was probably much slower. Disks have gotten faster at a greater pace than processors during the last 10 years.
vletal 7y ago
Has anyone already tried to compile the kernel with these patches for their desktop/laptop with encrypted drive? https://github.com/cloudflare/linux/tree/master/patches https://github.com/cloudflare/linux/tree/master/patches
- asymptotically2 7y ago
  Yes, I'm running them on kernel 5.5.13 (which came out today)
  - 3r8riacz 7y ago
    Wow, I'd be so happy if you could share the steps you took to achieve this. Let's say I have a Debian machine, how could I try it out?
    - asymptotically2 7y ago
      I use Gentoo Linux so it was as easy as putting the patches in a directory, and then rebuilding the kernel with the option to enable the synchronous cipher. For Debian it's almost as easy: https://passthroughpo.st/patch-kernel-debian/ https://passthroughpo.st/patch-kernel-debian/
pmorici 7y ago
Interesting. One other thing they don't mention that I found interesting when doing my own digging on dmcrypt speeds a while back is that the 'cryptosetup benchmark' command is only showing the single core performance of each of those encryption algorithms. You can verify this by watching the processor load as it performs the benchmark. That lead me to find that if you have a Linux software RAID you can get much better performance by having 1 dmcrpyt volume per disk and then software RAID the dm devices instead of putting a single dmcrypt on top of the software RAID. Curious if that would stack performance wise with what they found here or if that just happened to help with the queuing issue they identified.
- mercora 7y ago
  i remember somewhat recently efforts to parallelize the work of dm-crypt where applicable had been merged. However, i guess having multiple separate encryption parameters and states (read: disks) leaves more opportunity for parallelization of the work especially if disk access patterns are not spread wide enough.
est31 7y ago
Wow those speed improvements are very neat. And an awesome blog post accompanying them. Prior to reading this, I've considered Linux disk encryption adding negligible latency because no HDDs/SSD can be fast enough for a CPU equipped with AES-NI, but that view has changed. Two questions: 1. are there any efforts to upstream them? 2. Invoking non-hw-accelerated AES decryption routines sounds quite expensive. Has it been tried out to save the FPU registers only if there is the need for decryption?
- knorker 7y ago
  At least your first question is answered in the article: Yes
- andyjpb 7y ago
  The existing Linux system is useful for hardware that does less than 200MB/s, so you should be fine with HDDs. Cloudflare is optimising for SSDs. They don't talk about latency: all their crypto benchmarks measure throughput. Near the end they hint at response time for their overall cache system but there's no detailed discussion of latency issues. The takeaway for me is that I'm OK with what's currently in Linux for the HDDs I use for my backups but I'd probably lose out if I encrypted my main SSD with LUKS. At the end of the article they say that they're not going to upstream the patches as they are because they've only tested them with this one workload. I'd also be interested to see a benchmark comparing SW AES with FPU-saving + HW AES. Unfortunately their post does not include stats for how often their proxy falls into the HW or SW implementations. Whatever those numbers are, I'd expect FPU-saving + HW AES to be somewhere in the middle.
  - jlgaddis 7y ago
    > The takeaway for me is that I'm OK with what's currently in Linux for the HDDs I use for my backups but I'd probably lose out if I encrypted my main SSD with LUKS. Yep, when building my latest workstation, I went with a pair of ("regular") SSDs (RAID1) for my data. Later, I decided to add an NVMe for the OS for the additional speed. I then went and encrypted all of the drives (via LUKS), however, which basically killed any additional performance I would've gotten from the NVMe drive. I would have been just fine as well off with only the SSDs and without the NVMe drive.
dependenttypes 7y ago
> one can only encrypt the whole disk with a single key You can still use partitions. > not all cryptographic algorithms can be used as the block layer doesn't have a high-level overview of the data anymore I do not really understand this. Which cryptographic algorithms can't be used? > Most common algorithms require some sort of block chaining to be secure Nowadays I would say that from these only CTR is common, which does not require chaining. > Application and file system level encryption are usually the preferred choice for client systems because of the flexibility One big issue with "Application and file system level encryption" is that you often end up leaking metadata (such as the date edited, file name, file size, etc). Regardless I think that this is a really nice article. I can't wait to try their patches on my laptop.
- steerablesafe 7y ago
  > One big issue with "Application and file system level encryption" is that you often end up leaking metadata (such as the date edited, file name, file size, etc). I wonder how cryfs stacks up in this regard. https://www.cryfs.org https://www.cryfs.org
- richardwhiuk 7y ago
  > I do not really understand this. Which cryptographic algorithms can't be used? CBC - which is one the most common stream cipher algorithm. It's not clear me whether GCM would work or not.
  - dependenttypes 7y ago
    Apparently CBC is used by cryptsetup by default, see https://linux.die.net/man/8/cryptsetup https://linux.die.net/man/8/cryptsetup It might not be ideal but it still can be used. Though, I would not call CBC common at all. Pretty much everyone has switched to CTR or some variant of it (such as GCM). Also, CBC is not a stream cipher algorithm.
    - JoshTriplett 7y ago
      That online manpage is quite outdated; I recommend man7.org, which gets regularly auto-updated: http://man7.org/linux/man-pages/man8/cryptsetup.8.html http://man7.org/linux/man-pages/man8/cryptsetup.8.html . The current default for LUKS is XTS.
vbezhenar 7y ago
Offtopic, but why am I getting two scrollbars on this website? This is weird.
- tyingq 7y ago
  There is a scrollable div, the one that leads with: grep -A 11 'xts(aes)' /proc/crypto Is that what you mean?
  - zzzcpan 7y ago
    I can confirm, they broke scrolling with CSS setting overflow-x in #main-body which for me also shows two scrollbars.
  - vbezhenar 7y ago
    No, I literally getting two scroll bars for entire page. First scrollbar works, second scrollbar is disabled. Scrollable div is third scrollbar, but that's OK. It looks like that: https://i.imgur.com/Rs8a7m5.png https://i.imgur.com/Rs8a7m5.png
- zackbloom 7y ago
  Hi, I work on the Cloudflare Blog, we're working on deploying a fix now.
tyingq 7y ago
The blog post reads like this all happened recently, but their linked post to the dm-crypt mailing list is from September 2017[1]. I'm curious if they've interacted with the dm-crypt people more recently. [1]https://www.spinics.net/lists/dm-crypt/msg07516.html https://www.spinics.net/lists/dm-crypt/msg07516.html
- mattst88 7y ago
  Yeah, the time frame is somewhat unclear. The patch they link to in their tree is dated December 2019 however [1], so I assume this blog post is about stuff they've completed recently. [1] https://github.com/cloudflare/linux/blob/master/patches/0023-Add-DM_CRYPT_FORCE_INLINE-flag-to-dm-crypt-target.patch https://github.com/cloudflare/linux/blob/master/patches/0023...
unixhero 7y ago
Did they reach out to the Linux kernel mailing list? Or just the dm-crypt team, I found the answer they received rather arrogant and useless to be honest.
- jlgaddis 7y ago
  I'm a huge "fan" of F/OSS but, unfortunately, such condescending answers are all too common in this "community".
floatboth 7y ago
> Data encryption at rest is a must-have for any modern Internet company What is it protecting against — data recovery from discarded old disks? Very stupid criminals breaking into the datacenter, powering servers off and stealing disks? A breach in some web app would give the attacker access to a live system that has the encrypted disks already mounted…
- enitihas 7y ago
  I think mostly against breach in datacenter security. Most competent companies already have policies on how to deal with discarded old disks. The one that don't have might not be competent enough to use encryption on rest too. It's all about layers of defenses.
- mercora 7y ago
  being able to purge old disks confidently in a secure manner is a upside huge enough to make this statement true in my opinion. There have been numerous incidents even involving companies specializing in securely purging disks. If your data is encrypted there is basically nothing to do you could even outright sell those from your DC or something. Just delete the keys/headers from the disk and you are safe. Its also not possible to get data injected offline into your filesystem without having the keys. Without encryption you could just get the disk of the targeted server running somewhere and set your implants or what you have. When the server sees the disk back up it looks just like a hiccup or something.
  - brobinson 7y ago
    > Its also not possible to get data injected offline into your filesystem without having the keys. This is, in theory, possibly against volumes encrypted using AES XTS (which seems to the how the majority of FDE systems work) as the ciphertext is indeed malleable.
    - mercora 7y ago
      i am no expert on this but i was thinking it is only possible to inject noise which is likely corrupting the filesystem in the process. copying/moving valid blocks should be prevented by XTS as far as i understood (which might not be that much). I guess using a filesystem with integrity checks helps a bit although its still not authenticated or something.
lidHanteyk 7y ago
As usual, Cloudflare wants to pretend that they are community players, but they aren't. If they weren't hypocrites, then they'd submit their patches like [0] for upstream review, but they haven't. (I searched LKML.) I understand the underlying desire, which is to avoid using a queue for CPU-bound work that could be done immediately, but there doesn't appear to have been any serious effort to coordinate with other Linux contributors to figure out a solution to the problem. [0] https://github.com/cloudflare/linux/blob/master/patches/0024-Add-xtsproxy-Crypto-API-module.patch https://github.com/cloudflare/linux/blob/master/patches/0024...
- Majromax 7y ago
  The article discusses this in the conclusion: > We are going to submit this work for inclusion in the main kernel source tree, but most likely not in its current form. Although the results look encouraging we have to remember that Linux is a highly portable operating system: it runs on powerful servers as well as small resource constrained IoT devices and on many other CPU architectures as well. The current version of the patches just optimises disk encryption for a particular workload on a particular architecture, but Linux needs a solution which runs smoothly everywhere. That is, they think their current patch is too specialized for their own use-case to warrant inclusion in the mainline kernel without significant adaptation.
- deleted 7y ago
  [deleted]
- deleted 7y ago
  [deleted]
- manigandham 7y ago
  They are submitting their work, after they put in even more work to make it more universally applicable to all Linux users. They also did try to engage with the community who basically told them that they didn't know how fast crypto should be.
- marcinzm 7y ago
  >but there doesn't appear to have been any serious effort to coordinate with other Linux contributors to figure out a solution to the problem. Well when they reached out to the community they were told they're idiots and should f* off in only somewhat nicer language. Then they were simply ignored. When your community is toxic don't complain that people don't want to be part of it.
nullc 7y ago
> otherwise, we just forward the encryption request to the slower, generic C-based xts(ecb(aes-generic)) implementation This seems like at least something of a bad idea, because that implementation (if my search-fu is correct) is: https://github.com/torvalds/linux/blob/master/crypto/aes_generic.c https://github.com/torvalds/linux/blob/master/crypto/aes_gen... Which is obviously not constant time, and will leak information through cache/timing sidechannels. AES lends itself to a table based implementation which is simple, fairly fast, and-- unfortunately-- not secure if sidechannels matter. Fortunately, AES-NI eliminated most of the motivation for using such implementations on a vast collection of popular desktop hardware which has had AES-NI for quite a few years now. For the sake of also being constructive, here is a constant time implementation in naive C for both AES encryption and decryption (the latter being somewhat hard to find, because stream modes only use the former): https://github.com/bitcoin-core/ctaes https://github.com/bitcoin-core/ctaes (sadly, being single-block-at-a-time and constant time without hardware acceleration has a significant performance cost! ... better could be done for XTS mode, as the above algorithm could run SIMD using SSE2-- it isn't implemented in that implementation because the intended use was CBC mode which can't be parallelized like that) Can't the kernel aes-ni just be setup to save the fpu registers itself on the stack, if necessary?
- nemo1618 7y ago
  I wish the world could move on from AES. We have ciphers that are nearly as fast without requiring specialized hardware, just generic SIMD. Imagine how fast a ChaCha ASIC could run! There are other options for non-AES FDE too: most infamously Speck (suspected to be compromised by the NSA), but also Adiantum, which is now in Linux 5.0.
  - nullc 7y ago
    AES in an ASIC is pretty efficient, I'd expect the difference to flatten if both had good hardware implementations. Not that I wouldn't be happy to see faster chacha20 on systems.
  - dependenttypes 7y ago
    > Imagine how fast a ChaCha ASIC could run Not as fast. Chacha20 uses 32-bit additions which are fast in software but expensive and slow in hardware. In addition protecting Chacha20 from power analysis attacks is more difficult compared to AES. > just generic SIMD Constant-time AES with SSE2 is actually faster than the naive variable-time AES. See https://www.bearssl.org/constanttime.html#aes https://www.bearssl.org/constanttime.html#aes In addition Chacha20 is not nearly as fast as AES when using the AVX-512 Vector AES instructions. > but also Adiantum Which uses AES (once per sector).
tbrock 7y ago
Any chance of this patch making it to the mainline kernel?
- saagarjha 7y ago
  Not this one, specifically, but they've mentioned that they're working on upstreaming some derivative patches.
thedance 7y ago
All this seems to me a series of very strong arguments for doing the crypto in your application.
- saagarjha 7y ago
  That would be even slower and more complex.
  - thedance 7y ago
    Why? The slowness in this article comes from architectural brain damage inside the kernel. Doing the encryption and IO on your threads, when and where you choose to do it, is the solution. As your performance requirements increase, you are less and less likely to want kernel features.
LinuxBender 7y ago
Does CloudFlare plan to get their kernel patches merged upstream?
- yalooze 7y ago
  Second to last paragraph: > We are going to submit this work for inclusion in the main kernel source tree, but most likely not in its current form. Although the results look encouraging we have to remember that Linux is a highly portable operating system: it runs on powerful servers as well as small resource constrained IoT devices and on many other CPU architectures as well. The current version of the patches just optimises disk encryption for a particular workload on a particular architecture, but Linux needs a solution which runs smoothly everywhere.
  - LinuxBender 7y ago
    I missed that, thankyou.
gok 7y ago
That response from the dm-crypt mailing list is unreal.
ggregoire 7y ago
> Many companies, however, don't encrypt their disks, because they fear the potential performance penalty caused by encryption overhead. There is also the overhead of automatically unblocking a remote server during an unattended reboot. Reading the encryption password on a USB stick or fetching it through internet is a no from me. I think there are solutions about storing the password in RAM or in an unencrypted partition, but that's the overhead I'm talking about. I wonder how companies deal with that.
- jlgaddis 7y ago
  Red Hat's solution to this problem is NBDE. > The Network-Bound Disk Encryption (NBDE) allows the user to encrypt root volumes of hard drives on physical and virtual machines without requiring to manually enter a password when systems are restarted. [0] [0]: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Using_Network-Bound_Disk_Encryption.html https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
- gruez 7y ago
  Isn't this what TPMs are designed for? I think both intel and amd motherboards have them built-in by using the security processor in the CPU.
- mercora 7y ago
  i use kexec for reboots and store the keys for disks inside an initramfs which itself is stored on an encrypted boot partition. When i do a cold boot these systems boot into a recovery like OS so i can fix stuff when needed but mainly to do a kexec there (its not perfect but what is). If its possible to avoid this (i.e. i have physical access) i can decrypt the initramfs directly from grub using a passphrase entered locally. A warm reboot using kexec does not need any intervention from my side and directly boots into the already decrypted initramfs with the key already present and thus able to mount the encrypted volumes including the root volume.
- r1ch 7y ago
  Debian offers a dropbear shell in initramfs which you can use to SSH in and provide keys. I only have a handful of servers so currently I do this manually on a reboot but it would not be difficult to automate using for example SSH keys unlocking key material. The downside of this is your initramfs and kernel are on an unencrypted disk so a physical attacker could feasibly backdoor them. I'm sure there's some secure boot UEFI / TPM solution here.
convivialdingo 7y ago
Did this commercially for 15 years. Always the same problems. We ended up with several solutions- but all of them generally work the same conceptually. First off, separation of I/O layers. System calls into the FS stack should be reading and writing only to memory cache. Middle layer to schedule, synchronize and prioritize process IO. This layer fills the file system caché with cleartext and schedules writes back to disk using queues or journals. You also need a way to convert data without downtime. A simple block or file kernel thread to lock, encrypt, mark and writeback works well. Another beneficial technique is to increase blocksizes on disk. User Processes usually work in 4K blocks, but writing back blocks at small sizes is expensive. Better to schedule those writebacks later at 64k blocks so that hopefully the application is done with that particular stretch of data. Anyway, my 2 pennies.
gautamcgoel 7y ago
Does anyone know what the picture is like on FreeBSD? Is it faster?
justlexi93 7y ago
Neat. Poorly optimized queues can have a significant impact on performance, doubling throughput for disk encryption with some queue tweaks is pretty significant.
herpderperator 7y ago
> Unlike file system level encryption it encrypts all data on the disk including file metadata and even free space. Anyone have a source on how full disk aka block-level encryption encrypts free space? The only way I can imagine this could happen is by overwriting the entire disk initially with random data, so that you can't distinguish between encrypted and true "free space", i.e. on a brand new clean disk. Then, when a file (which, when written, would have been encrypted) is deleted (which by any conventional meaning of the word 'deleted' means the encrypted data is still present, but unallocated, thus indistinguishable from the random data in step 1), then gets overwritten again with random data? I would argue that overwriting an encrypted file with random data isn't really encrypting free space, but rather just overwriting the data, which already appeared random/encrypted. It is hardly any different to having a cleartext disk and overwriting deleted files with zeros, making them indistinguishable from actual free space.
- dlgeek 7y ago
  Debian does the first thing you discussed if you create an encrypted partition in the installer - it writes 0s through the crypto layer to fill the entire disk with encrypted data.
- koala_man 7y ago
  The point of encrypting free space is just so you can't say how full the drive is. This way, an attacker can't focus cracking on the fullest disk, match stolen backup disks to hosts based on non-sensitive health metrics, etc. >The only way I can imagine this could happen is by overwriting the entire disk initially with random data Traditionally, for speed, you'd write all zeroes to the encrypted volume (causing the physical volume to appear random), but yes >Then, when a file (which, when written, would have been encrypted) is deleted You'd just leave it. Crucially, you don't TRIM it. >I would argue that overwriting an encrypted file with random data isn't really encrypting free space Yup, that's why it's not done

491 points by jgrahamc 7y ago

>Being desperate we decided to seek support from the Internet and posted our findings to the dm-crypt mailing list When I see a company such as CloudFlare being so transparent about their difficulties, and trying to find an answer using their community members, it makes me love them even more. No ego, pure Professionalism

sneak 7y ago
Correspondingly, the response they received reflects just as strongly on the community itself.
- hyper_reality 7y ago
  Yep, the response they received was incredibly condescending. The follow-up from Cloudflare remained polite and added a lot more data, and was ignored. It's a shame because I've seen this condescending attitude quite frequently in the crypto open source community, and am not really sure how it arises. At least in this case it seems to have had the good outcome of motivating Cloudflare to dig in deeper and solve the problem by themselves.
  - dr_zoidberg 7y ago
    I'd say it comes from sitting at an ivory tower and giving 0 ducks about who you're talking to. In this case, the person/team asking was capable enough to go on their own, dig, test, change, and find a fix. It could've probably been easier for them if given proper directions. OTOH, perhaps those that answered from atop the tower had little idea of the mechanisms that the author(s?) dug out and changed. So double shame on them, for being condescending and not knowing. And it also falls on ourselves to be mindful of this behaviour, that can creep up on us without knowing. We sometimes think our time is super valuable and we don't have to spend it on some "newbie question" or this guy who doesn't understand. The past year I've been mentoring grad students in the lab I work at, and found myself once or twice going this route. I luckily caught it early, took a deep breath and gave them the time and explanations they needed. In the end I got a few nice surprises out of two amazing students, who were seeing a bit beyond what was evident.
    - belorn 7y ago
      I see many large project addressing this issue by not having a public list where you can talk directly to the developers. Instead there is user lists which public relation managers maintain and where the expected result from the original mail would either be no response or a polite and nicely written one about testing the user configuration options that they had already tested. That way the developer would not need to respond unless the user has shown enough proof of work to demonstrate to the public relation managers that the issue should be forwarded to a developer, in which case the answer the developer would reply with would be under the assumption that the person/team asking is capable enough to use the directions to dig, test, change, and create a patch which then later might be added to the project.
  - megous 7y ago
    Full response: https://www.spinics.net/lists/dm-crypt/msg07517.html https://www.spinics.net/lists/dm-crypt/msg07517.html From the PoV of the person who responded they didn't provide any relevant information that would indicate what platform they run, or what speed they expect, or why they think 800MiB/s seems slow to them. On many platforms this would be a pretty good result. At first look, it looks like they expected the speed of unencrypted storage, because that's what they tried to compare against. So the response seems reasonable at first glance to me. They got the answer to their main question. (which they omitted from their blog article)
    - gravitas 7y ago
      I disagree with your assessment. > If the numbers disturb you, then this is from lack of understanding on your side. This is arrogance on the part of the person replying that hand-waved away their problem "you just don't understand" when in fact, they (Cloudflare) do/did understand. They then went on to prove that it was due to queuing within the kernel, not the hardware as commented by this person in their flippant reply.
      - megous 7y ago
        Cloudflare did not understand at the time. Anyway, I'm not questioning that the reply was not very helpful, I just don't see it as unreasonable. I liked the technical parts of the CF writeup overall.
        marcinzm 7y ago
        >Cloudflare did not understand at the time And neither did the people who responded to them it seems.
        why_only_15 7y ago
        The next line after that is "You are probably unaware that encryption is a heavy-weight operation". If you're posting to the dm-crypt mailing list about encryption performance, you're probably very aware that encryption is a heavy-weight operation.
  - _jal 7y ago
    Honestly, read the original message and consider how you would have replied. They showed work in a vacuum - demonstrated that dm-crypt has costs over raw device access (I would hope so!) on some unknown hardware, and then asks "does this look right to you?" Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. People whine about technical mailing lists, I think because they don't get the context. Think of them as sort of like water coolers at an office that specializes in whatever the list is about. You get a short slice of expert attention in between doing whatever they actually have to get done. Throwing a bunch of data on the floor and saying "hey, is this expected?" is not going to work well. Seriously, what were they expecting?
    - rrss 7y ago
      > Well, yeah, that looks like it looks elsewhere, and by the way, there's a built-in command that also would have told you this. It's entirely possible to say both these things in a much more constructive and less condescending tone than was used.
      - _jal 7y ago
        People are so very weirdly sensitive to these things when it is a big company that comes calling. Wonder why that is. Context matters. If you don't take the time to understand the context you're walking in to and don't follow local rules, don't be surprised if people are rude to you back. Not that I even think what they said was all that rude. Do you also think you can slide in to a gam3r chat and expect business etiquette?
        manigandham 7y ago
        I don't think a crypto mailing list is the same as "gam3r" subreddit, but it wasn't that rude overall. It was the tone that they "dont understand" when in fact Cloudflare does understand crypto and performance very well, and went so far as to dive into kernel code and submit patches that fixed the problem that others didnt even realize existed. Even so I agree this isn't worth such a big discussion.
- ezoe 7y ago
  And according to the cloudflare, the current dm-crypt implementation is a horrible bit rot which got no reviewing for 15 years.
- crest 7y ago
  Especially because my old Haswell-E workstation running FreeBSD has no problem maxing out four encrypted SATA SSDs (>500MB/s each) at the same time with AES-NI. There is no excuse for slow cipher implementations and the queuing sounds insane saving and restoring the SSE registers can't be expensive enough to justify all those context switches between kernel threads.
- pengaru 7y ago
  It's a public mailing list, I see zero upstream kernel commits from arno@* so it doesn't appear the response came from someone who actually knows and works with the dm-crypt code. I'm on a number of public mailing lists and there's often a participant who tends to be both available/communicative and callous in their communication style. My assumption is there's a filter effect going on here where some folks who have very poor social abilities wind up at their computer alone all the time and public mailing lists become part of their few remaining human interactions. What I'd take away from this particular dm-crypt interaction isn't that the community is assholes, but that the community is small and the mailing list poorly attended/inactive. In the past I've reported my own dm-crypt problems upstream and it took years to get a bisected regression reverted. Just getting relevant people to pay attention was a challenge.
peterwwillis 7y ago
You won't know about their ego or professionalism until you work with them. Posting on a mailing list and making a blog post about it is not proof of either of these, it's brand marketing. They're trumpeting their engineering talent to build good will/nerd rep so people will love their company, spend money there, and apply for jobs. (But what it does show is that they're good at marketing, because it's working)
- brnt 7y ago
  Sounds like they're also pretty good at making disk encryption faster.
- darkwater 7y ago
  It is also marketing for sure, but it is well done marketing. It's like having a high PageRank in Google because you actually write meaningful, useful, well-written blog posts which Google happens (happened) to value vs link factory blog posts.
- the_duke 7y ago
  Writing informative blog posts and publishing patches to the Linux kernel is the rare kind of marketing I can 100% support.

beagle3 7y ago

Ages ago I benchmarked truecrypt overhead on my machine at the time (2006, I think?) and it was about 3%; I assumed that's a reasonable and still applicable number, also do dm-crypt and modern VeraCrypt. Guess I was get gradually more wrong through those years, according to the git archeology....

singlow 7y ago
Also, disk speed in 2006 was probably much slower. Disks have gotten faster at a greater pace than processors during the last 10 years.

vletal 7y ago

Has anyone already tried to compile the kernel with these patches for their desktop/laptop with encrypted drive? https://github.com/cloudflare/linux/tree/master/patches https://github.com/cloudflare/linux/tree/master/patches

asymptotically2 7y ago
Yes, I'm running them on kernel 5.5.13 (which came out today)
- 3r8riacz 7y ago
  Wow, I'd be so happy if you could share the steps you took to achieve this. Let's say I have a Debian machine, how could I try it out?
  - asymptotically2 7y ago
    I use Gentoo Linux so it was as easy as putting the patches in a directory, and then rebuilding the kernel with the option to enable the synchronous cipher. For Debian it's almost as easy: https://passthroughpo.st/patch-kernel-debian/ https://passthroughpo.st/patch-kernel-debian/

pmorici 7y ago

Interesting. One other thing they don't mention that I found interesting when doing my own digging on dmcrypt speeds a while back is that the 'cryptosetup benchmark' command is only showing the single core performance of each of those encryption algorithms. You can verify this by watching the processor load as it performs the benchmark. That lead me to find that if you have a Linux software RAID you can get much better performance by having 1 dmcrpyt volume per disk and then software RAID the dm devices instead of putting a single dmcrypt on top of the software RAID. Curious if that would stack performance wise with what they found here or if that just happened to help with the queuing issue they identified.

mercora 7y ago
i remember somewhat recently efforts to parallelize the work of dm-crypt where applicable had been merged. However, i guess having multiple separate encryption parameters and states (read: disks) leaves more opportunity for parallelization of the work especially if disk access patterns are not spread wide enough.

est31 7y ago

Wow those speed improvements are very neat. And an awesome blog post accompanying them. Prior to reading this, I've considered Linux disk encryption adding negligible latency because no HDDs/SSD can be fast enough for a CPU equipped with AES-NI, but that view has changed. Two questions: 1. are there any efforts to upstream them? 2. Invoking non-hw-accelerated AES decryption routines sounds quite expensive. Has it been tried out to save the FPU registers only if there is the need for decryption?

knorker 7y ago
At least your first question is answered in the article: Yes
andyjpb 7y ago
The existing Linux system is useful for hardware that does less than 200MB/s, so you should be fine with HDDs. Cloudflare is optimising for SSDs. They don't talk about latency: all their crypto benchmarks measure throughput. Near the end they hint at response time for their overall cache system but there's no detailed discussion of latency issues. The takeaway for me is that I'm OK with what's currently in Linux for the HDDs I use for my backups but I'd probably lose out if I encrypted my main SSD with LUKS. At the end of the article they say that they're not going to upstream the patches as they are because they've only tested them with this one workload. I'd also be interested to see a benchmark comparing SW AES with FPU-saving + HW AES. Unfortunately their post does not include stats for how often their proxy falls into the HW or SW implementations. Whatever those numbers are, I'd expect FPU-saving + HW AES to be somewhere in the middle.
- jlgaddis 7y ago
  > The takeaway for me is that I'm OK with what's currently in Linux for the HDDs I use for my backups but I'd probably lose out if I encrypted my main SSD with LUKS. Yep, when building my latest workstation, I went with a pair of ("regular") SSDs (RAID1) for my data. Later, I decided to add an NVMe for the OS for the additional speed. I then went and encrypted all of the drives (via LUKS), however, which basically killed any additional performance I would've gotten from the NVMe drive. I would have been just fine as well off with only the SSDs and without the NVMe drive.

dependenttypes 7y ago

> one can only encrypt the whole disk with a single key You can still use partitions. > not all cryptographic algorithms can be used as the block layer doesn't have a high-level overview of the data anymore I do not really understand this. Which cryptographic algorithms can't be used? > Most common algorithms require some sort of block chaining to be secure Nowadays I would say that from these only CTR is common, which does not require chaining. > Application and file system level encryption are usually the preferred choice for client systems because of the flexibility One big issue with "Application and file system level encryption" is that you often end up leaking metadata (such as the date edited, file name, file size, etc). Regardless I think that this is a really nice article. I can't wait to try their patches on my laptop.

steerablesafe 7y ago
> One big issue with "Application and file system level encryption" is that you often end up leaking metadata (such as the date edited, file name, file size, etc). I wonder how cryfs stacks up in this regard. https://www.cryfs.org https://www.cryfs.org
richardwhiuk 7y ago
> I do not really understand this. Which cryptographic algorithms can't be used? CBC - which is one the most common stream cipher algorithm. It's not clear me whether GCM would work or not.
- dependenttypes 7y ago
  Apparently CBC is used by cryptsetup by default, see https://linux.die.net/man/8/cryptsetup https://linux.die.net/man/8/cryptsetup It might not be ideal but it still can be used. Though, I would not call CBC common at all. Pretty much everyone has switched to CTR or some variant of it (such as GCM). Also, CBC is not a stream cipher algorithm.
  - JoshTriplett 7y ago
    That online manpage is quite outdated; I recommend man7.org, which gets regularly auto-updated: http://man7.org/linux/man-pages/man8/cryptsetup.8.html http://man7.org/linux/man-pages/man8/cryptsetup.8.html . The current default for LUKS is XTS.

vbezhenar 7y ago

Offtopic, but why am I getting two scrollbars on this website? This is weird.

tyingq 7y ago
There is a scrollable div, the one that leads with: grep -A 11 'xts(aes)' /proc/crypto Is that what you mean?
- zzzcpan 7y ago
  I can confirm, they broke scrolling with CSS setting overflow-x in #main-body which for me also shows two scrollbars.
- vbezhenar 7y ago
  No, I literally getting two scroll bars for entire page. First scrollbar works, second scrollbar is disabled. Scrollable div is third scrollbar, but that's OK. It looks like that: https://i.imgur.com/Rs8a7m5.png https://i.imgur.com/Rs8a7m5.png
zackbloom 7y ago
Hi, I work on the Cloudflare Blog, we're working on deploying a fix now.

tyingq 7y ago

The blog post reads like this all happened recently, but their linked post to the dm-crypt mailing list is from September 2017[1]. I'm curious if they've interacted with the dm-crypt people more recently. [1]https://www.spinics.net/lists/dm-crypt/msg07516.html https://www.spinics.net/lists/dm-crypt/msg07516.html

mattst88 7y ago
Yeah, the time frame is somewhat unclear. The patch they link to in their tree is dated December 2019 however [1], so I assume this blog post is about stuff they've completed recently. [1] https://github.com/cloudflare/linux/blob/master/patches/0023-Add-DM_CRYPT_FORCE_INLINE-flag-to-dm-crypt-target.patch https://github.com/cloudflare/linux/blob/master/patches/0023...

unixhero 7y ago

Did they reach out to the Linux kernel mailing list? Or just the dm-crypt team, I found the answer they received rather arrogant and useless to be honest.

jlgaddis 7y ago
I'm a huge "fan" of F/OSS but, unfortunately, such condescending answers are all too common in this "community".

floatboth 7y ago

> Data encryption at rest is a must-have for any modern Internet company What is it protecting against — data recovery from discarded old disks? Very stupid criminals breaking into the datacenter, powering servers off and stealing disks? A breach in some web app would give the attacker access to a live system that has the encrypted disks already mounted…

enitihas 7y ago
I think mostly against breach in datacenter security. Most competent companies already have policies on how to deal with discarded old disks. The one that don't have might not be competent enough to use encryption on rest too. It's all about layers of defenses.
mercora 7y ago
being able to purge old disks confidently in a secure manner is a upside huge enough to make this statement true in my opinion. There have been numerous incidents even involving companies specializing in securely purging disks. If your data is encrypted there is basically nothing to do you could even outright sell those from your DC or something. Just delete the keys/headers from the disk and you are safe. Its also not possible to get data injected offline into your filesystem without having the keys. Without encryption you could just get the disk of the targeted server running somewhere and set your implants or what you have. When the server sees the disk back up it looks just like a hiccup or something.
- brobinson 7y ago
  > Its also not possible to get data injected offline into your filesystem without having the keys. This is, in theory, possibly against volumes encrypted using AES XTS (which seems to the how the majority of FDE systems work) as the ciphertext is indeed malleable.
  - mercora 7y ago
    i am no expert on this but i was thinking it is only possible to inject noise which is likely corrupting the filesystem in the process. copying/moving valid blocks should be prevented by XTS as far as i understood (which might not be that much). I guess using a filesystem with integrity checks helps a bit although its still not authenticated or something.

lidHanteyk 7y ago

As usual, Cloudflare wants to pretend that they are community players, but they aren't. If they weren't hypocrites, then they'd submit their patches like [0] for upstream review, but they haven't. (I searched LKML.) I understand the underlying desire, which is to avoid using a queue for CPU-bound work that could be done immediately, but there doesn't appear to have been any serious effort to coordinate with other Linux contributors to figure out a solution to the problem. [0] https://github.com/cloudflare/linux/blob/master/patches/0024-Add-xtsproxy-Crypto-API-module.patch https://github.com/cloudflare/linux/blob/master/patches/0024...

Majromax 7y ago
The article discusses this in the conclusion: > We are going to submit this work for inclusion in the main kernel source tree, but most likely not in its current form. Although the results look encouraging we have to remember that Linux is a highly portable operating system: it runs on powerful servers as well as small resource constrained IoT devices and on many other CPU architectures as well. The current version of the patches just optimises disk encryption for a particular workload on a particular architecture, but Linux needs a solution which runs smoothly everywhere. That is, they think their current patch is too specialized for their own use-case to warrant inclusion in the mainline kernel without significant adaptation.
deleted 7y ago
[deleted]
deleted 7y ago
[deleted]
manigandham 7y ago
They are submitting their work, after they put in even more work to make it more universally applicable to all Linux users. They also did try to engage with the community who basically told them that they didn't know how fast crypto should be.
marcinzm 7y ago
>but there doesn't appear to have been any serious effort to coordinate with other Linux contributors to figure out a solution to the problem. Well when they reached out to the community they were told they're idiots and should f* off in only somewhat nicer language. Then they were simply ignored. When your community is toxic don't complain that people don't want to be part of it.

nullc 7y ago

> otherwise, we just forward the encryption request to the slower, generic C-based xts(ecb(aes-generic)) implementation This seems like at least something of a bad idea, because that implementation (if my search-fu is correct) is: https://github.com/torvalds/linux/blob/master/crypto/aes_generic.c https://github.com/torvalds/linux/blob/master/crypto/aes_gen... Which is obviously not constant time, and will leak information through cache/timing sidechannels. AES lends itself to a table based implementation which is simple, fairly fast, and-- unfortunately-- not secure if sidechannels matter. Fortunately, AES-NI eliminated most of the motivation for using such implementations on a vast collection of popular desktop hardware which has had AES-NI for quite a few years now. For the sake of also being constructive, here is a constant time implementation in naive C for both AES encryption and decryption (the latter being somewhat hard to find, because stream modes only use the former): https://github.com/bitcoin-core/ctaes https://github.com/bitcoin-core/ctaes (sadly, being single-block-at-a-time and constant time without hardware acceleration has a significant performance cost! ... better could be done for XTS mode, as the above algorithm could run SIMD using SSE2-- it isn't implemented in that implementation because the intended use was CBC mode which can't be parallelized like that) Can't the kernel aes-ni just be setup to save the fpu registers itself on the stack, if necessary?

nemo1618 7y ago
I wish the world could move on from AES. We have ciphers that are nearly as fast without requiring specialized hardware, just generic SIMD. Imagine how fast a ChaCha ASIC could run! There are other options for non-AES FDE too: most infamously Speck (suspected to be compromised by the NSA), but also Adiantum, which is now in Linux 5.0.
- nullc 7y ago
  AES in an ASIC is pretty efficient, I'd expect the difference to flatten if both had good hardware implementations. Not that I wouldn't be happy to see faster chacha20 on systems.
- dependenttypes 7y ago
  > Imagine how fast a ChaCha ASIC could run Not as fast. Chacha20 uses 32-bit additions which are fast in software but expensive and slow in hardware. In addition protecting Chacha20 from power analysis attacks is more difficult compared to AES. > just generic SIMD Constant-time AES with SSE2 is actually faster than the naive variable-time AES. See https://www.bearssl.org/constanttime.html#aes https://www.bearssl.org/constanttime.html#aes In addition Chacha20 is not nearly as fast as AES when using the AVX-512 Vector AES instructions. > but also Adiantum Which uses AES (once per sector).

tbrock 7y ago

Any chance of this patch making it to the mainline kernel?

saagarjha 7y ago
Not this one, specifically, but they've mentioned that they're working on upstreaming some derivative patches.

thedance 7y ago

All this seems to me a series of very strong arguments for doing the crypto in your application.

saagarjha 7y ago
That would be even slower and more complex.
- thedance 7y ago
  Why? The slowness in this article comes from architectural brain damage inside the kernel. Doing the encryption and IO on your threads, when and where you choose to do it, is the solution. As your performance requirements increase, you are less and less likely to want kernel features.

LinuxBender 7y ago

Does CloudFlare plan to get their kernel patches merged upstream?

yalooze 7y ago
Second to last paragraph: > We are going to submit this work for inclusion in the main kernel source tree, but most likely not in its current form. Although the results look encouraging we have to remember that Linux is a highly portable operating system: it runs on powerful servers as well as small resource constrained IoT devices and on many other CPU architectures as well. The current version of the patches just optimises disk encryption for a particular workload on a particular architecture, but Linux needs a solution which runs smoothly everywhere.
- LinuxBender 7y ago
  I missed that, thankyou.

gok 7y ago

That response from the dm-crypt mailing list is unreal.

ggregoire 7y ago

> Many companies, however, don't encrypt their disks, because they fear the potential performance penalty caused by encryption overhead. There is also the overhead of automatically unblocking a remote server during an unattended reboot. Reading the encryption password on a USB stick or fetching it through internet is a no from me. I think there are solutions about storing the password in RAM or in an unencrypted partition, but that's the overhead I'm talking about. I wonder how companies deal with that.

jlgaddis 7y ago
Red Hat's solution to this problem is NBDE. > The Network-Bound Disk Encryption (NBDE) allows the user to encrypt root volumes of hard drives on physical and virtual machines without requiring to manually enter a password when systems are restarted. [0] [0]: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Security_Guide/sec-Using_Network-Bound_Disk_Encryption.html https://access.redhat.com/documentation/en-US/Red_Hat_Enterp...
gruez 7y ago
Isn't this what TPMs are designed for? I think both intel and amd motherboards have them built-in by using the security processor in the CPU.
mercora 7y ago
i use kexec for reboots and store the keys for disks inside an initramfs which itself is stored on an encrypted boot partition. When i do a cold boot these systems boot into a recovery like OS so i can fix stuff when needed but mainly to do a kexec there (its not perfect but what is). If its possible to avoid this (i.e. i have physical access) i can decrypt the initramfs directly from grub using a passphrase entered locally. A warm reboot using kexec does not need any intervention from my side and directly boots into the already decrypted initramfs with the key already present and thus able to mount the encrypted volumes including the root volume.
r1ch 7y ago
Debian offers a dropbear shell in initramfs which you can use to SSH in and provide keys. I only have a handful of servers so currently I do this manually on a reboot but it would not be difficult to automate using for example SSH keys unlocking key material. The downside of this is your initramfs and kernel are on an unencrypted disk so a physical attacker could feasibly backdoor them. I'm sure there's some secure boot UEFI / TPM solution here.

convivialdingo 7y ago

Did this commercially for 15 years. Always the same problems. We ended up with several solutions- but all of them generally work the same conceptually. First off, separation of I/O layers. System calls into the FS stack should be reading and writing only to memory cache. Middle layer to schedule, synchronize and prioritize process IO. This layer fills the file system caché with cleartext and schedules writes back to disk using queues or journals. You also need a way to convert data without downtime. A simple block or file kernel thread to lock, encrypt, mark and writeback works well. Another beneficial technique is to increase blocksizes on disk. User Processes usually work in 4K blocks, but writing back blocks at small sizes is expensive. Better to schedule those writebacks later at 64k blocks so that hopefully the application is done with that particular stretch of data. Anyway, my 2 pennies.

gautamcgoel 7y ago

Does anyone know what the picture is like on FreeBSD? Is it faster?

justlexi93 7y ago

Neat. Poorly optimized queues can have a significant impact on performance, doubling throughput for disk encryption with some queue tweaks is pretty significant.

herpderperator 7y ago

> Unlike file system level encryption it encrypts all data on the disk including file metadata and even free space. Anyone have a source on how full disk aka block-level encryption encrypts free space? The only way I can imagine this could happen is by overwriting the entire disk initially with random data, so that you can't distinguish between encrypted and true "free space", i.e. on a brand new clean disk. Then, when a file (which, when written, would have been encrypted) is deleted (which by any conventional meaning of the word 'deleted' means the encrypted data is still present, but unallocated, thus indistinguishable from the random data in step 1), then gets overwritten again with random data? I would argue that overwriting an encrypted file with random data isn't really encrypting free space, but rather just overwriting the data, which already appeared random/encrypted. It is hardly any different to having a cleartext disk and overwriting deleted files with zeros, making them indistinguishable from actual free space.

dlgeek 7y ago
Debian does the first thing you discussed if you create an encrypted partition in the installer - it writes 0s through the crypto layer to fill the entire disk with encrypted data.
koala_man 7y ago
The point of encrypting free space is just so you can't say how full the drive is. This way, an attacker can't focus cracking on the fullest disk, match stolen backup disks to hosts based on non-sensitive health metrics, etc. >The only way I can imagine this could happen is by overwriting the entire disk initially with random data Traditionally, for speed, you'd write all zeroes to the encrypted volume (causing the physical volume to appear random), but yes >Then, when a file (which, when written, would have been encrypted) is deleted You'd just leave it. Crucially, you don't TRIM it. >I would argue that overwriting an encrypted file with random data isn't really encrypting free space Yup, that's why it's not done