12 ms·
What is the purpose of the lost+found folder in Linux and Unix? (2014)
- pmontra 5mo agoI had a lost+found folder in all Unix file systems I used since the 80s. It's where fsck places files that it found during a scan and can't figure out to which directory they belong. Sometimes I found stuff in there. From what I googled XFS, Btrfs and ZFS don't use lost+found. It's a thing of the old not journaled filesystems and of the ext family.
- jcalvinowens 5mo agoXFS does use /lost+found, it calls it the "orphanage directory" and xfs_repair reparents children of corrupt directories there. Based on comments in the kernel source, it seems like the userspace fsck for JFS and F2FS will also sometimes create /lost+found. There might be more that do.
- adrian_b 5mo agoXFS filesystems do not have a "/lost+found" directory in their normal state. In the very rare occasions when one has to run "xfs_repair", it will create a "/lost+found" directory, if it is required for recovered files. After the repair and after investigating whether the recovered files contain useful data or not (and after moving the useful files elsewhere), one should normally delete the "/lost+found" directory, because it is no longer needed.
- Tsiklon 5mo agoXFS as implemented in RHEL8+ (the only places i've used it in anger) tends to handle being full very badly, leading to system lockups and blocked tasks necessitating a hard reboot. Worse yet is when it's in this state the journal fills and nothing can be done with the volume. To recover from this on a volume mounted at boot mandates going to either a live disk, or stopping boot in initramfs and running xfs_repair there, I've fruitlessly attempted to play back the journal on many separate occasions by attempting to mount the filesystem (again causing a lock up due to no space) in that state you have to drop the journal, run xfs_repair and then clean up the detritus from /lost+found (and then the location that caused the disk to fill altogether). EXT4 has other issues certainly, but at least it reserves blocks for the root user explicitly so the system doesn't stop.
- esseph 5mo agoXFS tip: If you truncate a file it doesn't update metadata. That's how you can get back space for the journal log to start cleaning up crap without rebooting the box and/or taking services down.
- adrian_b 5mo agoThe last time when I have used RHEL was decades ago, so I do not have any idea about what happens there. On the other hand, I have been using XFS since 2005 (since when I have transitioned from 32-bit Linux to 64-bit Linux), on a great variety of hardware systems, servers, desktops, mini-PCs, laptops. My file systems are typically mostly full and from time to time I had incidents when some job failed by filling completely the file system and no longer having any space left for writing the remainder of the files being written. Filling completely the HDD or SSD has never caused any problems. I have always just deleted some files or moved some files to other file systems, and I have continued working. Sometimes I had some downloading in progress, which was halted by the browser because of full disk, and in such cases, after making space, I just resumed the download in the browser. So I am puzzled by your experience, but I am not very surprised because in Linux there are many obscure configuration options, so the behavior can vary a lot between distributions (I typically use Gentoo). Perhaps your problems were caused by certain daemons that were continuing to make write attempts in the background, which I do not have. The only problems that I have ever encountered in XFS happened only in early XFS, i.e. 2 decades ago, which was extremely sensitive to power failures, despite being a journaled filesystem. In early XFS, after a power failure, some previously open files were erased, even if they had been open only for reading. Because of this, a power failure frequently bricked the system, by erasing "/etc/fstab". However, this stupid XFS feature has been corrected many years ago and nowadays power failures normally do not have any effect on XFS, i.e. xfs_repair is normally not needed, even after power failures. That was a bug at the conceptual level, not at the programming level, because the erasure of some files in early XFS was intentional, because it wrongly concluded that they might have been corrupted. While early XFS was notorious for its fragility against power failures, at that time none of the competing file systems was significantly better, all were buggy. Around the same time, more than two decades ago, I have seen a lot of other filesystems corrupted by power failures, regardless whether they were Windows NTFS or Linux EXT3 or JFS, despite the fact that all were advertised as being resistant to power failures by being journaled. At that time, only one filesystem was completely impervious to power failures, and it was non-journaled, the FreeBSD UFS with "soft updates" (i.e. with a careful ordering of the disk writes, to maintain a consistent state across power failures).
- Eikon 5mo agoEven with journaling, you might need one. ZeroFS [0] almost had a lost+found directory (even with the WAL enabled), because you might have consistency issues between your in-memory state and what was flushed, and especially in what order. ZeroFS ended up not needing recovery at all through atomic, strictly ordered commits [1], but it was far from trivial (and not just a matter of requiring a WAL). [0] https://github.com/Barre/ZeroFS https://github.com/Barre/ZeroFS [1] https://github.com/Barre/ZeroFS/blob/main/zerofs/src/fs/write_coordinator.rs https://github.com/Barre/ZeroFS/blob/main/zerofs/src/fs/writ...
- FerretFred 5mo agoIn a couple of decades running Linux installations of all flavours, I have never seen anything in lost+found!
- lokar 5mo agoYou need to use worse hardware and bad power :)
- FerretFred 5mo agoUmm .. how about a Raspberry Pi Zero 2W powered by a 2000MaH " lipstick style" powerbank?
- marcosdumay 5mo agoAnd more concurrent writes. But I think ext4 will only let things appear there if you change some default flags.
- pkaye 5mo agoI used to develop SSD firmware and one of things I worked on is making it robust to power failure. The power supplies have lots of capacitance so the voltage drop was slow so we would use a special test board that would disconnect from power and discharge fast to test it.
- doublepg23 5mo agoThank you for your service!
- lokar 5mo agoMy main experience is with pre-scsi/ide systems :)
- esafak 5mo agoESDI?
- undebuggable 5mo agolost+found is the Thumbs.db and .DS_Store of Linux
- pixelesque 5mo agoNot really, as it's only once per file system mount, whereas those Windows and MacOS files are sprinkled in most directories with images and almost every non-network drive directory respectively.
- dhosek 5mo agoI think it’s more akin to the Recovered Items (or something like that) folder that shows up in your home directory sometimes (but maybe not anymore?)
- autoexec 5mo agoMore like a FOUND.000 folder or a root directory filled with .CHK files
- gattilorenz 5mo agoI was thinking that it’s been a while since I saw those file names… but I guess your username checks out :)
- JdeBP 5mo agoYou can even tell which operating systems xe has used. Both of those are very specific. The \FOUND.nnn directories are something that IBM/Microsoft introduced when they re-did CHKDSK for OS/2, which was what Windows NT inherited. FILEnnnn.CHK in the root directory was the old Microsoft CHKDSK, and also Symantec/Norton SCANDISK, for MS/PC/DR-DOS.
- mesrik 5mo agoI've last seen those in older Cisco ASA firewall's up to OS version 8.something which had internal or external CompactFlash with VFAT16 filesystems. Usually caused by end of useful life of that CF or if already replaced by someone who did not quite understand device requirements and what it supports. Either because did not care or understand 32bit VFAT while it works for while, then when CF usage gets over VFAT16 supported 2GB fs gets corrupted, system fails after a while and then you got plenty of those FOUND.xxx files root of that CF drive after boot ran fsck. Those old ASA's did accept and work with 4GB CF's which were available much longer than 2GB versions, but you needed to make max 2GB VFAT16 primary partition and then format it with Linux mkfs.vfat with flags that made sure it's only 16bit version. Once that was done, you could copy files from old CF using something that copies also hidden files and directories too, which there were few there. ASA used to be some Cisco proprietary OS and was just rebranded PIX firewall, then from 8 oddly hacked linux with grub boot. Not much of my favourite box as a firewall, but Remote-access VPN features did work quite well quite long and when clustered it was easy to run upgrades each node at time without DTLS and IPsec clients even noticing it.
- robrain 5mo ago(2012), maybe?
- JdeBP 5mo agoIt is StackExchange. So in theory someone could modernize it at any time.
- robrain 5mo agoIndeed and I just saw it was edited in 2012 - originally posed in 2011, answers last updated in 2014.
- ciupicri 5mo agoYeah, it seems I improved a tiny tiny bit a couple of them.
- trumpdong 5mo agoNo, that would be reverted for "violating the author's intent" if you edited an existing answer, and if you posted a new answer it'd be permanently at the end of the list because it would never attract many votes due to being at the end of the list.
- josephcsible 5mo ago> No, that would be reverted for "violating the author's intent" if you edited an existing answer Can you link to an example of when that happened when it shouldn't have? > if you posted a new answer it'd be permanently at the end of the list because it would never attract many votes due to being at the end of the list. No it wouldn't. They added a new answer sort called "trending" and made it the default specifically to fix that problem.
- deleted 5mo ago[deleted]
- deleted 5mo ago[deleted]
- shevy-java 5mo agoI keep everything hidden there. In reallife I would rename this to "trash".
- JdeBP 5mo agoI have a book on my bookshelf, Eric Foxley's Unix for Super-Users. It was published in 1985, and it answers this question on page 52, the first page listed for the entry 'lost+found' in its index. This is surely not the earliest book mention, is it? (It'll be in earlier man pages, of course.) Google Books does not give me an earlier one, although it does yield another 1985 book. Fun fact: Foxley cautioned that lost+found must be pre-sized ahead of time, because the fsck of the time did not grow the directory to fit found files.
- valleyer 5mo agoHow do you size a directory? Just by creating entries in it and then unlinking them?
- necheffa 5mo agoUsually the filesystem driver provides an API call that sets the appropriate data structures so that the directory entry table is preallocated. The ext-family filesystems provide the mklost+found command to tap into this call if you need to recreate the lost+found directory specifically.
- Taniwha 5mo agoFrom memory mklost+found did exactly this
- JdeBP 5mo agoIt did. Foxley gives the manual procedure for sizing the lost+found directory on the aforementioned page 52. I have the 1986 edition of Fielder's and Hunter's UNIX System Administration and it does not mention any such command in its discussion of lost+found in chapter 3. It references AT&T Unix System 5 Release 2 (or 'UNIX 5.2' as the book puts it). But Google Books tells me that their later 1991 update, referencing Release 4 and with an updated title to match, does indeed mention mklost+found. So that looks like something that appeared in Release 3 or 4.
- 5mo ago
- anonu 5mo agoglad to see that Stack Overflow (or stackexchange.com) is still a thing.
- orwin 5mo agothey offer private instances to school too, where moderation is left to school policy, and mine seems to be good enough to use whenever i have frontend questions (i'm 10 year into my career and still use my juniors to answer my frontend questions, i think i won't ever change)
- gberger 5mo agoIt's a question from 2011...
- matheusmoreira 5mo agoWon't be for long. https://sherwood.news/tech/stack-overflow-forum-dead-thanks-ai-but-companys-still-kicking-ai/ https://sherwood.news/tech/stack-overflow-forum-dead-thanks-... https://news.ycombinator.com/item?id=48282709 https://news.ycombinator.com/item?id=48282709
- jmclnx 5mo agolost+found is still used on OpenBSD, seems it is created when needed. Only /home has that directory on my system. IIRC, it was created when a kernel panic happened a few releases ago. Plus some files were placed in it when fsck executed on /home
- SoftTalker 5mo agoSame, it's only in /home on my system also. Also /home is pretty much the only directory where I see fsck needing to do a lot of recovery after a power failure. Makes sense I guess, because that's where processes such as web browsers are likely to have lots of files open in RW mode.
- deadbabe 5mo agoHow do questions like this make it to the top? It is an obvious thing if you search for it or ask AI, but people seem to just ignore those in favor of generating new human responses. Thing is, any time I try to replicate something like that, I basically get a flippant response saying to go look elsewhere.
- yndoendo 5mo agoReaching HN or other sites allows for exposure to information that a person otherwise would not ask themselves. I also respect human responses over AI ones every day that ends in Y.
- Velocifyer 5mo agoWhy not on sundae?
- zer0zzz 5mo agoBack in the day I accidentally deleted all my stuff because I had it all in a special dir of this user in suse Linux. When I deleted the user, yast deleted everything. Fortunately I was using ReiserFS at the time and something about its murderous tree data structure made it trivial to undelete. Reiser_fsck found ALL my stuff, mostly with full dir tree structure in tact and put it all in lost+found
- NelsonMinar 5mo agoWe once deleted the lost+found folder on an old Unix system* by accident. Things went very badly the next time the system rebooted, fsck did not handle it at all well. * Probably DEC Ultrix 2.2, a BSD 4.2 derivative.
- amelius 5mo agoWhy can't a filesystem create the lost+found folder only when it needs to store files in it? That would be a much cleaner approach, imho. Added benefit is that you'd immediately see it if something is wrong with a disk.
- layer8 5mo agoAs the submission explains, the lost+found folder has pre-allocated space for the directory entries. From the mklost+found man page [0]: mklost+found pre-allocates disk blocks to the lost+found directory so that when e2fsck(8) is being run to recover a file system, it does not need to allocate blocks in the file system to store a large number of unlinked files. This ensures that e2fsck will not have to allocate data blocks in the file system during recovery. Pre-allocating space without making the directory visible would require more arcane file system magic. [0] https://man7.org/linux/man-pages/man8/mklost+found.8.html https://man7.org/linux/man-pages/man8/mklost+found.8.html
- amelius 5mo ago> Preallocating space without making the directory visible would require more arcane file system magic. If those filesystem engineers had a manager that said: make this nice for the user, then it would have been done. But these developers had no managers and were OK eating their own unpalatable dogfood.
- dhosek 5mo agoYou greatly overestimate the capabilities of computers of the era that this originated in. Major limitations of disk space, speed and addressability combined with limited CPU power and RAM mean that your hypothetical, “it could have been done” would require a significant investment in development and runtime resources all to avoid having a directory in root that would be empty until after a post-crash recovery. Also remember that these systems would have all been multi-user time-sharing systems, not desktop computers.
- 5mo ago
- Schnitz 5mo agoThe lost+found folder saved our backs once way back when. We had a 1TB NAS running Linux, an enormous amount of storage back then. It was shut down unexpectedly and disorderly and ran a multi hour fsck upon reboot. The volume must have had a shot root inode, as after the machine booted and mounted the volume it was empty. All directories were luckily under lost+found with all their contents.
- account42 5mo agormdir /lost+found is always my first step after creating and mounting a new file system. Reserving a user-visible directory just so that a 0.00001% chance recovery is slightly easier just doesn't make any sense.
- cobbaut 5mo agoI have recovered files several times after fsck on a failed partition. I may have a 75 percent success rate with this... did not keep statistics obviously. Though it could be less, cause I also remember recovering files with foremost on a dd copy of a disk.
- account42 5mo agoYou'll still be able to recover files - fsck will just recreate the directory. There is a slight chance that if the free list is fucked that this will loose you a couple of more blocks of data but then you're already well into fubar territory anyway.
- lanycrost 5mo agoThis is the way journaled filesystems place recovery files.