3 ms·

> Better to just use Git. How would this help with my example? Git is terrible for managing non-text files and has zero support for browsing such files interac

• laumars 8y ago
  > Turning off overcommit (and swap) does improve responsiveness but is not a viable option for a desktop system Back when I got my first SSD I ran Linux without a swap file/partition. I did this partly because it was only a 60GB SSD so I wanted to conserve space. I also did it because I didn't want to shorten the life of the SSD with (this was back when such a thing was a concern) and I ran that set up for years on a pretty modest 8GB RAM with KDE installed (ie not just a lightweight tiling WM). > unsurprising since a proper version needs FS integration It really doesn't and apfs (your file system in OSX) doesn't even do this. In fact it's probably better that your meta-data indexer isn't embedded into your file system driver because you're just going to slow down file system operations - which matters a lot on UNIX-based platforms because they do lots of file system operations. A far better approach is to have your indexer run as a separate process that monitors file writes (you can still have a kernel hook for that if you wish) thus you can then catalogue your files without interrupting your normal file system operations. You can also add more granularity like separate database per home directory (which would be much harder to do securely if you were embedding that code into the fs driver without then going down the route of having multiple tanks ala ZFS). It also makes it much easier to optimize your meta-data db since you can now dump everything into a RDBMS rather than attached to the space constrained inodes. For what it's worth, this is another area I have first hand experience with because I've written a few hobby file systems over the years. Nothing serious nor performant; just myself messing around with a few ideas. But it's still earned me a greater appreciation for the design decisions behind the file systems we do commonly use.
  • patrec 8y ago
    > Back when I got my first SSD I ran Linux without a swap file/partition You can turn off swap if you don't need hibernate, and from memory even turning off overcommit used to be OK-ish (of course most software written for linux doesn't try to deal with failing malloc requests gracefully, because there's no point since it never happens in the default configuration). You end up with a noticeably snappier system. However, this no longer seems to work in practice. Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory. > A far better approach is to have your indexer run as a separate process that monitors file writes (you can still have a kernel hook for that if you wish) This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). I agree that you don't want to synchronously update all indexing meta info on FS operations because everything will grind to a halt if you do that. But you still want OS support such as reliable notification and extended FS attributes to store things like "this was downloaded from here" or tagging. I don't think there is anything particularly magical about this (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) but in practice xattrs end up pretty much useless in linux because next to nothing uses them (baloo probably does) and as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though. I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days.
    • laumars 8y ago
      > You can turn off swap if you don't need hibernate Do people still hibernate? I thought these days suspending was a solved problem. > Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory. I thought the point of this discussion was talking about sane defaults? Of course if you're going to mess with kernel parameters then you run the risk of getting undesired behaviour. It's no different to when we used to tweak the BIOS in the 90s. So I'm not going to disagree with you there. But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? > This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). That's not file system integration though. What you were actually describing was a completely different behaviour. Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. > (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) Again, you don't want that information in the file system table. Storing every little bit of information like that in xattr would slow down standard file system operations. What you actually want to do is store that information in a separate RDBMS (eg sqlite3, MySQL/MariaDB, etc). To be honest even something like Redis might work as long as it has a persistent backup. > as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though. I've not spent a great amount of time with inotify but from my limited exposure I do recall it wasn't great with nested hierarchies. There's probably some better ways that I don't know of but this is a particular problem I've not needed to solve before so I'm as in the dark as you are. > I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days. Honestly, I think the perceived differences are all imaginary. Like wine tasting when you're told one bottle is expensive and another is moderately priced - lots of people will start to imagine deeper flavours in the more expensive bottle even if those flavours don't exist. So much of our perceptions are based on expectations rather than experiences and I think that's what's happening here because I've used both Krunner and Spotlight and my honest impression is that they're both much the same.