3 ms·

> Back when I got my first SSD I ran Linux without a swap file/partition You can turn off swap if you don't need hibernate, and from memory even turning off ov

by patrec 8y ago

> Back when I got my first SSD I ran Linux without a swap file/partition You can turn off swap if you don't need hibernate, and from memory even turning off overcommit used to be OK-ish (of course most software written for linux doesn't try to deal with failing malloc requests gracefully, because there's no point since it never happens in the default configuration). You end up with a noticeably snappier system. However, this no longer seems to work in practice. Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory. > A far better approach is to have your indexer run as a separate process that monitors file writes (you can still have a kernel hook for that if you wish) This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). I agree that you don't want to synchronously update all indexing meta info on FS operations because everything will grind to a halt if you do that. But you still want OS support such as reliable notification and extended FS attributes to store things like "this was downloaded from here" or tagging. I don't think there is anything particularly magical about this (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) but in practice xattrs end up pretty much useless in linux because next to nothing uses them (baloo probably does) and as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though. I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days.

laumars 8y ago
> You can turn off swap if you don't need hibernate Do people still hibernate? I thought these days suspending was a solved problem. > Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory. I thought the point of this discussion was talking about sane defaults? Of course if you're going to mess with kernel parameters then you run the risk of getting undesired behaviour. It's no different to when we used to tweak the BIOS in the 90s. So I'm not going to disagree with you there. But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? > This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). That's not file system integration though. What you were actually describing was a completely different behaviour. Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. > (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) Again, you don't want that information in the file system table. Storing every little bit of information like that in xattr would slow down standard file system operations. What you actually want to do is store that information in a separate RDBMS (eg sqlite3, MySQL/MariaDB, etc). To be honest even something like Redis might work as long as it has a persistent backup. > as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though. I've not spent a great amount of time with inotify but from my limited exposure I do recall it wasn't great with nested hierarchies. There's probably some better ways that I don't know of but this is a particular problem I've not needed to solve before so I'm as in the dark as you are. > I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days. Honestly, I think the perceived differences are all imaginary. Like wine tasting when you're told one bottle is expensive and another is moderately priced - lots of people will start to imagine deeper flavours in the more expensive bottle even if those flavours don't exist. So much of our perceptions are based on expectations rather than experiences and I think that's what's happening here because I've used both Krunner and Spotlight and my honest impression is that they're both much the same.
- patrec 8y ago
  > That's not file system integration though. What you were actually describing was a completely different behaviour. I should probably have phrased this differently, "kernel file system layer integration" maybe. The relevant (and presently, I believe, lacking part) in linux would be VFS. It also relies on applications making consistent use of xattrs for some functionality, something that does also not happen on linux. > Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. [...] Honestly, I think the perceived differences are all imaginary. Right. I'm not a file system expert, but I'm increasingly wondering if your confident pronouncements are backed up by sufficient knowledge what you're talking about. Spotlight is implemented with major kernel support in the form of fsevents. This allows the user space portions of it to receive fairly reliable and timely notification of file system changes efficiently. This is a key ingredient to make it work as well as it does. (see e.g. https://eclecticlight.co/2017/09/12/watching-macos-file-systems-fsevents-and-volume-journals/ https://eclecticlight.co/2017/09/12/watching-macos-file-syst...) Now the thing is, linux doesn't have a direct equivalent (or at least if it now has, it's a pretty recent thing, more than a decade after spotlight). Quoting from lkml (https://lkml.org/lkml/2016/12/20/312 https://lkml.org/lkml/2016/12/20/312) Other operating systems have a scalable way of watching changes on a large file system. Windows has USN Journal, macOS has FSEvents and BSD has kevents. The only way in Linux to monitor file system namei events (e.g. create/delete/move) is the recursive inotify watch way and this method scales very poorly for large enough directory trees. In light of this apparent disparity can you provide more detail on how linux based indexers work just the same and just as well as spotlight on macOS? What's the equivalent to fsevents they're all using? > But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? Let me try again: with default settings my high spec linux box ground to unusable state (and no I'm not making it up) frequently enough that I got sick off it. So unlike you (somewhat rudely) continue to imply, I'm not some bozo who randomly screwed around with system settings he didn't grok on a whim and then started whining after everything broke. > Again, you don't want that information in the file system table. Yeah, you do because that way it stays around when you copy, move or archive the file. You probably only want to do that with a few select metainfo fields (like the examples I gave earlier: download origin info and user supplied tags), but that's exactly what macOS does. Also, whilst I agree that storing search indexes and everything directly in the file system is probably not ideal, there is historical precedent of a systems that did exactly that, fairly successfully from what i hear (BeOS/BFS). P.S. maybe a more productive direction: what is you recommended way for setting up some baloo or some other linux indexer for running mdfind-style commandline queries with it (I don't want KDE or Gnome and I think baloosearch vs mdfind is also easier to compare directly)?

by patrec 8y ago

laumars 8y ago

> You can turn off swap if you don't need hibernate Do people still hibernate? I thought these days suspending was a solved problem. > Try turning off overcommit completely and see how long it takes Chrome to crash even if you have a lot of available memory. I thought the point of this discussion was talking about sane defaults? Of course if you're going to mess with kernel parameters then you run the risk of getting undesired behaviour. It's no different to when we used to tweak the BIOS in the 90s. So I'm not going to disagree with you there. But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? > This is how spotlight works though, no? It's a separate process gets notified by the kernel on file system changes and then indexes them (that's what I meant with FS integration). That's not file system integration though. What you were actually describing was a completely different behaviour. Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. > (spotlight is 15 years old tech and linux has had xattr support in all major file systems for ages) Again, you don't want that information in the file system table. Storing every little bit of information like that in xattr would slow down standard file system operations. What you actually want to do is store that information in a separate RDBMS (eg sqlite3, MySQL/MariaDB, etc). To be honest even something like Redis might work as long as it has a persistent backup. > as far as I'm aware there is no robust file system change notification API (you can use inotify for some stuff, but it's limited in various ways). I'd love to be wrong about this though. I've not spent a great amount of time with inotify but from my limited exposure I do recall it wasn't great with nested hierarchies. There's probably some better ways that I don't know of but this is a particular problem I've not needed to solve before so I'm as in the dark as you are. > I think the situation is better on macOS, but it might just be that spotlight is more polished and there is no fundamental difficulty in writing the same for linux these days. Honestly, I think the perceived differences are all imaginary. Like wine tasting when you're told one bottle is expensive and another is moderately priced - lots of people will start to imagine deeper flavours in the more expensive bottle even if those flavours don't exist. So much of our perceptions are based on expectations rather than experiences and I think that's what's happening here because I've used both Krunner and Spotlight and my honest impression is that they're both much the same.

patrec 8y ago
> That's not file system integration though. What you were actually describing was a completely different behaviour. I should probably have phrased this differently, "kernel file system layer integration" maybe. The relevant (and presently, I believe, lacking part) in linux would be VFS. It also relies on applications making consistent use of xattrs for some functionality, something that does also not happen on linux. > Moreover, you claimed that Spotlight works differently from other tools of it's ilk and that is also untrue. [...] Honestly, I think the perceived differences are all imaginary. Right. I'm not a file system expert, but I'm increasingly wondering if your confident pronouncements are backed up by sufficient knowledge what you're talking about. Spotlight is implemented with major kernel support in the form of fsevents. This allows the user space portions of it to receive fairly reliable and timely notification of file system changes efficiently. This is a key ingredient to make it work as well as it does. (see e.g. https://eclecticlight.co/2017/09/12/watching-macos-file-systems-fsevents-and-volume-journals/ https://eclecticlight.co/2017/09/12/watching-macos-file-syst...) Now the thing is, linux doesn't have a direct equivalent (or at least if it now has, it's a pretty recent thing, more than a decade after spotlight). Quoting from lkml (https://lkml.org/lkml/2016/12/20/312 https://lkml.org/lkml/2016/12/20/312) Other operating systems have a scalable way of watching changes on a large file system. Windows has USN Journal, macOS has FSEvents and BSD has kevents. The only way in Linux to monitor file system namei events (e.g. create/delete/move) is the recursive inotify watch way and this method scales very poorly for large enough directory trees. In light of this apparent disparity can you provide more detail on how linux based indexers work just the same and just as well as spotlight on macOS? What's the equivalent to fsevents they're all using? > But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? But what are you actually proving aside how easy it is to break things if you mess with core settings that are designed for experts? Let me try again: with default settings my high spec linux box ground to unusable state (and no I'm not making it up) frequently enough that I got sick off it. So unlike you (somewhat rudely) continue to imply, I'm not some bozo who randomly screwed around with system settings he didn't grok on a whim and then started whining after everything broke. > Again, you don't want that information in the file system table. Yeah, you do because that way it stays around when you copy, move or archive the file. You probably only want to do that with a few select metainfo fields (like the examples I gave earlier: download origin info and user supplied tags), but that's exactly what macOS does. Also, whilst I agree that storing search indexes and everything directly in the file system is probably not ideal, there is historical precedent of a systems that did exactly that, fairly successfully from what i hear (BeOS/BFS). P.S. maybe a more productive direction: what is you recommended way for setting up some baloo or some other linux indexer for running mdfind-style commandline queries with it (I don't want KDE or Gnome and I think baloosearch vs mdfind is also easier to compare directly)?