4 ms·

File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets

File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).

warrenm 4y ago
So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))
didgetmaster 4y ago
Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.
- warrenm 4y ago
  >It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help. My comments are aimed at your claims that either don't hold water, or where you try to defend your work as something brand new, when clearly other things already do what you describe :)
- warrenm 4y ago
  > Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. But that's not what you claimed until just now - and it flies in the face of other claims Is it a "database"? Or a filesystem replacement? If you "import files into Didgets just like you would copy files from one file system to a new one", then it's sitting on top of a file system, and acts as a database (whether it's actually new or not is up for debate (but so far...it doesn't sound "new" to me, based on everything you've said) "data is stored on disk in its new format" - not from what you've described elsewhere, where you claim you're just putting metadata around file names for faster searching (eg of file name attributes of binary objects like images) If you've built a file system, no one in their right mind is going to rely on it for a very long time - based on your demo being a Windows application where you need to import CSVs etc into it...it's pretty apparent it's not a file system (at least, it's not a native file system - it's still a file format hosted on the underlying OS' access to whatever file system(s) it's using) If you've built a database or filesystem mapper, someone may (or may not) want to use it - but so far you've provided nothing compelling in any of your descriptions to sound actually new ...you've described core features of plenty of existing tools/concepts - columnar databases, slocate, kvstores, ELK, Splunk, BeFS, etc

warrenm 4y ago

So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))

didgetmaster 4y ago

Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.

warrenm 4y ago
>It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help. My comments are aimed at your claims that either don't hold water, or where you try to defend your work as something brand new, when clearly other things already do what you describe :)
warrenm 4y ago
> Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. But that's not what you claimed until just now - and it flies in the face of other claims Is it a "database"? Or a filesystem replacement? If you "import files into Didgets just like you would copy files from one file system to a new one", then it's sitting on top of a file system, and acts as a database (whether it's actually new or not is up for debate (but so far...it doesn't sound "new" to me, based on everything you've said) "data is stored on disk in its new format" - not from what you've described elsewhere, where you claim you're just putting metadata around file names for faster searching (eg of file name attributes of binary objects like images) If you've built a file system, no one in their right mind is going to rely on it for a very long time - based on your demo being a Windows application where you need to import CSVs etc into it...it's pretty apparent it's not a file system (at least, it's not a native file system - it's still a file format hosted on the underlying OS' access to whatever file system(s) it's using) If you've built a database or filesystem mapper, someone may (or may not) want to use it - but so far you've provided nothing compelling in any of your descriptions to sound actually new ...you've described core features of plenty of existing tools/concepts - columnar databases, slocate, kvstores, ELK, Splunk, BeFS, etc