4 ms·

I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as

I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as the regular file API. What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. If you go through the hassle of indexing all your files using one of the OS vendors preferred indexing tools, the gap can be narrowed but I think my system is still faster.

warrenm 4y ago
>What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed
- didgetmaster 4y ago
  File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).
  - warrenm 4y ago
    So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))
  - didgetmaster 4y ago
    Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.
    - warrenm 4y ago
      >It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help. My comments are aimed at your claims that either don't hold water, or where you try to defend your work as something brand new, when clearly other things already do what you describe :)
    - 4y ago

warrenm 4y ago

>What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed

didgetmaster 4y ago
File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).
- warrenm 4y ago
  So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))
- didgetmaster 4y ago
  Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.
  - warrenm 4y ago
    >It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help. My comments are aimed at your claims that either don't hold water, or where you try to defend your work as something brand new, when clearly other things already do what you describe :)
  - 4y ago