3 ms·

The Didgets system is a general-purpose data manager that can store unstructured data (file data); highly structured data (relational DB data); and semi-structu

The Didgets system is a general-purpose data manager that can store unstructured data (file data); highly structured data (relational DB data); and semi-structured data (NoSQL data) all within a single system. The same key-value stores that are used to form relational tables are also used to attach meta-data tags to file objects. So you can create many relational tables in one of our containers (a pod) along with importing a hundred million files. Just like a relational query where you want to look at every row in a table where the firstName column value starts with the letter 'A', you can also query the system for every JPEG file where any folder in its path contains 'Microsoft', for example. The system can return the rows in the table that match more quickly than other DB systems; but it can also show you all the files that match hundreds of times faster than a conventional file system. I have loaded in DB tables that have hundreds of millions of rows. I have loaded in tables with thousands of columns. I have loaded in over 200 million files. The system can drill down to subsets of these objects faster and easier than any system I have used before. I have worked with many file systems and databases. I have built tools (e.g. PartitionMagic and Drive Image) that manipulate data. I think I have built something that would appeal to other data enthusiasts like myself, but getting them to try it out for themselves is a real challenge.

warrenm 4y ago
>but it can also show you all the files that match hundreds of times faster than a conventional file system I call baloney You can't return data faster than the filesystem unless it's all in RAM
- didgetmaster 4y ago
  I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as the regular file API. What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. If you go through the hassle of indexing all your files using one of the OS vendors preferred indexing tools, the gap can be narrowed but I think my system is still faster.
  - warrenm 4y ago
    >What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed
    - didgetmaster 4y ago
      File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).
      - warrenm 4y ago
        So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))
      - didgetmaster 4y ago
        Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.
warrenm 4y ago
You just described Splunk :)

warrenm 4y ago

>but it can also show you all the files that match hundreds of times faster than a conventional file system I call baloney You can't return data faster than the filesystem unless it's all in RAM

didgetmaster 4y ago
I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as the regular file API. What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. If you go through the hassle of indexing all your files using one of the OS vendors preferred indexing tools, the gap can be narrowed but I think my system is still faster.
- warrenm 4y ago
  >What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed
  - didgetmaster 4y ago
    File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).
    - warrenm 4y ago
      So ... you're preprocessing the filesystem's contents into another file and/or keeping it in memory >Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos). Just like I said You are not exceeding the performance of the filesystem, you're hiding the filesystem behind your preprocessed abstraction Like locate does on *nix (https://en.wikipedia.org/wiki/Locate_(Unix) https://en.wikipedia.org/wiki/Locate_(Unix))
    - didgetmaster 4y ago
      Didgets is not some abstraction between a file system and applications where it just caches data in RAM. It is designed to replace file systems as an on-disk storage solution. Yes, you import files into Didgets just like you would copy files from one file system to a new one; but the data is stored on disk in its new format. Just like other file systems do when you mount the volume from disk, Didgets will cache some system structures in memory to speed things up; but our on-disk structures are much smaller so we can read and cache what we need faster and using less memory to do it. As I pointed out earlier, the system is designed to manage DB tables as well as unstructured file data so it is much more than just a different kind of file system. It seems obvious from your comments that you have no intention of downloading and actually trying the software so it seems pointless to try and address all your objections. If on the other hand you try it and run into a real problem instead of just a perceived one...then I will be happy to help.

warrenm 4y ago

You just described Splunk :)