4 ms·

Any new data technology that can manipulate relational data will 'sound like' a lot of things. A Tesla sounds a lot like a Honda if you never look under the hoo

Any new data technology that can manipulate relational data will 'sound like' a lot of things. A Tesla sounds a lot like a Honda if you never look under the hood. All I am asking is for people to approach this technology with an open mind and take a few minutes to watch a couple demo videos or download the software and try it before they dismiss it out of hand because it seems at first glance to be similar to something else.

warrenm 4y ago
Explain how what you've built isn't one of the myriad options folks here have pointed-out that it "sounds like" ...so far you haven't :)
- didgetmaster 4y ago
  The Didgets system is a general-purpose data manager that can store unstructured data (file data); highly structured data (relational DB data); and semi-structured data (NoSQL data) all within a single system. The same key-value stores that are used to form relational tables are also used to attach meta-data tags to file objects. So you can create many relational tables in one of our containers (a pod) along with importing a hundred million files. Just like a relational query where you want to look at every row in a table where the firstName column value starts with the letter 'A', you can also query the system for every JPEG file where any folder in its path contains 'Microsoft', for example. The system can return the rows in the table that match more quickly than other DB systems; but it can also show you all the files that match hundreds of times faster than a conventional file system. I have loaded in DB tables that have hundreds of millions of rows. I have loaded in tables with thousands of columns. I have loaded in over 200 million files. The system can drill down to subsets of these objects faster and easier than any system I have used before. I have worked with many file systems and databases. I have built tools (e.g. PartitionMagic and Drive Image) that manipulate data. I think I have built something that would appeal to other data enthusiasts like myself, but getting them to try it out for themselves is a real challenge.
  - warrenm 4y ago
    >but it can also show you all the files that match hundreds of times faster than a conventional file system I call baloney You can't return data faster than the filesystem unless it's all in RAM
    - didgetmaster 4y ago
      I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as the regular file API. What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. If you go through the hassle of indexing all your files using one of the OS vendors preferred indexing tools, the gap can be narrowed but I think my system is still faster.
      - warrenm 4y ago
        >What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed
        didgetmaster 4y ago
        File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).

warrenm 4y ago

Explain how what you've built isn't one of the myriad options folks here have pointed-out that it "sounds like" ...so far you haven't :)

didgetmaster 4y ago
The Didgets system is a general-purpose data manager that can store unstructured data (file data); highly structured data (relational DB data); and semi-structured data (NoSQL data) all within a single system. The same key-value stores that are used to form relational tables are also used to attach meta-data tags to file objects. So you can create many relational tables in one of our containers (a pod) along with importing a hundred million files. Just like a relational query where you want to look at every row in a table where the firstName column value starts with the letter 'A', you can also query the system for every JPEG file where any folder in its path contains 'Microsoft', for example. The system can return the rows in the table that match more quickly than other DB systems; but it can also show you all the files that match hundreds of times faster than a conventional file system. I have loaded in DB tables that have hundreds of millions of rows. I have loaded in tables with thousands of columns. I have loaded in over 200 million files. The system can drill down to subsets of these objects faster and easier than any system I have used before. I have worked with many file systems and databases. I have built tools (e.g. PartitionMagic and Drive Image) that manipulate data. I think I have built something that would appeal to other data enthusiasts like myself, but getting them to try it out for themselves is a real challenge.
- warrenm 4y ago
  >but it can also show you all the files that match hundreds of times faster than a conventional file system I call baloney You can't return data faster than the filesystem unless it's all in RAM
  - didgetmaster 4y ago
    I didn't say that I could read the actual file contents faster than a FS. My system still takes about the same amount of time to read in a 1GB file off disk as the regular file API. What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. If you go through the hassle of indexing all your files using one of the OS vendors preferred indexing tools, the gap can be narrowed but I think my system is still faster.
    - warrenm 4y ago
      >What I did say is that searching for all files that meet specific criteria (e.g. files with a .jpg extension, larger than 2GB, and whose parent folder contains the word 'Microsoft' in it) is much, much faster on my system. Baloney Unless you preprocess your filesystem into some kind of flatfile or in-memory map, you cannot beat the filesystem for access speed
      - didgetmaster 4y ago
        File systems perform 2 main functions. The first is to allocate blocks and maintain the integrity of the data stream (file contents) of each file. While Didgets has a few algorithms that I think are superior for allocating and tracking extents for file contents; the benefits in this area are just a few percentage points at best (nothing to write home about). The second is to create and maintain metadata about each file. This includes file names, date and time stamps, access permissions and extra metadata like tags. This is where Didgets excels greatly over conventional systems. We store file metadata completely different than file systems. Their designs are decades old and very inefficient in many cases. Didgets uses the same key-value stores for attaching metadata tags to each file as we use to create columns in relational database tables. Finding files with certain tags is extremely fast as is just finding files of a certain type (e.g. documents or photos).