4 ms·

Didgets is a little startup trying to get some traction. I am definitely not 'afraid to share implementation details', but I have not done a great job so far of

Didgets is a little startup trying to get some traction. I am definitely not 'afraid to share implementation details', but I have not done a great job so far of prominently posting a lot of technical specifications and details on the website. So...are you promising to try it out if I tell you how I did it?

nawgz 4y ago
I never make promises. I also never use tech that doesn't share "how it works". Clear?
- didgetmaster 4y ago
  Crystal! I have have never met someone who knows all the details about every piece of technology they use. You must be very smart.
  - nawgz 4y ago
    > someone who knows all the details about every piece of technology they use > "how it works" in quotes Well, see, I can reason about SQL semantics via knowledge of B-trees, and I can understand how there's a time vs space tradeoff made in SQL implementations around indexing - add indexes to save time, but they write to disk. Didgets... I don't even know what it is. Do I query it with SQL? Does it only work from your UI DB browser? I could watch videos to find out but I find you a bit combative, so my desire to bridge the gap from my end is pretty low atm
  - setr 4y ago
    I think it’s more of If someone tells you they’re faster than their well-funded, well-developed peers, then either: 1. They’re dramatically better engineers 2. They’ve constrained their data model to make it faster, likely at the cost of some usecases 3. They’ve developed some novel theory or set of tricks 4. Their benchmarks are limited/biased, or plainly wrong 5. They’ve gotten to drop backwards compatibility constraints their competitors couldn’t With usually the answer being 2 or 5 if they’re legitimately faster. In your case, based on glancing at videos 1. You seem to have widened the data model significantly 2. You’ve got no claims to novelty except the notion of didgets, which is an abstraction I’d expect to cost a pretty perf penny 3. You’ve got no claims to particular competency So… I’m instinctively betting on 4.
    - didgetmaster 4y ago
      I don't blame you at all for betting on 4. There are plenty of snake-oil salesmen out there. That is why I invite you to download the software; load in your own data set; and perform whatever benchmark test you like. Don't trust me. Trust your own eyes.
      - setr 4y ago
        The problem is that I need to be first convinced it’s worth bothering with (though you won’t convince me personally, because I don’t have any need for a better-performing DB and I’m lazy). But if my theory is right, you’ll have a lot more success getting people to test it if you resolve the other items. Which I think you could do in probably two paragraphs on the site. Even if I could convince myself that it’s faster, I still wouldn’t be comfortable with it if I couldn’t explain, even at a high level, why it’s faster. And I would never assume it’s faster for anything except exactly what I tested (which inevitably is only somewhat representative of what I actually use it for)
parker_mountain 4y ago
>So...are you promising to try it out if I tell you how I did it? Think of it this way - that kind of information is required for me to even consider something like this.
- didgetmaster 4y ago
  Didgets is a new technology. While the core technology has proven to be very flexible and performant, my resource constraints have limited what the current product can do. It is not yet a 'drop-in replacement' for an enterprise, distributed database. It can't yet do everything a file system can do. Instead, I have built a tool on the technology that can do some very interesting analytics against large DB tables. It can create pivot tables in seconds. It can help you find anomalies and clean your data quickly. But some other features like file indexing and log management features still need a lot of work. If you have an immediate need for the polished features, then it can be very useful as it is. If you have a need for features that are only partially finished, then I need feedback, testing, and resources to get them working properly.
  - jiggawatts 4y ago
    > analytics against large DB tables. It can create pivot tables in seconds. There are dozens, if not hundreds of products in this space. A popular example is Power BI, where even the desktop edition can produce extremely fast pivot table reports over about 10 GB of data. The server edition can produce sub-second reports on terabytes of data, not a mere couple of gigabytes like in your demos. You'd have to demonstrate a significant improvement over the existing state of the art for anyone to be interested.

nawgz 4y ago

I never make promises. I also never use tech that doesn't share "how it works". Clear?

didgetmaster 4y ago
Crystal! I have have never met someone who knows all the details about every piece of technology they use. You must be very smart.
- nawgz 4y ago
  > someone who knows all the details about every piece of technology they use > "how it works" in quotes Well, see, I can reason about SQL semantics via knowledge of B-trees, and I can understand how there's a time vs space tradeoff made in SQL implementations around indexing - add indexes to save time, but they write to disk. Didgets... I don't even know what it is. Do I query it with SQL? Does it only work from your UI DB browser? I could watch videos to find out but I find you a bit combative, so my desire to bridge the gap from my end is pretty low atm
- setr 4y ago
  I think it’s more of If someone tells you they’re faster than their well-funded, well-developed peers, then either: 1. They’re dramatically better engineers 2. They’ve constrained their data model to make it faster, likely at the cost of some usecases 3. They’ve developed some novel theory or set of tricks 4. Their benchmarks are limited/biased, or plainly wrong 5. They’ve gotten to drop backwards compatibility constraints their competitors couldn’t With usually the answer being 2 or 5 if they’re legitimately faster. In your case, based on glancing at videos 1. You seem to have widened the data model significantly 2. You’ve got no claims to novelty except the notion of didgets, which is an abstraction I’d expect to cost a pretty perf penny 3. You’ve got no claims to particular competency So… I’m instinctively betting on 4.
  - didgetmaster 4y ago
    I don't blame you at all for betting on 4. There are plenty of snake-oil salesmen out there. That is why I invite you to download the software; load in your own data set; and perform whatever benchmark test you like. Don't trust me. Trust your own eyes.
    - setr 4y ago
      The problem is that I need to be first convinced it’s worth bothering with (though you won’t convince me personally, because I don’t have any need for a better-performing DB and I’m lazy). But if my theory is right, you’ll have a lot more success getting people to test it if you resolve the other items. Which I think you could do in probably two paragraphs on the site. Even if I could convince myself that it’s faster, I still wouldn’t be comfortable with it if I couldn’t explain, even at a high level, why it’s faster. And I would never assume it’s faster for anything except exactly what I tested (which inevitably is only somewhat representative of what I actually use it for)

parker_mountain 4y ago

>So...are you promising to try it out if I tell you how I did it? Think of it this way - that kind of information is required for me to even consider something like this.

didgetmaster 4y ago
Didgets is a new technology. While the core technology has proven to be very flexible and performant, my resource constraints have limited what the current product can do. It is not yet a 'drop-in replacement' for an enterprise, distributed database. It can't yet do everything a file system can do. Instead, I have built a tool on the technology that can do some very interesting analytics against large DB tables. It can create pivot tables in seconds. It can help you find anomalies and clean your data quickly. But some other features like file indexing and log management features still need a lot of work. If you have an immediate need for the polished features, then it can be very useful as it is. If you have a need for features that are only partially finished, then I need feedback, testing, and resources to get them working properly.
- jiggawatts 4y ago
  > analytics against large DB tables. It can create pivot tables in seconds. There are dozens, if not hundreds of products in this space. A popular example is Power BI, where even the desktop edition can produce extremely fast pivot table reports over about 10 GB of data. The server edition can produce sub-second reports on terabytes of data, not a mere couple of gigabytes like in your demos. You'd have to demonstrate a significant improvement over the existing state of the art for anyone to be interested.