3 ms·

How confident are you that your database abstraction is going to fulfill the needs of someone with anything more than a few trivial query workloads? I didn't h

How confident are you that your database abstraction is going to fulfill the needs of someone with anything more than a few trivial query workloads? I didn't hear much about the database at all other than "it just works." This sounds like a recipe for full table scans falling over once more than ~10K records are in a given table. What about transactions? Can I do joins?

pbiggar 7y ago
The model here is that we own the responsibility of making it scale. We haven't solved the problem of significant workload, but there doesn't seem to be any reason why this wouldn't work - certainly our current (low-scale) users have not had any scaling issues with Dark. If there's a fatal flaw we haven't thought of, please let me know. We deliberately cut relational features for now because they didn't mesh with our language. We want to add joins back - right now you're doing stuff the NoSQL way (which is not my preferred way, but it's where we are today). We haven't figured out or transactional model, but we want to have a good story around that.
- ryanworl 7y ago
  Sent you a Twitter DM. Too much to type out, but I have some ideas for you.
  - pbiggar 7y ago
    Thanks!
- vosper 7y ago
  > We want to add joins back - right now you're doing stuff the NoSQL way Can you expand on the database capabilities? By NoSQL do you mean no relational data, no constraints, etc? You can do joins in Mongo, but it's still not an RDBMS. And you can have relational data and constraints in databases that support non-SQL query languages. NoSQL is a bit of an overloaded term these days :)
  - pbiggar 7y ago
    Yes, that was a bit vague, sorry. What I meant was that our datastores are simple key-value stores at the moment.
    - eternalny1 7y ago
      > What I meant was that our datastores are simple key-value stores at the moment. And if you go down the route of trying to expand that, then you are no longer a tool for easily building APIs, you are now a database tool also ... and before you know it you are trying to replicate features that RDBMs have had for 40 years. I wouldn't envy trying to have to implement ACID transactions. I realize it seems like this is hyperbole but have you drawn a fine line in the sand at where the features for Dark end? Because if not you will quickly paint yourself into a corner with this.
- eternalny1 7y ago
  > We deliberately cut relational features for now because they didn't mesh with our language. We want to add joins back You're going to get caught in that age-old trap. Try to make something to "make programming easy", then realize it meets the needs of very few due to lack of features, and then start adding features until it's not "easy" any more it's just programming. And then you realize you're decades behind what people are already using to do programming.
- kaens 7y ago
  It's very easy to paint yourself into a corner if you haven't figured out at least a plan of attack for these topics up-front, especially if you're starting to ramp up actual use of your product.
cryptonector 7y ago
Right, the demo video lost me at the datastore point. This is a fancy ORM. Now, I do get that making app dev so much easier is a very good thing, but ORMs are a double-edged sword. You need an option for the dev to have much more control over the schema and queries!
eternalny1 7y ago
> How confident are you that your database abstraction is going to fulfill the needs of someone with anything more than a few trivial query workloads? At this point, why does a tool like this need to exist at all? To a full-stack developer writing API's isn't difficult ... and if it is difficult then you are going to need a lot more than a tool like this. Should non-programmers be creating JSON REST APIs in the first place? I don't really see where this is going.

pbiggar 7y ago

The model here is that we own the responsibility of making it scale. We haven't solved the problem of significant workload, but there doesn't seem to be any reason why this wouldn't work - certainly our current (low-scale) users have not had any scaling issues with Dark. If there's a fatal flaw we haven't thought of, please let me know. We deliberately cut relational features for now because they didn't mesh with our language. We want to add joins back - right now you're doing stuff the NoSQL way (which is not my preferred way, but it's where we are today). We haven't figured out or transactional model, but we want to have a good story around that.

ryanworl 7y ago
Sent you a Twitter DM. Too much to type out, but I have some ideas for you.
- pbiggar 7y ago
  Thanks!
vosper 7y ago
> We want to add joins back - right now you're doing stuff the NoSQL way Can you expand on the database capabilities? By NoSQL do you mean no relational data, no constraints, etc? You can do joins in Mongo, but it's still not an RDBMS. And you can have relational data and constraints in databases that support non-SQL query languages. NoSQL is a bit of an overloaded term these days :)
- pbiggar 7y ago
  Yes, that was a bit vague, sorry. What I meant was that our datastores are simple key-value stores at the moment.
  - eternalny1 7y ago
    > What I meant was that our datastores are simple key-value stores at the moment. And if you go down the route of trying to expand that, then you are no longer a tool for easily building APIs, you are now a database tool also ... and before you know it you are trying to replicate features that RDBMs have had for 40 years. I wouldn't envy trying to have to implement ACID transactions. I realize it seems like this is hyperbole but have you drawn a fine line in the sand at where the features for Dark end? Because if not you will quickly paint yourself into a corner with this.
eternalny1 7y ago
> We deliberately cut relational features for now because they didn't mesh with our language. We want to add joins back You're going to get caught in that age-old trap. Try to make something to "make programming easy", then realize it meets the needs of very few due to lack of features, and then start adding features until it's not "easy" any more it's just programming. And then you realize you're decades behind what people are already using to do programming.
kaens 7y ago
It's very easy to paint yourself into a corner if you haven't figured out at least a plan of attack for these topics up-front, especially if you're starting to ramp up actual use of your product.

cryptonector 7y ago

Right, the demo video lost me at the datastore point. This is a fancy ORM. Now, I do get that making app dev so much easier is a very good thing, but ORMs are a double-edged sword. You need an option for the dev to have much more control over the schema and queries!

eternalny1 7y ago

> How confident are you that your database abstraction is going to fulfill the needs of someone with anything more than a few trivial query workloads? At this point, why does a tool like this need to exist at all? To a full-stack developer writing API's isn't difficult ... and if it is difficult then you are going to need a lot more than a tool like this. Should non-programmers be creating JSON REST APIs in the first place? I don't really see where this is going.