3 ms·

We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost

We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost and performance. The problem you're dedscribing is political: How to get everyone across cultural and language barriers agree upon what the bits mean? There is no technical obstacle to everyone accepting a universal API. In many ways, we're on our way there already. The Internet is full of standards and protocols. Yet it shunts knowledge around the world caring only of a few numbers. It might be something fundamental, like that each aspect of reality has to be attended to by a sentience. Like every person is responsible for maintaining some part of the universal standard. Maybe general-purpose-anything is flawed as a concept, and we must come to understand that what we seek cannot be automated in a meaningful way. Maybe in the age of Turing et al there was so little knowledge it all seemed computable. Now we're drowning in it.

bob1029 5y ago
> The problem you're describing is political This is my experience as well. At the end of the day, most line of business applications can just use SQLite and be done with it. One project, one executable, one database, one model, one machine. Very rarely does the business actually need the kinds of fantastical things that are routinely bandied about on hackernews. Now, that isnt to say these things arent fuckin awesome and that we should stop talking about them, we just need to be realistic about the scope and applicability of our tools. In terms of a universal API, we are kind of doing this in our shop. We are building 1 unified domain model, which is approximately in 3rd normal form. All of our software is required to use this model to represent the facts/state of whatever is happening. Changes to this model go through a semi-formal review process, just like you would expect with an old school SQL setup in a big shop. Our model can be mapped one-way to SQL for purposes of online business rule evaluation or offline reporting. The benefits of having 1 standardized model are really cool after its been in place for a while. You will notice your non-technical project managers developing an awareness for which top-level types may be implicated in certain matters. Having one slow-changing way to represent the business at hand makes it really easy for people to start picking things up. When its scattered to 50 repositories and 10 different languages, this is a bit trickier.
- hypertele-Xii 5y ago
  It's kind of the same argument for/against a universal programming language. The sheer creativity of all the peoples working in the same language, eclipsing the few lost concepts we are unable to represent with our current model. If we want a robust model, we have to include uncertainty within it. As long as computers fail by being sure about the error, a human is always needed to fix the problem. It would have to be an almost philosophically mathematical standard. How we do that when half the world measures physics by completely different metrics than the other one? At least we have Unicode. God bless Unicode. Solving the problem from a computer-linguistic viewpoint: Let's get the language into the computer and we can worry about what it means later!
- hypertele-Xii 5y ago
  It would have to be a programmable, distributed, eventually consistent wiki, that literally everyone can understand. Sounds... fantastical?
lmm 5y ago
> We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost and performance. And they're even more limited than relational databases when it comes to the ability to scale out, or have good consistency semantics without deadlocks. Just implementing a basic mutex on a (general) filesystem is nigh-on impossible. And while AFS is impressive, there still aren't really any true master-master filesystems, much less filesystems that can autoscale.
- hypertele-Xii 5y ago
  Consider the hubris behind a term like autoscale. You're talking about a machine that makes more machines, adapting to its environment, and all that without human intervention. I think it's far more likely we'll just discover that, indeed, no general solution exists, and that humans must take personal responsibility for tending to their digital gardens. We certainly have a lot of spam to weed out.
  - socceroos 5y ago
    I detect wisdom of the eternal variety
  - lmm 5y ago
    That's nice but autoscaling mongodb clusters have been working for what, 5+ years at this point? There are limits to what you can do safely with autoscaling, sure. But a datastore without that capability will never be a universal datastore that can cover all use cases.

bob1029 5y ago

> The problem you're describing is political This is my experience as well. At the end of the day, most line of business applications can just use SQLite and be done with it. One project, one executable, one database, one model, one machine. Very rarely does the business actually need the kinds of fantastical things that are routinely bandied about on hackernews. Now, that isnt to say these things arent fuckin awesome and that we should stop talking about them, we just need to be realistic about the scope and applicability of our tools. In terms of a universal API, we are kind of doing this in our shop. We are building 1 unified domain model, which is approximately in 3rd normal form. All of our software is required to use this model to represent the facts/state of whatever is happening. Changes to this model go through a semi-formal review process, just like you would expect with an old school SQL setup in a big shop. Our model can be mapped one-way to SQL for purposes of online business rule evaluation or offline reporting. The benefits of having 1 standardized model are really cool after its been in place for a while. You will notice your non-technical project managers developing an awareness for which top-level types may be implicated in certain matters. Having one slow-changing way to represent the business at hand makes it really easy for people to start picking things up. When its scattered to 50 repositories and 10 different languages, this is a bit trickier.

hypertele-Xii 5y ago
It's kind of the same argument for/against a universal programming language. The sheer creativity of all the peoples working in the same language, eclipsing the few lost concepts we are unable to represent with our current model. If we want a robust model, we have to include uncertainty within it. As long as computers fail by being sure about the error, a human is always needed to fix the problem. It would have to be an almost philosophically mathematical standard. How we do that when half the world measures physics by completely different metrics than the other one? At least we have Unicode. God bless Unicode. Solving the problem from a computer-linguistic viewpoint: Let's get the language into the computer and we can worry about what it means later!
hypertele-Xii 5y ago
It would have to be a programmable, distributed, eventually consistent wiki, that literally everyone can understand. Sounds... fantastical?

lmm 5y ago

> We have general purpose databases. They're called filesystems. You can literally store anything and everything you like, and hardware is ever-shrinking in cost and performance. And they're even more limited than relational databases when it comes to the ability to scale out, or have good consistency semantics without deadlocks. Just implementing a basic mutex on a (general) filesystem is nigh-on impossible. And while AFS is impressive, there still aren't really any true master-master filesystems, much less filesystems that can autoscale.

hypertele-Xii 5y ago
Consider the hubris behind a term like autoscale. You're talking about a machine that makes more machines, adapting to its environment, and all that without human intervention. I think it's far more likely we'll just discover that, indeed, no general solution exists, and that humans must take personal responsibility for tending to their digital gardens. We certainly have a lot of spam to weed out.
- socceroos 5y ago
  I detect wisdom of the eternal variety
- lmm 5y ago
  That's nice but autoscaling mongodb clusters have been working for what, 5+ years at this point? There are limits to what you can do safely with autoscaling, sure. But a datastore without that capability will never be a universal datastore that can cover all use cases.