3 ms·

I don't think anyone has solved the issues raised in Out of the Tar Pit in the data storage space. We still spend a ton of effort, perhaps more than ever before

by gfodor 9y ago

I don't think anyone has solved the issues raised in Out of the Tar Pit in the data storage space. We still spend a ton of effort, perhaps more than ever before, wrangling incidental complexity. In 2018, I should be able to define a simple relational schema in 30 seconds and start using it with effectively zero constraints around data access patterns, transaction volume, and data scale, and with no design decisions needing to be made around these bits of incidental complexity. There should be one gigantic knob I can turn to spend more money to increase more resource consumption of the system, and it should be damned cheap. We are so far from that reality that it seems obvious there is still a ton of work to do. You can see bits and pieces of solutions but if your product asks me to define indexes, tune queries, determine sharding keys, write against computer-centric (vs human-centric) APIs, cannot deal with change easily, requires babysitting, or falls over after hitting some incidental tipping point relative to the resources available to a single machine, you've not hit it. I have not used Google Spanner but at 30k/ft it seems like the closest thing out there to the idealized case -- but being closed source and centralized it is not really a "solved problem" imho.

foobarchu 9y ago
This has always been my biggest complaint with Cassandra. It performs and works fantastic if you have the right usage patterns, but if you don't fit into that pattern then you need to mold your usage so that it will fit, or risk the consequences. Using Cassandra effectively is pretty much impossible without knowledge of it's internals and consideration of what effect your usage patterns will have on those internals, which IMO kind of defeats the purpose of a database (keep my data, give it back when I ask).
chubot 9y ago
Personally, I would like something like sqlite that you can sync like git. Though it probably should enforce its schema more than sqlite does. I would use it for replicated / distributed storage for data frames (e.g. R or Pandas), which is somewhat related to my other request [1] The difference here is that there is structure to the files. [1] https://news.ycombinator.com/item?id=16394048 https://news.ycombinator.com/item?id=16394048
ariwilson 9y ago
Doesn't this exist? Isn't it called hiring consultants / an engineering team?
mamcx 9y ago
I totally agree. I'm toying with the idea of build a relational-centric language (and maybe storage) because I think the same. I wanna to create a table/relation as easy as: customers = [id:int, name:str; 1 "jhon"; 2 "doe"] in memory. Other things as triggers, PK, FK, views, etc obscure the simplicity of the relational model and make people say weird stuff as "relational database not scale" or "are too inflexible". I think exist a LOT of easy things to enrich the model and make it more usefull. For example, is ok to say: inv = [id:int, lines:Lines; 1 [id:1, qty:1, price:$10]; 2 [id:1, qty:3, price:$40];] Is totally ok to nest relations on relations (this alone could make wonders for ORM :) ).
mamcx 9y ago
> I should be able to define a simple relational schema in 30 seconds and start using it with effectively zero constraints around data access patterns, transaction volume, and data scale, and with no design decisions needing to be made around these bits of incidental complexity Oh, this also is something that need a better way! RDBMS are too coupled. I remember to ask if I can ditch the SQL parsing of sqlite and call directly the storage (so I can put my own query engine on top) and that was like if I'm nuts!. I think is possible to build a RDBMS that is semi-plugable. To make things as swap parts of the engine on "user land". If someone for example wanna create a new kind of index is must be as easy as write: fun CoolIndex.get(key) -> Value and plug it into the engine just fine. I think do something like flask/django-esque framework where is possible to have custom fields, validations, middlewares, etc on top of a core storage layer and a default implementation. So, instead of go with redis or something else I could build my "redis-like" api INSIDE the engine and get the advantages of locality, integrity, etc. ie: See a RDBMS as an API Backend. This could be model (instead of MVC in common web frameworks) as CQRS or something similar.
- Fins 9y ago
  Wasn't that what Btrieve was doing back in the 90s? Or was it PervasiveSQL on top of Btrieve engine you could access directly?..
noir_lord 9y ago
This leads me to one of my constant bugbears. Why is RDBMS tooling so completely shite. I have an IDE that can index several million lines of code and apply accurately timely context dependent intelligent code completion - I have yet to find a good way to debug a MySQL stored procedure (I hate them, I inherited them, except for very simple use cases they are getting slow replaced).
TheHam 9y ago
AWS just release AppSync. You can define a relational GraphQL Schema and hook up to a number of sources to feed data to it. Then, AppSync will automatically turn your schema into a GraphQL API where clients can make queries, mutations, and subscribe to data changes with no constraints regarding data access patterns. Subscriptions scales without any work from you. All of these operations use the relational schema, which is self-documenting. You can change the schema any time you want. Using the graphQL API requires no knowledge of the backend data store. Therefore, clients and consumers of your API do not need to know about shards, indices, SQL queries, etc. If you have an existing DynamoDB table, you can automatically generate a GraphQL schema/API from it too. https://aws.amazon.com/appsync/ https://aws.amazon.com/appsync/ disclaimer: i work on aws appsync
- marsanyi 9y ago
  This looks pretty nice.
- davidjnelson 9y ago
  This looks awesome, except that it's still in preview. Seems a bit early to bet on it - any idea when there will be assurances that it will continue to be supported long term?

by gfodor 9y ago

foobarchu 9y ago

This has always been my biggest complaint with Cassandra. It performs and works fantastic if you have the right usage patterns, but if you don't fit into that pattern then you need to mold your usage so that it will fit, or risk the consequences. Using Cassandra effectively is pretty much impossible without knowledge of it's internals and consideration of what effect your usage patterns will have on those internals, which IMO kind of defeats the purpose of a database (keep my data, give it back when I ask).

chubot 9y ago

Personally, I would like something like sqlite that you can sync like git. Though it probably should enforce its schema more than sqlite does. I would use it for replicated / distributed storage for data frames (e.g. R or Pandas), which is somewhat related to my other request [1] The difference here is that there is structure to the files. [1] https://news.ycombinator.com/item?id=16394048 https://news.ycombinator.com/item?id=16394048

ariwilson 9y ago

Doesn't this exist? Isn't it called hiring consultants / an engineering team?

mamcx 9y ago

I totally agree. I'm toying with the idea of build a relational-centric language (and maybe storage) because I think the same. I wanna to create a table/relation as easy as: customers = [id:int, name:str; 1 "jhon"; 2 "doe"] in memory. Other things as triggers, PK, FK, views, etc obscure the simplicity of the relational model and make people say weird stuff as "relational database not scale" or "are too inflexible". I think exist a LOT of easy things to enrich the model and make it more usefull. For example, is ok to say: inv = [id:int, lines:Lines; 1 [id:1, qty:1, price:$10]; 2 [id:1, qty:3, price:$40];] Is totally ok to nest relations on relations (this alone could make wonders for ORM :) ).

mamcx 9y ago

> I should be able to define a simple relational schema in 30 seconds and start using it with effectively zero constraints around data access patterns, transaction volume, and data scale, and with no design decisions needing to be made around these bits of incidental complexity Oh, this also is something that need a better way! RDBMS are too coupled. I remember to ask if I can ditch the SQL parsing of sqlite and call directly the storage (so I can put my own query engine on top) and that was like if I'm nuts!. I think is possible to build a RDBMS that is semi-plugable. To make things as swap parts of the engine on "user land". If someone for example wanna create a new kind of index is must be as easy as write: fun CoolIndex.get(key) -> Value and plug it into the engine just fine. I think do something like flask/django-esque framework where is possible to have custom fields, validations, middlewares, etc on top of a core storage layer and a default implementation. So, instead of go with redis or something else I could build my "redis-like" api INSIDE the engine and get the advantages of locality, integrity, etc. ie: See a RDBMS as an API Backend. This could be model (instead of MVC in common web frameworks) as CQRS or something similar.

Fins 9y ago
Wasn't that what Btrieve was doing back in the 90s? Or was it PervasiveSQL on top of Btrieve engine you could access directly?..

noir_lord 9y ago

This leads me to one of my constant bugbears. Why is RDBMS tooling so completely shite. I have an IDE that can index several million lines of code and apply accurately timely context dependent intelligent code completion - I have yet to find a good way to debug a MySQL stored procedure (I hate them, I inherited them, except for very simple use cases they are getting slow replaced).

TheHam 9y ago

AWS just release AppSync. You can define a relational GraphQL Schema and hook up to a number of sources to feed data to it. Then, AppSync will automatically turn your schema into a GraphQL API where clients can make queries, mutations, and subscribe to data changes with no constraints regarding data access patterns. Subscriptions scales without any work from you. All of these operations use the relational schema, which is self-documenting. You can change the schema any time you want. Using the graphQL API requires no knowledge of the backend data store. Therefore, clients and consumers of your API do not need to know about shards, indices, SQL queries, etc. If you have an existing DynamoDB table, you can automatically generate a GraphQL schema/API from it too. https://aws.amazon.com/appsync/ https://aws.amazon.com/appsync/ disclaimer: i work on aws appsync

marsanyi 9y ago
This looks pretty nice.
davidjnelson 9y ago
This looks awesome, except that it's still in preview. Seems a bit early to bet on it - any idea when there will be assurances that it will continue to be supported long term?