7 ms·

F1: A Distributed SQL Database That Scales

131 points by alec 13y ago

dvliman 13y ago
This is not new by any mean...
- jahewson 13y ago
  The paper was only published in conference proceedings yesterday.
  - paulsamways 13y ago
    See http://research.google.com/pubs/pub38125.html http://research.google.com/pubs/pub38125.html
- packetslave 13y ago
  "not new" != "not interesting"
jacques_chester 13y ago
Impressive, as one might expect. Some reactions: -- Secret sauce A lot of the magic of F1 comes from Spanner, the distributed storage system. The name "F1" itself is an allusion to "inheriting" some of the properties of Spanner. -- Hierarchical tables What they call hierarchical tables, I would think best be viewed as one-to-many relationships. In guess they've privileged this model in their storage because that's what a lot of their AdWords schema looks like. -- Change History I like the observation that keeping full histories is relatively straightforward with atomic, granular timestamping and indeed that it should be baked in. Every database schema I've ever worked with always goes through a similar evolutionary cycle: 1. We only need to capture the current state of the model. 2. Wait, we do need to capture historical states of the model. 3. Wait, the model is changed, we need to capture historical states and the models that were current. (You can think of this as taking progressive differentials of incoming transactions). The F1 designers have baked that right into the database, where it belongs. Weak temporal support has long been the sore point in SQL. -- Remote data I was struck by their observation that most database storage engines are built around the concept of seeks and reads, whereas theirs is necessarily built around batching and pipelining over a network. If I am reading them correctly, their engine takes advantage of having multiple copies of data by sending reads to multiple disk nodes and then working from the first copy that is returned.
- drunkpotato 13y ago
  How do hierarchical tables compare to Postgres' hstore? I think recursive structures like lists and trees are one weakness of the relational model, and haven't found a fully satisfying relational answer to the limitations. It seems to me like the assumption of atomic column types is a major weakness inherent in the relational model itself when it comes to recursion. Any thoughts/comments?
  - jacques_chester 13y ago
    > How do hierarchical tables compare to Postgres' hstore? If I am reading them correctly, it's a storage strategy, not a specific "feature" per se. The closest analogy to hstore is that they provide native support for storing and querying protobuf blobs. > I think recursive structures like lists and trees are one weakness of the relational model, and haven't found a fully satisfying relational answer to the limitations. It depends on why you're using trees or graphs. If it's inherent in the data, then modern SQL has recursive queries that make it much easier than the old methods. If it's inherent in the model, you will find it harder. You might need to pick a non-standard approach, such as PostgreSQL's inherited tables. I'd think long and hard before saying it's inherent in the model, by the way. Strictly speaking you can represent the same thing as sets of relations or as a graph; it's better to utilise the strengths of the tool in front of you.
  - zapov 13y ago
    Both Oracle and Postgres have objects and collections. This allows you to build hierarchical structures. Postgres unfortunately doesn't allow for recursive structures ;( If you are interested in how to do it take a look here: https://blog.dsl-platform.com/postgres-bridge-between-worlds/ https://blog.dsl-platform.com/postgres-bridge-between-worlds...
    - jacques_chester 13y ago
      It should be noted that Oracle's object types are a feature of PL/SQL, their imperative programming language. PostgreSQL's concept of object-relational databases is baked into the SQL. Into the logical model of the data itself. Put another way: Oracle doesn't have table inheritance.
fintler 13y ago
I posted this question on StackOverflow regarding TrueTime (used by Spanner) a few days ago and haven't received any responses: <http://stackoverflow.com/questions/18384883/why-is-googles-truetime-api-hard-to-duplicate> http://stackoverflow.com/questions/18384883/why-is-googles-t... However, I think this HN thread seems like it might be a good place to get comments on why my line of reasoning may be incorrect. Does anyone have any thoughts on why building something like Spanner on top of basic Paxos quorums and NTP would be a bad idea?
- mad44 13y ago
  Check this out. http://muratbuffalo.blogspot.com/2013/08/beyond-truetime-using-augmentedtime-for.html http://muratbuffalo.blogspot.com/2013/08/beyond-truetime-usi...
  - fintler 13y ago
    Thanks for the link -- I'm working through it now.
capkutay 13y ago
Can someone enlighten me as to why google doesn't want to make an enterprise play? Many companies tout that they are offering data infrastructures similar to those used at google..why wouldn't google commercialize that technology themselves? It should be pretty obvious as to how that could be of huge financial benefit to the company.
- dragonwriter 13y ago
  > Can someone enlighten me as to why google doesn't want to make an enterprise play? They do. They have a range of commercial, including enterprise, offerings (many of which are based on originally-internal technologies, including their MySQL-based distributed database that preceded F1.) > why wouldn't google commercialize that technology themselves? They probably will, just as they have commercialized their previous internal storage technologies.

131 points by alec 13y ago

dvliman 13y ago

This is not new by any mean...

jahewson 13y ago
The paper was only published in conference proceedings yesterday.
- paulsamways 13y ago
  See http://research.google.com/pubs/pub38125.html http://research.google.com/pubs/pub38125.html
packetslave 13y ago
"not new" != "not interesting"

jacques_chester 13y ago

Impressive, as one might expect. Some reactions: -- Secret sauce A lot of the magic of F1 comes from Spanner, the distributed storage system. The name "F1" itself is an allusion to "inheriting" some of the properties of Spanner. -- Hierarchical tables What they call hierarchical tables, I would think best be viewed as one-to-many relationships. In guess they've privileged this model in their storage because that's what a lot of their AdWords schema looks like. -- Change History I like the observation that keeping full histories is relatively straightforward with atomic, granular timestamping and indeed that it should be baked in. Every database schema I've ever worked with always goes through a similar evolutionary cycle: 1. We only need to capture the current state of the model. 2. Wait, we do need to capture historical states of the model. 3. Wait, the model is changed, we need to capture historical states and the models that were current. (You can think of this as taking progressive differentials of incoming transactions). The F1 designers have baked that right into the database, where it belongs. Weak temporal support has long been the sore point in SQL. -- Remote data I was struck by their observation that most database storage engines are built around the concept of seeks and reads, whereas theirs is necessarily built around batching and pipelining over a network. If I am reading them correctly, their engine takes advantage of having multiple copies of data by sending reads to multiple disk nodes and then working from the first copy that is returned.

drunkpotato 13y ago
How do hierarchical tables compare to Postgres' hstore? I think recursive structures like lists and trees are one weakness of the relational model, and haven't found a fully satisfying relational answer to the limitations. It seems to me like the assumption of atomic column types is a major weakness inherent in the relational model itself when it comes to recursion. Any thoughts/comments?
- jacques_chester 13y ago
  > How do hierarchical tables compare to Postgres' hstore? If I am reading them correctly, it's a storage strategy, not a specific "feature" per se. The closest analogy to hstore is that they provide native support for storing and querying protobuf blobs. > I think recursive structures like lists and trees are one weakness of the relational model, and haven't found a fully satisfying relational answer to the limitations. It depends on why you're using trees or graphs. If it's inherent in the data, then modern SQL has recursive queries that make it much easier than the old methods. If it's inherent in the model, you will find it harder. You might need to pick a non-standard approach, such as PostgreSQL's inherited tables. I'd think long and hard before saying it's inherent in the model, by the way. Strictly speaking you can represent the same thing as sets of relations or as a graph; it's better to utilise the strengths of the tool in front of you.
- zapov 13y ago
  Both Oracle and Postgres have objects and collections. This allows you to build hierarchical structures. Postgres unfortunately doesn't allow for recursive structures ;( If you are interested in how to do it take a look here: https://blog.dsl-platform.com/postgres-bridge-between-worlds/ https://blog.dsl-platform.com/postgres-bridge-between-worlds...
  - jacques_chester 13y ago
    It should be noted that Oracle's object types are a feature of PL/SQL, their imperative programming language. PostgreSQL's concept of object-relational databases is baked into the SQL. Into the logical model of the data itself. Put another way: Oracle doesn't have table inheritance.

fintler 13y ago

I posted this question on StackOverflow regarding TrueTime (used by Spanner) a few days ago and haven't received any responses: <http://stackoverflow.com/questions/18384883/why-is-googles-truetime-api-hard-to-duplicate> http://stackoverflow.com/questions/18384883/why-is-googles-t... However, I think this HN thread seems like it might be a good place to get comments on why my line of reasoning may be incorrect. Does anyone have any thoughts on why building something like Spanner on top of basic Paxos quorums and NTP would be a bad idea?

mad44 13y ago
Check this out. http://muratbuffalo.blogspot.com/2013/08/beyond-truetime-using-augmentedtime-for.html http://muratbuffalo.blogspot.com/2013/08/beyond-truetime-usi...
- fintler 13y ago
  Thanks for the link -- I'm working through it now.

capkutay 13y ago

Can someone enlighten me as to why google doesn't want to make an enterprise play? Many companies tout that they are offering data infrastructures similar to those used at google..why wouldn't google commercialize that technology themselves? It should be pretty obvious as to how that could be of huge financial benefit to the company.

dragonwriter 13y ago
> Can someone enlighten me as to why google doesn't want to make an enterprise play? They do. They have a range of commercial, including enterprise, offerings (many of which are based on originally-internal technologies, including their MySQL-based distributed database that preceded F1.) > why wouldn't google commercialize that technology themselves? They probably will, just as they have commercialized their previous internal storage technologies.