7 ms·

Announcing SurrealDB 1.0

63 points by tinrab 3y ago

robertlagrant 3y ago
> Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself thinks of Oracle APEX thinks of IBM AS400 shudders
- beders 3y ago
  Hahaha, loved coding RPG on the AS400. The file system on top of the database layer! Awesome stuff
fredthedeadhead 3y ago
Fireship did a 'SurrealDB in 100 seconds' in case anyone would like a quick video summary: https://youtu.be/C7WFwgDRStM?si=va17q7zagn4lIoi3 https://youtu.be/C7WFwgDRStM?si=va17q7zagn4lIoi3
esafak 3y ago
I like where they are going. Are there benchmarks out yet?
dwb 3y ago
Not Open Source (in an OSI way): https://github.com/surrealdb/surrealdb/blob/ed60a35b9b539e1b4be0725d909ba9d83e45ce07/LICENSE https://github.com/surrealdb/surrealdb/blob/ed60a35b9b539e1b...
- benrutter 3y ago
  Is it claiming anywhere to be open source? I've heard an interview with the creators and they seemed pretty open about which aspects where open source and which aspects where shut down? I could be missing something from their branding but I'd rather see projects with non-open source licenses from the beginning that bait-and-switch license change when they want to become profitable. (Although, I'd much rather just see a fully open source project)
  - dwb 3y ago
    Sure. I wasn't really trying to insinuate anything, although I think in a more-ideal world the licensing arrangements of all projects would be in a prominent place, especially if they use a "Star us on GitHub" call-to-action. It's a vital part of evaluating this sort of thing.
  - khuey 3y ago
    > Is it claiming anywhere to be open source? They have a page on their website where they claim to be "an open source company" (https://surrealdb.com/opensource https://surrealdb.com/opensource)
    - dwb 3y ago
      Certainly debatable copy, but in total fairness that page is a list of actually-Open-Source licensed projects.
- anonzzzies 3y ago
  It would indeed be better to state that clearly, then I don’t have to check it (our company has a policy which won’t get this passed).
canadiantim 3y ago
My biggest concern with SurrealDB is the license, otherwise I'd be very interested in adopting it. I know the license is okay, but from my reading of it, sounds like if SurreaDB really wanted to shut you down they could. Otherwise, I find the architecture of it to be beautiful and compelling.
- gpm 3y ago
  Unless you're operating a "database service", a well defined term (in the license) that excludes almost every use-case A “Database Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties. I don't see how SurrealDB could shut you down?
  - auggierose 3y ago
    So if you create schemas on the fly, you cannot? Why would I use such a db?
    - sroussey 3y ago
      Right, so could I use it to creat an Airtable clone?
      - fiddlerwoaroof 3y ago
        Generating a table schema per airtable spreadsheet sounds like an anti-pattern to me.
        fiddlerwoaroof 3y ago
        That being said, I generally agree: I’ve been bitten by enough database products that are some startup’s main product to really trust them. I’d rather use a database that’s either maintained by something like the Apache Foundation (Druid, Pinot, etc.) or one that’s a side project of some big company (FoundationDB) or one that’s been around long enough to be unlikely to surprise us when the team is acquihired.
stallmanwasrigh 3y ago
It benches very very slow and no options to horizontally scale outside of manual sharding. API has a cool design though. The dev team needs to look at perf before the complexity hole gets too insane if it hasn't already.
- jitl 3y ago
  What’s your source on the benchmark thing? It seems like they have a lot of backend KV store options including FoundationDB and TiKV which are both distributed, so I’d be curious how the backends perform relative to one another. https://surrealdb.com/features https://surrealdb.com/features
vlovich123 3y ago
Is there any trick to implement live updates in a scaleable way? In the limit for a naive implementation, every mutation would have to check every subscriber to see if the change is relevant which seems like it would cause large bottlenecks for writes.
- jitl 3y ago
  One of the guys behind Convex explained the rough idea; I hope I do it justice. The strategy is to break the subscription up into listens based on the read-set ranges of the query. Then you put the individual read-set ranges into a system table that you index. Finally when writes happen, you notify all queries who's read-set intersects with the write-set. For example, say I have a query `SELECT * from block WHERE parent_id = X AND last_modified_at > Y`. This query might create two subscriptions for its read sets: { query_id: Q, table: block, column: parent_id, min: X, max: X } { query_id: Q, table: block, column: last_modified_at, min: Y, max: Infinity } Now a write happens: `INSERT INTO block { id: Z, parent_id: X, last_modified_at: Y + 10, title: "hi" }` We can find our subscriptions to notify by doing queries like these: SELECT query_id FROM subscription WHERE table = block AND ( (column = id AND min <= new_block.id AND max >= new_block.id ) OR (column = parent_id AND min <= new_block.parent_id AND max >= new_block.parent_id) OR (column = last_modified_at AND min <= new_block.last_modified_at AND max >= new_block.last_modified_at) ) Then you notify all those query_ids. I'm sure there's a lot of details and optimizations you can do on top of this; finding the right read sets seems pretty tricky for complex queries. Plus stuff like batching/debouncing, etc.
  - vlovich123 3y ago
    Is there a talk you can point me to? There’s several challenges with this approach that come up for me (which unless I’m mistaken is the naive approach of checking each write against a notification set). The first is that maintaining the read set seems very expensive since it scales with the number of live queries installed. In a multi tenant DB, that seems tricky to manage in terms of memory used. The second is that the computation of intersection with a read range seems potentially very expensive. Imagine you have a string column. You now have to do a string comparison for every insertion that might hypothetically match the query. Finally, computing that read set correctly seems challenging (as you mention) and it’s not immediately clear to me it’s always tractable (eg could you do this with arbitrary complicated joins?). Additionally, in your description, each write has an implicit table scan to do the select to find the intersection. That will tank write throughput even for small total intersection sets (eg there’s a reason LSM databases do deletes by inserting a tombstone instead of checking whether the data exists first, same with the merge operator in RocksDB - a db read in the write path significantly kills performance)
jauntywundrkind 3y ago
Live queries & change-feeds! Reactive data stores are great; neat to see.
omneity 3y ago
Strong vibes from RethinkDB. I hope SurrealDB will have a different fate, I'm a big big fan of live queries and change feeds and don't like most existing implementations.
- drdaeman 3y ago
  This immediately sprung into my mind. Already got bitten by RethinkDB (trusted it too much to do some logic, rewrite was painful), so I’d rather have reasonably easily replaceable third-party components than depend on something unique, especially without any guarantees.
- grounder 3y ago
  Can you give a quick elaboration on what you mean about the fate of RethinkDB? I'm not familiar with it.
  - omneity 3y ago
    The project didn't really take off as a business and the team joined Stripe. See the "History" section. https://en.wikipedia.org/wiki/RethinkDB https://en.wikipedia.org/wiki/RethinkDB
ThinkBeat 3y ago
This feels like satire. and written by someone who should have spent more time studying the history of database systems. > Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself This is not a good idea. It has been done many times and never quite caught on because it is not a good idea. From a security perspective it is a nightmare. Or if you do put all the correct isolation around the code to protect the database, then you have basically created an "app server" (old term) inside a database, and it would happily, run outside of the database since in essence it is already doing so.
- anonzzzies 3y ago
  > never quite caught on It was quite normal in 70s 80s and even 90s; all the ms sql, db2, oracle and as400 systems I encountered in those days had all or almost all logic as stored procs. Very large ones.
  - threeseed 3y ago
    During the Java era the standard quickly became ORM instead of stored procedures. Largely thanks to NeXT showing the way with EOF (the original ORM).
    - foobiekr 3y ago
      ORMs basically exist because people doing coding refuse to learn the relational model and want to program with OO instead. Most of the time they are, at best, irrelevant but eventually become expensive, slow, complex and painful. Everyone in coding knows, once they get a lot of experience, that translation/mapping layers are a waste of cycles, memory, and are complex and where assumptions mismatch results in tons of bugs, and that applies to ORMs in partcular.
      - happymellon 3y ago
        I've just spent the past week refactoring code that was too slow because joins were being performed by a loop in the code rather than just doing a join in the first place. Replacing 3 classes, and numerous methods with two 6 line queries that does the same task in a thousandth of the time is quite satisfying.
zamazan4ik 3y ago
1.0 version but https://github.com/surrealdb/surrealdb/issues/1548 https://github.com/surrealdb/surrealdb/issues/1548 is still open :)
- obinnaspeaks 3y ago
  A lot of the issues here have been resolved in documentation. For example there’s now new docs on permissions and deployment also more information on backups. Others are currently being worked ;).. The issue is still open as we haven’t resolved all of it yet so makes a good reference.
h1fra 3y ago
I feel like databases are the new JS framework, there is a new one every week.
pavlov 3y ago
Is it a database where the only date/time type is a melting clock, and a join is defined as “the chance encounter of a sewing machine and an umbrella on an operating table”?
pryelluw 3y ago
“Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself.” Don’t need to. Majority of my work is de-coupling systems “designed” around this idea. Why can’t the database be the database, the backend be the backend, etc ? You can’t have a toolbox full of multi use tools. Sometimes you just need a fucking Phillips head screwdriver.
dgudkov 3y ago
While I applaud any attempts to innovate in this area, I'd be more interested in seeing the opposite approach - integrating persistent storage in a programming language. If we already have default implementations of dictionaries, linked lists, and other in-memory data structures in programming languages, why not have default implementations of permanent data structures such as object collections and KV-stores?
- PedroBatista 3y ago
  Check this out: https://microstream.one/products/microstream-for-java/ https://microstream.one/products/microstream-for-java/ Io also has persistence baked into it: https://iolanguage.org/reference/ https://iolanguage.org/reference/ But I'm sure for most of the HN crowd one has Java in the URL and the other is a esoteric cool language from the early 00's so.. not practical. :)
- thunderbong 3y ago
  There have been many attempts over the years. There's Prevayler [0], in Java, and Madeleine [1], in Ruby. Neither of them became wildly popular. I worked with Madeleine for a few years and while the performance was amazing, RDBM systems are ahead because of the many years of standardisation, tooling and capabilities (e.g. migrations) The other very important thing is - DB latency is an issue only in the hands of a novice. In the real world, network latency is a couple of magnitudes higher. [0]: https://github.com/prevayler/prevayler https://github.com/prevayler/prevayler [1]: https://github.com/ghostganz/madeleine https://github.com/ghostganz/madeleine

63 points by tinrab 3y ago

> Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself thinks of Oracle APEX thinks of IBM AS400 shudders

beders 3y ago
Hahaha, loved coding RPG on the AS400. The file system on top of the database layer! Awesome stuff

fredthedeadhead 3y ago

Fireship did a 'SurrealDB in 100 seconds' in case anyone would like a quick video summary: https://youtu.be/C7WFwgDRStM?si=va17q7zagn4lIoi3 https://youtu.be/C7WFwgDRStM?si=va17q7zagn4lIoi3

esafak 3y ago

I like where they are going. Are there benchmarks out yet?

dwb 3y ago

Not Open Source (in an OSI way): https://github.com/surrealdb/surrealdb/blob/ed60a35b9b539e1b4be0725d909ba9d83e45ce07/LICENSE https://github.com/surrealdb/surrealdb/blob/ed60a35b9b539e1b...

benrutter 3y ago
Is it claiming anywhere to be open source? I've heard an interview with the creators and they seemed pretty open about which aspects where open source and which aspects where shut down? I could be missing something from their branding but I'd rather see projects with non-open source licenses from the beginning that bait-and-switch license change when they want to become profitable. (Although, I'd much rather just see a fully open source project)
- dwb 3y ago
  Sure. I wasn't really trying to insinuate anything, although I think in a more-ideal world the licensing arrangements of all projects would be in a prominent place, especially if they use a "Star us on GitHub" call-to-action. It's a vital part of evaluating this sort of thing.
- khuey 3y ago
  > Is it claiming anywhere to be open source? They have a page on their website where they claim to be "an open source company" (https://surrealdb.com/opensource https://surrealdb.com/opensource)
  - dwb 3y ago
    Certainly debatable copy, but in total fairness that page is a list of actually-Open-Source licensed projects.
anonzzzies 3y ago
It would indeed be better to state that clearly, then I don’t have to check it (our company has a policy which won’t get this passed).

canadiantim 3y ago

My biggest concern with SurrealDB is the license, otherwise I'd be very interested in adopting it. I know the license is okay, but from my reading of it, sounds like if SurreaDB really wanted to shut you down they could. Otherwise, I find the architecture of it to be beautiful and compelling.

gpm 3y ago
Unless you're operating a "database service", a well defined term (in the license) that excludes almost every use-case A “Database Service” is a commercial offering that allows third parties (other than your employees and contractors) to access the functionality of the Licensed Work by creating tables whose schemas are controlled by such third parties. I don't see how SurrealDB could shut you down?
- auggierose 3y ago
  So if you create schemas on the fly, you cannot? Why would I use such a db?
  - sroussey 3y ago
    Right, so could I use it to creat an Airtable clone?
    - fiddlerwoaroof 3y ago
      Generating a table schema per airtable spreadsheet sounds like an anti-pattern to me.
      - fiddlerwoaroof 3y ago
        That being said, I generally agree: I’ve been bitten by enough database products that are some startup’s main product to really trust them. I’d rather use a database that’s either maintained by something like the Apache Foundation (Druid, Pinot, etc.) or one that’s a side project of some big company (FoundationDB) or one that’s been around long enough to be unlikely to surprise us when the team is acquihired.

stallmanwasrigh 3y ago

It benches very very slow and no options to horizontally scale outside of manual sharding. API has a cool design though. The dev team needs to look at perf before the complexity hole gets too insane if it hasn't already.

jitl 3y ago
What’s your source on the benchmark thing? It seems like they have a lot of backend KV store options including FoundationDB and TiKV which are both distributed, so I’d be curious how the backends perform relative to one another. https://surrealdb.com/features https://surrealdb.com/features

vlovich123 3y ago

Is there any trick to implement live updates in a scaleable way? In the limit for a naive implementation, every mutation would have to check every subscriber to see if the change is relevant which seems like it would cause large bottlenecks for writes.

jitl 3y ago
One of the guys behind Convex explained the rough idea; I hope I do it justice. The strategy is to break the subscription up into listens based on the read-set ranges of the query. Then you put the individual read-set ranges into a system table that you index. Finally when writes happen, you notify all queries who's read-set intersects with the write-set. For example, say I have a query `SELECT * from block WHERE parent_id = X AND last_modified_at > Y`. This query might create two subscriptions for its read sets: { query_id: Q, table: block, column: parent_id, min: X, max: X } { query_id: Q, table: block, column: last_modified_at, min: Y, max: Infinity } Now a write happens: `INSERT INTO block { id: Z, parent_id: X, last_modified_at: Y + 10, title: "hi" }` We can find our subscriptions to notify by doing queries like these: SELECT query_id FROM subscription WHERE table = block AND ( (column = id AND min <= new_block.id AND max >= new_block.id ) OR (column = parent_id AND min <= new_block.parent_id AND max >= new_block.parent_id) OR (column = last_modified_at AND min <= new_block.last_modified_at AND max >= new_block.last_modified_at) ) Then you notify all those query_ids. I'm sure there's a lot of details and optimizations you can do on top of this; finding the right read sets seems pretty tricky for complex queries. Plus stuff like batching/debouncing, etc.
- vlovich123 3y ago
  Is there a talk you can point me to? There’s several challenges with this approach that come up for me (which unless I’m mistaken is the naive approach of checking each write against a notification set). The first is that maintaining the read set seems very expensive since it scales with the number of live queries installed. In a multi tenant DB, that seems tricky to manage in terms of memory used. The second is that the computation of intersection with a read range seems potentially very expensive. Imagine you have a string column. You now have to do a string comparison for every insertion that might hypothetically match the query. Finally, computing that read set correctly seems challenging (as you mention) and it’s not immediately clear to me it’s always tractable (eg could you do this with arbitrary complicated joins?). Additionally, in your description, each write has an implicit table scan to do the select to find the intersection. That will tank write throughput even for small total intersection sets (eg there’s a reason LSM databases do deletes by inserting a tombstone instead of checking whether the data exists first, same with the merge operator in RocksDB - a db read in the write path significantly kills performance)

jauntywundrkind 3y ago

Live queries & change-feeds! Reactive data stores are great; neat to see.

omneity 3y ago

Strong vibes from RethinkDB. I hope SurrealDB will have a different fate, I'm a big big fan of live queries and change feeds and don't like most existing implementations.

drdaeman 3y ago
This immediately sprung into my mind. Already got bitten by RethinkDB (trusted it too much to do some logic, rewrite was painful), so I’d rather have reasonably easily replaceable third-party components than depend on something unique, especially without any guarantees.
grounder 3y ago
Can you give a quick elaboration on what you mean about the fate of RethinkDB? I'm not familiar with it.
- omneity 3y ago
  The project didn't really take off as a business and the team joined Stripe. See the "History" section. https://en.wikipedia.org/wiki/RethinkDB https://en.wikipedia.org/wiki/RethinkDB

ThinkBeat 3y ago

This feels like satire. and written by someone who should have spent more time studying the history of database systems. > Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself This is not a good idea. It has been done many times and never quite caught on because it is not a good idea. From a security perspective it is a nightmare. Or if you do put all the correct isolation around the code to protect the database, then you have basically created an "app server" (old term) inside a database, and it would happily, run outside of the database since in essence it is already doing so.

anonzzzies 3y ago
> never quite caught on It was quite normal in 70s 80s and even 90s; all the ms sql, db2, oracle and as400 systems I encountered in those days had all or almost all logic as stored procs. Very large ones.
- threeseed 3y ago
  During the Java era the standard quickly became ORM instead of stored procedures. Largely thanks to NeXT showing the way with EOF (the original ORM).
  - foobiekr 3y ago
    ORMs basically exist because people doing coding refuse to learn the relational model and want to program with OO instead. Most of the time they are, at best, irrelevant but eventually become expensive, slow, complex and painful. Everyone in coding knows, once they get a lot of experience, that translation/mapping layers are a waste of cycles, memory, and are complex and where assumptions mismatch results in tons of bugs, and that applies to ORMs in partcular.
    - happymellon 3y ago
      I've just spent the past week refactoring code that was too slow because joins were being performed by a loop in the code rather than just doing a join in the first place. Replacing 3 classes, and numerous methods with two 6 line queries that does the same task in a thousandth of the time is quite satisfying.

zamazan4ik 3y ago

1.0 version but https://github.com/surrealdb/surrealdb/issues/1548 https://github.com/surrealdb/surrealdb/issues/1548 is still open :)

obinnaspeaks 3y ago
A lot of the issues here have been resolved in documentation. For example there’s now new docs on permissions and deployment also more information on backups. Others are currently being worked ;).. The issue is still open as we haven’t resolved all of it yet so makes a good reference.

h1fra 3y ago

I feel like databases are the new JS framework, there is a new one every week.

pavlov 3y ago

Is it a database where the only date/time type is a melting clock, and a join is defined as “the chance encounter of a sewing machine and an umbrella on an operating table”?

pryelluw 3y ago

“Imagine a world where the majority of your backend logic is seamlessly embedded within the database itself.” Don’t need to. Majority of my work is de-coupling systems “designed” around this idea. Why can’t the database be the database, the backend be the backend, etc ? You can’t have a toolbox full of multi use tools. Sometimes you just need a fucking Phillips head screwdriver.

dgudkov 3y ago

While I applaud any attempts to innovate in this area, I'd be more interested in seeing the opposite approach - integrating persistent storage in a programming language. If we already have default implementations of dictionaries, linked lists, and other in-memory data structures in programming languages, why not have default implementations of permanent data structures such as object collections and KV-stores?

PedroBatista 3y ago
Check this out: https://microstream.one/products/microstream-for-java/ https://microstream.one/products/microstream-for-java/ Io also has persistence baked into it: https://iolanguage.org/reference/ https://iolanguage.org/reference/ But I'm sure for most of the HN crowd one has Java in the URL and the other is a esoteric cool language from the early 00's so.. not practical. :)
thunderbong 3y ago
There have been many attempts over the years. There's Prevayler [0], in Java, and Madeleine [1], in Ruby. Neither of them became wildly popular. I worked with Madeleine for a few years and while the performance was amazing, RDBM systems are ahead because of the many years of standardisation, tooling and capabilities (e.g. migrations) The other very important thing is - DB latency is an issue only in the hands of a novice. In the real world, network latency is a couple of magnitudes higher. [0]: https://github.com/prevayler/prevayler https://github.com/prevayler/prevayler [1]: https://github.com/ghostganz/madeleine https://github.com/ghostganz/madeleine