6 ms·

hey hn, supabase ceo here we're excited about this release - it adds a few neat features to our Realtime server[0]. For those who don't read the post (it's qui

hey hn, supabase ceo here we're excited about this release - it adds a few neat features to our Realtime server[0]. For those who don't read the post (it's quite long), our Realtime server was initially built for subscribing to Postgres changes via websockets. We've added Broadcast functionality (for "ephemeral" messages that don't need to be persisted to Postgres) and Presence (similar to Broadcasts, but it retains an eventually-consistent state between users). We were planning a larger write-up with all the technical details but it became a bit long. Some highlights: - Realtime is built with Elixir an Phoenix[1] - We've moved to a single, global cluster - We're using `Phoenix.Tracker` to globally distribute the Erlang PIDs of all of our Postgres connections. This guarantees there is only a single connection to each database. - We've implemented a "leaky-bucket" rate-limiter using Erlang's `:counters` module[3] for "per tenant" rate-limiting. We'll move this to a global rate-limiter in the future - We're currently seeing ~2000 concurrent connected users - it will be interesting to see what this spikes up to now that this is on HN It's still in heavy development, and we'll make sure to detail all the choices we take as we continue solve problems. A few of the Realtime team will be here answer questions - @chasers, @wenbo, @abc3erl. [0] Realtime: https://github.com/supabase/realtime https://github.com/supabase/realtime [1] Phoenix Framework: https://www.phoenixframework.org/ https://www.phoenixframework.org/ [2] rate limiter: https://en.wikipedia.org/wiki/Leaky_bucket https://en.wikipedia.org/wiki/Leaky_bucket [3] :counters: https://www.erlang.org/doc/man/counters.html https://www.erlang.org/doc/man/counters.html

guykdm 4y ago
Hey Kiwi, any chance of getting temporal.io style features for supabase edge functions? would love to have a one-stop-shop solution for all my real and not-real time needs.
- kiwicopple 4y ago
  (I'm assuming you want a queue/workflow engine) Temporal has done an incredible job of ensuring there will be no side-effects (even with external systems), so I doubt we'll be able to do something as extensive as they have. That said, we will probably build a simple Postgres queue/workflow engine (similar to pgboss[0]). It will live in the Postgres database (as an extension), but will be nicely integrated with Edge Functions. We haven't started this yet, so I don't have any timelines. [0] pgboss: https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss
  - joisig 4y ago
    Take a look at Oban, it very well might fit the bill. https://github.com/sorentwo/oban https://github.com/sorentwo/oban
    - kiwicopple 4y ago
      we used Oban for a very early product (which we put on ice) called Supabase Workflows: https://supabase.com/blog/supabase-workflows https://supabase.com/blog/supabase-workflows The reason we paused it was the realisation that it's far more powerful to co-locate this sort of product with the data. In this case, Workflows would be more powerful as a Postgres extension - you can store the workflows into your database, create them using SQL, and the workflows would have direct access to your operational data. We'll probably re-write this using pgx[0] at some stage (if anyone is reading this and wants to attempt this, please reach out). [0] pgx: https://github.com/tcdi/pgx https://github.com/tcdi/pgx
- swyx 4y ago
  have you tried Azure Durable Functions? just curious if youve evaluated all the durable workflow solutions out there and what you came away with
ForrestN 4y ago
Love to see Barbz running/naming cool things!! Why be simple when you could be stunning???
cpursley 4y ago
Any chance of open sourcing the rate limiter bit for Elixir?
- maxloh 4y ago
  Isn't it open source? https://github.com/supabase/realtime https://github.com/supabase/realtime
  - cpursley 4y ago
    As a stand-alone Elixir package.
    - nextaccountic 4y ago
      Isn't the whole supabase open source? Oh actually you mean, you want to run this without supabase? The code is already open source, and anyone can extract it into a more reusable package. And it's developed in the open, anyone can fork or send PRs or anything. That's what open sourcing a piece of code is really about.
      - cpursley 4y ago
        Yeah, I already extracted the WAL listening bit out (which is now far behind the realtime changes). And for that reason would rather have those pieces as individual packages for the wider Elixir community and so that it's easier to contribute back. https://github.com/cpursley/walex https://github.com/cpursley/walex
- chasers 4y ago
  Yeah I plan to. Have some things to prep next week for ElixirConf but am going to get back to it after that.
samokhvalov 4y ago
It would be great to see monitoring of this thing under load – I mean the Postgres part of it, how it's loaded, how many connections used, query analysis, etc.
- kiwicopple 4y ago
  cool - we can include that in the next blog post
tandyman 4y ago
LOVE Supabase. Building a project over the last several months for a client and very excited to launch the beta soon. Thanks for all you are doing...very exciting stuff. Creating dev and staging environments is a feature that I am most looking forward to. Hopefully that is on the road map.
fintechjock 4y ago
I have almost zero coding experience. Seriously, I've read a couple of books and I've taken a couple of MOOCs and that is about it. I've never 'launched' any project of my own, and I have zero front-end or back-end experience. I say that because I just used Supabase for the first time, and I was able to build and launch the Slack clone in ~15 minutes. Seriously cool. Thank you for sharing
- kiwicopple 4y ago
  That's awesome. I've shared your comment with the team - they will really enjoy your feedback. Good luck building
pixelbyindex 4y ago
Hi Supabase & Team What are the chances of an official C# library being in the works? I think this would greatly fill the needs for unity devs. I use firebase on almost all game backends because of how quickly I can get set up with auth and a db. I used supabase in my latest project and it served my needs but there were a few difficulties to overcome. The greatest shortcoming was the lack of real time support (C#), and I had to do everything via REST api. I am planning on publishing the game as OSS and making some learning videos about it
- kiwicopple 4y ago
  They're unofficial, but @acupofjose has been working hard on some community C# libraries: https://github.com/supabase-community/#client-libraries https://github.com/supabase-community/#client-libraries They won't support the functionality we outline in our blog post yet (we are still figuring it out on our official JS libraries), but I'll send him your message. As an aside, if anyone is reading this and wants to help maintain the C# libraries, please reach out. [0] acupofjose: https://github.com/orgs/supabase-community/people/acupofjose https://github.com/orgs/supabase-community/people/acupofjose
jahewson 4y ago
Given that Phoenix.Tracker is eventually consistent, how does it guarantee that there's only one Postgres connection?
- chasers 4y ago
  Yeah the database ultimately guarantees that. This lets us distribute the fact that a connection exists (in the form of a PID) so that we don't have to go back to the db on every connection request. This helps quite a bit in the case of a lot of connections coming in very quickly, especially when those connections are geographically far from the database or the database is under load from some other system. If no connection exists yet, you're right, two connections could try to establish themselves at the same time. In which case, whoever gets to the db first wins. But then the loser is still connected to PubSub, so they'll now start getting changes just the same.
timwis 4y ago
Really exciting! I’ve been checking your blog every couple weeks waiting for this release. Does the global cluster architecture make it much harder to self-host supabase? Not that I plan to, but having the promise of portability is really important.
matthewfcarlson 4y ago
I’m working on a live trivia type app. What’s the latency on the new real-time stuff like? I’m currently rolling my own thing with socket.io and
- lawik 4y ago
  I can speak some to the Elixir tech it is built on. It should be low, even with load. The Erlang VM is very good with concurrency and keeping latencies consistent and low. The presence feature could be slower since that propagates a bit more complexly than the broadcasts would. Of course the latency can go up with load but it shouldn't get very spiky thanks to the preemptive multitasking.

guykdm 4y ago

Hey Kiwi, any chance of getting temporal.io style features for supabase edge functions? would love to have a one-stop-shop solution for all my real and not-real time needs.

kiwicopple 4y ago
(I'm assuming you want a queue/workflow engine) Temporal has done an incredible job of ensuring there will be no side-effects (even with external systems), so I doubt we'll be able to do something as extensive as they have. That said, we will probably build a simple Postgres queue/workflow engine (similar to pgboss[0]). It will live in the Postgres database (as an extension), but will be nicely integrated with Edge Functions. We haven't started this yet, so I don't have any timelines. [0] pgboss: https://github.com/timgit/pg-boss https://github.com/timgit/pg-boss
- joisig 4y ago
  Take a look at Oban, it very well might fit the bill. https://github.com/sorentwo/oban https://github.com/sorentwo/oban
  - kiwicopple 4y ago
    we used Oban for a very early product (which we put on ice) called Supabase Workflows: https://supabase.com/blog/supabase-workflows https://supabase.com/blog/supabase-workflows The reason we paused it was the realisation that it's far more powerful to co-locate this sort of product with the data. In this case, Workflows would be more powerful as a Postgres extension - you can store the workflows into your database, create them using SQL, and the workflows would have direct access to your operational data. We'll probably re-write this using pgx[0] at some stage (if anyone is reading this and wants to attempt this, please reach out). [0] pgx: https://github.com/tcdi/pgx https://github.com/tcdi/pgx
swyx 4y ago
have you tried Azure Durable Functions? just curious if youve evaluated all the durable workflow solutions out there and what you came away with

ForrestN 4y ago

Love to see Barbz running/naming cool things!! Why be simple when you could be stunning???

cpursley 4y ago

Any chance of open sourcing the rate limiter bit for Elixir?

maxloh 4y ago
Isn't it open source? https://github.com/supabase/realtime https://github.com/supabase/realtime
- cpursley 4y ago
  As a stand-alone Elixir package.
  - nextaccountic 4y ago
    Isn't the whole supabase open source? Oh actually you mean, you want to run this without supabase? The code is already open source, and anyone can extract it into a more reusable package. And it's developed in the open, anyone can fork or send PRs or anything. That's what open sourcing a piece of code is really about.
    - cpursley 4y ago
      Yeah, I already extracted the WAL listening bit out (which is now far behind the realtime changes). And for that reason would rather have those pieces as individual packages for the wider Elixir community and so that it's easier to contribute back. https://github.com/cpursley/walex https://github.com/cpursley/walex
chasers 4y ago
Yeah I plan to. Have some things to prep next week for ElixirConf but am going to get back to it after that.

samokhvalov 4y ago

It would be great to see monitoring of this thing under load – I mean the Postgres part of it, how it's loaded, how many connections used, query analysis, etc.

kiwicopple 4y ago
cool - we can include that in the next blog post

tandyman 4y ago

LOVE Supabase. Building a project over the last several months for a client and very excited to launch the beta soon. Thanks for all you are doing...very exciting stuff. Creating dev and staging environments is a feature that I am most looking forward to. Hopefully that is on the road map.

fintechjock 4y ago

I have almost zero coding experience. Seriously, I've read a couple of books and I've taken a couple of MOOCs and that is about it. I've never 'launched' any project of my own, and I have zero front-end or back-end experience. I say that because I just used Supabase for the first time, and I was able to build and launch the Slack clone in ~15 minutes. Seriously cool. Thank you for sharing

kiwicopple 4y ago
That's awesome. I've shared your comment with the team - they will really enjoy your feedback. Good luck building

pixelbyindex 4y ago

Hi Supabase & Team What are the chances of an official C# library being in the works? I think this would greatly fill the needs for unity devs. I use firebase on almost all game backends because of how quickly I can get set up with auth and a db. I used supabase in my latest project and it served my needs but there were a few difficulties to overcome. The greatest shortcoming was the lack of real time support (C#), and I had to do everything via REST api. I am planning on publishing the game as OSS and making some learning videos about it

kiwicopple 4y ago
They're unofficial, but @acupofjose has been working hard on some community C# libraries: https://github.com/supabase-community/#client-libraries https://github.com/supabase-community/#client-libraries They won't support the functionality we outline in our blog post yet (we are still figuring it out on our official JS libraries), but I'll send him your message. As an aside, if anyone is reading this and wants to help maintain the C# libraries, please reach out. [0] acupofjose: https://github.com/orgs/supabase-community/people/acupofjose https://github.com/orgs/supabase-community/people/acupofjose

jahewson 4y ago

Given that Phoenix.Tracker is eventually consistent, how does it guarantee that there's only one Postgres connection?

chasers 4y ago
Yeah the database ultimately guarantees that. This lets us distribute the fact that a connection exists (in the form of a PID) so that we don't have to go back to the db on every connection request. This helps quite a bit in the case of a lot of connections coming in very quickly, especially when those connections are geographically far from the database or the database is under load from some other system. If no connection exists yet, you're right, two connections could try to establish themselves at the same time. In which case, whoever gets to the db first wins. But then the loser is still connected to PubSub, so they'll now start getting changes just the same.

timwis 4y ago

Really exciting! I’ve been checking your blog every couple weeks waiting for this release. Does the global cluster architecture make it much harder to self-host supabase? Not that I plan to, but having the promise of portability is really important.

matthewfcarlson 4y ago

I’m working on a live trivia type app. What’s the latency on the new real-time stuff like? I’m currently rolling my own thing with socket.io and

lawik 4y ago
I can speak some to the Elixir tech it is built on. It should be low, even with load. The Erlang VM is very good with concurrency and keeping latencies consistent and low. The presence feature could be slower since that propagates a bit more complexly than the broadcasts would. Of course the latency can go up with load but it shouldn't get very spiky thanks to the preemptive multitasking.