3 ms·

What I would absolutely love is to have a proxy server that would act as postgres, and use this new driver to connect to a remote postgres. This would avoid us

What I would absolutely love is to have a proxy server that would act as postgres, and use this new driver to connect to a remote postgres. This would avoid us having to rewrite our remote agents (which require a postgres connection to the main Database) to use an http api. Instead, we could just keep them as it is, using rust's sqlx queries, point them to the local proxy. Essentially pgBouncer but through http. Postgres <> Neon HTTP Proxy <---HTTP---> Local Fake Postgres Server <--unix socket--> Agents We need it for the open-source project windmill.dev and we will build this if no one else does it.

gmac 3y ago
This sounds cool (but it's not something we've built at Neon). In case it's useful, I can clarify: * Every Neon database is accessible over an ordinary TCP Postgres connection (as well as WebSockets and http). * Our JavaScript serverless driver is based on node-postgres and is a drop-in replacement for it. So for JS, no rewriting is needed. Assuming that doesn't help, do you specifically need http (rather than WebSockets), and a proxy that speaks the pg protocol? If not, the simplest thing might be to (1) use Neon, or run a WebSocket proxy next to your own database (e.g. https://github.com/neondatabase/wsproxy); https://github.com/neondatabase/wsproxy); and (2) find/write a similar WebSocket proxy, but in reverse — i.e. one that listens out for TCP/socket connections and tunnels them over WebSockets — to run next to the agents.
- rubenfiszel 3y ago
  Windmill is written in Rust so cannot use the javascript serverless driver. I love sqlx and sqlx would not be easy to replace so short of making it an extension for sqlx, the only route is to make believe the sqlx connection pool that it's talking to a pg server, when it's actually just talking to the local proxy. If we were using javascript/typescript, then yes we would probably just be able to use that library. > (2) find/write a similar WebSocket proxy, but in reverse — i.e. one that listens out for TCP/socket connections and tunnels them over WebSockets — to run next to the agents. It's that part that I would love to see. My assumption is that this proxy next to my agents would need to speak pg protocol and then transcribe it to wss. If there is a simpler way, let me know :)
habibur 3y ago
First check if that can be done with a port forwarder or socks proxy.

gmac 3y ago

This sounds cool (but it's not something we've built at Neon). In case it's useful, I can clarify: * Every Neon database is accessible over an ordinary TCP Postgres connection (as well as WebSockets and http). * Our JavaScript serverless driver is based on node-postgres and is a drop-in replacement for it. So for JS, no rewriting is needed. Assuming that doesn't help, do you specifically need http (rather than WebSockets), and a proxy that speaks the pg protocol? If not, the simplest thing might be to (1) use Neon, or run a WebSocket proxy next to your own database (e.g. https://github.com/neondatabase/wsproxy); https://github.com/neondatabase/wsproxy); and (2) find/write a similar WebSocket proxy, but in reverse — i.e. one that listens out for TCP/socket connections and tunnels them over WebSockets — to run next to the agents.

rubenfiszel 3y ago
Windmill is written in Rust so cannot use the javascript serverless driver. I love sqlx and sqlx would not be easy to replace so short of making it an extension for sqlx, the only route is to make believe the sqlx connection pool that it's talking to a pg server, when it's actually just talking to the local proxy. If we were using javascript/typescript, then yes we would probably just be able to use that library. > (2) find/write a similar WebSocket proxy, but in reverse — i.e. one that listens out for TCP/socket connections and tunnels them over WebSockets — to run next to the agents. It's that part that I would love to see. My assumption is that this proxy next to my agents would need to speak pg protocol and then transcribe it to wss. If there is a simpler way, let me know :)

habibur 3y ago

First check if that can be done with a port forwarder or socks proxy.