4 ms·

>> If you have to manage 100-1000s of connections > Which I said, usually you are not. Postgresql connections are expensive. You do not want 100s-1000s. I was

by 1st1 10y ago

>> If you have to manage 100-1000s of connections > Which I said, usually you are not. Postgresql connections are expensive. You do not want 100s-1000s. I was talking about a micro-service that has to support a lot of connections and work with a DB. >> some of each can be long-living sessions such as websockets > you use websockets to talk to your Postgresql database? Where did I say I was talking about HTTP web services ? So you were talking about programs that don't communicate with the outside world except to a DB?

zzzeek 10y ago
If you write a program that has thousands of persistent websockets open, you do not want all those thousands of connections to each have their own, persistent, long lived database transaction. You will not like your process listing nor will you be happy about the amount of table and row locking going on. Transactions and connection scope should be short lived, and if you're using a connection pool, you definitely are going for that. As my blog post stated, you'd use a thread pool for the database access part, given that the http-facing side is async.
1st1 10y ago
> If you write a program that has thousands of persistent websockets open, you do not want all those thousands of connections to each have their own, persistent, long lived database transaction. [..] Mike, I've never suggested to have a DB connection per HTTP request. Please re-read my other comment in this thread.
zzzeek 10y ago
> Mike, I've never suggested to have a DB connection per HTTP request. right, so my suggestion is, keep the DB code in a thread pool. So you can use the async for your 1000-client frontend and not force it into the DB layer where it makes things much more complicated (or what I do, just use gevent :) ).
- allendoerfer 10y ago
  Sorry to interrupt: You two can click on the timestamp to answer your posts. Please continue, we are listening carefully. And since I am already here @1st1: How hard would it be to create a synchronous version of your adapter?
  - 1st1 10y ago
    A lot of code in asyncpg is datatypes parsers, buffer abstractions, API facade and other IO independent things. So good news is that it would be possible to reuse a lot of code. Protocol is implemented in two layers -- the base layer is IO and framework independent. The higher level is designed for asyncio. Long story short, it's possible to add a synchronous version, but it still will require a lot of work. The API will also have to be split in two.
    - allendoerfer 10y ago
      I think you should open a bounty for it. As questions like "Can I use this with Django/SQLAlchemy/…?" showed, people love drop-in replacements that speed up their code, especially when it is at a low level like this. Do not want to sound ungrateful, though. Thank you for your work also on you other contributions. Would definitely use this when applicable. Are more performance related projects like this and uvloop planned?
      - 1st1 10y ago
        > I think you should open a bounty for it. As questions like "Can I use this with Django/SQLAlchemy/…?" showed, people love drop-in replacements that speed up their code, especially when it is at a low level like this. We'll consider doing that. We have to keep in mind that supporting two IO modes will add a lot of maintenance overhead. Anyways, we can certainly continue the discussion on GH, and maybe we can find a champion to do the heavy-lifting. > Do not want to sound ungrateful, though. Thank you for your work also on you other contributions. Would definitely use this when applicable. Are more performance related projects like this and uvloop planned? Thank you. Right now most of our resources are consumed by getting EdgeDB alpha out as soon as possible. We also have plans to create a micro web framework based on httptools and few other exciting ideas. Stay tuned :) Also check out PEP 525 -- another cool thing we're trying to make happen in Python 3.6.
- 1st1 10y ago
  > right, so my suggestion is, keep the DB code in a thread pool. So you can use the async for your 1000-client frontend and not force it into the DB layer where it makes things much more complicated (or what I do, just use gevent :) ). Looks like a totally valid suggestion for folks who write async code and want to continue to use SQL Alchemy. However, if you don't use SQL Alchemy (or another ORM) and just want to work with the DB directly in an async/await code base, having a threadpool will only needlessly complicate things.

by 1st1 10y ago

zzzeek 10y ago

If you write a program that has thousands of persistent websockets open, you do not want all those thousands of connections to each have their own, persistent, long lived database transaction. You will not like your process listing nor will you be happy about the amount of table and row locking going on. Transactions and connection scope should be short lived, and if you're using a connection pool, you definitely are going for that. As my blog post stated, you'd use a thread pool for the database access part, given that the http-facing side is async.

1st1 10y ago

> If you write a program that has thousands of persistent websockets open, you do not want all those thousands of connections to each have their own, persistent, long lived database transaction. [..] Mike, I've never suggested to have a DB connection per HTTP request. Please re-read my other comment in this thread.

zzzeek 10y ago

> Mike, I've never suggested to have a DB connection per HTTP request. right, so my suggestion is, keep the DB code in a thread pool. So you can use the async for your 1000-client frontend and not force it into the DB layer where it makes things much more complicated (or what I do, just use gevent :) ).

allendoerfer 10y ago
Sorry to interrupt: You two can click on the timestamp to answer your posts. Please continue, we are listening carefully. And since I am already here @1st1: How hard would it be to create a synchronous version of your adapter?
- 1st1 10y ago
  A lot of code in asyncpg is datatypes parsers, buffer abstractions, API facade and other IO independent things. So good news is that it would be possible to reuse a lot of code. Protocol is implemented in two layers -- the base layer is IO and framework independent. The higher level is designed for asyncio. Long story short, it's possible to add a synchronous version, but it still will require a lot of work. The API will also have to be split in two.
  - allendoerfer 10y ago
    I think you should open a bounty for it. As questions like "Can I use this with Django/SQLAlchemy/…?" showed, people love drop-in replacements that speed up their code, especially when it is at a low level like this. Do not want to sound ungrateful, though. Thank you for your work also on you other contributions. Would definitely use this when applicable. Are more performance related projects like this and uvloop planned?
    - 1st1 10y ago
      > I think you should open a bounty for it. As questions like "Can I use this with Django/SQLAlchemy/…?" showed, people love drop-in replacements that speed up their code, especially when it is at a low level like this. We'll consider doing that. We have to keep in mind that supporting two IO modes will add a lot of maintenance overhead. Anyways, we can certainly continue the discussion on GH, and maybe we can find a champion to do the heavy-lifting. > Do not want to sound ungrateful, though. Thank you for your work also on you other contributions. Would definitely use this when applicable. Are more performance related projects like this and uvloop planned? Thank you. Right now most of our resources are consumed by getting EdgeDB alpha out as soon as possible. We also have plans to create a micro web framework based on httptools and few other exciting ideas. Stay tuned :) Also check out PEP 525 -- another cool thing we're trying to make happen in Python 3.6.
1st1 10y ago
> right, so my suggestion is, keep the DB code in a thread pool. So you can use the async for your 1000-client frontend and not force it into the DB layer where it makes things much more complicated (or what I do, just use gevent :) ). Looks like a totally valid suggestion for folks who write async code and want to continue to use SQL Alchemy. However, if you don't use SQL Alchemy (or another ORM) and just want to work with the DB directly in an async/await code base, having a threadpool will only needlessly complicate things.