4 ms·

Hey everyone, I work on PGlite. Excited to see this on HN again. If you have any questions I'll be sure to answer them. We recently crossed a massive usage mi

Hey everyone, I work on PGlite. Excited to see this on HN again. If you have any questions I'll be sure to answer them. We recently crossed a massive usage milestone with over 3M weekly downloads (we're nearly at 4M!) - see https://www.npmjs.com/package/@electric-sql/pglite https://www.npmjs.com/package/@electric-sql/pglite While we originally built this for embedding into web apps, we have seen enormous growth in devtools and developer environments - both Google Firebase and Prisma have embedded PGlite into their CLIs to emulate their server products.

oulipo2 10mo ago
I'm interested to use Pglite for local unit-testing, but I'm using timescaledb in prod, do you think you will have this extension pre-built for Pglite?
- samwillis 10mo ago
  I'm not aware of anything trying to compile timescale for it. Some extensions are easer than other, if there is limited (or ideally no) network IO and its written in C (Timescale is!) with minimal dependencies then its a little easer to get them working.
- rel 10mo ago
  I’ve had incredible success with testcontainers for local unit-testing
- tdrz 10mo ago
  We have a walk-through on porting extensions to PGlite: https://pglite.dev/extensions/development#building-postgres-extensions https://pglite.dev/extensions/development#building-postgres-...
mentalgear 10mo ago
Yupp, this has big potential for local-first !
nnnnico 10mo ago
This is awesome, thanks for your work! Could this work with the file system api in the bowser to write to user disk instead of indexeddb? I'm interested in easy ways for syncing fot local-first single user stuff <3 thanks again
- tdrz 10mo ago
  That's a very nice idea, we will look into it!
reachableceo 10mo ago
Well downloads doesn’t equal usage does it ? How do you know how many deployments you actually have in the wild?
- pixelatedindex 10mo ago
  True downloads don’t equal usage but there’s a correlation. I also doubt deployment equals usage - I can deploy to some env and not make any requests. Additionally, how you can get data on how many deployments without telemetry? The only telemetry that I’m interested in is for my uses, and don’t really care about sending data on deployment count to a third party. So the download count becomes a “good enough” metric.
TheDataMaverick 10mo ago
Amazing work! It makes setting up CI so much easier.
- lame_lexem 10mo ago
  huh. could you tell how you use it in ci?
  - tln 10mo ago
    I'm using it for a service that has DB dependencies. Instead of using SQLite in tests and PG in production, or spinning up a Postgres container, you use Postgres via pglite. In my case, the focus is on DX ie faster tests. I load shared database from `pglite-schema.tgz` (~1040ms) instead of running migrations from a fresh DB and then use transaction rollback isolation (~10ms per test). This is a lot faster and more convenient than spinning up a container. Test runs are 5x faster. I'm hoping to get this working on a python service soon as well (with py-pglite).
    - TheTaytay 10mo ago
      Thank you for the details. This makes a lot of sense!
mpweiher 10mo ago
This looks really interesting...but why WASM-only? Naively it seems like WASM-ification would be a 2nd step, after lib-ification. Obviously missing something...
- intrasight 10mo ago
  WASM means you only need to develop for one target run time. That's my guess as to why.
- OvbiousError 10mo ago
  If I understand correctly, what this project does is take the actual postgresql sources, which are written in C, compile them to wasm and provide typescript wrappers. So you need the wasm to be able to use the C code from js/ts.
  - mpweiher 10mo ago
    Yes. I would like to use the code as a library from something other than js/ts.
    - monster_truck 10mo ago
      So compile it and use it?
    - embedding-shape 10mo ago
      Sounds you only need to create the APIs for calling into WASM if so, so as long as your language of choice can do that, you're good to go.
      - dboreham 10mo ago
        That adds extra unnecessary complexity. The code is written in C. There are C compilers for all CPUs. So just call the C code from <other language that's not JS>.
phplovesong 10mo ago
This looks REALLY awesome. Could you name a few usecases when i would want to use this. Is the goal to be an sqlite/duckdb alternative?
JackC 10mo ago
Thanks for your work! Is the project interested in supporting http-vfs readonly usecases? I'm thinking of tools like DuckDB or sql.js-httpvfs that support reading blocks from a remote url via range requests. Curious because we build stuff like this https://news.ycombinator.com/item?id=45774571 https://news.ycombinator.com/item?id=45774571 at my lab, and the current ecosystem for http-vfs is very slim — a lot of proofs of concept, not many widely used and optimized libraries. I have no idea if this makes sense for postgres — are the disk access patterns better or worse for http-vfs in postgres than they are in sqlite?
glenjamin 10mo ago
Does pglite in memory outperform “normal” postgres? If so then supporting the network protocol so it could be run in CI for non-JS languages could be really cool
- jitl 10mo ago
  Look into libeatmydata LD_PRELOAD. it disables fsync and other durability syscalls, fabulous for ci. Materialize.com uses it for their ci that’s where i learned about it.
  - deleted 10mo ago
    [deleted]
- allan_s 10mo ago
  for CI you can already use postgresql with "eat-my-data" library ? I don't know if there's more official image , but in my company we're using https://github.com/allan-simon/postgres-eatmydata https://github.com/allan-simon/postgres-eatmydata
  - anarazel 10mo ago
    You can just set fsync=off if you don't want to flush to disk and are ok with corruption in case of a OS/hw level crash.
    - ffsm8 10mo ago
      Huh, i always just mounted the data directory as tmpfs/ramdisk. Worked nicely too
DonnyV 10mo ago
I see you guys are working on supporting the postgis extension. This would be HUGE!!! The gis community would be all over this. If anyone wants to help out who has compiled the postgis extension and is familiar with WASM. You can help out here. https://github.com/electric-sql/pglite/pull/807 https://github.com/electric-sql/pglite/pull/807
sgt 10mo ago
Any chance for a Flutter library?
SteveLauC 10mo ago
Hi there, would you like to share the progress of converting PGlite into a native system library? I can see there is a repo for that, but it hasn't been updated for 5 months
- tdrz 10mo ago
  We are actively looking into it. But as you can see from the comments here, there are quite a lot of other features that users want and we have limited bandwidth. We will do it!
dcgudeman 10mo ago
Small world! We spoke about this at the QCon dinner.

oulipo2 10mo ago

I'm interested to use Pglite for local unit-testing, but I'm using timescaledb in prod, do you think you will have this extension pre-built for Pglite?

samwillis 10mo ago
I'm not aware of anything trying to compile timescale for it. Some extensions are easer than other, if there is limited (or ideally no) network IO and its written in C (Timescale is!) with minimal dependencies then its a little easer to get them working.
rel 10mo ago
I’ve had incredible success with testcontainers for local unit-testing
tdrz 10mo ago
We have a walk-through on porting extensions to PGlite: https://pglite.dev/extensions/development#building-postgres-extensions https://pglite.dev/extensions/development#building-postgres-...

mentalgear 10mo ago

Yupp, this has big potential for local-first !

nnnnico 10mo ago

This is awesome, thanks for your work! Could this work with the file system api in the bowser to write to user disk instead of indexeddb? I'm interested in easy ways for syncing fot local-first single user stuff <3 thanks again

tdrz 10mo ago
That's a very nice idea, we will look into it!

reachableceo 10mo ago

Well downloads doesn’t equal usage does it ? How do you know how many deployments you actually have in the wild?

pixelatedindex 10mo ago
True downloads don’t equal usage but there’s a correlation. I also doubt deployment equals usage - I can deploy to some env and not make any requests. Additionally, how you can get data on how many deployments without telemetry? The only telemetry that I’m interested in is for my uses, and don’t really care about sending data on deployment count to a third party. So the download count becomes a “good enough” metric.

TheDataMaverick 10mo ago

Amazing work! It makes setting up CI so much easier.

lame_lexem 10mo ago
huh. could you tell how you use it in ci?
- tln 10mo ago
  I'm using it for a service that has DB dependencies. Instead of using SQLite in tests and PG in production, or spinning up a Postgres container, you use Postgres via pglite. In my case, the focus is on DX ie faster tests. I load shared database from `pglite-schema.tgz` (~1040ms) instead of running migrations from a fresh DB and then use transaction rollback isolation (~10ms per test). This is a lot faster and more convenient than spinning up a container. Test runs are 5x faster. I'm hoping to get this working on a python service soon as well (with py-pglite).
  - TheTaytay 10mo ago
    Thank you for the details. This makes a lot of sense!

mpweiher 10mo ago

This looks really interesting...but why WASM-only? Naively it seems like WASM-ification would be a 2nd step, after lib-ification. Obviously missing something...

intrasight 10mo ago
WASM means you only need to develop for one target run time. That's my guess as to why.
OvbiousError 10mo ago
If I understand correctly, what this project does is take the actual postgresql sources, which are written in C, compile them to wasm and provide typescript wrappers. So you need the wasm to be able to use the C code from js/ts.
- mpweiher 10mo ago
  Yes. I would like to use the code as a library from something other than js/ts.
  - monster_truck 10mo ago
    So compile it and use it?
  - embedding-shape 10mo ago
    Sounds you only need to create the APIs for calling into WASM if so, so as long as your language of choice can do that, you're good to go.
    - dboreham 10mo ago
      That adds extra unnecessary complexity. The code is written in C. There are C compilers for all CPUs. So just call the C code from <other language that's not JS>.

phplovesong 10mo ago

This looks REALLY awesome. Could you name a few usecases when i would want to use this. Is the goal to be an sqlite/duckdb alternative?

JackC 10mo ago

Thanks for your work! Is the project interested in supporting http-vfs readonly usecases? I'm thinking of tools like DuckDB or sql.js-httpvfs that support reading blocks from a remote url via range requests. Curious because we build stuff like this https://news.ycombinator.com/item?id=45774571 https://news.ycombinator.com/item?id=45774571 at my lab, and the current ecosystem for http-vfs is very slim — a lot of proofs of concept, not many widely used and optimized libraries. I have no idea if this makes sense for postgres — are the disk access patterns better or worse for http-vfs in postgres than they are in sqlite?

glenjamin 10mo ago

Does pglite in memory outperform “normal” postgres? If so then supporting the network protocol so it could be run in CI for non-JS languages could be really cool

jitl 10mo ago
Look into libeatmydata LD_PRELOAD. it disables fsync and other durability syscalls, fabulous for ci. Materialize.com uses it for their ci that’s where i learned about it.
- deleted 10mo ago
  [deleted]
allan_s 10mo ago
for CI you can already use postgresql with "eat-my-data" library ? I don't know if there's more official image , but in my company we're using https://github.com/allan-simon/postgres-eatmydata https://github.com/allan-simon/postgres-eatmydata
- anarazel 10mo ago
  You can just set fsync=off if you don't want to flush to disk and are ok with corruption in case of a OS/hw level crash.
  - ffsm8 10mo ago
    Huh, i always just mounted the data directory as tmpfs/ramdisk. Worked nicely too

DonnyV 10mo ago

I see you guys are working on supporting the postgis extension. This would be HUGE!!! The gis community would be all over this. If anyone wants to help out who has compiled the postgis extension and is familiar with WASM. You can help out here. https://github.com/electric-sql/pglite/pull/807 https://github.com/electric-sql/pglite/pull/807

sgt 10mo ago

Any chance for a Flutter library?

SteveLauC 10mo ago

Hi there, would you like to share the progress of converting PGlite into a native system library? I can see there is a repo for that, but it hasn't been updated for 5 months

tdrz 10mo ago
We are actively looking into it. But as you can see from the comments here, there are quite a lot of other features that users want and we have limited bandwidth. We will do it!

dcgudeman 10mo ago

Small world! We spoke about this at the QCon dinner.