3 ms·

I found compiling with Clozure CL instead of SBCL leads to way better memory allocation/usage. It's worth getting it going, it's one of the greatest tools in m

I found compiling with Clozure CL instead of SBCL leads to way better memory allocation/usage. It's worth getting it going, it's one of the greatest tools in migrating things into and out of pgsql I've ever used. If you have pgsql in your pipeline, get it working. I used this and debezium to really make my ETL pipeline absolutely bulletproof.

dfsegoat 7y ago
> I used this and debezium This looks very cool, was not aware of it: "Debezium is a set of distributed services that capture row-level changes in your databases so that your applications can see and respond to those changes." https://debezium.io/documentation/faq/#what_is_debezium https://debezium.io/documentation/faq/#what_is_debezium
- chucky_z 7y ago
  It has a lot of required components, however I ran most of them in a single-component setup (e.g.: 1 zookeeper server , 1 kafka server; using only their provided containers) and got extremely far (think 1 billion rows a day ingested, then transferred to pgsql and s3) with some extra overhead still.