4 ms·
We compress Pub/Sub messages and more, saving a load of money
- sa46 6y agoSince you’re on btrfs, a copy-on-write file system, you should also investigate disabling full page writes for Postgres WAL. I disabled it for our cluster on ZFS and got a nice reduction in write volume.
- lawrjone 6y agoAuthor here, happy to answer any questions people may have!
- agup006 6y agoGreat blog! I saw you are now using fluentd to do gzip in real-time, any notes or feedback on that portion? (I’m the PM of the fluentd project)
- lawrjone 6y agoHey! I didn't implement the fluentd piece, but I can point you at the PR that did just that for our fork of the gcloud-pubsub plugin: https://github.com/gocardless/fluent-plugin-gcloud-pubsub-custom/pull/1 https://github.com/gocardless/fluent-plugin-gcloud-pubsub-cu... One comment I can make (from my own perspective, of course) is that we would normally upstream these changes, but fluentd tends to have a problem with a variety of different packages for the same thing, and no clear official choice. That means it's difficult to know where to put your efforts, and can lead to individuals forking plugins like we have as we don't know where to focus the upstream efforts. We didn't try too hard, truth be told, but thought it was worth mentioning. Thanks for all your efforts on fluentd!