5 ms·

1.5M PDFs in 25 Minutes

nesarkvechnep 3y ago
Bringing in libraries for problems already solved on the BEAM. I don't know why people keep using Go for distributed work.
sss111 3y ago
is 1.5M PDFs in 25 minutes, supposed to be crazy? I can't wrap my head around it (like it seems incredibly slow)
- koliber 3y ago
  That’s generating 60,000 PDFs per minute, or 1,000 per second. That is not slow. As a very rough benchmark, print this page to a PDF file to see how long it takes.
  - ByQuyzzy 3y ago
    They don't even say what hardware they're using. It could be impressive, or slow and bloated but just massively parallel.
shreyansh_k 3y ago
This seems to be a standard "serial processing to parallel distributed processing" story, with job generator, broker, processor etc, every grandma used to share.
bionhoward 3y ago
I’m somebody who never really used go but one thing that struck me when parsing go was the way it seems simpler than almost any other language, GC issues aside I can definitely see why Golang is a solid option for projects like this where you have some wrangling task and need something faster than Python but Rust is too finicky
rtpg 3y ago
Weasyprint is your friend for replacing puppeteer and company for PDF generation. And if you’re accompany using it, sponsor the project!!
- mmh0000 3y ago
  To save everyone else a search: https://weasyprint.org/ https://weasyprint.org/ https://github.com/Kozea/WeasyPrint https://github.com/Kozea/WeasyPrint
dzhiurgis 3y ago
> taking some 8 hours daily > By converting our HTML templates into TEX formats and utilizing pdflatex for PDF generation, we observed a 10X increase in speed compared to the original Puppeteer way of generating 8*60/10 = 48 minutes. So this speedup alone would've been enough, wouldn't it?
- tmnvix 3y ago
  That would just be for the PDF generation. The 8 hours would also include the signing and emailing.
- rbdixon 3y ago
  I have a complex document generation tool based on latex. I’ve been working on replacing it with typst. Typst is simply 10x faster. Crazy example of how fast typst is: 30”x30” document with 2600 tiny images generates a 40MB PDF in under a half second. Check typst out. It’s amazing. Not quite latex equivalent in some ways but moving fast on an amazing foundation.
  - tehbeard 3y ago
    Gave this a look at, as PDF generation has been the bane of web app development for us. This looks much more reasonable than fpdf's bare bones nature, or wkhtmltopdf / snappy / puppeteer needing esoteric, old CSS specs and having a lot of quirks because of it. Particularly happy at how it can swap page size as needed.
  - singhrac 3y ago
    They should absolutely switch to Typst for programmatically generating PDFs. pdflatex is notoriously slow.
d_sc 3y ago
does anyone have experience using haraka vs a service like sendgrid? I'm curious if there are any challenges with mail deliverability such as the haraka instance being flagged for spam.
- Avamander 3y ago
  There's definitely more work required to set up, warm up and to deal with blocklists when self-hosting. Basic stuff like DKIM/SPF/DMARC is a must in both cases.
- giovannibonetti 3y ago
  As a Postmark customer, I strongly recommend switching to their service if you want better deliverability than Sendgrid, for example. However, if you have a poorly sanitized email audience, you'll suffer a lot of pain initially to reduce the bounce ratio. It's like switching from a gym where you're on your own and no one cares to one where you're yelled at if you miss your training. Naturally, the former delivers much better results, but there is an adaptation period which makes a lot of people give up.
ByQuyzzy 3y ago
We need a standard platform for benchmarking, they don't even tell us their hardware. I vote rPi 5. It's easy to get, it's the same everywhere, and you can't just blast out whatever numbers you pull from your ass.
- mr-karan 3y ago
  Co author of the blog post. Quoting from the article > For the contract note generation job, we spawn about 40 instances in total currently, a mix of c6a.8xlarge, c6a.2xlarge, and c6a.4xlarge.
Shakahs 3y ago
This looks pretty reasonable overall, but it seems like it could have been implemented far more simply. It appears they are emailing PDF copies of stock market transactions to fulfill a regulatory requirement, so the PDF generation should be deterministic and email delivery of a receipt is an idempotent operation. Instead of all that scheduled orchestration complexity they could just queue their PDF jobs to SQS and do the work in a Lamba function, this would also let them queue the PDFs as the transactions are happening instead of continuing to run this as a nightly batch job. It's also unclear why render + sign + add to mail queue need to be 3 separate queued tasks. They have to execute serially, so a single worker could do the whole process and checkpoint to S3 after each step, or not because the output is deterministic anyway. It would also eliminate their issues with S3 request limits by keeping all the data relevant to a PDF local to 1 worker for the duration of processing.

Bringing in libraries for problems already solved on the BEAM. I don't know why people keep using Go for distributed work.

sss111 3y ago

is 1.5M PDFs in 25 minutes, supposed to be crazy? I can't wrap my head around it (like it seems incredibly slow)

koliber 3y ago
That’s generating 60,000 PDFs per minute, or 1,000 per second. That is not slow. As a very rough benchmark, print this page to a PDF file to see how long it takes.
- ByQuyzzy 3y ago
  They don't even say what hardware they're using. It could be impressive, or slow and bloated but just massively parallel.

shreyansh_k 3y ago

This seems to be a standard "serial processing to parallel distributed processing" story, with job generator, broker, processor etc, every grandma used to share.

bionhoward 3y ago

I’m somebody who never really used go but one thing that struck me when parsing go was the way it seems simpler than almost any other language, GC issues aside I can definitely see why Golang is a solid option for projects like this where you have some wrangling task and need something faster than Python but Rust is too finicky

rtpg 3y ago

Weasyprint is your friend for replacing puppeteer and company for PDF generation. And if you’re accompany using it, sponsor the project!!

mmh0000 3y ago
To save everyone else a search: https://weasyprint.org/ https://weasyprint.org/ https://github.com/Kozea/WeasyPrint https://github.com/Kozea/WeasyPrint

dzhiurgis 3y ago

> taking some 8 hours daily > By converting our HTML templates into TEX formats and utilizing pdflatex for PDF generation, we observed a 10X increase in speed compared to the original Puppeteer way of generating 8*60/10 = 48 minutes. So this speedup alone would've been enough, wouldn't it?

tmnvix 3y ago
That would just be for the PDF generation. The 8 hours would also include the signing and emailing.
rbdixon 3y ago
I have a complex document generation tool based on latex. I’ve been working on replacing it with typst. Typst is simply 10x faster. Crazy example of how fast typst is: 30”x30” document with 2600 tiny images generates a 40MB PDF in under a half second. Check typst out. It’s amazing. Not quite latex equivalent in some ways but moving fast on an amazing foundation.
- tehbeard 3y ago
  Gave this a look at, as PDF generation has been the bane of web app development for us. This looks much more reasonable than fpdf's bare bones nature, or wkhtmltopdf / snappy / puppeteer needing esoteric, old CSS specs and having a lot of quirks because of it. Particularly happy at how it can swap page size as needed.
- singhrac 3y ago
  They should absolutely switch to Typst for programmatically generating PDFs. pdflatex is notoriously slow.

d_sc 3y ago

does anyone have experience using haraka vs a service like sendgrid? I'm curious if there are any challenges with mail deliverability such as the haraka instance being flagged for spam.

Avamander 3y ago
There's definitely more work required to set up, warm up and to deal with blocklists when self-hosting. Basic stuff like DKIM/SPF/DMARC is a must in both cases.
giovannibonetti 3y ago
As a Postmark customer, I strongly recommend switching to their service if you want better deliverability than Sendgrid, for example. However, if you have a poorly sanitized email audience, you'll suffer a lot of pain initially to reduce the bounce ratio. It's like switching from a gym where you're on your own and no one cares to one where you're yelled at if you miss your training. Naturally, the former delivers much better results, but there is an adaptation period which makes a lot of people give up.

ByQuyzzy 3y ago

We need a standard platform for benchmarking, they don't even tell us their hardware. I vote rPi 5. It's easy to get, it's the same everywhere, and you can't just blast out whatever numbers you pull from your ass.

mr-karan 3y ago
Co author of the blog post. Quoting from the article > For the contract note generation job, we spawn about 40 instances in total currently, a mix of c6a.8xlarge, c6a.2xlarge, and c6a.4xlarge.

Shakahs 3y ago

This looks pretty reasonable overall, but it seems like it could have been implemented far more simply. It appears they are emailing PDF copies of stock market transactions to fulfill a regulatory requirement, so the PDF generation should be deterministic and email delivery of a receipt is an idempotent operation. Instead of all that scheduled orchestration complexity they could just queue their PDF jobs to SQS and do the work in a Lamba function, this would also let them queue the PDFs as the transactions are happening instead of continuing to run this as a nightly batch job. It's also unclear why render + sign + add to mail queue need to be 3 separate queued tasks. They have to execute serially, so a single worker could do the whole process and checkpoint to S3 after each step, or not because the output is deterministic anyway. It would also eliminate their issues with S3 request limits by keeping all the data relevant to a PDF local to 1 worker for the duration of processing.