7 ms·
Typst has been pretty amazing, and at my organization, we’re very happy with it. We needed to generate over 1.5 million PDFs every night and experimented with v
by mr-karan 2y ago
Typst has been pretty amazing, and at my organization, we’re very happy with it. We needed to generate over 1.5 million PDFs every night and experimented with various solutions—from Puppeteer for HTML to PDF conversions, to pdflatex and lualatex. Typst has been several orders of magnitude faster and has a lighter resource footprint. Also, templating the PDFs in LaTeX wasn’t a pleasant developer experience, but with Typst templates, it has been quite intuitive.
We’ve written more about this large-scale PDF generation stack in our blog here: https://zerodha.tech/blog/1-5-million-pdfs-in-25-minutes https://zerodha.tech/blog/1-5-million-pdfs-in-25-minutes
- sitkack 2y agoWhat is the use case for generating that many PDFs?
- mr-karan 2y agoRegulatory requirements mandate that. Stock brokers in India are required to generate this document called “Contract Notes” which includes all the trades done by the user on the stock exchanges. It also contains a breakdown of all charges incurred by the user (brokerage, various taxes etc). And this has to be emailed to every user before the next trading session begins.
- bjourne 2y agoThere are of course way more efficient methods for generating templated pdfs than using a typesetter.
- cameroncooper 2y agoI'm interested to hear what you would propose.
- abound 2y agoNot sure what GP had in mind, but one can programmatically generate PDFs directly, without using something like Typst as a "middleman".
- the_mitsuhiko 2y agoHave you tried doing that? It’s no fun at all and far from easy. I don’t quite see a benefit in doing it without some utility.
- a57721 2y agoBesides, the generation of PDF reports is usually decoupled from the templates, so you will have to work on your own "middleman".
- wolfi1 2y agoI guess some webkit solution like wkhtmltopdf
- mr_mitm 2y agoHow is that more efficient than Typst exactly?
- bjourne 2y agoApache iText, for example.
- IshKebab 2y agoDoes the law specify PDF? I would have thought pain text or even HTML would be sufficient.
- the_mitsuhiko 2y agoI don’t know the situation in India but brokers in Austria and Germany do the same. The law does not stipulate the format but PDF is what everyone uses. I assume it’s because it can be signed and archived and will outlast pretty much anything. You need to keep these for 7 years.
- mr-karan 2y agoYes, in India, the law mandates that ECNs (electronic contract notes) need to be digitally signed with a valid certifying authority. While it's true that XML/docx/xls files could also support digital signatures, but I think PDFs are prevalent and also allow clients to verify this on their end, quite easily.
- yukeabu 2y agoPDF is less likely to contain executable malicious code than other formats.
- radicality 2y agoIs it? More so than say .csv file ? I was under the impressions that pdfs are not that safe. I thought they can do stuff like execute a subset of PostScript and Javascript.
- yukeabu 2y agoLook, when it comes to corporate reporting, PDFs are pretty much the gold standard. Sure, they've got some potential security issues, but any decent company's IT department has them well in hand. Think about it - you want your reports to look sharp, right? PDFs deliver that professional look every time, no matter who opens them or on what device. Plus, they've got all those nifty features like password protection and digital signatures that the big guys love. CSV files? They're great for crunching numbers, but let's face it - they look about as exciting as a blank wall. Try sending a CSV report to the board of directors and watch their eyes glaze over. So, yes, for reporting in a company that's got its security act together, PDFs are your best bet. They're like the well-dressed, security-savvy cousin of other file formats - they look good and keep things safe.
- happimess 2y agoThis is a really great write up. Kudos for the obvious effort, both on the technical side and sharing the process with the rest of us.
- dudus 2y agoHave you tried reportlab as well? It was a good solution when I had to deal with a similar problem many moons ago. Not quite the same volume you have but still.
- kerkeslager 2y agoHaving used ReportLab a bunch, I'd agree it's a good solution, but not maybe on the more mediocre side of good. Generating LaTeX was a better solution for me, and while I haven't used it, Typst looks a lot better.
- znpy 2y agoJust wondering: did your organisation contribute anything back to the project, or supported it financially in any way?
- mr-karan 2y agoHey. Yes we did support them financially from our FOSS fund. We’ll be happy to do it once more as well.
- mstijak 2y agoNever heard that someone is generating PDF documents at that pace. I'm working on a product that is used for mass PDF reporting based on Puppeteer. With nightly jobs, caching, and parallel processing, the performance is ok. https://www.cx-reports.com https://www.cx-reports.com
- tecleandor 2y agoI think we don't reach that much quantity, but we do a hefty number. What for? Invoices! We're having problems because until now PDFs are being generated by the ERP system, and it can't keep pace. I know there's a Dev team working on a microservice for PDF generation, but never thinked about doing it with Typst. I think I'm going to send them @mr-karan link.
- 1oooqooq 2y agopdf/ps can easily be created in a way that data for text and qr code fields are easily in plain text. seems like yall focusing too much on the higher level tools instead of what's right in front of you.
- mstijak 2y agoCompany branding is an important aspect of PDF creation that many tools struggle to handle correctly. PDF documents often need to include logos, company colors, fonts, and other branding elements. Puppeteer is popular because you can control these aspects through CSS. However, Puppeteer can be challenging to work with for larger documents, as each change requires programming effort or when your software needs to needs to serve multiple clients each with different requirements.
- tecleandor 2y agoYep, it's mostly about branding and control. It needs certain concrete layout and logos, and has to be relatively easy to change them. We also render shipping labels in PDF, and we have to be VERY strict with that. But we're still not touching that, as that process is not at slow and problematic as the invoicing one.
- iamgopal 2y agoI was reading second sentence, and I knew it was zerodha. It’s good to see more open source in your tech stack.
- stevekemp 2y agoThanks for your writeup, that was exceptionally well-presented.