40 ms·
Have you considered using GitHub Actions to automate the data scraping? A GitHub Actions workflow is allowed to run for up to 6 hours so even with the PDF proce
by simonw 5y ago
Have you considered using GitHub Actions to automate the data scraping? A GitHub Actions workflow is allowed to run for up to 6 hours so even with the PDF processing you are doing it may be enough time to generate the full site.
- underyx 5y agoYes! I actually had that set up on a previous project (which I just migrated into this one at https://visawhen.com/nvc https://visawhen.com/nvc): https://github.com/underyx/nvc-backlog/blob/main/.github/workflows/update.yml https://github.com/underyx/nvc-backlog/blob/main/.github/wor... This setup was lovely and worked quite well. Except it broke due to an absolutely obscure, insane, hopeless bug: https://github.community/t/cannot-resolve-travel-state-gov-hostname-in-github-actions-with-default-dns-server/180625 https://github.community/t/cannot-resolve-travel-state-gov-h...
- simonw 5y agoOh what a pain! I have onset if you could work around that by making HTTP calls to http://IP.OF.TRAVEL.GOV/ http://IP.OF.TRAVEL.GOV/ and manually setting the host header? No idea if you could get TLS working like that though.