3 ms·
It adds up. I once tried to scrape an internal company website / API because we had no PDF exports and wanted to download + export to PDF, with `httpie`. It's a
by pdimitar 6y ago
It adds up. I once tried to scrape an internal company website / API because we had no PDF exports and wanted to download + export to PDF, with `httpie`. It's an amazing tool but all the startup times compounded pretty badly. I switched to `curl` (which is a bit more pain until I got the full command line right, granted) and the script finished in ~95 minutes as opposed to the ~202 minutes it took with `httpie`.
With about 0.5s startup time, it means every 120 requests add a full minute to the final time. And I had to scrape ~125K URLs back then.
For a daily casual flow 0.5s startup time might not be much (although people like myself get irritated by that as well but I do recognize it's a minor inconvenience). But when doing mass-scripting such delays can very easily compound to non-trivial time inefficiencies.
- Scarbutt 6y agoTo be honest, you approached this problem pretty badly.
- michaelmrose 6y agoThis might be a useful comment if you had spent the time to offer thoughts on how it could be better. As it is this comment serves only to gratify your ego whereas advice might help readers. It's worse than adding nothing at all.
- pdimitar 6y agoIt was supposed to be a one-off task and I was confident at the time that I can script it quickly. I still scripted it but it took at least half a day. I'll still behind the idea that one has to be able to do one-off tasks without starting a dedicated programming project. It's what scripting languages are meant for. But I did misjudge the speed at which I'll be able to finish the task, that much is true. Ironically I knew exactly what to do from the start but several small and annoying quirks of the shell scripting languages lost me quite a bit of time.
- killingtime74 6y agoYour comment is approached pretty badly.
- dharmab 6y agoFor mass-scripting I'd prefer to use a native HTTP library with connection pooling.
- pdimitar 6y agoSure, I can agree with that somewhat. Still, it was an one-off thing and I didn't want to turn it into a project. Using a for loop, `curl`, `wkhtmltopdf` and `parallel` eventually did the job.
- j0057 6y agoI for one am totally on board with writing atrocious grep/sed/awk pipeline mashups, doing whatever it takes to get a one-off job done with as little programming as possible. Sometimes the important parameter to optimize is brain cycles spent, or time to result, not CPU time or optimal usage of network resources.
- pdimitar 6y agoAbsolutely. I'd even argue that the brain cycles spent should be the first and most important metric for one-off tasks. That's why I recently asked people here about what to use if not shell scripting. I think I'll either use a less confusing scripting language that is almost like bash/zsh -- namely `rc` -- or I'll brush off my OCaml skills. That need was borne out of my frustration that I routinely spent non-trivial amounts of time on one-off scripts so at one point I got fed up and started looking for alternatives.
- inshadows 6y agoWhy don't you use cURL then? cURL is written in C, which sure is teh performance you're looking for.
- pdimitar 6y agoI did just that in the end. It took me some fiddling with headers and form parameters and file uploads, and it worked. It's just that httpie's (and thus ht's) CLI usage is a bit more ergonomic.
- mekster 6y agoI think people need to start distinguishing between CLI and TUI as TUI aren't really meant for scripting but CLI tools are.
- pdimitar 6y agoBe that as it may, I still liked `httpie` (and `ht`) more than `curl` even though `curl` seems superior in terms of configurability.