4 ms·
> Caution: That command took just over 30 hours to complete on my macbook. (it also killed Finder a couple times and I had to disable spotlight on the folder I
by yeldarb 10y ago
> Caution: That command took just over 30 hours to complete on my macbook. (it also killed Finder a couple times and I had to disable spotlight on the folder I was saving all the .json files to)
I had a similar job I needed to do a few months ago and used AWS lambda to massively parallelize the work.
I was able to bring down what I estimated would take my laptop 30 days down to about an hour by sharding to a ton of small instances.
Might be worth a look if you plan on updating this with any regularity.
- curiousgal 10y agoHow much did it end up costing you?
- yeldarb 10y agoIt looks like I barely went over the free tier. My lambda bill in September (when I did all this processing) was 4 cents.
- aaronhoffman 10y agoThat's a great idea, and I'll definitely look into that for next time. When I started that command, I had no idea how long it would take :-) lucky for me, I could continue to work on a different machine.