4 ms·
It also happens every time it retries, so it spends most of the time doing nothing. Retrying after 60 seconds... Is that really normal? The downloads fro
by jannes 3y ago
It also happens every time it retries, so it spends most of the time doing nothing.
Retrying after 60 seconds...
Is that really normal?
The downloads from reddit work fine, but if the upload doesn't work then I don't see the point of running this.
- toomuchtodo 3y agoPlease allow it time to catch up. It always does.
- capableweb 3y agoYes, it is really normal when lots of people try to upload at the same time. Bandwidth is limited, so when lots of people start to run the warrior, the servers need some space to do their thing. Also, IA has limited bandwidth, so sometimes that's the bottleneck too. If you give it time, it'll work eventually. Up the concurrency to max, so you can have more items in the upload state, as long as you don't start hitting rate limits from reddit, it'll be fine.
- codetrotter 3y agoThe point of having many people run it is to maximise the number of different IP addresses scraping the data. Even if you are only using a small percentage of your available bandwidth you are still helping out by running this. If they attempted to max out the download bandwidth of all clients they’d only end up getting everyone IP banned by Reddit, and then the scraping would not be successful. So even if most time is waiting to upload the scraped data, it’s still good. Slow and steady.