3 ms·
Looks really cool. Thanks for sharing! I think what Uberneo is asking is whether Parallec would handle the html parsing like Scrapy. I believe the answer is no
by jstoiko 11y ago
Looks really cool. Thanks for sharing!
I think what Uberneo is asking is whether Parallec would handle the html parsing like Scrapy. I believe the answer is no. You wouldn't want to slow down Parallec with parsing though, you would rather send the html output to some other process for that, right?
- jeffpeiyt 11y agoThanks jstoiko! You are right. Parallec is not specifically built to do crawling or parse website pages recursively (however you may build such crawler on top of it) We mostly use it to manage (HTTP/S) agents on every production machine in the cloud for software deployment, remediation, asset discovery etc. (Parallec like a kubenate master) to manage all the kubelet (agents) Yes, we may or may not want to slow down. Sometimes if it is just regex or simple parsing we just put the parser inside of the worker. We can send the results out to Kafka etc so that some other process/machine can process them. http://www.parallec.io/docs/submit-task/#apis-on-response-handling http://www.parallec.io/docs/submit-task/#apis-on-response-ha...