4 ms·

Do you have an example of that? I'm curious how PowerShell is better than other languages for this type of problem. Does it just have good built-in parallelis

by chubot 9y ago

Do you have an example of that? I'm curious how PowerShell is better than other languages for this type of problem. Does it just have good built-in parallelism?

Numberwang 9y ago
You can see my entire thinking outlined in my reddit thread. https://www.reddit.com/r/PowerShell/comments/5f4r10/web_scraping_newb_question/ https://www.reddit.com/r/PowerShell/comments/5f4r10/web_scra... I miss-remembered the number of sited scraped but it's all the same. There is no parallelism (I really really am a complete non-techie) but actually in this case I didn't want to hammer away as the sites were all under the same domain and I imagine there might be trouble (which I why I also played it very safe and added the pause). I know there are better and smarter ways of doing things and I did further modifications and experiments after my thread, but in any case, I had a task and was able to quickly complete it in PowerShell starting from absolute 0.
joeyaiello 9y ago
Disclaimer: I'm a PM who works on PowerShell at Microsoft There's a bunch of 3rd-party modules out there that do a great job of offering parallelism[1][2][3], and we also support the notion of "PowerShell Jobs"[4] (which one or two of those third-party implementations actually build on to achieve what they're doing). However, we decided that strong parallelism support is important enough to build into the language, so we're mulling on a first-party design. If you're interested, we've got an RFC out where we're looking for feedback on the parallelism/concurrency design.[5] [1]: https://www.powershellgallery.com/packages/PSParallel/ https://www.powershellgallery.com/packages/PSParallel/ [2]: https://www.powershellgallery.com/packages/SplitPipeline https://www.powershellgallery.com/packages/SplitPipeline [3]: https://github.com/RamblingCookieMonster/Invoke-Parallel https://github.com/RamblingCookieMonster/Invoke-Parallel [4]: https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_jobs https://msdn.microsoft.com/en-us/powershell/reference/5.1/mi... [5]: https://github.com/PowerShell/PowerShell-RFC/issues/85 https://github.com/PowerShell/PowerShell-RFC/issues/85
- noinsight 9y ago
  In addition to jobs there are workflows which also offer parallelism but they're a little fickly.

by chubot 9y ago

Do you have an example of that? I'm curious how PowerShell is better than other languages for this type of problem. Does it just have good built-in parallelism?

Numberwang 9y ago

You can see my entire thinking outlined in my reddit thread. https://www.reddit.com/r/PowerShell/comments/5f4r10/web_scraping_newb_question/ https://www.reddit.com/r/PowerShell/comments/5f4r10/web_scra... I miss-remembered the number of sited scraped but it's all the same. There is no parallelism (I really really am a complete non-techie) but actually in this case I didn't want to hammer away as the sites were all under the same domain and I imagine there might be trouble (which I why I also played it very safe and added the pause). I know there are better and smarter ways of doing things and I did further modifications and experiments after my thread, but in any case, I had a task and was able to quickly complete it in PowerShell starting from absolute 0.

joeyaiello 9y ago

Disclaimer: I'm a PM who works on PowerShell at Microsoft There's a bunch of 3rd-party modules out there that do a great job of offering parallelism[1][2][3], and we also support the notion of "PowerShell Jobs"[4] (which one or two of those third-party implementations actually build on to achieve what they're doing). However, we decided that strong parallelism support is important enough to build into the language, so we're mulling on a first-party design. If you're interested, we've got an RFC out where we're looking for feedback on the parallelism/concurrency design.[5] [1]: https://www.powershellgallery.com/packages/PSParallel/ https://www.powershellgallery.com/packages/PSParallel/ [2]: https://www.powershellgallery.com/packages/SplitPipeline https://www.powershellgallery.com/packages/SplitPipeline [3]: https://github.com/RamblingCookieMonster/Invoke-Parallel https://github.com/RamblingCookieMonster/Invoke-Parallel [4]: https://msdn.microsoft.com/en-us/powershell/reference/5.1/microsoft.powershell.core/about/about_jobs https://msdn.microsoft.com/en-us/powershell/reference/5.1/mi... [5]: https://github.com/PowerShell/PowerShell-RFC/issues/85 https://github.com/PowerShell/PowerShell-RFC/issues/85

noinsight 9y ago
In addition to jobs there are workflows which also offer parallelism but they're a little fickly.