Run parallel Invoke-WebRequest jobs in PowerShell v3

Question:

Running simultaneous background jobs in PowerShell in pretty straightforward, but I cannot seem to get it working with the new (in v3) cmdlet Invoke-WebRequest.

I have several thousand files I am downloading scriptomagically via PowerShell. Works great, but doing them in series is taking days:

I’ve made several attempts at leveraging the the ‘AsJob’ method, but either they error-out or complete fine yet no local files get saved. Here’s an example of the latter:

Output examples:

The weird thing is, the jobs above complete roughly as fast as it would take to download the associated file…so it seems that data is going somewhere — just not to my save location. I assume it’s going to memory but being dumped without being saved. Tried adding and removing the ‘PassThru’ option, but get the same results either way. Also tried piping it to Out-File, still no joy. Not sure why this particular cmdlet is being such a bugger.

Answer:

You need to pass in the parameters in such as way that they actually make it into the job. In V3, you can use the $using: syntax to do this:

BTW you don’t need to use Invoke-Expression. Just put the code you want to run in a job inside a scriptblock i.e. a set of curly braces {...}.

Source:

Run parallel Invoke-WebRequest jobs in PowerShell v3 by licensed under CC BY-SA | With most appropriate answer!

Leave a Reply