Script Download Files From Web

Sep 26, 2018 Download files from websites programatically via powershell This script can be used to define a file parameter path on a website and a 'save' location in the script, when run the script will download the specified file to the set location.The script may be amended and used for any other purposes.I have not yet amended this script to utili. Mar 31, 2006 Script to download a file from a website automatically. Does anyone know how I could make a script that would go to a given web address and download a text file.

Does anyone know how I could make a script that would go to a given web address and download a text file from that address and then store it at a given location?
Say: website.com/documents/reports/daily/report.txt
Then take report.txt and dump it to c:documentsreportstoday's date
Something like that?
It would need to run daily, and capture about 18 files.
Right now I get to go to the site daily, and download all 18 files and well... there HAS to be a better way -- View image here: http://episteme.arstechnica.com/groupee_common/emoticons/icon_smile.gif --
Thanks

In PowerShell, you can download a file via HTTP, HTTPS, and FTP with the Invoke-WebRequest cmdlet

Michael Pietroforte

Michael Pietroforte is the founder and editor in chief of 4sysops. He has more than 35 years of experience in IT management and system administration.

Latest posts by Michael Pietroforte (see all)

Results of the 4sysops member and author competition in 2018 - Tue, Jan 8 2019
Why Microsoft is using Windows customers as guinea pigs - Reply to Tim Warner - Tue, Dec 18 2018
PowerShell remoting with SSH public key authentication - Thu, May 3 2018

Download with SMB ^

If you are working in a hybrid IT environment, you often need to download or upload files from or to the cloud in your PowerShell scripts. If you only use Windows servers that communicate through the Server Message Block (SMB) protocol, you can simply use the Copy-Item cmdlet to copy the file from a network share:

</div><table><tbody><tr><td><div>2</div></td><td><div><span>$WebClient</span><span>.</span><span>DownloadFile</span><span>(</span><span>'https://www.contoso.com/file'</span><span>,</span><span>'C:pathfile'</span><span>)</span></div></td></tr></tbody></table><p>As of PowerShell 3, we have the <i>Invoke-WebRequest</i> cmdlet, which is more convenient to work with. It is PowerShell’s counterpart to GNU <i>wget</i>, a popular tool in the Linux world, which is probably the reason Microsoft decided to use its name as an alias for <i>Invoke-WebRequest</i>. This is perhaps an understatement; <i>Invoke-WebRequest </i>is more powerful than <i>wget</i> because it allows you to not only download files but also parse them. But this is a topic for another post.<br /></p><h2>Download with Invoke-WebRequest ^</h2><p>To simply download a file through HTTP, you can use this command:</p><div><textarea wrap='soft' readonly='>Invoke-WebRequest -Uri 'http://www.contoso.com' -OutFile 'C:pathfile'

Invoke-WebRequest-Uri'http://www.contoso.com'-OutFile'C:pathfile'

In the example, we just download the HTML page that the web server at www.contoso.com generates. Note that, if you only specify the folder without the file name, as you can do with Copy-Item, PowerShell will error:

Invoke-WebRequest : Could not find a part of the path

The shorter version for the command line is:

If you omit the local path to the folder, Invoke-WebRequest will just use your current folder. The -Outfile parameter is always required if you want to save the file. The reason is that, by default, Invoke-WebRequest sends the downloaded file to the pipeline.

However, the pipeline will then not just contain the contents of the file. Instead, you will find an object with a variety of properties and methods that allow you to analyze text files. If you send a binary file through the pipeline, PowerShell will treat it as a text file and you won’t be able to use the data in the file.

To only read the contents of the text file, we need to read the Content property of the object in the pipeline:

</div><h2>Windows Script To Download File</h2><table><tbody><tr><td><div><span>Invoke-WebRequest</span><span>'http://www.contoso.com'</span><span>|</span><span>Select-Object</span><span>-ExpandProperty</span><span>Content</span><span>|</span><span>Out-File</span><span>'file'</span></div></td></tr></tbody></table><p>This command does the same thing as the previous one. The <i>-ExpandProperty</i> parameter ensures that the header (in this case, “Content”) won’t be stored in the file.</p><p>If you want to have the file in the pipeline <i>and</i> store it locally, you have to use <i>-PassThru</i> parameter:</p><div><textarea wrap='soft' readonly='>Invoke-WebRequest 'http://www.contoso.com' -OutFile 'file' -PassThru | Select-Object -ExpandProperty Content

Invoke-WebRequest'http://www.contoso.com'-OutFile'file'-PassThru|Select-Object-ExpandPropertyContent

This command stores the web page in a file and displays the HTML code.

Download and display file

Powershell Script To Download Files

Authenticating at a web server ^

If the web server requires authentication, you have to use the -Credential parameter:

</div><table><tbody><tr><td><div><span>Invoke-WebRequest</span><span>-Uri</span><span>https</span><span>:</span><span>/</span><span>/</span><span>www</span><span>.</span><span>contoso</span><span>.</span><span>com</span><span>/</span><span>-OutFile</span><span>C</span><span>:</span><span>'pathfile'</span><span>-Credential</span><span>'yourUserName'</span></div></td></tr></tbody></table><p>Note that, if you omit the <i>-Credential</i> parameter, PowerShell will not prompt you for a user name and password and will throw this error:</p><p><i>Invoke-WebRequest : Authorization Required</i></p><p>You have to at least pass the user name with the <i>-Credential</i> parameter. PowerShell will then ask for the password. If you want to avoid a dialog window in your script, you can store the credentials in a <i>PSCredential</i> object:</p><div><textarea wrap='soft' readonly='>$Credentials = Get-CredentialInvoke-WebRequest -Uri 'https://www.contoso.com' -OutFile 'C:pathfile' -Credential $Credentials

2	Invoke-WebRequest-Uri'https://www.contoso.com'-OutFile'C:pathfile'-Credential$Credentials

You can use the -UseDefaultCredentials parameter instead of the -Credential parameter if you want to use the credentials of the current user. To add a little extra security, you might want to encrypt the password. Make sure to always use HTTPS instead of HTTP if you have to authenticate on a remote server. If the web server uses basic authentication, your password will be transmitted in clear text if you download via HTTP.

Note that this method only works if the web server manages authentication. Nowadays, most websites use the features of a content management system (CMS) to authenticate users. Usually, you then have to fill out an HTML form. I will explain in one of my next posts how you can do this with Invoke-WebRequest.

Downloading files through FTP works analogous to HTTP. You also shouldn’t use this protocol if security matters. To download multiple files securely, you had better work with SFTP or SCP. Invoke-WebRequest doesn’t support these protocols. However, third-party PowerShell modules exist that step into the breach.

Script Download Files From Websites

In my next post I will show you can use Invoke-WebRequest to parse HTML pages and scrape content from websites.

Script To Download All Files From Website

Users who have LIKED this post: