Support Centre

Using wget on Linux

Introduction

wget is a Linux file downloader. It is a free utility for non-interactive download of files from the Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval through HTTP proxies.

Use Cases

We'll go through a few examples on using wget.

Download a Single File

# wget http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

This command will download the CentOS 7 ISO file in the user’s current working directory.

Download File and Save with a Different File Name

We can save the file we downloaded with a different filename. We can use the uppercase "-O" option.

# wget -O CentOS7.iso http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

Resume Partially Downloaded File

There are instances where we were downloading a file but in the middle, the connection was interrupted. Using the "-c" option with wget will allow us to continue where we left off.

# wget -c http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

If you do not specify the "-c" option, wget will append a .X (where X is a number starting from 1) extension and consider it as a new download.

Download Multiple Files

To download multiple files using wget, you need to create a text file first and add the URLs in the file separated by a new line.

# cat download-list.txt
url1
url2
url3
url4

To download these files, use the following command:

# wget -i download-list.txt

Download Files in the Background

We can download files in the background using the "-b" option.

# wget -b http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

# Continuing in background, pid 4505.
# Output will be written to ‘wget-log’.

As you can see, it says the output will be written to a file named "wget-log". This file will be in the current directory you executed wget from.

Limit Download Speed

By default, wget will attempt to use all available bandwidth. This can be an issue if you are using a shared connection. You can use the "--limit-rate" option to limit the bandwidth used by wget.

# wget --limit-rate=200k http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

In this example, we are limiting the download speed to 200k.

Increase Retry Attempts

By default, wget only retries up to 20 times to make the download successful. This is useful in problematic connections and you are downloading a large file. Chances are there will be failure in downloading. Use the "--tries" option.

# wget --tries=75 http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

Logging

We can have wget save its output to a logfile using the "-o" option.

# wget -o download.log http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

This will save the output from wget to "download.log" in the user's current directory.

Downloading from Password Protected Sites

We can download from password protected sites using the "--http-user" and "--http-password" options. For FTP sites, replace "http" with "ftp".

# wget --http-user=username --http-password=password download.log http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

Setting the User Agent

Sites may block the wget user agent from accessing. In this case, we can change the user agent by using the "-U" option. This allows an arbitrary string to be set for the user agent. 

# wget -U 'MyUserAgent' download.log http://mirror.nbrc.ac.in/centos/7.0.1406/isos/x86_64/CentOS-7.0-1406-x86_64-DVD.iso

A list of User Agents is available here. 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.