For probing an http server and identifying its response, we can use the –spider option.

wget --spider http://www.google.com

–06:24:36– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
200 OK

It will say you the http response code and return a corresponding exit code. If the server responded properly with a successful response, the program will exit with error code 0. When you add the switch –spider, the real file will not be downloaded to local.

To analyse the http server response header, use -S (–server-response) switch. It prints the header sent by the server.

wget --spider -S http://www.google.com

–06:23:14– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.0 200 OK
Cache-Control: private, max-age=0
Date: Tue, 16 Jun 2009 13:23:14 GMT
Expires: -1
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=640b4463b5aaaadf:TM=1245158594:LM=1245158594:S=35L2K0_MlEo7Cka5; expires=Thu, 16-Jun-2011 13:23:14 GMT; path=/; domain=.google.com
Server: gws
Length: unspecified [text/html]
200 OK

This is useful to check any additional parameters like the MIME type sent by server, charset and last updated time of the file.

Wget supports sending custom/altered header fields in its request header.

wget --header="Host: www.mysite.com" --spider http://192.168.0.1

This command will request for site www.mysite.com hosted on server 192.168.0.1 using name based virtual hosting.

wget --spider --header="Accept-Encoding: compress, gzip" http://www.mysite.com

This request will tell the server, accepting compress and gzip encoding methods. If server supports sending compressed http packets, it will respond with a Content-Encoding flag in its response header.

HTTP/1.1 200 OK
Date: Tue, 16 Jun 2009 13:32:08 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8b DAV/2 mod_jk/1.2.26
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5
Connection: Keep-Alive
Content-Type: text/html

Also try various timeouts and re-try values to benchmark your server response and performance.

wget --spider --wait=0 --waitretry=0 -T 0.1 -t 10 http://www.mysite.com

This checks your server for faster response and if it could not get a page within the specified time, returns a non-zero exit value.

-t number
–tries=number
Set number of retries to number.

-T seconds
–timeout=seconds
Set the network timeout to seconds seconds. This is equivalent to specifying –dns-timeout, –connect-timeout, and –read-time-out, all at the same time.

-w seconds
–wait=seconds
Wait the specified number of seconds between the retrievals.

–waitretry=seconds

If you don’t want Wget to wait between every retrieval, but only between retries of failed downloads, you can use this option.