If you want to give an informative message saying “We are under maintenance”, create a custom HTML page containing the message and put it in the website root. For e.g., we create an HTML page called “under-maintenance.html” and place it inside the website root, so that we can access it through the URL http://www.yourdomain.com/under-maintenance.html

Setup status “503″ and header “Retry-After”

Depending on the maintenance duration, we also have to care about the search engines which may visit the website during maintenance. For doing this, we will send an http status code 503, “Service Unavailable”, along with the HTML page containing maintenance information. Also, we will inform search engines, visit the website only after the maintenance is completed.

First, setup an error document for http status 503. Specify your maintenance page as the error document as shown below.

ErrorDocument 503 /under-maintenance.html

Now add a small trick using mod rewrite to flag all http responses from the website using status 503.

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/under-maintenance.html$
RewriteRule .* - [R=503,L]

Note: The ‘R’ flag can be used to redirect to a destination page only when the status code specified is ranging from 300 to 399. If it’s outside the range, rewrite engine will skip the destination URL and we just gave a – (dash) to specify the destination URL will not be changed.

We have used here a conditional rewrite where the URI is not equal to “/under-maintenance.html”. This is to avoid looping errors which can occur when the rule is enabled.

Additionally, if you are using images or external style sheets inside your maintenance page, don’t forget to exclude them from being redirected. For e.g., add an additional condition per external resource below the existing condition.

RewriteCond %{REQUEST_URI} !^/styles/maintenance.css$

The effective rule will be, URI is not equal to “/under-maintenance.html” AND “/styles/maintenance.css”.

Now, we have one more thing to do, tell search engines visiting your site during maintenance period, come back only after specified period of time to re-check the site. Add below block of code to your .htaccess file to set a “Retry-After” response header indicating a time period or a fixed time.

To re-visit the site only after 10 hours,

<IfModule mod_headers.c>
Header set Retry-After: 36000
</IfModule>

To re-visit after a fixed time,

<IfModule mod_headers.c>
Header set Retry-After: "Wed, 25 Nov 2009 06:30:00 GMT"
</IfModule>

Note: “headers_module” must be enabled to use the “Header” directive inside your .htaccess file. Make sure a similar line of code is enabled inside your httpd.conf file in apache.

LoadModule headers_module modules/mod_headers.so

At last, the final block of code to be inserted at the end of .htaccess file is:

<IfModule mod_rewrite.c>
 ErrorDocument 503 /under-maintenance.html
 RewriteEngine On
 RewriteCond %{REQUEST_URI} !^/under-maintenance.html$
 RewriteCond %{REQUEST_URI} !^/styles/maintenance.css$
 RewriteRule .* - [R=503,L]
 <IfModule mod_headers.c>
 Header set Retry-After: 36000
 </IfModule>
</IfModule>

The mechanism of content caching is pretty simple but an effective configuration of server will only allow utilizing this feature fully and properly.

Mainly five response headers and their values control the caching behavior in an http client-server communication.

1. Date
2. Last-Modified
3. Cache-Control
4. ETag
5. Expires

Date & Last-Modified

A web server sets the Date and Last-Modified headers default in its response unless directed otherwise. The Last-Modified header value helps a browser to identify weather the content is modified or not on server. Browsers can send conditional requests and verify the freshness of a resource whenever requires. This header is a minimum requirement at least when no other cache control headers are present.

Cache-Control

Cache-Control header values are a set of control instructions which one end can force the other to obey. A browser can request its cache preferences to server. Also, a web server can force the browser to follow some caching behaviors. This may include commonly used directives like “no-cache”, “no-store”, “max-age” or “must-revalidate”. These control instructions always get high preference over any other cache control header values and browser algorithms.

ETag & Expires

They are two different methods to implement improved client side caching. ETag (Entity Tag) is actually a checksum of the resource’s attributes and it is a unique value dynamically generated by server or application. For a ‘static file’ resource, this can be the ‘sum’ of size, modification time and inode number. An application can generate this tag value based on some specific criterias and set in resource’s response header.

Expires is an entirely different technology which guarantees a valid cached resource until the expiry time is reached. A combination of Expires and Last-Modified values help managing the cache control of static files simply and more efficiently.

How to setup my server to use appropriate cache control headers

Many people prefer avoiding ETag unless it is required for any specific reason. Also many believe it can help improve the server’s performance itself if removed. The concept of ETag is similar to what Last-Modified header does. A conditional request based on the last known ETag value can send to server to verify the freshness of a resource. Server will answer with a 304 response (Not Modified) if not modified. ETag ensures more accuracy when compared to Last-Modified header, but for static file resources, most of the time it is not necessary to generate this extra header. ETag is useful when you need strong validation of some kind of dynamic generated contents. There is a known issue too with ETag where you use multiple servers to load balance the traffic. The generated ETag will be different on different servers and this will create confusion to clients.

You can set or unset the ETag header as shows below:
To generate ETag for static files in Apache using all available attributes,

FileETag All

To unset this behavior explicitly,

<IfModule headers_module>
   Header unset ETag
</IfModule>
FileETag None

Note: ‘headers_module’ needs to be enabled this to work.

The combination of Expires and Last-Modified headers with appropriate values give the best result. See below how to set Expires directives in Apache

<IfModule expires_module>
   ExpiresActive On
   ExpiresByType application/javascript A604800
   ExpiresByType application/x-javascript A604800
   ExpiresByType text/css A604800
   ExpiresByType image/gif A604800
   ExpiresByType image/jpeg A604800
   ExpiresByType image/png A604800
   ExpiresByType image/x-icon A604800
   ExpiresByType application/x-shockwave-flash A604800
</IfModule>

If ‘expires_module’ is enabled and ExpiresActive set to ‘On’, Expires functionality turns on. ExpiresByType directive allows specifying different ‘interval’ values for different MIME Types. See http://httpd.apache.org/docs/2.2/mod/mod_expires.html to understand how to use the interval syntax and a default expiry time using ExpiresDefault.

Expires module sets a ‘max-age’ value equal to the ‘interval’ specified and adds to the Cache-Control header value.

Cache-Control max-age=604800

Note, if you remove or unset Last-Modified header when Expires is active, browsers will be forced to not use any conditional checking at all. This behavior is not recommended even through if we can eliminate the extra overhead of sending a conditional checking. Browsers will handle this situation intelligently and they will only re-validate the cached content when you re-request (Refresh) a web page or max-age is reached. RFC standards recommend Last-Modified header to be send with every response header.

For probing an http server and identifying its response, we can use the –spider option.

wget --spider http://www.google.com

–06:24:36– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
200 OK

It will say you the http response code and return a corresponding exit code. If the server responded properly with a successful response, the program will exit with error code 0. When you add the switch –spider, the real file will not be downloaded to local.

To analyse the http server response header, use -S (–server-response) switch. It prints the header sent by the server.

wget --spider -S http://www.google.com

–06:23:14– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.0 200 OK
Cache-Control: private, max-age=0
Date: Tue, 16 Jun 2009 13:23:14 GMT
Expires: -1
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=640b4463b5aaaadf:TM=1245158594:LM=1245158594:S=35L2K0_MlEo7Cka5; expires=Thu, 16-Jun-2011 13:23:14 GMT; path=/; domain=.google.com
Server: gws
Length: unspecified [text/html]
200 OK

This is useful to check any additional parameters like the MIME type sent by server, charset and last updated time of the file.

Wget supports sending custom/altered header fields in its request header.

wget --header="Host: www.mysite.com" --spider http://192.168.0.1

This command will request for site www.mysite.com hosted on server 192.168.0.1 using name based virtual hosting.

wget --spider --header="Accept-Encoding: compress, gzip" http://www.mysite.com

This request will tell the server, accepting compress and gzip encoding methods. If server supports sending compressed http packets, it will respond with a Content-Encoding flag in its response header.

HTTP/1.1 200 OK
Date: Tue, 16 Jun 2009 13:32:08 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8b DAV/2 mod_jk/1.2.26
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5
Connection: Keep-Alive
Content-Type: text/html

Also try various timeouts and re-try values to benchmark your server response and performance.

wget --spider --wait=0 --waitretry=0 -T 0.1 -t 10 http://www.mysite.com

This checks your server for faster response and if it could not get a page within the specified time, returns a non-zero exit value.

-t number
–tries=number
Set number of retries to number.

-T seconds
–timeout=seconds
Set the network timeout to seconds seconds. This is equivalent to specifying –dns-timeout, –connect-timeout, and –read-time-out, all at the same time.

-w seconds
–wait=seconds
Wait the specified number of seconds between the retrievals.

–waitretry=seconds

If you don’t want Wget to wait between every retrieval, but only between retries of failed downloads, you can use this option.