Using this code within a shell script is very simple, just needs to define a global variable named ARGS_INPUT_FORMAT and set appropriate values to it. The argument parsing code will read this variable and set appropriate inputs as well as outputs.

Set values to the variable ARGS_INPUT_FORMAT as shown below.

ARGS_INPUT_FORMAT="OPTION STRING 1|OPTION STRING 2(OPTIONAL)|OPTION STRING N(OPTIONAL):VARIABLE NAME WHICH HOLDS OPTION VALUE:DEFAULT OPTION VALUE(OPTIONAL):DEFAULT OPTION VALUE IF USED AS SWITCH(OPTIONAL)"

Example Usage

These are some examples:

ARGS_INPUT_FORMAT="--ignore-case|-i:check_case:NO:YES"
ARGS_INPUT_FORMAT="--user|-u|-U:username:-1:-1,--ignore-case|-i:ignore_case:NO:YES"
ARGS_INPUT_FORMAT="--color:bgcolor"

Some example script usages are:

./main-script.sh --ignore-case="YES"
./main-script.sh -i -u "Find Linux Help"
./main-script.sh --color="Yellow"

Now access the value of color using variable bgcolor as shown below.

echo "You have chosen $bgcolor color."

There are some advantages of this code; it is not utilizing the capabilities of an external binary but parse using shell script itself. It never shifts or alters the original arguments supplied to main script. It retains the position and count of arguments and can be re-used them inside the main script again.

Argument Parsing Code

for arg_opts in $(echo "$ARGS_INPUT_FORMAT" | tr "," " ") ; do
  opt_names=$(echo "$arg_opts" | cut -d':' -f1)
  opt_var=$(echo "$arg_opts" | cut -d':' -f2)
  opt_def_val=$(echo "$arg_opts" | cut -d':' -f3)
  opt_switch_val=$(echo "$arg_opts" | cut -d':' -f4)
  eval "$opt_var=$opt_def_val"
  for opt_name in $(echo "$opt_names" | tr "|" " ") ; do
    arg_pos=1
    while [ $arg_pos -le $# ] ; do
      case "$(eval "echo $$arg_pos")" in
        "$opt_name="*)
              eval "$opt_var="$(echo "$(eval "echo $$arg_pos")" | sed 's/^'"$opt_name"'=//')""
              ARGS_STAT[$arg_pos]=0
              ;;
        "$opt_name")
              if [ $(expr $arg_pos + 1) -le $# ] && [ $(echo "$(eval "echo $$(expr $arg_pos + 1)")" | grep -c '^-') -eq 0 ] ; then
                ARGS_STAT[$arg_pos]=0
                arg_pos=$(expr $arg_pos + 1)
                eval "$opt_var="$(eval "echo $$arg_pos")""
              else
                eval "$opt_var="$opt_switch_val""
              fi
              ARGS_STAT[$arg_pos]=0
              ;;
        "$opt_name"*)
              eval "$opt_var="$(echo "$(eval "echo $$arg_pos")" | sed 's/^'"$opt_name"'//')""
              ARGS_STAT[$arg_pos]=0
              ;;
        *)
              [ -z "${ARGS_STAT[$arg_pos]}" ] && ARGS_STAT[$arg_pos]=1
              ;;
      esac
      arg_pos=$(expr $arg_pos + 1)
    done
  done
done

Download the script file: [download id="5"]

Script Usage

Define a global variable as ARGS_INPUT_FORMAT and set an appropriate value as shown above. Define this variable on top of the main script.

Save the argument parsing script to an external file and source it into the main script. The same source script can be included in as many scripts wherever  you need argument processing. Even the code block can be directly pasted below the global variable definition inside your main script for better portability.

About Debugging

The script initializes an array variable named ARGS_STAT and set debug values to it. If it finds an invalid argument in commandline, the corresponding array index is updated with a dirty flag “1″.

Here is a sample code which can be included in the main script to display invalid arguments supplied and their respective position numbers.

echo "**Invalid args**"
count=1
while [ $count -le $# ] ; do
  if [ ${ARGS_STAT[$count]} -eq 1 ] ; then
    echo "$(eval echo $$count) ($count)"
  fi
  count=$(expr $count + 1)
done

Download the script file: [download id="6"]

If you want to give an informative message saying “We are under maintenance”, create a custom HTML page containing the message and put it in the website root. For e.g., we create an HTML page called “under-maintenance.html” and place it inside the website root, so that we can access it through the URL http://www.yourdomain.com/under-maintenance.html

Setup status “503″ and header “Retry-After”

Depending on the maintenance duration, we also have to care about the search engines which may visit the website during maintenance. For doing this, we will send an http status code 503, “Service Unavailable”, along with the HTML page containing maintenance information. Also, we will inform search engines, visit the website only after the maintenance is completed.

First, setup an error document for http status 503. Specify your maintenance page as the error document as shown below.

ErrorDocument 503 /under-maintenance.html

Now add a small trick using mod rewrite to flag all http responses from the website using status 503.

RewriteEngine On
RewriteCond %{REQUEST_URI} !^/under-maintenance.html$
RewriteRule .* - [R=503,L]

Note: The ‘R’ flag can be used to redirect to a destination page only when the status code specified is ranging from 300 to 399. If it’s outside the range, rewrite engine will skip the destination URL and we just gave a – (dash) to specify the destination URL will not be changed.

We have used here a conditional rewrite where the URI is not equal to “/under-maintenance.html”. This is to avoid looping errors which can occur when the rule is enabled.

Additionally, if you are using images or external style sheets inside your maintenance page, don’t forget to exclude them from being redirected. For e.g., add an additional condition per external resource below the existing condition.

RewriteCond %{REQUEST_URI} !^/styles/maintenance.css$

The effective rule will be, URI is not equal to “/under-maintenance.html” AND “/styles/maintenance.css”.

Now, we have one more thing to do, tell search engines visiting your site during maintenance period, come back only after specified period of time to re-check the site. Add below block of code to your .htaccess file to set a “Retry-After” response header indicating a time period or a fixed time.

To re-visit the site only after 10 hours,

<IfModule mod_headers.c>
Header set Retry-After: 36000
</IfModule>

To re-visit after a fixed time,

<IfModule mod_headers.c>
Header set Retry-After: "Wed, 25 Nov 2009 06:30:00 GMT"
</IfModule>

Note: “headers_module” must be enabled to use the “Header” directive inside your .htaccess file. Make sure a similar line of code is enabled inside your httpd.conf file in apache.

LoadModule headers_module modules/mod_headers.so

At last, the final block of code to be inserted at the end of .htaccess file is:

<IfModule mod_rewrite.c>
 ErrorDocument 503 /under-maintenance.html
 RewriteEngine On
 RewriteCond %{REQUEST_URI} !^/under-maintenance.html$
 RewriteCond %{REQUEST_URI} !^/styles/maintenance.css$
 RewriteRule .* - [R=503,L]
 <IfModule mod_headers.c>
 Header set Retry-After: 36000
 </IfModule>
</IfModule>

The mechanism of content caching is pretty simple but an effective configuration of server will only allow utilizing this feature fully and properly.

Mainly five response headers and their values control the caching behavior in an http client-server communication.

1. Date
2. Last-Modified
3. Cache-Control
4. ETag
5. Expires

Date & Last-Modified

A web server sets the Date and Last-Modified headers default in its response unless directed otherwise. The Last-Modified header value helps a browser to identify weather the content is modified or not on server. Browsers can send conditional requests and verify the freshness of a resource whenever requires. This header is a minimum requirement at least when no other cache control headers are present.

Cache-Control

Cache-Control header values are a set of control instructions which one end can force the other to obey. A browser can request its cache preferences to server. Also, a web server can force the browser to follow some caching behaviors. This may include commonly used directives like “no-cache”, “no-store”, “max-age” or “must-revalidate”. These control instructions always get high preference over any other cache control header values and browser algorithms.

ETag & Expires

They are two different methods to implement improved client side caching. ETag (Entity Tag) is actually a checksum of the resource’s attributes and it is a unique value dynamically generated by server or application. For a ‘static file’ resource, this can be the ‘sum’ of size, modification time and inode number. An application can generate this tag value based on some specific criterias and set in resource’s response header.

Expires is an entirely different technology which guarantees a valid cached resource until the expiry time is reached. A combination of Expires and Last-Modified values help managing the cache control of static files simply and more efficiently.

How to setup my server to use appropriate cache control headers

Many people prefer avoiding ETag unless it is required for any specific reason. Also many believe it can help improve the server’s performance itself if removed. The concept of ETag is similar to what Last-Modified header does. A conditional request based on the last known ETag value can send to server to verify the freshness of a resource. Server will answer with a 304 response (Not Modified) if not modified. ETag ensures more accuracy when compared to Last-Modified header, but for static file resources, most of the time it is not necessary to generate this extra header. ETag is useful when you need strong validation of some kind of dynamic generated contents. There is a known issue too with ETag where you use multiple servers to load balance the traffic. The generated ETag will be different on different servers and this will create confusion to clients.

You can set or unset the ETag header as shows below:
To generate ETag for static files in Apache using all available attributes,

FileETag All

To unset this behavior explicitly,

<IfModule headers_module>
   Header unset ETag
</IfModule>
FileETag None

Note: ‘headers_module’ needs to be enabled this to work.

The combination of Expires and Last-Modified headers with appropriate values give the best result. See below how to set Expires directives in Apache

<IfModule expires_module>
   ExpiresActive On
   ExpiresByType application/javascript A604800
   ExpiresByType application/x-javascript A604800
   ExpiresByType text/css A604800
   ExpiresByType image/gif A604800
   ExpiresByType image/jpeg A604800
   ExpiresByType image/png A604800
   ExpiresByType image/x-icon A604800
   ExpiresByType application/x-shockwave-flash A604800
</IfModule>

If ‘expires_module’ is enabled and ExpiresActive set to ‘On’, Expires functionality turns on. ExpiresByType directive allows specifying different ‘interval’ values for different MIME Types. See http://httpd.apache.org/docs/2.2/mod/mod_expires.html to understand how to use the interval syntax and a default expiry time using ExpiresDefault.

Expires module sets a ‘max-age’ value equal to the ‘interval’ specified and adds to the Cache-Control header value.

Cache-Control max-age=604800

Note, if you remove or unset Last-Modified header when Expires is active, browsers will be forced to not use any conditional checking at all. This behavior is not recommended even through if we can eliminate the extra overhead of sending a conditional checking. Browsers will handle this situation intelligently and they will only re-validate the cached content when you re-request (Refresh) a web page or max-age is reached. RFC standards recommend Last-Modified header to be send with every response header.

For probing an http server and identifying its response, we can use the –spider option.

wget --spider http://www.google.com

–06:24:36– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: unspecified [text/html]
200 OK

It will say you the http response code and return a corresponding exit code. If the server responded properly with a successful response, the program will exit with error code 0. When you add the switch –spider, the real file will not be downloaded to local.

To analyse the http server response header, use -S (–server-response) switch. It prints the header sent by the server.

wget --spider -S http://www.google.com

–06:23:14– http://www.google.com/
Resolving www.google.com… 74.125.53.103, 74.125.53.99, 74.125.53.104, …
Connecting to www.google.com|74.125.53.103|:80… connected.
HTTP request sent, awaiting response…
HTTP/1.0 200 OK
Cache-Control: private, max-age=0
Date: Tue, 16 Jun 2009 13:23:14 GMT
Expires: -1
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=640b4463b5aaaadf:TM=1245158594:LM=1245158594:S=35L2K0_MlEo7Cka5; expires=Thu, 16-Jun-2011 13:23:14 GMT; path=/; domain=.google.com
Server: gws
Length: unspecified [text/html]
200 OK

This is useful to check any additional parameters like the MIME type sent by server, charset and last updated time of the file.

Wget supports sending custom/altered header fields in its request header.

wget --header="Host: www.mysite.com" --spider http://192.168.0.1

This command will request for site www.mysite.com hosted on server 192.168.0.1 using name based virtual hosting.

wget --spider --header="Accept-Encoding: compress, gzip" http://www.mysite.com

This request will tell the server, accepting compress and gzip encoding methods. If server supports sending compressed http packets, it will respond with a Content-Encoding flag in its response header.

HTTP/1.1 200 OK
Date: Tue, 16 Jun 2009 13:32:08 GMT
Server: Apache/2.2.8 (Unix) mod_ssl/2.2.8 OpenSSL/0.9.8b DAV/2 mod_jk/1.2.26
Vary: Accept-Encoding
Content-Encoding: gzip
Keep-Alive: timeout=5
Connection: Keep-Alive
Content-Type: text/html

Also try various timeouts and re-try values to benchmark your server response and performance.

wget --spider --wait=0 --waitretry=0 -T 0.1 -t 10 http://www.mysite.com

This checks your server for faster response and if it could not get a page within the specified time, returns a non-zero exit value.

-t number
–tries=number
Set number of retries to number.

-T seconds
–timeout=seconds
Set the network timeout to seconds seconds. This is equivalent to specifying –dns-timeout, –connect-timeout, and –read-time-out, all at the same time.

-w seconds
–wait=seconds
Wait the specified number of seconds between the retrievals.

–waitretry=seconds

If you don’t want Wget to wait between every retrieval, but only between retries of failed downloads, you can use this option.

Screenshot

kSar CPU Statistics

Sar is a system activity reporting tool which comes with the sysstat rpm package. Sar will collect your system activities every 10 minutes and store in /var/log/sa/saXX where XX is the zero-padded two digit day of month. Once you have sar is ready in your linux system, kSar can read the daily report file generated by sar from /var/log/sa/sarXX and generate the graphical report.

kSar can work in GUI as well as in CLI modes. GUI mode gives you the visualization of sar generated report with customizable easy to read graphs.

The CLI mode gives you the flexibility of reading sar reports from the specified input file and create graphical output in various formats like PDF, HTML or images. The command line interface will be useful if you want to generate a report on daily basis and send it via email using cron job.

kSar supports a variety of platforms like Solaris, Linux, HP-UX, AIX and Mac OS/X

Usage

GUI only :

java -jar kSar-x.x.x.jar

GUI and collect :

java -jar kSar-x.x.x.jar -input /var/log/sa/sarXX

To run kSar on the command line :

java -jar kSar-x.x.x.jar -input /var/log/sa/sarXX -outputPDF today.pdf

kSar can support reading input from different sources like ‘ssh:// or file:// or cmd://’. It can read the input from the output of a command (cmd://) or from a remote host (ssh ://)

Download kSar athttp://ksar.atomique.net/

Here is a simple solution using iptables to overcome this situation by allowing all high ports in server. (configuration will be saved in /etc/sysconfig/iptables)

-A INPUT -p tcp -m tcp -s 0/0 -d 0/0 --dport 21 -j ACCEPT
-A INPUT -p tcp -m tcp -s 0/0 -d 0/0 --dport 1024: -j ACCEPT

The INPUT chain will accept all incoming connections on any high port in server. But it is less secure and not recommended to leave your server ports open to everyone.

So here is a better solution implemented using the connection tracking mechanism for FTP in iptables. This is achieved in iptables by loading an additional module called “ip_conntrack_ftp”. This module will keep track of your established control connections and will find out the required data connections to be opened by analysing PORT command sent over the control channel. ip_conntrack_ftp module will open those required high ports only in server and allow data transfer.

You can enable this module in two ways. For loading the module on demand, execute

/sbin/modprobe ip_conntrack_ftp

And for loading it when iptable starts, modify the iptables configuration file “/etc/sysconfig/iptables-config” as given below

IPTABLES_MODULES="ip_conntrack_netbios_ns ip_conntrack_ftp"

Addition modules can be loaded by placing their names separated by space.

So the new iptables entry will be like this

-A INPUT -p tcp -m tcp -s 0/0 -d 0/0 --dport 21 -j ACCEPT

Here you are specifying only the control connection in iptables and data connection will be opened on demand by ip_conntrack_ftp module.

rsync is basically used for two filesystem/directory tree synchronization which are placed in two different network locations. It also can be used for website mirroring.

Here find some simple example usages of rsync :

1. Copying/syncing the contents of one directory called “/src” to another directory called “/dst” :

rsync -a /src/ /dst

Note: “-a” option represents archive mode, which preserves most of the file attributes when copying.

If you miss a trailing slash (/) after source directory, it will create the directory as “/dst/src” in the destination and will transfer the files into it.

2. Copying the contents of a directory called “/src” in computer 1 to another directory called “/dst” in computer 2 :

rsync -a -e ssh /src/ computer2:/dst

Assume you are in computer 1, you can execute the above command to copy files to computer 2 using “ssh”. It will establish an ssh connection from computer 1 to computer 2 and transfer files over that connection. Also, it will ask you ssh login details on computer 2 to establish the connection before start transferring the data.