Cluster filesystem utilization alerts

This is a quick and raw method to setup alerts when the filesystem fill above threshold.

Pre-requisites

1. Monitored filesystems should be consistent, meaning available across all nodes
2. passwordless ssh should be setup between the nodes.
   Node where the alert script is running should be able to talk to other nodes over ssh.
3. Prepare the list of hostnames to be monitored.
   This will be passed to pdsh using WCOLL variable.

Steps

1. Create a bash script .sh file with below content

   #!/bin/bash -x
   
   export PDSH_RCMD_TYPE='ssh'
   export WCOLL='/path/to/hosts-file'
   
   pdsh "df -Ph /tmp /var /home | grep /" | sed s/%//g | awk '$6 >= 85 {printf("%s%s\tis %d percent.\n", $1, $7, $6) }' | mail -s "Environment filesystem(s) almost full" -E -r [email protected] -c [email protected] [email protected]

2. Change variables in above script, example
   85 above, is threshold value
   WCOLL param to the file containing list of hosts (one per line)
   email addresses – add system admin and the group distribution list
3. Setup cron for the script file

HTH