Managing File Counts

One resource consists of the directory queues for e-mail and print servers.. The number of files in these directories varies depending on the volume of work on the server.. The script pr

■ ■ ■

C H A P T E R 3 2

Managing File Counts

System administrators have to manage the computing resources available to their users

One resource consists of the directory queues for e-mail and print servers These

directo-ries contain the files waiting to be processed by the server The number of files in these

directories varies depending on the volume of work on the server There are times when

no one is using the service and the queue empties out There are also times when the

server is heavily loaded and the queue will be full Usually, though, the number of files will

stay within stipulated limits

Of course, from time to time you will have problems with a queue filling up faster than

the server can process the requests This might be due to a bubble of extremely heavy load

or because of some type of problem that prevents the server from processing the requests

You will want to monitor the queue directories so you can take the proper measures when

a problem arises

The script presented in this chapter is a simple monitor that counts the files in the

specified directories The script gives you the necessary core functionality; you will,

how-ever, want to modify the directory locations and notification methods for your own site

and needs

File-Count Monitor

First we define the directories to be monitored It would be a bit cleaner to place this

definition in a separate file so you don’t have to modify the script to make configuration

changes However, to keep the script as simple as possible I have not done that Each

entry in the FILENUM variable is a colon-delimited set of fields The fields specify the

direc-tory name to monitor, the warning threshold, the maximum limit, and finally the e-mail

address to which to send notifications

#!/bin/sh

FILENUM="/var/spool/mqueue:50:100:root \

/var/spool/lp:50:100:root@yourdomain.net /:50:100:rbpeters"

212 C H A P T E R 3 2 ■ M A N A G I N G F I L E C O U N T S

Now we start looping through each of the configured directories to check the file counts First sed swaps colons with spaces in the configuration entry; then the set com-mand assigns, to each of the positional parameters $1 through $4, the corresponding element of the output, which consists of the items in the original colon-separated fields Once the fields are assigned to the variables, the script determines the number of files in the directory

for monitor in $FILENUM

do

set `echo $monitor | sed -e 's/:/ /g'`

file_count=`ls $1 | wc -l`

The script completes by comparing the current file count to the three possible situations:

Everything is normal The file count is lower than the warning threshold

A warning state The count is between the warning threshold and maximum limit

An error state The count is above the maximum limit

In either the warning or error state, the script will send a notification via e-mail specify-ing the condition of the directory There is one inherent problem with the e-mail notification type If you’re monitoring a mail queue directory that is filling up, it is likely that the e-mail notification will not be sent In this case, you may want to change your notification method to use something other than e-mail such as an alpha-numeric pager or phone

if [ $file_count -lt $2 ]

then

continue

elif [ $file_count -ge $2 -a $file_count -lt $3 ]

then

echo "Warning: File count in $1 is $file_count, should be less \

than $2" | mail -s "Directory file count warning for $1" $4

else

echo "Error: File count in $1 is $file_count, should be less \

than $2" | mail -s "Directory file count error for $1" $4

fi

done

To make this script more useful, you would need to call it with cron or possibly sur-round the code with an infinite loop that a perpetual process would run to continually monitor your machine The script’s main function is carried out in the assignment of the file_count variable by performing an ls on the target directory and then piping the out-put to the wc utility This is one of a number of ways to get queue-status information Two

C H A P T E R 3 2 ■ M A N A G I N G F I L E C O U N T S 213

other ways of determining the file count in a directory are to use the echo and find

com-mands Here are some possible solutions:

file_count=`ls $1 | wc -l`

file_count=`find $1 -type f | wc -l`

file_count=`echo $1/* | wc -w`

Note that the wc -l switch has been replaced with -w for the last solution This is

because the result of the shell’s expansion of the asterisk will appear in the form of a single

line of filenames separated by spaces

Testing File-Count Methods

The different methods of counting the files in a directory are not all created equal Some

older operating systems cannot handle large numbers of files when the ls or find

com-mand is used Sometimes the find-oriented solution will return incorrect results on

directories containing between 12,000 and 15,000 files Also, I have worked on systems

with file counts of more than one million in a single directory; in these cases ls returns

an “argument list too long” error I have not yet found a situation in which the echo

solu-tion fails

On my current Gentoo Linux system, both find and ls seem not to suffer from these

earlier limitations, and I have used both to successfully count files in a single directory

numbering well above 800,000 files While testing, I noticed that there is a speed

differ-ence between the three solutions

The find solution is by far the fastest It took about 4 seconds with just over 300,000

files in a directory The second fastest is echo * It took about 1.5 seconds This measure is

most accurate if you work in the directory whose files are to be counted If a full path is

supplied with the asterisk (e.g., /some/full/path/*), the command takes longer because

each element in the shell expansion contains the full path and all that extra text needs to

be processed The slowest in my tests was ls It took about 4 seconds to complete Both

find and ls seemed to take similar amounts of time regardless of whether the full path is

supplied

Realistically speaking, with a current operating system any of these solutions should

work fine It is not very likely that you will deal with this extreme number of files in a single

directory, so the performance differences between the solutions will almost surely be

insignificant