Linux File System Monitoring

A lot of the embedded systems I work on lately run Linux. I enjoy that, because I’m a long-time UNIX user and I run nothing but Linux on my personal computers and servers these days (well, assuming you count Android as a form of Linux, which I do).

However, I’m struck by how embedded Linux development is often more akin to dealing with a Linux server and not a Linux desktop. On the desktop you expect GUI software and grand interactions with DBUS. On a server, you are just trying to get stuff done, usually without human intervention on the server-side. I find I keep pulling my server tricks out to serve on my embedded systems.

The other day I was working on a system where an external entity was to transfer a file to an embedded board (say, for example, via FTP). When the file shows up, the embedded board was supposed to do something with it. To protect the innocent, I can’t give you all the details, but as an illustration, let’s say that you were working on a book reader and when a book file shows up in a certain directory, you want to add it to the book reader’s database.

The simple but ugly way to do this is just scan the directory periodically. Here’s a really dumb shell script:

  while true
        for I in 'ls'
           do cat $I; rm $I
   sleep 10

Just for an example, I dump the file to the console and remove it, but in real life you’d do something more interesting. This is really not a good script because it executes all the time (although most of it sleeping) and it just isn’t a very elegant solution. By the way, if you think I should use for I in * try doing that in an empty directory and you’ll see why I use the ls command instead.

Honestly, though. You want something more elegant right? Modern kernels (2.6.13 and later) have filesystem notifications in the form of an interface called inotify. You can use these calls programmatically with the sys/inotify.h header file. There is also a set of command-line programs you can install (packaged as inotify-tools, usually).

One of those tools is inotifywait and it makes for a nicer script. For example:

while true
       if FN='inotifywait –e close_write,moved_to --format %f .'
          cat $FN
          rm $FN

That’s better, I think. It doesn’t wake up frequently, just when something’s changed. I figure any sane program putting something in the directory will either open the file for writing and close it, or it will move it. Either way will work and the %f tells the command to report the file name. There are other events you can wait for as well, of course.

The other command line, inotifywatch, also outputs file change events on its output, but I won’t talk about it any further. If you think you need that capability, you can read the man page.

The script is still less than ideal, though. Presumably a system might have lots of different directories it wants to monitor. You really don’t want to repeat this script, or a variation of it, for each case.

There is another program for that, called incron (you will almost surely have to install this one). The incron program is like cron but instead of time-based events, the events are based on file notifications. Once you install it, you will probably have to change /etc/incron.allow and /etc/incron.deny if you want to actually use it, especially as a normal user.

Since I mentioned electronic books, I’ll show you how I use incron to automatically have Calibre (the book reader software) catalog incoming electronic books. You use the command “incrontab -e” to edit your incron table. The format is very picky (it wants spaces, not tabs, for example). Here’s a line from mine (this is all one line, but will probably wrap when you read it):

/home/alw/Downloads/books IN_CLOSE_WRITE,IN_MOVED_TO /home/alw/bin/autocalibre $@/$#

This runs the autocalibre script with the full path name any time incron sees a file closed for writing or moved to the download directory (you can guess that incron uses inotify, and it does).

If you are really curious, here’s the autocalibre script:

# Automatically add a file in a book directory
# First we want to make sure it really is a kind we want
if [ "$1" == "" ]
  exit 1
if [ "$EXT" == "epub" -o "$EXT" == "EPUB" -o "$EXT" == "pdf" -o "$EXT" == "PDF" ]
    calibredb add "$1"
exit 0

This is the most elegant solution yet. A system program does all the waiting and our script only runs when necessary. You can do a lot with these tools, and not just in the embedded space. How are you going to use them? Leave a comment and share your ideas.