Taming your Temporary Files

I don’t know about you, but there have been many times when I wanted to download something only temporarily with the intent to delete it later, like saving an image to upload elsewhere or saving email attachments that I didn’t need to keep to open in an external program. There have also been times when I wanted to test something out and had to temporarily create files with the intent to delete them later, whether it’s creating a test program, extracting a compressed archive, or even creating a named pipe (FIFO). I used to use my disk for that, using a ~/tmp directory, but it would slowly fill up when I neglected to delete these temporary files, thinking that they “might be useful later”; even to this day, I still have about 12GB of supposedly “temporary” files.

The largest files and folders in my tmp folder. Sorry, but I’m not going to show you my files!

How do you prevent these temporary files from filling up your disk? The obvious answer is to just delete them, and good for you if that works for you, but it evidently doesn’t work for me. You could stop downloading things or generating temporary files, but that’s unreasonable. One solution that I came up with when I used to run on Windows was to create a fixed-size container and put my temporary files there. I now had a maximum size for my temporary files so they wouldn’t overrun my disk. It worked for a while, but I made the mistake of making the container too small (only 256 MB), so in time, I was putting my larger temporary files in a tmp directory outside my container. I was also still not really deleting my temporary files, so in time, the container filled up.

So space-boxing the files is a good thing, but how do you make sure that your temporary files stay temporary and actually get deleted?

I can think of two ways to solve this problem: set up a scratch mount, or run a cron job. I use the first one, but the second one is equally valid.

Setting up a scratch mount

The goal of a scratch mount is to have a separate file system mounted on your temporary directory that will handle space-boxing the directory and automatically deleting the files in it. To do this, you need an appropriate file system.

tmpfs

After my switch to Arch, I learned about tmpfs. To quote the Arch wiki (emphasis mine):

tmpfs is a temporary filesystem that resides in memory and/or your swap partition(s), depending on how much you fill it up. Mounting directories as tmpfs can be an effective way of speeding up accesses to their files, or to ensure that their contents are automatically cleared upon reboot.

That’s it, then: use tmpfs to get the system to delete the files for you.

For systems using systemd, the /tmp directory is usually a tmpfs mount. Using /tmp worked for a while, but I was soon finding that my system wouldn’t function properly because there wasn’t enough space on /tmp. A tmpfs file system is, by default, half the size of your total memory. When I used /tmp as my temporary directory, I only had 4 GB of memory, so my /tmp could only hold 2 GB. Programs use /tmp for temporary files as well, so there was competition for space and when the space ran out, my system would stop functioning properly.

/scratch

The solution is to make a separate tmpfs mount on another directory, separate from /tmp. I set mine up on /scratch.

To set up this tmpfs mount, create the directory on which you want to mount. For example, to set up /scratch, run the following as root:

# mkdir /scratch

Next, add the following line to /etc/fstab, replacing 3G with the maximum size of the tmpfs mount. Remember that tmpfs is backed by RAM and swap space, so make sure you leave ample room for the system in case it ever gets filled; for reference, I chose 3G for 8G of total RAM and swap space. tmpfs will only use as much as it needs, so this size is just a limit.

tmpfs   /scratch    tmpfs   size=3G 0 0

/etc/fstab defines what should be mounted at boot time, so adding this line will mount a 3G-sized tmpfs file system on /scratch at boot time.

To test that this works, run the following as root:

# mount /scratch

If this works, you should now be able to save files under /scratch, and you should see them disappear when you reboot. You should also see that a fresh tmpfs file system is mounted under /scratch upon reboot.

Running a cron job

As an alternative to using a scratch mount, you may opt instead to use a cron job that will delete files that are older than some set relative time period.

Note: The current monospace font makes ~ look like a dash in small text sizes (~). Wherever you see something that looks like ~/tmp it’s actually ~/tmp. Unfortunately, I have no easy way to change the font.

Create your temporary directory. For example, to use ~/tmp as your temporary directory:

$ mkdir ~/tmp

The command

The command that will do the actual deleting is find:

$ find ~/tmp -atime +2 -delete

The above command will search for all files under ~/tmp not accessed within the last 3 days (yes, that is +2; see the man page for find).

To test this, you can issue a touch command, then run the find command, then see that the file you created with touch has been deleted:

$ touch ~/tmp/newfile
$ touch -d "4 days ago" ~/tmp/oldfile
$ ls ~/tmp/newfile ~/tmp/oldfile
/home/user/tmp/newfile
/home/user/tmp/oldfile
$ find ~/tmp -atime +2 -delete
$ ls ~/tmp/newfile ~/tmp/oldfile
/home/user/newfile
ls: cannot access '/home/user/tmp/oldfile': No such file or directory

If you find that checking the access time does not work, possibly because your file system does not keep track of it, you may instead replace -atime with -ctime to check the “changed” time or with -mtime to check the modified time.

The cron job

You have the command to delete old files. Now it’s time to turn it into a cron job so that it runs regularly.

Edit your crontab using the following command:

$ crontab -e

Assuming you know how to use the editor that comes up, add something like the following line:

30 07 * * * find ~/tmp -atime +2 -delete

This will set up a cron job that will delete all files under ~/tmp not accessed within the last 3 days every day at 07:30. Adjust the values to your liking.

To know if you’ve set it up properly, you will just have to wait.

Comparison

Scratch Mount Cron Job
Storage RAM + swap Disk
Size Fixed, limited to available RAM+swap Limited to available disk space
Time deleted On shutdown After a fixed amount of time
Files survive reboot? No Yes
R/W Speed As fast as RAM/swap As fast as disk
Scope System-wide (requires root) Per-user (does not require root)

There are advantages and disadvantages for both methods, and which one you use will be determined by your workflow. I use a scratch mount because I prefer to be in control of deleting files, but I wouldn’t discount using the cron job if that works better for you. You could even combine the two strategies if you wanted to: set up a tmpfs mount to provide a fast fixed-size file system that doesn’t keep files after a reboot, but also set up a cron job so that it won’t get filled over time.

Whichever method you decide to use, you still have to remember to put your temporary files there. Once you get into that habit, you can say goodbye to unmanaged temporary files!


Advertisements

$ cat your_comment

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s