Delete all but the most recent X files in bash


Question

Is there a simple way, in a pretty standard UNIX environment with bash, to run a command to delete all but the most recent X files from a directory?

To give a bit more of a concrete example, imagine some cron job writing out a file (say, a log file or a tar-ed up backup) to a directory every hour. I'd like a way to have another cron job running which would remove the oldest files in that directory until there are less than, say, 5.

And just to be clear, there's only one file present, it should never be deleted.

1
134
8/25/2008 8:37:03 AM

Accepted Answer

The problems with the existing answers:

  • inability to handle filenames with embedded spaces or newlines.
    • in the case of solutions that invoke rm directly on an unquoted command substitution (rm `...`), there's an added risk of unintended globbing.
  • inability to distinguish between files and directories (i.e., if directories happened to be among the 5 most recently modified filesystem items, you'd effectively retain fewer than 5 files, and applying rm to directories will fail).

wnoise's answer addresses these issues, but the solution is GNU-specific (and quite complex).

Here's a pragmatic, POSIX-compliant solution that comes with only one caveat: it cannot handle filenames with embedded newlines - but I don't consider that a real-world concern for most people.

For the record, here's the explanation for why it's generally not a good idea to parse ls output: http://mywiki.wooledge.org/ParsingLs

ls -tp | grep -v '/$' | tail -n +6 | xargs -I {} rm -- {}

The above is inefficient, because xargs has to invoke rm once for each filename.
Your platform's xargs may allow you to solve this problem:

If you have GNU xargs, use -d '\n', which makes xargs consider each input line a separate argument, yet passes as many arguments as will fit on a command line at once:

ls -tp | grep -v '/$' | tail -n +6 | xargs -d '\n' -r rm --

-r (--no-run-if-empty) ensures that rm is not invoked if there's no input.

If you have BSD xargs (including on OS X), you can use -0 to handle NUL-separated input, after first translating newlines to NUL (0x0) chars., which also passes (typically) all filenames at once (will also work with GNU xargs):

ls -tp | grep -v '/$' | tail -n +6 | tr '\n' '\0' | xargs -0 rm --

Explanation:

  • ls -tp prints the names of filesystem items sorted by how recently they were modified , in descending order (most recently modified items first) (-t), with directories printed with a trailing / to mark them as such (-p).
  • grep -v '/$' then weeds out directories from the resulting listing, by omitting (-v) lines that have a trailing / (/$).
    • Caveat: Since a symlink that points to a directory is technically not itself a directory, such symlinks will not be excluded.
  • tail -n +6 skips the first 5 entries in the listing, in effect returning all but the 5 most recently modified files, if any.
    Note that in order to exclude N files, N+1 must be passed to tail -n +.
  • xargs -I {} rm -- {} (and its variations) then invokes on rm on all these files; if there are no matches at all, xargs won't do anything.
    • xargs -I {} rm -- {} defines placeholder {} that represents each input line as a whole, so rm is then invoked once for each input line, but with filenames with embedded spaces handled correctly.
    • -- in all cases ensures that any filenames that happen to start with - aren't mistaken for options by rm.

A variation on the original problem, in case the matching files need to be processed individually or collected in a shell array:

# One by one, in a shell loop (POSIX-compliant):
ls -tp | grep -v '/$' | tail -n +6 | while IFS= read -r f; do echo "$f"; done

# One by one, but using a Bash process substitution (<(...), 
# so that the variables inside the `while` loop remain in scope:
while IFS= read -r f; do echo "$f"; done < <(ls -tp | grep -v '/$' | tail -n +6)

# Collecting the matches in a Bash *array*:
IFS=$'\n' read -d '' -ra files  < <(ls -tp | grep -v '/$' | tail -n +6)
printf '%s\n' "${files[@]}" # print array elements
94
3/25/2018 2:03:40 PM

Remove all but 5 (or whatever number) of the most recent files in a directory.

rm `ls -t | awk 'NR>5'`

Licensed under: CC-BY-SA with attribution
Not affiliated with: Stack Overflow
Icon