Listing

  • Count the number of files in a folder:

    1 $ find ./ -type f | wc -l
  • List all file extensions found in a folder:

    1 $ find ./ -type f | rev | cut -d "." -f 1 | sort | uniq | rev
  • List all files sharing the same name within the sub folders:

    1 $ find . -type f -printf "%f\n" | sort | uniq --repeated --all-repeated=separate
  • List number of files across all subfolders sharing the same name, whatever their extension is:

    1 $ find . -type f -exec basename {} \; | sed 's/\(.*\)\..*/\1/' | sort | uniq -c | grep -v "^[ \t]*1 "

Size & Space

  • List size in MiB of subfolders and files in the current folder and display them sorted by size:

    1 $ du -cm * | sort -nr
  • Show the 10 biggest files in MiB found among the current directory and its subfolders:

    1 $ find . -type f -exec du -m "{}" \; | sort -nr | head -n 10
  • Display the total size used by all PNG files in sub-directories:

    1 $ find ./ -iname "*.png" -exec du -k "{}" \; | awk '{c+=$1} END {printf "%s KB\n", c}'

Creation

  • Create several folder with a similar pattern:

    1 $ mkdir -p ./folder/subfolder{001,002,003}
  • Create a symbolic link (source):

    1 $ ln -s target link_name

Copy

  • Dump a disk to an image while monitoring the copy progression:

    1 $ pv /dev/da0 > '/mnt/tank/my-data/HDD-part1+part2.img'
  • Same as above but over SSH:

    1 $ pv /dev/da0 | ssh [email protected] "cat > /mnt/tank/my-data/HDD-part1+part2.img"

Move

  • Move files containing the date: “2005- string from the ./posts/ subfolder to the other ./2005/ subfolder:

    1 $ grep -R -l 'date: "2005-' ./posts/ | xargs -I '{}' mv '{}' ./2005/

Renaming

  • Convert all files in the current folder to lower case:

    1 $ rename 'y/A-Z/a-z/' *
  • Prefix all files in the current folder:

    1 $ rename 's/(.*)$/prefix-$1/' *
  • Rename all mp3 files in the current folder by adding a “sub-extension”:

    1 $ rename 's/\.mp3/\.my-sub-extension\.mp3/' *.mp3
  • Renaming based on regular expression, for files matching another regular expression. The particular example below was used to fix some Dropbox conflicting files:

    1 $ find ./Dropbox -type f -name "* (kev-laptop's conflicted copy 2013-02-01)*" -execdir rename -f -v "s/(.*) \(kev-laptop's conflicted copy 2013-02-01\)(.*)/\1\2/" {} \;
  • Strip filenames of their leading dot and extension composed of 6 alphanumeric characters. These are temporary files created by rsync:

    1 $ rename --force --dry-run 's/^\.(.+)\.[0-9a-zA-Z]{6}$/$1/' *

Cleaning-up

  • Delete all empty files and folders (run this command several times to remove nested empty directories):

    1 $ find ./ -empty -print -delete
  • Remove empty directories found in all subfolders starting with prefix:

    1 $ find ./ -type d -empty -ipath "./prefix*" -print -delete
  • Delete files ending with .thumbnail.jpg or .thumbnail.png files (case insensitive):

    1 $ find ./ -iregex ".*\.thumbnail\.\(jpg\|png\)$" -print -delete
  • Same as above but instead for files ending with their dimensions, like image-640x480.jpg or photo-2400x3200.png:

    1 $ find ./ -iregex ".*-[0-9]+x[0-9]+\.\(jpg\|png\)$" -print -delete
  • Here is how I clean-up copies of external drives from accumulated cruft over the past decades:

     1 # Remove metadata at volume's root.
     2 $ find . -name "System Volume Information"  -type d -depth 1 -mount -print -delete
     3 $ find . -name ".DocumentRevisions-V*"      -type d -depth 1 -mount -print -delete
     4 $ find . -name ".TemporaryItems"            -type d -depth 1 -mount -print -delete
     5 $ find . -name "\$AVG8.VAULT\$"             -type d -depth 1 -mount -print -delete
     6 $ find . -name ".Spotlight-V*"              -type d -depth 1 -mount -print -delete
     7 $ find . -name "\$RECYCLE.BIN"              -type d -depth 1 -mount -print -delete
     8 $ find . -name ".VolumeIcon.*"              -type f -depth 1 -mount -print -delete
     9 $ find . -name "autorun.inf"                -type f -depth 1 -mount -print -delete
    10 $ find . -name ".fseventsd"                 -type d -depth 1 -mount -print -delete
    11 $ find . -name ".Trash-*"                   -type d -depth 1 -mount -print -delete
    12 $ find . -name "RECYCLER"                   -type d -depth 1 -mount -print -delete
    13 $ find . -name "Recycled"                   -type d -depth 1 -mount -print -delete
    14 $ find . -name "found.*"                    -type d -depth 1 -mount -print -delete
    15 $ find . -name "\$AVG"                      -type d -depth 1 -mount -print -delete
    16 
    17 # Remove metadata file and folders artifacts.
    18 $ find . -name "desktop.ini"    -type f -mount -print -delete
    19 $ find . -name "__MACOSX"       -type d -mount -print -delete
    20 $ find . -name "Thumbs.db"      -type f -mount -print -delete
    21 $ find . -name ".DS_Store"      -type f -mount -print -delete
    22 $ find . -name "._*"            -type f -mount -print -delete
    23 
    24 # Remove empty directories (repeat until none left).
    25 $ find -type d -empty -mount -print -delete
    26 $ find -type d -empty -mount -print -delete
    27 $ find -type d -empty -mount -print -delete
  • Delete all files and folders in the current directory except the README.txt file:

    1 $ ls ./ -I "README.txt" | xargs rm -rf
  • Remove all duplicates within the whole pool of files (including —hidden ones) build up from folder-1, folder-2 and folder-3 directories. In a set of duplicates, the first file in alphabeticcaly sorted named path is kept (-S p option).

    1 $ rmlint --progress --hidden -S p ./folder-1 ./folder-2 ./folder-3
    2 $ ./rmlint.sh
  • Remove all duplicates in backup-set1 and backup-set2 if and only if they’re already present in backup-set3 (i.e. the reference folder tagged after the // separator), but do not alter the latter in anyway (thanks to the —keep-all-tagged option). To make things extra-safe we use —no-crossdev to not jump to other physical file systems:

    1 $ rmlint --progress --hidden --no-crossdev --keep-all-tagged ./backup-set1/ ./backup-set2/ // ./backup-set3/
    2 $ ./rmlint.sh

Antivirus

  • Download and refresh local ClamAV virus definition database:

    1 $ freshclam
  • Check all files, only display infected files and ring a bell when found (really slow scan):

    1 $ clamscan -r --bell -i /

Backups

  • Initialize and start a backup with restic:

    1 $ restic init
    2 $ restic backup --one-file-system ~/
  • Remove old backups:

    1 $ restic forget --keep-hourly 24 --keep-daily 15 --keep-weekly 13 --keep-monthly 12 --keep-yearly 3 --prune