Data Migration Shell Script Example » cyberfella ltd, since 2012.

A nice little script written around the rsync command used to successfully migrate large amounts of data between NFS filesystems, avoiding .snapshot folders in the process. A simple script in essence but a nice reference example nonetheless on the use of variables, functions, if statements, case statements, patterns and some useful commands, e.g. using sed to remove whitespace at the front of a variable returned by wc.

A simple but proper shell script that can almost certainly be built/improved upon using tee to write std output to a log file as well as the screen for instance, and using find to subsequently count the number of files afterwards because df is unlikely match to the nearest megabyte across different filesystems served by different NAS’s for comparison/verification.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
SOURCE=$1
DEST=$2

#Functions section
function migratenonhiddenfolders(){

echo “Re-Synchronising non-hidden top level folders only…”

#Synchronise the data
ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR}”
timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR}
done
}

#Code section
if [[ -z $1 ]];then
echo “No Source or Destination specified”
echo “Usage: migrate.sh /<source_fs> /<destination_fs>”
exit
fi
if [[ -z $2 ]];then
echo “No Destination specified”
echo “Usage: migrate.sh /source_fs> /<destination_fs>”
exit
fi

#Source and Destination filesystems have been specified
echo “Source filesystem: $SOURCE”
FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`
echo “The $FOLDERCOUNT source folders are…”
ls -l $SOURCE | grep ^d | awk {‘print $9’}
echo
echo “Destination filesystem: $DEST”
echo
echo -n “Please confirm the details are correct [Yes/No] > “
read CONFIRM
case $CONFIRM in
[Yy] | [Yy][Ee][Ss])
migratenonhiddenfolders
;;

*)
echo
echo “User aborted.”
exit
;;
esac

#Clean exit
exit

Improved version (with logging) shown below.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
SOURCE=$1
DEST=$2

#Functions section
function migratenonhiddenfolders(){

echo “Migrating ${SOURCE} to ${DEST} at `date`” >> ~/migration.log
echo “Re-Synchronising non-hidden top level folders only…” | tee -a ~/migration.log
#Synchronise the data
ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR} | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
echo “Completed migrating to ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
done
}

#Clean exit
exit

###########################################################
##
## Data Migration script by M.D.Bradley, Cyberfella Ltd
## http://www.cyberfella.co.uk/2013/08/09/data-migration/
##
## Version 1.0 9th August 2013
###########################################################

Did you like this?
Tip cyberfella with Cryptocurrency