As a compliment to my recent post “Data Migration using emcopy”
http://www.cyberfella.co.uk/2014/05/02/emcopy/
I thought it only fair to follow up with an equivalent post for good ol’ robocopy. This has mainly come about having discovered an annoying bug in emcopy whereby it doesn’t ignore the directories specified by more than one /xd exclusion – it always excludes the last one specified, but none of the others?!
Robocopy on the other hand, does allow exclusion of more than one directory, each one specified using the /xd switch and can be a full path (to exclude very specific directories) or just one word (to exclude any directories with that name anywhere in the directory tree).
The switch worth mentioning the most though, is the /FFT switch. Update: AND THE /B SWITCH (use backup rights). Uh, and come to think of it, the /XO switch too (especially if you’re running a repeated backup of data to a USB HDD).
Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.
When migrating data into a Celerra/VNX/NetApp CIFS Server, the act of copying data from a NTFS volume on a Windows Server to a Linux based Filesystem on a NAS is enough to throw the timestamps on the files out just enough to make robocopy think that the source file is newer, even when it’s not. This means that subsequent copies of the changed files take just as long as the initial copy.
By appending /FFT to the long list of switches used in your robocopy command, it allows for a discrepancy of up to 2 seconds – enough to provide a convenient workaround to this problem.
In practice, this brought down troublesome 36 hour copy operations requiring a weekend cut-over to be arranged, down to just over 1 hour – cue cliche – saving time and money.
An example command is given below…
robocopy d:\source e:\dest *.* /xd “System Volume Information” d:\Migration homedirs profiles wtsprofiles /e /np /fft /xo /r:1 /w:1
There are many more switches available in robocopy, including the ability to use multiple threads in newer versions (highly recommended). Just type robocopy /? from the Windows command line to see the other options.
In practice I found emcopy to be inconsistent with copying ACE’s across to large filesystems, completely skipping some folders when creating an empty folder structure using the /xf * /create method. This means that file data (and the missing subfolders) subsequently copied into place with /nosec would be forced to inherit the parent permissions. Most likely not a problem, but if the data has lots of bespoke permissions then it becomes a huge problem as data is generally more “open” at the parent levels.
To re-sync permissions, the following command was useful.
for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy /E /Copy:S /IS /IT q:\%%f g:\root\%%f
This has since been updated. To re-sync folder perms between source and dest trees, this works…
robocopy s:\ d:\ /lev:3 /MIR /SEC /SECFIX /V /B /TIMFIX /xo /xn /xc /r:1 /w:1
Due to folders being missed, I never deal with a file system using a single command. I always break it up and handle each top level folder as an individual job by placing the command in a for loop as shown above.
Alternatively use the following to replicate changed files and their security, and also set the security on unchanged files. The /V shows the unchanged files being fixed.
for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy q:\ g:\root\ /MIR /SEC /SECFIX /V /B /TIMFIX /r:1 /w:1
I found myself fighting for a day or two with an apparent intermittent problem copying NTFS security when robocopying data from NAS to NAS. Despite using all the methods described above, sometimes the NTFS permissions just weren’t being copied across. I have since discovered that using the /B switch with every other method mentioned already, fixes this annoying problem and the ACE’s come across perfectly.
I’ve since encountered odd behaviour using for loops that has resulted in a mistrust of them. So, I code each top level folder as an individual line in a batch file. The problems were encountered where there were spaces in the folder names irrespective of using “delims=”, robocopy didn’t always get it right thereafter.
Robocopy doesn’t let you copy certain folders. It lets you exclude certain folders but that’s not much use if you only want to copy folders starting with u6* for example. In this situation, e.g. migrating all users whose usernames begin with u6 to a separate filesystem, you need to use the for loop.
for /f “delims=” %f IN (‘dir s:\root\u5* /ad /b’) DO robocopy s:\root\%f t:\root\%f /COPYALL /R:1 /W:1 /ZB /NP /L /FFT /LOG+:D:\cyberfellaltd\u6mig.log
Update: 28/2/2017 Real World Example: Migration of a subset of users to new filesystem. Two passes, two different approaches. One does initial copy of just usernames beginning with u5, the second generates a list of missing users after the first pass and does a second pass targeting the missing users. Note that this is a hybrid Bash/Batch script and requires the installation of GNUWin32 on Windows in order to work. This is covered here.
for /f “delims=” %%f IN (‘dir s:\root\u5* /ad /b/ o’) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does first pass on all u5 users)
dir /ad /b /o s:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\src.txt | wc –l (counts 2113 and writes list of all u5 users to src.txt)
dir /ad/b /o t:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\dest.txt | wc –l (counts 2113 and writes list of all u5 users to dest.txt)
comm -23 t:\src.txt t:\dest.txt | tee t:\missing.txt | wc –l (counts 0 differences and writes list of any missing u5 users to missing.txt)
for /f “delims=” %%f IN (cat t:\missing.txt) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does 2nd pass on any missing users only)
If you have folders containing ampersand characters in the name, your copies can fail. This post here covers a way to deal with it using variables.
Tip cyberfella with Cryptocurrency
Donate Bitcoin to cyberfella
Donate Bitcoin Cash to cyberfella
Donate Ethereum to cyberfella
Donate Litecoin to cyberfella
Donate Monero to cyberfella
Donate ZCash to cyberfella
