Category: SAN/NAS/Storage

When migrating from one domain to another (usually new) domain, the users and groups are migrated so as to retain their historical SID.  This is so that the migrated account in the new domain can still access resources in the old domain that were secured against that users unique SID or groups to which that user was once a member.  When a user account is migrated, a new account is effectively created with a new, unique SID, but the old SID is also retained.

This is convenient so that everything continues to “hang together” in terms of newly migrated users being able to access new data and old data pending its inevitable migration to file servers or NAS in the new domain.

Once the data is migrated to the new domain, then the ACL’s will still likely contain ACE’s of user and group SIDs from the legacy domain.  The migrated users will still be able to access this data provided the old domain continues to exist for the purposes of authenticating old SIDs but at some point, the old domain will likely be scheduled for decomissioning to free up money and space associated with cooling, electricity and ongoing support overheads.  Another option would be to virtualize an old DC and have done with it.

The legacy user and group SID’s contained in the ACE’s will need to be translated to the primary SIDs of the new migrated users and groups to remove the need for legacy DC’s to remain in the new environment.  This can be done using SetACL – the freely available tool mentioned in other posts such as Removing Orphaned SIDs.

The following prose is a combination of my own notes and those found on the Helge Klein website where SetACL can be downloaded.

Migration – the Flexible Way

SetACL 3 comes with a more flexible way of handling permissions migrations. Its enhanced migration engine can be used to migrate intra-domain, inter-domain or any mix of the two. To start a migration using the flexible model use a command line similar to the following:

SetACL -on D:\Data -ot file -actn trustee -trst csv:C:\temp\mappings.csv;ta:repltrst -rec cont

This instructs SetACL to copy permissions between individual users or groups for the D:\Data directory tree. Which accounts the migration is performed for is read from a simple CSV file, mappings.csv in the example, whose contents might look like this:

MyOldDomain\Joe,MyNewDomain\Peter
MyOldDomain\Mary,MyNewDomain\Mary
MyOldDomain\DG-OldDom-HelpDesk,MyNewDomain\DG-NewDom-HelpDesk
MyOtherDom\Fred,MyNewDomain\Fred

or like this… (both formats are supported, I prefer this one i.e. sddl format)

S-1-5-21-1552345877-4092344333-3107638942-187127,S-1-5-21-3615417180-1828292988-6264
S-1-5-21-1552345877-4092344333-3107638942-187157,S-1-5-21-3615417180-1828292988-6294
S-1-5-21-1552345877-4092344333-3107638942-187177,S-1-5-21-3615417180-1828292988-6394

As you can see, the system is flexible, powerful and easy to use. All you have to do is create a mapping table pairing old account with new account. If you do not want to do that or if you think it is just too much bother: the simpler method presented above still works and is still considered useful in many scenarios.

Command reference
https://helgeklein.com/setacl/documentation/command-line-version-setacl-exe

Working example

Back up directory security first.  Always use a starting folder, not the root of the drive.  The restores do not work to the root of a drive as the export file has an empty first line that upsets the restore process.  This means you may need to back up each top level folder individually so that your restores would work or you can just delete the first empty line (but be aware you’ll not recover its permissions and applying them manually will cause them to subsequently ripple down through the tree (no)thanks to Windows’ behaviour.)

icacls D:\sourcedata /save ACL_Dep.txt /T /C    (Traverse and continue on errors )

backups of acl’s can be restored as follows:

icacls D:\ /restore ACL_Dep.txt /c   (continue on errors such as missing files)

Map files can be created as follows.
Dump SID + SID history to a file for Migrated Groups: (change the OU path with the correct one for your AD)

dsquery * “OU=Groups,OU=MigratedGroups,OU=Cromford,OU=UK,OU=DEV,OU=VMFARM,DC=cyberfella,DC=co,DC=uk” -filter “(&(objectClass=Group)(sidHistory=*))” -attr sidHistory ObjectSID -limit 20000 > Sidmapping.txt

Check if –limit option is set large enough given the number of groups in your domain. Perform necessary edits to the sidmapping.txt file until it’s format looks like this (below).

S-1-5-21-2121000310-530922351-1601650009-38709,S-1-5-21-1477100029-3748294062-1861910009-37469
S-1-5-21-2121000310-530922351-1601650009-171643,S-1-5-21-1477100029-3748294062-1861910009-37983
S-1-5-21-2121000310-530922351-1601650009-67673,S-1-5-21-1477100029-3748294062-1861910009-38089
S-1-5-21-2121000310-530922351-1601650009-67965,S-1-5-21-1477100029-3748294062-1861910009-38585
S-1-5-21-2121000310-530922351-1601650009-68256,S-1-5-21-1477100029-3748294062-1861910009-38867
S-1-5-21-2121000310-530922351-1601650009-54379,S-1-5-21-1477100029-3748294062-1861910009-39244
S-1-5-21-2121000310-530922351-1601650009-127676,S-1-5-21-1477100029-3748294062-1861910009-39397

Re-ACL Migrated Group ACE’s from the historical SIDs currently securing the data to the primary object SIDs in the new domain using the freely available SetACL tool.

setacl -on D:\destinationdata -ot file -actn trustee -trst csv:sidmapping.txt;ta:repltrst -rec cont_obj –log reacl_log.txt

The following command will delete any orphaned SIDs (ACE’s that to don’t resolve to any group in AD and thus serve no purpose)

setacl -on D:\destinationdata -ot file -actn delorphanedsids -rec cont_obj

EXAMPLE TIMINGS

(Obviously your results may vary depending on many variables such as number of folder, files, rows in sid map csv, disk speed, cpu speed, etc)

Exporting ACLs for 128GB filesystem consisting of 30,000 files in 2,750 folders on an HP ProBook CORE i5 laptop took 15 seconds.and processed every file successfully although problems are generally encountered if files have been created using file explorer run as administrator (the ownership changes from the logged on user to admnistrators).

5 new groups were added to the acl’s at the top of the tree and re-exporting the security took 17 seconds (a further 3 seconds)

Re-ACLing the filesystem using a SID Map containing 5 rows to translate ALL ACE’s referencing the newly added groups to a single new group, took 5 minutes, 45 seconds.

Copying the folder structure only but leaving all files behind (create duplicate empty folder structure with security using emcopy), took 14 seconds.

Re-ACLing the empty folder structure (2,750 folders), avoiding the overhead of scanning and re-acling 30,000 files took 25 seconds.

It is therefore recommended that re-acling take place on a duplicate empty folder structure and data (files) are subsequently copied into it using emcopy with the /nosec switch.

For large filesystems, say over 1TB, you may want to process each top level folder as its own individual job, looping through all top level folders sequentially.  It’s definitely my preferred approach with pretty much everything, not just setacl.  You can use a for loop to do this, e.g.

for /f %%f in (‘dir s:\ /ad /b’) DO setacl.exe -on “d:\root\%%f”  -ot file -actn trustee -trst csv:sidmap.txt;ta:repltrst -rec cont_obj

INHERITANCE PROTECTION ISSUE

As is often the case, the real-world (the source of all my blog posts) presents some unforseen issues around the use of these complex tools, and setacl appears to have an issue whereby re-acling using the trustee action above (the same method as documented on Helge Klein’s website) also performs a reset on the inheritance attribute, potentially opening up hardened subfolders where the original inherited ace’s have been partially retained and inheritance turned off.

The following sequence of commands appear to correct the issue by re-breaking inheritance, removing inherited ace’s, applying specific permissions to the subfolder before turning inheritance back on on the parent.

Consider the following structure.  T:\level 1\level 2\level 3  where level 2 inherits from level 1 but level 3 stops inheriting and has its own permissions set.  If these permissions were originally ace’s that were inherited, then re-permissioning can reset inheritance.  The following sequence of commands seems to correct it.  Your mileage may vary.

setacl -on “t:\level 1\level 2” -ot file -actn setprot -op “dacl:p_nc;sacl:p_nc”          (Protection, No Copy Inherited ACEs)

setacl -on “t:\level 1\level 2\level 3” -ot file -actn ace -ace “n:S-1-5-21-12345-12345-12345-1234;p:full” -ace “n:S-1-5-21…. and so on.

setacl -on “t:\level 1\level 2\level 3” -ot file -actn setprot -op “dacl:p_nc;sacl:p_nc”  (Protection, Remove Inherited ACEs)

setacl -on “t:\level 1\level 2” -ot file -actn setprot -op “dacl:np;sacl:np”                  (No Protection)

Did you like this?
Tip cyberfella with Cryptocurrency

• Bitcoin
• Monero

Donate Bitcoin to cyberfella

Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Bitcoin Cash to cyberfella

Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Ethereum to cyberfella

Scan the QR code or copy the address below into your wallet to send some Ether:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Litecoin to cyberfella

Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Monero to cyberfella

Scan the QR code or copy the address below into your wallet to send some Monero:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate ZCash to cyberfella

Scan the QR code or copy the address below into your wallet to send some ZCash:

Copy ZCash Address

[X] Click Here to Hide Donation Details

As a compliment to my recent post “Data Migration using emcopy”

http://www.cyberfella.co.uk/2014/05/02/emcopy/

I thought it only fair to follow up with an equivalent post for good ol’ robocopy.  This has mainly come about having discovered an annoying bug in emcopy whereby it doesn’t ignore the directories specified by more than one /xd exclusion – it always excludes the last one specified, but none of the others?!

Robocopy on the other hand, does allow exclusion of more than one directory, each one specified using the /xd switch and can be a full path (to exclude very specific directories) or just one word (to exclude any directories with that name anywhere in the directory tree).

The switch worth mentioning the most though, is the /FFT switch.  Update:  AND THE /B SWITCH (use backup rights). Uh, and come to think of it, the /XO switch too (especially if you’re running a repeated backup of data to a USB HDD).

Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.

When migrating data into a Celerra/VNX/NetApp CIFS Server, the act of copying data from a NTFS volume on a Windows Server to a Linux based Filesystem on a NAS is enough to throw the timestamps on the files out just enough to make robocopy think that the source file is newer, even when it’s not.  This means that subsequent copies of the changed files take just as long as the initial copy.

By appending /FFT to the long list of switches used in your robocopy command, it allows for a discrepancy of up to 2 seconds – enough to provide a convenient workaround to this problem.

In practice, this brought down troublesome 36 hour copy operations requiring a weekend cut-over to be arranged, down to just over 1 hour – cue cliche  – saving time and money.

An example command is given below…

robocopy d:\source e:\dest *.* /xd “System Volume Information” d:\Migration homedirs profiles wtsprofiles /e /np /fft /xo /r:1 /w:1

There are many more switches available in robocopy, including the ability to use multiple threads in newer versions (highly recommended).  Just type robocopy /? from the Windows command line to see the other options.

In practice I found emcopy to be inconsistent with copying ACE’s across to large filesystems, completely skipping some folders when creating an empty folder structure using the /xf * /create method.  This means that file data (and the missing subfolders) subsequently copied into place with /nosec would be forced to inherit the parent permissions.  Most likely not a problem, but if the data has lots of bespoke permissions then it becomes a huge problem as data is generally more “open” at the parent levels.

To re-sync permissions, the following command was useful.

for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy /E /Copy:S /IS /IT q:\%%f g:\root\%%f

This has since been updated.  To re-sync folder perms between source and dest trees, this works…

robocopy s:\ d:\ /lev:3 /MIR /SEC /SECFIX /V /B /TIMFIX /xo /xn /xc /r:1 /w:1

Due to folders being missed, I never deal with a file system using a single command.  I always break it up and handle each top level folder as an individual job by placing the command in a for loop as shown above.

Alternatively use the following to replicate changed files and their security, and also set the security on unchanged files.  The /V shows the unchanged files being fixed.

for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy q:\ g:\root\ /MIR /SEC /SECFIX /V /B /TIMFIX /r:1 /w:1

I found myself fighting for a day or two with an apparent intermittent problem copying NTFS security when robocopying data from NAS to NAS.  Despite using all the methods described above, sometimes the NTFS permissions just weren’t being copied across.  I have since discovered that using the /B switch with every other method mentioned already, fixes this annoying problem and the ACE’s come across perfectly.

I’ve since encountered odd behaviour using for loops that has resulted in a mistrust of them.  So, I code each top level folder as an individual line in a batch file.  The problems were encountered where there were spaces in the folder names irrespective of using “delims=”, robocopy didn’t always get it right thereafter.

Robocopy doesn’t let you copy certain folders.  It lets you exclude certain folders but that’s not much use if you only want to copy folders starting with u6* for example.  In this situation, e.g. migrating all users whose usernames begin with u6 to a separate filesystem, you need to use the for loop.

for /f “delims=” %f IN (‘dir s:\root\u5* /ad /b’) DO robocopy s:\root\%f t:\root\%f /COPYALL /R:1 /W:1 /ZB /NP /L /FFT /LOG+:D:\cyberfellaltd\u6mig.log

Update: 28/2/2017  Real World Example: Migration of a subset of users to new filesystem.  Two passes, two different approaches.  One does initial copy of just usernames beginning with u5, the second generates a list of missing users after the first pass and does a second pass targeting the missing users.  Note that this is a hybrid Bash/Batch script and requires the installation of GNUWin32 on Windows in order to work.  This is covered here.

for /f “delims=” %%f IN (‘dir s:\root\u5* /ad /b/ o’) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does first pass on all u5 users)

dir /ad /b /o s:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\src.txt | wc –l      (counts 2113 and writes list of all u5 users to src.txt)

dir /ad/b /o t:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\dest.txt | wc –l    (counts 2113 and writes list of all u5 users to dest.txt)

comm -23 t:\src.txt t:\dest.txt | tee t:\missing.txt | wc –l  (counts 0 differences and writes list of any missing u5 users to missing.txt)

for /f “delims=” %%f IN (cat t:\missing.txt) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does 2^nd pass on any missing users only)

If you have folders containing ampersand characters in the name, your copies can fail.  This post here covers a way to deal with it using variables.

Did you like this?
Tip cyberfella with Cryptocurrency

• Bitcoin
• Monero

Donate Bitcoin to cyberfella

Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Bitcoin Cash to cyberfella

Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Ethereum to cyberfella

Scan the QR code or copy the address below into your wallet to send some Ether:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Litecoin to cyberfella

Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate Monero to cyberfella

Scan the QR code or copy the address below into your wallet to send some Monero:

Donate via Installed Wallet Copy

[X] Click Here to Hide Donation Details

Donate ZCash to cyberfella

Scan the QR code or copy the address below into your wallet to send some ZCash:

Copy ZCash Address

[X] Click Here to Hide Donation Details