Missing sidHistory attributes on migrated accounts

Log onto a server and open a command prompt as Administrator.

Issue the following dsquery to create a four column, comma separated text file of all users names, logon names, primary object SID and if applicable, sidHistory SID.  Then open this .csv file in Excel and Auto Filter the sidHistory column to show all blanks.  This is the list of accounts that have NOT been subject to an inter-domain user account migration.

dsquery * “OU=Groups,OU=MigratedGroups,OU=Cromford,OU=UK,OU=DEV,OU=VMFARM,DC=cyberfella,DC=co,DC=uk” -filter “(&(objectClass=User))” -attr samAccountName cn ObjectSID sidHistory -limit 20000 > missing-sidhistories.txt



Deleting inaccessible data such as users homedirectories

Users Home Directories are often hardened such that even Domain Administrators have problems migrating them and subsequently deleting them.  A way to deal with that is already documented here so this post is really just about the subsequent cleanup of the stubborn source data.

You can sit in Windows Explorer taking ownership and rattling the new permissions down each users tree if you like, but it’s a laborious process when you have 2000 users.  It doesn’t always work out 100% successful either.

This is my way of clearing out all users home directories that begin with the characters u5 for example.  You can adapt or scale it up it to suit your own requirements easily and save yourself a lot of time and effort.

First, make a list of the directories you want to delete.  Whether you have access to them or not is irrelevant at this stage.

dir /ad /b | findstr ^u5 > mylist.txt

dir /ad /b findstr ^U5 >> mylist.txt

Create an empty folder if you dont have one already.

mkdir empty

Now mirror that empty folder over the top of the users in the list, exploiting the operating backup right in robocopy that conveniently bypasses the NTFS security

for /f %F in (mylist.txt) DO robocopy empty %F /MIR /B /TIMFIX

This will leave empty folders behind but the security on them will have been overwritten with that of your empty folder, giving you the permission to delete it.

for /f %F in (mylist.txt) DO rmdir %F


Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.


Robocopy folders with ampersands in the name

Don’t use & in file and folder names.

With that little pearl of wisdom out of the way, what about when your users have used ampersand characters in their folder names and you’re trying to robocopy the data to it’s new home, only to have the copy fail?

Try this…

SET “source=dogs & cats”

SET “destination=dogs & cats”

or if you can get away with it without breaking links…

SET “destination=dogs and cats”

robocopy.exe “%source%” “%destination%” /MIR

For more robocopy wisdom, check this post here

In real-world practice, I have found that robocopy is woefully unreliable when it comes to copying permissions (using the /e /sec /xf * switches).  I recommend using emcopy to copy folder structures and their NTFS permissions.  Similar to the robocopy commands above, these emcopy commands worked almost* perfectly for me

SET “source=dogs & cats”

SET “destination=dogs and cats”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

*Note how I’ve changed the destination folder to not include the ampersand character.  In practice, permissions were not copied to folders with ampersands in the name using robocopy or emcopy – in fact robocopy didn’t copy permissions at all!

If you’re copying a subset of data from a bigger source set of data, then never use /MIR or you will run a high risk of loosing data.   Oh yes you will.  Use the above emcopy commands one folder at a time to get your destination folder structure in place, before finally syncing the subfolder you want into the new destination.  This saves a potentially troublesome cleanup exercise later, deleting superfluous data, e.g.


SET “source=dogs & cats”

SET “destination=dogs and cats”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

Followed by…

SET “source=dogs & cats\spaniels”

SET “destination=dogs and cats\spaniels”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

Followed by…

SET “source=dogs & cats\spaniels\springer”

SET “destination=dogs and cats\spaniels\springer”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

and finally sync your file data into the new secured folder structure…

SET “source=dogs & cats\spaniels\springer”

SET “destination=dogs and cats\spaniels\springer”

Synchronise all file data using your preferred robocopy or emcopy command here.


Export all users in ActiveDirectory

If you’re tasked with generating a list / creating a spreadsheet of all user accounts in AD but are worried you might miss out an OU when manually going through and exporting the list using the Active Directory Users and Computers MMC Snap-in, then use Powershell to generate a list instead, safe in the knowledge it’ll find everything.

If you’re really keen you can subsequently use GNUWin32 to give you neat command line tools usually only available to a bash command prompt on a Linux/UNIX OS to chop columns out of the exported csv file using cut, awk, sort and uniq.  Or just use Excel to achieve it.  More on GNUWin32 here.

Open a Powershell and type the following to export all users in the directory to a csv file…

Import-module activedirectory

get-aduser -filter * | Export-Csv c:\myusers.csv

Since the OU Path’s are themselves comma separated, it throws the keys in the csv out of alignment, making it challenging to extract the columns to the right of it that contains the samAccountName  “Logon Name”.  To get over this hurdle, go back to PowerShell and be more specific about the exact key (or Label) you want, e.g. if you just want a list of Logon Names for all users in AD, then this command works…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

Some other useful Labels you may want to use are shown below for your convenience (including a neat If statement for extracting Disabled Accounts).

@{Label = “First Name”;Expression = {$_.GivenName}}
@{Label = “Last Name”;Expression = {$_.Surname}}
@{Label = “Display Name”;Expression = {$_.DisplayName}}
@{Label = “Logon Name”;Expression = {$_.sAMAccountName}}
@{Label = “Full address”;Expression = {$_.StreetAddress}}
@{Label = “City”;Expression = {$_.City}}
@{Label = “State”;Expression = {$_.st}}
@{Label = “Post Code”;Expression = {$_.PostalCode}}
@{Label = “Country/Region”;Expression = {if (($_.Country -eq ‘GB’) ) {‘United Kingdom’} Else {”}}}
@{Label = “Job Title”;Expression = {$_.Title}}
@{Label = “Company”;Expression = {$_.Company}}
@{Label = “Description”;Expression = {$_.Description}}
@{Label = “Department”;Expression = {$_.Department}}
@{Label = “Office”;Expression = {$_.OfficeName}}
@{Label = “Phone”;Expression = {$_.telephoneNumber}}
@{Label = “Email”;Expression = {$_.Mail}}
@{Label = “Manager”;Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}}
@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}}
@{Label = “Last LogOn Date”;Expression = {$_.lastlogondate}}

You can combine the Labels above in a single command with a comma in the select-object section, for example to extract all logon names and whether or not the account is disabled…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}},@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

I had some trouble with the LastLogon Label, so have included the working example used to obtain this information below.

get-aduser -filter * -properties * | select-object @{Label = “LogonName”;Expression = {$_.saMAccountName}},@{Label = “LastLogonDate”;Expression = {$_.LastLogonDate}}| Export-Csv D:\ADUsers\ADUsers.LastLogon.csv



How to cable up VNX SP Ports (Dual Fabric topology)

So your VNX has two SP’s and you have two fabric switches.  You already know you have to connect each SP to each fabric for resilience, but you’re still a bit confused.  Fear not.  Use this as a guide.  It can be used no matter how many front-end port modules and SFP’s you have so that you get it right first time for all your designated Storage Ports, Mirrorview Ports, Sancopy Ports required for your project.  The FC Switch Ports you choose are not set in stone, but keep it the same on each side at least.

 What’s important is that the correct SP port, goes to the correct switch.

When you know you’ve got it right, you can get verify the WWN in Unisphere corresponds with the WWN logged in on the FLOGI database on the switch to check before you create the requisite fcaliases etc.


It’s much easier to build it right first time than sort it out afterwards.



Inject Administrators/Full Control permissions into inaccessible folders.

Note:  This can also be used to inject Everyone/Full Control, or a specific user, using the username or SID.  The Administrators Group SID is always S-1-5-32-544.  Other well-known SIDs are listed here.

Download the command line version of SetACL.exe from here.  Like all the best things in life, it’s free.

Open a command prompt as Adminstrator (right click cmd.exe, run as admin)

setacl -on “C:\Private No Entry” -ot file -actn ace -ace “n:Administrators;p:full” -rec cont_obj -ignoreerr

The “Private No Entry” folder should now have Administrators, Full Control Permissions.  If not, don’t fret, read on…

The following command gives Administrators the “dream ticket” to accessing all data by setting ownership to Administrators on all folders and files and forcing subdirectories to re-inherit inheritable Administrators:Full Control permissions from the parent.

setacl -on “C:\Private No Entry” -ot file -actn setprot -op “dacl:np;sacl:nc” -rec cont_obj -actn setowner -ownr “n:S-1-5-32-544”

If you still receive “Operating System Message:Access Denied” or similar, then you’ll need to take a robocopy of the “inaccessible” data using the /B switch to exploit OS Backup Right, leaving permissions behind using /COPY:DAT (instead of /COPY:DATSOU or /COPYALL) then repeat the process above on the copied data instead.

robocopy “C:\Private No Entry” “T:\Cracked Data” /B /COPY:DAT /E /NP /R:1 /W:1

Now view the Inherited permissions on the copied data…  You’ll see it has a whole bunch of new, open permissions that it’s got from the parent folder T:.

cacls “T:\Cracked Data”

The cracked data could be robocopied back over the original inaccessible source data using /MIR /COPYALL /SEC /SECFIX switches if required.  If it doesn’t allow it, then note that I have successfully robocopied an empty folder over the top of an inaccessible folder before using just /MIR  (in order to delete it), then robocopied the cracked data back into place, e.g.

robocopy “T:\Empty Folder” “C:\Private No Entry” /MIR /B

robocopy “T:\Cracked Data” “C:\Private No Entry” /MIR /SEC /B

Finally, if you want to re-harden the folder whilst retaining the access you’ve granted Administrators, then use the following commands…

Presently, access has been attained via inherited permissions so before removing inheritance, first inject a non-inherited ACE that allows administrators access, i.e.

setacl -on “C:\Private No Entry” -ot file -actn ace -ace “n:S-1-5-32-544;p:full” -rec cont_obj

Verify the Administrators:Full Control permissions are present on the folder

cacls “C:\Private No Entry”

Finally it is safe to remove inheritance without losing access (strictly speaking, you are “protecting the child object from inherited permissions on the parent object”)

setacl -on u567149 -ot file -actn setprot -op “dacl:p_nc;sacl:p_nc”

This sequence of commands can be used to copy users home directories that are typically hardened to only permit the user themselves access to the data contained within.  If you are using it to migrate home dierctories, there is a loop to re-apply user-specific permissions to each homedirectory afterwards here


Deleting Windows data where the path length exceeds 260 characters

After migrating Windows data, it can be a royal pain cleaning up the source data using del *.* /s /q /f, especially when the path length exceeds 260 (or thereabouts) characters.  You can manually shorten the folder names and keep trying, but this may be time consuming, tiring and ultimately futile.

The simplest way I’ve found to reliably delete data, irrespective of path length, is to use robocopy.

  1. cd into the directory that you want to empty
  2. create a new empty subdirectory called empty
  3. rename all other adjacent folders 1, 2, 3, 4 etc if possible
  4. robocopy empty 1 /mir /r:1 /w:1
  5. repeat for each adjacent folder, 2, 3, 4 etc.



Robocopy leaves some NTFS permissions behind?

And so does emcopy and icacls /save & /restore doesn’t work either…

Googling doesn’t help – every solution that promises to work, doesn’t.



Sorry for shouting, but I’m really rather excited to have cracked this major show stopper for my clients data migration.  The /B switch uses the Backup right to perform the copy.  That’s presumably running with system level privs, rather than my meager admin account in cmd run as administrator.  Magic.  Data integrity restored!  Professional reputation saved!

Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.


Find ACL’s that don’t match parent with AccessEnum

When data begins its life, the permissions are invariably set at one of the top two or three levels, Some examples of some types of data you wouldn’t want everyone to get their hands on in a given company might be…

X:\Payroll\Cost Centre 1\Salaries

X:\Human_Resources\Cost Centre 2\Disciplinaries

X:\Pensions\Cost Centre 3\Compulsory Redundancy Quotes

All users would have to have access to the top level share, then all users might still have access to the Human Resources, Payroll and Pensions folders too.   There may or may not be a cost centre folder to aid the business in understanding what parts of what department are consuming the most storage space (for internal billing), then there will be the “parent” folders (top level in terms of the point where specific permissions are set) whereby only certain individuals in a global group will have access.  ABE (Access Based Enumeration) might be enabled also, preventing users who don’t have access to certain folders from even seeing that the folder exists in the first place.

For the purposes of data migration and reporting, the IT department should have some kind of data admins global group that has full control from the top, all the way down.  Backup admins will also need modify access in order to perform file recoveries back to their original locations.

Over time, chunks of data get moved about as a result of departmental “tidy ups” and restructuring.  The effect this has on permissions is that when data is “copied” it inherits the security of the parent folders on the destination side, however when it is “moved” it takes its permissions with it – sort of.  I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.

I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.  And therein lies a problem for subsequent data migrations.

The effect this has, is to cause the folders on the migrated side (where a copy operation has been used) to subsequently re-inherit their permissions from their new parent.  Only non inherited permissions and inheritance attributes are ever copied, so these permissions that were once inherited but can no longer be inherited are likened to “ghosts” and are lost during any kind of copy operation (data migration operation using emcopy/robocopy or some other method).

I’d like to use “orphans” to describe these children with no matching parent permissions but the term is already taken to describe SIDs in ACEs that no longer resolve to a group in AD (occurs when a folder is secured then the group subsequently deleted), so I’ve come up with “ghosts” to describe them.  I can’t use “zombies” as that is taken to refer to a child process on a UNIX system that has completed execution but remains in the process table so until its parent process can read its exit status.  “Ghosts” is quite apt as they are not tangible and vanish when you migrate the data.

Technically, the copy operation is correcting anomalies but in the real world, that means loss of access, or worse, open access, depending on the nature of the change of permissions on the new parent.

The best way to deal with these ghosts is at source, but you need to know about them first, in order to deal with them.  Sysinternals accessenum GUI tool is a neat way to scan filesystems looking for children where the security differs from that of its parent.  Be sure to set the options accordingly.



Powershell and the NTFSSecurity Module

The powershell NTFSSecurity module provides cmdlets to export and import security.  Unlike icacls which sticks to using sddl format (for a 10 fold increase in speed exporting security for large filesystems), powershell will resolve the SIDs in sddl format into human friendly names by chatting to the DC as it goes.  Some useful commands are noted below.

Download from here
NTFSSecurity Module for Powershell

Just create the folder “NTFSSecurity” in the folder set according to the environment variable %PSModulePath%

The module should now be listed in “Get-Module -ListAvailable” and can be imported using “Import-Module NTFSSecurity“.

For example, all the files in the zip file have to be in “%SystemRoot%\system32\WindowsPowerShell\v1.0\Modules\NTFSSecurity\“.
If you did this then the module should be listed in “Get-Module -ListAvailable” and can be imported using “Import-Module NTFSSecurity“.

Note that running Windows Powershell Modules (look for the powershell icon with the admin shield in the corner in your start menu), will automatically load the module upon CLI startup.

#to backup permissions just pipe what Get-Ace returns to Export-Csv
get-childitem -Recurse | Get-Ace -ExcludeInherited | Export-Csv permissions.csv

#to restore the permissions pipe the imported data to Add-Ace
#As the imported data also contains the path you do not need to specify the item
Restore: Import-Csv .\permissions.csv | Add-Ace

get-childitem -Recurse | get-inheritance | export-CSV C:\inheritanceon.csv -NoTypeInformation

get-childitem -recurse -exclude *.* | get-ace | export C:\migrationscripts\incinherited.csv -notypeinformation