Missing sidHistory attributes on migrated accounts

Log onto a server and open a command prompt as Administrator.

Issue the following dsquery to create a four column, comma separated text file of all users names, logon names, primary object SID and if applicable, sidHistory SID.  Then open this .csv file in Excel and Auto Filter the sidHistory column to show all blanks.  This is the list of accounts that have NOT been subject to an inter-domain user account migration.

dsquery * “OU=Groups,OU=MigratedGroups,OU=Cromford,OU=UK,OU=DEV,OU=VMFARM,DC=cyberfella,DC=co,DC=uk” -filter “(&(objectClass=User))” -attr samAccountName cn ObjectSID sidHistory -limit 20000 > missing-sidhistories.txt



Deleting inaccessible data such as users homedirectories

Users Home Directories are often hardened such that even Domain Administrators have problems migrating them and subsequently deleting them.  A way to deal with that is already documented here so this post is really just about the subsequent cleanup of the stubborn source data.

You can sit in Windows Explorer taking ownership and rattling the new permissions down each users tree if you like, but it’s a laborious process when you have 2000 users.  It doesn’t always work out 100% successful either.

This is my way of clearing out all users home directories that begin with the characters u5 for example.  You can adapt or scale it up it to suit your own requirements easily and save yourself a lot of time and effort.

First, make a list of the directories you want to delete.  Whether you have access to them or not is irrelevant at this stage.

dir /ad /b | findstr ^u5 > mylist.txt

dir /ad /b findstr ^U5 >> mylist.txt

Create an empty folder if you dont have one already.

mkdir empty

Now mirror that empty folder over the top of the users in the list, exploiting the operating backup right in robocopy that conveniently bypasses the NTFS security

for /f %F in (mylist.txt) DO robocopy empty %F /MIR /B /TIMFIX

This will leave empty folders behind but the security on them will have been overwritten with that of your empty folder, giving you the permission to delete it.

for /f %F in (mylist.txt) DO rmdir %F


Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.


Download the full Firefox stand-alone installer

There’s nothing more frustrating than downloading an installer that assumes that you’re going to have internet access on the machine that you subsequently intend to run the installer on (called a stub installer).

For example, downloading firefox so that you can get to your enterprise storage arrays java based admin interface without the agony presented by internet explorer’s tendency to throw its toys out the pram over the certificate and the settings are locked down by IE policy, this policy, that policy and the other policy that all exist to make the environment so much more “secure” but actually just don’t allow anything, anywhere, ever.  It’s secure!, it’s been signed off as being suitably unusable to prevent exposing ourselves to any kind of imaginary threat!  Aren’t we clever?.  No.  Rant over.

It’s secure!, it’s been signed off as being suitably unusable to prevent exposing ourselves to any kind of imaginary threat!

I’ve probably digressed, I can’t tell.  I’m too angry.  And you are too probably, if you’ve ended up here.  Installers that assume an internet connection are completely useless in the enterprise environment (best read in the voice of Clarkson).

Whats even more frustrating is that the stub installer is the only apparent option, judging by mozillas website.  Well it isn’t the only option – you can still download the full-fat, stand-alone installer from their ftp site – but ftp is blocked by your firewall!

No bother, just replace ftp:// with http:// at the beginning of the URL, or even better just click here for the 64 bit version (or here for the 32 bit version).



Users home directory folders displayed as “My Documents”

When viewing a mounted shared filesystem that contains users home directories, many of the folders will be displayed in Windows Explorer as “My Documents” instead of the logon name e.g. bloggsj.  When you’re looking for a particular users home directory and they’re all called “My Documents” it can get quite frustrating.

This occurs as a result of the users home directory containing a desktop.ini file and your windows client is designed to automatically assume you’re looking at your own home directory.  It’s basically not smart enough to figure out it’s not yours but somebody elses.  You’d think they’d patch this but they haven’t yet and it’s been this way now for years.

So, what to do (other than use command line to do everything)?

In Windows Explorer…

  1. Navigate to share eg \\Server\Users
  2. Right click on column SIZE
  3. Click on More at the bottom
  4. tick Filename
  5. Drag Filename column to the leftmost column and sort on it.  (optional)

You then get an extra column showing the real filename that will totally overcome the problem and give you the visibility you want.

You could make this the default for folders by doing this.

In Windows Explorer

  1. Press ALT (to display the old fashioned menu)
  2. choose Tools->Folder Options->View
  3. Click Apply to Folders



Export all users in ActiveDirectory

If you’re tasked with generating a list / creating a spreadsheet of all user accounts in AD but are worried you might miss out an OU when manually going through and exporting the list using the Active Directory Users and Computers MMC Snap-in, then use Powershell to generate a list instead, safe in the knowledge it’ll find everything.

If you’re really keen you can subsequently use GNUWin32 to give you neat command line tools usually only available to a bash command prompt on a Linux/UNIX OS to chop columns out of the exported csv file using cut, awk, sort and uniq.  Or just use Excel to achieve it.  More on GNUWin32 here.

Open a Powershell and type the following to export all users in the directory to a csv file…

Import-module activedirectory

get-aduser -filter * | Export-Csv c:\myusers.csv

Since the OU Path’s are themselves comma separated, it throws the keys in the csv out of alignment, making it challenging to extract the columns to the right of it that contains the samAccountName  “Logon Name”.  To get over this hurdle, go back to PowerShell and be more specific about the exact key (or Label) you want, e.g. if you just want a list of Logon Names for all users in AD, then this command works…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

Some other useful Labels you may want to use are shown below for your convenience (including a neat If statement for extracting Disabled Accounts).

@{Label = “First Name”;Expression = {$_.GivenName}}
@{Label = “Last Name”;Expression = {$_.Surname}}
@{Label = “Display Name”;Expression = {$_.DisplayName}}
@{Label = “Logon Name”;Expression = {$_.sAMAccountName}}
@{Label = “Full address”;Expression = {$_.StreetAddress}}
@{Label = “City”;Expression = {$_.City}}
@{Label = “State”;Expression = {$_.st}}
@{Label = “Post Code”;Expression = {$_.PostalCode}}
@{Label = “Country/Region”;Expression = {if (($_.Country -eq ‘GB’) ) {‘United Kingdom’} Else {”}}}
@{Label = “Job Title”;Expression = {$_.Title}}
@{Label = “Company”;Expression = {$_.Company}}
@{Label = “Description”;Expression = {$_.Description}}
@{Label = “Department”;Expression = {$_.Department}}
@{Label = “Office”;Expression = {$_.OfficeName}}
@{Label = “Phone”;Expression = {$_.telephoneNumber}}
@{Label = “Email”;Expression = {$_.Mail}}
@{Label = “Manager”;Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}}
@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}}
@{Label = “Last LogOn Date”;Expression = {$_.lastlogondate}}

You can combine the Labels above in a single command with a comma in the select-object section, for example to extract all logon names and whether or not the account is disabled…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}},@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

I had some trouble with the LastLogon Label, so have included the working example used to obtain this information below.

get-aduser -filter * -properties * | select-object @{Label = “LogonName”;Expression = {$_.saMAccountName}},@{Label = “LastLogonDate”;Expression = {$_.LastLogonDate}}| Export-Csv D:\ADUsers\ADUsers.LastLogon.csv



What groups am I a member of?

Need to know what groups you’re a member of in Active Directory, but don’t have access to AD Users and Groups management snap-in?  Try this command.  It may help to run cmd.exe as Administrator if that privilege is available to you, but may not be necessary.

gpresult /r

The output at the bottom will be something like this, along with any additional Global group names you’re a member of.


An alternative is whoami /groups which provides an output similar to this…


Note: whoami also works on Linux/UNIX systems.



Inject Administrators/Full Control permissions into inaccessible folders.

Note:  This can also be used to inject Everyone/Full Control, or a specific user, using the username or SID.  The Administrators Group SID is always S-1-5-32-544.  Other well-known SIDs are listed here.

Download the command line version of SetACL.exe from here.  Like all the best things in life, it’s free.

Open a command prompt as Adminstrator (right click cmd.exe, run as admin)

setacl -on “C:\Private No Entry” -ot file -actn ace -ace “n:Administrators;p:full” -rec cont_obj -ignoreerr

The “Private No Entry” folder should now have Administrators, Full Control Permissions.  If not, don’t fret, read on…

The following command gives Administrators the “dream ticket” to accessing all data by setting ownership to Administrators on all folders and files and forcing subdirectories to re-inherit inheritable Administrators:Full Control permissions from the parent.

setacl -on “C:\Private No Entry” -ot file -actn setprot -op “dacl:np;sacl:nc” -rec cont_obj -actn setowner -ownr “n:S-1-5-32-544”

If you still receive “Operating System Message:Access Denied” or similar, then you’ll need to take a robocopy of the “inaccessible” data using the /B switch to exploit OS Backup Right, leaving permissions behind using /COPY:DAT (instead of /COPY:DATSOU or /COPYALL) then repeat the process above on the copied data instead.

robocopy “C:\Private No Entry” “T:\Cracked Data” /B /COPY:DAT /E /NP /R:1 /W:1

Now view the Inherited permissions on the copied data…  You’ll see it has a whole bunch of new, open permissions that it’s got from the parent folder T:.

cacls “T:\Cracked Data”

The cracked data could be robocopied back over the original inaccessible source data using /MIR /COPYALL /SEC /SECFIX switches if required.  If it doesn’t allow it, then note that I have successfully robocopied an empty folder over the top of an inaccessible folder before using just /MIR  (in order to delete it), then robocopied the cracked data back into place, e.g.

robocopy “T:\Empty Folder” “C:\Private No Entry” /MIR /B

robocopy “T:\Cracked Data” “C:\Private No Entry” /MIR /SEC /B

Finally, if you want to re-harden the folder whilst retaining the access you’ve granted Administrators, then use the following commands…

Presently, access has been attained via inherited permissions so before removing inheritance, first inject a non-inherited ACE that allows administrators access, i.e.

setacl -on “C:\Private No Entry” -ot file -actn ace -ace “n:S-1-5-32-544;p:full” -rec cont_obj

Verify the Administrators:Full Control permissions are present on the folder

cacls “C:\Private No Entry”

Finally it is safe to remove inheritance without losing access (strictly speaking, you are “protecting the child object from inherited permissions on the parent object”)

setacl -on u567149 -ot file -actn setprot -op “dacl:p_nc;sacl:p_nc”

This sequence of commands can be used to copy users home directories that are typically hardened to only permit the user themselves access to the data contained within.  If you are using it to migrate home dierctories, there is a loop to re-apply user-specific permissions to each homedirectory afterwards here


Deleting Windows data where the path length exceeds 260 characters

After migrating Windows data, it can be a royal pain cleaning up the source data using del *.* /s /q /f, especially when the path length exceeds 260 (or thereabouts) characters.  You can manually shorten the folder names and keep trying, but this may be time consuming, tiring and ultimately futile.

The simplest way I’ve found to reliably delete data, irrespective of path length, is to use robocopy.

  1. cd into the directory that you want to empty
  2. create a new empty subdirectory called empty
  3. rename all other adjacent folders 1, 2, 3, 4 etc if possible
  4. robocopy empty 1 /mir /r:1 /w:1
  5. repeat for each adjacent folder, 2, 3, 4 etc.



Robocopy leaves some NTFS permissions behind?

And so does emcopy and icacls /save & /restore doesn’t work either…

Googling doesn’t help – every solution that promises to work, doesn’t.



Sorry for shouting, but I’m really rather excited to have cracked this major show stopper for my clients data migration.  The /B switch uses the Backup right to perform the copy.  That’s presumably running with system level privs, rather than my meager admin account in cmd run as administrator.  Magic.  Data integrity restored!  Professional reputation saved!

Note: Use /TIMFIX with /B to correct non-copying of datestamps on files, resulting in 02/01/1980 datestamps on all files copied with /B Backup rights.


Find ACL’s that don’t match parent with AccessEnum

When data begins its life, the permissions are invariably set at one of the top two or three levels, Some examples of some types of data you wouldn’t want everyone to get their hands on in a given company might be…

X:\Payroll\Cost Centre 1\Salaries

X:\Human_Resources\Cost Centre 2\Disciplinaries

X:\Pensions\Cost Centre 3\Compulsory Redundancy Quotes

All users would have to have access to the top level share, then all users might still have access to the Human Resources, Payroll and Pensions folders too.   There may or may not be a cost centre folder to aid the business in understanding what parts of what department are consuming the most storage space (for internal billing), then there will be the “parent” folders (top level in terms of the point where specific permissions are set) whereby only certain individuals in a global group will have access.  ABE (Access Based Enumeration) might be enabled also, preventing users who don’t have access to certain folders from even seeing that the folder exists in the first place.

For the purposes of data migration and reporting, the IT department should have some kind of data admins global group that has full control from the top, all the way down.  Backup admins will also need modify access in order to perform file recoveries back to their original locations.

Over time, chunks of data get moved about as a result of departmental “tidy ups” and restructuring.  The effect this has on permissions is that when data is “copied” it inherits the security of the parent folders on the destination side, however when it is “moved” it takes its permissions with it – sort of.  I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.

I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.  And therein lies a problem for subsequent data migrations.

The effect this has, is to cause the folders on the migrated side (where a copy operation has been used) to subsequently re-inherit their permissions from their new parent.  Only non inherited permissions and inheritance attributes are ever copied, so these permissions that were once inherited but can no longer be inherited are likened to “ghosts” and are lost during any kind of copy operation (data migration operation using emcopy/robocopy or some other method).

I’d like to use “orphans” to describe these children with no matching parent permissions but the term is already taken to describe SIDs in ACEs that no longer resolve to a group in AD (occurs when a folder is secured then the group subsequently deleted), so I’ve come up with “ghosts” to describe them.  I can’t use “zombies” as that is taken to refer to a child process on a UNIX system that has completed execution but remains in the process table so until its parent process can read its exit status.  “Ghosts” is quite apt as they are not tangible and vanish when you migrate the data.

Technically, the copy operation is correcting anomalies but in the real world, that means loss of access, or worse, open access, depending on the nature of the change of permissions on the new parent.

The best way to deal with these ghosts is at source, but you need to know about them first, in order to deal with them.  Sysinternals accessenum GUI tool is a neat way to scan filesystems looking for children where the security differs from that of its parent.  Be sure to set the options accordingly.