Category: Data Migration

Apr 06

Robocopy folders with ampersands in the name

Don’t use & in file and folder names.

With that little pearl of wisdom out of the way, what about when your users have used ampersand characters in their folder names and you’re trying to robocopy the data to it’s new home, only to have the copy fail?

Try this…

SET “source=dogs & cats”

SET “destination=dogs & cats”

or if you can get away with it without breaking links…

SET “destination=dogs and cats”

robocopy.exe “%source%” “%destination%” /MIR

For more robocopy wisdom, check this post here

In real-world practice, I have found that robocopy is woefully unreliable when it comes to copying permissions (using the /e /sec /xf * switches).  I recommend using emcopy to copy folder structures and their NTFS permissions.  Similar to the robocopy commands above, these emcopy commands worked almost* perfectly for me

SET “source=dogs & cats”

SET “destination=dogs and cats”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

*Note how I’ve changed the destination folder to not include the ampersand character.  In practice, permissions were not copied to folders with ampersands in the name using robocopy or emcopy – in fact robocopy didn’t copy permissions at all!

If you’re copying a subset of data from a bigger source set of data, then never use /MIR or you will run a high risk of loosing data.   Oh yes you will.  Use the above emcopy commands one folder at a time to get your destination folder structure in place, before finally syncing the subfolder you want into the new destination.  This saves a potentially troublesome cleanup exercise later, deleting superfluous data, e.g.


SET “source=dogs & cats”

SET “destination=dogs and cats”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

Followed by…

SET “source=dogs & cats\spaniels”

SET “destination=dogs and cats\spaniels”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

Followed by…

SET “source=dogs & cats\spaniels\springer”

SET “destination=dogs and cats\spaniels\springer”

emcopy “%source%” “%destination%” /secfix /xf * /lev:1

and finally sync your file data into the new secured folder structure…

SET “source=dogs & cats\spaniels\springer”

SET “destination=dogs and cats\spaniels\springer”

Synchronise all file data using your preferred robocopy or emcopy command here.

Jan 11

Export all users in ActiveDirectory

If you’re tasked with generating a list / creating a spreadsheet of all user accounts in AD but are worried you might miss out an OU when manually going through and exporting the list using the Active Directory Users and Computers MMC Snap-in, then use Powershell to generate a list instead, safe in the knowledge it’ll find everything.

If you’re really keen you can subsequently use GNUWin32 to give you neat command line tools usually only available to a bash command prompt on a Linux/UNIX OS to chop columns out of the exported csv file using cut, awk, sort and uniq.  Or just use Excel to achieve it.  More on GNUWin32 here.

Open a Powershell and type the following to export all users in the directory to a csv file…

Import-module activedirectory

get-aduser -filter * | Export-Csv c:\myusers.csv

Since the OU Path’s are themselves comma separated, it throws the keys in the csv out of alignment, making it challenging to extract the columns to the right of it that contains the samAccountName  “Logon Name”.  To get over this hurdle, go back to PowerShell and be more specific about the exact key (or Label) you want, e.g. if you just want a list of Logon Names for all users in AD, then this command works…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

Some other useful Labels you may want to use are shown below for your convenience (including a neat If statement for extracting Disabled Accounts).

@{Label = “First Name”;Expression = {$_.GivenName}}
@{Label = “Last Name”;Expression = {$_.Surname}}
@{Label = “Display Name”;Expression = {$_.DisplayName}}
@{Label = “Logon Name”;Expression = {$_.sAMAccountName}}
@{Label = “Full address”;Expression = {$_.StreetAddress}}
@{Label = “City”;Expression = {$_.City}}
@{Label = “State”;Expression = {$}}
@{Label = “Post Code”;Expression = {$_.PostalCode}}
@{Label = “Country/Region”;Expression = {if (($_.Country -eq ‘GB’) ) {‘United Kingdom’} Else {”}}}
@{Label = “Job Title”;Expression = {$_.Title}}
@{Label = “Company”;Expression = {$_.Company}}
@{Label = “Description”;Expression = {$_.Description}}
@{Label = “Department”;Expression = {$_.Department}}
@{Label = “Office”;Expression = {$_.OfficeName}}
@{Label = “Phone”;Expression = {$_.telephoneNumber}}
@{Label = “Email”;Expression = {$_.Mail}}
@{Label = “Manager”;Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}}
@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}}
@{Label = “Last LogOn Date”;Expression = {$_.lastlogondate}}

You can combine the Labels above in a single command with a comma in the select-object section, for example to extract all logon names and whether or not the account is disabled…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}},@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

I had some trouble with the LastLogon Label, so have included the working example used to obtain this information below.

get-aduser -filter * -properties * | select-object @{Label = “LogonName”;Expression = {$_.saMAccountName}},@{Label = “LastLogonDate”;Expression = {$_.LastLogonDate}}| Export-Csv D:\ADUsers\ADUsers.LastLogon.csv


Nov 25

How to cable up VNX SP Ports (Dual Fabric topology)

So your VNX has two SP’s and you have two fabric switches.  You already know you have to connect each SP to each fabric for resilience, but you’re still a bit confused.  Fear not.  Use this as a guide.  It can be used no matter how many front-end port modules and SFP’s you have so that you get it right first time for all your designated Storage Ports, Mirrorview Ports, Sancopy Ports required for your project.  The FC Switch Ports you choose are not set in stone, but keep it the same on each side at least.

 What’s important is that the correct SP port, goes to the correct switch.

When you know you’ve got it right, you can get verify the WWN in Unisphere corresponds with the WWN logged in on the FLOGI database on the switch to check before you create the requisite fcaliases etc.


It’s much easier to build it right first time than sort it out afterwards.


Sep 07

Inject Everyone/Full Control ACE into NTFS Folder

Download SetACL.exe from here

Open a command line as Adminstrator (right click cmd.exe, run as admin)

setacl -on “C:\Private No Entry” -ot file -actn ace -ace “n:Everyone;p:full” -rec cont_obj -ignoreerr

The “Private No Entry” folder should now have Everyone, Full Control Permissions.


Dec 04

Deleting Windows data where the path length exceeds 260 characters

After migrating Windows data, it can be a royal pain cleaning up the source data using del *.* /s /q /f, especially when the path length exceeds 260 (or thereabouts) characters.  You can manually shorten the folder names and keep trying, but this may be time consuming, tiring and ultimately futile.

The simplest way I’ve found to reliably delete data, irrespective of path length, is to use robocopy.

  1. cd into the directory that you want to empty
  2. create a new empty subdirectory called empty
  3. rename all other adjacent folders 1, 2, 3, 4 etc if possible
  4. robocopy empty 1 /mir /r:1 /w:1
  5. repeat for each adjacent folder, 2, 3, 4 etc.


Sep 10

Robocopy leaves some NTFS permissions behind?

And so does emcopy and icacls /save & /restore doesn’t work either…

Googling doesn’t help – every solution that promises to work, doesn’t.



Sorry for shouting, but I’m really rather excited to have cracked this major show stopper for my clients data migration.  The /B switch uses the Backup right to perform the copy.  That’s presumably running with system level privs, rather than my meager admin account in cmd run as administrator.  Magic.  Data integrity restored!  Professional reputation saved!

Aug 19

Find ACL’s that don’t match parent with AccessEnum

When data begins its life, the permissions are invariably set at one of the top two or three levels, Some examples of some types of data you wouldn’t want everyone to get their hands on in a given company might be…

X:\Payroll\Cost Centre 1\Salaries

X:\Human_Resources\Cost Centre 2\Disciplinaries

X:\Pensions\Cost Centre 3\Compulsory Redundancy Quotes

All users would have to have access to the top level share, then all users might still have access to the Human Resources, Payroll and Pensions folders too.   There may or may not be a cost centre folder to aid the business in understanding what parts of what department are consuming the most storage space (for internal billing), then there will be the “parent” folders (top level in terms of the point where specific permissions are set) whereby only certain individuals in a global group will have access.  ABE (Access Based Enumeration) might be enabled also, preventing users who don’t have access to certain folders from even seeing that the folder exists in the first place.

For the purposes of data migration and reporting, the IT department should have some kind of data admins global group that has full control from the top, all the way down.  Backup admins will also need modify access in order to perform file recoveries back to their original locations.

Over time, chunks of data get moved about as a result of departmental “tidy ups” and restructuring.  The effect this has on permissions is that when data is “copied” it inherits the security of the parent folders on the destination side, however when it is “moved” it takes its permissions with it – sort of.  I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.

I say “sort-of” because although it takes the security ACE’s with it, it also takes the attribute that says those ACE’s were originally inherited from the parent.  And therein lies a problem for subsequent data migrations.

The effect this has, is to cause the folders on the migrated side (where a copy operation has been used) to subsequently re-inherit their permissions from their new parent.  Only non inherited permissions and inheritance attributes are ever copied, so these permissions that were once inherited but can no longer be inherited are likened to “ghosts” and are lost during any kind of copy operation (data migration operation using emcopy/robocopy or some other method).

I’d like to use “orphans” to describe these children with no matching parent permissions but the term is already taken to describe SIDs in ACEs that no longer resolve to a group in AD (occurs when a folder is secured then the group subsequently deleted), so I’ve come up with “ghosts” to describe them.  I can’t use “zombies” as that is taken to refer to a child process on a UNIX system that has completed execution but remains in the process table so until its parent process can read its exit status.  “Ghosts” is quite apt as they are not tangible and vanish when you migrate the data.

Technically, the copy operation is correcting anomalies but in the real world, that means loss of access, or worse, open access, depending on the nature of the change of permissions on the new parent.

The best way to deal with these ghosts is at source, but you need to know about them first, in order to deal with them.  Sysinternals accessenum GUI tool is a neat way to scan filesystems looking for children where the security differs from that of its parent.  Be sure to set the options accordingly.


Aug 19

Powershell and the NTFSSecurity Module

The powershell NTFSSecurity module provides cmdlets to export and import security.  Unlike icacls which sticks to using sddl format (for a 10 fold increase in speed exporting security for large filesystems), powershell will resolve the SIDs in sddl format into human friendly names by chatting to the DC as it goes.  Some useful commands are noted below.

Download from here
NTFSSecurity Module for Powershell

Just create the folder “NTFSSecurity” in the folder set according to the environment variable %PSModulePath%

The module should now be listed in “Get-Module -ListAvailable” and can be imported using “Import-Module NTFSSecurity“.

For example, all the files in the zip file have to be in “%SystemRoot%\system32\WindowsPowerShell\v1.0\Modules\NTFSSecurity\“.
If you did this then the module should be listed in “Get-Module -ListAvailable” and can be imported using “Import-Module NTFSSecurity“.

Note that running Windows Powershell Modules (look for the powershell icon with the admin shield in the corner in your start menu), will automatically load the module upon CLI startup.

#to backup permissions just pipe what Get-Ace returns to Export-Csv
get-childitem -Recurse | Get-Ace -ExcludeInherited | Export-Csv permissions.csv

#to restore the permissions pipe the imported data to Add-Ace
#As the imported data also contains the path you do not need to specify the item
Restore: Import-Csv .\permissions.csv | Add-Ace

get-childitem -Recurse | get-inheritance | export-CSV C:\inheritanceon.csv -NoTypeInformation

get-childitem -recurse -exclude *.* | get-ace | export C:\migrationscripts\incinherited.csv -notypeinformation

Jul 23

Share all subfolders as individual hidden shares

Continuing on from my previous post about setting permissions on all migrated users home directories here…

Re-permissioning Users Home Directories

Re-permission each users subdirectory so only their user account has access (note that the homedir name and username must match),

for /f %%f IN (‘dir /ad /b E:\MigratedData\homedirs\’) DO cacls E:\MigratedData\homedirs\%%f /e /p %%f:F

It is also possible to share each migrated home directory (or any other set of subfolders) as its own hidden share, without the repetitious click, click,click of the share wizard and copious amounts of your time (that you’ll never get back).  Let the command line take the strain!  And the best bit?  You don’t even need PowerShell to do it!


Create a hidden share for each users home directory (Note: home directory must have appropriate NTFS security in place)

for /f %%f IN (‘dir /ad /b E:\MigratedData\homedirs\’) DO net share %%f$=E:\MigratedData\homedirs\%%f /GRANT:Everyone,FULL

You could tighten up the share security only allowing the user themselves to have full control (in terms of share permissions – the underlying NTFS perms to that effect should already be in place), by changing /GRANT:Everyone,FULL to /GRANT:%%f,FULL

Note that double %% is necessary for using these commands in a batch file,  If you want to run them straight on the command line, you’ll need to drop one of the % e.g. for /f %f IN…



Jun 15

Finding SIDs that haven’t changed after SetACL

This post follows on from a previous post sidmaps and setacl describing how you can use SetACL with a SID map generated using dsquery to translate sidHistory attributes to primarySIDs for migrated groups used in the permissions on your data.  It is used in the final stages of a domain migration so cut the reliance on the continued existence of the old domain controllers in order to verify that the sidHistory attributes of migrated groups in the new domain still correspond with a valid object in the old domain (often referred to as ‘lookback’ or ‘reachback’).

Before an old domain can be decommissioned, the SIDs securing the NTFS folders need to be switched to use only the primary SIDs of the migrated groups.

Once a first pass of SetACL with a sidmap has been run, you’ll need to check that there are no legacy SIDs still being used to secure any folders.

To achieve this, I re-run an export of the security using a command similar to

icacls E:\rootfolder /save H:\exports\E_export.txt 

then split the potentially large log file up into 10Mb pieces using a free filesplitter such as the one available here

I open each file and re-save in ANSI encoded format using Notepad++ available here Notepad++ This is necessary for the subsequent pattern matching steps to work correctly.

I make sure I have GNUWin32 utilities installed available here GNUWin32 so that I can use powerful command line utilities such as cut, sort and uniq ordinarily only available in Linux/UNIX, in Windows.

The following command is then used to strip the old sids from the files and create a new unique list of old SIDs

cut -d; -f6 file_01(10).txt | cut -d) -f1 | findstr S-1-5-1-123456789 | sort | uniq >> sidlist.txt      

(can use | wc -l to just count lines)

I repeat the command on the first file of the set of ten split files i.e file_01(10).txt, for the following fields (we’ve already done field 6 i.e. -f6)

11, 16, 21, 26, 31, 36, 41, 46, 51, 56 and so on (increments of 5) until I get nothing more out for file_01(10).txt before moving on and repeating the process for file 2 i.e file_02(10).txt and so on until I’ve exhausted all 10 files.  This doesn’t take as long as it sounds.  It’s even quicker if you put it in a loop like this…

REM Requires GNUWin32 installed as a prereq.
REM Use fsplit.exe to break large icacls exports into 10MB pieces.
REM Usage Example: extractsids.bat exportfile_01(16).txt 1552345678
SET /a i=6
IF %i%==56 GOTO END
echo Extracting sids matching %2 from field %i% in %1…
cut -d; -f%i% %1 | cut -d) -f1 | findstr %2 | sort | uniq >> %1_sidsextracted.txt
SET /a i=%i%+5
sort %1_sidsextracted.txt | uniq > %1_%2_extractedsids.txt
if exist %1_sidsextracted.txt | uniq > %1_%2_extractedsids.txt
if exist %1_sidsextracted.txt (
del %1_sidsextracted.txt
) else (
echo “Somethings gone wrong.
echo Done! Extracted sids written to %1_extractedsids.txt

I then re-sort and unique the list again to end up with my final list of SIDs that were not changed as part of the initial Re-ACL process. (Step is included in batch file above).

sort sidlist.txt | uniq > unchanged_sids.txt

If I need to identify what users or groups these unique SIDs correspond to in AD, then I can use the free tool available here sidtoname.exe in conjunction with the following batch file that I’ll call get_names.bat

@echo off
REM Usage:  From a command line…
REM groupnames.bat unchanged_sids.txt > names.txt
REM Dependencies: sidtoname.exe
for /f %%a in (%1) do (
sidtoname.exe %%a

And there you have it.  A list of groups whose SID’s were not changed after an initial pass of SetACL with a sidmap.  You now need to take the list of sids or the list of groups and determine the sidHistory and primarySIDs of them, then append them to your original sidmap before re-running another pass of SetACL.  Note that the groups listed in names.txt could be user names as well as group names (or aliases too).  They will be conveniently prepended with a label of User, Group or Alias accordingly.