Search Result for gnuwin32 — 4

Jan 11

Export all users in ActiveDirectory

If you’re tasked with generating a list / creating a spreadsheet of all user accounts in AD but are worried you might miss out an OU when manually going through and exporting the list using the Active Directory Users and Computers MMC Snap-in, then use Powershell to generate a list instead, safe in the knowledge it’ll find everything.

If you’re really keen you can subsequently use GNUWin32 to give you neat command line tools usually only available to a bash command prompt on a Linux/UNIX OS to chop columns out of the exported csv file using cut, awk, sort and uniq.  Or just use Excel to achieve it.  More on GNUWin32 here.

Open a Powershell and type the following to export all users in the directory to a csv file…

Import-module activedirectory

get-aduser -filter * | Export-Csv c:\myusers.csv

Since the OU Path’s are themselves comma separated, it throws the keys in the csv out of alignment, making it challenging to extract the columns to the right of it that contains the samAccountName  “Logon Name”.  To get over this hurdle, go back to PowerShell and be more specific about the exact key (or Label) you want, e.g. if you just want a list of Logon Names for all users in AD, then this command works…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

Some other useful Labels you may want to use are shown below for your convenience (including a neat If statement for extracting Disabled Accounts).

@{Label = “First Name”;Expression = {$_.GivenName}}
@{Label = “Last Name”;Expression = {$_.Surname}}
@{Label = “Display Name”;Expression = {$_.DisplayName}}
@{Label = “Logon Name”;Expression = {$_.sAMAccountName}}
@{Label = “Full address”;Expression = {$_.StreetAddress}}
@{Label = “City”;Expression = {$_.City}}
@{Label = “State”;Expression = {$}}
@{Label = “Post Code”;Expression = {$_.PostalCode}}
@{Label = “Country/Region”;Expression = {if (($_.Country -eq ‘GB’) ) {‘United Kingdom’} Else {”}}}
@{Label = “Job Title”;Expression = {$_.Title}}
@{Label = “Company”;Expression = {$_.Company}}
@{Label = “Description”;Expression = {$_.Description}}
@{Label = “Department”;Expression = {$_.Department}}
@{Label = “Office”;Expression = {$_.OfficeName}}
@{Label = “Phone”;Expression = {$_.telephoneNumber}}
@{Label = “Email”;Expression = {$_.Mail}}
@{Label = “Manager”;Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}}
@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}}
@{Label = “Last LogOn Date”;Expression = {$_.lastlogondate}}

You can combine the Labels above in a single command with a comma in the select-object section, for example to extract all logon names and whether or not the account is disabled…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}},@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

I had some trouble with the LastLogon Label, so have included the working example used to obtain this information below.

get-aduser -filter * -properties * | select-object @{Label = “LogonName”;Expression = {$_.saMAccountName}},@{Label = “LastLogonDate”;Expression = {$_.LastLogonDate}}| Export-Csv D:\ADUsers\ADUsers.LastLogon.csv


Jun 15

Finding SIDs that haven’t changed after SetACL

This post follows on from a previous post sidmaps and setacl describing how you can use SetACL with a SID map generated using dsquery to translate sidHistory attributes to primarySIDs for migrated groups used in the permissions on your data.  It is used in the final stages of a domain migration so cut the reliance on the continued existence of the old domain controllers in order to verify that the sidHistory attributes of migrated groups in the new domain still correspond with a valid object in the old domain (often referred to as ‘lookback’ or ‘reachback’).

Before an old domain can be decommissioned, the SIDs securing the NTFS folders need to be switched to use only the primary SIDs of the migrated groups.

Once a first pass of SetACL with a sidmap has been run, you’ll need to check that there are no legacy SIDs still being used to secure any folders.

To achieve this, I re-run an export of the security using a command similar to

icacls E:\rootfolder /save H:\exports\E_export.txt 

then split the potentially large log file up into 10Mb pieces using a free filesplitter such as the one available here

I open each file and re-save in ANSI encoded format using Notepad++ available here Notepad++ This is necessary for the subsequent pattern matching steps to work correctly.

I make sure I have GNUWin32 utilities installed available here GNUWin32 so that I can use powerful command line utilities such as cut, sort and uniq ordinarily only available in Linux/UNIX, in Windows.

The following command is then used to strip the old sids from the files and create a new unique list of old SIDs

cut -d; -f6 file_01(10).txt | cut -d) -f1 | findstr S-1-5-1-123456789 | sort | uniq >> sidlist.txt      

(can use | wc -l to just count lines)

I repeat the command on the first file of the set of ten split files i.e file_01(10).txt, for the following fields (we’ve already done field 6 i.e. -f6)

11, 16, 21, 26, 31, 36, 41, 46, 51, 56 and so on (increments of 5) until I get nothing more out for file_01(10).txt before moving on and repeating the process for file 2 i.e file_02(10).txt and so on until I’ve exhausted all 10 files.  This doesn’t take as long as it sounds.  It’s even quicker if you put it in a loop like this…

REM Requires GNUWin32 installed as a prereq.
REM Use fsplit.exe to break large icacls exports into 10MB pieces.
REM Usage Example: extractsids.bat exportfile_01(16).txt 1552345678
SET /a i=6
IF %i%==56 GOTO END
echo Extracting sids matching %2 from field %i% in %1…
cut -d; -f%i% %1 | cut -d) -f1 | findstr %2 | sort | uniq >> %1_sidsextracted.txt
SET /a i=%i%+5
sort %1_sidsextracted.txt | uniq > %1_%2_extractedsids.txt
if exist %1_sidsextracted.txt | uniq > %1_%2_extractedsids.txt
if exist %1_sidsextracted.txt (
del %1_sidsextracted.txt
) else (
echo “Somethings gone wrong.
echo Done! Extracted sids written to %1_extractedsids.txt

I then re-sort and unique the list again to end up with my final list of SIDs that were not changed as part of the initial Re-ACL process. (Step is included in batch file above).

sort sidlist.txt | uniq > unchanged_sids.txt

If I need to identify what users or groups these unique SIDs correspond to in AD, then I can use the free tool available here sidtoname.exe in conjunction with the following batch file that I’ll call get_names.bat

@echo off
REM Usage:  From a command line…
REM groupnames.bat unchanged_sids.txt > names.txt
REM Dependencies: sidtoname.exe
for /f %%a in (%1) do (
sidtoname.exe %%a

And there you have it.  A list of groups whose SID’s were not changed after an initial pass of SetACL with a sidmap.  You now need to take the list of sids or the list of groups and determine the sidHistory and primarySIDs of them, then append them to your original sidmap before re-running another pass of SetACL.  Note that the groups listed in names.txt could be user names as well as group names (or aliases too).  They will be conveniently prepended with a label of User, Group or Alias accordingly.




May 18

Using Linux commands on WIndows

Wouldn’t it be nice if you could pipe the output from windows commands into non-windows commands like grep, cut, awk, sort etc that are available to you on alternative unix-based operating systems?


Download and install GNUWin32 from here and the CoreUtils package here and Grep here that should do it.  There are more packages available though here

Once installed, add the path to the bin directory to your Windows System Environment Variable Path

Environment_variables Path

A few useful commands will now be available on the command line.  My favourite is comm which compares files and can be quite flexible with the output with the -1 -2 or -3 switches to suppress lines that appear in file1, file2 or both files respectively.   You can also combine them e.g. -12 -23, 13 to affect the output, so that only the desired output is achieved.  This takes a bit of playing around with, but is very powerful and very simple.  So much so, that it is my number 1 go to tool for file comparison.  Examples shown the in the screenshots below.

comm-helpcomm-3 comm_windows

Note:  Some Windows tools such as icacls export text to a format other than ANSI.  When viewed using Notepad or Notepad++, all appears fine, but if you cat them , you’ll see there are effectively spaces between each character, meaning grep won’t work.  Such text files will need to be saved in ANSI format first.  You can do this using Notepad++.  After selecting Encode in ANSI, save it, then retry grep for a more successful pattern match!



May 22

Data Migration using robocopy

As a compliment to my recent post “Data Migration using emcopy”

I thought it only fair to follow up with an equivalent post for good ol’ robocopy.  This has mainly come about having discovered an annoying bug in emcopy whereby it doesn’t ignore the directories specified by more than one /xd exclusion – it always excludes the last one specified, but none of the others?!

Robocopy on the other hand, does allow exclusion of more than one directory, each one specified using the /xd switch and can be a full path (to exclude very specific directories) or just one word (to exclude any directories with that name anywhere in the directory tree).

The switch worth mentioning the most though, is the /FFT switch.  Update:  AND THE /B SWITCH (use backup rights). Uh, and come to think of it, the /XO switch too (especially if you’re running a repeated backup of data to a USB HDD).

When migrating data into a Celerra/VNX/NetApp CIFS Server, the act of copying data from a NTFS volume on a Windows Server to a Linux based Filesystem on a NAS is enough to throw the timestamps on the files out just enough to make robocopy think that the source file is newer, even when it’s not.  This means that subsequent copies of the changed files take just as long as the initial copy.

By appending /FFT to the long list of switches used in your robocopy command, it allows for a discrepancy of up to 2 seconds – enough to provide a convenient workaround to this problem.

In practice, this brought down troublesome 36 hour copy operations requiring a weekend cut-over to be arranged, down to just over 1 hour – cue cliche  – saving time and money.

An example command is given below…

robocopy d:\source e:\dest *.* /xd “System Volume Information” d:\Migration homedirs profiles wtsprofiles /e /np /fft /xo /r:1 /w:1

There are many more switches available in robocopy, including the ability to use multiple threads in newer versions (highly recommended).  Just type robocopy /? from the Windows command line to see the other options.

In practice I found emcopy to be inconsistent with copying ACE’s across to large filesystems, completely skipping some folders when creating an empty folder structure using the /xf * /create method.  This means that file data (and the missing subfolders) subsequently copied into place with /nosec would be forced to inherit the parent permissions.  Most likely not a problem, but if the data has lots of bespoke permissions then it becomes a huge problem as data is generally more “open” at the parent levels.

To re-sync permissions, the following command was useful.

for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy /E /Copy:S /IS /IT q:\%%f g:\root\%%f

This has since been updated.  To re-sync folder perms between source and dest trees, this works…

robocopy s:\ d:\ /lev:3 /MIR /SEC /SECFIX /V /B /xo /xn /xc /r:1 /w:1

Due to folders being missed, I never deal with a file system using a single command.  I always break it up and handle each top level folder as an individual job by placing the command in a for loop as shown above.

Alternatively use the following to replicate changed files and their security, and also set the security on unchanged files.  The /V shows the unchanged files being fixed.

for /f “delims=” %%f IN (‘dir g:\root\ /ad /b’) DO robocopy q:\ g:\root\ /MIR /SEC /SECFIX /V /B /r:1 /w:1

I found myself fighting for a day or two with an apparent intermittent problem copying NTFS security when robocopying data from NAS to NAS.  Despite using all the methods described above, sometimes the NTFS permissions just weren’t being copied across.  I have since discovered that using the /B switch with every other method mentioned already, fixes this annoying problem and the ACE’s come across perfectly.

I’ve since encountered odd behaviour using for loops that has resulted in a mistrust of them.  So, I code each top level folder as an individual line in a batch file.  The problems were encountered where there were spaces in the folder names irrespective of using “delims=”, robocopy didn’t always get it right thereafter.

Robocopy doesn’t let you copy certain folders.  It lets you exclude certain folders but that’s not much use if you only want to copy folders starting with u6* for example.  In this situation, e.g. migrating all users whose usernames begin with u6 to a separate filesystem, you need to use the for loop.

for /f “delims=” %f IN (‘dir s:\root\u5* /ad /b’) DO robocopy s:\root\%f t:\root\%f /COPYALL /R:1 /W:1 /ZB /NP /L /FFT /LOG+:D:\cyberfellaltd\u6mig.log

Update: 28/2/2017  Real World Example: Migration of a subset of users to new filesystem.  Two passes, two different approaches.  One does initial copy of just usernames beginning with u5, the second generates a list of missing users after the first pass and does a second pass targeting the missing users.  Note that this is a hybrid Bash/Batch script and requires the installation of GNUWin32 on Windows in order to work.  This is covered here.

for /f “delims=” %%f IN (‘dir s:\root\u5* /ad /b/ o’) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does first pass on all u5 users)

dir /ad /b /o s:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\src.txt | wc –l      (counts 2113 and writes list of all u5 users to src.txt)

dir /ad/b /o t:\root\u5* | tr ‘[:upper:]’ ‘[:lower:]’ | tee t:\dest.txt | wc –l    (counts 2113 and writes list of all u5 users to dest.txt)

comm -23 t:\src.txt t:\dest.txt | tee t:\missing.txt | wc –l  (counts 0 differences and writes list of any missing u5 users to missing.txt)

for /f “delims=” %%f IN (cat t:\missing.txt) DO robocopy s:\root\%%f t:\root\%%f /COPYALL /R:1 /W:1 /ZB /NP /FFT /LOG+d:\mattb\u5mig.log (does 2nd pass on any missing users only)

If you have folders containing ampersand characters in the name, your copies can fail.  This post here covers a way to deal with it using variables.

1 comment