Category: Cool Bash Tricks

Jan 11

Export all users in ActiveDirectory

If you’re tasked with generating a list / creating a spreadsheet of all user accounts in AD but are worried you might miss out an OU when manually going through and exporting the list using the Active Directory Users and Computers MMC Snap-in, then use Powershell to generate a list instead, safe in the knowledge it’ll find everything.

If you’re really keen you can subsequently use GNUWin32 to give you neat command line tools usually only available to a bash command prompt on a Linux/UNIX OS to chop columns out of the exported csv file using cut, awk, sort and uniq.  Or just use Excel to achieve it.  More on GNUWin32 here.

Open a Powershell and type the following to export all users in the directory to a csv file…

Import-module activedirectory

get-aduser -filter * | Export-Csv c:\myusers.csv

Since the OU Path’s are themselves comma separated, it throws the keys in the csv out of alignment, making it challenging to extract the columns to the right of it that contains the samAccountName  “Logon Name”.  To get over this hurdle, go back to PowerShell and be more specific about the exact key (or Label) you want, e.g. if you just want a list of Logon Names for all users in AD, then this command works…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

Some other useful Labels you may want to use are shown below for your convenience (including a neat If statement for extracting Disabled Accounts).

@{Label = “First Name”;Expression = {$_.GivenName}}
@{Label = “Last Name”;Expression = {$_.Surname}}
@{Label = “Display Name”;Expression = {$_.DisplayName}}
@{Label = “Logon Name”;Expression = {$_.sAMAccountName}}
@{Label = “Full address”;Expression = {$_.StreetAddress}}
@{Label = “City”;Expression = {$_.City}}
@{Label = “State”;Expression = {$_.st}}
@{Label = “Post Code”;Expression = {$_.PostalCode}}
@{Label = “Country/Region”;Expression = {if (($_.Country -eq ‘GB’) ) {‘United Kingdom’} Else {”}}}
@{Label = “Job Title”;Expression = {$_.Title}}
@{Label = “Company”;Expression = {$_.Company}}
@{Label = “Description”;Expression = {$_.Description}}
@{Label = “Department”;Expression = {$_.Department}}
@{Label = “Office”;Expression = {$_.OfficeName}}
@{Label = “Phone”;Expression = {$_.telephoneNumber}}
@{Label = “Email”;Expression = {$_.Mail}}
@{Label = “Manager”;Expression = {%{(Get-AdUser $_.Manager -server $ADServer -Properties DisplayName).DisplayName}}}
@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}}
@{Label = “Last LogOn Date”;Expression = {$_.lastlogondate}}

You can combine the Labels above in a single command with a comma in the select-object section, for example to extract all logon names and whether or not the account is disabled…

get-aduser -filter * | select-object @{Label = “Logon Name”;Expression ={$_.saMAccountName}},@{Label = “Account Status”;Expression = {if (($_.Enabled -eq ‘TRUE’) ) {‘Enabled’} Else {‘Disabled’}}} | Export-Csv D:\ADUsers\ADUsers.LogonNames.csv

I had some trouble with the LastLogon Label, so have included the working example used to obtain this information below.

get-aduser -filter * -properties * | select-object @{Label = “LogonName”;Expression = {$_.saMAccountName}},@{Label = “LastLogonDate”;Expression = {$_.LastLogonDate}}| Export-Csv D:\ADUsers\ADUsers.LastLogon.csv

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
May 18

Using Linux commands on WIndows

Wouldn’t it be nice if you could pipe the output from windows commands into non-windows commands like grep, cut, awk, sort etc that are available to you on alternative unix-based operating systems?

 

Download and install GNUWin32 from here and the CoreUtils package here and Grep here that should do it.  There are more packages available though here

Once installed, add the path to the bin directory to your Windows System Environment Variable Path

Environment_variables Path

A few useful commands will now be available on the command line.  My favourite is comm which compares files and can be quite flexible with the output with the -1 -2 or -3 switches to suppress lines that appear in file1, file2 or both files respectively.   You can also combine them e.g. -12 -23, 13 to affect the output, so that only the desired output is achieved.  This takes a bit of playing around with, but is very powerful and very simple.  So much so, that it is my number 1 go to tool for file comparison.  Examples shown the in the screenshots below.

comm-helpcomm-3 comm_windows

Note:  Some Windows tools such as icacls export text to a format other than ANSI.  When viewed using Notepad or Notepad++, all appears fine, but if you cat them , you’ll see there are effectively spaces between each character, meaning grep won’t work.  Such text files will need to be saved in ANSI format first.  You can do this using Notepad++.  After selecting Encode in ANSI, save it, then retry grep for a more successful pattern match!

ANSI

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Nov 14

nohup and disown your long running jobs

Ever started a job and thought “this is running on a bit longer than I expected”, then wondered whats going to happen when you go home and come back in to work tomorrow morning to find your remote session gone, which leaves you wondering “did that job complete?”.

Mmm, me too, which is where the nohup or disown commands come in.  nohup (no hangup) is a well known job control utility which will prevent a process from reacting to the hangup signal when a shell is disconnected.  Usually you’d preceed your actual command with it, e.g.

nohup rsync -auvc /Source/* /Destination/

but if your command is already running, you’re left wishing you’d nohup‘d it to start with – unless you’re running Solaris or AIX in which case the nohup command has a convenient -p switch to specify the process id (use ps -ef | grep rsync to obtain the PID of that long running data migration process, then nohup -p 9675 (or whatever the PID is of your running job).

If you’re not running Solaris or AIX, then pray you started the command in the bash shell (Linux default shell so more likely than not).  If you did, then you can

CTRL-Z

to pause the current job running in the foreground, then use the

jobs

command to determine its job number (most likely 1 if there’s no other sysadmins running backgrounded jobs), then background the process with

bg 1 

then finally

disown %1

to disconnect the process from the current shell.  Running

jobs 

again will show that your job is no longer in the list, but

ps -ef

will reveal that it is in fact still running.

Your shell can now be closed without the fear of your running job being killed with it.  Yay.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Oct 11

Linux Commands A-Z

I can’t take credit for this one –  that goes to Rahul Nag http://techosolutions.wordpress.com/2013/10/10/a-z-linux-commands/ but it’s just too useful to not have in my notes.  I’ve also added some extras and will continue to do so as I think of them/use them myself.

Command Description
alias Create an alias opposite of unalias
apropos Search manual for keyword
at Schedule a job to run in the future.
awk Find and Replace text within file(s) or show specific columns only

basename Opposite of dirname

break Exit from a loop
builtin Run a shell builtin
bunzip2 Decompress file from bzip2 format
bzip2 Compress file to bzip2 format
cal Display a calendar
case Conditionally perform a command
cat Concatenate files to standard output opposite of tac
cd Change Directory
cfdisk Partition table manipulator for Linux
chgrp Change group ownership
chmod Change access permissions
chown Change file owner and group
chroot Run a command with a different root directory
chvt Change the virtual Terminal
cksum Print CRC checksum and byte counts
clear Clear terminal screen
cmp Compare two files
comm Compare two sorted files line by line
command Run a command – ignoring shell functions
compress Compress file(s) to old Unix compress format
continue Resume the next iteration of a loop
convmv A perl script that converts filenames from one encoding to another
cp Copy one or more files to another location

cpio Copy files to and from archives

cron Daemon to execute scheduled commands at predefined time
crontab Schedule a command to run at a later time
csplit Split a file into context-determined pieces
cut Divide a file into several parts
date Display or change the date & time
dc Desk Calculator
dd Data Dump – Convert and copy a file
declare Declare variables and give them attributes
df Display free disk space
diff Display the differences between two files
diff3 Show differences among three files
dir Briefly list directory contents
dircolors Colour setup for `ls’
dirname Convert a full pathname to just a path
dirs Display list of remembered directories

df  Disk space free stats.

du Estimate file space usage
echo Display message on screen
ed A line-oriented text editor (edlin)
egrep Search file(s) for lines that match an extended expression
eject Eject CD-ROM
enable Enable and disable builtin shell commands
env Disp, set, or remove environment variables
eval Evaluate several commands/arguments
exec Execute a command (used with find)
exit Exit the shell
expand Convert tabs to spaces opposite of unexpand
export Set an environment variable
expr Evaluate expressions
factor Print prime factors
false Do nothing, unsuccessfully opposite of true
fdformat Low-level format a floppy disk
fdisk Partition table manipulator for Linux
fgrep Search file(s) for lines that match a fixed string
file Determine type of file
find Search for files that meet a desired criteria
fmt Reformat paragraph text
fold Wrap text to fit a specified width.
for Expand words, and execute commands
format Format disks or tapes
free Disp, s memory usage
fsck Filesystem consistency check and repair.
fstat List open files
function Define Function Macros
fuser Identify process using file
gawk Find and Replace text within file(s)
getopts Parse positional parameters
grep Search file(s) for lines that match a given pattern
groups Print group names a user is in
gunzip Decompress file(s) from GNU zip format
gzcat Show contents of compressed file(s)
gzip Compress file(s) to GNU zip format
hash Remember the full pathname of a name argument
head Output the first part of file(s)
history Command History
hostname Print or set system name
iconv Converts the encoding of characters from one code page encoding scheme to another.
id Print user and group id’s
if Conditionally perform a command
import Capture an X server screen and save the image to file
info Help info
install Copy files and set attributes
join Join lines on a common field
kill Stop a process from running
less Display output one screen at a time
let Perform arithmetic on shell variables
ln Make links between files
local Create variables
locate Find files
logname Print current login name
logout Exit a login shell
lpc Line printer control program
lpr Off line print
lprint Print a file
lprintd Abort a print job
lprintq List the print queue
lprm Remove jobs from the print queue
ls List information about file(s)
ll #ls -l List information about file(s)
lsof List open files
m4 Macro processor
makewhatis Rebuild whatis database
man Print manual pages
mkdir Create new folder(s)
mkfifo Make FIFOs (named pipes)
mknod Make block or character special files
more Display output one screen at a time
mount Mount a file system
mtools Manipulate MS-DOS files
mv Move or rename files or directories
netconfig Configure your network
nice Set the priority of a command or job
nl Number lines and write files
nohup Run a command immune to hangup
od View binary files
passwd Modify a user password
paste Merge lines of files
pathchk Check file name portability
popd Restore the previous value of the current directory opposite of pushd
pr Convert text files for printing
printcap Printer capability database
printenv Print environment variables
printf Format and print data
ps Process status
pushd Save and then change the current directory
pwd Print Working Directory
quota Display disk usage and limits
quotacheck Scan a file system for disk usage
quotactl Set disk quotas
pax Archive file(s)
ram ram disk device
rcp Copy files between two machines.
read read a line from standard input
readonly Mark variables/functions as readonly
remsync Synchronize remote files via email
return Exit a shell function
rm Remove (delete) files
rmdir Remove folder(s)
rpm RPM Package Manager (was RedHat Package Manager)
rsync Remote file copy (Synchronize file trees)
screen Terminal window manager
sdiff Merge two files interactively
sed Stream Editor used to perform search and replace
select Accept keyboard input
seq Print numeric sequences
set Manipulate shell variables and functions opposite of unset
shift Shift positional parameters
shopt Shell Options
shutdown Shutdown or restart linux
sleep Delay for a specified time
sort Sort text files often used with | uniq but can just sort -u
source Run commands from a file `.’
split Split a file into fixed-size pieces
strings print the strings of printable characters in (binary) files.
su Substitute user identity
sum Print a checksum for a file
symlink Make a new name for a file
sync Synchronize data on disk with memory
tac Print files out in reverse line order opposite of cat
tail Output the last part of files
tar Tape ARchiver
tee Redirect output to multiple files as well as to the screen -better than just using > or >>(tee -a)
test Evaluate a conditional expression
time Measure Program Resource Use
times User and system times

timex preceed a command to show how long it ran for upon completion

timidity Play midi files and set up software synth to play midi files with other commands.
touch Change file timestamps
top List processes running on the system
traceroute Trace Route to Host
trap Run a command when a signal is set(bourne)
tr Translate, squeeze, and/or delete characters change uppercase to lowercase
true Do nothing, successfully opposite of false
tsort Topological sort
tty Print filename of terminal on stdin
type Describe a command
ulimit Limit user resources
umask Users file creation mask
umount Unmount a filesystem
unalias Remove an alias opposite of alias
uname Print system information
unexpand Convert spaces to tabs opposite of expand
uniq Uniquify files (remove all duplicate lines) can also use sort -u to sort | uniq
units Convert units from one scale to another
unset Remove variable or function names opposite of set
unshar Unpack shell archive scripts
until Execute commands (until error)
useradd Create new user account
usermod Modify user account
users List users currently logged in
uuencode Encode a binary file into 7-bit ASCII characters
uudecode Decode a file created by uuencode
v Verbosely list directory contents (`ls -l -b’)
vdir Verbosely list directory contents (`ls -l -b’)

vi  The greatest shell text editor ever

vim The greatest shell text editor ever – improved

watch Execute/display a program periodically
whatis List manual pages by name
wc Print byte, word, and line counts of a file
whereis Report all known instances of a command
which Locate a program file in the user’s path.
while Execute commands
who Print all usernames currently logged in
whoami Print the current user id and name (`id -un’)
xargs Execute utility, passing constructed argument list(s)
yes Print a string until interrupted
zcat Show contents of compressed file(s)
zip Compress and archive file(s) to zip format

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 22

Fun Linux Commands

Although it has a reputation for having a steep initial learning curve, in fact Linux / UNIX is great fun to learn and use, and obviously much more is possible with its command line due to the many decades of constant evolution and improvement that have been put into it by the thousands of active developers in the open source community.

This makes it very rewarding to use.

Here are some commands that are purely about fun and serve little to no purpose in the serious workplace.  They can each be installed using apt-get install or yum install depending on whether you’re using a debian or redhat based distribution.

Enjoy.

moo – displays a cow

sl – lists files in a directory like ls but if you get the syntax wrong, a train will drive across your screen

figlet – useful to turn text into big chunky letters

cowsay (and xcowsay) -displays a cow that says whatever you tell it to say as text (or graphic)

oneko – displays a little ASCII cat that chases your mouse around the screen

rev – reverses text

fortune – displays random quotes/nonsense.  Can be used in conjunction with cowsay e.g. fortune -s | cowsay

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 13

Shell Scripting “test”

Here’s a quick reference guide to the tests performed on a variable as part of a shell script.

test expr or [ expr ]

 

Example (if $var is not set to any value)

if [[ -z $var ]]; then

echo “variable has no value”

fi

 

-n file                true if variable has a value set

-z file                true if variable is empty

-L                      true if file is symbolic link

file1 -nt file2      true if file1 newer than file2

file1 -ot file2      true if file1 older than file2

file1 -ef file2      true if file1 and file2 are same device and inode number.

-e file                true if file exists (NOT ON HPUX use f instead)

-x file                true if file is executable

-r file                true if file is readable

-w file               true if file is writable

-f file                 true if file is a regular file

-d directory       true if directory is a directory

-c file                true if character special file

-b file                true if block special file

-p file                true if a named pipe

-u file                true if set UID bit is set

-g file                true if set GID bit is set

-k file                true if sticky bit is set

-s file                true if filesize > 0

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 09

Data Migration Shell Script Example

A nice little script written around the rsync command used to successfully migrate large amounts of data between NFS filesystems, avoiding .snapshot folders in the process.  A simple script in essence but a nice reference example nonetheless on the use of variables, functions, if statements, case statements, patterns and some useful commands, e.g. using sed to remove whitespace at the front of a variable returned by wc.

A simple but proper shell script that can almost certainly be built/improved upon using tee to write std output to a log file as well as the screen for instance, and using find to subsequently count the number of files afterwards because df is unlikely match to the nearest megabyte across different filesystems served by different NAS’s for comparison/verification.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
  SOURCE=$1
  DEST=$2

#Functions section
  function migratenonhiddenfolders(){

    echo “Re-Synchronising non-hidden top level folders only…”

  #Synchronise the data
    ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
      echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR}”
      timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR}
  done
  }

#Code section
  if [[ -z $1 ]];then
    echo “No Source or Destination specified”
    echo “Usage: migrate.sh /<source_fs> /<destination_fs>”
    exit
  fi
  if [[ -z $2 ]];then
    echo “No Destination specified”
    echo “Usage: migrate.sh /source_fs> /<destination_fs>”
    exit
  fi

#Source and Destination filesystems have been specified
  echo “Source filesystem: $SOURCE”
  FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`
  echo “The $FOLDERCOUNT source folders are…”
  ls -l $SOURCE | grep ^d | awk {‘print $9’}
  echo
  echo “Destination filesystem: $DEST”
  echo
  echo -n “Please confirm the details are correct [Yes/No] > “
  read CONFIRM
    case $CONFIRM in
        [Yy] | [Yy][Ee][Ss])
          migratenonhiddenfolders
          ;;

       *)
        echo
        echo “User aborted.”
        exit
       ;;
    esac

#Clean exit
exit

Improved version (with logging) shown below.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
  SOURCE=$1
  DEST=$2

#Functions section
  function migratenonhiddenfolders(){

    echo “Migrating ${SOURCE} to ${DEST} at `date`” >> ~/migration.log
    echo “Re-Synchronising non-hidden top level folders only…” | tee -a ~/migration.log
    #Synchronise the data
    ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
    echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR} | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    echo “Completed migrating to ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    done
  }

#Code section
  if [[ -z $1 ]];then
    echo “No Source or Destination specified”
    echo “Usage: migrate.sh /<source_fs> /<destination_fs>”
    exit
  fi
  if [[ -z $2 ]];then
    echo “No Destination specified”
    echo “Usage: migrate.sh /source_fs> /<destination_fs>”
    exit
  fi

#Source and Destination filesystems have been specified
  echo “Source filesystem: $SOURCE”
  FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`
  echo “The $FOLDERCOUNT source folders are…”
  ls -l $SOURCE | grep ^d | awk {‘print $9’}
  echo
  echo “Destination filesystem: $DEST”
  echo
  echo -n “Please confirm the details are correct [Yes/No] > “
  read CONFIRM
  case $CONFIRM in
    [Yy] | [Yy][Ee][Ss])
      migratenonhiddenfolders
     ;;
  *)
      echo
      echo “User aborted.”
      exit
      ;;
  esac

#Clean exit
  exit

###########################################################
##
## Data Migration script by M.D.Bradley, Cyberfella Ltd
## http://www.cyberfella.co.uk/2013/08/09/data-migration/
##
## Version 1.0 9th August 2013
###########################################################

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 08

Trimming whitespace off the beginning of a variable in bash

Sometimes a variable can be returned with a few empty (whitespace) characters in front of it.  This is not too much of an issue when running individual commands ending in a | wc -l to count the number of lines, but when you want to sandwich the variable in between two words in a more meaningful echo statement in a script, that “gap” between the words and the numbers looks untidy.  To remove the preceeding whitespace, append the following into the command used to build the variable (between the back ticks `   `).

     sed -e ‘s/^[ \t]*//’

e.g. to count the number of folders…

     FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`

     echo “The $FOLDERCOUNT source folders are…”

Will output this…

     The 4 source folders are…

instead of this…

     The            4 source folders are…

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
May 15

Translate text from lowercase to uppercase

When comparing files on Linux, there are a bunch of tools available to you, which are covered in separate posts on my blog.  This neat trick deserves its own post though -namely converting between upper and lowercase.

Before comparing two text files that have been sorted, duplicates removed with uniq and grepped etc, remember to convert to lower or upper case prior to making final comparison with another file.

tr ‘[:lower:]’ ‘[:upper:]’ <input-file > output-file

My preferred way to compare files isn’t using diff or comm but to use grep…  More often than not it gives me the result I want.

grep -Fxv -f first-file second-file

This returns lines in the second file that are not in the first file.

When comparing files, remember to remove any BLANK LINES.

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Mar 19

Comparing two lists in Linux/UNIX

Deserves its own blog post this one.  A new favourite of mine, that seems more reliable than using diff or comm on account of its being easy to understand, and thus less likely to get wrong (matter of opinion).

grep -Fxv -f masterlist backupclients which would list any lines in a list of backupclients that were not found in the master list.

Note:  This lists any lines in the second file that are not matched in the first file (not the other way around).

-F pattern to match is a list of fixed strings

-x select only matches that match the whole line

-v reverses it, i.e. does not match the whole line

-f list of strings to (not) match are in this file

Result: filename output entire lines from the file specified that are not in the file specified by –f

 

It can be useful to convert all text to UPPERCASE and don’t forget to REMOVE EMPTY LINES in files.

Text can be converted to uppercase with tr ‘[:lower:] [:upper:]’ <infile >outfile

If you want to list only the lines that appear in two files, then comm -12 firstfile secondfile works well.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
1 comment
Social Media Auto Publish Powered By : XYZScripts.com