Anti-virus on VNX CIFS Servers

To scan viruses on your Windows File Servers using local or block (SAN) storage is easy – you just install an AV agent on the Windows Server and voila.  But what if your Windows File Server is replaced by an emc VNX CIFS Server?

The VNX uses an optional agent called CAVA (Common Anti-virus Agent) that enables a filter driver on the CIFS Server that sends  the file off to a third party AV server for scanning.  If a virus signature is found, the VNX subsequently deletes the file.

Here’s everything you need to set it up…   (Note that versions described below may change over time).

emc CAVA for VNX Installation, Configuration and Administration

Create a Windows Server, preferably 2 or a couple of VMs and add to the domain.

Download VNX Common Event Enabler from here (291MB)…

You’ll need to register an account on support.emc.com if you don’t already have one (Powerlink account).

https://download.emc.com/downloads/DL48037_Common-Event-Enabler-6.3.1-for-Windows.iso

Install VNX Common Event Event Enabler 6.3.1 (includes CAVA) and a 3rd party AV product of your choice.
emc_VEE_Pack_x64_6.3.1.exe

You will also need to install <vnx nas version>_VNXFileCifsMgmt.exe which sadly is only available on CD2 of the Tools Pack that came with your VNX.  If you’ve subsequently upgraded the NAS to a more recent version, you’ll need to obtain the latest software from EMC.  I was able to download the elusive software from a link sent to me by EMC support, even though I couldn’t find it or search for it on Powerlink.  The links below may work for you, it may not.  Try it.

https://support.emc.com/search/?text=”cifs%20tools”&facetResource=ST

or try this one…

https://support.emc.com/search/?text=Dl48750%20DL32448

Start, Administrative Tools, Celerra Management,
Expand Data Mover Management (you’ll need to point it at the IP address of your CIFS interface)
Expand Anti-virus
Set file masks (don’t use *.*), and exclude files that don’t harbor viruses, configure CAVA CIFS Server name to exactly match that on the VNX CIFS Server name (may need to be in caps!), and IP addresses of CAVA AV Servers.  Example viruschecker.conf shown below.  How you get this into your viruschecker.conf is your problem.  Personally, I’d take the easy option of using the gui, then manually edit the viruschecker.conf file using vi to fix any problems, remove square brackets and stuff.  To edit the viruschecker.conf file manually on the datamover over ssh, log on as nasadmin, su to root and use these commands…

server_file server_2 -get viruschecker.conf viruschecker.conf

vi viruschecker.conf (and tidy it up)

server_file server_2 -put viruschecker.conf viruschecker.conf

CIFSserver=globalcifsserver  -Note that this CIFS Server must reside on physical DM, not your CIFS Server on VDM
Addr=<IP addresses of AV engines separated by semi colons> eg 10.1.1.1:10.1.1.2
shutdown=viruschecking

excl=*.dwl:*.edb:*.fmb:*.fmt:*.fmx:*.frm:*.inp:*.ldb:*.ldf:*.mad:*.maf:*.mam:*.maq:*.mar:*.mat:*.mda:*.mdb:*.mde:*.mdf:*.mdn:*.mdw:*.mdz:*.ndf:*.ora:*.orc:*.ost:*.pst:*.sc:*.sqc:*.sql:*.sqr:*.stm:*.tar:*.tmp:*.zip:????????:*RECYCLER*

masks=*.386:*.ace:*.acm:*.acv:*.acx:*.add:*.ade:*.adp:*.adt:*.app:*.asd:*.asp:*.asx:*.avb:*.ax:*.ax?:*.bas:*.bat:*.bin:*.bo?:*.btm:*.cbt:*.cdr:*.cer:*.cfm:*.chm:*.cla:*.class:*.cmd:*.cnv:*.com:*.cpl:*.cpy:*.crt:*.csc:*.csh:*.css:*.dat:*.dbx:*.der:*.dev:*.dl?:*.dll:*.do?:*.do??:*.doc:*.docx:*.dot:*.drv:*.dvb:*.dwg:*.eml:*.exe:*.fon:*.fxp:*.gadget:*.gms:*.gvb:*.hlp:*.hta:*.htm:*.html:*.htt:*.htw:*.htx:*.im?:*.inf:*.ini:*.ins:*.ins:*.isp:*.its:*.js:*.js?:*.jse:*.jtd:*.lgp:*.lib:*.lnk:*.lnk:*.mad:*.maf:*.mag:*.mam:*.maq:*.mar:*.mas:*.mat:*.mau:*.mav:*.maw:*.mb?:*.mda:*.mdb:*.mde:*.mdt:*.mdw:*.mdz:*.mht:*.mhtm:*.mhtml:*.mod:*.mp?:*.mpd:*.mpp:*.mpt:*.mrc:*.ms?:*.msc:*.msg:*.msh:*.msh1:*.ksh:*.msh1xml:*.msh2:*.msh2xml:*.mshxml:*.msi:*.mso:*.msp:*.mst:*.nch:*.nws:*.obd:*.obj:*.obz:*.ocx:*.oft:*.olb:*.ole:*.ops:*.otm:*.ov?:*.pcd:*.pcd:*.pci:*.pdb:*.pdf:*.pdr:*.php:*.pif:*.pl:*.plg:*.pm:*.pnf:*.pnp:*.pot:*.pot:*.pp?:*.pp??:*.ppa:*.pps:*.pps:*.ppt:*.prc:*.prf:*.prg:*.ps1:*.ps1xml:*.ps2:*.ps2xml:*.psc2:*.pwz:*.qlb:*.qpw:*.reg:*.rtf:*.sbf:*.scf:*.sco:*.scr:*.sct:*.sh:*.shb:*.shs:*.sht:*.shtml:*.shw:*.sis:*.smm:*.swf:*.sys:*.td0:*.tlb:*.tmp:*.tsk:*.tsp:*.tt6:*.url:*.vb:*.vb?:*.vba:*.vbe:*.vbs:*.vbx:*.vom:*.vs?:*.vsd:*.vsmacros:*.vss:*.vst:*.vsw:*.vwp:*.vxd:*.vxe:*.wbk:*.wbt:*.wiz:*.wk?:*.wml:*.wms:*.wpc:*.wpd:*.ws:*.ws?:*.wsc:*.wsf:*.wsh:*.xl?:*.xl??:*.xla:*.xls:*.xlt:*.xlw:*.xml:*.xnk:*.xtp

Create a service account in the domain and check the user rights

Create a local group viruscheckers on the CIFS Server using the local users and groups snap-in, and add your service account in.

Make your service account a local admin on the CAVA Servers and double check that the debug programs right in group policy has local administrators in it (windows default setting) or put the cava service account in it.  This is needed for the CAVA service to query the OS on the VM to determine the AV engine.

GPO_name\Computer Configuration\Windows Settings\Security Settings\Local Policies\User Rights Assignment        Debug Programs

Restart the EMC CAVA service on the CAVA Vm’s using this service account – note: it’ll get assigned Log On As A Service rights automatically.

If you need to re-add rights to the CAVA service account in group policy for any reason (they’ve been stripped out in an update), then you’ll need to also restart the CAVA Service on the VM before the CAVA Agent on the Datamover will re-recognise the AV engine.

In the EMC Celerra Management snap-in

Expand User Rights Assignment
Expand EMC Virus Check
Add
Select the service account in the Domain to give virus checking right to, Add, OK, OK

PuTTY/SSH to VNX Control Station
Login as nasadmin
server_viruschk server_2
You should see ONLINE, plus details of file masks and AV server used.

If you get Unknown AV Engine or Third Party AV engine, even though you’re using McAfee or Sophos or one of the other supported AV engines, then something is up – HP Protect Tools can get in the way of the DM authenticating to the CAVA VM’s.  I’m using McAfee and although mcshield.exe is a known av engine and its running, it didn’t pick it up because the password was getting scrambled by ProtectTools.  Check your AV policy being applied to the AV engine includes Network Drives.  It may not.  Until you solve this problem, set shutdown=viruschecking in your viruschecker.conf to shutdown=no to prevent it from stopping all the time.  Use the snap-in to adjust this setting.  Also make sure your viruschecker.conf is pointing as a global cifs server permanently resident on the physical datamover and not your cifs server on a virtual data mover thats actually sharing your filesystems.

server_viruschk server_2 -audit
Should see details of viruses caught. This can be tested using EICAR test virus and dropping the file into the CIFS Share on the CIFS Server.
The file should get automatically deleted by your anti-virus software.

Reboot everything once it’s all set up (CAVA Vm’s).  A reboot can cure most problems.

Common Commands via the CLI

Replace server_x with the data mover you are accessing eg server_2

server_viruschk server_x Shows if virus checking is running and scanning rules
server_viruschk server_x -audit Shows CAVA scanning stats and scan queue. Very useful to see if the CAVA queue is blocked
server_log server_x To see if there are any errors on the data movers
server_setup server_x –P viruschk –o start=64 Start the virus checker service on the data mover
server_setup server_x –P viruschk –o stop Stop the virus checker service on the data mover
server_viruschk server_x –fsscan fs1 –create Starts a virus scanning job a on file system
server_viruschk server_x –fsscan fs1 –delete Stops a virus scanning job on a file system
server_viruschk server_x –fsscan fs1 –list Show the scanning status

Debugging CAVA

You can set debug logging on the data mover

.server_config server_2 “param viruschk Traces=0x00000004” #turns on debug for AV in the server_log
.server_config server_2 “param viruschk Traces=0x00000000” #turns off debug for AV in the server_log

server_log server_x To see if there are any errors logged on the data movers.

[paypal-donation]

Facebooktwitterredditpinterestlinkedinmail

Configure VNX for NDMP

Filesystems on your VNX can be backed up using NDMP in emc Networker.  The Networker side of things is already concisely documented here… under the heading Adding a NAS filesystem to backup (using NDMP)

http://www.cyberfella.co.uk/2012/08/28/emc-networker/

 


In order for NDMP backups to work, some configuration of the VNX is necessary.

Create network interface for Backup

server_ifconfig server_2 -create -Device FSN01_NDMP_Backup -name Backup -protocol IP iii.iii.iii.iii sss.sss.sss.sss bbb.bbb.bbb.bbb mtu=1500 vlan=902

If you’re unsure of the correct broadcast address bbb.bbb.bbb.bbb, you can go through the motions of configuring an interface using Unisphere first, then use the pre-determined broadcast address in your command line before committing the change.  If you do commit the change, you’ll just need to delete the interface before recreating it using the CLI.  The advantage to using the CLI is that you can assign a name to your interface e.g. “Backup”.  Using unisphere, it’ll assign a name automatically (the IP address separated by hyphens) which is less meaningful.

 

Check for ndmp user and add user and reset password

/nas/sbin/server_user server_2 -list

/nas/sbin/server_user -add -md5 -passwd ndmp

/nas/sbin/server_user -passwd ndmp

********

 

Check port range

server_param server_2 -facility NDMP -info portRange

1024-65535

 

Check data streams value is consistent with parallelism value in Networker

server_param server_2 –facility NDMP –info concurrentDataStreams

server_param server_2 –facility NDMP –modify concurrentDataStreams –value 8

 

List full paths required to configure NDMP backup clients (emc VNX)

server_mount server_2

e.g. /root_vdm_2/CYBERFELLA_Test_FS

 

[paypal-donation]

Facebooktwitterredditpinterestlinkedinmail

Data Migration Shell Script Example

A nice little script written around the rsync command used to successfully migrate large amounts of data between NFS filesystems, avoiding .snapshot folders in the process.  A simple script in essence but a nice reference example nonetheless on the use of variables, functions, if statements, case statements, patterns and some useful commands, e.g. using sed to remove whitespace at the front of a variable returned by wc.

A simple but proper shell script that can almost certainly be built/improved upon using tee to write std output to a log file as well as the screen for instance, and using find to subsequently count the number of files afterwards because df is unlikely match to the nearest megabyte across different filesystems served by different NAS’s for comparison/verification.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
  SOURCE=$1
  DEST=$2

#Functions section
  function migratenonhiddenfolders(){

    echo “Re-Synchronising non-hidden top level folders only…”

  #Synchronise the data
    ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
      echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR}”
      timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR}
  done
  }

#Code section
  if [[ -z $1 ]];then
    echo “No Source or Destination specified”
    echo “Usage: migrate.sh /<source_fs> /<destination_fs>”
    exit
  fi
  if [[ -z $2 ]];then
    echo “No Destination specified”
    echo “Usage: migrate.sh /source_fs> /<destination_fs>”
    exit
  fi

#Source and Destination filesystems have been specified
  echo “Source filesystem: $SOURCE”
  FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`
  echo “The $FOLDERCOUNT source folders are…”
  ls -l $SOURCE | grep ^d | awk {‘print $9’}
  echo
  echo “Destination filesystem: $DEST”
  echo
  echo -n “Please confirm the details are correct [Yes/No] > “
  read CONFIRM
    case $CONFIRM in
        [Yy] | [Yy][Ee][Ss])
          migratenonhiddenfolders
          ;;

       *)
        echo
        echo “User aborted.”
        exit
       ;;
    esac

#Clean exit
exit

Improved version (with logging) shown below.

#!/usr/bin/bash

#Generic script for migrating file systems.
#Variables Section
  SOURCE=$1
  DEST=$2

#Functions section
  function migratenonhiddenfolders(){

    echo “Migrating ${SOURCE} to ${DEST} at `date`” >> ~/migration.log
    echo “Re-Synchronising non-hidden top level folders only…” | tee -a ~/migration.log
    #Synchronise the data
    ls -l $SOURCE | grep ^d | awk {‘print $9’} | while read EACHDIR; do
    echo “Syncing ${SOURCE}/${EACHDIR} with ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    timex /usr/local/bin/rsync -au ${SOURCE}/${EACHDIR}/* ${DEST}/${EACHDIR} | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    echo “Completed migrating to ${DEST}/${EACHDIR} at `date`” | tee -a ~/${DEST}_${EACHDIR}.log ~/${DEST}.log ~/migration.log
    done
  }

#Code section
  if [[ -z $1 ]];then
    echo “No Source or Destination specified”
    echo “Usage: migrate.sh /<source_fs> /<destination_fs>”
    exit
  fi
  if [[ -z $2 ]];then
    echo “No Destination specified”
    echo “Usage: migrate.sh /source_fs> /<destination_fs>”
    exit
  fi

#Source and Destination filesystems have been specified
  echo “Source filesystem: $SOURCE”
  FOLDERCOUNT=`ls -l $SOURCE | grep ^d | wc -l | sed -e ‘s/^[ \t]*//’`
  echo “The $FOLDERCOUNT source folders are…”
  ls -l $SOURCE | grep ^d | awk {‘print $9’}
  echo
  echo “Destination filesystem: $DEST”
  echo
  echo -n “Please confirm the details are correct [Yes/No] > “
  read CONFIRM
  case $CONFIRM in
    [Yy] | [Yy][Ee][Ss])
      migratenonhiddenfolders
     ;;
  *)
      echo
      echo “User aborted.”
      exit
      ;;
  esac

#Clean exit
  exit

###########################################################
##
## Data Migration script by M.D.Bradley, Cyberfella Ltd
## http://www.cyberfella.co.uk/2013/08/09/data-migration/
##
## Version 1.0 9th August 2013
###########################################################

Facebooktwitterredditpinterestlinkedinmail

A Waste of Space

I recently witnessed a huge 40% increase in disk space used moving some files from one SAN to another despite the number of files being the same on both the source and destination filesystems/SANs.

The reason this happens is primarily down to block size.

Ordinarily you wouldn’t notice much difference as all files are different sizes and most files are bigger than the minimum block size of the filesystem, so although some space always gets wasted as the last bit of data is written to the last block before starting the next file in a new block, the amount of wasted space shouldn’t be hugely different between filesystems.

But, what if the files are all roughly the same size, and that size is smaller than the minimum block size, i.e. you’re writing millions of 4Kb files into a filesystem that has an 8Kb block size?  Ouch is the answer, as over half of your filesystem capacity will be wasted.

wasted-space

 

 

 

 

 

 

The answer is to make sure that you know a bit more about the data that’s going to be written to the destination filesystem than just it’s top level capacity (returned by a command such as df -h, which actually tells you how many blocks of data are used on the entire filesystem multiplied by the block size to give a more humanly meaningful answer in KB/MB/GB instead of number of just blocks occupied).

Facebooktwitterredditpinterestlinkedinmail

Obtain Status Summary from FC Switches in your SAN

Keep SAN switch status info in one place and analyse it daily.  This can be useful for SAN reporting purposes.

It’s a little script I knocked together that looks “bigger” than it actually is (I could improve it further by introducing another loop to eliminate a lot of repetitive code).

It has the pre-requisite of copying ssh keys to each switch to allow a remote nasadmin user to authenticate without passing a password across the network prior to running the show interface brief command and passing the output back to the remote script over an encrypted connection.

Don’t put passwords in your scripts, especially for user accounts that have superuser access to important stuff like your fc switches.  Hacking (in the context of Cracking) is an opportunist crime that exploits bad practices like this to gain entry to the parts of your infrastructure that a denial of service attack would cause the most damage.

#!/bin/sh

#Variables
MYDIR=/local/home/nasadmin/switches/
# Substitue names of fc-switches below…
FC_SWITCHES=( fc-switch-1 fc-switch-2 fc-switch-3 fc-switch-4 )

#Code Section
#Obtain detail from each switch
#(requires ssh keys to be set up on each switch to enable passwordless ssh authentication of admin user on central node)
for EACHFCSWITCH in ${FC_SWITCHES[@]};
# begin first loop
do
/usr/bin/ssh -q admin@${EACHFCSWITCH} “show interface brief” | tee ${MYDIR}/ShowInterfaceBrief_${EACHFCSWITCH}
done

#All Show Interface Brief information collected.

#Process information to summarize it in /local/home/nasadmin/switches/SwitchSummary
rm ${MYDIR}/SwitchSummary} 2>&1
cd ${MYDIR}
ls -1 ${MYDIR} | grep ^Sh | while read EACHSWITCH
do
VAR_in=`grep “in” ${EACHSWITCH} | wc -l`
VAR_fc=`grep “fc” ${EACHSWITCH} | wc -l`
VAR_up=`grep “up” ${EACHSWITCH} | wc -l`
VAR_notConnected=`grep “notConnected” ${EACHSWITCH} | wc -l`
VAR_down=`grep “down” ${EACHSWITCH} | wc -l`
VAR_trunking=`grep “trunking” ${EACHSWITCH} | wc -l`
VAR_sfpAbsent=`grep “sfpAbsent” ${EACHSWITCH} | wc -l`
VAR_errDisabled=`grep “errDisabled” ${EACHSWITCH} | wc -l`

echo ${EACHSWITCH} >> ${MYDIR}/SwitchSummary
echo “in ${VAR_in}” >> ${MYDIR}/SwitchSummary
echo “fc ${VAR_fc}” >> ${MYDIR}/SwitchSummary
echo “up ${VAR_up}” >> ${MYDIR}/SwitchSummary
echo “notConnected ${VAR_notConnected}” >> ${MYDIR}/SwitchSummary
echo “down ${VAR_down}” >> ${MYDIR}/SwitchSummary
echo “trunking ${VAR_trunking}” >> ${MYDIR}/SwitchSummary
echo “sfpAbsent ${VAR_sfpAbsent}” >> ${MYDIR}/SwitchSummary
echo “errDisabled ${VAR_errDisabled}” >> ${MYDIR}/SwitchSummary
echo ” ” >> ${MYDIR}/SwitchSummary
done

#Process information to summarize it in /local/home/nasadmin/switches/SwitchSummary.csv

echo -n “Switch,” > ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | cut -d_ -f2 | while read EACHSWITCH
do
echo -n “${EACHSWITCH},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “in,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_in=`grep “in” ${EACHSWITCH} | wc -l`
echo -n “${VAR_in},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “fc,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_fc=`grep “fc” ${EACHSWITCH} | wc -l`
echo -n “${VAR_fc},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “notConnected,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_notConnected=`grep “notConnected” ${EACHSWITCH} | wc -l`
echo -n “${VAR_notConnected},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “down,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_down=`grep “down” ${EACHSWITCH} | wc -l`
echo -n “${VAR_down},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “trunking,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_trunking=`grep “trunking” ${EACHSWITCH} | wc -l`
echo -n “${VAR_trunking},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “sfpAbsent,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_sfpAbsent=`grep “sfpAbsent” ${EACHSWITCH} | wc -l`
echo -n “${VAR_sfpAbsent},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “errDisabled,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_errDisabled=`grep “errDisabled” ${EACHSWITCH} | wc -l`
echo -n “${VAR_errDisabled},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

sed ‘s/,$//’ ${MYDIR}/SwitchSummary.csv >${MYDIR}/SwitchSummaryExcelFormat.csv
rm ${MYDIR}SwitchSummary.csv

#Optionally copy to windows share–> scp S${MYDIR}/witchSummaryExcelFormat.csv windows-server:/windows-share

#Clean exit (before Comments section)
cd –
exit

Facebooktwitterredditpinterestlinkedinmail

Passwordless authentication to remote systems

Sometimes, you’ll be faced with writing a script that goes off, collects some information from a remote system, and copies that information to another remote system for off-site storage.  Such an example would be the configuration information of remote unix systems, a backup server or the configuration of a san/nas storage device on your network.

In order to run this job as an automated task, you need to be able to connect securely to the remote systems before copying configuration data over the network, but don’t want to have to enter passwords manually, and shouldn’t include passwords in your scripts either (for security reasons), so how do you do it?

You can use ssh keys to do it.

ssh keys are a pair of keys that can be generated for any given user account (called public key and private key), and the private key is then securely copied to the remote system, so that when a connection attempt is made to that remote system by the user offering their public key, the two keys are put together on the remote system to form a successful means of authenticating to that remote system, just like a normal password, but it’s all done by the systems with no interaction required by the user.

If that sounded complicated, it wasn’t meant to.  And it isn’t.  In summary, setting this passwordless authentication mechanism up goes like this…

1. Generate keys for my user using ssh-keygen

2. Copy the keys to the remote system using ssh-copy-id

3. ssh to the remote system to test.  Voila!

A real-world working example of copying the contents of someuser‘s ~/.ssh/id-rsa.pub public key to a remote nas device’s /home/someuser/.ssh/authorized-keys file is shown below with the input required from the sysadmin shown in bold.

myuser@myserver.cyberfella.co.uk 5$ su – someuser
Password:
[someuser@myserver ~]$ pwd
/local/home/someuser
[someuser@myserver ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/local/home/someuser/.ssh/id_rsa):
Created directory ‘/local/home/someuser/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /local/home/someuser/.ssh/id_rsa.
Your public key has been saved in /local/home/someuser/.ssh/id_rsa.pub.
The key fingerprint is:
d2:4b:53:80:4e:17:32:b1:1b:d3:d3:5c:9c:41:6a:94 someuser@myserver.cyberfella.co.uk
[someuser@myserver ~]$ cd .ssh
[someuser@myserver ~/.ssh]$ ls
id_rsa  id_rsa.pub
[someuser@myserver ~/.ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mynas
36
Warning: Permanently added ‘mynas,192.168.0.69’ (RSA) to the list of known hosts.
A customized version of the Linux operating system is used on the
EMC(R) VNX(TM) Control Station.  The operating system is
copyrighted and licensed pursuant to the GNU General Public License
(“GPL”), a copy of which can be found in the accompanying
documentation.  Please read the GPL carefully, because by using the
Linux operating system on the EMC Celerra you agree to the terms
and conditions listed therein.

EXCEPT FOR ANY WARRANTIES WHICH MAY BE PROVIDED UNDER THE TERMS AND
CONDITIONS OF THE APPLICABLE WRITTEN AGREEMENTS BETWEEN YOU AND EMC,
THE SOFTWARE PROGRAMS ARE PROVIDED AND LICENSED “AS IS” WITHOUT
WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT
NOT LIMITED TO, THE IMPLIED MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.  In no event will EMC Corporation be liable to
you or any other person or entity for (a) incidental, indirect,
special, exemplary or consequential damages or (b) any damages
whatsoever resulting from the loss of use, data or profits,
arising out of or in connection with the agreements between you
and EMC, the GPL, or your use of this software, even if advised
of the possibility of such damages.

EMC, VNX, Celerra, and CLARiiON are registered trademarks or trademarks of
EMC Corporation in the United States and/or other countries. All
other trademarks used herein are the property of their respective
owners.

EMC VNX Control Station Linux release 3.0 (NAS 7.0.53)
someuser@mynas’s password:
Warning: No xauth data; using fake authentication data for X11 forwarding.
Now try logging into the machine, with “ssh ‘mynas'”, and check in:

.ssh/authorized_keys

to make sure we haven’t added extra keys that you weren’t expecting.

[someuser@myserver ~/.ssh]$

myserver.cyberfella.co.uk 102# cd /etc
myserver.cyberfella.co.uk 103# vi cron.allow
myserver.cyberfella.co.uk 104# su – someuser
[someuser@myserver ~]$ crontab -e
no crontab for someuser – using an empty one

crontab: installing new crontab
[someuser@myserver ~]$ crontab -l
45 23 * * * /local/home/someuser/myscript.sh >/dev/null 2>&1
[someuser@myserver ~]$

 

Example commands in myscript.sh that work due to the passwordless authentication mechanism being in place are…

Backup myNAS information…

/usr/bin/scp -p -o ConnectTimeout=300 someuser@mynas:/celerra/backup/nasdb_backup.1.tar.gz /home/someuser/nasdb_backup.1.tar.gz

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_mount mydatamover ” | grep rw | grep -v “^root_fs_” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_export  ALL ” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_replicate -list ” >> ~/mynas_db_backup

Backup myNAS Network Information…

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_ifconfig server_2 -all ” >> ~/mynas_db_backup

Backup myNAS Quota information…

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_quotas -list -mover mydatamover” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_quotas -report -mover mydatamover” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_df ALL | grep -iv CKPT” >> ~/mynas_db_backup

 

Adding SSH Key to Cisco MDS Switch

If you have a script that goes off to a Cisco MDS switch, retreives some information and writes it back to a server, then you’d need to copy the ssh key to the switch, just like you would to a remote unix server for the purpose of passwordless authentication during the initial ssh connection.  Adding a public key to a cisco switch is done like this…

config t

username admin sshkey ssh-rsa <contents of id-rsa.pub here>

It’ll tell you if the key has been added successfully or not.

Deleting a SSH Key from Cisco MDS Switch

To subsequently remove an ssh public key from the switch, use…

no username admin sshkey ssh-rsa <contents of id-rsa.pub here>

 

Troubleshooting SSH connections

You may find yourself having issues with connecting as root or any other user for that matter.  Despite having created and copied you public keys to the remote systems, you’re still being prompted for passphrases or passwords for the user, defeating the whole point of setting up passwordless authentication.

Here’s a quick checklist of things to look out for and ways to troubleshoot the connection.

service stop sshd && /usr/sbin/sshd -d  (restart sshd in debug mode on the remote machine)

ssh -vv <remote-host> (connect to the remote host using ssh in verbose mode)

Before Googling the errors, make sure you can confirm the following:

When you generated the public keys using ssh-keygen you left the passphrase blank.

When you copied the keys over to the remote machine using ssh-copy-id you used the full path to the id_rsa.pub file.  If you’re root, it’s quite probable you copied another users ssh keys over instead of your own!

The .ssh directory in the users home directory has 700 permissions and the authorized-keys file has 600 permissions.

Facebooktwitterredditpinterestlinkedinmail

DataDomain Deduplication

Every year there’s a technology that really stands out and impresses me more than any other.  This year, that technology is Data Domain.  In short it’s like a NAS device, but performs de-duplication of common segments of incoming data in real time, massively reducing how many truly unique blocks of data are ultimately written down to the disks.

When used in conjunction with Networker backup solution, the overhead of de-duplicating all the incoming backup data is offloaded to…

Data Domain can also be used as a virtual tape library more on this in my Networker Cheatsheet here http://www.cyberfella.co.uk/2012/08/28/emc-networker/

…a process DDBoost running on the Networker Storage Node receiving the data from the client over the network prior to forwarding on the partially de-duplicated data stream to the DataDomain device.  DDBoost on the storage node performs the first 3 of the 5 stages of de-duplication, with the final 2 stages performed in real-time (as mentioned above) on the DataDomain device itself.

The overhead of all this de-duplication turns out to be insignificant too as it is ultimately less than the overhead of backing up all the data being received from the client in the first place.  Indexing of the backup data occurs at the Networker Backup and Recover Server, usually a separate server to the Storage Node which is blissfully unaware of the fact that the data passing through the storage node on it’s way to the “volume” is being de-duplicated.

You may think that all this makes for fast backups but will make for slow recoveries when the de-duplicated data has to be reconstituted, but in tests, recoveries are very fast, possibly due to the reduced number of seek times reading from the hard disks?.  It’s damn impressive anyhow.  Here’s a real example of just how impressive over 120 days of backing up many Terabytes of data daily…

Yes, you read it right.  It makes for a huge 96.9% reduction in the data ultimately being written to disk as it turns 43Tb of data into just 1Tb on disk.  Like I said, that’s impressive, and should make you realise how expensive and inefficient it is to carry on storing many copies of the same / static data on your expensive SANs as you can use a DataDomain for NFS and CIFS shares too.

Facebooktwitterredditpinterestlinkedinmail

Tuning an SSD powered Linux PC

So you’ve bought an SSD to give your everyday computing device a performance boost?  Well done.

The good news is, if you’re running Linux, there’s a handful of things you can do to make the most of your new super-powered block storage device.  My results below speak for themselves.  The bad news is, if you’re just a gadget consumer who has to have the latest and greatest, then simply buying it, fitting it and reinstalling the OS / cloning your previous drive is not going to cut it.  It’s more common sense than out-and-out rocket science, but whatever your OS, you can use my guide to give you ideas on what you can do to improve both performance and possibly the longevity of your device.  Being relatively new to the consumer market, the longevity of solid state block storage devices is yet to be seen.  At least you can do your bit to reduce the number of writes going to the device and (one would think) extend its life.

I chose to buy two relatively small capacity Intel SSD’s, connected each one to its own SATA controller on the system board and mount / on one and /home on the other.  I don’t see the point in buying large capacity SSD’s when it’s the performance you’re after rather than huge capacity to store your documents, photos, mp3, movie and software collections on – Thats what relatively cheap 2TB USB HDD’s and Cloud Storage providers like DropBox and Ubuntu One are for.  Oh and buy two of those two external HDD’s too because nobody wants to see 2TB of their data go irretrievably down the pan.

Incidentally, if you do lose data there is a nice previous blog entry on data forensics that will help you get it back. Search for forensics at the top or follow this link for that…

Disk Recovery and Forensics

Anyway, here’s the comparison of HDD performance to whet your appetite.

single hard disk in Lenovo IdeaCentre Q180

New dual SSD’s in HP dc7900 SFF PC

Tuning your SSD powered system…

Make sure the partitions are aligned.  This means that when a block is written to the filesystem, there are far fewer boundaries crossed on the ssd with each block written.

Much is written on the web about how to achieve this, I found the easiest way was to create a small ext2 /boot partition on the front of one drive, swap on the front of the other, and create my big / and /home partitions at the end of the disks (I have two remember) when using the manual partitioning tool gparted during installation.  By doing this, when i divided my starting sector number (returned by fdisk -l) by 512, i found the number was perfectly divisable – which is indicative of properly aligned partitions.  Job done then.

For each ssd in your computer, prepend noatime and discard to the options, leaving errors=remount-ro or defaults on the end.

/dev/sda   /   ext4   noatime,discard,errors=remount-ro 0 1

Change the scheduler from noop to deadline.
Add the following line for each SSD in your system:

echo deadline >/sys/block/sda/queue/scheduler

Make it do this each time you reboot
As root, vi /etc/rc.local and add these above the exit 0 line at the end of the file

echo deadline > /sys/block/sda/queue/scheduler
echo 1 > /sys/block/sda/queue/iosched/fifo_batch

GRUB Boot loader

vi /etc/default/grub    and change the following line…

GRUB_CMDLINE_LINUX_DEFAULT=”elevator=deadline quiet splash”

sudo update-grub

Reduce how aggressive swap is on the system.  A linux system with 2GB or more RAM will hardly ever swap.

echo 0 > /proc/sys/vm/swappiness
sudo vi /etc/sysctl.conf
change vm.swappiness=1

vm.vfs_cache_pressure=50

Move tmp areas to memory instead of ssd.  You’ll lose the contents of these temporary filesystems between boots, but on a desktop that may not be important.
In your /etc/fstab, add the following:

tmpfs   /tmp       tmpfs   defaults,noatime,mode=1777   0  0
tmpfs   /var/spool tmpfs   defaults,noatime,mode=1777   0  0
tmpfs   /var/tmp   tmpfs   defaults,noatime,mode=1777   0  0
tmpfs   /var/log   tmpfs   defaults,noatime,mode=0755   0  0

Move firefox cache (you’ll loose this between boots)
in firefox, type about:config and right click and create a new variable

browser.cache.disk.parent_directory

set it to /tmp

Boot from a live usb stick so the disks aren’t mounted, and as root, deactivate the journals on your ext4 partitions of your internal ssd’s e.g.

sudo tune2fs -O ^has_journal /dev/sda1

Add TRIM command to /etc/rc.local for each SSD i.e.

Above the line exit 0, add the following

fstrim -v /

fstrim -v /home     (only if your /home is mounted on a second SSD)

For computers that are always on, add trim command to /etc/cron.daily/trim

#!/bin/sh

fstrim -v / && fstrim -v /home

chmod +x /etc/cron.daily/trim

BIOS Settings

Set SATA mode to AHCI.  It will probably be set to IDE.  You’ll need to hunt for this setting is it varies between BIOS types.

SSD Firmware

Use lshw to identify your SSD and download the latest firmware from the manufacturer.  For Intel SSDs, go here

https://downloadcenter.intel.com/confirm.aspx?httpDown=http://downloadmirror.intel.com/18363/eng/issdfut_2.0.10.iso&lang=eng&Dwnldid=18363

 

That’s it.  I’ll add other tips to this list as and when I think of them or see them on the net.  You could reboot using a live usb and delete the residual files left behind in the tmp directories that you’ll be mounting in RAM from here on, but that’s up to you.  If you do, DO NOT remove the directories themselves or the system won’t boot.  If you do remove them, then fix it by booting using a live usb stick, mount the / partition into say, /ssd and mkdir the directories you deleted in /ssd/var/tmp and /ssd/tmp.  Be aware though that /tmp and /var/tmp have special permissions set on them.  chmod 777, followed by chmod +t to set the sticky bit -drwxrwxrwt (sticky bit set – with execution) on them.

Facebooktwitterredditpinterestlinkedinmail

Create filesystem on emc VNX

USEFUL CLI EXAMPLES for emc VNX NAS

Information

List all filesystems on the NAS:   nas_fs -list -all

Delete a filesystem:   nas_fs -delete -force

View info for all filesystems on the NAS:   nas_fs -info -all

View human friendly error message description:  nas_message -info 2216

List inodes:   server_df -inode   (Data Mover name obtained on filesystem properties in unisphere.  nas_fs -info -all doesn’t display it.

Create new filesystem

nas_fs -name NFS_Cyberfella_Dest -type uxfs -create size=20G pool=storage=SINGLE -thin no -option slice=y,nbpi=4096,mover=-mount_option mountmover=,mountpoint=/NFS_Cyberfella_Dest,mountmode=rw,accesspolicy=UNIX

Note: It’s best to delete filesystems via Unisphere GUI.  Use CLI for troublesome deletes.

Export/Unexport Filesystem over NFS

server_export-P nfs -list -all

server _export-P nfs -unexport -perm

Facebooktwitterredditpinterestlinkedinmail

Logical Volume Management

Logical volume management on Linux is a different way of using available disks (block devices), allowing for more flexible allocation of space on filesystems stored on one or more disks by grouping disks into volume groups, then creating logical volumes within those volume groups without boundaries.  Filesystems are created on the logical volumes which may use a portion of a disks full capacity, or the capacity of more than one disk.  The only limit is the total space in the volume group which is determined by the sum of the total physical space of all the physical block storage devices in the volume group.

You can add more devices to an existing volume group and even take them away (be careful!).  Volume groups, Logical Volumes in them, and the Filesystems on them can also be extended, resized or removed as necessary.

Scan for block devices that can be used for Logical Volume Management

lvmdiskscan

Display current physical volumes and their LVM Status

pvdisplay

Use block device /dev/hdb for LVM

pvcreate /dev/hdb

This command also takes multiple devicenames in one go, separated by spaces.

Create a volume group consisting of the physical volumes

vgcreate VolGroup01 /dev/hdb

Extend an existing volume group onto the block device just added

vgextend VolGroup00 /dev/hdb

Read the man page on vgextend for all available options.

Create a logical volume within the volume group

lvcreate -L +25G /dev/VolGroup01/LogVol00

Extend existing logical volume into the new unused space in the volume group

lvextend -L +25G /dev/VolGroup00/LogVol00

Expand the filesystem into the new free space in the logical volume.

resize2fs /dev/VolGroup00/LogVol00/

Verify the new size using df

df -kh

Display volume group information

vgdisplay VolGroup01

Display physical volume information

pvdisplay /dev/hdb

Display logical volume information

lvdisplay /dev/VolGroup01/LogVol00

Un-mount the filesystem

umount /filesystem

Check and Fix errors on the filesystem

e2fsck -f /dev/VolGroup01/LogVol00   

or  /dev/mapper/LogVol00

Resize a logical volume

lvresize -L new-size /dev/VolGroup01/LogVol00

Resize a filesystem

resize2fs /dev/mapper/LogVol00

e.g. 1500M or 44G

Facebooktwitterredditpinterestlinkedinmail