Category: Backup / Disaster Recovery

Feb 06

Accidentally formatted hard disk recovery

So you had more than one hard disk plugged into your nice new Humax FreeSat set top box, one containing all your existing downloaded media and the other, an empty one intended for recording.

Upon formatting the drive intended for recording you subsequently discover that your other FAT32 disk with all your media on it, now has a nice new, empty NTFS partition on it too.  A real WTF moment that absolutely is not your fault.  It happens to the best of us.  It’s just happened to me.

It’s in these moments that having a can-do attitude is of the utmost importance.  Congratulations are in order, because Life has just presented a real challenge for you to overcome.

The likelihood is 95% of your friends will feign sympathy and tell you…

“there’s nothing you can do if you’ve re-formatted the drive”

the largely self-appointed “tech experts” (on the basis they have all-the-gear) will likely tell you…

“you’ve reformatted your FAT32 partition with NTFS so you’ve lost everything.”

…like you’d have stood a chance if you’d gone over it with a like-for-like file system format and they could have got all your data back for you (yeah, right).

Well, if you’ve been sensible enough to not make any writes to the drive, then I can tell you that you absolutely can recover all your data.  In fact, there’s no data to recover as it’s all still on the drive, so “recovery” will be instantaneous.   I’m here to tell you…

You need a computer running Linux and you need to install the testdisk package.

In a console window, run sudo testdisk

You may need to unmount the disk first using gparted but leave it plugged in.

In testdisk, you need to list partitions and it’ll display the new high performance file system NTFS partition and nothing else at this point.  There is an option to do a “deeper scan”.  This walks the cylinders looking for any evidence that a previous file system was here.  If you’ve not done any writes to the drive since it got reformatted with NTFS, then it’ll instantly find details of a previous FAT32 partition.  You can cancel the scan at this point as it’s found all it needs (see below)

What you need to do now is tell the disk that it’s this format you want on the primary partition, not the current NTFS one.  You can select it, and even list the files on it (P).

This can in someways be the most frustrating part, as you can see that the files and the index are there, but your file manager will still show an empty NTFS disk.  Now you need to switch the NTFS structured disk back over to FAT32 by writing the previously discovered FAT32 structure over the top of the primary partition.

You’ll receive a message along the lines of needing a reboot.  You just need to quit testdisk, and remove and re-add the hard disk (if it’s USB) or reboot if it’s an internal drive and re-run test disk after to see that the NTFS partition structure has been replaced with the FAT32 one that existed before.

Like before, you can list the files on the partition using testdisk.  Seeing as this partition is now the current live one, the files should also appear in your file manager.  In my case, I’m using the Nemo file manager on Linux Mint 18.1 Serena, Cinnamon 3.0 edition (and I can highly recommend it).

So there you go.  There are a few lessons to be learned here -for all of us, but like many things in life, things are not always as they seem.  Your computers file manager does not show you what data is on the disk – it is merely reading the contents of the current known good file allocation table from an address on the front of the disk that the partition is known to begin at.  Such file allocation tables will exist all over the disk from previous lives in between re-formatted for re-use.  When you re-format a disk, you’re just giving the file allocation table a fresh start with a new address but the old one will still exist somewhere and in multiple places on the disk.  The file allocation table is the index of disk contents that is read by the file manager in order to give you a representation of what it believes to be retrievable data on your disk.  The data itself can then be found starting at the addresses contained in that index for each file.  The data is still there on parts of the disk that have not yet been written over with replacement blocks of data, hence if you’ve not performed any writes, then all your data is all still there.  So if you want your data to be truly irrecoverable, then you must perform multiple random writes over the top of all cylinders using a tool like DBAN that will take hours to complete, or better, take an angle grinder to it.  Just remember to take a backup first.

So if you want your data to be truly irrecoverable, then you can perform multiple random writes over the top using a tool like DBAN, or better, take an angle grinder to it.

So the real proof that the data is indeed readable once again would be to open and play a movie file.  So as proof, here’s a little screenie of VLC Media Player playing Penelope Spheeris’ 1981 punk rock documentary “The Decline Of Western Civilization”.

Coincidentally, 1981 is quite a significant year for me, I was 6 years old and my parents had just bought me my first computer -a BBC Model B micro computer that had just been released.  I began teaching myself BASIC right away.

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
May 29

Reset sysadmin password on VNX

If the sysadmin password is lost or unknown, but you can ssh to the control station and log on as nasadmin, then you can run the following commands to view any additional admin accounts (if any were created).

/nas/sbin/navicli -h spa security -list -type

If there is another admin account on the system, try using the username as the password and see if that works.

If not, then you can create a temporary administrator account using the following command

/nas/sbin/navicli -h spa security -adduser -user tempadmin -password tempadmin -scope 0 -role administrator

This will allow you to log into the Unisphere global scope at the control stations IP address as tempadmin, where you can then reset the sysadmin password on VNX.

After verifying that you can now log back in to Unisphere as sysadmin, don’t forget to delete the tempadmin account from Settings, Global Users in Unisphere.

Official emc video available here https://community.emc.com/videos/5642

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Dec 03

Configure VNX for NDMP

Filesystems on your VNX can be backed up using NDMP in emc Networker.  The Networker side of things is already concisely documented here… under the heading Adding a NAS filesystem to backup (using NDMP)

http://www.cyberfella.co.uk/2012/08/28/emc-networker/

 

In order for NDMP backups to work, some configuration of the VNX is necessary.

Create network interface for Backup

server_ifconfig server_2 -create -Device FSN01_NDMP_Backup -name Backup -protocol IP iii.iii.iii.iii sss.sss.sss.sss bbb.bbb.bbb.bbb mtu=1500 vlan=902

If you’re unsure of the correct broadcast address bbb.bbb.bbb.bbb, you can go through the motions of configuring an interface using Unisphere first, then use the pre-determined broadcast address in your command line before committing the change.  If you do commit the change, you’ll just need to delete the interface before recreating it using the CLI.  The advantage to using the CLI is that you can assign a name to your interface e.g. “Backup”.  Using unisphere, it’ll assign a name automatically (the IP address separated by hyphens) which is less meaningful.

 

Check for ndmp user and add user and reset password

/nas/sbin/server_user server_2 -list

/nas/sbin/server_user -add -md5 -passwd ndmp

/nas/sbin/server_user -passwd ndmp

********

 

Check port range

server_param server_2 -facility NDMP -info portRange

1024-65535

 

Check data streams value is consistent with parallelism value in Networker

server_param server_2 –facility NDMP –info concurrentDataStreams

server_param server_2 –facility NDMP –modify concurrentDataStreams –value 8

 

List full paths required to configure NDMP backup clients (emc VNX)

server_mount server_2

e.g. /root_vdm_2/CYBERFELLA_Test_FS

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
2 comments
Jul 25

Obtain Status Summary from FC Switches in your SAN

Keep SAN switch status info in one place and analyse it daily.  This can be useful for SAN reporting purposes.

It’s a little script I knocked together that looks “bigger” than it actually is (I could improve it further by introducing another loop to eliminate a lot of repetitive code).

It has the pre-requisite of copying ssh keys to each switch to allow a remote nasadmin user to authenticate without passing a password across the network prior to running the show interface brief command and passing the output back to the remote script over an encrypted connection.

Don’t put passwords in your scripts, especially for user accounts that have superuser access to important stuff like your fc switches.  Hacking (in the context of Cracking) is an opportunist crime that exploits bad practices like this to gain entry to the parts of your infrastructure that a denial of service attack would cause the most damage.

#!/bin/sh

#Variables
MYDIR=/local/home/nasadmin/switches/
# Substitue names of fc-switches below…
FC_SWITCHES=( fc-switch-1 fc-switch-2 fc-switch-3 fc-switch-4 )

#Code Section
#Obtain detail from each switch
#(requires ssh keys to be set up on each switch to enable passwordless ssh authentication of admin user on central node)
for EACHFCSWITCH in ${FC_SWITCHES[@]};
# begin first loop
do
/usr/bin/ssh -q admin@${EACHFCSWITCH} “show interface brief” | tee ${MYDIR}/ShowInterfaceBrief_${EACHFCSWITCH}
done

#All Show Interface Brief information collected.

#Process information to summarize it in /local/home/nasadmin/switches/SwitchSummary
rm ${MYDIR}/SwitchSummary} 2>&1
cd ${MYDIR}
ls -1 ${MYDIR} | grep ^Sh | while read EACHSWITCH
do
VAR_in=`grep “in” ${EACHSWITCH} | wc -l`
VAR_fc=`grep “fc” ${EACHSWITCH} | wc -l`
VAR_up=`grep “up” ${EACHSWITCH} | wc -l`
VAR_notConnected=`grep “notConnected” ${EACHSWITCH} | wc -l`
VAR_down=`grep “down” ${EACHSWITCH} | wc -l`
VAR_trunking=`grep “trunking” ${EACHSWITCH} | wc -l`
VAR_sfpAbsent=`grep “sfpAbsent” ${EACHSWITCH} | wc -l`
VAR_errDisabled=`grep “errDisabled” ${EACHSWITCH} | wc -l`

echo ${EACHSWITCH} >> ${MYDIR}/SwitchSummary
echo “in ${VAR_in}” >> ${MYDIR}/SwitchSummary
echo “fc ${VAR_fc}” >> ${MYDIR}/SwitchSummary
echo “up ${VAR_up}” >> ${MYDIR}/SwitchSummary
echo “notConnected ${VAR_notConnected}” >> ${MYDIR}/SwitchSummary
echo “down ${VAR_down}” >> ${MYDIR}/SwitchSummary
echo “trunking ${VAR_trunking}” >> ${MYDIR}/SwitchSummary
echo “sfpAbsent ${VAR_sfpAbsent}” >> ${MYDIR}/SwitchSummary
echo “errDisabled ${VAR_errDisabled}” >> ${MYDIR}/SwitchSummary
echo ” ” >> ${MYDIR}/SwitchSummary
done

#Process information to summarize it in /local/home/nasadmin/switches/SwitchSummary.csv

echo -n “Switch,” > ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | cut -d_ -f2 | while read EACHSWITCH
do
echo -n “${EACHSWITCH},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “in,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_in=`grep “in” ${EACHSWITCH} | wc -l`
echo -n “${VAR_in},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “fc,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_fc=`grep “fc” ${EACHSWITCH} | wc -l`
echo -n “${VAR_fc},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “notConnected,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_notConnected=`grep “notConnected” ${EACHSWITCH} | wc -l`
echo -n “${VAR_notConnected},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “down,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_down=`grep “down” ${EACHSWITCH} | wc -l`
echo -n “${VAR_down},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “trunking,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_trunking=`grep “trunking” ${EACHSWITCH} | wc -l`
echo -n “${VAR_trunking},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “sfpAbsent,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_sfpAbsent=`grep “sfpAbsent” ${EACHSWITCH} | wc -l`
echo -n “${VAR_sfpAbsent},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

echo -n “errDisabled,” >> ${MYDIR}/SwitchSummary.csv
ls -1 ${MYDIR} | grep Show | while read EACHSWITCH
do
VAR_errDisabled=`grep “errDisabled” ${EACHSWITCH} | wc -l`
echo -n “${VAR_errDisabled},” >> ${MYDIR}/SwitchSummary.csv
done
echo “” >> ${MYDIR}/SwitchSummary.csv

sed ‘s/,$//’ ${MYDIR}/SwitchSummary.csv >${MYDIR}/SwitchSummaryExcelFormat.csv
rm ${MYDIR}SwitchSummary.csv

#Optionally copy to windows share–> scp S${MYDIR}/witchSummaryExcelFormat.csv windows-server:/windows-share

#Clean exit (before Comments section)
cd –
exit

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
May 30

Exchange Management Shell

Some useful client-side commands to run in the Exchange Management Shell on the MBX (Mailbox Server) when diagnosing failed Exchange backups in emc Networker or other product (due to VSS Writer issues) and examples of the expected output.

Get-MailboxDatabase GB-DAG* -Status | select Server,name,LastFullBackup

Server Name LastFullBackup
—— —- ————–
Server1 DAG1-MDB01 23/05/2013 14:10:32
Server1 DAG1-MDB07 28/05/2013 00:06:25

Get-MailboxDatabase GB-DAG* -Status | select Server,name,LastIncrementalBackup

Server Name LastIncrementalBackup
—— —- ———————
Server1 MDB01 04/07/2013 19:44:04
Server1 MDB07 04/07/2013 19:44:04

Get-MailboxDatabase GB* -status | sort-object name | ft name, server, lastf*, lasti*, backupinprogress -auto

Name                        Server                              LastFullBackup              LastIncrementalBackup BackupInProgress
—- —— ————– ——————— —————-
MDB01 Server1 02/07/2013 14:45:11 04/07/2013 19:44:04    False
MDB02 Server1 02/07/2013 14:38:56 04/07/2013 19:42:35     False

Get-MailboxDatabase -Identity GB-DAG1* -Status | sort-object name| ft server,name,backupinprogress -auto

Server Name BackupInProgress
—— —- —————-
Server1-MBX001 DAG1-MDB01 False
Server2-MBX002 DAG1-MDB02 True

Get-MailboxDatabase GB* -status | sort-object name | ft name, server, activationpreference, lastf* -auto

Name Server ActivationPreference LastFullBackup
—- —— ——————– ————–
DAG1-MDB01 Server1-MBX001 {[Server1-MBX001, 1], [Server2-MBX001, 2]} 23/05/2013 14:10:32
DAG1-MDB02 Server2-MBX002 {[Server2-MBX002, 1], [Server1-MBX002, 2]} 28/05/2013 00:06:26

Get-MailboxDatabaseCopyStatus “*\GB*” | sort-object name

DAG1-MDB01\Server2-MBX001 Healthy 0 2 30/05/2013 07:50:56 Healthy
DAG1-MDB01\Server1-MBX001 Mounted 0 0 Healthy

vssadmin list writers
vssadmin 1.1 – Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: ‘Task Scheduler Writer’
Writer Id: {d61d61c8-d73a-4eee-8cdd-f6f9786b7124}
Writer Instance Id: {1bddd48e-5052-49db-9b07-b96f96727e6b}
State: [1] Stable
Last error: No error

Writer name: ‘VSS Metadata Store Writer’
Writer Id: {75dfb225-e2e4-4d39-9ac9-ffaff65ddf06}
Writer Instance Id: {088e7a7d-09a8-4cc6-a609-ad90e75ddc93}
State: [1] Stable
Last error: No error

 

See the active/passive status of each mailbox database how Networker will see it prior to backing it up (or ignoring it if it’s active and passive option is set)

nsrsnap_vss_save -v -?

To reset databases that have backups “In Progress” and reset any “Restartable errors” on VSS Writers, restart the Microsoft Exchange Replication Service on all servers in the DAG.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Jan 08

Passwordless authentication to remote systems

Sometimes, you’ll be faced with writing a script that goes off, collects some information from a remote system, and copies that information to another remote system for off-site storage.  Such an example would be the configuration information of remote unix systems, a backup server or the configuration of a san/nas storage device on your network.

In order to run this job as an automated task, you need to be able to connect securely to the remote systems before copying configuration data over the network, but don’t want to have to enter passwords manually, and shouldn’t include passwords in your scripts either (for security reasons), so how do you do it?

You can use ssh keys to do it.

ssh keys are a pair of keys that can be generated for any given user account (called public key and private key), and the private key is then securely copied to the remote system, so that when a connection attempt is made to that remote system by the user offering their public key, the two keys are put together on the remote system to form a successful means of authenticating to that remote system, just like a normal password, but it’s all done by the systems with no interaction required by the user.

If that sounded complicated, it wasn’t meant to.  And it isn’t.  In summary, setting this passwordless authentication mechanism up goes like this…

1. Generate keys for my user using ssh-keygen

2. Copy the keys to the remote system using ssh-copy-id

3. ssh to the remote system to test.  Voila!

A real-world working example of copying the contents of someuser‘s ~/.ssh/id-rsa.pub public key to a remote nas device’s /home/someuser/.ssh/authorized-keys file is shown below with the input required from the sysadmin shown in bold.

myuser@myserver.cyberfella.co.uk 5$ su – someuser
Password:
[someuser@myserver ~]$ pwd
/local/home/someuser
[someuser@myserver ~]$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/local/home/someuser/.ssh/id_rsa):
Created directory ‘/local/home/someuser/.ssh’.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /local/home/someuser/.ssh/id_rsa.
Your public key has been saved in /local/home/someuser/.ssh/id_rsa.pub.
The key fingerprint is:
d2:4b:53:80:4e:17:32:b1:1b:d3:d3:5c:9c:41:6a:94 someuser@myserver.cyberfella.co.uk
[someuser@myserver ~]$ cd .ssh
[someuser@myserver ~/.ssh]$ ls
id_rsa  id_rsa.pub
[someuser@myserver ~/.ssh]$ ssh-copy-id -i ~/.ssh/id_rsa.pub mynas
36
Warning: Permanently added ‘mynas,192.168.0.69’ (RSA) to the list of known hosts.
A customized version of the Linux operating system is used on the
EMC(R) VNX(TM) Control Station.  The operating system is
copyrighted and licensed pursuant to the GNU General Public License
(“GPL”), a copy of which can be found in the accompanying
documentation.  Please read the GPL carefully, because by using the
Linux operating system on the EMC Celerra you agree to the terms
and conditions listed therein.

EXCEPT FOR ANY WARRANTIES WHICH MAY BE PROVIDED UNDER THE TERMS AND
CONDITIONS OF THE APPLICABLE WRITTEN AGREEMENTS BETWEEN YOU AND EMC,
THE SOFTWARE PROGRAMS ARE PROVIDED AND LICENSED “AS IS” WITHOUT
WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT
NOT LIMITED TO, THE IMPLIED MERCHANTABILITY AND FITNESS FOR A
PARTICULAR PURPOSE.  In no event will EMC Corporation be liable to
you or any other person or entity for (a) incidental, indirect,
special, exemplary or consequential damages or (b) any damages
whatsoever resulting from the loss of use, data or profits,
arising out of or in connection with the agreements between you
and EMC, the GPL, or your use of this software, even if advised
of the possibility of such damages.

EMC, VNX, Celerra, and CLARiiON are registered trademarks or trademarks of
EMC Corporation in the United States and/or other countries. All
other trademarks used herein are the property of their respective
owners.

EMC VNX Control Station Linux release 3.0 (NAS 7.0.53)
someuser@mynas’s password:
Warning: No xauth data; using fake authentication data for X11 forwarding.
Now try logging into the machine, with “ssh ‘mynas'”, and check in:

.ssh/authorized_keys

to make sure we haven’t added extra keys that you weren’t expecting.

[someuser@myserver ~/.ssh]$

myserver.cyberfella.co.uk 102# cd /etc
myserver.cyberfella.co.uk 103# vi cron.allow
myserver.cyberfella.co.uk 104# su – someuser
[someuser@myserver ~]$ crontab -e
no crontab for someuser – using an empty one

crontab: installing new crontab
[someuser@myserver ~]$ crontab -l
45 23 * * * /local/home/someuser/myscript.sh >/dev/null 2>&1
[someuser@myserver ~]$

 

Example commands in myscript.sh that work due to the passwordless authentication mechanism being in place are…

Backup myNAS information…

/usr/bin/scp -p -o ConnectTimeout=300 someuser@mynas:/celerra/backup/nasdb_backup.1.tar.gz /home/someuser/nasdb_backup.1.tar.gz

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_mount mydatamover ” | grep rw | grep -v “^root_fs_” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_export  ALL ” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_replicate -list ” >> ~/mynas_db_backup

Backup myNAS Network Information…

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_ifconfig server_2 -all ” >> ~/mynas_db_backup

Backup myNAS Quota information…

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_quotas -list -mover mydatamover” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/nas_quotas -report -mover mydatamover” >> ~/mynas_db_backup

/usr/bin/ssh -q someuser@mynas “export NAS_DB=/nas;/nas/bin/server_df ALL | grep -iv CKPT” >> ~/mynas_db_backup

 

Adding SSH Key to Cisco MDS Switch

If you have a script that goes off to a Cisco MDS switch, retreives some information and writes it back to a server, then you’d need to copy the ssh key to the switch, just like you would to a remote unix server for the purpose of passwordless authentication during the initial ssh connection.  Adding a public key to a cisco switch is done like this…

config t

username admin sshkey ssh-rsa <contents of id-rsa.pub here>

It’ll tell you if the key has been added successfully or not.

Deleting a SSH Key from Cisco MDS Switch

To subsequently remove an ssh public key from the switch, use…

no username admin sshkey ssh-rsa <contents of id-rsa.pub here>

 

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Nov 16

DataDomain Deduplication

Every year there’s a technology that really stands out and impresses me more than any other.  This year, that technology is Data Domain.  In short it’s like a NAS device, but performs de-duplication of common segments of incoming data in real time, massively reducing how many truly unique blocks of data are ultimately written down to the disks.

When used in conjunction with Networker backup solution, the overhead of de-duplicating all the incoming backup data is offloaded to…

Data Domain can also be used as a virtual tape library more on this in my Networker Cheatsheet here http://www.cyberfella.co.uk/2012/08/28/emc-networker/

…a process DDBoost running on the Networker Storage Node receiving the data from the client over the network prior to forwarding on the partially de-duplicated data stream to the DataDomain device.  DDBoost on the storage node performs the first 3 of the 5 stages of de-duplication, with the final 2 stages performed in real-time (as mentioned above) on the DataDomain device itself.

The overhead of all this de-duplication turns out to be insignificant too as it is ultimately less than the overhead of backing up all the data being received from the client in the first place.  Indexing of the backup data occurs at the Networker Backup and Recover Server, usually a separate server to the Storage Node which is blissfully unaware of the fact that the data passing through the storage node on it’s way to the “volume” is being de-duplicated.

You may think that all this makes for fast backups but will make for slow recoveries when the de-duplicated data has to be reconstituted, but in tests, recoveries are very fast, possibly due to the reduced number of seek times reading from the hard disks?.  It’s damn impressive anyhow.  Here’s a real example of just how impressive over 120 days of backing up many Terabytes of data daily…

Yes, you read it right.  It makes for a huge 96.9% reduction in the data ultimately being written to disk as it turns 43Tb of data into just 1Tb on disk.  Like I said, that’s impressive, and should make you realise how expensive and inefficient it is to carry on storing many copies of the same / static data on your expensive SANs as you can use a DataDomain for NFS and CIFS shares too.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 28

Networker Cheatsheet

Here is a handy cheatsheet in troubleshooting failing backups and recoveries using emc’s Networker all taken from real-world experience (and regularly updated).

If it’s helped you out of a pinch, and is worth a dollar, then please consider donating to help maintain this useful blog.

Is backup server running?

nsrwatch -s backupserver        -Gives a console version of the NMC monitoring screen

Check the daemon.raw is being written to…

cp /nsr/logs/daemon.raw ~/copyofdaemon.raw

nsr_render_log -l ~/copyofdaemon.raw > ~/copyofdaemon.log

tail -10 ~/copyofdaemon.log

You may find mminfo and nsradmin commands are unsuccessful.  the media database may be unavailable and/or you may receive “program not registered” error that usually implies the Networker daemons/services are not running on the server/client.  This can also occur during busy times such as clone groups running (even though this busy-ness is not reflected in the load averages on the backup server.

Client config.

Can you ping the client / resolve the hostname or telnet to 7937?

Are the static routes configured (if necessary).

Can the client resolve the hostnames for the backup interfaces? have connectivity to them?

Does the backup server appear in the nsr/res/servers file?

Can you run a save -d3 -s /etc on the client?

From the backup server (CLI)…

nsradmin -p 390113 -s client

Note:  If the name field is incorrect according to nsradmin (happens when machines are re-commissioned without being rebuilt) then you need to stop nsrexecd, rename /nsr/nsrladb folder to /nsr/nsrladb.old, restart nsrexecd, and most importantly, delete and recreate the client on the networker backup server, before retrying a savegrp -vc client_name group_name

Also check that all interface names are in the servers file for all interfaces on all backup servers and storage nodes likely to back the client up.

Can you probe the client?

savegrp -pvc client groupname

savegrp -D2 -pc client groupname (more verbose)

Bulk import of clients

Instead of adding clients manually one at a time in the NMC, you can perform an initial bulk import.

nsradmin -i bulk-import-file

where the bulk-import-file contains many lines like this

create type: NSR Client;name:w2k8r2;comment:SOME COMMENT;aliases:w2k8r2,w2k8r2-b,w2k8r2.cyberfella.co.uk;browse policy:Six Weeks;retention policy:Six Weeks;group:zzmb-Realign-1;server network interface:backupsvrb1;storage nodes:storagenode1b1;

Use excel to form a large csv, then use Notepad++ to remove commas.  Be aware there is a comma in the aliases field, so use an alternative character in excel to represent this then replace it with a comma once all commas have been removed from the csv.

Add user to admin list on bu server

nsraddadmin -u user=username, host=*     

where username is the username minus the domain name prefix (not necessary).

Reset NMC Password (Windows)

The default administrator password is administrator.  If that doesn’t work, check to see that the GST service is started using a local system account (it is by default), then in Computer Management, Properties, Advanced Properties, create a System Environment Variable; GST_RESET_PW=1

Stop and start the GST Service and attempt to logon to the NMC using the default username and password pair above.

When done, set GST_RESET_PW=<null>

Starting a Backup / Group from the command line

On the backup server itself:  savegrp -D5 -G <group_name>

Ignore the index save sets if you are just testing a group by adding  -I

Just backing up the :index savesets in a group: savegrp -O -G <group_name>

On a client: save -s <backup_server_backupnic_name> <path>

Reporting with mminfo

List names of all clients backed up over the last 2 weeks (list all clients)

mminfo -q “savetime>2 weeks ago” -r ‘client’ | sort | uniq

mminfo -q ‘client=client-name, level=full’ -r ‘client,savetime,ssid,name,totalsize’

in a script with a variable, use double quotes so that the variable gets evaluated, and to sort on american date column…

mminfo -q client=${clientname},level=full -r ‘client,savetime,ssid,level,volume’ | sort -k 2.7,2.10n -k 2.1,2.5n -k 2.4,2.5n

mminfo -ot -c client -q “savetime>2 weeks ago”

mminfo -r “ssid,name,totalsize,savetime(16),volume” -q “client=client_name,savetime >10/01/2012,savetime <10/16/2012”

List the last full backup ssid’s for subsequent use with recover command (unix clients)

mminfo -q ‘client=server1,level=full’ -r ‘client,savetime,ssid’

Is the client configured properly in the NMC? (see diagram above  for hints on what to check in what tabs)

How many files were backed up in each saveset (useful for counting files on a NetApp which is slow using the find command at host level)

sudo mminfo -ot -q ‘client=mynetappfiler,level=full,savetime<7 days ago’ -r ‘name,nfiles’

name                         nfiles

/my_big_volume          894084

You should probably make use of the ssflags option in the mminfo report too, which adds an extra column regarding the status of the saveset displaying one or more of the following characters CvrENiRPKIFk with the common fields shown in bold below along with their meanings.

C Continued, v valid, r purged, E eligible for recycling, N NDMP generated, i incomplete, R raw, P snapshot, K cover, I in progress, F finished, k checkpoint restart enabled.

Check Client Index

nsrck -L7 clientname

Backing up Virtual Machines using Networker,VCentre and VADP

To back up virtual machine disk files on vmfs volumes at the vmware level (as opposed to the individual file level backups of the individual vm’s), networker can interface with the vcenter servers to discover what vm’s reside on the esxi clusters managed by them, and their locations on the vmfs shared lun.  For this to work, the shared lun’s also need to be presented/visible to the VADP Proxy (Windows server with Networker client and/or Server running as a storage node) in the fc switch fabric zone config.

The communication occurs as shown in blue.  i.e.

The backup server starts backup group containing vadp clients.

The vadp proxy asks vcentre what physical esxi host has the vm, and where the files reside on the shared storage luns.

The vadp proxy / networker storage node then tells the esxi host to maintain a snapshot of the vm while the vmdk files are locked for backup.

the vmdk files are written to the storage device (in my example, a data domain dedup device)

when the backup is complete, the client index is updated on the backup server, and the changes logged by the snapshot are applied to the now unlocked vmdk and then the snapshot is deleted on the esxi host.

Configuring Networker for VADP Backups via a VADP Proxy Storage Node

The VADP Proxy is just a storage node with fibre connectivity to the SAN and access to the ESXi DataStore LUNs.

In Networker, right click Virtualisation, Enable Auto Discovery

VADP-enable

Complete the fields, but notice there is an Advanced tab.  This is to be completed as follows…  not necessarily like you’d expect…

vadp-advanced

Note that the Command Host is the name of the VADP Proxy, NOT the name of the Virtual Center Server.

Finally, Run Auto Discovery.  A map of the infrastructure should build in the Networker GUI

vadp-gui

Ensure vc, proxy and networker servers all have network comms and can resolve each others names.

You should now be ready to configure a VADP client.

Configuring a VADP client (Checklist)

GENERAL TAB

vadp-client-general

IDENTITY
COMMENT
application_name – VADP
VIRTUALIZATION
VIRTUAL CLIENT
(TICK)
PHYSICAL HOST
client_name
BACKUP
DIRECTIVE
VCB DIRECTIVE
SAVE SET
*FULL*
SCHEDULE
Daily Full

APPS AND MODULES TAB

vadp-client-appsmods

 

 

 

 

 

 

BACKUP
BACKUP COMMAND
nsrvadp_save -D9
APPLICATION INFORMATION
VADP_HYPERVISOR=fqdn_of_vcenter (hostname in caps)
VADP_VM_NAME=hostname_of_vm (in caps)
VADP_TRANSPORT_MODE=san
DEDUPLICATION
Data Domain Backup
PROXY BACKUP
VMWare
hostname_of_vadp_proxy:hostname_of_vcenter.fqdn(VADP)

GLOBALS 1 OF 2 TAB
ALIASES
hostname
        hostname.fqdn
        hostname_backup
        hostname_backup.fqdn
        ip_front
        ip_back

GLOBALS 2 OF 2 TAB
REMOTE ACCESS
user=svc_vvadpb,host=hostname_vadp_proxy
        user=SYSTEM,host=hostname_vadp_proxy
        *@*

OWNER NOTIFICATION
  /bin/mail -s “client completion : hostname_client” nwmonmail

Recovery using recover on the backup client

sudo recover -s backup_server_backup_interface_name

Once in recover, you can cd into any directory irrespective of permissions on the file system.

Redirected Client Recovery using the command line of the backup server.

Initiate the recover program on the backup server…
sudo recover -s busvr_interface -c client_name -iR -R client_name

or use…  -iN (No Overwrite / Discard)
-iY (Overwrite)

-iR (Rename ~ )

Using recover> console

Navigate around the index of recoverable files just like a UNIX filesystem

Recover>    ls    pwd cd\

Change Browsetime
Recover>    changetime yesterday
1 Nov 2012 11:30:00 PM GMT

Show versions of a folder or filename backed up
Recover>      versions     (defaults to current folder)
Recover>    versions myfile

Add a file to be recovered to the “list” of files to be recovered
Recover>    add
Recover>     add myfile

List the marked files in the “list” to be recovered
Recover>    list

Show the names of the volumes where the data resides
Recover>    volumes

Relocate recovered data to another folder
Recover>    relocate /nsr/tmp/myrecoveredfiles

Recover>  relocate “E:\\Recovered_Files”     (for Redirected Windows Client Recovery from Linux Svr)

View the folder where the recovered files will be recovered to
Recover>    destination

Start Recovery
Recover>    recover

SQL Server Recovery (database copy) on a SQL Cluster

First, rdc to cluster name and run command prompt as admin on cluster name (not cluster node)
nsrsqlrc -s <bkp-server-name> -d MSSQL:CopyOfMyDatabase -A <sql cluster name> -C MyDatabase_Data=R:\MSSQL10_50.MSSQLSERvER\MSSQL\Data\CopyOfMyDatabase.mdf,MyDatabase_log=R:\MSSQL_10_50\MSSQLSERVER\MSSQL\Data\CopyOfMyDatabase.ldf MSSQL:MyDatabase

Delete the NSR Peer Information of the NetWorker Server on the client/storage node.

Please follow the steps given below to delete the NSR peer information on NetWorker Server and on the Client.

1. At NetWorker server command line, go to the location /nsr/res

2. Type the command:

nsradmin -p nsrexec
print type:nsr peer information; name:client_name
delete
y

Delete the NSR Peer Information for the client/storage node from the NetWorker Server.

Specify the name of the client/storage node in the place of client_name.

1. At the client/storage node command line, go to the location /nsr/res

2. Type the command:

nsradmin -p nsrexec
print type:nsr peer information
delete

y

VADP Recovery using command line

Prereqs to a successful VADP restore are that the virtual machine be removed from the Inventory in VCenter (right click vm, remove from Inventory), and the folder containing the virtual machines files in the vmware datastore be renamed or removed. If the vm still exists in vmware or in the datastore, VADP will not recover it.

Log onto the backup server over ssh and obtain the save set ID for your VADP “FULLVM” backup.

mminfo –avot –q “name=FULLVM,level=full”

Make a note of the SSID for the vm/backup client (or copy it to the cut/paste buffer)

e.g. 1021210946

Log onto the VADP Proxy (which has SAN connectivity over fibre necessary to recover the files back to the datastore using the san VADP recover mode)

recover.exe –S 1021210946 –o VADP:host=VC_Svr;VADP:transmode=san

Note that if you want to recover a VM back to a different vCenter,Datastore,ESX host and/or different resource pool, you can do that from the recover command too, rather than waiting to do it using the vsphere client.  this can be used if your vm still exists in vmware and you don’t want to overwrite it.  You can additionally specify VADP:host=  VADP:datacenter=  VADP:resourcepool=  VADP:hostsystem= and VADP:datastore= fields in the recover command, separated by semicolons and no spaces.

I’ve found that whilst the minimal command above may work on some environments, others demand a far more detailed recover.exe command with all VADP parameters set before it’ll communicate with the VC.  A working example is shown below (with each VADP parameter separated on a newline for readability – you’ll need to put it into a single line, and remove any spaces between each .

recover.exe -S 131958294 -o

VADP:host=vc.fqdn;

VADP:transmode=san;

VADP:datacenter=vmware-datacenter-name;

VADP:hostsystem=esxihost.fqdn;

VADP:displayname=VM_DISPLAYNAME;

VADP:datastore=“config=VM_DataStore#Hard disk 2=VM_DataStore_LUN_Name#Hard disk 1=VM_DataStore_LUN_Name”;

VADP:user=mydomain\vadp_user;

VADP:password=vadp_password

Creating new DataDomain Devices in Networker

In Networker Administrator App from NMC Console, Click Devices button at the top.
Right click Devices in the Left hand pane, New Device Wizard (shown)

Select Data Domain, Next, Next

 Use an existing data domain system
Choose a data domain system in the same physical location to your backup server!
Enter the Data Domain OST username and password

Browse and Select
Create a New Folder in sequence, e.g. D25, tick it.

Highlight the automatically generated Device Name, Copy to clipboard (CTRL-C), Next

Untick Configure Media Pools (label device afterwards using Paste from previous step), Next

Select Storage Node to correspond with device locality from “Use an existing storage node”, Next

Agree to the default SNMP info (unless reconfiguration for custom monitoring environment is required), Next

Configure, Finish

Select new device (unlabelled, Volume name blank), right click, Label

Paste Device Name in clipboard buffer (CTRL-V)
Select Pool to add the Device into, OK.

 

 

 

 

Slow backups of large amounts of data to DataDomain deduplication device

If you have ridiculously slow backups of large amounts of data, check in Networker NMC to see the name of the storage node (Globals2 tab of the client configuration), then connect to the DataDomain and look under the Data Management, DD Boost screen for “Clients” of which your storage node will be one.  Check how many CPU’s and Memory it has.  e.g. Guess which one is the slow one (below)

 

 

 

 

 

Then SSH to the storage node and check what processes are consuming the most CPU and Memory (below)

 

 

 

 

 

 

 

In this example (above), despite dedicating a storage node backup a single large applications data, the fact that it only has 4 cpu’s and is scanning every file that ddboost is attempting to deduplicate means that a huge bottleneck is introduced.  This is a typical situation whereby decommissioned equipment has been re-purposed.

Networker Server

ssh to the networker server and issue the nsrwatch command.  It’s a command line equivalent to connecting to the Enterprise app in the NMC and looking at the monitoring screen.  Useful if you can’t connect to the NMC.

 

 

 

 

 

 

Blank / Empty Monitoring Console

If you’re NMC is displaying a blank monitoring console, try this before restarting the NMC…

Tick or Un-tick and Re-tick Archive Requests.

monitoring-refresh

 

 

 

 

 

 

 

 

 

 

Tape Jukebox Operations

ps -ef | grep nsrjb     -Maybe necessary to kill off any pending nsrjb processes before new ones will work.

nsrjb -C | grep <volume>    -Identify the slot that contains the tape (volume)

nsrjb -w -S <slot>      -Withdraw the tape in slot <slot>

nsrjb -d       -Deposit all tapes in the cap/load port into empty slots in the jukebox/library.

Note:  If you are removing and replacing tapes you should take note what pools the removed tapes belong it and allocate new blank tapes deposited into the library to the same pools to eliminate impact on backups running out of tapes.

Exchange Backups

The application options of the backup client (exchange server in DAG1 would be as follows

NSR_SNAP_TYPE=vss

NSR_ALT_PATH=C:\temp

NSR_CHECK_JET_ERRORS=none

NSR_EXCH2010_BACKUP=passive

NSR_EXCH_CHECK=no

NSR_EXCH2010_DAG=GB-DAG1

NSR_EXCH_RETAIN_SNAPSHOTS=no

NSR_DEVICE_INTERFACE=DATA_DOMAIN

NSR_DIRECT_ACCESS=no

Adding a NAS filesystem to backup (using NDMP)

Some pre-reqs on the VNX need to be satisfied before NDMP backups will work.  This is explained here

General tab

general-tab

 

 

 

 

 

 

 

 

 

 

 

The exported fs name can be determined by logging onto the VNX as nasadmin and issuing the following command

server_mountpoint server_2 -list

Apps and Modules tab

apps_modules_tab

 

 

 

 

 

 

 

 

 

Application Options that have worked in testing NDMP Backups.

Leave datadomain unticked in Networker 8.x and ensure you’ve selected a device pool other than default, or Networker may just sit waiting for a tape while you’re wondering why NDMP backups aren’t starting!

HIST=y
UPDATE=y
DIRECT=y
DSA=y
SNAPSURE=y
#OPTIONS=NT
#NSR_DIRECT_ACCESS=NO
#NSR_DEVICE_INTERFACE=DATA_DOMAIN

Backup Command: nsrndmp_save -s backup_svr -c nas_name -M -T vbb -P storage_node_bu_interface or don’t use -P if Backup Server acts as SN.

To back up an NDMP client to a non-NDMP device, use the -M option.

The value for the NDMP backup type depends on the type of NDMP host. For example, NetApp, EMC, and Procom all support dump, so the value for the Backup Command attribute is:

nsrndmp_save -T dump

Globals 1 tab

globals1

 

 

 

 

Globals2 tab

globals2

 

 

 

 

 

 

 

 

List full paths of VNX filesystems required for configuring NDMP save client on Networker (run on VNX via SSH)

server_mount server_2

List full paths required to configure NDMP backup clients (emc VNX)

server_mount server_2

e.g. /root_vdm_2/CYBERFELLA_Test_FS

Important:  If the filesystem being backd up contains more than 5 million files, set the timeout attribute to zero in the backup group’s properties.

Command line equivalent to the NMC’s Monitoring screen

nsrwatch

Command line equivalent to the NMC’s Alerts pane

printf “show pending\nprint type:nsr\n” | /usr/sbin/nsradmin -i-

Resetting Data Domain Devices

Running this in one go if you’ve not done it before is not advised.  Break it up into individual commands (separated here by pipes) and ensure the output is what you’d expect, then re-join commands accordingly so you’re certain you’re getting the result you want.  This worked in practice though.  It will only reset Read Only (.RO) devices so it won’t kill backups, but will potentially kill recoveries or clones if they are in progress.

nsr_render_log -lacedhmpty -S “1 hour ago” /nsr/logs/daemon.raw | grep -i critical | grep RO | awk {‘print $10’} | while read eachline; do nsrmm | grep $eachline | cut -d, -f1 | awk {‘print $7’}; done | while read eachdevice; do nsrmm -HH -v -y -f “${eachdevice}”; done

Identify OS of backup clients via CLI

The NMC will tell you what the Client OS is, but it won’t elaborate and tell you what type, e.g. Solaris, not Solaris 11 or Linux, not Linux el6.  Also, as useful as the NMC is, it continually drives me mad how you cant export the information on the screen to excel.  (If someone figures this out, leave a comment below).

So, here’s how I got what I wanted using the good ol’ CLI on the backup server.  Luckily for me the backup server is Linux.
Run the following command on the NetWorker server, logging the putty terminal output to a file:

nsradmin
. type: nsr client
show client OS type
show name
show os type
p

This should get you a list of client names and what OS they’re running according to Networker in your putty.log file.  Copy and paste the list into a new file called mylist.  Extract just the Solaris hosts…

grep -i -B1 solaris >mylist
grep name mylist | cut -d: -f2 | cut -d\; -f1 >mysolarislist

sed ‘s/^ *//’ mysolarislist | grep -v \\-bkp > solarislist

You’ll now have a nice clean list of solaris networker client hostnames.  You can remove any backup interface names by using

grep -v b$

to remove all lines ending in b.

One liner…

grep -i -B1 solaris mylist | grep name | cut -d: -f2 | cut -d\; -f1 | sed ‘s/^ *//’ | grep -v \\-bkp | grep -v b$ | sort | uniq > solarislist

Now this script will use that list of hostnames to ssh to them and retrieve more OS detail with the uname -a command.  Note that if SSH keys aren’t set up, you’ll need to enter your password each time a new SSH session is established.  This isn’t as arduous as it sounds.  use PuTTY right click to paste the password each time, reducing effort to a single mouse click.
#!/bin/bash

cat solarislist | while read eachhost; do
echo “Processing ${eachhost}”
ssh -n -l cyberfella -o StrictHostKeyChecking=no ${eachhost} ‘uname -a’ >> solaris_os_ver 2>&1
done

This generates a file solaris_os_ver that you can just grep for ^SunOS and end up with a list of all the networker clients and the full details of the OS on them.

grep ^SunOS solaris_os_ver | awk ‘{print $1 $3 $2}’

Facebooktwittergoogle_plusredditpinterestlinkedinmail
Comments Off on Networker Cheatsheet
Aug 06

List UIDs of failed files

If you’re copying data from an NFS device, the local root user of your NFS client will not have omnipotent access over the data, and so if the permissions are set with everyone noaccess, i.e. r-wr-w— or similar (ending in — instead of r–) then even root will fail to copy some files.

To capture the outstanding files after the initial rsync run as root, you’ll need to determine the UID of the owner(s) of the failed files, create dummy users for those uids and perform subsequent rsync’s su’d to those dummy users.  You won’t get read access any other way.

The following shell script will take a look at the log file of failures generated by rysnc -au /src/* /dest/ 2> rsynclog and list uid’s of user accounts that have read access to the failed-to-copy data.  (Note: when using rsync, appending a * will effectively miss .hidden files.  Lose the * and use trailing slashes to capture all files including hidden files and directories).

subsequent rsync operations can be run by each of these users in turn to catch the failed data.  This requires the users to be created on the system performing the copy, e.g. useradd -o -u<UID> -g0 -d/home/dummyuser -s/bin/bash dummyuser

This could also easily be incorporated into the script of course.

#!/usr/bin/bash

#Variables Section

    SRC=”/source_dir”
    DEST=”/destination_dir”
    LOGFILE=”/tmp/rsynclog”
    RSYNCCOMMAND=”/usr/local/bin/rsync -au ${SRC}/* ${DEST} 2> ${LOGFILE}”
    FAILEDDIRLOG=”/tmp/faileddirectorieslog”
    FAILEDFILELOG=”/tmp/failedfileslog”
    UIDLISTLOG=”/tmp/uidlistlog”
    UNIQUEUIDS=”/tmp/uniqueuids”

#Code Section

    #Create a secondary list of all the failed directories
    grep -i opendir ${LOGFILE} | grep -i failed ${LOGFILE} | cut -d\” -f2 > ${FAILEDDIRLOG}

    #Create a secondary list of all the failed files
    grep -i “send_files failed” ${LOGFILE} | cut -d\” -f2 > ${FAILEDFILELOG}

    #You cannot determine the UID of the owner of a directory, but you can for a file
    
    #Remove any existing UID list log file prior to writing a new one
    if [ -f ${UIDLISTLOG} ]; then
        rm ${UIDLISTLOG}
    fi

    #Create a list of UID’s for failed file copies    
    cat ${FAILEDFILELOG} | while read EACHFILE; do
        ls -al ${EACHFILE} | awk {‘print $3’} >> ${UIDLISTLOG}
    done

    #Sort and remove duplicates from the list
    cat ${UIDLISTLOG} | sort | uniq > ${UNIQUEUIDS}    

    cat ${UNIQUEUIDS}

exit

Don’t forget to chmod +x a script before executing it on a Linux/UNIX system.

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Aug 06

Counting number of files in a Linux/UNIX filesystem

cd to the starting directory, then to count how many files and folders exist beneath,

find . -depth | wc -l

although in practice find . | wc -l works just as well leaving off -depth.  Or to just count the number of files

find . -type f | wc -l

Note that on Linux, a better way to compare source and destination directories, might be to count the inodes used by either filesystem.

df -i

Exclude a hidden directory from the file count, e.g. .snapshots directory on a NetApp filer

#find ./ -type f \( ! -name “.snapshot” -prune \) -print | wc -l – Note:  had real trouble with this!

New approach…  :o(

ls -al | grep ^d | awk {‘print $9’} | grep -v “^\.” | while read eachdirectory; do

     find ./ -depth | wc -l

done

Then add up numbers at the end.

Another way to count files in a large filesystem is to ask the backup software.  If you use emc Networker, the following example may prove useful.

sudo mminfo -ot -q ‘client=mynas,level=full,savetime<7 days ago’ -r ‘name,nfiles’

name                         nfiles

/my-large-volume          894084

Facebooktwittergoogle_plusredditpinterestlinkedinmail
comment?
Social Media Auto Publish Powered By : XYZScripts.com