List UIDs of failed files

If you’re copying data from an NFS device, the local root user of your NFS client will not have omnipotent access over the data, and so if the permissions are set with everyone noaccess, i.e. r-wr-w— or similar (ending in — instead of r–) then even root will fail to copy some files.

To capture the outstanding files after the initial rsync run as root, you’ll need to determine the UID of the owner(s) of the failed files, create dummy users for those uids and perform subsequent rsync’s su’d to those dummy users.  You won’t get read access any other way.

The following shell script will take a look at the log file of failures generated by rysnc -au /src/* /dest/ 2> rsynclog and list uid’s of user accounts that have read access to the failed-to-copy data.  (Note: when using rsync, appending a * will effectively miss .hidden files.  Lose the * and use trailing slashes to capture all files including hidden files and directories).

subsequent rsync operations can be run by each of these users in turn to catch the failed data.  This requires the users to be created on the system performing the copy, e.g. useradd -o -u<UID> -g0 -d/home/dummyuser -s/bin/bash dummyuser

This could also easily be incorporated into the script of course.

#!/usr/bin/bash

#Variables Section

    SRC=”/source_dir”
    DEST=”/destination_dir”
    LOGFILE=”/tmp/rsynclog”
    RSYNCCOMMAND=”/usr/local/bin/rsync -au ${SRC}/* ${DEST} 2> ${LOGFILE}”
    FAILEDDIRLOG=”/tmp/faileddirectorieslog”
    FAILEDFILELOG=”/tmp/failedfileslog”
    UIDLISTLOG=”/tmp/uidlistlog”
    UNIQUEUIDS=”/tmp/uniqueuids”

#Code Section

    #Create a secondary list of all the failed directories
    grep -i opendir ${LOGFILE} | grep -i failed ${LOGFILE} | cut -d\” -f2 > ${FAILEDDIRLOG}

    #Create a secondary list of all the failed files
    grep -i “send_files failed” ${LOGFILE} | cut -d\” -f2 > ${FAILEDFILELOG}

    #You cannot determine the UID of the owner of a directory, but you can for a file
    
    #Remove any existing UID list log file prior to writing a new one
    if [ -f ${UIDLISTLOG} ]; then
        rm ${UIDLISTLOG}
    fi

    #Create a list of UID’s for failed file copies    
    cat ${FAILEDFILELOG} | while read EACHFILE; do
        ls -al ${EACHFILE} | awk {‘print $3’} >> ${UIDLISTLOG}
    done

    #Sort and remove duplicates from the list
    cat ${UIDLISTLOG} | sort | uniq > ${UNIQUEUIDS}    

    cat ${UNIQUEUIDS}

exit

Don’t forget to chmod +x a script before executing it on a Linux/UNIX system.

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash:

Counting number of files in a Linux/UNIX filesystem

cd to the starting directory, then to count how many files and folders exist beneath,

find . -depth | wc -l

although in practice find . | wc -l works just as well leaving off -depth.  Or to just count the number of files

find . -type f | wc -l

Note that on Linux, a better way to compare source and destination directories, might be to count the inodes used by either filesystem.

df -i

Exclude a hidden directory from the file count, e.g. .snapshots directory on a NetApp filer

#find ./ -type f \( ! -name “.snapshot” -prune \) -print | wc -l – Note:  had real trouble with this!

New approach…  :o(

ls -al | grep ^d | awk {‘print $9’} | grep -v “^\.” | while read eachdirectory; do

     find ./ -depth | wc -l

done

Then add up numbers at the end.

Another way to count files in a large filesystem is to ask the backup software.  If you use emc Networker, the following example may prove useful.

sudo mminfo -ot -q ‘client=mynas,level=full,savetime<7 days ago’ -r ‘name,nfiles’

name                         nfiles

/my-large-volume          894084

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash:

Copying the contents of one filesystem to another.

Sometimes on older operating systems, rsync (first choice for copying files from one filesystem to another) may not be available.  In such circumstances, you can use tar.  If it’s an initial copy of a large amount of data you’re doing, then this may actually be 2 – 4 times faster due to the lack of rsync’s checksum calculations, although rsync would be faster for subsequent delta copies.

timex tar -cf – /src_dir | ( cd /dest_dir ; tar -xpf – )

Add a v to the tar -xpf command if you want to see a scrolling list of files as the files are copied but be aware that this will slow it down.  I prefer to leave it out and just periodically ls -al /dest_dir in another terminal to check the files are being written correctly.  timex at the front of the command will show you how long it ran for once it completes (may be useful to know).

With the lack of verbose output, if you need confirmation that the command is still running, use ps -fu user_name | grep timex although the originating terminal should not have returned a command prompt unless you backgrounded the process with an & upon execution, or CTRL Z, jobs, bg job_id subsequently. Note that backgrounding the process may hinder your collection of timings so is not recommended if you are timing the operation.

Another alternative would be to pipe the contents of find . -depth into cpio -p thus using cpio’s passthru mode…

timex find . -depth | cpio -pamVd /destination_dir

Note that this command can appear to take a little while to start, before printing a single dot to the screen per file copied (the capital V verbose option as opposed to the lowercase v option)

If you wish to copy data from one block storage device to another, it’d be faster to do it at block level rather than file level.  To do this, ensure the filesystems are unmounted, then use the dd command dd if=/dev/src_device of=/dev/dest_device

Do not use dd on mounted filesystems.  You will corrupt the data.

Overall progress can be monitored throughout the long copy process with df -h in a separate command windowprepending the cpio command with timex will not yield any times once the command has completed – but it is faster than both tar or rsync for initial large copies of data.

To perform a subsequent catch-up copy of new or changed files, simultaneously deleting any files from the Destination that no longer exist on the Source for a true “syncronisation” of the two sides, much like a mirror synchronisation, use…

timex ./rsync -qazu –delete /src_dir/* /dest_dir  

Note this will not include hidden files.  To do that, lose the * off the source fs and add a trailing slash to the destination fs

or to catch up the new contents on the Src side to the Dest side and not delete any files on the Dest side that have been deleted on Src, use

rsync -azu –progress /NFS_Src/* /NFS_Dest

a= archive mode; equals –rlptgoD (recursive, links, permissions, times, group, owner and device files preserved)

z = compress file during transfer (optional but generally best practice)

u = update

–progress in place of v (verbose) or q (quiet).  A touch faster and more meaningful than a scrolling list of files going up the screen.

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash:

Booting an in-band VirtualCentre Server VM from the ESXi console

If your VirtualCentre server is itself a VM, then it’ll be running on an ESXi host.  In the event that the ESXi host is restarted without vMotioning the VirtualCenter Server first (such as when the management network is irrecoveraby unresponsive), then depending on your environment, you may not be able to get a remote connection to the vm after the host has restarted.  In this scenario, you’d need to be able to boot the VM from the unsupported console.  This is how to do it.

Connect to the iLo or equivalent management interface to the ESX host, send an Alt-F1 and type unsupported followed by the root password to obtain a prompt on the unsupported console.

Identify the VM’s resident on the host

vim-cmd vmsvc/getallvms

Identify the current power state of the vm running virtual centre

vim-cmd vmsvc/power.getstate ##           where ## is the number of the vm identified above

Power on the vm

vim-cmd vmsvc/power.on ##                       where ## is the number of the vm identified above

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash:

Troubleshooting Openfiler (missing NFS shares)

I came home on Friday evening to find my DLNA server wasn’t available :(.  It’s not the scenario I needed after an intense few days squeezing 5 days worth of work into a 4 day week due to the Easter bank holiday weekend, plus the 3 hour drive home.

Firstly, my DLNA server is simply Serviio running on a Xubuntu VM which mounts an NFS share containing my media files.

The virtual infrastructure in my lab that underpins it is a two node ESXi cluster with a third node running Openfiler to provide the shared storage to ESXi.  This includes a RAID 0 (not recommended I might add) iSCSI target for maximum IO within a constrained home budget and a 1TB USB HDD containing a NFS Datastore where I store my ISO’s and vm backups so as to save space on the relatively expensive, high performance iSCSI target intended for the VM’s disk files, which are also thinly provisioned to further save on space.  The Openfiler NAS also has a second 1TB USB HDD containing a second NFS Media Store share, mounted by Serviio/Xubuntu VM already mentioned (as well as any other machine in the network). The network is an 8 port, 1 GB/s managed switch with two VLANs and two Networks, one which joins the rest of the LAN, and one which just contains VMotion and iSCSI traffic.

 

So, like I said, my Serviio DLNA server was u/a and some troubleshooting was in order.

My first reaction was that something was wrong in VMWare Land, but this turned out not to be the case – however, the storage configuration tab revealed that the NFS datastores were not available, and df -h on my workstation confirmed it, so almost immediately my attention switched from VMWare to Openfiler.

Now, I won’t go into it too much here, but I’m torn with Openfiler.  The trouble is most folks would only ever interface with the web-based GUI, and they’d quickly come unstuck, since conary updateall to install all the latest updates or not, certain changes don’t seem to get written back.  I had to perform all my LVM configuration manually at the command line as root, not via the web-gui as openfiler.  I’ve yet to investigate this any further as it’s now working OK for me, but my guess would be a permissions issue.

I connected to the Openfiler web interface and could see that the shared folders (shown below) were missing, so the NFS shares were not being shared but more importantly it also implied that the logical volumes containing the filesystems exported via NFS were not mounted.  df -h on Openfiler’s command line interface confirmed this.

In order to check that Openfiler could see the hard drives at all, I issued the command fdisk -l but because the USB HDD’s are LVM physical volumes, they have gpt partition tables on them, not msdos, so fdisk does not support it, but is kind enough to recommend using GNU Parted instead.  Despite the recommendation, I used lshw > /tmp/allhardware and just used vi to go looking for the hard drive information.  The USB HDD’s are Western Digital, so I just :/WD to find them amongst the reams of hardware information, and find them I did.  Great, so the OS could see the disks, but they weren’t mounted.  I quickly checked /etc/fstab and sure enough, the devices were in there, but mount -a wasn’t fixing the problem.

Remember I mentioned that the drives had a gpt partition table, and that they were LVM physical volumes?  Well therein lies the problem.  You can’t mount a filesystem on a logical volume if the volume group that it is a part of is not activated.  Had my volume groups deactivated?  Yes, they had.

vgchange -ay /dev/vg_nfs

vgchange -ay /dev/vg_vmware

Now my volume groups were active, mount -a should work, confirmed by df -h showing that the /dev/mapper/vg_vmware-lv_vmware and /dev/mapper/vg_nfs-lv_nfs block storage devices were now mounted into /mnt/vg_vmware/lv_vmware and /mnt/vg_nfs/lv_nfs respectively.  exportfs -a should reshare the NFS shares provided the details were still in /etc/exports which they were.  Going back to the Openfiler web-interface, the shares tab now revealed the folders shown in blue (above) and their mount points needed by any NFS clients in order to mount them.  Since the mountpoint details were already in /etc/fstab on my workstation, mount -a re-mounted them and into /nfs/nfsds and /nfs/nfsms and ls -al showed that the files were all there.

rdesktop to my VirtualCenter server, mount -a in the Xubuntu terminal to remount them on the DLNA server, re-run serviio.sh and that’s it.

So that’s how I diagnosed what was wrong and how I fixed it.  Now I just need to investigate the system logs on Openfiler to see why the volume groups deactivated in the first place.  After continuous uptime without issue for 4 months, I must admit that it did come as a surprise.

 

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash:

Disk Recovery and Forensics

Who doesn’t love the word “Forensics”?  It’s a word that brings out the inner geek in all of us, yet the reality is usually pretty grim – like when your only hard drive containing all your important files and photos fails.

The first thing you should do if you suspect your hard drive is failing or has failed is not attempt to write to it and if necessary hard shut the machine down asap by pushing and holding the power button on your PC.  Any further writes could lunch the drive for good making recovery impossible.  In otherwords STOPPP!!

Anyway, here’s some notes from recent tinkerings with Ubuntu Rescue Remix (Google it, Download it).  It’s a bootable Live CD which boots a computer into a command line only Linux environment, and for the remaining 2% who are still reading, provides you with a good handful of tools that stand you the best chance of recovering data from a failing hard disk.

Assuming you’ve just booted it and your hard disk(s) are attached, the first thing to do is identify which disk corresponds to which device name in /dev.  This can be done using lshw or fdisk -l

lshw > /tmp/hardware

cat /tmp/hardware | less

The next step is to clone the dodgy disk to either another disk, or to an image file or both.  You choose.

ddrescue /dev/sda /dev/sdc

or (restartable clone to an image file)

ddrescue –direct –retrim –max-retries=3 /dev/sda imagefile logfile

If you’ve cloned to another healthy disk, then you should fsck /dev/sdc to fix any errors, then attempt to mount it with mount /dev/sdc1 /mnt/mydisk and see if you can read any data on it.  You may be as good as done at this point with no further need to go on to employing other more targeted tools for recovering data off an unmountable drive.  Failing that, try to stay calm (really – it helps), clone the disk to an imagefile the best you can, then read on.  If you can’t stay calm, then run testdisk and benefit from a more intuitive menu driven interface of various recovery options.

testdisk

Or if you’re enjoying this new found challenge of getting the photos back before the missus finds out, read on about using foremost and other similar, powerful recovery commands.

sudo foremost -i imagefile -o /recovery/foremost -w       (list recoverable files only)

sudo foremost -i imagefile -o /recovery/foremost -t jpg           (recover jpg files only)

If you suspect that the partitioning information on the drive is gone, then you can replace it using gpart to guess what the previous partitioning scheme was based upon whats on the drive.  This is good if you’re an overzealous techy who blanked the drive to install the latest OS without thinking about who else had an account on the computer and what they may have had stored.  Not good.  Don’t do it again.

sudo gpart /dev/sda

Or instead of using foremost, you could try scalpel.  Like foremost, but configurable and well, a bit better.

vi /etc/scalpel/scalpel.conf     (to configure options)

sudo scalpel imagefile -o /recovery/scalpel/

Or maybe try magicrescue on the cloned disk if there’s multiple file types to be recovered (requires the presence of recipes for the filetypes to be recovered).

/usr/share/magicrescue/recipes

Enable DMA on the cloned disk first to speed things up.

hdparm -d 1 -c 1 -u 1 /dev/hdc

sudo magicrescue  -r gzip -r png  -d /recovery/magicrescue /dev/sdc

If it’s specificly photos you’re wanting to recover, then there are two tools to choose from; photorec and recoverjpeg.

sudo photorec imagefile         (imagefile is the disk imagefile, not an image as in picture)

sudo recoverjpeg /dev/sdc1       (recovers any obvious jpeg files on partition /dev/sdc1)

If the files you want to recover were deleted on the original drive, then assuming the drive has come from a windows computer and was formatted with NTFS, then you can use ntfsundelete to recover the deleted files.

ntfsundelete -s /dev/sdc1     (scans for inodes of deleted files which can be subsequently recovered)

ntfsundelete /dev/sdc1 -u -i 3689 -o work.doc -d /recovered/ntfsundelete

If you want to recover old files previously written to a disk containing a new FAT filesystem, then you’re into using autopsy and dls, fls, icat and sorter from sleuthkit to create a secondary image of unallocated blocks contained in the image and list the inodes of files apparently contained within them, recover those files and optionally sort them by filetype, respectively.

sudo autopsy -d /media/disk/autopsy 192.168.0.1      (use your local ip address)

dls imagefile > imagefile_deletedblocks        (create secondary, smaller imagefile)

fls imagefile_deletedblocks -r -f fat -i raw      (list inode numbers of any deleted files found)

icat -r -f fat -i raw imagefile_deletedblocks inode_number > myfile.doc    (recover a file)

sudo sorter -h -s -i raw -f fat -d out -C /usr/share/sleuthkit/windows.sort /imagefile

This just touches upon ways you can recover lost data, with a few useful examples, but remember each command in it’s own right has a multitude of options which can be perused using the man command and reading the accompanying manual.  You can also google man sorter for example, and read the man page in a web browser.  I hope you get some data back!

Did you like this?
Tip cyberfella with Cryptocurrency

Donate Bitcoin to cyberfella

Scan to Donate Bitcoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some bitcoin:

Donate Bitcoin Cash to cyberfella

Scan to Donate Bitcoin Cash to cyberfella
Scan the QR code or copy the address below into your wallet to send bitcoin:

Donate Ethereum to cyberfella

Scan to Donate Ethereum to cyberfella
Scan the QR code or copy the address below into your wallet to send some Ether:

Donate Litecoin to cyberfella

Scan to Donate Litecoin to cyberfella
Scan the QR code or copy the address below into your wallet to send some Litecoin:

Donate Monero to cyberfella

Scan to Donate Monero to cyberfella
Scan the QR code or copy the address below into your wallet to send some Monero:

Donate ZCash to cyberfella

Scan to Donate ZCash to cyberfella
Scan the QR code or copy the address below into your wallet to send some ZCash: