Make bootable USB from .iso in Linux

The following command will write a downloaded .iso file of your favourite distro to a USB stick.  You can then boot off it and install to hardware.

sudo dd bs=4M if=./manjaro-xfce-18.0-stable-x86_64.iso of=/dev/sdb status=progress

Note that in my example, there was no partition on the usb stick to start with.  I’d removed it using gparted (not necessary though).

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Conky

One of my first ever posts was about conky and wbar on crunchbang linux.

Crunchbang has since been replaced with a community led fork, Bunsenlabs, and it’s well worth checking out.  I’m so impressed with it that it’s my laptop OS of choice, giving me very little grief installing onto my disappointingly-not-particularly-linux-friendly Dell XPS 15, unlike other popular distros.  Suffice to say, Bunsenlabs has saved my XPS15 from the financial damage limitation exercise known as ebay.

In any case, I thought I’d include a link to my own .conkyrc file.  It’s simple and neat, nothing too fancy.

The download file is called conkyrc.  Once downloaded, just rename it to .conkyrc i.e. put the dot in front (hidden file and the conky default), and copy it to your home directory, remembering to back up any existing .conkyrc file already in your home directory first.

If you want to edit yours to make it your own, the man page for conky is very good, but I find this better.

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Linux disk space consumption analysis.

Desktop distro’s have wonderful graphical disk space analysis programs such as Baobab (KDirStat), QDirStat, xdiskusage, duc, JDiskReport and with your desktop distro being connected to the internet, even if you dont already have them installed, installing them from your repositories is easy.   You can quickly drill down using these treemapper programs and find the culprit for filling your disk up.

In the datacentre, things are never so easy.  You have no internet access, and no local repository configured, and even if you did, you have no change control to install it on a live system, and even if you did, no GUI to view it. All you have is a production problem, a stressed out ops manager and a flashing cursor winking at you -oh and native tools.

Sure, you can use the find command to go looking for files over a certain size,

find ./ -type f -size +1000000M -exec ls -al {} \;

removing a zero and re-running as required until it starts finding something, but you’ll fight with the find command syntax for 15 minutes trying to get it to work, only to be unconvinced of the results.  As good as find is, it’s not exactly easy trying to put together a command that does something that should be simple.

Here is a much simpler solution.  Just use du.  In particular…

du -h –max-depth=1

This will summarize the size of the top level sub-directories underneath your present working directory.  You then cd in to the biggest one, run it again and repeat until you basically end up digging down and arriving at the largest file on disk – in my case a 32GB mysql database in /var/lib/mysql/zabbix.

So there you go.  Have a play with it and you’ll see what I mean.  It’s my favourite way of finding out what’s eating all my disk space.

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Stopping and Disabling Services in Linux

Things are a little different between Centos/RHEL6 and Centos/RHEL7 when it comes to starting and stopping services.

Having grown up on /etc/init.d/ scripts, it’s enough of a challenge using service blah stop instead of /etc/init.d/blah stop, but I guess service blah stop was added to Centos/RHEL6 to simplify things.

And then systemd comes alone they go and change it.  Way to go in terms of keeping things simple – even though it does bring some consistency across redhat and debian based distros going forward tbf…

Now it feels like every time I try to do something as simple as start and stop a service on a redhat based distro, Sod’s Law kicks in and I always get the command wrong, making me feel like a total noob, despite having rocked the command line for over 20 years.

As you can probably gather, I’m not a fan of the landscape changing (which is what drove me away from Windowz and into Linux in the first place – the longevity of the marketable skills set was better).  In my defence, Linus Torvalds isn’t that happy about it either.

Hence this little post.  A quick reminder on which command to use.  Now I’ve written it, I won’t need it of course.  Funny how the brain works, eh?

RHEL/CENTOS 6
chkconfig | grep zabbix – lists all services in all runlevels
chkconfig zabbix-agent off – toggle it on/off at startup
service zabbix-agent stop   -stop the service

RHEL/CENTOS 7
chkconfig
systemctl status zabbix-agent.service
systemctl disable zabbix-agent
systemctl stop zabbix-agent.service

 

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Retro Terminal

This is just too good.

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Firefox Bookmarks location (Linux)

At some point you may find yourself wishing you could get your firefox boomarks from another user home directory on your laptop or on backup.

I want to be able to find them, and restore them into Firefox in my new profile.

This is the location in the filesystem where the Firefox bookmarks are kept.  Firefox appears to retain backups of the bookmarks on its own, which is very convenient indeed.

/home/matt/.mozilla/firefox/n5ufmg0e.default/bookmarkbackups

Simply copy the most recent file to the same location in your new profile.  The filename is fairly cryptic looking, e.g.

‘bookmarks-2018-09-13_15_AWyxn1GRB8WngmWVnMswTg==.jsonlz4’

so my advice would be to not change it at all, just copy it as it is into the same location in your new profile.

You’ll need to restart firefox before you’ll see it as a recoverable item in  the Firefox Bookmarks, Show All Bookmarks, Backup and Restore dialog.

In newer versions of Firefox, the menu bar and bookmarks toolbar are missing by default (I hate all this modern minimalism as it smacks of form over function) but you can enable it again by right clicking on a blank bit of GUI and selecting Menu bar and Bookmarks bar (Highly recommended).

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Firefox Bookmarks location (Windows)

At some point you may find yourself wishing you could get your firefox boomarks from an old profile still resident on your laptop, but no longer the one that you’re logging into.

For example, I have an old profile here for user matthewbradley, but upon getting my account reactivated, I’m now finding myself logging into matthewbradley.UK – i.e. same human, different local profile on the laptop.

The frustrating thing is that all my bookmarks are in Firefox in my old profile and I want to be able to back them up and restore them into Firefox in my new profile.

This is the location in the filesystem where the Firefox bookmarks are kept.  Firefox appears to retain backups of the bookmarks on its own, which is very convenient indeed.  You should be able to access it via Explorer or Cmd console.

C:\Users\matthewbradley\AppData\Roaming\Mozilla\Firefox\Profiles\jrc8wnze.default\bookmarkbackups\

Note that AppData is a hidden folder so if you’re using Explorer, you’ll need to change the View options to show Hidden Items before you’ll see it.

Simply copy the most recent file to the same location in your new profile.  The filename is fairly cryptic looking, e.g.

bookmarks-2018-09-13_15_AWyxn1GRB8WngmWVnMswTg==.jsonlz4

so my advice would be to not change it at all, just copy it as it is, into your new profile.

You’ll need to restart firefox before you’ll see it as a recoverable item in  the Firefox Bookmarks, Show All Bookmarks, Backup and Restore dialog.

In newer versions of Firefox, the menu bar and bookmarks toolbar are missing by default (I hate all this modern minimalism as it smacks of form over function) but you can enable it again by right clicking on a blank bit of GUI and selecting Menu bar and Bookmarks bar (Highly recommended).

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Simplify linux find command using shell functions

Do you need to find a file and then perform some action on it and get caught up in curly brackets, back slashes and syntax errors when you could swear “this command worked in the past?”.  It’s one of the joys of Linux I guess, but quickly becomes tedious when you’re working against a problem and are under stress.

Here is a reference find command that works.  I hope it helps.  It’ll no doubt help me at some point (the entire purpose of my blog is to actually remind myself how to do half of this stuff from time to time).

sudo find ./ -name *.mkv -exec ls {} \;

Something I like to do is create shell functions in the .bashrc file in your home directory to simplify commonly used commands that are long to type and quite syntax sensitive.

#SHELL FUNCTIONS FOR .bashrc
f() { find . -name “*$1*”; }

This is a nice useful one that can be used to find any files that have the specified string anywhere in the filename.  Just type f All  to find any files with the word All occurring anywhere in the filename.

You could create other versions such as this one, that will find and remove files with a specified string in the filename – but I’d really not recommend it.

fr() { find ./ -name “*$1*” -exec rm {} \; }

Be sure to run man fr first to check that your shell function name isn’t the name of an existing binary on the system!

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Dropbox alternative for Linux users

With the recent announcement that Dropbox is dropping its support for linux filesystems (other than ext4) in November, you’ll no doubt be searching for an alternative cloud storage provider that supports linux file system synchronisation.

Look no further than MEGA.

50GB for free, local filesystem synchronisation, download and retain your own private key,  a great, easy to use web browser client.

File system sync client: https://mega.nz/sync

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail

Bare metal DR and Linux Migration with Relax and Recover (rear)

INTRODUCTION

In short, Relax and Recover (rear) is a tool that creates .tar.gz images of the running server and creates bootable rescue media as .iso images

Relax and Recover (ReaR) generates an appropriate rescue image from a running system and also acts as a migration tool for Physical to Virtual or Virtual to Virtual migrations of running linux hosts.

It is not per se, a file level backup tool.  It is akin to Clonezilla – another popular bare metal backup tool also used to migrate Linux into virtual environments.

There are two main commands, rear mkbackuponly and rear mkrescue to create the backup archive and the bootable image respectively.  They can be combined in the single command rear mkbackup.

The bootable iso provides a configured bootable rescue environment that, provided your backup is configured correctly in /etc/rear/local.conf, will make recovery as simple as typing rear recover from the recovery prompt.

You can back up to NFS or CIFS Share or to a USB block storage device pre-formatted by running rear format /dev/sdX

A LITTLE MORE DETAIL

A professional recovery system is much more than a simple backup tool.
Experienced admins know they must control and test the entire workflow for the recovery process in advance, so they are certain all the pieces will fall into place in case of an emergency.
Versatile replacement hardware must be readily available, and you might not have the luxury of using a replacement system that exactly matches the original.
The partition layout or the configuration of a RAID system must correspond.
If the crashed system’s patch level was not up to date, or if the system contained an abundance of manually installed software, problems are likely to occur with drivers, configuration settings, and other compatibility issues.
Relax and Recover (ReaR) is a true disaster recovery solution that creates recovery media from a running Linux system.
If a hardware component fails, an administrator can boot the standby system with the ReaR rescue media and put the system back to its previous state.
ReaR preserves the partitioning and formatting of the hard disk, the restoration of all data, and the boot loader configuration.

ReaR is well suited as a migration tool, because the restoration does not have to take place on the same hardware as the original.
ReaR builds the rescue medium with all existing drivers, and the restored system adjusts automatically to the changed hardware.
ReaR even detects changed network cards, as well as different storage scenarios with their respective drivers (migrating IDE to SATA or SATA to CCISS) and modified disk layouts.
The ReaR documentation provides a number of mapping files and examples.
An initial full backup of the protected system is the foundation.
ReaR works in collaboration with many backup solutions, including Bacula/Bareos SEP SESAM, Tivoli Storage Manager, HP Data Protector, Symantec NetBackup, CommVault Galaxy, and EMC Legato/Networker.

WORKING EXAMPLE

Below is a working example of rear in action, performed on fresh Centos VM’s running on VirtualBox in my own lab environment.

Note: This example uses a Centos 7 server and a NFS Server on the same network subnet.

INSTALLATION
Add EPEL repository
yum install wget
wget http://dl.fedoraproject.org/pub/epel/beta/7/x86_64/epel-release-7-0.2.noarch.rpm
rpm -ivh epel-release-7-0.2.noarch.rpm
yum install rear

START A BACKUP
On the CentOS machine
Add the following lines to /etc/rear/local.conf:
OUTPUT=iso
BACKUP=NETFS
BACKUP_TYPE=incremental
BACKUP_PROG=tar
FULLBACKUPDAY=”Mon”
BACKUP_URL=”nfs://NFSSERVER/path/to/nfs/export/servername”
BACKUP_PROG_COMPRESS_OPTIONS=”–gzip”
BACKUP_PROG_COMPRESS_SUFFIX=”.gz”
BACKUP_PROG_EXCLUDE=( ‘/tmp/*’ ‘/dev/shm/*’ )
BACKUP_OPTIONS=”nfsvers=3,nolock”

Now make a backup
[root@centos7 ~]# rear mkbackup -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
mkdir: created directory ‘/var/lib/rear/output’
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
Copying resulting files to nfs location
Encrypting disabled
Creating tar archive ‘/tmp/rear.QnDt1Ehk25Vqurp/outputfs/centos7/2014-08-21-1548-F.tar.gz’
Archived 406 MiB [avg 3753 KiB/sec]OK
Archived 406 MiB in 112 seconds [avg 3720 KiB/sec]

Now look on your NFS server
You’ll see all the files you’ll need to perform the disaster recovery.
total 499M
drwxr-x— 2 root root 4.0K Aug 21 23:51 .
drwxr-xr-x 3 root root 4.0K Aug 21 23:48 ..
-rw——- 1 root root 407M Aug 21 23:51 2014-08-21-1548-F.tar.gz
-rw——- 1 root root 2.2M Aug 21 23:51 backup.log
-rw——- 1 root root 202 Aug 21 23:49 README
-rw——- 1 root root 90M Aug 21 23:49 rear-centos7.iso
-rw——- 1 root root 161K Aug 21 23:49 rear.log
-rw——- 1 root root 0 Aug 21 23:51 selinux.autorelabel
-rw——- 1 root root 277 Aug 21 23:49 VERSION

INCREMENTAL BACKUPS
ReaR is not a file level Recovery tool (Look at fwbackups) however, you can perform incremental backups, in fact, in the “BACKUP_TYPE=incremental” parameter which takes care of that.
As you can see from the file list above, it shows the letter “F” before the .tar.gz extension which is an indication that this is a full backup.
Actually it’s better to make the rescue ISO seperately from the backup.
The command “rear mkbackup -v” makes both the bootstrap ISO and the backup itself, but running “rear mkbackup -v” twice won’t create incremental backups for some reason.

So first:
[root@centos7 ~]# time rear mkrescue -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
Creating disk layout
Creating root filesystem layout
TIP: To login as root via ssh you need to set up /root/.ssh/authorized_keys or SSH_ROOT_PASSWORD in your configuration file
Copying files and directories
Copying binaries and libraries
Copying kernel modules
Creating initramfs
Making ISO image
Wrote ISO image: /var/lib/rear/output/rear-centos7.iso (90M)
Copying resulting files to nfs location

real 0m49.055s
user 0m15.669s
sys 0m10.043s

And then:
[root@centos7 ~]# time rear mkbackuponly -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
Creating disk layout
Encrypting disabled
Creating tar archive ‘/tmp/rear.fXJJ3VYpHJa9Za9/outputfs/centos7/2014-08-21-1605-F.tar.gz’
Archived 406 MiB [avg 4166 KiB/sec]OK
Archived 406 MiB in 101 seconds [avg 4125 KiB/sec]

real 1m44.455s
user 0m56.089s
sys 0m16.967s

Run again (for incrementals)
[root@centos7 ~]# time rear mkbackuponly -v
Relax-and-Recover 1.16.1 / Git
Using log file: /var/log/rear/rear-centos7.log
Creating disk layout
Encrypting disabled
Creating tar archive ‘/tmp/rear.Tk9tiafmLyTvKFm/outputfs/centos7/2014-08-21-1608-I.tar.gz’
Archived 85 MiB [avg 2085 KiB/sec]OK
Archived 85 MiB in 43 seconds [avg 2036 KiB/sec]

real 0m49.106s
user 0m10.852s
sys 0m3.822s

Now look again at those backup files: -F.tar.gz is the Full Backup, -I.tar.gz is the Incremental. There’s also basebackup.txt and timestamp.txt files.
total 585M
drwxr-x— 2 root root 4.0K Aug 22 00:09 .
drwxr-xr-x 3 root root 4.0K Aug 22 00:04 ..
-rw-r–r– 1 root root 407M Aug 22 00:07 2014-08-21-1605-F.tar.gz
-rw-r–r– 1 root root 86M Aug 22 00:09 2014-08-21-1608-I.tar.gz
-rw-r–r– 1 root root 2.6M Aug 22 00:09 backup.log
-rw-r–r– 1 root root 25 Aug 22 00:05 basebackup.txt
-rw——- 1 root root 202 Aug 22 00:05 README
-rw——- 1 root root 90M Aug 22 00:05 rear-centos7.iso
-rw——- 1 root root 161K Aug 22 00:05 rear.log
-rw-r–r– 1 root root 0 Aug 22 00:09 selinux.autorelabel
-rw-r–r– 1 root root 11 Aug 22 00:05 timestamp.txt
-rw——- 1 root root 277 Aug 22 00:05 VERSION

RECOVERY
ReaR is designed to create bootable .iso, making recovery very easy and flexible in terms of options. .iso files can be booted from CD/DVD optical media, USB Block Storage Devices & Hard disks and also in VMWare & Virtual Box.
To recover a system, you first need to boot to the .ISO that was created with the backup.
You may use your favorite method for booting to the .ISO whether it’s creating a bootable USB sick, burning it to a CD, mounting it in iDRAC, etc.
Just boot to it on the server in which you want to restore to.
When the recovery screen loads, select the top option to recover.
Type root to log in.
To start recovery, type
rear -v recover

TROUBLESHOOTING RECOVERY
# Create missing directory:
mkdir /run/rpcbind

# Manually start networking:
chmod a+x /etc/scripts/system-setup.d/60-network-devices.sh
/etc/scripts/system-setup.d/60-network-devices.sh

# Navigate to and list files in /var/lib/rear/layout/xfs
# Edit each file ending in .xfs with vi and remove “sunit=0 blks” from the “log” section.
# In my case, the following files, then save them:
vi /var/lib/rear/layout/xfs/fedora_serv–build-root.xfs
vi /var/lib/rear/layout/xfs/sda1.xfs
vi /var/lib/rear/layout/xfs/sdb2.xfs

# Run the following commands to get a list of LVs and VGs:
lvdisplay
vgdisplay

# Run the following commands to remove the above listed LVs and VGs:
lvremove
vgremove

# Now run recovery again:
rear recover

USEFUL URLs / FURTHER READING
ReaR Project Page:
ReaR on Github:
ReaR in OpenSuse:
YaST Module for Suse:
ReaR User Guide:
SEP-SESAM Support:
ReaR1.15 Release Notes:

image_pdfCreate PDF of this post...
Facebooktwittergoogle_plusredditpinterestlinkedinmail