Installing different Java JRE on Linux

It you need to install a newer/alternative/multiple version(s) of the Java Runtime Environment on a RHEL Server, read the following guide.  It will enable you to switch between multiple installed JRE’s.  This can be useful for development / pre-prod servers, where prod is running a different version.

If you have a requirement to run multiple JREs on a single RHEL server, then use the “alternatives” package to facilitate switching between them via a convenient menu system.  Notes on configuring alternatives are at the end of this post.

CHECK PROD JRE VER
[matt@CyberfellaProdSvr ~]$ java -version
java version “1.7.0_67”
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

CHECK PRE-PROD JRE VER
[matt@CyberfellaPreSvr JRE 1.7.65]$ java -version
java version “1.6.0_34
OpenJDK Runtime Environment (IcedTea6 1.13.6) (rhel-1.13.6.1.el6_6-x86_64)
OpenJDK 64-Bit Server VM (build 23.25-b01, mixed mode)

CHECK EXISTING JRE(s) INSTALLED
rpm -qa | sort | grep ^j
java-1.5.0-gcj-1.5.0.0-29.1.el6.x86_64
java-1.6.0-openjdk-1.6.0.34-1.13.6.1.el6_6.x86_64
java-1.6.0-openjdk-devel-1.6.0.34-1.13.6.1.el6_6.x86_64

DOWNLOAD JRE VERSION
The latest version of JRE (Java Runtime Environment) is easy to find. The older ones, not so much. You’ll need an Oracle Support Account. Don’t panic, it’s free to set up.

WINSCP JRE TO LINUX SERVER
Install WinSCP on Windows. It will prompt to import all your PuTTY Sessions! How Convenient!  I love stuff that saves time.
Using WinSCP on Windows, SCP JRE 1.7.67 to the server.

INSTALL/UPGRADE JRE 1.7.67
rpm -Uvh jre-7u67-linux-x64.rpm

ADDING JAVA to ALTERNATIVES
View current selection of different JREs in alternatives
alternatives –config java
Add new version after installing rpm
alternatives –install <exe path> <binary> <install path> <selection>
alternatives –install /usr/bin/java java /usr/java/jre1.7.0_67/bin/java 3
Switch to new version
update-alternatives –config java
or just “alternatives –config java” seems to do the same thing.
Show current JRE
java -version

Facebooktwitterredditpinterestlinkedinmail

Automation with Ansible

The DevOps revolution has no end of brilliant projects and products that promise to get you closer to the “Infrastructure as Code” ideology.   I’ve briefly introduced the rapid deployment of virtual machines using Vagrant here and now it’s time to introduce Ansible.

Ansible is a tool that should be available from your repositories already, just like the afore mentioned Vagrant.  It’s a RedHat project, but is available across the Linux distribution landscape.  So on Debian/Ubuntu/Mint, installation is as easy as…

sudo apt-get install ansible

It is agentless, which is great as it radically simplifies the process of getting up and running, but like many agentless tools (the ones that don’t require the installation of a client daemon on all machines), you will either need to be using a directory admin account that is already set up to have privileges on all other servers in the domain, or else copy SSH keys out to all the machines that you intend to use ansible against in order to bring automation and consistency to your linux network.  The process of setting up passwordless authentication has already been covered here but it’s simple enough so I’ll summarise it here for convenience.

Lets say you have a machine linux1 with a user matt that you want to use as your ansible “server” to run commands against servers linux10, linux11 and linux12.  The other servers also have a user matt but it’s a local user, not a user in a directory.  In order for matt on linux1 to be accepted as being synonymous with matt on the other servers linux10, linux11 and linux12, matt‘s SSH keys will need to be generated and the public key copied to the other machines.

In order for matt on linux1 to be accepted as being synonymous with matt on the other servers linux10, linux11 and linux12, matt‘s SSH keys will need to be generated on linux1 and the public key copied to the other linux10, 11 and 12 machines.

su – matt

ssh-keygen  (Note:  do not use passphrase, leave blank or you’ll be prompted every time you attempt a passwordless connection to a remote host and this will obstruct using your public key authentication as root on remote system. )

cd .ssh

ssh-copy-id -i id_rsa.pub linux10 linux11 linux12  (Note:  On reflection, use the full path to the id_rsa file, e.g. /home/root/.ssh/id_rsa.pub.  This is because there is the potential to su to root and land in the previous users .ssh folder, and subsequently copy that users keys instead of the root users.  You’ll be hours figuring that one out).

Now that we’ve got that out of the way, we can get back to the subject in hand, namely ansible.  Ansible is a way of doing away with having to ssh to every machine in order to execute something locally on that remote machine in order to make it consistent with the other machines in your enterprise environment.

Ansible is a way of doing away with having to ssh to every machine in order to execute something locally on that remote machine in order to make it consistent with the other machines in your enterprise environment.

There are many modules available in ansible, documented  here but in order to keep this introduction to ansible simple, we’ll just demo the command module.

There is just one last thing to set up before that, and that is a hosts file that groups together your hosts in your network.  hosts can belong to more than one group, but in this simple demo, we need to create a file called hosts and in it, create a group called [group1] with linux10, linux11 and linux12 hosts as members…

vi hosts

[group1]

linux10

linux11

linux12

With this group created, we can now execute a command against each of the hosts in the group using ansible.

The syntax is ansible, followed by the group name, followed by -i (information), in our case the hosts file (not to be confused with /etc/hosts) , followed by -m (module name, in our case command module), followed by -a (arguments to be passed to the module, in our case “uname -a”).

ansible group1 -i ./hosts -m command -a “uname -a”

This will return the results of running uname -a on each of the servers listed in the group in our hosts file, to stdout just as if we had ssh’d to each of them in the same terminal and executed the command.

The example below shows the results of executing uptime against my laptop from a centos vm running on virtualbox, as user matt, where the ssh keys have been prior copied to my laptop, then again as the root user where the ssh keys have not.  Note also that once the passphrase has been entered once for the user, that’s it from that point on and the ansible host is effectively trusted to execute commands on remote hosts.  Powerful and Convenient stuff.

If you want to be able to use ansible as root to execute commands remotely (using ansibles -b option, i.e. become) then you’ll need to copy the root users ssh keys over to the remote hosts too.  You can do this the exact same way as you copy over any other users ssh keys, only this one comes with an added obstacle – ssh as root is not permitted by default in most modern linux distributions as a way of hardening against a brute force attack as root.  Sensible stuff, and not that difficult to overcome.  You just need to edit the /etc/ssh/sshd-config file on the remote host to permit root login while you copy the keys across.

Just comment out the existing PermitRootLogin prohibit-password line and replace it with PermitRootLogin yes.  Note: not PermitRootLogin PermitRootLogin as in the example above – I couldn’t restart sshd.

service sshd restart

And voila, the root users ssh keys copy across fine.

Now you need to change the ssh-config file back to PermitRootLogin prohibit-password and restart sshd again to put the system back to it’s secure default state whereby the root user is allowed to attempt a connection, it’s just not allowed to send a password.  If ssh keys are in place of course, passwords don’t need to be sent – that’s the whole point of ssh keys, after all!

Voila, I can now ssh to the remote system as root, even thought the ssh daemon on the remote system is configured to not permit password authentication for inbound connections by the user root.   If that’s the case, then you will now be able to use the ansible -b option (become) to execute commands or playbooks to configure remote systems as root.

At this point, you may find yourself saying “not on my system it doesn’t!”

If that’s the case, please go to the end of the post and read the Troubleshooting SSH connections section for tips on what to do.

Although ansible now works as root on remote systems, you’ll find that sudoers throws you an error when attempting to use the -b (become root) option when running the ansible command as a user other than root on the ansible server.

Adding the user to the sudo group on the remote host doesn’t fix this either, since the sudoers mechanism will still (by default) ask for the users password in order to run a command as root.

Once this edit has been made to sudoers using visudo then you can see below, that re-running the same ansible -b command as the vagrant user, successfully executes the uptime command as root on the remote system.

And therein ends my initial introduction to ansible and hopefully some tips on getting it working the way you want.  Playbooks will be covered in a separate post.

Troubleshooting SSH connections

You may find yourself having issues with connecting as root or any other user for that matter.  Despite having created and copied you public keys to the remote systems, you’re still being prompted for passphrases or passwords for the user, defeating the whole point of setting up passwordless authentication.

Here’s a quick checklist of things to look out for and ways to troubleshoot the connection.

service stop sshd && /usr/sbin/sshd -d  (restart sshd in debug mode on the remote machine)

ssh -vv <remote-host> (connect to the remote host using ssh in verbose mode)

Before Googling the errors, make sure you can confirm the following:

When you generated the public keys using ssh-keygen you left the passphrase blank.

When you copied the keys over to the remote machine using ssh-copy-id you used the full path to the id_rsa.pub file.  If you’re root, it’s quite probable you copied another users ssh keys over instead of your own!

The .ssh directory in the users home directory has 700 permissions and the authorized-keys file has 600 permissions.

Facebooktwitterredditpinterestlinkedinmail

Rapid VM Deployment using Vagrant

I need a VM and I need it asap.

Vagrant is designed to be the quickest way to a running VM, and I’m impressed.  I have VirtualBox running on my trusty Dell XPS 13  laptop; “Sputnik” (named after the collaboration between Ubuntu and Dell).

Installing Virtualbox and Vagrant on Linux Mint (or any  Debian/Ubuntu derivative) is as easy as typing…

sudo apt-get install virtualbox vagrant

…and thanks to Vagrant and the many virtual machines available for VirtualBox and VMWare platforms, getting your first VM up and running is as simple as typing…

vagrant init centos/7 or vagrant init debian/jessie64

or vagrant init hashicorp/precise64 the latter hashicorp Ubuntu LTS build being the one that Vagrant’s own documentation is based upon.  For my example here, I’m going to start with a RHEL based Centos 7 offering..

This creates a text file called Vagrantfile in the current directory.

Rather than have this file in the root of my home directory, I’ve relocated it to a subdirectory ~/Vagrant/Centos7.  This will allow me to have other Vagrantfiles for other types of VM all stored under ~/Vagrant in their own subdirectory.  Probably not a bad idea as I’ll likely want to spin up a few different VM’s over time.

I’m now ready to “up” my VM…

vagrant up

Since I don’t already have a copy of the image downloaded, it goes off to sort all that for you.  While it’s doing that, there’s nothing stopping me from spinning up an Ubuntu Precise64 VM in another terminal window…

Since I already had the hashicorp/precise64 “Box” image from a previous deployment, it procured this VM in seconds while it continued to download the Centos Box image in the other terminal.

In my other terminal window, Centos 7 has now also been procured, along with some helpful tips should any issues arise around non-installation of VirtualBox Guest Additions on my host  (In my case, I’m running VirtualBox version 5.1.34 at the time of writing).

Flick across to VirtualBox Manager and you can see the two new running VMs based on the downloaded Boxes have been added to the Inventory.  Note: Do not rename them.

To connect to them, simply use the command…

vagrant ssh

Both VM’s allow you to log on instantly over SSH with just this minimalist command run from within the directory containing the Vagrantfile.

So there you have it, a Centos VM and a Ubuntu VM up and running in seconds.  Not hours.  Not Days.  Not Weeks.

It is that simple.  From Zero to Virtualbox, Vagrant and logged on to a running VM of your choice in three commands and dare I say it, about three seconds.

To shut them down, or bring them online again, use the following commands, just make sure you run them from within the correct subdirectory or you could shut the wrong VM down…

vagrant halt

vagrant up

It’s worth checking out the Vagrantfile and the documentation online as you can copy and re-use the Vagrantfile and make useful modifications to it.  Here are some more vagrant box commands to explore.

You can see here that although the vagrant box list command shows all boxes/images downloaded on your host system, if you execute vagrant box outdated, it’ll only check for updated box images for the box image specified in your local Vagrantfile, not all Boxes on the host system at once.

Note that this is not the same thing as performing sudo apt-get update && sudo apt-get dist-upgrade (or redhat equivalent yum update command) on the VM built using the Box image (shown below).

As with any new VM or Server, you will probably want to bring all packages up to date using the VM’s own OS package management system.

Vagrant Boxes and Shared Folders

As already established, Vagrant images for VMWare or VirtualBox can be downloaded from the internet using the vagrant command line or as a quick google search will reveal, from here.

The image files (boxes) are stored in .vagrant.d/boxes

Once an image has been downloaded, a “box” has been “created”.  This doesn’t mean a server (VM) has been created.  It just means that your local installation of vagrant has a box, ready to be deployed as a VM.

Before this can be done, it is prudent to create a “Project” for your VM, to put a structure in place to allow for some separation, given that you’re likely to want more than one VM.  This is very easy to do.  Just create some folder e.g. vagrant-project1 in your home directory, or anything you like.

cd into that directory and initialise a new project,

cd ~/vagrant-project1

vagrant init

This will create a file called Vagrantfile in your project folder.  Edit this file to read as follows…

Vagrant.configure(“2”) do |config|
   config.vm.box = “hashicorp/precise64”
   config.vm.box_version = “1.1.0”
   config.vm.box_url = “https://vagrantcloud.com/hashicorp/precise64”
end

You don’t have to put all three lines in it, just the first one will do, but why not while you’re in there?

You are not at a point where you have a project, you have a box and you’ve configured your new project to use that box.  Now you can bring up a VM in Virtualbox by running the following command from inside your Project folder,

vagrant up

You can log in with no password, by typing

vagrant ssh

Then create your first user and add him/her to the sudoers file, just as you would with any linux server,

adduser matt && adduser matt sudo

or adduser matt && usermod -aG sudo matt

whatever’s your way of doing it, it doesn’t matter.

Shared folders

You can edit files on your VM locally, you don’t need to ssh to the server in order to access files on it.  Vagrant has mounted your Project folder from inside the VM, so if you’re on the VM and you cd into /vagrant, you’ll be in the same folder as if you’re on your host machine and you cd into ~/vagrant-project1.  Really cool!

Provisioning

Vagrant can also be configured to automatically provision.  For example, the Vagrantfile can be edited as follows to automatically execute a script – in this case, the script installs Apache if it is not already present.

The bootstrap.sh script referenced in the Vagrantfile above, looks as follows.

Since it was created in my project folder, it is automatically also on the VM since that folder is automatically mounted as mentioned previously.

To effect the changes, you can vagrant reload –provision a running VM to quickly restart it, or if you’ve not yet started the VM, vagrant up will automatically do it.

You can see the VM getting restarted by Vagrant (white text shown above) and Apache getting installed (green text) in the console.

The Apache webserver will not be available from your local web browser, but it can be tested on the VM command line with wget -q0- 127.0.0.1

Networking

Port Forwarding

We can forward the webserver port 80 to our local machine on say, port 4567 and test the webserver accordingly using our own web browser.

We can see that due to the tunnel we’ve created from the VM, by browsing local port 4567, we’re seeing what’s being served on port 80 on our VM.

 

 

 

 

Facebooktwitterredditpinterestlinkedinmail

Linux Cheatsheets

The following post is for convenience where solutions and answers to your everyday IT challenges are not found in the many posts published on the site.

It serves as a single point of download for many useful cheat sheets freely published by other linux systems admins – not me.

The original authors are credited on each cheatsheet.

Redhat Linux 5 6 7

Regular Expressions

Centos

Linux Command Line

Bash

Bash and ZSH

Basic Systems Admin

Linux Cluster

Pocket Guide Linux Commands

Linux Network Commands

Things I Forget

Linux Systems Admin

Users and Groups

Vim Editor

Fstab and NFS

Puppet

Shell Scripting

Metasploit

Rsync

Yum

LVM Logical Volume Manager

Awk

Logrotate and Cron

Wget

Bash Script Colours

Docker

Git

SSH

Find

Aircrack

DevOps and SecOps

 

Facebooktwitterredditpinterestlinkedinmail

Ping a list of hosts

The following shell script automates a ping test across a list of hosts.  The format of the expected host-list file is…

<hostname1> <ipaddress1>

<hostname2> <ipaddress2>

<hostname3> <ipaddress3>

…etc

You can easily tailor the script to suit your list if you only have  a list of hostnames or ip addresses.

The hosts that respond are logged to a file, ping_log.

Note that the script was written in Bash on a Red Hat Linux server, and the syntax may differ from a fully POSIX compliant script written in Ksh on HPUX, where variables are encapsulated in {} brackets and tests are double [[ ]] bracketed.

Facebooktwitterredditpinterestlinkedinmail

Whats filled up my filesystem?

If df -h reveals that one of your linux filesystems is full, you’ll be asking yourself whats filled it.

find /myfs -mtime +1 -type f -size +1000000 -exec ls -al {} \;
Replace /myfs with the the name of the filesystem thats filled according to df -h
Knock a zero off -size and repeat to drill down into the largest files
Add 1 to -mtime to go back 1, 2, 3 days etc to see the largest files written to in the last 1, 2, 3 days.
If you’re lucky enough to have a GUI and internet connectivity (highly unlikely on a linux server in the datacenter), install a tree mapper such as baobab (equivalent to sequoia view for Windows).
baobab
Facebooktwitterredditpinterestlinkedinmail

RHEL Server using Powerpath won’t boot

If you have a Red Hat Enterprise Linux server with the emc Powerpath driver installed, and find it won’t boot due to the LVM subsystem not being able to find the volume group, it may be because the Powerpath driver hasn’t finished loading and scanning for block storage before the LVM subsystem has started.  There are two ways around it, one if to set the dump and fsck order numbers in /etc/fstab to 0 and 0 respectively, but that means the filesystem will never be checked for errors during boot time.  The other is the correct way around it, as recommended by emc in the Powerpath guide, and involves using the _netdev option in the /etc/fstab file so that the OS know to give it some time.  It works too!

Use the _netdev option in /etc/fstab instead of defaults

Use a higher number than 1 for the fsck check order too.

The emc Powerpath guide states…

For RHEL 5, PowerPath devices should be mounted with the _netdev
option instead of the defaults option in the /etc/fstab file. This will
ensure that fsck is run later in the boot sequence.

Facebooktwitterredditpinterestlinkedinmail

nohup and disown your long running jobs

Ever started a job and thought “this is running on a bit longer than I expected”, then wondered whats going to happen when you go home and come back in to work tomorrow morning to find your remote session gone, which leaves you wondering “did that job complete?”.


Mmm, me too, which is where the nohup or disown commands come in.  nohup (no hangup) is a well known job control utility which will prevent a process from reacting to the hangup signal when a shell is disconnected.  Usually you’d preceed your actual command with it, e.g.

nohup rsync -auvc /Source/* /Destination/

but if your command is already running, you’re left wishing you’d nohup‘d it to start with – unless you’re running Solaris or AIX in which case the nohup command has a convenient -p switch to specify the process id (use ps -ef | grep rsync to obtain the PID of that long running data migration process, then nohup -p 9675 (or whatever the PID is of your running job).

If you’re not running Solaris or AIX, then pray you started the command in the bash shell (Linux default shell so more likely than not).  If you did, then you can

CTRL-Z

to pause the current job running in the foreground, then use the

jobs

command to determine its job number (most likely 1 if there’s no other sysadmins running backgrounded jobs), then background the process with

bg 1 

then finally

disown %1

to disconnect the process from the current shell.  Running

jobs 

again will show that your job is no longer in the list, but

ps -ef

will reveal that it is in fact still running.

Your shell can now be closed without the fear of your running job being killed with it.  Yay.

Facebooktwitterredditpinterestlinkedinmail

Quick surface plots using GNUPlot

GNUPlot is a free and very neat little graphing tool.  Upon installing it and running gnuplot you’ll be presented with a flashing command line prompt gnuplot>_ at which point you’ll be asking yourself “now what?”

gnuplot either plots data using plot or plots 3D surfaces using splot.

In this example, I’ll plot a surface plot (3D splot) of weekday (x), hour (y), number of jobs running (z).

gnuplot likes to be fed its data in text column format, separated with spaces or tabs but not commas e.g.

#day #hour #jobs
1 1 5
1 2 5
1 3 5
1 4 5
1 5 5
1 6 5
1 7 5
1 8 5
1 9 5
1 10 5

To create a surface plot of this data (my sample data used has values for all 24 hours in all 7 days), simply type

splot ‘path_to_data.dat’ to point to your text file containing your columns of numbers.

The results will be something like this.  Good, but not quite there yet.

gnuplot1

 

 

 

 

 

 

 

 

 

 

Some extra commands in the gnuplot command line window will improve the visual representation of the data, giving us the surface plot we’re ultimately after.

set dgrid3d

set grid

set view 50

set style data lines

set contour base

set hidden3d trianglepattern 7

set autoscale

Finally, use the command replot to update the graph.  The results are now much more usable, with contour lines on the base of the 3D graph to further highlight the “hot spots”, i.e. the hours of what day the most jobs are running (in my example).

gnuplot3

 

 

 

 

 

 

 

 

 

 

 

There’s much more fun to be had tweaking GNUPlot but I’ll leave that up to you and your imagination.  It’s worth finally mentioning  that the commands entered into gnuplot can be scripted and saved as a .plt file to compliment your .dat data file.  Then, to plot the surface maps again, you just need to load the script using…

load ‘path_to_script.plt’

Remember, the final line in your script should be

splot ‘path_to_data.dat’

so that the graph is actually generated, with all the options preceeding it.  e.g.

#My GNUPlot surface map script surf-map.plt

set dgrid3d

set grid

set view 50

set style data lines

set contour base

set hidden3d trianglepattern 7

set autoscale

splot ‘data.dat’

How you generate your actual data to be plotted is up to you.  A scheduled task/cron job which collects the data and appends it to the data.dat file is generally run as a separate shell script, e.g.

#!/bin/sh

#Insert newline into data.dat
HOURVAL=`date | awk {‘print $4’} | cut -d: -f1`
DAYVAL=`date | awk {‘print $1’}
RUNNINGGROUPSVAL=`ps -ef | grep savegrp | wc -1`

echo “${HOURVAL} ${DAYVAL} ${RUNNINGGROUPSVAL}” >> ~/data.dat

and the graphs generated at will using gnuplot.

Depending on the shapes generated by the surface map, it’s a nice touch that GNUPlot allows you to left-click on the graph and drag it around in 3 dimensions to achieve the best possible viewing angle, prior to saving the .png file, conveniently colouring the underside of the surface a different colour to the upper, visible side of the surface.

Facebooktwitterredditpinterestlinkedinmail