Automation with Ansible

The DevOps revolution has no end of brilliant projects and products that promise to get you closer to the “Infrastructure as Code” ideology.   I’ve briefly introduced the rapid deployment of virtual machines using Vagrant here and now it’s time to introduce Ansible.

Ansible is a tool that should be available from your repositories already, just like the afore mentioned Vagrant.  It’s a RedHat project, but is available across the Linux distribution landscape.  So on Debian/Ubuntu/Mint, installation is as easy as…

sudo apt-get install ansible

It is agentless, which is great as it radically simplifies the process of getting up and running, but like many agentless tools (the ones that don’t require the installation of a client daemon on all machines), you will either need to be using a directory admin account that is already set up to have privileges on all other servers in the domain, or else copy SSH keys out to all the machines that you intend to use ansible against in order to bring automation and consistency to your linux network.  The process of setting up passwordless authentication has already been covered here but it’s simple enough so I’ll summarise it here for convenience.

Lets say you have a machine linux1 with a user matt that you want to use as your ansible “server” to run commands against servers linux10, linux11 and linux12.  The other servers also have a user matt but it’s a local user, not a user in a directory.  In order for matt on linux1 to be accepted as being synonymous with matt on the other servers linux10, linux11 and linux12, matt‘s SSH keys will need to be generated and the public key copied to the other machines.

In order for matt on linux1 to be accepted as being synonymous with matt on the other servers linux10, linux11 and linux12, matt‘s SSH keys will need to be generated on linux1 and the public key copied to the other linux10, 11 and 12 machines.

su – matt

ssh-keygen  (Note:  do not use passphrase, leave blank or you’ll be prompted every time you attempt a passwordless connection to a remote host and this will obstruct using your public key authentication as root on remote system. )

cd .ssh

ssh-copy-id -i linux10 linux11 linux12  (Note:  On reflection, use the full path to the id_rsa file, e.g. /home/root/.ssh/  This is because there is the potential to su to root and land in the previous users .ssh folder, and subsequently copy that users keys instead of the root users.  You’ll be hours figuring that one out).

Now that we’ve got that out of the way, we can get back to the subject in hand, namely ansible.  Ansible is a way of doing away with having to ssh to every machine in order to execute something locally on that remote machine in order to make it consistent with the other machines in your enterprise environment.

Ansible is a way of doing away with having to ssh to every machine in order to execute something locally on that remote machine in order to make it consistent with the other machines in your enterprise environment.

There are many modules available in ansible, documented  here but in order to keep this introduction to ansible simple, we’ll just demo the command module.

There is just one last thing to set up before that, and that is a hosts file that groups together your hosts in your network.  hosts can belong to more than one group, but in this simple demo, we need to create a file called hosts and in it, create a group called [group1] with linux10, linux11 and linux12 hosts as members…

vi hosts





With this group created, we can now execute a command against each of the hosts in the group using ansible.

The syntax is ansible, followed by the group name, followed by -i (information), in our case the hosts file (not to be confused with /etc/hosts) , followed by -m (module name, in our case command module), followed by -a (arguments to be passed to the module, in our case “uname -a”).

ansible group1 -i ./hosts -m command -a “uname -a”

This will return the results of running uname -a on each of the servers listed in the group in our hosts file, to stdout just as if we had ssh’d to each of them in the same terminal and executed the command.

The example below shows the results of executing uptime against my laptop from a centos vm running on virtualbox, as user matt, where the ssh keys have been prior copied to my laptop, then again as the root user where the ssh keys have not.  Note also that once the passphrase has been entered once for the user, that’s it from that point on and the ansible host is effectively trusted to execute commands on remote hosts.  Powerful and Convenient stuff.

If you want to be able to use ansible as root to execute commands remotely (using ansibles -b option, i.e. become) then you’ll need to copy the root users ssh keys over to the remote hosts too.  You can do this the exact same way as you copy over any other users ssh keys, only this one comes with an added obstacle – ssh as root is not permitted by default in most modern linux distributions as a way of hardening against a brute force attack as root.  Sensible stuff, and not that difficult to overcome.  You just need to edit the /etc/ssh/sshd-config file on the remote host to permit root login while you copy the keys across.

Just comment out the existing PermitRootLogin prohibit-password line and replace it with PermitRootLogin yes.  Note: not PermitRootLogin PermitRootLogin as in the example above – I couldn’t restart sshd.

service sshd restart

And voila, the root users ssh keys copy across fine.

Now you need to change the ssh-config file back to PermitRootLogin prohibit-password and restart sshd again to put the system back to it’s secure default state whereby the root user is allowed to attempt a connection, it’s just not allowed to send a password.  If ssh keys are in place of course, passwords don’t need to be sent – that’s the whole point of ssh keys, after all!

Voila, I can now ssh to the remote system as root, even thought the ssh daemon on the remote system is configured to not permit password authentication for inbound connections by the user root.   If that’s the case, then you will now be able to use the ansible -b option (become) to execute commands or playbooks to configure remote systems as root.

At this point, you may find yourself saying “not on my system it doesn’t!”

If that’s the case, please go to the end of the post and read the Troubleshooting SSH connections section for tips on what to do.

Although ansible now works as root on remote systems, you’ll find that sudoers throws you an error when attempting to use the -b (become root) option when running the ansible command as a user other than root on the ansible server.

Adding the user to the sudo group on the remote host doesn’t fix this either, since the sudoers mechanism will still (by default) ask for the users password in order to run a command as root.

Once this edit has been made to sudoers using visudo then you can see below, that re-running the same ansible -b command as the vagrant user, successfully executes the uptime command as root on the remote system.

And therein ends my initial introduction to ansible and hopefully some tips on getting it working the way you want.  Playbooks will be covered in a separate post.

Troubleshooting SSH connections

You may find yourself having issues with connecting as root or any other user for that matter.  Despite having created and copied you public keys to the remote systems, you’re still being prompted for passphrases or passwords for the user, defeating the whole point of setting up passwordless authentication.

Here’s a quick checklist of things to look out for and ways to troubleshoot the connection.

service stop sshd && /usr/sbin/sshd -d  (restart sshd in debug mode on the remote machine)

ssh -vv <remote-host> (connect to the remote host using ssh in verbose mode)

Before Googling the errors, make sure you can confirm the following:

When you generated the public keys using ssh-keygen you left the passphrase blank.

When you copied the keys over to the remote machine using ssh-copy-id you used the full path to the file.  If you’re root, it’s quite probable you copied another users ssh keys over instead of your own!

The .ssh directory in the users home directory has 700 permissions and the authorized-keys file has 600 permissions.


Rapid VM Deployment using Vagrant

I need a VM and I need it asap.

Vagrant is designed to be the quickest way to a running VM, and I’m impressed.  I have VirtualBox running on my trusty Dell XPS 13  laptop; “Sputnik” (named after the collaboration between Ubuntu and Dell).

Installing Virtualbox and Vagrant on Linux Mint (or any  Debian/Ubuntu derivative) is as easy as typing…

sudo apt-get install virtualbox vagrant

…and thanks to Vagrant and the many virtual machines available for VirtualBox and VMWare platforms, getting your first VM up and running is as simple as typing…

vagrant init centos/7 or vagrant init debian/jessie64

or vagrant init hashicorp/precise64 the latter hashicorp Ubuntu LTS build being the one that Vagrant’s own documentation is based upon.  For my example here, I’m going to start with a RHEL based Centos 7 offering..

This creates a text file called Vagrantfile in the current directory.

Rather than have this file in the root of my home directory, I’ve relocated it to a subdirectory ~/Vagrant/Centos7.  This will allow me to have other Vagrantfiles for other types of VM all stored under ~/Vagrant in their own subdirectory.  Probably not a bad idea as I’ll likely want to spin up a few different VM’s over time.

I’m now ready to “up” my VM…

vagrant up

Since I don’t already have a copy of the image downloaded, it goes off to sort all that for you.  While it’s doing that, there’s nothing stopping me from spinning up an Ubuntu Precise64 VM in another terminal window…

Since I already had the hashicorp/precise64 “Box” image from a previous deployment, it procured this VM in seconds while it continued to download the Centos Box image in the other terminal.

In my other terminal window, Centos 7 has now also been procured, along with some helpful tips should any issues arise around non-installation of VirtualBox Guest Additions on my host  (In my case, I’m running VirtualBox version 5.1.34 at the time of writing).

Flick across to VirtualBox Manager and you can see the two new running VMs based on the downloaded Boxes have been added to the Inventory.  Note: Do not rename them.

To connect to them, simply use the command…

vagrant ssh

Both VM’s allow you to log on instantly over SSH with just this minimalist command run from within the directory containing the Vagrantfile.

So there you have it, a Centos VM and a Ubuntu VM up and running in seconds.  Not hours.  Not Days.  Not Weeks.

It is that simple.  From Zero to Virtualbox, Vagrant and logged on to a running VM of your choice in three commands and dare I say it, about three seconds.

To shut them down, or bring them online again, use the following commands, just make sure you run them from within the correct subdirectory or you could shut the wrong VM down…

vagrant halt

vagrant up

It’s worth checking out the Vagrantfile and the documentation online as you can copy and re-use the Vagrantfile and make useful modifications to it.  Here are some more vagrant box commands to explore.

You can see here that although the vagrant box list command shows all boxes/images downloaded on your host system, if you execute vagrant box outdated, it’ll only check for updated box images for the box image specified in your local Vagrantfile, not all Boxes on the host system at once.

Note that this is not the same thing as performing sudo apt-get update && sudo apt-get dist-upgrade (or redhat equivalent yum update command) on the VM built using the Box image (shown below).

As with any new VM or Server, you will probably want to bring all packages up to date using the VM’s own OS package management system.

Vagrant Boxes and Shared Folders

As already established, Vagrant images for VMWare or VirtualBox can be downloaded from the internet using the vagrant command line or as a quick google search will reveal, from here.

The image files (boxes) are stored in .vagrant.d/boxes

Once an image has been downloaded, a “box” has been “created”.  This doesn’t mean a server (VM) has been created.  It just means that your local installation of vagrant has a box, ready to be deployed as a VM.

Before this can be done, it is prudent to create a “Project” for your VM, to put a structure in place to allow for some separation, given that you’re likely to want more than one VM.  This is very easy to do.  Just create some folder e.g. vagrant-project1 in your home directory, or anything you like.

cd into that directory and initialise a new project,

cd ~/vagrant-project1

vagrant init

This will create a file called Vagrantfile in your project folder.  Edit this file to read as follows…

Vagrant.configure(“2”) do |config| = “hashicorp/precise64”
   config.vm.box_version = “1.1.0”
   config.vm.box_url = “”

You don’t have to put all three lines in it, just the first one will do, but why not while you’re in there?

You are not at a point where you have a project, you have a box and you’ve configured your new project to use that box.  Now you can bring up a VM in Virtualbox by running the following command from inside your Project folder,

vagrant up

You can log in with no password, by typing

vagrant ssh

Then create your first user and add him/her to the sudoers file, just as you would with any linux server,

adduser matt && adduser matt sudo

or adduser matt && usermod -aG sudo matt

whatever’s your way of doing it, it doesn’t matter.

Shared folders

You can edit files on your VM locally, you don’t need to ssh to the server in order to access files on it.  Vagrant has mounted your Project folder from inside the VM, so if you’re on the VM and you cd into /vagrant, you’ll be in the same folder as if you’re on your host machine and you cd into ~/vagrant-project1.  Really cool!


Vagrant can also be configured to automatically provision.  For example, the Vagrantfile can be edited as follows to automatically execute a script – in this case, the script installs Apache if it is not already present.

The script referenced in the Vagrantfile above, looks as follows.

Since it was created in my project folder, it is automatically also on the VM since that folder is automatically mounted as mentioned previously.

To effect the changes, you can vagrant reload –provision a running VM to quickly restart it, or if you’ve not yet started the VM, vagrant up will automatically do it.

You can see the VM getting restarted by Vagrant (white text shown above) and Apache getting installed (green text) in the console.

The Apache webserver will not be available from your local web browser, but it can be tested on the VM command line with wget -q0-


Port Forwarding

We can forward the webserver port 80 to our local machine on say, port 4567 and test the webserver accordingly using our own web browser.

We can see that due to the tunnel we’ve created from the VM, by browsing local port 4567, we’re seeing what’s being served on port 80 on our VM.






DevOps in a Nutshell

 What is DevOps?

DevOps is the application of the Development life cycle to your Infrastructure Operations, Datacentre and Cloud computing environments beneath.

Yes, the Developers are coming over the hill and are taking the SysOps jobs! Everything will be managed a single grand unified way.  Sysadmins look out!  They’ll automate you out of existence with self-serve apps!

Now that servers run in VMWare and in Containers of isolated UNIX and Linux software stacks sharing a common underlying kernel, servers that were once hardware, are more often-than-not, now software entities or “microservices”.

As such, development processes can be applied to the management of their lifecycle, coining the term “Infrastructure As Code” and not just to the upper Application Layer in the OSI Model.


DevOps means doing better and proper processes.  The first thing to know is what processes exist and then to check if and how you implement them in your organisation/IT department.

  • Development Process
  • Requirements Engineering
  • Testing and QA
  • System Integration
  • Release Management
  • Change Management
  • Deployment
  • Configuration Management
  • Update Management
  • Incident Management
  • System Provisioning
  • Installation Automation
  • Security Policies
  • Monitoring
  • Learning and Training

Solutions per Process

Development Process:    Scrum, Kanban, IBM Rational Suite …
Testing:   QA Jenkins, Selenium, …
System Integration:   Mozilla Tinderbox, …
Release Management:   Redmine, Trac, SourceForge, Bugzilla…
Change Management:   idoit, itop, project-open
Deployment:   Fabric, Garnison, YADT, ..

Configuration Management:   Trebutchet, …
Update Management:   lpvs, debsecan
Incindent Management:   idoit, itop, …
Installation Automation:   puppet, chef, cfengine, ansible
Security Policies:   FIXME
Monitoring:   Nagios & Co, Munin, Cacti, NewRelic, Splunk, Netflow, …
Learning and Training:   any spreadsheet

Commercial Solutions


  • Atlassian Confluence: Good for startups due to small user licensing, but beware the 30 and 100 user steps!


  • Atlassian Jira: Classical ticketing, same licensing advantages and disadvantages as Confluence
  • Jira Greenhopper Plugin: Scrum Board for Jira



Linux Cheatsheets

The following post is for convenience where solutions and answers to your everyday IT challenges are not found in the many posts published on the site.

It serves as a single point of download for many useful cheat sheets freely published by other linux systems admins – not me.

The original authors are credited on each cheatsheet.

Redhat Linux 5 6 7

Regular Expressions


Linux Command Line


Bash and ZSH

Basic Systems Admin

Linux Cluster

Pocket Guide Linux Commands

Linux Network Commands

Things I Forget

Linux Systems Admin

Users and Groups

Vim Editor

Fstab and NFS


Shell Scripting




LVM Logical Volume Manager


Logrotate and Cron


Bash Script Colours






DevOps and SecOps