DevOps in a Nutshell

 What is DevOps?

DevOps is the application of the Development life cycle to your Infrastructure Operations, Datacentre and Cloud computing environments beneath.

Yes, the Developers are coming over the hill and are taking the SysOps jobs! Everything will be managed a single grand unified way.  Sysadmins look out!  They’ll automate you out of existence with self-serve apps!

Now that servers run in VMWare and in Containers of isolated UNIX and Linux software stacks sharing a common underlying kernel, servers that were once hardware, are more often-than-not, now software entities or “microservices”.

As such, development processes can be applied to the management of their lifecycle, coining the term “Infrastructure As Code” and not just to the upper Application Layer in the OSI Model.

Processes

DevOps means doing better and proper processes.  The first thing to know is what processes exist and then to check if and how you implement them in your organisation/IT department.

  • Development Process
  • Requirements Engineering
  • Testing and QA
  • System Integration
  • Release Management
  • Change Management
  • Deployment
  • Configuration Management
  • Update Management
  • Incident Management
  • System Provisioning
  • Installation Automation
  • Security Policies
  • Monitoring
  • Learning and Training

Solutions per Process

Development Process:    Scrum, Kanban, IBM Rational Suite …
Testing:   QA Jenkins, Selenium, …
System Integration:   Mozilla Tinderbox, …
Release Management:   Redmine, Trac, SourceForge, Bugzilla…
Change Management:   idoit, itop, project-open
Deployment:   Fabric, Garnison, YADT, ..

Configuration Management:   Trebutchet, …
Update Management:   lpvs, debsecan
Incindent Management:   idoit, itop, …
Installation Automation:   puppet, chef, cfengine, ansible
Security Policies:   FIXME
Monitoring:   Nagios & Co, Munin, Cacti, NewRelic, Splunk, Netflow, …
Learning and Training:   any spreadsheet

Commercial Solutions

Documentation

  • Atlassian Confluence: Good for startups due to small user licensing, but beware the 30 and 100 user steps!

Ticketing

  • Atlassian Jira: Classical ticketing, same licensing advantages and disadvantages as Confluence
  • Jira Greenhopper Plugin: Scrum Board for Jira

Suites

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Linux Cheatsheets

The following post is for convenience where solutions and answers to your everyday IT challenges are not found in the many posts published on the site.

It serves as a single point of download for many useful cheat sheets freely published by other linux systems admins – not me.

The original authors are credited on each cheatsheet.

Redhat Linux 5 6 7

Regular Expressions

Centos

Linux Command Line

Bash

Bash and ZSH

Basic Systems Admin

Linux Cluster

Pocket Guide Linux Commands

Linux Network Commands

Things I Forget

Linux Systems Admin

Users and Groups

Vim Editor

Fstab and NFS

Puppet

Shell Scripting

Metasploit

Rsync

Yum

LVM Logical Volume Manager

Awk

Logrotate and Cron

Wget

Bash Script Colours

Docker

Git

SSH

Find

Aircrack

DevOps and SecOps

 

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Ping a list of hosts

The following shell script automates a ping test across a list of hosts.  The format of the expected host-list file is…

<hostname1> <ipaddress1>

<hostname2> <ipaddress2>

<hostname3> <ipaddress3>

…etc

You can easily tailor the script to suit your list if you only have  a list of hostnames or ip addresses.

The hosts that respond are logged to a file, ping_log.

Note that the script was written in Bash on a Red Hat Linux server, and the syntax may differ from a fully POSIX compliant script written in Ksh on HPUX, where variables are encapsulated in {} brackets and tests are double [[ ]] bracketed.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Protect your privacy with a VPN

Protecting your privacy doesn’t need to be as complicated as using all manner of CIA-beating tech to hide yourself and your computer from the evils that lurk on the interwebs these days, where literally nobody is to be trusted.  It’s fun setting all that stuff up, if that’s what you’re into, but for most of you, you just want a nice, easy solution that works and doesn’t affect your day-to-day online experience.

Frankly, everyone should be using a VPN, whether they realise it or not and whether they think they have anything to hide or not.

My personal favourite service (there are a few very good ones) is ExpressVPN.

Sign up for a small monthly fee and download the software for your given operating system – in my case Linux Mint (so I downloaded the Ubuntu 64bit .deb package).

The commands to install it, activate it using the code supplied when you subscribe, and connect to it are shown below….

Does it get any easier than that?  I don’t think so.

Once it’s installed and running, you should add it to your startup applications, so that it starts automatically when you log in for convenience.

Lastly and for completeness, you can add the extension for Firefox (not essential but why wouldn’t you?).

You can activate up to 3 devices with your subscription.  All major operating systems and phone operating systems are supported.

It just works.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Fix Windows 10 Slowness/Lag

Windows 10 Slowness seems to bug most users, with a constant degree of lag when flicking between tasks.  It can be stressful and disruptive to your reaching a productive, flow state.

Most modern mid-range laptops are more than capable of running multiple operating systems simultaneously, rocking Intel core i5 processors with vPro technology just like their floor standing, desk based or rack mounted workstation and server brethren so why then, is your laptop slow when all it has to do is run a single instance of Windows 10, – often on a SSD (Solid State Drive) that claims to be faster than it’s mechanical, spinning counterpart?

Many users have reverted to Windows 7 seeing as Windows 8 was such an abomination (Windows 10 is a long way from winning me over too, tbf) and are left wondering what the future holds for them from here on in, in terms of upgrade path and acceptable snappy performance.

Aside from the mammoth processors and supercharged block storage devices inside the modern affordable laptop, there is also the huge amount of RAM too.  Large RAM requirements have arisen out of the desire to run multiple applications simultaneously and flick between them, but also as a result of a widespread transition from 32-bit operating systems and applications to 64-bit – a move that in itself warrants double the amount of physical RAM in order to match the performance.

And just look at all that %Idle time?  HOW CAN IT BE SO SLOWWW?!!

Back in the 32-bit 1GB RAM days when RAM was expensive, page files were used to write pages of memory to disk, to free up precious, fast volatile RAM (You can still run many light weight distributions of Linux on that spec with very little to no slowness and minimal to no swapping to virtual memory too.  Same is true of 64 bit with 2GB RAM.)

The size of the pagefile defaulted to the amount of RAM.  This was due to the assumption that if the amount of pagefile needs to be any bigger than that, then you really do need more physical RAM as your system would have undoubtedly ground to a snails pace already.

And that is still the default.  The problem with that, is that in a laptop with a single partition, a single 8GB file used for regularly paging the chosen contents of 8GB physical memory out to it, puts a fair bit of strain on the IO subsystem – and it’s unnecessary strain.  With 8GB RAM, you’ll not need much if any page file at all, for most tasks.

So with that in mind, change your virtual memory settings from being “system managed”, so a fixed size pagefile set at the “Recommended” size.

In Control Panel, System…

These settings will need a reboot to take effect.  You should notice snappier performance as a result.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

vCenter Server Appliance installer fails on Linux

If you’ve downloaded the vCenter Server Appliance .iso file, unpacked it to a folder on your Linux workstation, then hit a problem during installation reading the .ovf file during deployment to your VMWare ESXi hypervisor

./vcsa-ui-installer/lin64/installer

The end of the installation log will read something like this

There were a couple additional steps I had to do in order to get it to run from my filesystem, rather than from a mounted .iso.

firstly, chmod -R 777 the whole lot, e.g. if you’ve unpacked the iso into a folder called /vCentre-deployment then chmod -R 777 /vCentre -deployment

You will likely have to chmod +x the  ./vcsa-ui-installer/lin64/installer file too.  I didn’t need to run it using sudo since the installation is to a remote ESXi host on the network, not the local machine.

Upon re-running the installer, you should progress past the point where the installer throws the error shown above and see the following screen.

Note that even for a “tiny” deployment, 10GB of RAM is required on the ESXi host.  A frankly obscene minimum requirement and hence where this blog post subsequently ends.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Separate IP Address Octets in Excel

If you find yourself with a spreadsheet containing hundreds of servers, and there’s an IP Address Column, you may want to create 4 additional columns, each containing each of the four octets of the IP Address.

This may be useful for grouping large numbers of hosts in a spreadsheet by subnet.

And this may be useful in grouping together servers in legacy infrastructure by application where that information doesn’t exist, since deployments of solutions across multiple servers are often grouped together in a range of IP addresses allocated to the projects by the Network admin.

These formulae are used to separate out each of the four octets.

A2:=LEFT(A1,FIND(".",A1)-1)
A3:=MID(A1,FIND(".",A1)+1,FIND(".",A1,FIND(".",A1)+1)-FIND(".",A1)-1)
A4:=MID(A1,FIND(".",A1,FIND(".",A1)+1)+1,FIND(".",A1,FIND(".",A1,FIND(".",A1)+1)+1)-(FIND(".",A1,FIND(".",A1)+1)+1))
A5:=MID(A1,FIND(".",A1,FIND(".",A1,FIND(".",A1)+1)+1)+1,LEN(A1)-FIND(".",A1,FIND(".",A1,FIND(".",A1)+1)+1))
Facebooktwittergoogle_plusredditpinterestlinkedinmail

PS4 Won’t log in to PSN

A major annoyance with the Sony Playstation 4 is that intermittently the PS4 won’t log in to PSN.  And unless it successfully logs into the PlayStation Network, pretty much all your apps are useless.    It’s really annoying that the PSN needlessly gets in the way of my ability to watch a programme on All4 or BBC iPlayer etc that have no dependency on the PSN whatsoever.

This is extremely frustrating when you know its connected to the internet and iPlayer or All4 service is operating normally, but you can’t do what you want to do, because you have a stupid (No)Play Station.  GRRRR!!!  It’s so Sony to do this too.  Not a fan.

If it wasn’t for my Gran Turismo addiction, I’d bin it.

So.  To improve matters, you need to set up your network connection again, only this time do it a little differently, specifying the DNS and MTU settings manually.

Primary DNS: 8.8.8.8

Secondary DNS: 8.8.4.4

MTU: 1473

Settings, Network, Set Up Internet Connection

Choose Wi-Fi or Cabled connection

Choose Custom

Automatic should be fine but I find it connects to the internet faster using a static IP Address.   If you do specify an address make sure it an address outside the scope on the DHCP server to prevent a potential IP address conflict on the network.  You may need to log onto the router and reduce the default DHCP scope accordingly, or set a reservation for the PS4.  Make sure your subnet mask is 255.255.255.0 unless you know better and your gateway likely looks like 192.168.x.1 where x is either 0 or 1.

Manually set the DNS addresses to those of the Google DNS Servers

Leave DHCP settings alone unless you want to receive an IP address from a DHCP server on your LAN other than the one on your wifi & router supplied by your ISP.

Set the MTU manually.  This pertains to the message transfer unit size on the network.  The default is 1500, but 1473 works better.

That’s it for the networking config.

Make sure it’s set to connect to the Internet

Something else you can check is in Settings, Users, Login Settings for your User, make sure the check box “Automatically log in to PS4 automatically” is checked.

Reset PSN Account Password

The thing that ultimately made the difference for me, was in Settings, Account Settings, Re-enter your PSN Password.  I don’t know what happens “behind the scenes” when you do this, but boom!  it logged straight into the PlayStation Network no problem and my apps all work perfectly.

Facebooktwittergoogle_plusredditpinterestlinkedinmail

Solaris P2V Process

This post is a work-in-progress.  The initial content is taken directly from Oracle’s own knowledge base here.  In time, it is my intention to augment the fundamental steps laid out below with real-world observations, screenshots, modifications and additional steps.

The P2V (Physical Server to Virtual Machine Migration is effectively the technical process of decommissioning a physical SPARC server by relocating its workload to an isolated instance of Solaris, running in software (known as a non-global zone) on a more powerful server, like a T8 or M8 running Solaris 11 and Oracle VM Server for SPARC (known as the global zone).

COLLECT REQUIRED INFORMATION FROM THE SOURCE SYSTEM.

Obtain the hostname:
# hostname

Obtain the hostid:
# hostid

Also see Host ID Emulation.

Obtain the root password.

View the software being run on the system:
# ps -eaf

Check the networking configuration on the system:
# ifconfig -a

View the storage utilized, for example, by viewing the contents of /etc/vfstab.

View the amount of local disk storage in use, which determines the size of the archive:
# df -k

Determine the packages and patches that are on the system. See pkginfo(1) for more information.
Examine the contents of /etc/system.

USE THE FLARCREATE COMMAND TO CREATE THE SYSTEM IMAGE.

This example procedure uses NFS to place the flash archive on the target Solaris system, but you could use any method to move the file.

You must be the global administrator in the global zone to perform this procedure.

Become superuser, or assume the Primary Administrator role.

Log in to the source system to be archived.

Change directories to the root directory.
# cd /

Use flarcreate to create a flash archive image file named s10-system on the source system, and place the archive onto the target system:
source-system
# flarcreate -S -n s10-system -L cpio /net/target/export/s10-system.flar
Determining which filesystems will be included in the archive…
Creating the archive…
cpio: File size of “etc/mnttab” has
increased by 4352068650 blocks
1 error(s)
Archive creation complete.

The target machine will require root write access to the /export file system. Depending on the size of the file system on the host system, the archive might be several gigabytes in size, so enough space should be available in the target filesystem.

Tip –
In some cases, flarcreate can display errors from the cpio command. Most commonly, these are messages such as File size of etc/mnttab has increased by 435. When these messages pertain to log files or files that reflect system state, they can be ignored. Be sure to review all error messages thoroughly.

CONFIGURING THE ZONE

Note that the only required elements to create a native non-global zone are the zonename and zonepath properties.  Other resources and properties are optional.  Some optional resources also require choices between alternatives, such as the decision to use either the dedicated-cpu resource or the capped-cpu resource.  See Zone Configuration Data for information on available zonecfg properties and resources.

You must be the global administrator in the global zone to perform this procedure.

Become superuser, or assume the Primary Administrator role.

To create the role and assign the role to a user, see Using the Solaris Management Tools With RBAC (Task Map) in System Administration Guide: Basic Administration.

Set up a zone configuration with the zone name you have chosen.

The name my-zone is used in this example procedure.
global# zonecfg -z my-zone

If this is the first time you have configured this zone, you will see the following system message:
my-zone: No such zone configured

Use ‘create’ to begin configuring a new zone.
Create the new zone configuration.

This procedure uses the default settings.
zonecfg:my-zone> create

Set the zone path, /export/home/my-zone in this procedure.
zonecfg:my-zone> set zonepath=/export/home/my-zone

Do not place the zonepath on ZFS for releases prior to the Solaris 10 10/08 release.
Set the autoboot value.

If set to true, the zone is automatically booted when the global zone is booted. Note that for the zones to autoboot, the zones service svc:/system/zones:default must also be enabled. The default value is false.
zonecfg:my-zone> set autoboot=true

Set persistent boot arguments for a zone.
zonecfg:my-zone> set bootargs=”-m verbose”

Dedicate one CPU to this zone.
zonecfg:my-zone> add dedicated-cpu

Set the number of CPUs.
zonecfg:my-zone:dedicated-cpu> set ncpus=1-2
(Optional) Set the importance.
zonecfg:my-zone:dedicated-cpu> set importance=10
The default is 1.

End the specification.
zonecfg:my-zone:dedicated-cpu> end

Revise the default set of privileges.
zonecfg:my-zone> set limitpriv=”default,sys_time”

This line adds the ability to set the system clock to the default set of privileges.
Set the scheduling class to FSS.
zonecfg:my-zone> set scheduling-class=FSS

Add a memory cap.
zonecfg:my-zone> add capped-memory

Set the memory cap.
zonecfg:my-zone:capped-memory> set physical=50m

Set the swap memory cap.
zonecfg:my-zone:capped-memory> set swap=100m

Set the locked memory cap.
zonecfg:my-zone:capped-memory> set locked=30m

End the memory cap specification.
zonecfg:my-zone:capped-memory> end

Add a file system.
zonecfg:my-zone> add fs

Set the mount point for the file system, /usr/local in this procedure.
zonecfg:my-zone:fs> set dir=/usr/local

Specify that /opt/zones/my-zone/local in the global zone is to be mounted as /usr/local in the zone being configured.
zonecfg:my-zone:fs> set special=/opt/zones/my-zone/local

In the non-global zone, the /usr/local file system will be readable and writable.
Specify the file system type, lofs in this procedure.
zonecfg:my-zone:fs> set type=lofs

The type indicates how the kernel interacts with the file system.

End the file system specification.
zonecfg:my-zone:fs> end
This step can be performed more than once to add more than one file system.
(Optional) Set the hostid.
zonecfg:my-zone> set hostid=80f0c086

Add a ZFS dataset named sales in the storage pool tank.
zonecfg:my-zone> add dataset

Specify the path to the ZFS dataset sales.
zonecfg:my-zone> set name=tank/sales

End the dataset specification.
zonecfg:my-zone> end

(Sparse Root Zone Only) Add a shared file system that is loopback-mounted from the global zone.

Do not perform this step to create a whole root zone, which does not have any shared file systems. See the discussion for whole root zones in Disk Space Requirements.
zonecfg:my-zone> add inherit-pkg-dir

Specify that /opt/sfw in the global zone is to be mounted in read-only mode in the zone being configured.
zonecfg:my-zone:inherit-pkg-dir> set dir=/opt/sfw
Note – The zone’s packaging database is updated to reflect the packages. These resources cannot be modified or removed after the zone has been installed using zoneadm.

End the inherit-pkg-dir specification.
zonecfg:my-zone:inherit-pkg-dir> end

This step can be performed more than once to add more than one shared file system.

Note –If you want to create a whole root zone but default shared file systems resources have been added by using inherit-pkg-dir, you must remove these default inherit-pkg-dir resources using zonecfg before you install the zone:
zonecfg:my-zone> remove inherit-pkg-dir dir=/lib
zonecfg:my-zone> remove inherit-pkg-dir dir=/platform
zonecfg:my-zone> remove inherit-pkg-dir dir=/sbin
zonecfg:my-zone> remove inherit-pkg-dir dir=/usr
(Optional) If you are creating an exclusive-IP zone, set the ip-type.
zonecfg:my-zone> set ip-type=exclusive

Note –Only the physical device type will be specified in the add net step.

Add a network interface.
zonecfg:my-zone> add net

(shared-IP only) Set the IP address for the network interface, 192.168.0.1 in this procedure.
zonecfg:my-zone:net> set address=192.168.0.1

Set the physical device type for the network interface, the hme device in this procedure.
zonecfg:my-zone:net> set physical=hme0

Solaris 10 10/08: (Optional, shared-IP only) Set the default router for the network interface, 10.0.0.1 in this procedure.
zonecfg:my-zone:net> set defrouter=10.0.0.1

End the specification.
zonecfg:my-zone:net> end

This step can be performed more than once to add more than one network interface.

Add a device.
zonecfg:my-zone> add device
Set the device match, /dev/sound/* in this procedure.
zonecfg:my-zone:device> set match=/dev/sound/*
End the device specification.
zonecfg:my-zone:device> end
This step can be performed more than once to add more than one device.

Add a zone-wide resource control by using the property name.
zonecfg:my-zone> set max-sem-ids=10485200
This step can be performed more than once to add more than one resource control.

Add a comment by using the attr resource type.
zonecfg:my-zone> add attr
Set the name to comment.
zonecfg:my-zone:attr> set name=comment
Set the type to string.
zonecfg:my-zone:attr> set type=string
Set the value to a comment that describes the zone.
zonecfg:my-zone:attr> set value=”This is my work zone.”
End the attr resource type specification.
zonecfg:my-zone:attr> end

Verify the zone configuration for the zone.
zonecfg:my-zone> verify

Commit the zone configuration for the zone.
zonecfg:my-zone> commit
Exit the zonecfg command.
zonecfg:my-zone> exit
Note that even if you did not explicitly type commit at the prompt, a commit is automatically attempted when you type exit or an EOF occurs.

Using Multiple Subcommands From the Command Line
Tip –The zonecfg command also supports multiple subcommands, quoted and separated by semicolons, from the same shell invocation.
global# zonecfg -z my-zone “create ; set zonepath=/export/home/my-zone”

INSTALL THE ZONE

The zoneadm command described in Part II, Zones and in the zoneadm(1M) man page is the primary tool used to install and administer non-global zones. Operations using the zoneadm command must be run from the global zone on the target system.

In addition to unpacking files from the archive, the install process performs checks, required postprocessing, and other functions to ensure that the zone is optimized to run on the host.

You can use an image of a Solaris system that has been fully configured with all of the software that will be run in the zone. See Creating the Image Used to Directly Migrate A Solaris System Into a Zone.

If you created a Solaris system archive from an existing system and use the -p (preserve sysidcfg) option when you install the zone, the zone will have the same identity as the system used to create the image.

If you use the -u (sys-unconfig) option when you install the zone on the target, the zone produced will not have a hostname or name service configured.

Caution –
You must specify either the -p option or the -u option, or an error results.

Installer options and description

-a archive

Location of archive from which to copy system image. Full flash archive and  cpio, gzip compressed cpio, bzip compressed cpio, and level 0 ufsdump are    supported. Refer to the gzip man page available in the SUNWsfman              package.

-d path

Location of directory from which to copy system image.

-d —

Use the -d option with the dash parameter to direct that the existing directory layout be used in the zonepath. Thus, if the administrator manually sets up the zonepath directory before the installation, the -d — option can be used to indicate that the directory already exists.

-p Preserve system identity.

-s Install silently.

-u sys-unconfig the zone.

-v Verbose output.

-b patchid

One or more -b options can be used to specify a patch ID for a patch installed in the system image. These patches will be backed out during the installation process.

The -a and -d options are mutually exclusive. The -p, -s, -u and -v options are only allowed when either -a or -d is provided.

PROCEDURE HOW TO INSTALL THE ZONE
Become superuser, or assume the Primary Administrator role.

Install the configured zone s-zone by using the zoneadm command with the install -a option and the path to the archive.
global# zoneadm -z s-zone install -u -a /net/machine_name/s-system.flar

You will see various messages as the installation completes. This can take some time.

When the installation completes, use the list subcommand with the -i and -v options to list the installed zones and verify the status.

Troubleshooting
If an installation fails, review the log file. On success, the log file is in /var/log inside the zone. On failure, the log file is in /var/tmp in the global zone.

If a zone installation is interrupted or fails, the zone is left in the incomplete state. Use uninstall -F to reset the zone to the configured state.

BOOT THE ZONE
You must be the global administrator in the global zone to perform this procedure.

If the -u option was used, you must also zlogin to the zone console and perform system configuration as described in Performing the Initial Internal Zone Configuration.

Become superuser, or assume the Primary Administrator role.

Use the zoneadm command with the -z option, the name of the zone, which is s-zone, and the boot subcommand to boot the zone.
global# zoneadm -z s-zone boot

When the boot completes, use the list subcommand with the -v option to verify the status.
global# zoneadm list -v

Facebooktwittergoogle_plusredditpinterestlinkedinmail