Building in isolation with Docker

by Bruce Szalwinski

Background

Over at Device Detection, I wrote about creating an Apache Handler that could be used to do real time device detection.  The handler has a number of dependencies on other Perl modules, 51Degrees, JSON, Apache2::Filter, Apache2:RequestRec, Apache2::RequestUtil, etc.  And those modules have dependencies as well.  I wanted our build server, Bamboo, to do the building of my module without the nasty side effects of having to install third party libs into all of the build agents.  In the Maven world, I would just add all of these dependencies to my pom.xml and maven would download the dependencies from the repository into my local environment.  At build time, Bamboo would take care of establishing a clean environment, download my dependencies and most importantly, when the build was complete, the plates would be wiped clean ready to serve another guest leaving no traces of my build party.  The challenge then, how to do this in Perl world.  Spoiler alert, full source code is available at DeviceDetection.

Enter Docker

homer-isolation-tank

The fancy new way for developers to deliver applications to production is via Docker.  Developers define the application dependencies via a plain old text file, conventionally named Dockerfile.  Using the Docker toolkit, the Dockerfile is used to build a portable image that can be deployed to any environment.  A running image is known as a container, which behaves like an operating system running inside of a host operating system.  This is a lot like VMs but lighter weight.  Developers are now empowered to deliver an immutable container to production.  Let’s see if this can also be used to provide an isolated build environment.

docker-big-picture

For the big picture folks, here is what we are trying to do.  We’ll start by defining all of the application dependencies in our Dockerfile.  We’ll use the Docker toolkit to build an image from this Dockerfile.  We’ll run the image to produce a container.  The container’s job will be to build and test the Perl modules and when all tests are successful, produce an RPM.

Building the image was an iterative process and I had fun dusting off my sysadmin hat.  Here is where I finally ended up.

FROM google/debian:wheezy
RUN    apt-get -y install make gcc build-essential sudo
RUN    apt-get -y install apache2-threaded-dev
RUN    apt-get -y install libapache2-mod-perl2
RUN    apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl
RUN apt-get -y install rpm

Let’s break this down.   The first non-comment line in the Dockerfile must be the “FROM” command.  This defines the image upon which our image will be based.  I’m using the “google/debian” image tagged as “wheezy”.   Think of images as layers.  Each image may have dependencies on images below it.  Eventually, you get to a base image, which is defined as an image without a parent.

FROM google/debian:wheezy

The RUN command is used to add layers to the image, creating a new image with each successful command.  The 51Degrees Perl module is built using the traditional Makefile.PL process, so we start by installing the make, gcc and build-essentials.  Containers generally run as root so we wouldn’t normally need to install sudo, but our handler uses Apache::Test for its unit test and Apache::Test doesn’t allow root to create the required httpd process.  So we will end up running our install as a non-root user and give that user sudo capabilities.  More about that in a bit.

RUN     apt-get -y install make gcc build-essential sudo

Next, we install our apache environment.  With Apache2, there is a pre-fork and a threaded version which has to do with how apache handles multi-processing.  For my purposes, I didn’t really care which one I picked. It was important however to pickup the -dev version as this includes additional testing features.

RUN     apt-get -y install apache2-threaded-dev

Next, we install mod-perl since the device detector is a mod perl handler.

RUN     apt-get -y install libapache2-mod-perl2

Next, add our Perl dependencies.  Each Linux distro has its own way of naming Perl modules.  Why?  Because they can.  Debian uses “lib” prefix and “-perl” suffix, converts “::” to “-“, and lower cases everything.  To install the Perl module known as “Test::Harness”, you would request “libtest-harness-perl”.

RUN     apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl

And since we’ll be delivering a couple of RPMs at the end of this, we install the rpm package.

RUN apt-get -y install rpm

With the Dockerfile in place, it is time to build our image.  We tell docker to build our image and tag it as “device-detection”.  We tell docker to look in the current directory for a file named Dockerfile.

$ docker build -t device-detection .

Time for some coffee as docker downloads the internet and builds our image.  Here is the pretty version of the log produced after the initial construction of the image.  If there are no changes to the Dockerfile, then the image is just assembled from the cached results.  The 12 character hex strings are the ids (full ids are really  64 characters long) of the images that are saved after each step.

Sending build context to Docker daemon 2.048 kB
Sending build context to Docker daemon
Step 0 : FROM google/debian:wheezy
 ---> 11971b6377ef
Step 1 : RUN apt-get -y install make gcc build-essential sudo
 ---> Using cache
 ---> 2438117da917
Step 2 : RUN apt-get -y install apache2-threaded-dev
 ---> Using cache
 ---> 41f878809025
Step 3 : RUN apt-get -y install libapache2-mod-perl2
 ---> Using cache
 ---> 43eadc4ec9eb
Step 4 : RUN apt-get -y install libtest-harness-perl libtap-formatter-junit-perl libjson-perl
 ---> Using cache
 ---> 106d5f017b5c
Step 5 : RUN apt-get -y install rpm
 ---> Using cache
 ---> fd0dc5f192d6
Successfully built fd0dc5f192d6

Use the docker images to see the images that have been built.  The ubuntu/14.04 was before I got religion and started using google/debian.

$ docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             VIRTUAL SIZE
device-detection    latest              2f2b3e38e8c1        29 minutes ago      358.6 MB
ubuntu              14.04               d0955f21bf24        2 weeks ago         188.3 MB
google/debian       wheezy              11971b6377ef        9 weeks ago         88.2 MB

Running the Image

At this point, we have an image that contains our isolated build environment.  Now we are ready to do some building by running the image.  In Docker terms, a running image is known as a container.  The build-docker script will be used to produce a container.   When we create our Bamboo build plan, this is the script that we will execute.

#!/bin/bash
docker run --rm -v $PWD:/opt/51d device-detection:latest /opt/51d/entry.sh

The –rm removes the container when finished. The -v mounts the current directory as /opt/51d inside of the container.  The device-detection:latest refers to our image that we just built.  And finally, the /opt/51d/entry.sh is the command to execute inside of the container.

#!/bin/bash
adduser --disabled-password --gecos '' r
adduser r sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
su -m r -c /opt/51d/build

The entry.sh script will be executed inside of the container.  To test the handler, we’ll need the vendor’s module installed.  And to install modules, we need to have root privileges.  We are going to use Apache::Test to test the handler but Apache::Test won’t let us start the httpd process as root.  The solution is to create a new user, r, and give him sudo capabilities.  With that in place, we hand off execution to the next process /opt/51d/build.

That all worked well on my local environment, but something interesting happened when I went to deploy this from Bamboo. The owner of the files in the container turned out to be the user that built the container.  I was the one building the container in my local environment, but I wasn’t the one building the container inside of Bamboo.  When the user ‘r’ attempted to create a file, it got a permission denied error because the directories are not owned by him.  I discovered this by having Bamboo list the files from inside the running container. They are owned by the mysterious user with UID:GID of 3366:777.

build   08-Apr-2015 09:28:35    /opt/51d:
build   08-Apr-2015 09:28:35    total 28
build   08-Apr-2015 09:28:35    drwxr-xr-x 7 3366 777 4096 Apr  8 16:28 51Degrees-PatternWrapper-Perl
build   08-Apr-2015 09:28:35    drwxr-xr-x 5 3366 777 4096 Apr  8 16:28 CDK-51DegreesFilter
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  530 Apr  8 16:28 build
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  120 Apr  8 16:28 build-docker
build   08-Apr-2015 09:28:35    drwxr-xr-x 2 3366 777 4096 Apr  8 16:28 docker
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  146 Apr  8 16:28 entry.sh
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 3366 777  497 Apr  8 16:28 rpm.sh

We can use this UID:GID information when creating our user. The stat command can be used to return the UID and GID of a file.  We’ll create a group associated with the group that owns the /opt/51d directory and then we’ll create our user with the UID and GID associated with the owner of the directory.  Our modified entry.sh script is then:

#!/bin/bash
addgroup --gid=$(stat -c %g /opt/51d) r
adduser --disabled-password --gecos '' --uid=$(stat -c %u /opt/51d) --gid=$(stat -c %g /opt/51d) r
adduser r sudo
echo '%sudo ALL=(ALL) NOPASSWD:ALL' >> /etc/sudoers
su -m r -c /opt/51d/build

And we can see that user ‘r’ is now the “owner” of the files.

build   08-Apr-2015 09:28:35    /opt/51d:
build   08-Apr-2015 09:28:35    total 28
build   08-Apr-2015 09:28:35    drwxr-xr-x 7 r r 4096 Apr  8 16:28 51Degrees-PatternWrapper-Perl
build   08-Apr-2015 09:28:35    drwxr-xr-x 5 r r 4096 Apr  8 16:28 CDK-51DegreesFilter
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  530 Apr  8 16:28 build
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  120 Apr  8 16:28 build-docker
build   08-Apr-2015 09:28:35    drwxr-xr-x 2 r r 4096 Apr  8 16:28 docker
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  146 Apr  8 16:28 entry.sh
build   08-Apr-2015 09:28:35    -rwxr-xr-x 1 r r  497 Apr  8 16:28 rpm.sh

With the user setup, entry.sh hands off control to the build script to do the heavy lifting.  Here we setup our apache environment and start building the two Perl modules.  The () is a convient bash-ism that creates a sub-process, leaving us in the current directory when completed.  And the PERL_TEST_HARNESS_DUMP_TAP is an environment variable recognized by Tap::Formatter:Junit package.  Unit tests will live at the location specified by this variable.

#!/bin/bash

source /etc/apache2/envvars
export APACHE_TEST_HTTPD=/usr/sbin/apache2
export PERL_TEST_HARNESS_DUMP_TAP=/opt/51d/CDK-51DegreesFilter/dist/results

(cd /opt/51d/51Degrees-PatternWrapper-Perl && \
        perl Makefile.PL && \
        make && \
        make dist && \
        sudo make install && \
        ../rpm.sh FiftyOneDegrees-PatternV3-0.01.tar.gz)

(cd /opt/51d/CDK-51DegreesFilter && \
        perl Build.PL && \
        ./Build && \
        ./Build test && \
        ./Build dist && \
        ../rpm.sh CDK-51DegreesFilter-0.01.tar.gz)

When the build script completes, we are done and the container is stopped and removed.  Because we have mounted the current directory inside of container, artifacts produced by the container are available after the build completes.  This is exactly the side effect we need to have.  We can publish the tests results produced by the build process as well as the RPMs.  And we have accomplished the goal of having an isolated build environment, D’oh!

Fun things learned along the way

Inside of the container, Apache::Test starts an httpd server on port 8529. It then tries to setup the mod_cgi library by binding a socket to a filehandle in the /opt/51d/CDK-51DegreesFilter/t/logs directory via this directive:


<IfModule mod_cgid.c>
    ScriptSock /opt/51d/CDK-51DegreesFilter/t/logs/cgisock
</IfModule>

The httpd server had issues with this, not sure why, perhaps because Docker is binding the file system as well.  I resolved it by moving the ScriptSock location to /tmp/cgisock.  More details on this conundrum are available at stackoverflow where I asked and answered my own question,  http://stackoverflow.com/questions/29424132/error-accessing-cgi-script-inside-docker-container-operation-not-permitted-cou.

Device Detection

by Bruce Szalwinski

Background

The good folks that power Apache Mobile Filter use version 2 of the Device Repository from 51Degrees and currently have no plans for updating their software to use version 3.  Since we currently use the AMF handler to do device detection and since 51Degrees has announced the end of life for version 2, this provides an opportunity for us to write our own handler.  We attempted to do this in version 2 days, but there was no Perl API offered from 51Degrees and the C code was pretty shaky.  With version 3, the 51 Degrees folks now offer a Perl API that wraps around a much more robust C API.  With that, the stage is set to tackle writing our own Apache Handler.   I’ll use the Apache::Test module to help drive the development.  This article from last decade was very helpful in learning how to use this powerful module.  Full source code is available at DeviceDetection.

Requirements

Analyze web traffic by a user specified set of device properties.

Implementation

An Apache Handler allows for the customization of the default behavior of the web server.  We will write a handler that reads the user agent from the request, detects the device associated with the user agent, creates environment variables for each requested device property and writes the values to a log file.  Let’s get started.

Write tests first

To test our handler, we’ll send requests to an apache server, passing in various user agent strings and validating that we receive known device id values.

use strict;
use warnings FATAL => 'all';

use Apache::TestTrace;
use Apache::Test qw(plan ok have_lwp);
use Apache::TestRequest qw(GET);
use Apache::TestUtil qw(t_cmp);
use Apache2::Const qw(HTTP_OK);

use JSON;

plan tests => 6, have_lwp;

detect_device('','15364-5690-17190-18092');
detect_device('unknown', '15364-5690-17190-18092');
detect_device(
 "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36",
  "15364-18110-25377-18092");

sub detect_device {
  my ($user_agent, $device_id) = @_;

  Apache::TestRequest::user_agent(
    reset => 1,
    agent => $user_agent
  );

  my $response = GET '/cgi-bin/index.cgi';
  my $json = decode_json $response->content;

  debug "response", $response;

  ok defined($json->{_51D_ID}) eq 1;
  ok $json->{_51D_ID} eq $device_id;
}

Great, we have a unit test and it fails miserably because we don’t have a apache server.  We need an apache server that we can start, stop and configure for every test run.  Conveniently, the Apache::Test module provides a “whole, pristine and isolated” apache server at our disposal.  Cool, we have a server.  Next, the handler will push device properties into the environment via the subprocess environment table so we need a way to capture those values.  Line 28 shows how the unit tests make a call to a CGI script.  The CGI script will simply grab the variables pushed into the environment by the handler and return them to the test.


#!/usr/bin/perl

use CGI qw(:standard -no_xhtml -debug);
use JSON;

print header('application/json');

my %properties;

while ( my ($key, $value) = each(%ENV)) {
  if ( $key =~ /^_51D/) {
    $properties{$key} = $value;
  }
}

print encode_json \%properties;

Ok, so we have a failing test, a web server, and a way of communicating between the two.  We have a little bit of wiring to do to let the server know about our CGI script as well as our handler.  By convention, Apache::Test will look for a file called t/conf/extra.conf.in.  This file contains configuration directives that will be added to httpd.conf before starting the server.  We’ll take this opportunity to configure the execution of our index.cgi test harness, configure our log format and setup our handler.


PerlSwitches -w

ScriptAlias /cgi-bin @ServerRoot@/cgi-bin
<Location /cgi-bin>
  SetHandler cgi-script
  Options +ExecCGI +Includes
</Location>

LogFormat "%{_51D_ID}e|%{User-Agent}i" combined

PerlTransHandler +CDK::51DegreesFilter
PerlSetEnv DeviceRepository @ServerRoot@/data/51Degrees-Lite.dat
PerlSetEnv DevicePropertyList ScreenPixelsHeight,BatteryCapacity
PerlSetEnv DevicePrefix _51D

Man, when is this guy ever going to get around to writing some code?  Almost there. The Apache::TestRunPerl and Apache::TestMM modules combine together to provide all that is necessary to start, configure and stop Apache, as well as run all of the individual unit tests.  These get added into our Build.PL script.  The test action normally just executes tests.  We need to subclass this action so that we can start the server before the  tests executes and stop it when complete.   It would also be nice to produce Junit style output of the test results so that they can be published by the build server.


use Module::Build;
use ModPerl::MM ();
use Apache::TestMM qw(test clean);
use Apache::TestRunPerl ();
use IO::File;

my $class = Module::Build->subclass(
    class => 'CDK::Builder',
    code => q{
	sub ACTION_test {
	    my $self = shift;
	    $self->do_system('t/TEST -start-httpd');
	    $self->SUPER::ACTION_test();
	    $self->do_system('t/TEST -stop-httpd');
	}
    }
);

my $build = $class->new (
  module_name => 'CDK::51DegreesFilter',
  license => 'perl',
  test_file_exts => [qw(.t)],
  use_tap_harness => 1,
  tap_harness_args => {
    sources => {
      File => {
        extensions => ['.tap', '.txt'],
      },
    },
    formatter_class => 'TAP::Formatter::JUnit',
  },
  build_requires => {
      'Module::Build' => '0.30',
      'TAP::Harness'  => '3.18',
  },
  test_requires => {
      'Apache::Test' => 0,
  },
  requires => {
      'mod_perl2' => 0,
      'FiftyOneDegrees::PatternV3' => 0,
      'JSON' => 0,
      'Apache2::Filter' => 0,
      'Apache2::RequestRec' => 0,
      'Apache2::RequestUtil' => 0,
      'Apache2::Log' => 0,
      'Apache2::Const' => 0,
      'APR::Table' => 0
  }
);

Apache::TestMM::filter_args();
Apache::TestRunPerl->generate_script();

$build->create_build_script;

 

Handler

Finally.  At this point, writing the handler is pretty anti climatic.  It reads the user agent from the header and passes it to the getMatch method from 51Degrees.  A set of device properties are returned as a JSON object.  Each requested property, defined by DevicePropertyList, is added to the environment, via subprocess_env().  The AMF handler used a caching mechanism to avoid detection costs for previously seen user agents.  The 51D folks said the new version was faster, so I wouldn’t need it.  Performance testing will prove this out.


sub handler {
  my $f = shift;

  my $user_agent=$f->headers_in->{'User-Agent'} || '';
  my $json = FiftyOneDegrees::PatternV3::getMatch($dataset, $user_agent);
  my %properties = %{ decode_json($json) };

  while ( my ($key, $value) = each(%properties) ) {
    my $dkey = uc("${prefix}_${key}");
    $f->subprocess_env($dkey => $value);
  }

  return Apache2::Const::DECLINED;
}

 

Performance

To test performance, I setup Jmeter with 5 threads on a sandbox machine and looped over a set of 350K unique user agents.  The Jmeter instance made requests to apache running on a second sandbox machine with the new handler installed.  With 2,428,264 requests under its belt, the average response time is 10ms.  For v2, with caching, the average response time was 16ms.

How to Find Happiness In Your Job

By Kris Young (@thehybridform)

Are you happy with your job? Do you come into work, sit down, and then sigh with your head hung low? If so then you are not alone.

According to Forbes[1] 52% of Americans are not satisfied with their work. Some surveys place this number above 80%. This begs the question why? Do we study the wrong subject in school or apply for the wrong job? Maybe we followed the wrong advice from our high school guidance councilor.

Psychology Today [2] says we spend 90,000 hours of our lifetime working. Why not enjoy our time. The question is, “How do we find a job we love?”

Traits of a Job to Love

There are three areas that people who love their job have in common. These can be used as a measurement of job satisfaction:

  • Creativity
  • Impact
  • Control

The first two areas feed our natural human desire to belong. We are social beings and seek confirmation and a place in our community to achieve self worth. We do this by helping others, which in turn helps ourselves.

At the same time we wish for freedom, a way to disengage from the pack. The last area, Control, addresses this instinct.

Creativity

Using one’s imagination or original ideas to express oneself is a human trait we all share. You can find creativity in all professions. Artists and musicians have obvious expressive qualities. How about IT professionals? Or bartenders? Absolutely!

Conjuring an internal thought to realization for an audience is the heart of creativity. Even if that audience is you. Be it a new algorithm or a business process. Think about what you do at work and how you can be creative. You may already be on the road to job satisfaction.

Impact

Knowing our creativity matters to other people is important. Who wants to toil away on a problem nobody cares about? An impact gives you validation of your hard work and results in a feedback loop, which in turn compels you continue creating. And on the loop goes.

Feedback can be in the form of pay raise, bonus, more autonomy, or peer recognition of a job well done. All feedback gives us a chance to adjust into the next loop. We will tend to veer towards creativity that gives us the most positive outcome and the biggest impact.

Control

Living life on your terms is a very seductive prospect. Having the choice to do what you want when you want in you own time is a right we all share. This aspect of your job can be summed up as career capitol. The more career capitol you have the more control you will have in you daily life. It’s the freedom to express yourself and control your destiny.

You want to introduce a new process to the business or a new ingredient to a recipe? Can you leave early on Friday to pick the kids? If you feel you need permission to start a new creative endeavor then you need more career capitol in the bank. But how do you acquire this?

The Advice

At some point in your life you will hear someone say, “Follow your passion”. This could come form someone very successful or smart. A person you respect or wish to live up to. He or she seems legit; why not give the advice a try?

I’m here to tell you this is the worst advice to give or follow.

Sure there are people out there who have an inherit passion at a young age; those people are few and far between. Chances are you have no idea what you passion is anyway. And it will change overtime. If you don’t have a passion then you may feel left out or somehow not able to be happy with your work. This is demoralizing and contributes to the dissatisfaction with your job, a vicious cycle. FYI passion is a synonym for suffering and who want’s to suffer.

Better advice is to adopt a craftsmen mindset and become “So good they cannot ignore you.”[3] Having a craftsmen mindset is far more important than a passion mindset.

The Craftsmen Mindset

Achieving craftsmanship of an in demand and/or rare skill will present you with career capitol. Which goes a lot further in creating work you love.

This means working hard, learning as much as you can, and becoming the expert. Some people say the trick is to choose something you like, but that’s is not really important. You may not know what you like or where to start. If you do great! If not try something new. What ever you choose adopt a craftsmen mindset. But remember the more rare and in demand the skill, the faster you will accrue career capitol.

Adopting a craftsmen mindset will lead you to your passion. You will have creativity, impact, and control. You will find yourself creating the work you love to do. Your passion will find you.

Short Story

If Steve Jobs followed his passion we would not have the iPhones, iPads, or MacBooks. His passion was studying Zen Buddhism. Though a fluke meeting and noticing some people were excited about model-kit computers an empire was born.

He became passionate about technology and science. He became so good you could not ignore him.

[1] http://www.forbes.com/sites/susanadams/2014/06/20/most-americans-are-unhappy-at-work/

[2] https://www.psychologytoday.com/blog/thrive/201102/finding-happiness-work

[3] http://lifehacker.com/5947649/steve-martins-advice-for-building-a-career-you-love

Follow

Get every new post delivered to your Inbox.

Join 33 other followers

%d bloggers like this: