Thoughts and Tutorials about Continuous Integration, Application Integration, SOA, IT Architecture, WebSphere and Liferay

Nov 25, 2014

Using Docker behind a proxy


Docker draws currently a lot of attention on the (quite old) linux container concept.
Docker is a lightwight solution for system level virtualisiation which allows the deployment of operating systems and applications to be more portable (e.g. to different cloud platforms).

One usecase is to use docker behind a enterprise proxy system. Some times this proxy systems do SSL/TLS inspection, which means that this systems repleace SSL/TLS certificates.
Docker can be configured to use this proxy systems.

Configuration

To configure docker to work with an http(s) proxy edit the docker system configuration file.

vi /etc/sysconfig/docker

Add the HTTP_PROXY / HTTPS_PROXY environment variables. Authentication is also supported but optional.


If your proxy system uses NTLM / NTLMv2 authentication you have to use an intermediate proxy like cntlm, that handles the actual NTLM authentication. After setting up cntlm simply add the cntlm proxy adress to the docker system configuration file.

export HTTP_PROXY="http://localhost:3128"
export HTTPS_PROXY="http://localhost:3128"
HTTP_PROXY="http://localhost:3128"
HTTPS_PROXY="http://localhost:3128"

It is currently the best to add both variants - one time with "export" and one time without it.
The "export" is needed when starting docker with the services utitility (service docker start). When starting docker with the old init.d scripts (/etc/rc.d/init.d/docker start) export is not needed.
When the proxy settings are ignored docker prints out:
dial tcp 162.242.195.84:443: connection timed out

If the proxy system does SSL/TLS inspection, the proxy usually signs the particular server certificate with its own proxy root certificates.
This leads to the following docker exception when you try to pull images from the public docker repository:

x509: certificate signed by unknown authority

To solve this add the proxy root certificate to the trusted certificates of your docker host (underlying linux systems that hosts docker binaries). However the setup depends on your linux distribution.
I will explain it based on CentOS Linux (and Red Hat Enterprise Linux).
First copy the proxy root certificate to the ca-trust area.

cp /tmp/<proxy-root-certificate>.crt /etc/pki/ca-trust/source/anchors/
 
Then update the trusted certificates 

update-ca-trust extract
  
After that docker needs to be restarted.

service docker restart


Further readings:

http://kb.kerio.com/product/kerio-connect/server-configuration/ssl-certificates/adding-trusted-root-certificates-to-the-server-1605.html 
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=767441 

Nov 16, 2014

Continuous Delivery with WebSphere



Modern application environments demand a fast release cycle for new versions of an application.
Continuous Integration defines that every change (commit, etc.) is iteratily tested to not break a deployable application. So every code change is tested to be integrable in the existing code base.

Continuous Delivery as an extension of Continuous Integration, demands that every deployable artifact resulting from Continuous Integration is actually automatically deployed to a staging enviroment for some kind of (manual) acceptance tests.

As for most time in IT terms are not excactily defined. So don't be confused when searching accurate definitions for that terms. Software vendors often use Continuous Delivery and Continuous Integration (and Continuous Deployment) in different ways.

In this post i will describe how to setup a Continuous Delivery cycle with a IBM WebSphere Application Server as deployment target. Though WebSphere is an closed source properteriy product, it offers interfaces for automatic deployments. So its possible to bring the closed source and the open source world a bit more closer together.


The basic setup


I will use Git 1.8.1.2 as SCM, Jenkins 1.587 as Continuous Integration-Server and of course WebSphere 8.5.5. as the deployment target. For directly triggering a build after a commit i will use the git post commit hook. For deploying the application i will use the WebSphere Deployer Plugin.

I also created a maven based sample application with a basic JUnit test. The project can be downloaded here.

Installation of the tools and plugins

Basically it doesnt matter if the tools will be installed on seperate machines. However i installed them all on my local PC to keep the complexity and the debug effort low.
I skip the installation of the basic tools as they are straight forward and continue with the plugin installation for Git and Jenkins.

Git post commit hook

Git has some built-in extensions called hooks.
To activate a hook locate a sample script in your Git installation and simply edit it. In this example we need the post-commit.sample file.

<git-installation>/share/git-core/templates/hooks/post-commit.sample

The file just needs to contain a single line that tells git to call a jenkins url that triggers the build in jenkins.

curl http://localhost:8080/git/notifyCommit?url=<path-of-git-repository>

To actually install the post commit hook for your local git repository, simple rename the file to post-commit (without .sample) and copy it to

<git-repository>/.git/hooks/

WebSphere Deployer Plugin

The easiest way to deploy your changes to WebSphere is the WebSphere Deployer Plugin for Jenkins.
To install the plugin go the jenkins web interface and navigate to

Manage Jenkins > Manage Plugins > Available

and search the "WebSphere Deployer Plugin". Select the plugin and click "Download now and install after restart". Choose "Restart Jenkins when installation is complete and no jobs are running" to restart Jenkins directly afterwards.
Installation of the WebSphere Deployer Plugin
The successfully connect to WebSphere, the WebSphere Deployer Plugin needs additional libraries.
For WebSphere Application Server 8.5.5 (used in this example) this libraries are the com.ibm.ws.admin.client_8.5.0.jar and the com.ibm.ws.orb_8.5.0.jar. Both files are located in

<WebSphere-Installation>/runtimes/

Copy the JARs to

<Jenkins-Installation-Path>/plugins/websphere-deployer/WEB-INF/lib

When working with WebSphere liberty, other libaries are required. Refer to the official documentation for more information.
After copying the libraries Jenkins needs another restart.

Building the pipeline

After the basic setup is completed and the plugins are installed the build pipeline in Jenkins can be configured.
Go to the Jenkins web interface and click "New Item". Type in your item  name and select "Maven project" as the item type.
New Maven Job in Jenkins

After clicking "OK" the job config dialog appears.

Under Source Management select Git and enter your git-repository. Since i am setting up this example on my local machine i just enter the local Git repository path. If you have a Git management tool like gerrit or Atlassian Stash you need to enter the approperiate URLs.
Source Code Management for Git


Under "Build triggers" select "Poll SCM". Ignore the warning Jenkins prints out for not specifying any scheduling interval. No schedule intervals are needed when using post-commit hooks. However it is necessary to active "Poll SCM".

Activate Poll SCM

Under "Build" enter the location of your base pom.xml file and your maven goals (most times "clean install")
Maven Settings and Goals


Since the Continous Delivery dogma demands to only deploy (stage) working builds i
recommand to activate post build steps (like deployments) only for working builds. So select "Run only if build succeeds" under "Post Steps".

Post Steps settings


To enable the deployment to WebSphere after a successfull build click "Add post-build action" and select "Deploy To IBM WebSphere Application Server" or "Deploy To IBM WebSphere Liberty Profile". In this example i will use the full blown IBM WebSphere Application Server so i choose the first option.
WebSphere Deployer Plugin as post-build action

The WebSphere Deployer Plugin is highly configurable. Need do not need to fill out all values. However basic settings are the WebSphere host name (localhost) the Connector Type (SOAP as default) and the corrosponding Port (8882 here - default is 8879). The port settings can be retrieved from the WebSphere settings (Application servers > server1 > Ports > SOAP_CONNECTOR_ADDRESS).

For the EAR Path you should select the last successful build artifact by specifing

modules/**/lastSuccessful/**/continuous-dev-cycle-example-*.war

The settings for the "Target Node", the "Target Cell" and the "Target Server" depend on your installation. However all values can be retrieved from WebSpheres Administrative Console.
Also choose Java EE 6 as the "Generated EAR Level".

WebSphere Deployer Plugin setup

Thats it. Click "Save" to create the job (aka the build pipeline).

The result

We build a basic continuous delivery pipeline for WebSphere.
Commiting a change
Changing the application
Commiting the change
results in a curl call on a Jenkins action url. The call on this url says Jenkins to start a poll for the corresponding git repository.

Jenkins builds the maven project, executes the JUnit-Tests.
Building and testing the change

After a successfull build the WebSphere Deployer Plugin deploys the application to the WebSphere Application Server.
Deploying the change