Continuous Integration with CruiseControl 2.7.1 and ClearCase

Abstract

Step-by-step installation of CruiseControl in Solaris 9 with ClearCase. This post details instructions on how to setup CruiseControl build directory in a flexible way and to write advanced CruiseControl project configuration files using plugins. Using the installation directory setup explained here we can run two continuous integration streams using 2 instances of CruiseControl running on 1 host machine. This installation guide can be suited to Windows machine with little efforts.

Common problems with solutions are listed at the end of this post.

Build environment and machine specs

The host machine architecture is SPARC with 2GB physical memory running Solaris 9, the SCM we are using is ClearCase.

Part 1: Installation and Directory Setup

Define two variables :
Let CC_INSTALL equal to /opt/cc2
and let CC_HOME equal to /opt/cc2/cruisecontrol-bin-2.7.1

Now download cruisecontrol 2.7.1 binary and extract it to /opt/cc2/cruisecontrol-bin-2.71

Create clearcase view for each product development streams (assuming you have not already created storage location for the views called asantoso_views, here I use asantoso_views storage location tag, and I have two ClearCase views to be integrated with CruiseControl):

% cleartool mkstgloc -view asantoso_views /private/asantoso/viewstore

% cleartool mkview -tag apc_cruisecontrol_3.0_stream1 -stgloc asantoso_views
% cleartool mkview -tag apc_cruisecontrol_3.0_stream2 -stgloc asantoso_views

We structure our directory like this (s1 and s2 are my stream names):

/opt/cc2
/opt/cc2/cruisecontrol-bin-2.7.1
/opt/cc2/streams/s1
/opt/cc2/streams/s1/logs
/opt/cc2/streams/s1/configs
/opt/cc2/streams/s2
/opt/cc2/streams/s2/logs
/opt/cc2/streams/s2/configs
Create this structure and set the directory read and write permissions to your user id.

The motivation using this setup:

• Upgrading to a new version of CruiseControl is a snap, just replace cruisecontrol-bin-2.7.1 and update the necessary path variables in file $CC_HOME/streams/s1/start.sh
• CruiseControl serializes a project into .ser file and as of 2.7.1 there’s no way to specify where the .ser files are stored, by default it is stored in the context where CruiseControl is started.
• We will have start.sh script in $CC_HOME/streams/s1 which executes CruiseControl so that the .ser files will be stored in there.
• We will store our configuration files in the $CC_HOME/streams/s1/configs directory and the build logs of each CruiseControl projects will be stored in $CC_HOME/streams/s1/logs directory
  The CruiseControl server log will be stored in $CC_HOME/streams/s1.

Now, create start.sh in $CC_HOME/streams/s1

#!/bin/bash

CC_INSTALL=/opt/cc2

CC_HOME=$CC_INSTALL/cruisecontrol-bin-2.7.1
CC_LOGS=$CC_INSTALL/streams/s1/logs
CC_CONFIG=$CC_INSTALL/streams/s1/configs/config.xml
CC_RMIPORT=1111
CC_WEBPORT=8081
CC_JMXPORT=8000
CC_NAME=APC_STREAM1

CLEARCASE_HOME=/opt/rational/clearcase/bin

PATH=$JAVA_HOME/bin:$CC_HOME:$PATH:$CLEARCASE_HOME
export PATH

cleartool startview apc_cruisecontrol_3.0_stream1

cruisecontrol.sh -jmxport $CC_JMXPORT -rmiport $CC_RMIPORT -cchome $CC_HOME
-webport $CC_WEBPORT -logDir $CC_LOGS $* -configfile $CC_CONFIG -ccname $CC_NAME

CruiseControl launcher options (case sensitive):

• -jmxport : jmx port
• -rmiport : rmi port
• -webport : jetty http server port
• -cchome : path to cc binary
• -logDir : path to base log directory where each CruiseControl project’s log is stored, if relative path is specifed, it is evaluated relative to context where CC is started.
• -configfile : the entry configuration file
• -ccname : the name to be displayed on jsp build reports page

Make sure you have JAVA_HOME environment variable set correctly.

The procedure for setting CC for second stream is nearly identical to the first, we need to assign distinct port and path values.

Open cruisecontrol.sh in $CC_HOME, add or modify the following lines on appropriate locations

Remove the line

CCDIR=`pwd`

Specify the cruisecontrol binary directory

CCDIR=`dirname $0`
export CCDIR

Optional step: delete the last two lines

$EXEC &
echo $! > cc.pid

Optional step: add two lines at the end of file

echo $EXEC
exec $EXEC

Uncomment CC_OPTS variable, allocate large JVM initial and maximum heap size depending on size of the host machine’s memory,

-Xms256m -Xmx1024m

Note that CruiseControl itself does not consume huge memory to run, but if you do not use fork in javac task it is advisable to set enough memory for CruiseControl

Part 2: CruiseControl Configuration File

Create config.xml in $CC_HOME/streams/s1

<cruisecontrol>  

<property name="cruisecontrol.settings.webport" value="8081"/>
<!-- the jetty port, as specified in the start.sh -->
<property name="cruisecontrol.settings.host" value="localhost"/>
<property name="cruisecontrol.web.url.base" value="http://${cruisecontrol.settings.host}:${cruisecontrol.settings.webport}" />
<property name="cruisecontrol.install" value="/opt/cc2" />
<!-- CC_INSTALL where cruisecontrol is installed -->
<property name="cruisecontrol.log.dir" value="logs" />
<!-- this path is evaluated starting from the context where CC is started -->

<system>
<configuration>
<threads count="1" />
</configuration>
</system>

<include.projects file="main.xml"/>

</cruisecontrol>

Lastly we include main.xml which is to define project templates.

Part 3: Configuration Templates and CruiseControl plugins

Create main.xml in $CC_HOME/streams/s1/configs
From this point on, it is specific to the type of projects that you have. In my case:
I have to abstract out the possible build project into three: periodic, manual, and timed into build templates through the use of CruiseControl plugins. The combination of using plugins and extraction of all the configuration values from these templates allows me to (for example) set the default email to be notified across all projects by modifying one property and furthermore I can set add email to be notified to a specific project.

Note: Similarly in here, if relative value is used in saveLogDir attribute, the path is evaluated from the context of where CC is started.
For example: If CC is started in /opt/cc2/streams/s1 by calling start.sh and saveLogDir value is logs then the final path is /opt/cc2/streams/s1/logs

A little thing to point at that in my build configuration contains the Apache Ivy ant task elements.

<cruisecontrol>

<!-- Global Default Properties -->

<property name="buildresult.url.base" value="${cruisecontrol.web.url.base}/buildresults" />
<property name="clearcase.viewname" value="apc_cruisecontrol_3.0_stream1" />
<property name="branch.name" value="dbapck3.0"/>
<property name="elisa.dev.dir" value="/view/${clearcase.viewname}/cm/elisa/dev" />
<property name="build.dir" value="${elisa.dev.dir}/ni/bigbuild/libs" /><!-- the place of compiled binaries -->
<property name="components.dir" value="${elisa.dev.dir}/ni/components" />
<property name="applications.dir" value="${elisa.dev.dir}/ni/applications" />
<property name="product.prefix" value="apc" />
<property name="extension" value="sh" /> <!-- ant script invoker file extension, example: sh or bat-->
<property name="mailhost" value="24.24.24.24" /> <!--  the ip of the outgoing mail server -->
<property name="buildmaster.email" value="agus.santoso@alcatel-lucent.com" />
<property name="email.subject.prefix" value="[CruiseControl]" />
<property name="project.manual.build.timeout" value="3600"/>
<property name="project.manual.build.target" value="publish_core_cruise_component" />
<property name="project.manual.id" value="apc-core"/>
<property name="project.manual.components.dir" value="${components.dir}" />
<property name="project.manual.mail.return" value="${buildmaster.email}"/>
<property name="project.manual.mail.subjectprefix" value="${email.subject.prefix}"/>
<property name="project.periodic.checkinterval" value="180"/>
<property name="project.periodic.components.dir" value="${applications.dir}" />
<property name="project.periodic.id" value="apc"/>
<property name="project.periodic.build.timeout" value="3600"/>
<property name="project.periodic.build.target" value="do_publish_components"/>
<property name="project.periodic.pause.start" value="2030"/>
<property name="project.periodic.pause.end" value="2230"/>
<property name="project.periodic.mail.return" value="${buildmaster.email}"/>
<property name="project.periodic.mail.subjectprefix" value="${email.subject.prefix}"/>
<property name="project.periodic.quietperiod" value="600"/>
<property name="project.periodic.requiremodification" value="true"/>
<property name="project.periodic.buildafterfailed" value="false" />
<property name="project.timed.components.dir" value="${applications.dir}" />
<property name="project.timed.timeout" value="3600" />
<property name="project.timed.time" value="2100" />
<property name="project.timed.id" value="apc"/>
<property name="project.timed.buildafterfailed" value="false" />
<property name="project.timed.mail.return" value="${buildmaster.email}"/>
<property name="project.timed.mail.subjectprefix" value="${email.subject.prefix}"/>
<property name="project.timed.build.target" value="do_publish_components" />

<!-- register cruisecontrol plugins -->
<plugin name="ivybuildstatus" classname="fr.jayasoft.ivy.cruise.IvyBuildStatus" />
<plugin name="ivyconditional" classname="fr.jayasoft.ivy.cruise.IvyConditionalPublisher" />
<plugin name="antlistener" classname="com.alcatel.util.cc.AntListener" />
<!--  mail publisher templates , add user emails here-->

<plugin name="project.m.emailset" skipusers="true"
classname="net.sourceforge.cruisecontrol.publishers.LinkEmailPublisher"
mailhost="${mailhost}" reportsuccess="always"
subjectprefix="${project.manual.mail.subjectprefix}"
buildresultsurl="${buildresult.url.base}/${project.name}"
returnaddress="${project.manual.mail.return}">
<ignore user="User" />
<always address="${project.manual.mail.return}" />
</plugin>

<plugin name="project.p.emailset" skipusers="true"
classname="net.sourceforge.cruisecontrol.publishers.LinkEmailPublisher"
mailhost="${mailhost}" reportsuccess="always"
subjectprefix="${project.periodic.mail.subjectprefix}"
buildresultsurl="${buildresult.url.base}/${project.name}"
returnaddress="${project.periodic.mail.return}">
<ignore user="User" />
<always address="${project.periodic.mail.return}" />
</plugin>

<plugin name="project.t.emailset" skipusers="true"
classname="net.sourceforge.cruisecontrol.publishers.LinkEmailPublisher"
mailhost="${mailhost}" reportsuccess="always"
subjectprefix="${project.timed.mail.subjectprefix}"
buildresultsurl="${buildresult.url.base}/${project.name}"
returnaddress="${project.timed.mail.return}" returnname="CruiseControl">
<ignore user="User" />
<always address="${project.timed.mail.return}" />
</plugin>

<plugin name="create.build.directories.bootstrapper"
classname="net.sourceforge.cruisecontrol.bootstrappers.AntBootstrapper"
antworkingdir="."
buildfile="build.xml"
uselogger="false"
usedebug="false"
target="create-build-directories">
<property name="target.dir" value="${build.dir}/${project.id}"/>
</plugin>

<!--
/***********************************************************************
*
*   Manual
*
************************************************************************
-->

<!-- MANUAL BUILD CONFIGURATION TEMPLATE -->

<plugin name="project.m.default.antbuilder"
classname="net.sourceforge.cruisecontrol.builders.AntBuilder"
buildfile="${project.manual.components.dir}/${project.manual.id}/build.xml"
target="${project.manual.build.target}"
uselogger="false"
antWorkingDir="${elisa.dev.dir}/ni"
antscript="${elisa.dev.dir}/ni/ant.${extension}"
saveLogDir="${cruisecontrol.log.dir}/${project.name}"
usedebug="false"
timeout="${project.manual.build.timeout}"/>
<plugin
name="project.m"
buildafterfailed="false"
classname="net.sourceforge.cruisecontrol.ProjectConfig">

<listeners>
<currentbuildstatuslistener file="${cruisecontrol.log.dir}/${project.name}/status.txt"/>
</listeners>

<log dir="${cruisecontrol.log.dir}/${project.name}">
<merge dir="${build.dir}/${project.manual.id}/test/testdata"/>
</log>

<modificationset>
<forceonly/>
</modificationset>

<bootstrappers>
<create.build.directories.bootstrapper>
<property name="target.dir" value="${build.dir}/${project.manual.id}" />
</create.build.directories.bootstrapper>
</bootstrappers>

<schedule interval="31536000" showProgress="true">
<project.m.default.antbuilder />
</schedule>

<publishers>

<artifactspublisher
dir="${build.dir}/${project.manual.id}/test/testreports"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="test"/>

<artifactspublisher
dir="${build.dir}/${project.manual.id}/coverageHistoryReport"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="coverageHistoryReport"/>

<artifactspublisher
dir="${build.dir}/${project.manual.id}/coverageReport"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="coverageReport"/>

<ivyconditional>
<project.m.emailset />
</ivyconditional>
</publishers>

</plugin>

<!--
/****************************************************************
*
*   Periodic
*
==***************************************************************
-->

<!-- PERIODIC BUILD CONFIGURATION TEMPLATE -->
<plugin name="project.p.default.antbuilder"
classname="net.sourceforge.cruisecontrol.builders.AntBuilder"
buildfile="${project.periodic.components.dir}/${project.periodic.id}/build.xml"
target="${project.periodic.build.target}"
uselogger="false"
antWorkingDir="${elisa.dev.dir}/ni"
antscript="${elisa.dev.dir}/ni/ant.${extension}"
saveLogDir="${cruisecontrol.log.dir}/${project.name}"
usedebug="false"
timeout="${project.periodic.build.timeout}"/>
<plugin
name="project.p"
requiremodification="${project.periodic.requiremodification}"
buildafterfailed="${project.periodic.buildafterfailed}"
classname="net.sourceforge.cruisecontrol.ProjectConfig">

<listeners>
<!--
<apc_lockfilelistener projectname="${project.name}"/>
-->

<currentbuildstatuslistener file="${cruisecontrol.log.dir}/${project.name}/status.txt" />
</listeners>

<log dir="${cruisecontrol.log.dir}/${project.name}">
<merge dir="${build.dir}/${project.periodic.id}/test/testdata" />
</log>

<bootstrappers>
<!--
<apc_lockfilebootstrapper projectname="${project.name}"/>
-->
<create.build.directories.bootstrapper>
<property name="target.dir" value="${build.dir}/${project.periodic.id}" />
</create.build.directories.bootstrapper>
</bootstrappers>

<modificationset quietperiod="${project.periodic.quietperiod}">
<ivybuildstatus ivyfile="${project.periodic.components.dir}/${project.periodic.id}/ivy.xml" />
<clearcase branch="${branch.name}"
viewpath="${project.periodic.components.dir}/${project.periodic.id}"/>
</modificationset>

<schedule interval="${project.periodic.checkinterval}">
<pause starttime="${project.periodic.pause.start}" endtime="${project.periodic.pause.end}" />
<project.p.default.antbuilder />
</schedule>
<publishers>

<artifactspublisher
dir="${build.dir}/${project.periodic.id}/test/testreports"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="test"/>
<artifactspublisher
dir="${build.dir}/${project.periodic.id}/coverageHistoryReport"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="coverageHistoryReport"/>

<artifactspublisher
dir="${build.dir}/${project.periodic.id}/coverageReport"
dest="${cruisecontrol.log.dir}/${project.name}" subdirectory="coverageReport"/>

<ivyconditional>
<project.p.emailset />
</ivyconditional>
</publishers>

</plugin>

<!--
/**************************************************************
*
*   Timed
*
***************************************************************
-->

<plugin name="project.t.default.antbuilder"
classname="net.sourceforge.cruisecontrol.builders.AntBuilder"
buildfile="${project.timed.components.dir}/${project.timed.id}/build.xml"
target="${project.timed.build.target}"
uselogger="false"
antWorkingDir="${elisa.dev.dir}/ni"
usedebug="false"
antscript="${elisa.dev.dir}/ni/ant.${extension}"
saveLogDir="${cruisecontrol.log.dir}/${project.name}"
timeout="${project.timed.timeout}"
time="${project.timed.time}" />
<!-- project template -->

<plugin name="project.t"
buildafterfailed="${project.timed.buildafterfailed}"
requiremodification="false"
classname="net.sourceforge.cruisecontrol.ProjectConfig">
<listeners>
<currentbuildstatuslistener file="${cruisecontrol.log.dir}/${project.name}/status.txt"/>
</listeners>

<log dir="${cruisecontrol.log.dir}/${project.name}">
</log>

<modificationset quietperiod="0">
<clearcase branch="${branch.name}"
viewpath="${project.timed.components.dir}/${project.timed.id}"/>
</modificationset>

<schedule>
<project.t.default.antbuilder/>
<!-- note : using composite as parent element of
antbuilder will render time based build of antbuilder not to take effect ,
therefore to execute two different targets instead of
using the composite tag, make a new target in the ant file that calls those two targets.
-->
</schedule>

<publishers>
<ivyconditional><project.t.emailset /></ivyconditional>
</publishers>

</plugin>

<!--
/******************************************************************
*
*   Project Stream
*
*******************************************************************
-->

<include.projects file="projects.xml"/>
</cruisecontrol>

Last on the bottom we include projects.xml

Part 4: Projects configuration

Create projects.xml in $CC_HOME/streams/s1/configs

Note that we can override the template and/or properties defined earlier in the parent config file but still bound to the context of the child config. The overriden template will still use properties in the parent config file unless overriden in the child.

<cruisecontrol>

<project
name="integration-periodic"
requiremodification="true"
buildafterfailed="false">

<listeners>
<currentbuildstatuslistener file="${cruisecontrol.log.dir}/${project.name}/status.txt" />
</listeners>

<log dir="${cruisecontrol.log.dir}/${project.name}">
</log>

<modificationset quietperiod="900">
<clearcase branch="${branch.name}" viewpath="${components.dir}"/>
<clearcase branch="${branch.name}" viewpath="${applications.dir}/apc"/>
</modificationset>

<schedule interval="${project.periodic.checkinterval}">
<pause starttime="${project.periodic.pause.start}" endtime="${project.periodic.pause.end}"/>
<ant
buildfile="${applications.dir}/apc/build.xml"
target="do_publish_components"
uselogger="false"
antWorkingDir="${elisa.dev.dir}/ni"
antscript="${elisa.dev.dir}/ni/ant.${extension}"
saveLogDir="${cruisecontrol.log.dir}/${project.name}"
usedebug="false"
timeout="${project.periodic.build.timeout}"
/>
</schedule>

<publishers>
<ivyconditional>
<project.p.emailset />
</ivyconditional>
</publishers>

</project>

<!-- daily build -->

<project.t name="integration-nightly">
<property name="project.timed.time" value="2100" />
<property name="project.timed.id" value="apc"/>
<property name="project.timed.components.dir" value="${applications.dir}"/>
<property name="project.timed.build.target" value="do_publish_components"/>
<modificationset quietperiod="0">
<clearcase branch="${branch.name}" viewpath="${components.dir}"/>
<clearcase branch="${branch.name}" viewpath="${applications.dir}/apc"/>
</modificationset>
<schedule interval="1">
<project.t.default.antbuilder />
</schedule>
</project.t>

<!-- override global properties in parent config -->

<property name="project.periodic.components.dir" value="${components.dir}" />
<property name="project.periodic.build.target" value="publish_core_cruise_component"/>

<!-- periodic component build -->

<project.p name="alcatel-mvc-plugins" >
<property name="project.periodic.id" value="alcatel-mvc-plugins"/>
</project.p>

<project.p name="apc-a7324-plug">
<property name="project.periodic.id" value="apc-a7324-plug"/>
</project.p>

<project.p name="apc-alcatel-plug">
<property name="project.periodic.id" value="apc-alcatel-plug"/>
</project.p>

<project.p name="apc-core" >
<property name="project.periodic.id" value="apc-core"/>
</project.p>

<project.p name="apc-g7342-plug" >
<property name="project.periodic.id" value="apc-g7342-plug"/>
</project.p>

<project.p name="apc-gpon-plug" >
<property name="project.periodic.id" value="apc-gpon-plug"/>
</project.p>

<project.p name="apc-inventory" >
<property name="project.periodic.id" value="apc-inventory"/>
</project.p>

<project.p name="apc-lkgplugin" >
<property name="project.periodic.id" value="apc-lkgplugin"/>
</project.p>

<project.p name="apc-northbound" >
<property name="project.periodic.id" value="apc-northbound"/>
</project.p>

<project.p name="application-framework" >
<property name="project.periodic.id" value="application-framework"/>
</project.p>

<project.p name="aware" >
<property name="project.periodic.id" value="aware"/>
</project.p>

<project.p name="axs-adapter" >
<property name="project.periodic.id" value="axs-adapter"/>
</project.p>

<project.p name="axs-utils" >
<property name="project.periodic.id" value="axs-utils"/>
</project.p>

<project.p name="belief-network" >
<property name="project.periodic.id" value="belief-network"/>
</project.p>

<project.p name="bulkcollector" >
<property name="project.periodic.id" value="bulkcollector"/>
</project.p>

<project.p name="bulkcollector-northbound" >
<property name="project.periodic.id" value="bulkcollector-northbound"/>
</project.p>

<project.p name="capabilities" >
<property name="project.periodic.id" value="capabilities"/>
</project.p>

</cruisecontrol>

Notice two important things:

• In integration nightly project I am overriding child elements of a template.
• I am overriding the global properties: project.periodic.components.dir and project.periodic.build.target to be used by the projects in this file. If I override one of the property twice or more, the last declared property will be used. This means that the order which the properties appear does not matter.

Common Problems and Solutions

• Invalid email user
  As far as 2.7.1 is concerned, when you click on the JSP build button, CC generates one fake modification created by fake “user” and it will send email to this address “user@mailhost” which is most probably does not exist. One way to suppress this is by adding <ignore user=”User” /> as a child of the email publisher, another way is by mapping this user to a valid email address <map user=”User” address=”validuser@validhost.com” />
• Time based build operates as a periodic build if declared inside a composite element.
  One solution is to create time based build as a separate project.
• logDir not found
  First note the context where CC is started, check the validity of the path -logDir and subsequently the validity of saveLogDir attribute value of the antbuilder. If relative path is used in saveLogDir the path is relative to the context of -logDir, and lastly check if read/write permission are set correctly.
• Killing CruiseControl JMX
  If some reasons, the JMX server is not terminated and stays in background you can kill its process
  get the pid by typing: ps -ef | grep mx4j