Busy times...

Yikes, Christmas is over us. And I never got around to posting anything since late summer.

Apologies! It is not that this blog is dead (it just smells a bit peculiar), but I have been hung up on a number of high priority issues.

I (seriously) hope the issues will be resolved early in the new year, so I can blog a little again.

In the pipeline (yes, let's make some vapor-blogging):
  • Enterprise deployment of IBM Rational products.
    As the name suggests, the "rational" approach would probably be to use IBM's installer. But no - I just had to go and invent our own deployment strategy. It is slim, it is fast, and we have full control of the deployment. What's not to like?
  • Authorized Job Service
    We do not use our Continuous Integration builds in production for various reasons. Instead we have a manually invoked build server. To make it work I needed a way to authenticate and authorize users between servers.
    And as I couldn't figure out how to make a single sign-on solution that would use the user's existing Windows session credentials (with JAAS), I had to roll my own using SSPI.
    I suspect this will be a blog entry where someone points out the obvious to me...
  • Our SVN backup and restore setup.
    The repositories are backed up at night and restored to a different server during the day. Lovely stuff. I hope we'll never have to make use of it - but if we do, I am much more comfortable with it than with our previous CVS setup.
  • And before the new year expires, something about our new documentation system (it needs to mature a bit yet).
    We use the Mylyn WikiText plugin to write documentation placed in the repository with the source. On project builds, the documentation is translated to HTML. We have extended the syntax (Confluence) a bit to add extra features...
Hopefully there will be more than four postings next year. But consider this an informal contract about deliveries for the next year :)

Have a nice one!


Migrating from CVS to SVN

We have started a migration of our projects from CVS to SVN.

Choice of VCS and RAD plugin

We looked at other VCS tools and would have liked a distributed VCS. But that would have required training of our developers to a different configuration management model, so we went for Subversion instead.

For the same reason, we chose to use the Subversive plugin in RAD instead of the tigris Subclipse plugin. While Subclipse is more true to how Subversion works, Subversive provides us with behavior that (for the developer) is much more like the CVS Team integration in RAD.

There are still a few minor bugs in the plugin, but on the whole it is quite usable.

The biggest change for our developers will be the new tagging behavior: before they could tag projects in the Project Explorer. But now they need to tag in the Subversion Repositories Explorer (because only it knows about the folder hierarchy in Subversion).

Repositories and Performance

Today we have 18 CVS repositories on a single server (in use by some 100 developers and Hudson).

While it has always been stable and running without problems, its performance is not good. Synchronization time for most projects is a minute or more, which is aggravating.

We expect Subversion to give us a much faster VCS experience - and that is certainly true for the 50-odd projects that have already been migrated.

In the new setup, we want to use fewer repositories, making it easier to move projects between departments when that is necessary.

However, as we have not been able to find some good data on Subversion performance, I am a little concerned about how the server will perform when we get all the projects migrated to SVN - to fewer repositories.

Fortunately, the ability (as admin) to move data with full history between repositories saves us (is the assumption :)

If many projects in few repositories becomes a problem, we can introduce more repositories. If load on the server becomes too high, we can use redirection in Apache to move some repositories to other machines.

So we feel safe at the moment (in our glorified ignorance).

Repository Layout

In CVS, a number of modules constitute a single RAD project. That makes it a (small) nuisance to check out a project. It may look like:



In Subversion, we address that by introducing a new logical layer in the folder hierarchy. We also introduce a layer for the departments:


The layout is ensured with a commit hook.

Subversion Hooks

We have a number of hook scripts implemented in Ruby. The scripts are launched from DOS bat scripts.

Layout Validator

The Layout Validator ensures the REPO/DEPT/LOGICAL/trunk|branches|tags/ hierarchy in Subversion.

Launcher pre-commit.bat, Script hook-layout-validator.rb

Commit Allower

In general all employees have read access to Subversion. But our build users are not allowed to commit (because they anonymize whomever started the job they run).

The list of users who cannot commit are listed in the script.

Launcher start-commit.bat, Script hook-commit-allower.rb

Repository Locker

When doing administrative stuff directly on a repository (imports, for example) we prevent write access for the developers.

The script looks for a file named jb.txt in the repository's root folder. If it exists, commit is prevented and the text in the file is presented to the user.

Launcher start-commit.bat, Script hook-repository-locker.rb

Integrity Checking

Each night we run a cron job that iterates over all repositories and runs this command:

svnadmin.exe verify -q $repo

The output is sent to a mailbox that is (more or less) constantly monitored.

So if there is data corruption on the server we will know within a short time. Unlike CVS, where we would only find out if the project containing a corrupted file was attempted built.

Per-Commit Backups

The next thing I will be looking at is creating per-commit diff files to another drive.

That should allow us to get up and running in hurry if the main server is hit by logical errors on the primary drive.

Interesting, and a new improvement over CVS where we only had the nightly backups to rely on...


Toolsmith Kaizen

Currently not being able to afford the time to prepare a technical-with-source blog entry, I thought I would write a little about a softer issue that I have been working on lately.

I am unable to get to the bottom of the problems on my plate. I am always playing catch up. That is how it has always been - here at Jyske Bank, at Systematic, and before that, at Red Hat and Cygnus Solutions. I guess it is the way the world works.

However, as my primary concern is improving the tools used by our developers, I badly need feedback from them to improve their world (or I'll be working blind).

In my experience, it is very easy as a Toolsmith to lose touch with the developers; they tend to suffer in (for the Toolsmith's ears) silence. While they may have gripes with the tools, they don't bother telling the Toolsmith about it unless it is really horrible or critical for their work. Their experience and assumption is that nothing will probably be done about the problems they see - so why waste time reporting them?

At least that is how I behave when my feedback has no effect. And how I have experienced developers I work with to behave.

And obviously, it is a self fulfilling prophecy; if problems go unreported, odds are they will not get fixed.

So I have decided to change the behavior of the developers here at Jyske Bank - to try and get some more feedback and, in the end, become a better Toolsmith.

I base this on the important observation: I will never ever get to the bottom of my task list. So tasks are always delayed to some extend (ultimately depending on their importance).

Which leads me to: if tasks are always delayed, they might as well become a little more delayed, in exchange for providing a better (perceived) service to my customers (the developers).

So I decide to actively prioritize improvement suggestions from the developers (issues have to be relevant and worthwhile, of course).

Specifically, for 2 weeks out of every 8 weeks, I will focus on implementing developer suggestions (as general firefighting allows). Votes in Jira serve as my guide for which issue to fix and in which order.

That is the foundation on which I want to build the change of behavior.

The rest is public relations; I need to get the word out and lower the bar of entry for the developers.

Enter my slogan: "Red Hat Kaizen"

I use my red fedora (and probably abuse RHAT's trademark - my apologies) as a gimmick; I wear it in the two-week developer-issues-have-priority period. And whenever I happen to be visiting developers to help with specific issues. It definitely creates some attention :)

Kaizen is for enticing the developers. I emphasize that there is no lower limit to the stuff they should report. Every little improvement matters - especially if it benefits 80 developers.

I have been pushing my little agenda for some weeks now, and while management is exuberant there has not been the reaction from the developers I hoped for (i.e. I am not swamped in new Jira issues).

But I have faith and patience. It'll come, I'm sure...

Have a nice vacation!


Sources moving to Buildmeister@Kenai

Given the trouble of recreating sources here in the blog, I have decided to move them into a proper VCS system at Kenai.

I will go through the existing blog entries, move the source and insert links to the VCS instead.

Go to the Buildmeister Kenai project to access the sources.


Configuration of Hudson

We use Hudson to continuously build our projects.

We have one Master and six Slaves.

There are 385 projects being built. 128 of these are simple projects where only some XML documents are validated. The other 257 projects are mostly JEE applications, but there are also a few regular Java projects.

Build times span from 12 seconds to 27 minutes.

We have the Hudson configuration expressed by a simply formatted property(ish) file. This file is under CVS control, and is manipulated by a number of support scripts independent of Hudson.

When the configuration is changed, it triggers a build in Hudson on the Master (the only project that is allowed on the Master). This translates the configuration to Hudson-lingo then restarts Hudson.

So we can track and undo changes to the configuration. If the configuration becomes corrupt, it will either fail to translate - or we can roll back to the previously working version.

Specifically, the configuration build process consists of these steps:

  1. Prepare Hudson for shutdown
  2. Set up a cron job to resume Hudson in 15 minutes
  3. Translate the property configuration into Hudson's configuration tree.
  4. Copy the generated configuration tree on top of the Hudson's active configuration.
  5. Set up a cron job to restart Hudson Service in 2 minutes

(1) tries to avoid having any build process running when Hudson restarts. (2) and (5) ensures that Hudson will resume (if the build fails) or be restarted with the new configuration.

(3) decouples us from Hudson. Should we decide to use a different continuous integration engine in the future, we can continue to use our configuration scripts. We just write a new translation script.

Much more robust, efficient and delicious than what I made at Systematic; there we had only a few (big) projects, so we used a manually configured LuntBuilt. But it was quickly growing unmaintainable by the time I left. Hopefully they are using automatically managed configurations by now :)

Lovely as this is (in my mind, anyway), step (1) does not work as I would have wished.

It could still cause an abort of running projects that take longer than the configuration project to build. And it did on occasion.

Today we use security on Hudson, so the Shutdown/Resume URLs are not accessible. So we almost always accidently terminate project builds when the configuration is changed.

What I think is needed is a special Hudson configuration bootstrap builder (beyond my available time, I'm afraid) that would:

  1. Prepare Hudson for shutdown
  2. Wait for all running jobs to complete
  3. Build the project
  4. Resume if the project failed, otherwise restart the Hudson Service.

Hopefully someone will make me such a builder :)


EjbDeploy on Windows

I do not think WebSphere was ever intended to run on Windows.

It is so easy to create applications with deep package hierarchies that result in deployed files that cannot be accessed on Windows via DOS path semantics. Allegedly UNC paths do not have the 254-ish character limit, so in theory there is a way to make it work, but in practice, it is not used.

We need to be able to catch this unfortunate situation at build time, because WebSphere is not really helpful in its error messages when this happens. Resulting in much head scratching and use of foul language before the cause is found. Followed by more foul language (I imagine) when the developers have to refactor their code to suit the platform limitations.

But that will be the issue of a future blog entry, because in this one I will deal with another DOS/Windows-imposed limit that causes frustrating breakage in WebSphere's support tools.

When generating Ejb stubs with the ejbdeploy script, it is not hard to exceed DOS's command line length limit if there is a big classpath.

I have experienced it at Systematic and here at Jyske Bank. At Systematic I think the solution we chose was to shorten the path to jars by mounting a DOS drive at a suitable location - possibly combined with merging jars into fewer files. I don't remember exactly.

But I remembered the problem when I saw it again, so I decided to fix the broken tool instead of working around it.

The result is a decompiled version of IBM's EjbDeploy class, where I have modified the -cp argument handling to read the classpath from a file instead of from the command line.

In front of that I have an Ant task that translates an Ant path reference into a string and writes it out to a file. It also runs IBM's setupCmdLine.bat script and records the defined environment variables. Then IBM's hacked class is invoked with the environment it expects and loads the long classpath from the file.

Works like a charm.

Which begs the question why IBM does not ship their expensive product with a similar solution. Probably because they see it as an AIX sales lead when customers complain...


The two files are placed at Buildmeister@Kenai.


Syntax testing

I have to post some python, ruby and java code in this blog. This is how the syntax highlighting is set up (documenting it for myself).

After changing the template:
  1. Copy contents of http://syntaxhighlighter.googlecode.com/svn/trunk/Styles/SyntaxHighlighter.css and paste it into the template before ]]></b:skin>
  2. Add before </head>:
    <script language="'javascript'" src="'http://syntaxhighlighter.googlecode.com/svn/trunk/Scripts/shCore.js'/">
    <script language="'javascript'" src="'http://syntaxhighlighter.googlecode.com/svn/trunk/Scripts/shBrushPython.js'/">
  3. Add before </body>:
    <script language="'javascript'">
Code pasted into the blog should:
  1. Have instances of < replaced with &lt;
  2. Be placed in a section like:
<pre name="code" class="py">

Add extra brushes as needed...

Note that the Blogger editor eats your line indentation if you edit a posting. So where indentation is important (say, in Python, but any scripts really), paste in the script as the last part of any edit action (gah!).

Configuration of WebSphere

Ah, always a hassle to set up, WebSphere.

Why must one programatically use an API to change the server configuration - which is then saved to XML files. Why not just document the configuration files, and let me poke them directly (as some other application servers allow)?

Anyway, such is the world I have chosen to live in. So I try to make it at least bearable.

Most WebSphere configuration scripts I have seen, whether Jython or Jacl, tend to be a glorious mix of code and data. Which I guess is fine, if the data does not change much. And if anyone who must change the data are fluent in the given language, so they do not mess up the
logic. Not cool.

Once More From the Top

I decided to sprinkle a dose of Separation of Concerns on the issue. I had to rewrite some of our WebSphere configuration magic to handle some new features, and we are starting to look at WAS7 migration. So we needed a new tool.

What I ended up with is a Jython script that takes a configuration file as input. The configuration file contains commands in brackets followed by optional properties. It looks a bit like Windows ini files:

[command, options, ...]

So the logic for interaction with WebSphere is stored in the script.

And the configuration you want to stuff into WebSphere is stored in the configuration file. (See the comment in the script for command documentation.)

Not really revolutionary. But it works and we can have different configuration files for WAS6, WAS61, and WAS7. Without polluting the logic with knowledge about the diffent versions. And without changing the logic when we find some new configuration node we must set.

Missing Feature

I know there is at least one logic feature missing: to change the console group permissions, I need to find a specific numbered node, and change it. That is, I need assignment from list (or something similar).

The current script cannot do that - because I did not need it for anything else, and because the changing the console group fails from Jacl and Jython in WAS6. But I will probably extend the script when I start to look at WAS7 in earnest.

Also, since we only use it to set up the initial configuration, there are probably some other stuff it cannot handle in its current form.

You are welcome to extend it...

Making Configurations

So how to make a configuration file? I make a new configuration by more or less following these steps:
  1. Use [types] and [attributes] to find the relevant configuration type to use and its attributes.
  2. Use [list] and [show]/[showall] to examine the default configuration.
  3. Use [create] and [modify] to make changes.
  4. Again use [list] and [show]/[showall] to see the result.
Loop a few times. Throw in [save] when it looks like it is working.

I have also been searching files in WebSphere's config folder to find the nodes/types I need to modify after making manual modifications in the Administration Console.

Or asked Google if everything else failed...


I have moved the script to Buildmeister@Kenai.

DOS Command Substitution

I have written many scripts in shell script (Bash mostly). By no means am I an expert in shell, but I like its power.

So when I have to do stuff in DOS, I feel powerless. DOS is such a poor and restricting environment for the unlucky shell programmer.

My pet peeve has been the lack of command substitution:

$ variable = `command`

where the variable is assigned the output from the executed command.

Well, as it turns out, it is possible to do something similar in DOS. It is just not very simple to do (in comparison).

Have a look at the FOR documentation. The /F variant allows you to execute a command. The output is assigned to a temporary variable, which can then be accessed in the loop part.

For example, assign the date to the variable TODAY with:

DOS> FOR /F "usebackq delims=^" %i IN (`date /t`) DO @set TODAY=%i

(yes, I know of %DATE% - it is an example!)

Two notes:
  1. the delims character must not appear in the output
  2. if you use it in a script, replace %i with %%i.

It would be a stretch to call it elegant. But it works.

I have also used one of the other FOR /F variants for reading values from a property file (using = for key/value separation):

DOS> FOR /F "usebackq delims== tokens=1,2" %i IN (c:\temp\file.properties) DO set PROP_%i=%j

Here FOO=BAR in the file will result in %PROP_FOO% being assigned the value BAR.

Welcome to the Buildmeister Blog

Hello Stranger!

My name is Jesper Skov. I'm 37 years of age and have a Master's degree in Software Engineering from AAlborg University, Denmark.

I am a buildmeister/toolsmith at heart: Understanding, analyzing, tinkering with, and, in particular, improving the tools used in the software developer's trade has always been an interest of mine.

Currently I am working at Jyske Bank, the 3rd largest bank in Denmark.

I spend most of my time developing and maintaining the build infrastructure and development tools for some 80 Java Enterprise developers.

With this blog, I hope to pave the ground for an exchange of knowledge and code with other buildmeisters and toolsmiths.

Before I started at Jyske Bank, I worked at Systematic doing pretty much the same. And I have solved many similar problems at both places.

Clearly, as I see it, there is a need for exchange of experience and code for solving common problems in the context of build and development tools.

Or, if this is old hat and everyone else have been exchanging buildmeister and toolsmith gems in some corner of the Internet I do not frequent, please let me know where it is. I need to know. Badly.

If you dabble in the same area of expertise, please consider creating your own blog so we can exchange knowledge and solutions. If there is more than one or two blogs on the issue, we should probably start a blogplanet.

Whether you intend to start your own blog, comment on my postings, or just lurk: Welcome!