Busy times...

Yikes, Christmas is over us. And I never got around to posting anything since late summer.

Apologies! It is not that this blog is dead (it just smells a bit peculiar), but I have been hung up on a number of high priority issues.

I (seriously) hope the issues will be resolved early in the new year, so I can blog a little again.

In the pipeline (yes, let's make some vapor-blogging):

• Enterprise deployment of IBM Rational products.
  As the name suggests, the "rational" approach would probably be to use IBM's installer. But no - I just had to go and invent our own deployment strategy. It is slim, it is fast, and we have full control of the deployment. What's not to like?
• Authorized Job Service
  We do not use our Continuous Integration builds in production for various reasons. Instead we have a manually invoked build server. To make it work I needed a way to authenticate and authorize users between servers.
  And as I couldn't figure out how to make a single sign-on solution that would use the user's existing Windows session credentials (with JAAS), I had to roll my own using SSPI.
  I suspect this will be a blog entry where someone points out the obvious to me...
• Our SVN backup and restore setup.
  The repositories are backed up at night and restored to a different server during the day. Lovely stuff. I hope we'll never have to make use of it - but if we do, I am much more comfortable with it than with our previous CVS setup.
• And before the new year expires, something about our new documentation system (it needs to mature a bit yet).
  We use the Mylyn WikiText plugin to write documentation placed in the repository with the source. On project builds, the documentation is translated to HTML. We have extended the syntax (Confluence) a bit to add extra features...

Hopefully there will be more than four postings next year. But consider this an informal contract about deliveries for the next year :)

Have a nice one!

Migrating from CVS to SVN

We have started a migration of our projects from CVS to SVN.

Choice of VCS and RAD plugin

We looked at other VCS tools and would have liked a distributed VCS. But that would have required training of our developers to a different configuration management model, so we went for Subversion instead.

For the same reason, we chose to use the Subversive plugin in RAD instead of the tigris Subclipse plugin. While Subclipse is more true to how Subversion works, Subversive provides us with behavior that (for the developer) is much more like the CVS Team integration in RAD.

There are still a few minor bugs in the plugin, but on the whole it is quite usable.

The biggest change for our developers will be the new tagging behavior: before they could tag projects in the Project Explorer. But now they need to tag in the Subversion Repositories Explorer (because only it knows about the folder hierarchy in Subversion).


Repositories and Performance

Today we have 18 CVS repositories on a single server (in use by some 100 developers and Hudson).

While it has always been stable and running without problems, its performance is not good. Synchronization time for most projects is a minute or more, which is aggravating.

We expect Subversion to give us a much faster VCS experience - and that is certainly true for the 50-odd projects that have already been migrated.

In the new setup, we want to use fewer repositories, making it easier to move projects between departments when that is necessary.

However, as we have not been able to find some good data on Subversion performance, I am a little concerned about how the server will perform when we get all the projects migrated to SVN - to fewer repositories.

Fortunately, the ability (as admin) to move data with full history between repositories saves us (is the assumption :)

If many projects in few repositories becomes a problem, we can introduce more repositories. If load on the server becomes too high, we can use redirection in Apache to move some repositories to other machines.

So we feel safe at the moment (in our glorified ignorance).

Repository Layout

In CVS, a number of modules constitute a single RAD project. That makes it a (small) nuisance to check out a project. It may look like:

REPO_DEPT_A/
 projectA.cfg
 projectA.ear
 projectA.ejb
 projectA.web

REPO_DEPT_B/
 projectB.ear
 projectB.ejb
 projectB.web

In Subversion, we address that by introducing a new logical layer in the folder hierarchy. We also introduce a layer for the departments:

REPO/
DEPT_A/
projectA/
  trunk/
    projectA.cfg
    projectA.ear
    projectA.ejb
    projectA.web
  branches/
  tags/
DEPT_B/
projectB/
  trunk/
    projectB.ear
    projectB.ejb
    projectB.web
  branches/
  tags/

The layout is ensured with a commit hook.


Subversion Hooks

We have a number of hook scripts implemented in Ruby. The scripts are launched from DOS bat scripts.

Layout Validator

The Layout Validator ensures the REPO/DEPT/LOGICAL/trunk|branches|tags/ hierarchy in Subversion.

Launcher pre-commit.bat, Script hook-layout-validator.rb

Commit Allower

In general all employees have read access to Subversion. But our build users are not allowed to commit (because they anonymize whomever started the job they run).

The list of users who cannot commit are listed in the script.

Launcher start-commit.bat, Script hook-commit-allower.rb

Repository Locker

When doing administrative stuff directly on a repository (imports, for example) we prevent write access for the developers.

The script looks for a file named jb.txt in the repository's root folder. If it exists, commit is prevented and the text in the file is presented to the user.

Launcher start-commit.bat, Script hook-repository-locker.rb


Integrity Checking

Each night we run a cron job that iterates over all repositories and runs this command:

svnadmin.exe verify -q $repo

The output is sent to a mailbox that is (more or less) constantly monitored.

So if there is data corruption on the server we will know within a short time. Unlike CVS, where we would only find out if the project containing a corrupted file was attempted built.


Per-Commit Backups

The next thing I will be looking at is creating per-commit diff files to another drive.

That should allow us to get up and running in hurry if the main server is hit by logical errors on the primary drive.

Interesting, and a new improvement over CVS where we only had the nightly backups to rely on...

Toolsmith Kaizen

Currently not being able to afford the time to prepare a technical-with-source blog entry, I thought I would write a little about a softer issue that I have been working on lately.

I am unable to get to the bottom of the problems on my plate. I am always playing catch up. That is how it has always been - here at Jyske Bank, at Systematic, and before that, at Red Hat and Cygnus Solutions. I guess it is the way the world works.

However, as my primary concern is improving the tools used by our developers, I badly need feedback from them to improve their world (or I'll be working blind).

In my experience, it is very easy as a Toolsmith to lose touch with the developers; they tend to suffer in (for the Toolsmith's ears) silence. While they may have gripes with the tools, they don't bother telling the Toolsmith about it unless it is really horrible or critical for their work. Their experience and assumption is that nothing will probably be done about the problems they see - so why waste time reporting them?

At least that is how I behave when my feedback has no effect. And how I have experienced developers I work with to behave.

And obviously, it is a self fulfilling prophecy; if problems go unreported, odds are they will not get fixed.


So I have decided to change the behavior of the developers here at Jyske Bank - to try and get some more feedback and, in the end, become a better Toolsmith.


I base this on the important observation: I will never ever get to the bottom of my task list. So tasks are always delayed to some extend (ultimately depending on their importance).

Which leads me to: if tasks are always delayed, they might as well become a little more delayed, in exchange for providing a better (perceived) service to my customers (the developers).

So I decide to actively prioritize improvement suggestions from the developers (issues have to be relevant and worthwhile, of course).

Specifically, for 2 weeks out of every 8 weeks, I will focus on implementing developer suggestions (as general firefighting allows). Votes in Jira serve as my guide for which issue to fix and in which order.



That is the foundation on which I want to build the change of behavior.

The rest is public relations; I need to get the word out and lower the bar of entry for the developers.

Enter my slogan: "Red Hat Kaizen"

I use my red fedora (and probably abuse RHAT's trademark - my apologies) as a gimmick; I wear it in the two-week developer-issues-have-priority period. And whenever I happen to be visiting developers to help with specific issues. It definitely creates some attention :)

Kaizen is for enticing the developers. I emphasize that there is no lower limit to the stuff they should report. Every little improvement matters - especially if it benefits 80 developers.


I have been pushing my little agenda for some weeks now, and while management is exuberant there has not been the reaction from the developers I hoped for (i.e. I am not swamped in new Jira issues).

But I have faith and patience. It'll come, I'm sure...

Have a nice vacation!

Sources moving to Buildmeister@Kenai

Given the trouble of recreating sources here in the blog, I have decided to move them into a proper VCS system at Kenai.

I will go through the existing blog entries, move the source and insert links to the VCS instead.

Go to the Buildmeister Kenai project to access the sources.