2009-07-03

Toolsmith Kaizen

Currently not being able to afford the time to prepare a technical-with-source blog entry, I thought I would write a little about a softer issue that I have been working on lately.

I am unable to get to the bottom of the problems on my plate. I am always playing catch up. That is how it has always been - here at Jyske Bank, at Systematic, and before that, at Red Hat and Cygnus Solutions. I guess it is the way the world works.

However, as my primary concern is improving the tools used by our developers, I badly need feedback from them to improve their world (or I'll be working blind).

In my experience, it is very easy as a Toolsmith to lose touch with the developers; they tend to suffer in (for the Toolsmith's ears) silence. While they may have gripes with the tools, they don't bother telling the Toolsmith about it unless it is really horrible or critical for their work. Their experience and assumption is that nothing will probably be done about the problems they see - so why waste time reporting them?

At least that is how I behave when my feedback has no effect. And how I have experienced developers I work with to behave.

And obviously, it is a self fulfilling prophecy; if problems go unreported, odds are they will not get fixed.


So I have decided to change the behavior of the developers here at Jyske Bank - to try and get some more feedback and, in the end, become a better Toolsmith.


I base this on the important observation: I will never ever get to the bottom of my task list. So tasks are always delayed to some extend (ultimately depending on their importance).

Which leads me to: if tasks are always delayed, they might as well become a little more delayed, in exchange for providing a better (perceived) service to my customers (the developers).

So I decide to actively prioritize improvement suggestions from the developers (issues have to be relevant and worthwhile, of course).

Specifically, for 2 weeks out of every 8 weeks, I will focus on implementing developer suggestions (as general firefighting allows). Votes in Jira serve as my guide for which issue to fix and in which order.



That is the foundation on which I want to build the change of behavior.

The rest is public relations; I need to get the word out and lower the bar of entry for the developers.

Enter my slogan: "Red Hat Kaizen"

I use my red fedora (and probably abuse RHAT's trademark - my apologies) as a gimmick; I wear it in the two-week developer-issues-have-priority period. And whenever I happen to be visiting developers to help with specific issues. It definitely creates some attention :)

Kaizen is for enticing the developers. I emphasize that there is no lower limit to the stuff they should report. Every little improvement matters - especially if it benefits 80 developers.


I have been pushing my little agenda for some weeks now, and while management is exuberant there has not been the reaction from the developers I hoped for (i.e. I am not swamped in new Jira issues).

But I have faith and patience. It'll come, I'm sure...

Have a nice vacation!

2009-06-10

Sources moving to Buildmeister@Kenai

Given the trouble of recreating sources here in the blog, I have decided to move them into a proper VCS system at Kenai.

I will go through the existing blog entries, move the source and insert links to the VCS instead.

Go to the Buildmeister Kenai project to access the sources.

2009-06-09

Configuration of Hudson

We use Hudson to continuously build our projects.

We have one Master and six Slaves.

There are 385 projects being built. 128 of these are simple projects where only some XML documents are validated. The other 257 projects are mostly JEE applications, but there are also a few regular Java projects.

Build times span from 12 seconds to 27 minutes.


We have the Hudson configuration expressed by a simply formatted property(ish) file. This file is under CVS control, and is manipulated by a number of support scripts independent of Hudson.

When the configuration is changed, it triggers a build in Hudson on the Master (the only project that is allowed on the Master). This translates the configuration to Hudson-lingo then restarts Hudson.

So we can track and undo changes to the configuration. If the configuration becomes corrupt, it will either fail to translate - or we can roll back to the previously working version.

Specifically, the configuration build process consists of these steps:

  1. Prepare Hudson for shutdown
  2. Set up a cron job to resume Hudson in 15 minutes
  3. Translate the property configuration into Hudson's configuration tree.
  4. Copy the generated configuration tree on top of the Hudson's active configuration.
  5. Set up a cron job to restart Hudson Service in 2 minutes


(1) tries to avoid having any build process running when Hudson restarts. (2) and (5) ensures that Hudson will resume (if the build fails) or be restarted with the new configuration.

(3) decouples us from Hudson. Should we decide to use a different continuous integration engine in the future, we can continue to use our configuration scripts. We just write a new translation script.


Much more robust, efficient and delicious than what I made at Systematic; there we had only a few (big) projects, so we used a manually configured LuntBuilt. But it was quickly growing unmaintainable by the time I left. Hopefully they are using automatically managed configurations by now :)


Lovely as this is (in my mind, anyway), step (1) does not work as I would have wished.

It could still cause an abort of running projects that take longer than the configuration project to build. And it did on occasion.

Today we use security on Hudson, so the Shutdown/Resume URLs are not accessible. So we almost always accidently terminate project builds when the configuration is changed.

What I think is needed is a special Hudson configuration bootstrap builder (beyond my available time, I'm afraid) that would:

  1. Prepare Hudson for shutdown
  2. Wait for all running jobs to complete
  3. Build the project
  4. Resume if the project failed, otherwise restart the Hudson Service.

Hopefully someone will make me such a builder :)

2009-05-19

EjbDeploy on Windows

I do not think WebSphere was ever intended to run on Windows.

It is so easy to create applications with deep package hierarchies that result in deployed files that cannot be accessed on Windows via DOS path semantics. Allegedly UNC paths do not have the 254-ish character limit, so in theory there is a way to make it work, but in practice, it is not used.

We need to be able to catch this unfortunate situation at build time, because WebSphere is not really helpful in its error messages when this happens. Resulting in much head scratching and use of foul language before the cause is found. Followed by more foul language (I imagine) when the developers have to refactor their code to suit the platform limitations.

But that will be the issue of a future blog entry, because in this one I will deal with another DOS/Windows-imposed limit that causes frustrating breakage in WebSphere's support tools.

When generating Ejb stubs with the ejbdeploy script, it is not hard to exceed DOS's command line length limit if there is a big classpath.

I have experienced it at Systematic and here at Jyske Bank. At Systematic I think the solution we chose was to shorten the path to jars by mounting a DOS drive at a suitable location - possibly combined with merging jars into fewer files. I don't remember exactly.

But I remembered the problem when I saw it again, so I decided to fix the broken tool instead of working around it.

The result is a decompiled version of IBM's EjbDeploy class, where I have modified the -cp argument handling to read the classpath from a file instead of from the command line.

In front of that I have an Ant task that translates an Ant path reference into a string and writes it out to a file. It also runs IBM's setupCmdLine.bat script and records the defined environment variables. Then IBM's hacked class is invoked with the environment it expects and loads the long classpath from the file.

Works like a charm.

Which begs the question why IBM does not ship their expensive product with a similar solution. Probably because they see it as an AIX sales lead when customers complain...

Source

The two files are placed at Buildmeister@Kenai.