Rational Deployment

We use a number of IBM Rational products (RAD, RSA, RSM, RDZ, and RDI at last count).

These are supposed to be installed using IBM's Installation Manager (IM). Unfortunately, IM and I have issues.

One Of Us Had To Go...

When I started at the bank, I had to make a new RAD7 (version 7.0.0.3 IIRC) deployment to our 75 developers.

This was early times for the IM so it did not do too well. In fact, its non-interactive installation was pretty much useless - and did not even get properly documented until RAD 7.0.0.4. But I tried. I really did.

With the result that when the installation package hit the first 20 developer machines it failed on almost all of them for various reasons. To be honest, we use a package installation system from Siemens (Sniik) that takes care of launching IM. And Sniik adds complications of its own, so it may not all have been IM's fault.

In any event, we reverted to the old package, and all was fine. All was fine, because it installed RAD7 (the initial release) and then let IM do an update to the latest version.

However, while I waste time trying to get IM and Sniik to install RAD in peaceful cooperation, IBM makes the next release. This release is so large, that IM times out during the download process - and so effectively prevents RAD installation on new machines using the old installation package.


This circus had lasted for 7-8 months. Fortunately, I was then allowed to do it my way.

An Alternative Approach

"My way" being an idea fostered at Systematic (Hi Anders!); install the product on one machine, zip it up and unpack it on all target machines. After the data is unpacked, replace host, domain and user name occurrences in all file names and files.

As far as I know, the idea was dropped at Systematic because of problems with the Windows Registry.

At the bank the developers do not have administrative rights on their machines. So it was imperative that I could make the solution work for non-administrators; use of Window's Registry would be a problem.

After some digging, it appeared to me that the Rational products only used the Registry for the Installation Manager. And the WebSphere Application Server may be set up to be started via the Windows Service system. But those were the only features requiring administrative rights I could find. And being happy to drop both, I could continue with the implementation.

Testing The Waters

I made a first simple implementation of the alternative installer using a bash script (for packaging) and a Ruby script for unpacking/replacing.

It worked well, and it did not take long before we could replace the broken Sniik/IM package with a new JB custom installer package for RAD.

As the months passed and more Rational products were deployed with the simple installer implementation, it started to show its problems; it took a long time to create packages, the installation packages took up much space (because I had to clone the whole thing when making minor installation script changes), and the users did not get good information when installing the package.

Then I volunteered to make a installation package for WebSphere Portal Server (WPS), because it took hours to install with IM - and I was pretty sure I could improve on that.

I knew I would have to re-implement the installer - both to address the difficulties with the old implementation, and to allow more flexibility for the problems I knew I would have with WPS. Both parts were rewritten in Ruby.

Rational Installer

The Rational Installer has two parts; the package creation script and the installer script.

create_package.rb

The create package script reads recipies for package creation. The recipies include information about the Windows rights the package should be created with, whether to search archives for strings to replace, and a list of folders to include in the package.

For each folder is described the strings to search for (host, user, and domain) and which sub-folders to exclude from the package (such as temp folders). It can also specify a list of files/patterns to exclude from string replacement (such as binary files).

The outcome is an archive (ZIP) file and a meta-data file used by the installer script.

It is possible to place pre- and post-install scripts next to the recipie which are copied into the resulting package.

install.rb

The installation script takes a specified list of packages to install. It allows existing installations to be replaced (if --force argument is used).

It presents an installation summary with expected installation time to the user. Checks that the user has the necessary Windows rights, that the destination disks have the necessary space and then runs any pre-install scripts.

Then the archives are unpacked and necessary text replacements and file renamings are made. The installed folders are given the required Windows rights.

Finally it runs any post-install scripts.

In Use

First I manually install the product I need to make a package for. RAD and friends are pretty simple to install manually. For WPS, I use scripts with IM. After IM has run, I have the base material to make my own packages.

Packaging RAD (without WAS) takes something like 6 minutes. Packaging WAS is around 3 minutes. Both ignore archive string searching, because I happen to know I can get away with it. For WPS it is necessary to scan archives for strings to be replaced. Combined with its large size, this results in a packaging time of 18 minutes.

When making an installation package for our RAD deployment, I break it down in one package for RAD itself and one for each of the nested WebSphere installations. This means I can update each WAS or RAD itself separately from the other elements by substituting the relevant packages in the installer.

The resulting package size are much smaller than the IM media. In part because our packages only contain what we really want to have installed - and in part because, well, IBM's installation media is just beyond obese.

For example, our RAD7.5.5 package is 1.9GB, while IBM's media is 3.1GB (7.5) + 2.9GB (7.5.5 update). Obviously, that in itself results in a massive time reduction when copying data to the machines.

Here is an example output from running the installer, installing RAD 7.5.5, WAS 6.0.2.33 and WAS 7.0.0.7 (the im/nodelock package contains the license files):

Loading information for package im/nodelock-rad75+3-servers
Forcing deletion of existing folder c:/udvikler/rational/rad75
Loading information for package rad/rad-7.5.5
Loading information for package was/was-6.0.2.33
Running preinstall script...
Loading information for package was/was-7.0.0.7
Running preinstall script...
-------------------------------------------------------------
Installing 4 packages:
Required space C: 4654MiB, D: 0MiB
Expected installation time 38 minutes
-------------------------------------------------------------
Installing package im/nodelock-rad75+3-servers starting at 2009.12.11 10:24
unpacking installation-manager-certificates to c:/udvikler/rational/rad75
startet at 2009.12.11 10:24, completion expected at 2009.12.11 10:25
replacing strings in 0 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Installing package rad/rad-7.5.5 starting at 2009.12.11 10:24
unpacking rad75 to c:/udvikler/rational/rad75
startet at 2009.12.11 10:24, completion expected at 2009.12.11 10:38
replacing strings in 0 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Making P2 backup
Installing package was/was-6.0.2.33 starting at 2009.12.11 10:35
unpacking was6 to c:/udvikler/rational/rad75/SDP/runtimes/base_v6
startet at 2009.12.11 10:35, completion expected at 2009.12.11 10:39
replacing strings in 5 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Running postinstall script...
Installing package was/was-7.0.0.7 starting at 2009.12.11 10:40
unpacking was7 to c:/udvikler/rational/rad75/SDP/runtimes/base_v7
startet at 2009.12.11 10:40, completion expected at 2009.12.11 10:49
replacing strings in 3 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Running postinstall script...
Installation successfully completed at 2009.12.11 10:46

So RAD 7.5.5 plus two WebSphere instances installed in less than 30 minutes. Not bad, if I have to say so myself :)

The post-install scripts can be used to tweak the installed product further. For instance, WebSphere contains certificates created on the installation host; these can be replaced by new certificates created on the client (we only do it for WPS though).

I also use the pre- and post-install scripts in combination to backup/restore WebSphere's profile database, so profiles survive a RAD/WAS reinstallation.

Push Or Pull

So I replaced a "push" deployment model with a "pull" variant; the developer has to manually install the Rational tools.

There were some concerns about this from the outset. And understandably so; when you want to deploy a new Rational product/version, each "receiving" developer has to spend ~30 minutes doing so. Before it happened at night at no cost to the developer.

Of course, what everybody tend to forget is that mostly everybody has experienced having to redo their installation during working hours for one reason or another. And now that pain has been reduced from half-a-wasted-day to something that can be done while at lunch.

The biggest advance, seen from my chair, is that I can do real fine grained staged deployments. Before, I had two stages - the first 20 developers, and then the rest.
Now I can ask individuals to update and test. And if they are happy, ask the remaining developers to install as it fits into their schedule. With two stages, there was always someone with a looming deadline who did not appreciate "push" deployments. Not an issue, anymore.


Scripts

The scripts will be made available soon(ish). I want to wait till Oracle/Sun has decided what to do about Kenai first. I may have to switch to SourceForge.

EjbDeploy on Windows

I do not think WebSphere was ever intended to run on Windows.

It is so easy to create applications with deep package hierarchies that result in deployed files that cannot be accessed on Windows via DOS path semantics. Allegedly UNC paths do not have the 254-ish character limit, so in theory there is a way to make it work, but in practice, it is not used.

We need to be able to catch this unfortunate situation at build time, because WebSphere is not really helpful in its error messages when this happens. Resulting in much head scratching and use of foul language before the cause is found. Followed by more foul language (I imagine) when the developers have to refactor their code to suit the platform limitations.

But that will be the issue of a future blog entry, because in this one I will deal with another DOS/Windows-imposed limit that causes frustrating breakage in WebSphere's support tools.

When generating Ejb stubs with the ejbdeploy script, it is not hard to exceed DOS's command line length limit if there is a big classpath.

I have experienced it at Systematic and here at Jyske Bank. At Systematic I think the solution we chose was to shorten the path to jars by mounting a DOS drive at a suitable location - possibly combined with merging jars into fewer files. I don't remember exactly.

But I remembered the problem when I saw it again, so I decided to fix the broken tool instead of working around it.

The result is a decompiled version of IBM's EjbDeploy class, where I have modified the -cp argument handling to read the classpath from a file instead of from the command line.

In front of that I have an Ant task that translates an Ant path reference into a string and writes it out to a file. It also runs IBM's setupCmdLine.bat script and records the defined environment variables. Then IBM's hacked class is invoked with the environment it expects and loads the long classpath from the file.

Works like a charm.

Which begs the question why IBM does not ship their expensive product with a similar solution. Probably because they see it as an AIX sales lead when customers complain...

Source

The two files are placed at Buildmeister@Kenai.

Configuration of WebSphere

Ah, always a hassle to set up, WebSphere.

Why must one programatically use an API to change the server configuration - which is then saved to XML files. Why not just document the configuration files, and let me poke them directly (as some other application servers allow)?

Anyway, such is the world I have chosen to live in. So I try to make it at least bearable.

Most WebSphere configuration scripts I have seen, whether Jython or Jacl, tend to be a glorious mix of code and data. Which I guess is fine, if the data does not change much. And if anyone who must change the data are fluent in the given language, so they do not mess up the
logic. Not cool.

Once More From the Top

I decided to sprinkle a dose of Separation of Concerns on the issue. I had to rewrite some of our WebSphere configuration magic to handle some new features, and we are starting to look at WAS7 migration. So we needed a new tool.

What I ended up with is a Jython script that takes a configuration file as input. The configuration file contains commands in brackets followed by optional properties. It looks a bit like Windows ini files:

[command, options, ...]
key1=value1
key2=value2
...
keyN=valueN

So the logic for interaction with WebSphere is stored in the script.

And the configuration you want to stuff into WebSphere is stored in the configuration file. (See the comment in the script for command documentation.)

Not really revolutionary. But it works and we can have different configuration files for WAS6, WAS61, and WAS7. Without polluting the logic with knowledge about the diffent versions. And without changing the logic when we find some new configuration node we must set.

Missing Feature

I know there is at least one logic feature missing: to change the console group permissions, I need to find a specific numbered node, and change it. That is, I need assignment from list (or something similar).

The current script cannot do that - because I did not need it for anything else, and because the changing the console group fails from Jacl and Jython in WAS6. But I will probably extend the script when I start to look at WAS7 in earnest.

Also, since we only use it to set up the initial configuration, there are probably some other stuff it cannot handle in its current form.

You are welcome to extend it...


Making Configurations

So how to make a configuration file? I make a new configuration by more or less following these steps:

1. Use [types] and [attributes] to find the relevant configuration type to use and its attributes.
2. Use [list] and [show]/[showall] to examine the default configuration.
3. Use [create] and [modify] to make changes.
4. Again use [list] and [show]/[showall] to see the result.
   

Loop a few times. Throw in [save] when it looks like it is working.

I have also been searching files in WebSphere's config folder to find the nodes/types I need to modify after making manual modifications in the Administration Console.

Or asked Google if everything else failed...

Script

I have moved the script to Buildmeister@Kenai.