2010-02-19

Rational Deployment

We use a number of IBM Rational products (RAD, RSA, RSM, RDZ, and RDI at last count).

These are supposed to be installed using IBM's Installation Manager (IM). Unfortunately, IM and I have issues.

One Of Us Had To Go...

When I started at the bank, I had to make a new RAD7 (version 7.0.0.3 IIRC) deployment to our 75 developers.

This was early times for the IM so it did not do too well. In fact, its non-interactive installation was pretty much useless - and did not even get properly documented until RAD 7.0.0.4. But I tried. I really did.

With the result that when the installation package hit the first 20 developer machines it failed on almost all of them for various reasons. To be honest, we use a package installation system from Siemens (Sniik) that takes care of launching IM. And Sniik adds complications of its own, so it may not all have been IM's fault.

In any event, we reverted to the old package, and all was fine. All was fine, because it installed RAD7 (the initial release) and then let IM do an update to the latest version.

However, while I waste time trying to get IM and Sniik to install RAD in peaceful cooperation, IBM makes the next release. This release is so large, that IM times out during the download process - and so effectively prevents RAD installation on new machines using the old installation package.


This circus had lasted for 7-8 months. Fortunately, I was then allowed to do it my way.

An Alternative Approach

"My way" being an idea fostered at Systematic (Hi Anders!); install the product on one machine, zip it up and unpack it on all target machines. After the data is unpacked, replace host, domain and user name occurrences in all file names and files.

As far as I know, the idea was dropped at Systematic because of problems with the Windows Registry.

At the bank the developers do not have administrative rights on their machines. So it was imperative that I could make the solution work for non-administrators; use of Window's Registry would be a problem.

After some digging, it appeared to me that the Rational products only used the Registry for the Installation Manager. And the WebSphere Application Server may be set up to be started via the Windows Service system. But those were the only features requiring administrative rights I could find. And being happy to drop both, I could continue with the implementation.

Testing The Waters

I made a first simple implementation of the alternative installer using a bash script (for packaging) and a Ruby script for unpacking/replacing.

It worked well, and it did not take long before we could replace the broken Sniik/IM package with a new JB custom installer package for RAD.

As the months passed and more Rational products were deployed with the simple installer implementation, it started to show its problems; it took a long time to create packages, the installation packages took up much space (because I had to clone the whole thing when making minor installation script changes), and the users did not get good information when installing the package.

Then I volunteered to make a installation package for WebSphere Portal Server (WPS), because it took hours to install with IM - and I was pretty sure I could improve on that.

I knew I would have to re-implement the installer - both to address the difficulties with the old implementation, and to allow more flexibility for the problems I knew I would have with WPS. Both parts were rewritten in Ruby.

Rational Installer


The Rational Installer has two parts; the package creation script and the installer script.

create_package.rb

The create package script reads recipies for package creation. The recipies include information about the Windows rights the package should be created with, whether to search archives for strings to replace, and a list of folders to include in the package.

For each folder is described the strings to search for (host, user, and domain) and which sub-folders to exclude from the package (such as temp folders). It can also specify a list of files/patterns to exclude from string replacement (such as binary files).

The outcome is an archive (ZIP) file and a meta-data file used by the installer script.

It is possible to place pre- and post-install scripts next to the recipie which are copied into the resulting package.

install.rb

The installation script takes a specified list of packages to install. It allows existing installations to be replaced (if --force argument is used).

It presents an installation summary with expected installation time to the user. Checks that the user has the necessary Windows rights, that the destination disks have the necessary space and then runs any pre-install scripts.

Then the archives are unpacked and necessary text replacements and file renamings are made. The installed folders are given the required Windows rights.

Finally it runs any post-install scripts.

In Use

First I manually install the product I need to make a package for. RAD and friends are pretty simple to install manually. For WPS, I use scripts with IM. After IM has run, I have the base material to make my own packages.

Packaging RAD (without WAS) takes something like 6 minutes. Packaging WAS is around 3 minutes. Both ignore archive string searching, because I happen to know I can get away with it. For WPS it is necessary to scan archives for strings to be replaced. Combined with its large size, this results in a packaging time of 18 minutes.

When making an installation package for our RAD deployment, I break it down in one package for RAD itself and one for each of the nested WebSphere installations. This means I can update each WAS or RAD itself separately from the other elements by substituting the relevant packages in the installer.

The resulting package size are much smaller than the IM media. In part because our packages only contain what we really want to have installed - and in part because, well, IBM's installation media is just beyond obese.

For example, our RAD7.5.5 package is 1.9GB, while IBM's media is 3.1GB (7.5) + 2.9GB (7.5.5 update). Obviously, that in itself results in a massive time reduction when copying data to the machines.

Here is an example output from running the installer, installing RAD 7.5.5, WAS 6.0.2.33 and WAS 7.0.0.7 (the im/nodelock package contains the license files):

Loading information for package im/nodelock-rad75+3-servers
Forcing deletion of existing folder c:/udvikler/rational/rad75
Loading information for package rad/rad-7.5.5
Loading information for package was/was-6.0.2.33
Running preinstall script...
Loading information for package was/was-7.0.0.7
Running preinstall script...
-------------------------------------------------------------
Installing 4 packages:
Required space C: 4654MiB, D: 0MiB
Expected installation time 38 minutes
-------------------------------------------------------------
Installing package im/nodelock-rad75+3-servers starting at 2009.12.11 10:24
unpacking installation-manager-certificates to c:/udvikler/rational/rad75
startet at 2009.12.11 10:24, completion expected at 2009.12.11 10:25
replacing strings in 0 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Installing package rad/rad-7.5.5 starting at 2009.12.11 10:24
unpacking rad75 to c:/udvikler/rational/rad75
startet at 2009.12.11 10:24, completion expected at 2009.12.11 10:38
replacing strings in 0 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Making P2 backup
Installing package was/was-6.0.2.33 starting at 2009.12.11 10:35
unpacking was6 to c:/udvikler/rational/rad75/SDP/runtimes/base_v6
startet at 2009.12.11 10:35, completion expected at 2009.12.11 10:39
replacing strings in 5 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Running postinstall script...
Installing package was/was-7.0.0.7 starting at 2009.12.11 10:40
unpacking was7 to c:/udvikler/rational/rad75/SDP/runtimes/base_v7
startet at 2009.12.11 10:40, completion expected at 2009.12.11 10:49
replacing strings in 3 files...
renaming 0 files...
setting permissions...
creating excluded (empty) folders
Running postinstall script...
Installation successfully completed at 2009.12.11 10:46


So RAD 7.5.5 plus two WebSphere instances installed in less than 30 minutes. Not bad, if I have to say so myself :)

The post-install scripts can be used to tweak the installed product further. For instance, WebSphere contains certificates created on the installation host; these can be replaced by new certificates created on the client (we only do it for WPS though).

I also use the pre- and post-install scripts in combination to backup/restore WebSphere's profile database, so profiles survive a RAD/WAS reinstallation.

Push Or Pull

So I replaced a "push" deployment model with a "pull" variant; the developer has to manually install the Rational tools.

There were some concerns about this from the outset. And understandably so; when you want to deploy a new Rational product/version, each "receiving" developer has to spend ~30 minutes doing so. Before it happened at night at no cost to the developer.

Of course, what everybody tend to forget is that mostly everybody has experienced having to redo their installation during working hours for one reason or another. And now that pain has been reduced from half-a-wasted-day to something that can be done while at lunch.

The biggest advance, seen from my chair, is that I can do real fine grained staged deployments. Before, I had two stages - the first 20 developers, and then the rest.
Now I can ask individuals to update and test. And if they are happy, ask the remaining developers to install as it fits into their schedule. With two stages, there was always someone with a looming deadline who did not appreciate "push" deployments. Not an issue, anymore.


Scripts

The scripts will be made available soon(ish). I want to wait till Oracle/Sun has decided what to do about Kenai first. I may have to switch to SourceForge.

21 comments:

  1. I'm in exactly the same boat .. pleeeeeeeeze release the scripts so that I can retain my sanity!!
    If I could avoid having to use the Installation Manager, my life would be so much better..

    ReplyDelete
  2. Ah, yes. I never got around to do that.
    I'll try to get it done this week.

    ReplyDelete
  3. There, I have committed the scripts at http://kenai.com/projects/buildmeister/sources/sources/show/blog-sources/rational/package-install

    I have tested that I can run them locally, so they have been properly disentangled from our build system.

    I'll leave it as a reader exercise to fully understand how it all works :)

    ReplyDelete
  4. Hello Jesper !
    It looks good !
    Unfortunately, I run in to a few issues while trying it out.. :
    - we install to C:\Program Files and the space caused a bit of confusion initially (as the path was passed to StringSearcher (it shifted the args to StringSearcher due to the whitespace between Program and Files).

    - When trying to install a created package (rad755), I get this error:
    "C:\Program Files\ruby\bin\ruby.exe" inst
    all.rb install_package rad75-package
    C:/Program Files/IBM/ruby/lib/ruby/1.8/i386-mswin32/Iconv.so: 127: The specified procedure could not be found. - Init_Iconv (LoadError)
    C:/Program Files/IBM/ruby/lib/ruby/1.8/i386-mswin32/Iconv.so
    from install.rb:41

    It seems to come from the require 'Iconv' statement.. but I have (so far) roughly 3 hours of Ruby experience so .. I think it looks as though I have Iconv.so (I can find a iconv.so and iconv.dll in my Ruby installation). Perhaps I'm using the wrong version?

    ReplyDelete
  5. Yes, spaces in folder names. Gotta love it. I'll add some quotes to the example script.

    I use ruby-1.9.1-p378-i386-mingw32

    ReplyDelete
  6. Tell me about it .. and now they're making it even more interesting with C:\Program Files (x86) on Win7

    It's a design decision that makes you wonder if MS recruited people from the Installation Manager team..

    Right - I'll see what happens when using 1.9.1 .. thanks !

    ReplyDelete
  7. Right, using 1.9.1 got rid of the Iconv issue..

    Unfortunately, the install.rb does not seem to pick up the package...

    I see this:
    "C:\Program Files\OSS\ruby\ruby-1.9.1-p430-i3
    86-mingw32\bin\ruby.exe" install.rb rad75
    ================================================================
    New installation started at 2010.10.07 15:26
    ================================================================
    -------------------------------------------------------------
    Installing 0 packages:
    Required space C: 0MiB, D: 0MiB
    Expected installation time 0 minutes
    -------------------------------------------------------------
    Installation successfully completed at 2010.10.07 15:26

    I've got PACKAGE_DIR (in install.rb) set to D:\temp\packages and it contains:
    ..
    D:\temp\packages\rad75.inf
    D:\temp\packages\rad75.zip
    ..
    I wonder if it's related to the "root" in the package-rad75.properties ( .. that whitespace again: C:/Program Files/IBM/SDP75 ) ?

    Or is it something you've seen before?

    ReplyDelete
  8. Oh - sorry - typo. I have PACKAGE_DIR set to "D:/temp/packages" - not D:\temp\packages

    ReplyDelete
  9. The PACKAGE_DIR must include folders (which you refer to by relative path name in the installer), containing .inf+.zip files.

    For example, our package repository looks something like this:

    rad
    rad/rad-7.5.4-yyyymmdd
    rad/rad-7.5.5-yyyymmdd
    rad/rad-7.5.5.1-yyymmdd
    rdi
    rdi/...
    rsa
    rsa/...
    rsm
    rsm/...
    was/
    was/7.0.0.11-yyyymmdd

    Each package can be chosen for installation separately or combined with others.

    As mentioned in the blog posting above, we bundle RAD with WAS6 and WAS7. After the posting was written, we have started also including WPS615 (which makes it fairly heavy weight, unfortunately).

    So you should put your rad75.inf+zip files in a nested folder. And do yourself the favor of giving it a date or a version, so you can do version controlled updates in the future.

    ReplyDelete
  10. I've now set PACKAGE_DIR to "D:/temp/packages/rad".

    In the rad directory - I now have a directory called "rad755-08102010" containing the rad75.zip and the rad75.inf

    I invoke install.rb with the argument: rad755-08102010/rad75

    .. but it's still not seeing any packages.

    Adding the --verbose (ie: install.rb --verbose rad755-08102010/rad75 ) flag does not seem to give me any more info and I cant see any errors in the log.

    I can see that @packages (after line 342 in install.rb) is empty (I output #{@packages.length()} - and it's 0 ) but its quite difficult to trace why its failing due to my current lack of Ruby knowledge and of a debugger.

    Can you see any obvious mistakes above ? (syntax of arguments etc.)

    Thanks for your help ! - I really appreciate it!

    ReplyDelete
  11. Set PACKAGE_DIR to d:/temp/packages.

    The installer argument should be rad/rad755-08102010

    That should do the trick.

    ReplyDelete
  12. It took a while until I could test again (as I had to wait for the Installation Manager to install RAD so that I would have something to package).
    I ran the packaging script successfully, set the PACKAGE_DIR to D:/temp/packages (in install.rb), then re-ran the install.rb.

    Unfortunately, I get the same:
    "C:\Program Files\OSS\ruby\ruby-1.9.1-p430-i3
    86-mingw32\bin\ruby.exe" install.rb rad/rad755-08102010
    ================================================================
    New installation started at 2010.10.12 12:39
    ================================================================
    -------------------------------------------------------------
    Installing 0 packages:
    Required space C: 0MiB, D: 0MiB
    Expected installation time 0 minutes
    -------------------------------------------------------------
    Installation successfully completed at 2010.10.12 12:39

    Ie: still not picking up any packages.

    I'll try to get some time to dig into the code.. but meanwhile, if you have any other ideas as to why it isn't picking up the packages, pls let me know.

    ReplyDelete
  13. I am just finishing packaging RAD7.5.5.2.

    My installation wrapper script reads (in full):

    -------------8<---------------8<------------
    @echo off

    set BT_VERSION=V7-2-5

    set FORCE=
    if "%1"=="--force" set FORCE=--force

    \\build\webtools\ruby-lang\ruby-1.9.1-p378-i386-mingw32\bin\ruby.exe \\build\lib\jb\jb.itu.ti.build.templates\%BT_VERSION%\bin\package-install\install.rb %~n0 im/nodelock-rational75+3-servers-2009.10.27#c:/udvikler/rational/rad75 rad/rad-7.5.5.2 was/was-6.0.2.37 wps/wps-6.1.5.4-was-7.0.0.11-2010.06.24 %FORCE%
    -------------8<---------------8<------------

    Our PACKAGE_DIR contains the new RAD7.5.5.2 package:

    rad/rad-7.5.5.2/rad75.inf
    rad/rad-7.5.5.2/rad75.zip

    The .inf file contains:

    -------------8<---------------8<------------
    # Package generated on 2010-10-12 12:45:53 +0100
    name=rad75
    time=16
    space=2746474247
    path=c:/udvikler/rational/rad75
    rename_files=
    replace_files=
    skipped_replace_files=
    excluded_folders=SDP/temp,SDP/runtimes/base_v6,SDP/runtimes/base_v61,SDP/runtimes/base_v7
    rights=GAPP_IBMVAJ
    archives_searched=false
    user=jb18110
    host=we8280
    domain=jbmain50
    -------------8<---------------8<------------

    Hope that contains a clue as to what your problem might be.


    What does your .inf file contain? How big is the archive?

    ReplyDelete
  14. Thanks for your ever-so-quick replies !

    Right, I've switched to using a target install directory with no spaces in the path. I packaged RAD755 and after double-checking your last posting, I noticed that I had left off the vital %~n0 (aagh!) from the call to install.rb. I also needed a bit of twiddling with the permissions used and it now *nearly* works.

    I'm now running into a diskspace issue.

    I'm trying to install to the C drive where I have approx 22Gb free - but install.rb is telling me that I need 7203 Mb more to be able to install (and this is after deleting the RAD755 installation that I used to create the package).

    According to my .inf file - "space=7501397713" which looks like 7.5 Gb to me .. but it still thinks its insufficient.

    Any ideas ?

    ReplyDelete
  15. I am pretty sure that is because my function for finding available space on drives does not handle your locale (or version of DOS/Windows).

    Look in util_fallback.rb, the function enough_free_space? The line you need to change is probably this one:
    m = last_line.match(/([\d.,]+) (bytes free|byte ledig)/)

    It gets the number of free bytes on the drive from the DOS dir command. "bytes free" is the English version, "byte ledig" the Danish one. You can replace either, or add another after a | if you want.

    ReplyDelete
  16. Ah .. should have spotted that one :-)

    This is the output from the command, using our locale:
    10 Dir(s) 23'760'867'328 bytes free

    .. so I changed util_fallback:170 from:
    m = last_line.match(/([\d.,]+) (bytes free|byte ledig)/)
    To:
    m = last_line.match(/([\d.,\']+) (bytes free|byte ledig)/)

    That got me past the space issue - thanks!

    Unfortunately, I now run into the situation that my version of unzip.exe is not able to handle the zipfile but fails during the extract (due to the size, I think). I tried using the wzunzip.exe from Winzip and that has no problem extracting the file.
    In the original util_fallback.rb, I could see that you are using info-zip unzip-6.00.
    I would prefer to not have to depend on winzip for this so I tried to find the unzip that you're using, on ftp://ftp.info-zip.org/pub/infozip/win32/
    Unfortunately, I couldn't find it.

    - Do you know where I can find (32/64 bit) unzip v. 6.00 for Windows XP / Win7) ?

    ReplyDelete
  17. unz60xn.exe is the one you are looking for. Run it to unpack - or rename to .zip and use another unzipper to access its contents.

    ReplyDelete
  18. Aaah.. gotit. Running the install now .. *fingers crossed* ! :-D

    ReplyDelete
  19. It worked !! FANTASTIC !!
    Now I just have to build the rest of the packages (license, WAS6/7 etc.) and integrate it into our provisioning process - but that shouldn't be too hard.

    Did you ever test it on Win 7 ?
    - I'll try it out on 64 bit Win7 today .. I can let you know what happens if you're interested (?)

    ReplyDelete
  20. Super, glad to hear it.

    I have never tried it on Windows 7. And yes, please do tell how it works out.

    ReplyDelete
  21. Just a quick update: installing the package on Windows 7 (64bit) works just fine - had no issues whatsoever.

    This is great for us, as we've had serious stability issues with Installation Manager on Windows 7 (it causes the machine to crash intermittently).

    Currently downloading 7552 fixpack .. 18 Gb !
    I'm always amazed by the size of IBM's fixpacks..

    ReplyDelete