I’m using Shipwright to come up with an easier method of installing Perl modules for the Starlink Software Collection. Normally it takes me the better part of a day to manually install everything for one platform. Given we support (at least) four different platforms, this is a waste of my time. Shipwright can (purportedly) automate this to some degree.

Unfortunately Shipwright’s documentation is lacking, so I thought I’d keep some notes about my process in hopes that it will help others in creating a Shipwright package (or vessel, or repository, or whatever it’s actually called). These notes were written with version 2.2.1.

By the way, I’m only showing output from Shipwright and not from any of the other commands I run. A command run on the command-line is prefixed with a %.

I first started by creating a directory (oracdr-perl-modules) that would hold everything. Then I changed into that directory and initialized a git repository:

 % cd oracdr-perl-modules
 % git init

I then initialized my Shipwright repository and commited the initial state to my git repository:

 % shipwright create -r fs:`pwd`
 % git add *
 % git commit -m 'Shipwright initialization.'

I am not using the git backend for Shipwright, as I will be handling my git repository manually.

As I will be making tweaks to modules to make sure they build correctly, pass tests, etc., I created a vendor branch that would hold the un-modified modules as downloaded from CPAN:

 % git branch vendor
 % git checkout vendor

Then I started adding modules from CPAN:

 % shipwright -r fs:`pwd` import cpan:Proc::Simple
 % git add *
 % git commit -m 'Added Proc::Simple.'

These three steps were repeated for all of the CPAN modules I needed.

Once I finished importing the CPAN modules, I switched to my master branch and merged from vendor, since I want my vendor branch to reflect the “base” state of modules as obtained from the vendor, and my master branch to hold any local changes I have to make to the modules to get them to build properly:

 % git checkout master
 % git merge vendor

Now it was time to make tweaks to some of the modules. As an example, ExtUtils::F77 doesn’t support the g95 Fortran compiler, so I had to make some changes to the F77.pm file:

 % cd sources/cpan-ExtUtils-F77/vendor
 % xemacs F77.pm

Once my changes were complete, I checked in the modified file:

 % git add F77.pm
 % git commit -m 'ExtUtils::F77: Tweaks to support g95.'

I had to add some compilation options for Astro::FITS::CFITSIO, adding a command-line option when running perl Makefile.PL and tweaking Makefile.PL. To do this, I modified the build file:

 % cd ../../../scripts/cpan-Astro-FITS-CFITSIO
 % xemacs build

As I needed to add a command-line option to the Makefile.PL line, I edited the configure: line to read:

configure: %%PERL%% Makefile.PL OPTIMIZE=-O LIB=%%INSTALL_BASE%%/lib/perl5/ PREFIX=%%INSTALL_BASE%% %%MAKEMAKER_CONFIGURE_EXTRA%%

In this case, I added OPTIMIZE=-O.

I then edited the Makefile.PL:

 % cd ../../sources/cpan-Astro-FITS-CFITSIO/vendor
 % xemacs Makefile.PL

Then, another commit to the git repository:

 % git add Makefile.PL ../../../sources/cpan-Astro-FITS-CFITSIO/build
 % git commit -m 'Astro::FITS::CFITSIO: Add OPTIMIZE=-O to Makefile.PL command-line, edit Makefile.PL to know about $ENV{STARLINK_DIR}'

At this point I thought I’d test out the build process, making sure that the modules I’d imported passed their tests. If they didn’t, I could either fix the tests or patch modules (or, in a worse-case scenario, tell Shipwright to skip testing specific modules). I had to make a clone of my git repository in a different directory:

 % cd ~/development
 % mkdir shipwright-test
 % cd shipwright-test
 % git clone file:///home/bradc/development/oracdr-perl-modules

Testing time! I’m testing with a private build of Perl, which is located in $STARLINK_DIR/Perl/bin/perl, thus the extra preamble. If I was just using the system Perl I’d leave that out.

 % cd oracdr-perl-modules
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder
no default install-base, will set it to /tmp/vessel_oracdr-perl-modules-pHWh30/oracdr-perl-modules
Building cpan-Proc-Simple
Building cpan-Tk

During Tk‘s tests it hung up. I’ve seen this before, so I cancelled the build by hitting Ctrl-C. I changed to my original Shipwright directory, deleted the offending test file, and updated my repository:

 % cd ~/development/oracdr-perl-modules/sources/cpan-Tk/vendor/t
 % git rm font.t
 % git commit -m 'Tk: font.t test locks up, remove it'

Then back to my Shipwright test directory and continue on merrily. I first cleaned up the Shipwright build by adding --clean — this might not be necessary, but it allows things to start from a fresh slate. When building XS modules like Tk sometimes header or object files can get into a state, so cleaning them up before rebuilding is good practice, I’ve found.

 % cd ~/development/shipwright-test/oracdr-perl-modules
 % git pull
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder --clean
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder
Building cpan-Tk
Use of uninitialized value in string at bin/shipwright-builder line 443.
build cpan-Tk test part failed.
 at bin/shipwright-builder line 456
        main::_install('cpan-Tk', 'GLOB(0x12f73e10)') called at bin/shipwright-builder line 346
        main::install() called at bin/shipwright-builder line 241

Another error. Rerunning with --verbose^[1] shows that the problem is with the t/listbox.t test, which I removed in a similar fashion to the t/font.t test. Update the repository, clean up, and start again:

 % git pull
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder --clean
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder
Building cpan-Tk
Building cpan-Params-Validate
Building cpan-Class-Singleton
Building cpan-Compress-Raw-Zlib
Building cpan-Compress-Raw-Bzip2
Building cpan-IO-Compress
Building cpan-IO-Zlib
Building cpan-Package-Constants
Building cpan-Archive-Tar
Building cpan-Regexp-Common
Building cpan-Pod-Escapes
Building cpan-Pod-Simple
Building cpan-podlators
Building cpan-Devel-Symdump
Building cpan-Pod-Coverage
Building cpan-Test-Pod-Coverage
Building cpan-Test-Pod
Building cpan-Test-Portability-Files
Building cpan-Pod-Readme
Building cpan-version
Building cpan-ExtUtils-CBuilder
Building cpan-Digest-SHA
Building cpan-Module-Signature
Building cpan-ExtUtils-ParseXS
Building cpan-Test-Harness
Building cpan-Module-Build
Building cpan-DateTime-TimeZone
Building cpan-List-MoreUtils
Building cpan-DateTime-Locale
Building cpan-DateTime
Building cpan-Time-Piece
Building cpan-IO-Tee
Building cpan-URI
Building cpan-HTML-Tagset
Building cpan-HTML-Parser
Building cpan-libwww-perl
Building cpan-Tk-TextANSIColor
Building cpan-XML-Parser
build cpan-XML-Parser make part failed.
 at bin/shipwright-builder line 456
        main::_install('cpan-XML-Parser', 'GLOB(0x116f0e10)') called at bin/shipwright-builder line 346
        main::install() called at bin/shipwright-builder line 241

Well, that’s a bit more successful. Running with --verbose showed that the expat.h file couldn’t be found. I then determined that XML::Parser is needed by SOAP::Lite, which we don’t really need after all, so I switched to my Shipwright directory, changed to the vendor branch, removed SOAP::Lite, changed back to the master branch, then merged in the changes:

 % cd ~/development/oracdr-perl-modules
 % get checkout vendor
 % shipwright -r fs:`pwd` delete cpan-SOAP-Lite
 % shipwright -r fs:`pwd` delete cpan-XML-Parser
 % git add *
 % git rm -r scripts/cpan-SOAP-Lite
 % git rm -r scripts/cpan-XML-Parser
 % git rm -r sources/cpan-SOAP-Lite
 % git rm -r sources/cpan-XML-Parser
 % git commit -m 'Remove SOAP::Lite and XML::Parser.'
 % git checkout master
 % git merge vendor

Note the change in naming convention when using shipwright delete versus shipwright import! For CPAN modules, you import using cpan:SOAP::Lite and delete using cpan-SOAP-Lite.

Now, back to the test directory to start again:

 % cd ~/development/shipwright-test/oracdr-perl-modules
 % git pull
 % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder --clean
akule % $STARLINK_DIR/Perl/bin/perl bin/shipwright-builder
Building cpan-Task-Weaken
Building cpan-DateTime-Format-Strptime
Use of uninitialized value in string at bin/shipwright-builder line 443.
build cpan-DateTime-Format-Strptime test part failed.

This error is because of DateTime::Locale. You’ll note that the most recent version (as of this writing) is 0.43. Apparently this version causes one of the tests in DateTime::Format::Strptime to fail, so we need to tell Shipwright that we only want version 0.42 of DateTime::Locale. In our Shipwright repository we do:

 % git checkout vendor
 % shipwright update -r fs:`pwd` cpan-DateTime-Locale --version 0.42
imported with success
updated with success

Don’t let those two lines fool you! Nothing has actually happened here^[2]. If you run the last command with -l info you’ll see a message stating:

path scripts/cpan-DateTime-Locale alreay exists, need to set overwrite arg to overwrite

I wouldn’t call this “success”, so I filed a bug^[3]. Turns out the overwrite argument isn’t implemented (another bug filed) so I just deleted DateTime::Locale and imported version 0.42:

 % shipwright -r fs:`pwd` delete cpan-DateTime-Locale
 % shipwright -r fs:`pwd` import cpan:DateTime::Locale --version 0.42
 % git add *
 % git commit -m 'Revert DateTime::Locale to v0.42 to...'
 % git checkout master
 % git merge vendor

At this point I’m not going to show every little error that popped up. Let’s skip ahead to the end:

Building cpan-Astro-FITS-CFITSIO
install finished, the dists are at /tmp/vessel_oracdr-perl-modules-GwEUP5/oracdr-perl-modules

Success!

Sort of. At this point I’ve only done the CPAN modules. Now it’s time to do some of our own modules. Because our modules aren’t stored in separate git repositories, I have to make a tarball of each one and import that:

 % shipwright import -r fs:`pwd` file:/home/bradc/development/starlink/perlmods/Starlink/Config/Starlink-Config-1.01.tar.gz

Starlink-Config-1.01.tar.gz is the result of doing perl Makefile.PL && make dist in our starlink/perlmods/Starlink/Config directory, which is the standard method of creating a tarball for distribution. Shipwright can import these directly.

Unfortunately when you import a module using a tarball, Shipwright doesn’t seem to realize that it exists for the purposes of being an installation requirement for future modules. For example, our Astro::Telescope module depends on Astro::SLA. I imported Astro::SLA from a tarball, then tried to import Astro::Telescope. Instead of realizing that Astro::SLA was already imported, Shipwright imported it again from CPAN. This is a particular problem when you’ve imported a module that doesn’t exist on CPAN, as Shipwright tries to find it, then fails to, and then crashes with a scary error:

 % shipwright import -r fs:`pwd` file:/home/bradc/development/starlink/perlmods/Starlink/HDSPACK/Starlink-HDSPACK-undef.tar.gz
CPAN related output will be at /tmp/shipwright_cpan.log
Use of uninitialized value in numeric eq (==) at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Source/CPAN.pm line 85.
invalid source: can't find 'NDF' in your CPAN mirror(s) [http://cpan.mirror.facebook.com http://cpan.knowledgematters.net http://cpan.yahoo.com]. at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Source/CPAN.pm line 97
        Shipwright::Source::CPAN::run('Shipwright::Source::CPAN=HASH(0x789c8d0)') called at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Source/Base.pm line 441
        Shipwright::Source::Base::_follow('Shipwright::Source::Compressed=HASH(0x6ca0770)', '/tmp/shipwright_source_gIMm9O/Starlink-HDSPACK-undef') called at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Source/Compressed.pm line 33
        Shipwright::Source::Compressed::run('Shipwright::Source::Compressed=HASH(0x6ca0770)', 'copy', 'HASH(0x661ac80)') called at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Script/Import.pm line 157
        Shipwright::Script::Import::run('Shipwright::Script::Import=HASH(0x639b500)', 'file:/home/bradc/development/starlink/perlmods/Starlink/HDSPA...') called at /local/perl-5.8/lib/site_perl/5.8.8/App/CLI/Command.pm line 53
        App::CLI::Command::run_command('Shipwright::Script::Import=HASH(0x639b500)', 'file:/home/bradc/development/starlink/perlmods/Starlink/HDSPA...') called at /local/perl-5.8/lib/site_perl/5.8.8/App/CLI.pm line 79
        App::CLI::dispatch('Shipwright::Script') called at /local/perl/bin/shipwright line 16

Did you spot the key piece of information there? Here it is:

invalid source: can't find 'NDF' in your CPAN mirror(s) [http://cpan.mirror.facebook.com http://cpan.knowledgematters.net http://cpan.yahoo.com]. at /local/perl-5.8/lib/site_perl/5.8.8/Shipwright/Source/CPAN.pm line 97

Even though I’ve already imported NDF to this Shipwright repository, Shipwright can’t find it. I’ve filed a bug about this one^[4].

The only work-around I can see at this point is to add --no-follow to my imports. This means that I have to have figured out the module requirements and dependency tree before importing every module from this point, which luckily I’ve already done. However, if this had been early in the process and I was relying on Shipwright to figure all of this out for me (which it’s supposed to do) I’d be screwed.

I found that the NDF module was missing a file in its MANIFEST so it wouldn’t build properly. I fixed the module, recreated the tarball, and deleted and imported the module again:

 % shipwright delete -r fs:`pwd` NDF
 % shipwright import -r fs:`pwd` file:/home/bradc/development/starlink/perlmods/Starlink/NDF/NDF-1.48.tar.gz

Now unfortunately when you do this sort of thing Shipwright thinks you want to build this module at the end, when actually you just want to build it where it was before. If you’ve already imported a bunch and want to go back and fix a module in this way, things will build out of order and your dependency tree will be all screwed up. Since I’m keeping this repository in git, I just have to revert a couple of files:

 % git checkout -- shipwright/order.yml
 % git checkout -- scripts/NDF/require.yml

The shipwright/order.yml file tells Shipwright which order to build modules in, and since we want to keep that order, we just revert it to lose the changes imposed by the delete/import pair of commands. The scripts/NDF/require.yml file lists requirements for NDF — I don’t know if this is used by Shipwright anywhere, but we might as well revert it anyhow.

I’m skipping ahead here to the point where I’ve ironed out all the kinks in my Shipwright repository, and it’s time to create a vessel on each platform and install the modules in my Starlink distribution directory. For future reference, Starlink has Perl installed in /star/Perl, so that’s the base directory for this case.

I start by checking out the repository via git:

 % git clone file:///net/akule/export/data/bradc/backup/development/oracdr-perl-modules perlmods-nanahope

Once this is done, I create my vessel:

 % cd perlmods-nanahope
 % /star/Perl/bin/perl bin/shipwright-builder --install-base /tmp/shipwright-vessel

This goes about and, since I’ve fixed up all of the tests and compiles and whatnot, creates the vessel in /tmp/shipwright-vessel.

Because the library directory structure doesn’t match that for Perl itself, I can’t install directly into /star/Perl like I’d want. Instead I have to change to my /star/Perl/lib/perl5/site_perl/5.10.0 directory, then copy the Shipwright-created files over:

 % cd /star/Perl/lib/perl5/site_perl/5.10.0
 % rsync -avz /tmp/shipwright-vessel/lib/perl5/ ./

And I’m done! Well, not quite, as I have some scripts in /tmp/shipwright-vessel/bin that need copying over. However, those scripts are merely wrappers around calls to a Shipwright-specific script. If you’re just installing into a Perl tree, you should copy the files in /tmp/shipwright-vessel/bin-wrapped into your Perl bin directory instead, as it keeps things clean.

1. sunnavy, the author of Shipwright, notes below that a build.log file is written, and that contains the error messages. Just check that for the error messages instead of having to re-run with --verbose! [↩]
2. Actually, this whole section doesn’t apply to version 2.3 of Shipwright, as this version bug has since been fixed. [↩]
3. Like I said, this bug was fixed in Shipwright v2.3. [↩]
4. …which has since been closed because the developer doesn’t consider this a bug. You can see his reply and rationale by clicking that link. [↩]