On Square's “Maven, Gems, and a JRuby Console for All”

Stephen Connolly's picture

Yesterday the guys a Square published an interesting post on how they use Maven to build an überjar which bundles JRuby and a pile of gems to give them an embedded JRuby console. When I read that post I immediately thought there must be a better way to do this…

So I did some digging… And the answer is of course that some of the plugins could do with improvement…

The first thing that smells bad is that they had to exclude the gem artifacts from the maven shade plugin… In fact if you don’t specify

<artifactSet>
 <excludes>
 <exclude>*:*:gem:*</exclude>
 </excludes>
</artifactSet>

then you get an error message like

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:2.0:shade (default) on project shade-gems: Error creating shaded jar: error in opening zip file ~/.m2/repository/rubygems/bundler/1.3.4/bundler-1.3.4.gem -> [Help 1]

That sound nasty… what is going on there?

Well the issue is that .gem files are not .zip files… they are actually .tar files, e.g.

$ unzip -l bundler-1.3.4.gem
Archive: bundler-1.3.4.gem
 End-of-central-directory signature not found. Either this file is not
 a zipfile, or it constitutes one disk of a multi-part archive. In the
 latter case the central directory and zipfile comment will be found on
 the last disk(s) of this archive.
unzip: cannot find zipfile directory in one of bundler-1.3.4.gem or
bundler-1.3.4.gem.zip, and cannot find bundler-1.3.4.gem.ZIP, period.
$ tar -tvf bundler-1.3.4.gem
-rw-r--r-- 0 wheel wheel 263115 1 Jan 1970 data.tar.gz
-rw-r--r-- 0 wheel wheel 2278 1 Jan 1970 metadata.gz

So the Maven Shade Plugin is complaining because a file that is part of the classpath is neither a directory nor a .zip file… well who exactly is putting that file on the classpath… My first thought is that the gem-maven-plugin is telling Maven that .gem files should be added to the Java classpath… but if we look we see that jar-gem files should be added to the classpath but gem files are excluded though I should add that without 

<plugin>
 <groupId>de.saumya.mojo</groupId>
 <artifactId>gem-maven-plugin</artifactId>
 <version>1.0.0-beta</version>
 <executions>
 <execution>
 <goals>
 <goal>initialize</goal>
 </goals>
 </execution>
 </executions>
 <extensions>true</extensions>
 </plugin>

Maven does not know the “gem” type of artifacts and an artifact of an unknown type will not be added to the classpath… so that shouldn’t be an issue… next point of call is the Maven Shade Plugin… and wait just a second… we have found a bug. The Maven Shade Plugin will blindly seek to add every dependency that is not of type pom to the shaded jar and assumes that the artifacts are .zip files. So I have gone and created MSHADE-152

The next bad smell is that you need to use build-helper:add-resource to add the gems as resources… Now I am not a ruby expert, let alone a JRuby expert, but it would seem to me that this would be something that was somewhat common… and in fact something that there should be a way of doing without resorting to build-helper:add-resource… so I went digging in the gem-maven-plugin source and found this nice parameter: includeRubygemsInResources. The bad news is that the parameter was added in April 2013 and the most recent release of the plugin is March 2013… now I cannot be certain that this parameter will do the same as the current build-helper:add-resource hack… but it certainly looks like it will do the right thing so that should be somewhat better…

So the long and the short of this…

Well when the next release of gem-maven-plugin is cut and MSHADE-152 gets fixed their pom configuration should end up looking a lot better e.g. 

 <plugin>
 <groupId>de.saumya.mojo</groupId>
 <artifactId>gem-maven-plugin</artifactId>
 <version>1.0.0-beta-1</version>
 <executions>
 <execution>
 <goals>
 <goal>initialize</goal>
 </goals>
 </execution>
 </executions>
+ <extensions>true</extensions>
+ <configuration>
+ <includeRubygemsInResources>true</includeRubygemsInResources>
+ <includeRubygemsInTestResources>false</includeRubygemsInTestResources>
+ </configuration>
 </plugin>
 
- <plugin>
- <groupId>org.codehaus.mojo</groupId>
- <artifactId>build-helper-maven-plugin</artifactId>
- <version>1.7</version>
- <executions>
- <execution>
- <id>add-resource</id>
- <phase>generate-sources</phase>
- <goals>
- <goal>add-resource</goal>
- </goals>
- <configuration>
- <resources>
- <resource>
- <directory>${project.build.directory}/rubygems</directory>
- <targetPath>rubygems</targetPath>
- </resource>
- </resources>
- </configuration>
- </execution>
- </executions>
- </plugin>
  
 <plugin>
 <groupId>org.apache.maven.plugins</groupId>
 <artifactId>maven-shade-plugin</artifactId>
 <version>3.0</version>
 <executions>
 <execution>
 <phase>package</phase>
 <goals>
 <goal>shade</goal>
 </goals>
 <configuration>
 <shadedArtifactAttached>true</shadedArtifactAttached>
 <shadedClassifierName>shaded</shadedClassifierName>
- <artifactSet>
- <excludes>
- <exclude>*:*:gem:*</exclude>
- </excludes>
- </artifactSet>
 ... 

Now that is a damn sight better (add 5 lines and remove 27, net improvement of 22 lines) and closer to the Maven way too… pity we’re not there just yet.

Update

On thinking about this some more, I actually think that the current direction the gem-maven-plugin is taking with its includeRubygemsInResources and includeRubygemsInTestResources parameters is also not aligned with the Maven way.

Maven dependencies have a <scope> which can be compile/provided/runtime/test… now I can (and do) argue that there are a couple of important scopes missing, but it is actually the dependency scope which should determine which classpath the gems get added to.

The compile and provided scope dependencies should be added up front at the beginning. The test and runtime scoped dependencies should be added immediately before the test-compile and test (or perhaps prepare-package) phases respectively.

Now I am not saying that is easy. for one thing gem is a dependency resolution framework, so the gem plugin would have to do some complex hoop jumping to decide which gems got which scope… or else it would need to rely on the gem poms listing the correct dependencies and that gem’s resolution of dependencies would agree with Maven’s… but that’s an issue for the gem maven plugin authors…

The point is that the “maven way” to handle it would be by inspecting the dependency scope and putting the gems on the corresponding classpaths automatically. 

—Stephen Connolly

CloudBees

www.cloudbees.com


Stephen Connolly has over 20 years experience in software development. He is involved in a number of open source projects, including JenkinsStephen was one of the first non-Sun committers to the Jenkins project and developed the weather icons. Stephen lives in Dublin, Ireland - where the weather icons are particularly useful. Follow Stephen on Twitter and on his blog.

Add new comment