Local Apache Ivy repository guide

Apache Ivy is a great dependency manager tool, mostly used for Java. It is an "extension" to Apache Ant, and many compare the combination of Ant+Ivy to Maven. The comparison is fair, because together they create similar build- and dependency- management. My observation is that while Maven is great for open source development, it is not as much preferred for repeatable builds, and more controlled in-house development.

Ant is great for software builds and even for distributions, while Ivy could precede Ant and handle the dependencies of a library or other artifacts.

Ivy is able to track software dependencies in a highly configurable manner. While Maven usually forces you to think the same way the Maven developers thought it was right, Ivy gives you freedom on how you define your repository. Of course, this freedom could be misused, but it pleases the IT directors and architects that they are able to control everything.

My requirements

While Ivy is able to use Maven’s (ibiblio) repository out of box, and they created a short tutorial on how to build a repository, I have found these options behind my expectations, which were the following:

• Many open source libraries are not present in Maven’s repositories, or even if they are, they might be out-dated, while the trunk version contains critical fixes (for example openid4java and scribe was such ones for me recently).

• Many open source libraries contain unreasonable dependencies (like pre-Java-5 libs that are part of the JDK anyway) or reference e.g. commons-logging, which I prefer to dodge in favor of slf4j.

• I want to own my repository. I want to be able to extend it at the time I wish, to introduce or remove dependencies as I see them fit. Yeah, I can be considered control-freak :)

The repository layout

I am suggesting the following layout:

repo root
|- organization
   |- revision
      |- module
         |- artifact-revision.jar
         |- artifact-sources-revision.zip
|- organization
   |- ...

This is not the usual organization / module / revision structure, because I have moved the revision (version) in a higher priority position, but I have my reasons:

• Most software libraries evolve over time and introduce new modules within the larger release. For example Apache Wicket introduced wicket-jmx module in 1.3, the wicket-util in 1.5. That means you have something like this:

repo-root
|- org.apache.wicket
   |- wicket-jmx
      |- 1.5
      |- 1.4.14
   |- wicket-util
      |- 1.5

No 1.4.14 for wicket-util? How shall one know if it is missing because a developer forgot to put it there, or because the release had no such artifact?

• If you download a new version, it is much easier to put it in the right place.

• Similarly, if you will delete a deprecated release of a given library, it is much easier to delete one single directory.

Therefore, I’d like to see something like the following:

repo-root
|- org.apache.wicket
   |- 1.4.14
      |- ...
      |- wicket-jmx
         |- wicket-jmx-1.4.14.jar
         |- wicket-jmx-sources-1.4.14.zip
   |- 1.5.0
      |- ...
      |- wicket-jmx
         |- wicket-jmx-1.5.0.jar
         |- wicket-jmx-sources-1.5.0.zip
      |- wicket-util
         |- wicket-util-1.5.0.jar
         |- wicket-util-sources-1.5.0.zip

It was a bit strange for me at first, but in a short time it became a second nature. There might be good reasons not to do this way, but for now, I can see more pros than cons.

Installing Ivy

Installing Ivy is not hard, although the official guide is a bit complex. Just download and unzip the latest release and copy the ivy.jar into ~/.ant/lib/.

Downloading libraries - the hard part

I hate to download libraries manually. This is a trivial task that any little script is capable of, and why not use Ivy to download most of the libraries from the Maven repositories?

The following Ant macro uses Ivy to retrieve the requested library and its dependencies:

<!-- this macro will retrieve the specified jars, sources and related javadocs -->
<macrodef name="dependency">
    <attribute name="org" />
    <attribute name="name" />
    <attribute name="rev" />
    <attribute name="conf" />
    <sequential>
        <ivy:retrieve inline="true" type="jar" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].[ext]" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
        <ivy:retrieve inline="true" type="src,source,sources" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].zip" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
        <ivy:retrieve inline="true" type="doc,docs,javadoc,javadocs" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].zip" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
    </sequential>
</macrodef>

I prefer to use SpringSource’s Enterprise Bundle Repository with Maven Central, this is my ivy-settings.xml file:

<ivysettings>
    <settings defaultCache="${repo.downloader.cache}" defaultResolver="spring-chain" />
    <resolvers>
        <chain name="spring-chain">
            <url name="com.springsource.repository.bundles.release">
                <ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
                <artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
            </url>
            <url name="com.springsource.repository.bundles.external">
                <ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
                <artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
            </url>
            <ibiblio name="ibiblio" m2compatible="true"/>
        </chain>
    </resolvers>
</ivysettings>

We need a few further variables in our ant build.xml and we are able to specify the libraries to download:

<project xmlns:ivy="antlib:org.apache.ivy.ant" default="download">

    <property file="${user.home}/.ant/${user.name}.properties" />
    <property file="${user.home}/.ant/defaults.properties" />

    <property name="repo.downloader.ivy" value="${basedir}/ivy-settings.xml" />
    <property name="repo.downloader.dir" value="${user.home}/.ivy2/download" />
    <property name="repo.downloader.cache" value="${user.home}/.ivy2/download-cache" />

    <macrodef name="dependency">
        <!-- see the definition above -->
    </macrodef>

    <target name="download">
        <ivy:settings file="${repo.downloader.ivy}" />

        <!-- Some SpringSource library -->
        <dependency org="org.springframework" name="org.springframework.core" rev="3.0.5.RELEASE" conf="runtime" />
        <dependency org="org.springframework" name="org.springframework.context" rev="3.0.5.RELEASE" conf="runtime" />
        <!-- ... -->
        <dependency org="javax.servlet" name="com.springsource.javax.servlet" rev="2.5.0" conf="runtime"/>

        <!-- Some Maven Central library -->
        <dependency org="com.hazelcast" name="hazelcast" rev="1.9" conf="default,sources"/>
        <dependency org="com.amazonaws" name="aws-java-sdk" rev="1.1.1" conf="default,sources"/>
        <!-- ... -->

        <!-- and list everything else here -->
    </target>
</project>

That’s it, we are able to download the files with the following command:

ant download

Moving files

The previous command should have downloaded the universe and a bit beyond, with every dependency that was specified anywhere. I think the most important part is to replace some of the organization parts into meaningful names. Spring’s EBR can be a good naming convention, here is a list of names from my current repository:

...
com.sun.xml.bind
com.sun.xml.fastinfoset
javax.jms
javax.mail
javax.servlet
net.sf.ehcache
org.aopalliance
org.apache.ant
org.apache.commons.codec
org.apache.commons.dbcp
org.apache.commons.httpclient
org.apache.commons.logging
org.apache.commons.pool
org.apache.httpcomponents
org.apache.lucene
org.apache.wicket
org.apache.xerces
org.cyberneko.html
org.freemarker
org.junit
...

As you have noticed, I’ve created a separate organization for each apache commons library, because they live a separate life (httpclient as a famous example that even small version changes can have dramatic effect).

And what is inside? This:

# find org.freemarker

org.freemarker
org.freemarker/2.3.16
org.freemarker/2.3.16/freemarker
org.freemarker/2.3.16/freemarker/freemarker-2.3.16.jar
org.freemarker/2.3.16/freemarker/freemarker-sources-2.3.16.zip
org.freemarker/2.3.16/freemarker/ivy.xml

Simple and clean, but how did I create the ivy.xml?

Generating the default ivy.xml

Although Ivy can be used to define large number of configurations, dependency rules and transition properties, for the main (external, open-source) repository, I prefer to keep things simple. My ivy.xml is usually like this:

<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="1.0">
    <info organisation="org.freemarker" module="freemarker" status="release" revision="2.3.16">
    </info>
    <configurations>
        <conf name="default" />
        <conf name="provided" />
        <conf name="test" />
    </configurations>
    <publications>
        <artifact name="freemarker" type="jar"/>
        <artifact name="freemarker-sources" type="src" ext="zip" />
    </publications>
</ivy-module>

This looks like a templatable file, and I’ve created an Ant task to shorten the required effort. You can access the related source codes at Agilord’s BitBucket repository, in the ivy-xml directory, or download the compiled jar.

It is something that was writtin in just a few minutes, and just for my own requirements, but you might adapt for your stuff too. Once you have downloaded, copy the jar file in ~/.ant/lib/ and you will be able to generate the ivy.xml files with the following build.xml:

<project default="ivy-xml">
    <property file="${user.home}/.ant/${user.name}.properties" />
    <property file="${user.home}/.ant/defaults.properties" />
    <property name="repo.dir" value="${user.home}/.ivy2/repo" />

    <taskdef name="ivy-xml" classname="com.agilord.build.ant.GenerateIvyXmlTask" />
    <target name="ivy-xml">
        <ivy-xml repo="${repo.dir}" />
    </target>
</project>

You will have ivy.xml in every directory and you may modify the file, because the script will not overwrite existing files.

Conclusions

I prefer to keep the number of customizations low, because library upgrades will became simpler:

• download from maven or compile from sources
• move to the right place
• run ivy.xml code generation
• start using it

If I would specify more dependencies, I would be required to update all of them in the ivy.xml files I own. There are cases and large organization when this is a strict requirement, but for small teams and startups, convention over configuration might work as well. Of course this solution is not a fit for everyone, but it might be a good starting point to adopt Ivy.