Create local Ivy repository guide
This article will introduce you how to create a local repository with Ivy, and if you have similar requirements that I had, you might use the supplemental tools I have developed. It is a bit longer than most would expect, but it explains a few automatic process, so the time you will invest in this will come back for sure.
Apache Ivy is a great dependency manager tool, mostly used for Java. It is an "extension" to Apache Ant, and many compare the combination of Ant+Ivy to Maven. The comparison is good and correct, because together they create similar build- and dependency- management. My observation is that while Maven is great for open source developments, it is lesser reliable and not preferred for in-house developments
And this is the place where Ivy+Ant shines. Ant is great for software builds and even for distributions, while Ivy could precede Ant and handle the dependencies of a library or other artifacts.
Ivy is able to track software dependencies in a highly configurable manner. While Maven usually forces you to think the same way the Maven developers thought it was right, Ivy gives you freedom on how you define your repository. Of course, this freedom could be misused, but it pleases the IT directors and architects that they are able to control everything.
My requirements
While Ivy is able to use Maven's (ibiblio) repository out of box, and they created a short tutorial on how to build a repository, I have found these options behind my expectations, which were the following:
-
Many open source libraries are not present in Maven's repositories, or even if they are, they might be out-dated, while the trunk version contains critical fixes (for example openid4java and scribe was such ones for me recently).
-
Many open source libraries contain unreasonable dependencies (like pre-Java-5 libs that are part of the JDK anyway) or reference e.g. commons-logging, which I prefer to dodge in favor of slf4j.
-
I want to own my repository. I want to be able to extend it at the time I wish, to introduce or remove dependencies as I see them fit. Yeah, I can be considered control-freak :)
The repository layout
I am suggesting the following layout:
repo root
|- organization
|- revision
|- module
|- artifact-revision.jar
|- artifact-sources-revision.zip
|- organization
|- ...
This is not the usual organization / module / revision structure, because I have moved the revision (version) in a higher priority position, but I have my reasons:
-
Most software libraries evolve over time and introduce new modules within the larger release. For example Apache Wicket introduced wicket-jmx module in 1.3, the wicket-util in 1.5. That means you have something like this:
repo-root |- org.apache.wicket |- wicket-jmx |- 1.5 |- 1.4.14 |- wicket-util |- 1.5No 1.4.14 for wicket-util? How shall one know if it is missing because Joe forgot to place it there (human might happen every time error), or because the release hadn't contained it?
-
If you download a new version, it is much easier to put it in the right place.
-
Similarly, if you will delete a deprecated release of a given library, it is much easier to delete one single library.
So instead of the above, I'd like to see something like the following:
repo-root
|- org.apache.wicket
|- 1.4.14
|- ...
|- wicket-jmx
|- wicket-jmx-1.4.14.jar
|- wicket-jmx-sources-1.4.14.zip
|- 1.5.0
|- ...
|- wicket-jmx
|- wicket-jmx-1.5.0.jar
|- wicket-jmx-sources-1.5.0.zip
|- wicket-util
|- wicket-util-1.5.0.jar
|- wicket-util-sources-1.5.0.zip
It was a bit strange for me too at first, but in a short time it became so evident that I wonder why haven't I seen this previously at other places. There might be good reasons not to do this way, but for now, I can see more pros than cons.
Installing Ivy
Installing Ivy is not hard, although the original documentation is a bit complex. Just download and unzip the latest release and copy the ivy.jar into ~/.ant/lib/
Downloading libraries - the hard work
I hate to download libraries manually. This is a trivial task that any little script is capable of, and why not use Ivy to download most of the libraries from the Maven repositories?
I've created a simple Ant macro that can handle Ivy's dependency configuration with copy-and-paste:
<!-- this macro will retrieve the specified jars, sources and related javadocs -->
<macrodef name="dependency">
<attribute name="org" />
<attribute name="name" />
<attribute name="rev" />
<attribute name="conf" />
<sequential>
<ivy:retrieve inline="true" type="jar" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].[ext]" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
<ivy:retrieve inline="true" type="src,source,sources" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].zip" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
<ivy:retrieve inline="true" type="doc,docs,javadoc,javadocs" transitive="true" pattern="${repo.downloader.dir}/[organisation]/[revision]/[module]/[artifact]-[revision].zip" organisation="@{org}" module="@{name}" revision="@{rev}" conf="@{conf}" />
</sequential>
</macrodef>
I prefer to use SpringSource's Enterprise Bundle Repository with Maven Central, so this is my ivy-settings.xml file:
<ivysettings>
<settings defaultCache="${repo.downloader.cache}" defaultResolver="spring-chain" />
<resolvers>
<chain name="spring-chain">
<url name="com.springsource.repository.bundles.release">
<ivy pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/release/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<url name="com.springsource.repository.bundles.external">
<ivy pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
<artifact pattern="http://repository.springsource.com/ivy/bundles/external/[organisation]/[module]/[revision]/[artifact]-[revision].[ext]" />
</url>
<ibiblio name="ibiblio" m2compatible="true"/>
</chain>
</resolvers>
</ivysettings>
We need a few further variables in our ant build.xml and we are able to specify the libraries to download:
<project xmlns:ivy="antlib:org.apache.ivy.ant" default="download">
<property file="${user.home}/.ant/${user.name}.properties" />
<property file="${user.home}/.ant/defaults.properties" />
<property name="repo.downloader.ivy" value="${basedir}/ivy-settings.xml" />
<property name="repo.downloader.dir" value="${user.home}/.ivy2/download" />
<property name="repo.downloader.cache" value="${user.home}/.ivy2/download-cache" />
<macrodef name="dependency">
<!-- see the definition above -->
</macrodef>
<target name="download">
<ivy:settings file="${repo.downloader.ivy}" />
<!-- Some SpringSource library -->
<dependency org="org.springframework" name="org.springframework.core" rev="3.0.5.RELEASE" conf="runtime" />
<dependency org="org.springframework" name="org.springframework.context" rev="3.0.5.RELEASE" conf="runtime" />
<!-- ... -->
<dependency org="javax.servlet" name="com.springsource.javax.servlet" rev="2.5.0" conf="runtime"/>
<!-- Some Maven Central library -->
<dependency org="com.hazelcast" name="hazelcast" rev="1.9" conf="default,sources"/>
<dependency org="com.amazonaws" name="aws-java-sdk" rev="1.1.1" conf="default,sources"/>
<!-- ... -->
<!-- and list everything else here -->
</target>
</project>
That's it, we are able to download the files with the following command:
ant download
You can access the related source codes at Agilord's BitBucket repository, in the repo-downloader directory.
Moving files
The previous command should have downloaded the universe and a bit beyond, with every dependency that was spacified anywhere. I think the most important part is to replace some of the organization parts into meaningful names. Spring's EBR can be a good naming convention, here is a few names from my actual repo:
...
com.sun.xml.bind
com.sun.xml.fastinfoset
javax.jms
javax.mail
javax.servlet
net.sf.ehcache
org.aopalliance
org.apache.ant
org.apache.commons.codec
org.apache.commons.dbcp
org.apache.commons.httpclient
org.apache.commons.logging
org.apache.commons.pool
org.apache.httpcomponents
org.apache.lucene
org.apache.wicket
org.apache.xerces
org.cyberneko.html
org.freemarker
org.junit
...
As you have noticed, I've created a separate organization for each apache commons library, because they live a separate life (httpclient as a famous example that even version changes can be dramatic).
And what is inside? This:
# find org.freemarker
org.freemarker
org.freemarker/2.3.16
org.freemarker/2.3.16/freemarker
org.freemarker/2.3.16/freemarker/freemarker-2.3.16.jar
org.freemarker/2.3.16/freemarker/freemarker-sources-2.3.16.zip
org.freemarker/2.3.16/freemarker/ivy.xml
Simple and clean, but how did I create the ivy.xml? :)
Generating the default ivy.xml
Although Ivy can be used to define large number of configurations, dependency rules and transition properties, for the main (external, open-source) repository, I prefer to keep things simple. My ivy.xml is usually like this:
<?xml version="1.0" encoding="ISO-8859-1"?>
<ivy-module version="1.0">
<info organisation="org.freemarker" module="freemarker" status="release" revision="2.3.16">
</info>
<configurations>
<conf name="default" />
<conf name="provided" />
<conf name="test" />
</configurations>
<publications>
<artifact name="freemarker" type="jar"/>
<artifact name="freemarker-sources" type="src" ext="zip" />
</publications>
</ivy-module>
I do not like to type such long and simple-minded text, so to shorten to repository I've created an Ant task to shorten the required effort. You can access the related source codes at Agilord's BitBucket repository, in the ivy-xml directory, or download the compiled jar from here.
It is hacky and really something that was writtin in just a few minutes, and just for my own requirements, but you might use it anyhow. Once you have downloaded, copy the jar file in ~/.ant/lib/ and you will be able to generate the ivy.xml files with the following little build.xml:
<project default="ivy-xml">
<property file="${user.home}/.ant/${user.name}.properties" />
<property file="${user.home}/.ant/defaults.properties" />
<property name="repo.dir" value="${user.home}/.ivy2/repo" />
<taskdef name="ivy-xml" classname="com.agilord.build.ant.GenerateIvyXmlTask" />
<target name="ivy-xml">
<ivy-xml repo="${repo.dir}" />
</target>
</project>
Now, you will have ivy.xml in every directory and you are free to overwrite/extend the defaults, because the above code will not overwrite existing files.
Conclusions
I prefer to keep the customizations low, because at the time I do library upgrades, it will happen easily:
- just downloading from maven or compiling from sources
- moving to the right place
- running ivy.xml generation
- start using it
If I would specify more dependency, I would be required to update all of them in the ivy.xml files I own. There are cases and large organization when this is a hard requirement, but for small teams and startups, convention over configuration might do as well. Of course others might have different requirements, and this solution does not fit for everyone, but it might be a good starting point for an organization to adopt Ivy and customize it as they move forward.
