Hack OTB on the Clojure REPL
The WHY and the WHAT
Lately, I have been playing with several configurations in order to be able to fully exploit the ORFEO Toolbox library from Clojure.
As explained in previous posts, I am using OTB through the Java bindings. Recently, the folks of the OTB team have made very cool things – the application wrappers – which enrich even further the ways of using OTB from third party applications. The idea of the application wrappers is that when you code a processing chain, you may want to use it on the command line, with a GUI or even as a class within another application.
What Julien and the others have done is to build an application engine which given a pipeline and its description (inputs, outputs and parameters) is able to generate a CLI interface and a Qt-based GUI interface. But once you are there, there is no major problem (well, if your name is Julien, at least) to generate a SWIG interface which will then allow to build bindings for the different languages supported by SWIG. This may seem a kitchen sink – usine à gaz in French – and actually it is if you don't pay attention (like me).
The application wrappers use advanced techniques which require a recent version of GCC. At least recent from a Debian point of view (GCC 4.5 or later). On the other hand, the bindings for the library (the OTB-Wrapping package) use CableSwig, which in turn uses its own SWIG interfaces. So if you want to be able to combine the application wrappers with the library bindings, odds are that you may end up with a little mess in terms of compiler versions, etc.
And this is exactly what happened to me when I had to build GCC from source in Debian Squeeze (no GCC 4.5 in the backports yet). So I started making some experiments with virtual machines in order to understand what was happening. Eventually I found that the simplest configuration was the best to sort things out. This is the HOW-TO for having a working installation of OTB with application wrappers, OTB-Wrapping and access everything from a Clojure REPL.
Installing Arch Linux on a virtual machine
I have chosen Arch Linux because with few steps you can have a minimal Linux install. Just grab the net-install ISO and create a new virtual machine on VirtualBox. In terms of packages just get the minimal set (the default selection, don't add anything else). After the install, log in as root, and perform a system update with:
If you used the net-install image, there should be no need for this step, but in case you used an ISO with the full system, it's better to make sure that your system is up to date. Then, create a user account and set up a password if you want to:
useradd -m jordi passwd jordi
Again, this is not mandatory, but things are cleaner this way, and you may want to keep on using this virtual machine. Don't forget to add your newly created user to the sudoers file:
pacman -S sudo vi /etc/sudoers
You can now log out and log in with the user name you created. Oh, yes, I forgot to point out that you don't have Xorg, or Gnome or anything graphical. Even the mouse is useless, but since you are a real hacker, you don't mind.
Install the packages you need
Now the fun starts. You are going to keep things minimal and only install the things you really need.
sudo pacman -S mercurial make cmake gcc gdal openjdk6 \ mesa swig cvs bison links unzip rlwrap
Yes, I know, Mesa (OpenGL) is useless without X, but we want to check that everything builds OK and OTB uses OpenGL for displaying images.
- Mercurial is needed to get the OTB sources, cvs is needed for getting CableSwig
- Make, cmake and gcc are the tools for building from sources
- Swig is needed for the binding generation, and bison is needed by CableSwig
- OpenJDK is our Java version of choice
- Links will be used to grab the Clojure distro, unzip to extract the corresponding jar file and rlwrap is used by the Clojure REPL.
And that's all!
Get the source code for OTB and friends
Start building everything
OTB and OTB-Wrapping
We create a directory for the OTB build and configure with CMake
mkdir OTB cd OTB ccmake ../../OTB
Don't forget to set to ON the application build and the Java wrappers. Then just make (literally):
By the way, I have noticed that the compilation of the application engine can blow up your gcc if you don't allocate enough RAM for your virtual machine. At this point, you should be able to use the Java application wrappers. But we want also the library bindings so we gon on. We can now build CableSwig which will be needed by OTB-Wrapping. Same procedure as before:
cd ../ mkdir CableSwig cd CableSwig/ ccmake ../../CableSwig/ make
And now, OTB-Wrapping. Same story:
cd ../ mkdir OTB-Wrapping cd OTB-Wrapping/ ccmake ../../OTB-Wrapping/
In the cmake configuration, I choose only to build Java, but even in this case a Python interpreter is needed. I think that CableSwig needs it to generate XML code to represent the C++ class hierarchies. If you did not install Python explicitly in Arch, you will have by default a 2.7 version. This is OK. If you decided to install Python with pacman, you will have both, Python 2.7 and 3.2 and the default Python executable will point to the latter. In this case, don't forget set the PYTHON\EXECUTABLE in CMake to /usr/bin/python2. Then, just make and cd to your home directory.
And you are done. Well not really. Right now, you can do Java, but what's the point? You might as well use the C++ version, right?
Land of Lisp
Since Lisp is the best language out there, and OTB is the best remote sensing image processing software (no reference here, but trust me), we'll do OTB in Lisp.
You may want to get some examples that I have gathered on Bitbucket.
mkdir Dev cd Dev/ hg clone http://bitbucket.org/inglada/pwob
PWOB stands for Playing With OTB Bindings. I have only put there 3 languages which run on the JVM for reasons I stated in a previous post. You will of course avoid the plain Java ones. I have mixed feelings about Scala. I definetly love Clojure since it is a Lisp.
The cool thing about this Lisp implementation is that it is contained into a jar file. You can get it with the text-based web browser links:
Go to the download page and grab the latest release. It is a zip file which contains, among other things the needed jar. You can unzip the file:
mkdir src mv clojure-1.3.0.zip src/ cd src/ unzip clojure-1.3.0.zip
Copy the jar file to the user .clojure dir:
cd clojure-1.3.0 mkdir ~/.clojure mv clojure-1.3.0.jar ~/.clojure/
Make a sym link so we have a clojure.jar:
ln -s /home/inglada/.clojure/clojure-1.3.0.jar /home/inglada/.clojure/clojure.jar
And clean up useless things
cd .. rm -rf clojure-1.3.0*
Final steps before hacking
Some final minor steps are needed before the fun starts. You may want to create a file for the REPL to store completions:
In order for Clojure to find all the jars and shared libraries, you have to define some environment variables. You may choose to set them into your .bashrc file:
export LD_LIBRARY_PATH="~/builds/OTB-Wrapping/lib/:~/builds/OTB/bin/" export ITK_AUTOLOAD_PATH="~/builds/OTB/bin/"
PWOB provides a script to run a Clojure REPL with everything set up:
cd ~/Dev/pwob/Clojure/src/Pwob ./otb-repl.sh
Now you should see something like this:
Clojure 1.3.0 user=>
Welcome to Lisp! If you want to use the applications you can for instance do:
(import '(org.otb.application Registry)) (def available-applications (Registry/GetAvailableApplications))
In the PWOB source tree you will find other examples.
Using the REPL is fun, but you will soon need to store the lines of code, test things, debug, etc. In this case, the best choice is to use SLIME, the Superior Lisp Interaction Mode for Emacs. There are many tutorials on the net on how to set it up for Clojure. Search for it using DuckDuckGo. In the PWOB tree (classpath.clj) you will find some hints on how to set it up for OTB and Clojure. A simpler config for Emacs is to use the inferior lisp mode, for which I have also written a config (otb-clojure-config.el). I may write a post some day about that. Have fun!