JPEG2000 Encoding with CUDA

For the last few days we have been struggling to efficiently write out very large JPEG-2000 files. In all their wisdom Erdas decided to supply only a 32-bit version of the ECW compressor. It seems to have trouble allocating memory for large JPEG-2000 files, is single threaded and dead-slow. Somewhere in the Ermapper install directory there is also the Kakadu core dll. So I decided to have a look at other methods of creating JPEG-2000, possibly faster and more scalable options. Kakadu seems to only supply an SDK and we can test the speed of compression with the SDK. GDAL also supports building against the Kakadu SDK and would be handy for creating the large mosaics we have been struggling with. It seems with modern hardware that is not the fastest option.

After some trial with Kakadu and a bit of reading I came across a few CUDA based implementations - CUJ2K and JPEG2K.  I decided to port the most recently maintained CUDA based JPEG-2000 implementation, claiming to be the fastest on commodity hardware to windows. The CMake based build system makes things easier. The code however was written for a POSIX based systems so I was missing a bunch of headers. I had to hack away at it and move variables around to cope with lack of C99 compliance in the MSVC C-Compiler or I am forced to used the /TP flag to upgrade the C-files to C++ files and allow variable declaration anywhere. Lots of moving variables around later, everything worked fine.

The compression library is built against FreeImage which functions as an I/O driver for the encoder/decoder. However only writing to the J2K profile is supported, a wrapper for the JP2 mode will be required to add-in all the projection metadata GIS Rasters typically require. To make this library more useful for GIS purposes we will need to add an I/O handler for GDAL instead of FreeImage to allow faster compression/decompression.

I have tested the JP2ECW, JP2OpenJPEG and JP2Kakadu drivers on a small file (26MB in .bmp format, since the Kakadu demo binary does not support LZW .tif I have lying around). The times averaged over a few runs on an i7-3930K @ 3.2GHz are shown below:

• LibECW - 0.8s
• Kakadu (1 Thread) - 1.5s
• Kakadu (8 Threads) - 0.5s
• OpenJPEG - 2s

Compared to these the CUDA library on the GTX580 on the same machine runs at 1.2s (with compression taking 0.75s, rest is I/O).  The logical next step is hooking up the compressor to GDAL for data-block I/O and building a JP2 wrapper in addition to the J2K profile which it currently uses. The Affero GPL licence also needs to be sorted out.

For those interested in the windows build, here is the SVN Diff against rev 84 and the pre-built release binaries against CUDA 5.5.

Raster to Vector - and things in between

Coming from the remote sensing world, your day revolves around processing datasets from raster to vector and back. Grab a stereo satellite image of Melbourne, trawl through all the buildings in Leica Stereo view and digitize ground footprint, save to shapefile/kml/whatever vector format is the flavour of the day, render back to raster in 2.5D/3D. There is always information between the cracks of every pixel and delineated polygons and lines that is important but gets mangled in the constant transformation.

Spending some time in the NetCDF world I have become aware of the variety in scientific data and how the rigid Raster/Vector mentality of the GIS world does not really fit with it. Last year I went to the ARSPC Conference in Alice Springs and a Chris Tweedie (Technical Sales from Erdas) gave a talk on the joys of JPEG2000 and how all sorts of data can be stored in this format. There was somebody in the back of the room form IMOS/TERN projects disapproving at the disparity between the expectation of the GIS world and the scientific datasets he is used to hosting.

NetCDF has nice and fast access to gridded data sets with co-ordinate conventions allowing storage of data not only using cartesian grids but also with curvilinear grids as long as the parameters producing the grid are properly defined. Now there is move towards making this rather rasterlike approach more vectorlike with the idea of unstructured grids. Then NetCDF will become an even more general (and more complex) representation of all possible data - raster, vectors and everything in-between. There is real need for generalised (non vector/raster) representation of geographic information. Something that emphasizes on utility rather than conformance. The software to handle this data and ultimately the memory models need to think differently (in parallel) to stay in touch with reality. It does not help at all when there is a nomenclature clash, vector mean winds and currents in CMAR and mean points and polygons in GIS.

That aside recently I have spent a fair bit of time migrating from SVN to Mercurial. The OTB project made me fall in love with Mercurial. Afterwards Google code has fostered the obsession with working offline while making commits - aeroplane mode. There are myriads of ways for migrating from SVN to Hg - the stock convert extension, hgsvn or hgsubversion. You can even stay attached to your beloved subversion using subrepository techniques. Not everything however is smooth and some strange empty merging takes place along the way due to the repeated rebasing needed to stay in sync with SVN, hopefully the limbo will be over soon.