JPEG2000 Encoding with CUDA

For the last few days we have been struggling to efficiently write out very large JPEG-2000 files. In all their wisdom Erdas decided to supply only a 32-bit version of the ECW compressor. It seems to have trouble allocating memory for large JPEG-2000 files, is single threaded and dead-slow. Somewhere in the Ermapper install directory there is also the Kakadu core dll. So I decided to have a look at other methods of creating JPEG-2000, possibly faster and more scalable options. Kakadu seems to only supply an SDK and we can test the speed of compression with the SDK. GDAL also supports building against the Kakadu SDK and would be handy for creating the large mosaics we have been struggling with. It seems with modern hardware that is not the fastest option.

After some trial with Kakadu and a bit of reading I came across a few CUDA based implementations - CUJ2K and JPEG2K.  I decided to port the most recently maintained CUDA based JPEG-2000 implementation, claiming to be the fastest on commodity hardware to windows. The CMake based build system makes things easier. The code however was written for a POSIX based systems so I was missing a bunch of headers. I had to hack away at it and move variables around to cope with lack of C99 compliance in the MSVC C-Compiler or I am forced to used the /TP flag to upgrade the C-files to C++ files and allow variable declaration anywhere. Lots of moving variables around later, everything worked fine.

The compression library is built against FreeImage which functions as an I/O driver for the encoder/decoder. However only writing to the J2K profile is supported, a wrapper for the JP2 mode will be required to add-in all the projection metadata GIS Rasters typically require. To make this library more useful for GIS purposes we will need to add an I/O handler for GDAL instead of FreeImage to allow faster compression/decompression.

I have tested the JP2ECW, JP2OpenJPEG and JP2Kakadu drivers on a small file (26MB in .bmp format, since the Kakadu demo binary does not support LZW .tif I have lying around). The times averaged over a few runs on an i7-3930K @ 3.2GHz are shown below:

• LibECW - 0.8s
• Kakadu (1 Thread) - 1.5s
• Kakadu (8 Threads) - 0.5s
• OpenJPEG - 2s

Compared to these the CUDA library on the GTX580 on the same machine runs at 1.2s (with compression taking 0.75s, rest is I/O).  The logical next step is hooking up the compressor to GDAL for data-block I/O and building a JP2 wrapper in addition to the J2K profile which it currently uses. The Affero GPL licence also needs to be sorted out.

For those interested in the windows build, here is the SVN Diff against rev 84 and the pre-built release binaries against CUDA 5.5.

A simple ray tracer in Python - with CGKit and RTree

First of all I am over-reaching by calling this a ray-tracer, it is more appropriately called a "Splatter". I am simply projecting all the top-triangles in an OBJ model onto a raster for True Ortho generation purposes. It was a fun project for playing with Kd-trees and line-triangle intersections. The OBJ models are geo-referenced so it does not make sense to take them into a purely CG/Game oriented ray-tracer like Blender and lose the geographic context, as well as fiddle around with setting up orthographic cameras and render resolution to match ground-sampling distance. Doing it through a proper ray-tracer typically also requires setting up appropriate lighting, while I just want to sample the pre-exisiting texture in the model.

Direct vertical projection render from script

Properly lit render using Terragen

As usual a bit of googling located the python batteries that will perform the intersection as well as parse the obj mesh I have at hand. CGKit fits the bill perfectly both in terms of parsing as well as ray-triangle intersection.

However testing every triangle for intersection for every grid-cell quickly becomes horridly slow and begs for a hierarchial index to speed up the look-up and only process the triangles in the path of the ray. Here I picked up RTree to build a 3D index , a kD-Tree of Bounding volume hierarchy to speed up intersection testing.

After building the tree I went back to only testing relevant triangles in CGKit and dug through the parsed OBJ data structure to extract the texture co-ordinates of the intersected face. The barycentric description of the intersection point is sufficient to sample the texture for the right pixel colour and transfer it to the raster grid. For those interested in the implementation, the code follows, improvement suggestions are welcome. Multiprocessing is thrown in for tile-by-tile processing.

The point of the ray-tracer was to generate true-orthos for GIS use, as a side effect of implementing the ray tracer we can now generate true-orthos from both the top and the bottom of the model. The main article about true-orthos can be found here.

Making 64-bit installers with CMake and NSIS

I was getting tired of my 64-bit programs getting installed by NSIS into the x86 folder under windows, just because the NSIS installer itself is 32-bit.

I started piecing the solution together from this bug report for CMake. NSIS has been ported to 64bit but the inclusion in CMake is only available in the latest release. So I had to get NSIS 64,  and latest version of CMake 2.8.11 and then make some modifications to CPack to enable the new 64-bit installer.

Change installer type to NSIS64: SET(CPACK_GENERATOR NSIS64) and re-package your application. Voila you get correct program file target location and here is a screenshot of the running application just for fun.

Some products such as PCL do a platform detection while building to suggest an install folder, using the snippet below. This method will also work if you don't feel like installing NSIS64.

if(CMAKE_CL_64)
    set(CPACK_NSIS_INSTALL_ROOT "$PROGRAMFILES64")
    set(win_system_name win64)
 else(CMAKE_CL_64)
    set(CPACK_NSIS_INSTALL_ROOT "$PROGRAMFILES32")
    set(win_system_name win32)
 endif(CMAKE_CL_64)

Medieval Fair 2013 - Making rings for chainmail

I have dressed up as Robin Hood and gone to the Medieval fair in Gumeracha sporting a bow. This year I decided to sport my camera with the classic 20mm pancake instead. The first thing I walked into was a sword holding competition. The challenge was to hold a 15 kilo sword with the arm extended horizontally for as long as possible. Plenty of fitness instructors and heavy lifters had a go at it, the record while I watched was around 3 minutes. I think I am going to practice with a cricket bat for next year.

Being a builder builds muscles

There were also jugglers and irish dancers, and to my dismay people with huge amounts of camera gear shadowing them. I felt really unprofessional with my dinky m43, then again this is the age of mobile photography and the battle between processing vs gear is on. It is now even being used as a marketing point by those Sony NEX ads. Granted a micro-four thirds sensor is smaller than the APS-C, still the weight advantages and flexibility bonuses do apply.

Juggler practicing

There were a few knights in chainmain and I walked into a conversation involving effort and time that goes into making the armor. Eventually the talk drifted to cheap Indian labour working for $1 a day so that people can have fun the fair, at that point people glanced at me and stopped talking. I should start tinkering with fencing wire to keep the stereotype going. Some things cannot be mechanised yet, clothing be it out of fibres or metals is one of them. The economics of hobbies is also interesting.

Riveted chainmail

The melee of kids with pugil sticks reminded me of a Schlock Mercenary episode. It will be good to get some excercise done while in costume.

Pugilists not boxing

There were also Middle eastern belly dancers and scribes doing calligraphy while complaining that the Belly dancers are not Medieval. Though the ululation accompanying the dancing is fairly universal.

Belly dancer and musician

Finally I managed to put the 850mm IR filter to good use and capture a panorama including the little lake at the Viking encampment. Without special IR modifications to the camera I had to use a pretty high ISO (1600) and a 2 second exposure. The processing through RawTherapee worked fine using some IR profiles I found floating around in Github, Hugin did a fair job of stitching as well.

850mm IR filter with unmodified camera panorama

NASA SpaceApps Adelaide 2013

A few weeks ago NASA hosted its second annual space apps challenge. I did not manage to participate on the first day and only managed to struggle in on the second day to witness the finale. The Adelaide Hackerspace provided the venue this year and Simon has very nice write-up on their blog.

 

There were a few more entries from around Australia including a slinky based greenhouse suitable for use on the moon and a meteor tracker integrated with Google sky. It was a couple of days of pizza and coffee, ending with beers by the time we finished.

 

Everybody went home pretty tired, but happy. I will definitely try to put in a couple of days of participation next year. Meanwhile feel free to use Mearth and look up all those cold place in Russia, Scandanavia and Hokkaido which have Mars like temperatures. Whoever is considering applying for Mars-One should visit one of these places and film on location.

Accurate positioning with UBlox Neo-6P and STM32

The Ublox Neo-6P provides raw clock phases and ephemeris suitable for use in post-processing based accurate positioning, or even RTK solutions using RTKLib.

STM32 with Eclipse was fairly messy to set up and depending on which version of GCC and CoreSupport libraries you use their are a few bugs to iron out. Used Code-Sourcery Lite toolchain, but compiled version did not behave as original firmware did. There are Eclipse based STM32 development commercial support from Atollic, and there were hacks to get the debugger Atollic packages working for ST-Link using the free options, but in the recent incarnation they have locked it down to TrueStudio only. I had to resort to OpenOCD for my debugging needs, it works like a charm with standard Eclipse settings, but I managed to nuke my STLink driver in the process and had to work hard to get it back.

Importing to GrafNav is fairly straight forward via the UBX converter, but it gives no indication on how it converts events, and you need sufficiently long logs to pick up any ephemeris. UBlox Receiver protocol is a must read for configuring the NEO-6P properly and monitoring the logs to ensure GPS lock. I am looking into the STM32 demonstration samples to add some more functionality to the project, including external interrupts from a camera for event marking and a GPS lock acquired status LED.

Adelaide Mini Maker Faire, Zoo and Indian Mela

I have been involved with the Adelaide Hackerspace on and off. Lately my attendance has dropped a lot due to work commitments and a casual addiction to photography. A few photography related hacks are stewing in my brain, so I might go back there. Anyway a few weeks back I went to the Adelaide Mini Makerfaire and it was great opportunity to see all the different things going on in Adelaide, from Dalek making, Metal working to 3D printing and Electric cars. I had a great time catching up with people and learning about new groups. The airstream guys had some hacked wireless AP's on display, including the Minitar on which I cut my teeth in cross-compiling and writing kernel modules to interact with the wifi-stack. There are a lot more photos.

I had planned a busy day, so I headed off to the zoo to attend the local photography group session and check out some pandas. It was my first time at the Adelaide zoo, so it took a while to get oriented. We covered a fair bit of ground in the hour and a half before closing. Managed a few shots of hippos, birds, otters and mountain goats. Lots of other animals which hardly stood still enough to compensate for my nascent photography skills.

We were trudging and hobbling back to the car when we noticed the bright lights and steady stream of people heading to Elder park. The local indian community was throwing their annual mela. So we headed there for a meal and to feed our cameras with some colour. The movement and low light made photography pretty difficult from the back of the crowd. It was a great day overall.