Compiling QGIS in MSVC in 2020

Compiling QGIS on Windows in 200x

I don't quite remember when I decided to help compile QGIS on Windows. It was somewhere between compiling GDAL with ECW support for Photoshop on Windows and getting carried into Direct3D and C# land with NASA WorldWind. It was sometime in the 2000's while still working at Apogee Imaging in Lobethal.

At that point I was manually building a database of the footprints of satellite imagery that filled up a wall cabinet with CD's and DVD's. The technique was something like open up the image, go around edges and trace a polygon. This was days before mature boolean thresholding and reliable/easy raster-to-vector logic.

I hopped on IRC on #qgis in Freenode and chatted with luminaries like timlinux, frankw and gsherman. Listened to the automated notifications from sigq, the commits bot. Things were heating up and instead of a Linux cross-compile to windows using MingW, something native to windows say using MSyS+MingW instead of Cygwin was desired. A lot of GDAL and Qt worked in MingW, so presumably QGIS would too. So I set myself to put together an MSYS environment with all the third-party dependencies that could be used to happily build QGIS. Eventually I built a release in NSIS as well.

My MSYS environment got packed in a zip and shared via FTP/HTTP on a VPS I had back then to the rest of the community. I earned myself a pin in the QGIS core contributor map in Adelaide. Something I am very proud of to this day. Eventually the MingW build got deprecated and native MSVC builds were supported. That's how contributions work, nothing lasts forever. In my IRC days, I helped on-board Nathan Woodrow to QGIS, who in turn I believe helped on-board Nyall Dawson. Nyall has surpassed us all in feature contributions and work on QGIS.

Getting started on QGIS on MSVC blog article. Trying to add some historical perspective on my involvement. Found this acknowledgement in the docs. Thanks to @timlinux . Back in the days of sharing mingw zip files via FTP. pic.twitter.com/NTkQgU5ibN

— Tisham Dhar (@whatnick) August 29, 2020

Fast forward to 2020, compiling QGIS in MSVC

I am getting back into doing lots of Open-source work after long hiatus in private industry with Aerometrex and start-up land with Lorisystems. It is great fun working on mostly in the open at Geoscience Australia. There is actually a recently archived opendatacube + qgis repository here. Seeing that repo and speaking to Nathan and LinuxConfAu inspired me to have a go and getting back into actively working on the Qgis code base. It has sprawled out, with lots and lots of new features. The build system is still familiar via CMake and actually much easier now with MSVC. I cast around for a recent guide and found this. The guide mostly works, however I made some refinements.

• Ditched bison and flex via Cygwin to using the one available via Msys2. These can be found here. Not needing the while Cygwin system helps in keeping the windows build system light. Simply download the binaries and add them to the Osgeo4W binaries directory.
• Captured my CMakeCache.txt to make it easier to reproduce and debug the build environment for others.
• Used Incredibuild in demo mode to use a few NUC's I have lying around to speed up the build. Recording while building failed the first time and worked the next. The whole build from scratch still tooks around 35minutes overall.

I am planning to throw some of my day to day DevOps skills towards the QGIS project and start helping again with Raster enhancements and windows release management. Perhaps getting Incredibuild in the hands of the windows maintainers will help tighten up the iteration cycle and make testing easier.

The twitter thread/ stream of consciousness edition of this is available as well.

Moving from babun to msys2 to get flex and bison required for compiling QGIS from source. https://t.co/W866AOM9Kt pic.twitter.com/fUmqpa9YHY

— Tisham Dhar (@whatnick) June 26, 2020

JPEG2000 Encoding with CUDA

For the last few days we have been struggling to efficiently write out very large JPEG-2000 files. In all their wisdom Erdas decided to supply only a 32-bit version of the ECW compressor. It seems to have trouble allocating memory for large JPEG-2000 files, is single threaded and dead-slow. Somewhere in the Ermapper install directory there is also the Kakadu core dll. So I decided to have a look at other methods of creating JPEG-2000, possibly faster and more scalable options. Kakadu seems to only supply an SDK and we can test the speed of compression with the SDK. GDAL also supports building against the Kakadu SDK and would be handy for creating the large mosaics we have been struggling with. It seems with modern hardware that is not the fastest option.

After some trial with Kakadu and a bit of reading I came across a few CUDA based implementations - CUJ2K and JPEG2K.  I decided to port the most recently maintained CUDA based JPEG-2000 implementation, claiming to be the fastest on commodity hardware to windows. The CMake based build system makes things easier. The code however was written for a POSIX based systems so I was missing a bunch of headers. I had to hack away at it and move variables around to cope with lack of C99 compliance in the MSVC C-Compiler or I am forced to used the /TP flag to upgrade the C-files to C++ files and allow variable declaration anywhere. Lots of moving variables around later, everything worked fine.

The compression library is built against FreeImage which functions as an I/O driver for the encoder/decoder. However only writing to the J2K profile is supported, a wrapper for the JP2 mode will be required to add-in all the projection metadata GIS Rasters typically require. To make this library more useful for GIS purposes we will need to add an I/O handler for GDAL instead of FreeImage to allow faster compression/decompression.

I have tested the JP2ECW, JP2OpenJPEG and JP2Kakadu drivers on a small file (26MB in .bmp format, since the Kakadu demo binary does not support LZW .tif I have lying around). The times averaged over a few runs on an i7-3930K @ 3.2GHz are shown below:

• LibECW - 0.8s
• Kakadu (1 Thread) - 1.5s
• Kakadu (8 Threads) - 0.5s
• OpenJPEG - 2s

Compared to these the CUDA library on the GTX580 on the same machine runs at 1.2s (with compression taking 0.75s, rest is I/O).  The logical next step is hooking up the compressor to GDAL for data-block I/O and building a JP2 wrapper in addition to the J2K profile which it currently uses. The Affero GPL licence also needs to be sorted out.

For those interested in the windows build, here is the SVN Diff against rev 84 and the pre-built release binaries against CUDA 5.5.