Managing Terrain in WorldWind Java - Trees in Denmark

Recently I talked to Thomas Knudsen from KMS, apart from discussing matlab vs octave and kriging vs other interpolation techniques. Finally we arrived at the topic he was presenting on, producing consistent DSM from Lidar. Lidar is notorious for the difficulty in tying multiple flightlines together, a problem KMS solved with their stock expertise in gridding (KMS pioneered a few variants of the kriging method). Interesting side note Picasa would not accept the picture as sent out of my iPhone - fix the bug in your Exif reader Google.

Thomas talked about the 2 Lidar companies they hired to collect the whole country since only 1 could not get the job done. I was reminded of the Scankort DSM data available via worldwind. Apparently Scankort went bust due to management failure and was gobbled up by Fugro. Somehow in the turmoil the data was released to NASA and is available for us to enjoy KMS's work.

For a few beers (not delivered yet) I hacked up a terrain manager to go with World Wind Java , so that you can turn the DSM on-off and compare with underlying SRTM. The results indeed show trees, while SRTM shows - well tree hills.

DSM

SRTM

 

When Licences hit the roof - Fuzzy Licencing policies

Being an Open Source guy software licencing always sounds iffy. There are though all these software licencing giants - they must be doing something right. The model hinges on selling using rights to the nuggety core code and surrounding it with a halo of hyped marketing and some very good service.

Oracle licences stuff per core with funny multipliers for diffrent CPU vendors, before the acquisition People Soft used to licence its product my a magic company valuation formula. The stock prices should have been set by how much a company's PeopleSoft licence costs.

Most vendors except the largest ones use a 3rd party licencing tool like FlexNet, whose older versions are very weak. The fixed number of licence idea be it attached to cores or to users creates an artificial ceiling which may halt production. In the financial industry the solution to limited cash is credits be it over drafts in cash accounts or margin lending for share trading. Since software licences are just as ephemeral and dynamic in value as currency, and most licence servers have extensive usage statistics measurements - software licencing should be fuzzy.

Fuzzy software licencing means degrading performance e.g. code efficiency and available runtime reduction when licences are overdrawn. If the financials can be sorted then an intermediate point to the Software as Service (SAS) model - bills are generated for extra usage.

Scalability in any process is always an issue, but with economics being like the weather maintaining operations at full scale is not feasible. May be the idea of pop-up shops will catch on.

Staying in production - heartbeats and jump starts

At FOSS4G I saw a very nice presentation by Andrea on maintaining Geoserver in production and optimizing performance. It is all nice and good with the jvm flags to squeeze more juice out of the hardware, maintain more objects in RAM, run parallel garbage collect and the lot. For the data server to keep running it needs :

• CPU(s) - Multi-threaded request fulfillment
• RAM - To do map composition, reprojection, format conversion
• Database - To draw spatial data out of, vector data greatly benefits from being in the database and being indexed
• HardDrive/File Share Service - Raster data is best drawn out of stripe-raid drives (0,5,0+1 depending on your budget).

With all these are loadbalancers and all important heartbeat and failover mechanisms. Heartbeat on each component is not sufficient. You need heartbeat on every bit of your infrastructure, and only a beat is not enough you need to hook up a full ECG machine and make sure consistent data is produced. All deployed layers need to be coming out at an acceptable  speed and not corrupted.

PS: Warning before touching anything (thanks arneke).

The importance of the generalist

In the fast moving technology industry things evolve fast and fast evolution entails fast extinction. Unfortunately humans have some 50+ year life span and can't afford to go extinct like databases and operating systems.

If you focus and develope extertise in one particular technology chances are its lifecycle will end in the near future. In the last few years of my professional life I have worked on and used myriads of open-source projects in the geospatial and scientific arena and seen plenty of them go extinct and the developers join the new kid on the block. There was a presentation on extinction, evolution, survival and dominance with respect to openlayers at the FOSS4G. I have seen similar evolution in World Wind .NET and Java. Once the java version became embeddable in other applications the viral mutations increased.

Jack of all trades, master of none, though often better than a master of one - Adam Savage

Knowing C# and Java is not a great deal but it helps transition. You have to know Oracle, MySQL, MSSQL and PostgreSQL since you never know what the next client or 3rd party software team you are working with will use. Your stuff has to work on Windows (2000, XP, Vista and 7) , Linux, Mac and possibly Open Solaris as well. The graphics engine on windows will need DirectX and OpenGL duality since some graphics cards are partial to one or the other. The server code will have to span PHP, Python, Java, Perl and some C++ based CGI/FastCGI. With all these you will have to keep learning all the countless new things Microsoft keeps churning out and the projects that keep on fighting for a foothold in the open-source ecosystem.

The best skill you can get yourself overall is learning how to learn, stop being piegeon-holed and look at the broad swath. Only coding will not get you there, programmers are not divas they don't succeed on sole capacity and reap huge salaries. They work in teams. Billy G in addition to being a decent coder is a ruthless and astute businessman to get the company where it is today. Oracle has to have an evil genius at the helm. Ignoring human relations and business mentality is not an option.

Just get the job done as best as you can, do whatever it takes - be a generalist.

Building the Orfeo Stack

Orfeo Toolbox (OTB) is a great symphony written in C++, or is it Opera ? I believe C++ in spite of its ceveats is one of the most portable langauages, a great choice for something demanding maximum output from the hardware such as an image processing application.

With inspiration from music with names such as orfeo and monteverdi, OTB puts together a fair few heavy weight projects. Not only in using them as libraries but being more than the sum of its parts. The developers also merge and maintain patched copies of these dependent projects in a mercurial repository ensuring a supported version is compiled against (a lot of the dependencies are merged from repository heads so much more recent than releases).

The dependencies are gdal, ossim, ITK, FLTK , OpenThreads, expat, liblas, etc. The cmake build system with the monolithic repository containing most of the other libraries makes getting up and running from a checkout relatively painless. Now I need to build the python and java bindings and try out some Qgis plugins and on demand image processing from GeoServer.

Cyber cafes and Upload Cafes

I blew a massive hole in my budget by using my Vodafone datacard in India to download 70MB at 1c a Kb. That's a huge load of waste for my net addiction. This could have been saved by using an internet cafe. Cyber cafes are ubiquitous in India, where household penetration of internet is low. Post Mumbai shootings, these cafes have become overcautious and if you show up even with a hint of a beard they will not let you use the machines without a full ID check (ID checks are everywhere actually - nobody believes you are who you say you are).

Moving back into another country - Australia. Internet penetration to homes and mobiles and everywhere in general is very high. The government does a lot of its business there (though I have heard of agencies which have 1 automated reply email saying they don't accept emails and provide you their physical address). Here the cyber cafe is very niche, possibly in the fragging niche in China town where people folk to play WoW and Counter Strike. The draw here is the low low lag times provided. The bandwidth is not a huge issue with gaming protocols these days since the maps are cached.

I was thinking of the business case for an upload cafe - a spot which provides high bandwidth upstream access similar to a data center - but on a casual basis. Large volume business data sharing is becoming the norm. There have been demonstrations in South Africa about the pigeon protocol being faster than the local ISP's upstream bandwidth. The download speeds in cable and adsl2+ are impressive. Yet the upstream needs a lot more work. You could choose symmetric ADSL if large uploads are your thing, but you are stuck with copper upstream on cable. I will use such a cafe to upload my hour long cooking videos to youtube or 100's of holiday photos to facebook.

Nearmap - Dense observation of space-time

Nearmap has finally had its public data release. Nice slippy maps view and you would expect in any web 2.0 mapping site. The biggest difference is you can make the time slip in addition to space - every capture date has a separate set of vertical and oblique views as well as separate stereo elevation model.

As with any large automated system e.g. Google the onus is on the users to find anomalies and QA the huge volume of data.

 

There are indeed some anomalies in the non-vertical modes over Adelaide. I hope they will take user feedback and improve the processing flow fulfill the great promise of recent aerial, elevation and city models this system holds.

 

Floating around in Singapore

Due to the night curfew in Adelaide, flights from Singapore to Adelaide can only leave for a 12 hour period. So if you are unfortunate enough to arrive at the wrong end of the half-day stick you will have to stooge about in the Singapore airport (not that it isn't huge and a shopping paradise) or the fine city of Singapore itself.

I booked a random hotel near little india from the airport, which turned out to be a stones throw from electronics Mecca of Singapore - Sim Lim. Most of my day got sucked in there, staring at gyros and brushless motor, some super compact hard disks and the newly affordable handheld FLIR sensors post H1N1 scare.

I managed to catch up with one of the top Orfeo Developers via Linked In. We walked around a bit dodging police cordons which had blocked off a fair few roads because of APEC. Managed find a decent bar with a sofa-bed next to the pier and chatted about IEEE standards for floating point over a couple of beers. I am indeed a splendid conversationalist.

Writing JUnit Tests in Geoserver

In order to get a community module promoted to extension status I had to write a couple of unit tests with JUnit. Geoserver strives to be production quality software so unit tests are quite handy, for agile programming as well as regression testing. Making tests pass through taking short-cuts is quite easy but usually very unwise. I did have to take the copy-paste short cut route though, since the generic DefaultMapProducerTest was hardcoded with the "image/gif" output format and I was not in the mood to make core changes (outside my tiny module).

Well now a couple of tests are written and the module code passes them and works as expected in general, time to petition for inclusion as an extension.

World Wind wins the best software of the year award

Another triumph for open source - NASA awarded World Wind the software of the year. More details are in Bull's blog. I obviously missed the boat on this one through family issues and trips to the Ganges to purify the body and soul - don't go by the grey mushy appearance of the river, if you believe it will purify you.

(8:52:08 PM) arneke: life is full of black boxes comes with a really crappy debugger ;)

At least World Wind is not a black box and the Java debugger is pretty good. You can quickly track down the issue and fix it, that is IF you are not using JNI and calling Native code. This can be the case with calls to the Native JOGL and various associated graphics card issues. Watch out for those - since death will soon follow.

Error message annoyance

One of the reasons users get frustrated with software is the inane error messages we programmers inject into it without consulting a sensible psychologist. This is the reason a lot of people have no clue what went wrong when something does go wrong - a stacktrace may be useful to us, but is garbage to the user. This is pointed out aptly in this comment.

Recently I was watching the demo video of an Enterprise document management platform - Documentum. The product video shows odd error messages and things you cannot do rather than things being simple, sort of product I would flinch from spending millions on.