CMAR Python Be-In - converting Matlab/R/VB folks

Last couple of days we have been running a coffee and cake fueled Be-In/Hackfest to learn Python at CSIRO CMAR. In the oceanography community a lot of coding goes on, the large models are written in Fortran and C. The throw-away prototyping is done in Matlab by physical oceanographers and in R by biologists.

R is open source and contribution/hacking friendly, but Matlab is another story altogether. Matlab however has a rich set of documentation and toolboxes which you will probably never read or use, just like you will never redline the tacho in your car - but one gets the comfortable feeling of knowing it is there in case something goes wrong. I was aiming for positive reviews like these.

In my true UI's are the best fashion I did a short IDE round up which you can find here. I learnt a lot about requirements of true large data processing, caveats in python and some backend magic for matplotlib. Also got a chance to look at the scripting API for Java and the pains with the SPI service registries and the death of com.sun.

Since we are on the Python topic I mucked around with running Matlab style functions from Python via Scilab (Java Matlab look alike). Data gets passed around Python Interpreter to Java VM for the sake of syntactic familiarity. After a couple of build hick-ups:

• locating includes due to scilab layout on Ubuntu
• launching with jvm and libjava on LD_LIBRARY_PATH
• proper setting of SCI env variable to /usr/share/scilab 

I had a matlab like API in Python. So I tried comparing computation of matrix inverses:





In [81]: from numpy.linalg import pinv,inv
In [82]: from scilab import scilab as sci
In [83]: x = sci.rand(100,100)
In [84]: from pylab import imshow,show
In [85]: a = inv(x)
In [86]: b = pinv(x)
In [87]: c = sci.inv(x)
In [88]: figure(1)
In [89]: imshow(a-b)
In [90]: figure(2)
In [91]: imshow(a-c)
In [92]: figure(3)
In [93]: imshow(b-c)


The inverse calculation is a fairly complex series of floating point operations and errors accumulate between the different techniques in the order of 4x10^-15 for the random array. No wonder results from matlab don't match up with those from Scilab and Numpy. Next I will grab a copy of Matlab and use mlabwrap to compare with the "real matlab" computed inverses.

PS: mlabwrap results are in, how do I tell which inverse is more "correct" suggestions welcome.
In [42]: mean(abs(sci.inv(x)-mlab.inv(x)))
Out[42]: 1.0797192783948417e-14
In [43]: mean(abs(sci.inv(x)-inv(x)))
Out[43]: 1.1512191793925341e-15
In [44]: mean(abs(inv(x)-mlab.inv(x)))
Out[44]: 1.0661985412661976e-14

Dragging Geotools kicking and screaming to OSGi land

Me and others have been making some noise on the Geotools list to convert the current modules in Geotools from being only modules in name to proper OSGi bundles with services being published for use by other bundles and packages being imported and exported cleanly.

There seem to have been several attempts to osgify the Geotools mud-ball with only the bits the users needed to get things done being taken care of and hard bit, particularly those using the Sun ServiceLoader interface being left out.

1. Classic example from Harald Wellman currently hosted on Berlios
2. Argeo attempt - Mathieu is putting the OSGi modification back into Geotools core
3. Humboldt project - Same idea using Bnd.

There are 2 solutions to the META-INF/services dilemma (introduced by the classloader separation inherent in OSGi)

1. A bundle lifecycle interceptor that picks up all the SPI based services and converts them to OSGi service.
2. A compile time mangler that does the same conversion and adds OSGI-INF/*.xml for the services.

The reference factories being registered look pretty nice in ASCII art.

The other issue plaguing the migration of geotools is the presence of split package, another ripple from the OSGi classloader per bundle approach. There are various hacks to solve this, but none of them is quite neat. As a stop-gap package can be merged using the split directive in the manifest or via Bnd.

Hope the transformation will be over soon and we will have truly modular Geotools.

Counting code - Ohcount vs Cloc

We have lots of evil csh which we want to convert to bash or even perl or python. So I decided to test some code line counting tools to estimate the effort involved in terms of pure lines of code. I tested 2 options: the old and venerable cloc and the child of web 2.0 ohcount. Like the definition of the metre, what constitutes a line of code can vary a bit.

Ohloh Line Count Summary

Examining 16633 file(s)
Language	Files	Code	Comment	Comment %	Blank	Total
java	2737	227551	102476	31.1%	61095	391122
xml	1814	87627	5470	5.9%	3134	96231
xmlschema	418	69194	8743	11.2%	753	78690
javascript	423	41392	9386	18.5%	5096	55874
html	429	19675	580	2.9%	2025	22280
css	86	8697	619	6.6%	1951	11267
sql	10	7289	246	3.3%	168	7703
xslt	47	3092	636	17.1%	569	4297
php	2	600	110	15.5%	102	812
bat	11	575	22	3.7%	166	763
scala	5	552	35	6.0%	99	686
python	29	544	786	59.1%	419	1749
shell	18	502	95	15.9%	99	696
make	6	356	24	6.3%	78	458
Total	6036	467707	129228	21.6%	75759	672694

Cloc gives similar numbers off by a few thousand and separated by csh vs bash. Ohcount could do with some more shell script discrimination.

Bringing back an old favourite - the Butteruino

After I had done some damage with AVR's bought from Dick Smith in 2002 on a breadboard, I moved onto the butterfly. I did a mini project measuring waterflow using thermistors for GroGuard. Then they lay disused in boxes until recently. I ordered a carrier board from Ecross and put together a handy proto-typing platform, except the serial port on the butterfly I have seems to be obstinately broken.



This did not stop me from getting some software together for loading code onto it. Since everybody has caught the Arduino fever. I decided to go with Wiring and the Arduino IDE to develop for the butterfly, fortunately the Butteruino project has already done a lot of the hardwork for me, I just upgraded the changes to v22 of the Arduino IDE release. The patch should go live on SVN soon. Till then if you have a butterfly and want to test it with the Arduino IDE get it here.

Now the assembled carrier will lie in boxes till I work up the courage to order some more old butterfly's from Digikey.
In spite of all their peripherals and fancy demo code, the butterflies suffer from the weirdness of the joystick, lack of FTDI for direct USB connection and overall fragility. Still they are excellent for long-term, low-power data logging applications, may be coupled with an Xbee. I believe given slightly different design choices, a better IDE than AvrStudio, no funny press the reset button and pray code uploads, the butterfly could have beaten the Arduino back then. I will look into making the flow sensor again once I get my hands on a working butterfly and develop some more Eagle skills.

Making wheel under the cameras - Beagle has 4 wheels

All the work with stereo vision and kinect hackery is of not much use without obstacles to sense and avoid. Hence the beagleboard is now mounted on 4 DC motors driven by the DFRobot Arduino Romeo board. The robot base is quite sturdy aluminum and has plenty of pre-drilled holes, but to mount the beagle I had to acquire my first rotary tool set.



ASUS is coming out with the PC version (Xtion PRO) of Kinect soon, which will be smaller than the original and will not have the RGB camera, it will also be purely USB powered. Perfect opportunity to use it along with the stereo vision cameras to increase the depth of field and build a platform which can use vision indoor and outdoors and in complete darkness, by combining the sensors.

Next comes getting data from the camera out wirelessly with sufficient bandwidth. I have the Beagle working with 3G modems, but this was going to be a first with a Ralink 2870 (which I got el-cheapo at DealExtreme) . With Koen's advice I decided to use the rt3070sta, the modinfo makes is obvious that this is going to work with the card:


root@beagleboard:~# modinfo rt3070sta
filename:       /lib/modules/2.6.32/kernel/drivers/net/wireless/rt3070sta.ko
version:        2.3.0.4
license:        GPL
description:    RT2870 Wireless Lan Linux Driver
author:         Paul Lin


It does not like pumping dhcp at boot time so I have to go the long way and pull it up with ifup later on. The usual gstreamer tricks work fine for sending the camera streams around:


In the beagle - gst-launch v4l2src device=/dev/video1 ! queue ! videorate ! video/x-raw-yuv,framerate=15/1 ! videoscale ! video/x-raw-yuv,width=320,height=240 !  ffmpegcolorspace ! jpegenc ! udpsink host=x.x.x.x port=xxxx

In the laptop - usr/bin/gst-launch udpsrc port=xxxx ! jpegdec ! autovideosink

Next comes pan-tilt and actual object avoidance and object following. Pass the dog a ball.

Expanding the Arm-AVR menagerie (Training Pandas)

I have recently acquired some more hardware - a Tincan tools Trainer for xM and a Pandaboard. Sometimes I feel embedded should start with an 'A' - for ARM and Arduino.

Word of caution in the new kernels the serial ports on the xM appear at /dev/ttyO1. Look in /proc/tty/drivers - OMAP-SERIAL  /dev/ttyO     253 0-3 serial.

You will need this little tid-bit to program the trainer. The programmer seemed to be atleast respond to avrdude when fresh out of packaging:

avrdude: stk500_getsync(): not in sync: resp=0x20avrdude -V -F -c stk500v1 -p m328p -P /dev/ttyO1 -b 57600 -U flash:w:blink.hex

avrdude: stk500_getsync(): not in sync: resp=0x20

It was not working anyway, but can I make it fail with 0x20 predictably ? So I got fancy and forwarded the serial port over the network. The beagle acts as a serial port server and the host machine acts as the client. This is plain tcp, but you can build an ssh-tunnel or whatever. The serial port is redirected with interceptty. The same failure was observed.



So I forked out some cash and got myself an ISP cable. This let me load the arduino bootloader using avrdude and the instructions here. After the first bootload, avrdude and avrgal can both push blink.hex to the arduino bootloader, but fail to perform further writes due to the lack of the software reset (stty -F /dev/ttyO1 hupcl). So I resorted to how the arduino visual IDE does things and eventually I used the Arduino.mk in /usr/share/arduino and built up a blink.hex which was significantly larger than the pure Makefile based one. This one supports loading and reloading via avrgal. The first attempt made an arduino bootsector virus as av500 put it.

Now that all is hunky-dory in the tincan + beagle-xM world, it is time to attach a few peripherals to the arduino and watch it squirm.

Merry with Maven - Maintaining GEOS-3586

I have been typing GEOS-3586 a lot of times this past week. The DDS/BIL plugin written for Geoserver 2.0.x had fallen by the wayside as things had changed on the WMS MapProducer(Response) front. This time it is going to be a serious push to get the plugin promoted from community module to extension status. I still need to get my head around GSIP-27 and GSIP-22 to efficiently continue maintaining this module.

Finally I was recommended to just keep trunk updated and work towards getting the module moved out of community into extensions space. The DDS/BIL module has indeed a special place since it is the only community module extending map respones, all other map reponses are part of core. The extensions in response are mostly in the WFS area (OGR, Excel etc.).

WorldWind in typical fashion throws some curve-balls at Geoserver: it uses BGCOLOR=-9999.0 for the elevation models instead of a hexcode. Fixing this will require dangerous interventions on the Geoserver side as outlined here by Andrea:

The kvp parsers are at work way before the output formats
get even into the picture.
The class that parses bgcolor is ColorKvpParser.

I think you can write a DispatcherCallback that hijacks the parsing and error
reporting mechanism, but you have to be very careful to do so to avoid
breaking the proper error reporting mechanism in other cases.

In the init(Response) method of the DispatcherCallback you check the error,
reset it to null if it's the one coming from the bgcolor parse failing,
it is that special value, and the output format is one of the two that worldwind
is using (use the raw kvp map to check the format and bgcolor values).

Then you register that dispatcher callback as a spring bean in
applicationContext.xml
and GeoServer should automatically pick it and use it

Please be very specific in the checks above or you'll leave users making
wrong request in a world of pain (no error reporting and software blowing up
in the most unexpected ways, or just ignoring what they set).

I am also planning to include WorldWind (in an applet form) as a 3D layer preview tool  in Geoserver alongside Openlayers. Chris Holmes floated this idea a while back. Since I will be doing lots of work on WorldWind anyway, I can practice browser integration on the side in Geoserver.

The plugin has some primitive documentation now as well - hacked together with my basic look-and-learn Sphinx-fu.

Talking about putting things together - I finally migrated the content from my old blog into this one. Now just need a way to redirect all the posts to duplicates here and just keep the blog being deleted one.