Why choose a language when you can be a Polyglot

Flamewars between language fanatics often gets in the way of "getting things done". I pick up languages fairly quickly and appreciate their benefits and short comings. After a fair bit of survey it seems like no language will instantly have familiarity and features concurrently, not unless it is also attached to a time machine and a brain washer. So why not call a truce and support them all (or most) - on a half-decent industry standard VM (no matter how much you dislike oracle).

Dynamic languages are a lot of fun to develop with, you can create prototypes quickly and have a tight feedback loop of your application taking shape. Sometimes the heuristics built into an application are not the right ones (u,v in vector interpretation are swapped, some format loader does not quite do what you expect). The expert/inquisitive user should be able to fix these vagaries at runtime and make the application behave. Any self-respecting application should include such dynamic scriptablity (Qgis does via CPython/SIP, Geoscript is nascent).

The REPL based languages that we are aiming to support via JSR 223 include:

1. Jython - with standard JSR 223.
2. JavaScript - with Mozilla Rhino, browser JNI bindings can be lobbed in as needed.
3. JRuby - again we get JSR 223 for free.
4. Groovy - the standard interface is not enough for delving in the depths of Groovy, but it does the job.
5. R - I hacked together a placeholder based on the JNI based rJava project, there are standard implementations but they don't play along nicely with our integration framework yet.
6. Perl - Another day another hack for the Inline::Java JNI based calls to the Perl interpreter.
7. Matlab - I had a chance to delve deep into the innards of Matlab and discovered a native and jvm based beast. There are several hacks to talk to Matlab from Java including running your Java code on the same VM Matlab is running (it even includes the felix runtime and scr). For now I am using matlabcontrol with some JSR-223 linkage.

The current integration tests with patchwork hooks to convert SUN Spi to OSGi Services looks something like below.


INFO: Testing Script Engine: jython
Jun 19, 2011 12:53:36 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: javascript
Jun 19, 2011 12:53:36 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: jruby
Jun 19, 2011 12:53:37 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: groovy
Jun 19, 2011 12:53:37 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: R
Jun 19, 2011 12:53:38 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: perl
Jun 19, 2011 12:53:38 PM org.trikend.script.test.ScriptTest testConsole
INFO: Testing Script Engine: matlab

A JSR-223 bridge may not do it justice and we may have to write our entire high-reliability portion of the system in Erjang, but that will not stop me from adding an Eshell to the integration test next. The java scripting project feels dead for a year, possibly due to Oracle turmoil, if there is any news on this front and Github forks alive and kicking I would really like to know. Other syntaxes and languages with fun features to evaluate would be: Scala + Akka, Clojure and Gosu.

CMAR Python Be-In - converting Matlab/R/VB folks

Last couple of days we have been running a coffee and cake fueled Be-In/Hackfest to learn Python at CSIRO CMAR. In the oceanography community a lot of coding goes on, the large models are written in Fortran and C. The throw-away prototyping is done in Matlab by physical oceanographers and in R by biologists.

R is open source and contribution/hacking friendly, but Matlab is another story altogether. Matlab however has a rich set of documentation and toolboxes which you will probably never read or use, just like you will never redline the tacho in your car - but one gets the comfortable feeling of knowing it is there in case something goes wrong. I was aiming for positive reviews like these.

In my true UI's are the best fashion I did a short IDE round up which you can find here. I learnt a lot about requirements of true large data processing, caveats in python and some backend magic for matplotlib. Also got a chance to look at the scripting API for Java and the pains with the SPI service registries and the death of com.sun.

Since we are on the Python topic I mucked around with running Matlab style functions from Python via Scilab (Java Matlab look alike). Data gets passed around Python Interpreter to Java VM for the sake of syntactic familiarity. After a couple of build hick-ups:

• locating includes due to scilab layout on Ubuntu
• launching with jvm and libjava on LD_LIBRARY_PATH
• proper setting of SCI env variable to /usr/share/scilab 

I had a matlab like API in Python. So I tried comparing computation of matrix inverses:





In [81]: from numpy.linalg import pinv,inv
In [82]: from scilab import scilab as sci
In [83]: x = sci.rand(100,100)
In [84]: from pylab import imshow,show
In [85]: a = inv(x)
In [86]: b = pinv(x)
In [87]: c = sci.inv(x)
In [88]: figure(1)
In [89]: imshow(a-b)
In [90]: figure(2)
In [91]: imshow(a-c)
In [92]: figure(3)
In [93]: imshow(b-c)


The inverse calculation is a fairly complex series of floating point operations and errors accumulate between the different techniques in the order of 4x10^-15 for the random array. No wonder results from matlab don't match up with those from Scilab and Numpy. Next I will grab a copy of Matlab and use mlabwrap to compare with the "real matlab" computed inverses.

PS: mlabwrap results are in, how do I tell which inverse is more "correct" suggestions welcome.
In [42]: mean(abs(sci.inv(x)-mlab.inv(x)))
Out[42]: 1.0797192783948417e-14
In [43]: mean(abs(sci.inv(x)-inv(x)))
Out[43]: 1.1512191793925341e-15
In [44]: mean(abs(inv(x)-mlab.inv(x)))
Out[44]: 1.0661985412661976e-14

Dragging Geotools kicking and screaming to OSGi land

Me and others have been making some noise on the Geotools list to convert the current modules in Geotools from being only modules in name to proper OSGi bundles with services being published for use by other bundles and packages being imported and exported cleanly.

There seem to have been several attempts to osgify the Geotools mud-ball with only the bits the users needed to get things done being taken care of and hard bit, particularly those using the Sun ServiceLoader interface being left out.

1. Classic example from Harald Wellman currently hosted on Berlios
2. Argeo attempt - Mathieu is putting the OSGi modification back into Geotools core
3. Humboldt project - Same idea using Bnd.

There are 2 solutions to the META-INF/services dilemma (introduced by the classloader separation inherent in OSGi)

1. A bundle lifecycle interceptor that picks up all the SPI based services and converts them to OSGi service.
2. A compile time mangler that does the same conversion and adds OSGI-INF/*.xml for the services.

The reference factories being registered look pretty nice in ASCII art.

The other issue plaguing the migration of geotools is the presence of split package, another ripple from the OSGi classloader per bundle approach. There are various hacks to solve this, but none of them is quite neat. As a stop-gap package can be merged using the split directive in the manifest or via Bnd.

Hope the transformation will be over soon and we will have truly modular Geotools.

Counting code - Ohcount vs Cloc

We have lots of evil csh which we want to convert to bash or even perl or python. So I decided to test some code line counting tools to estimate the effort involved in terms of pure lines of code. I tested 2 options: the old and venerable cloc and the child of web 2.0 ohcount. Like the definition of the metre, what constitutes a line of code can vary a bit.

Ohloh Line Count Summary

Examining 16633 file(s)
Language	Files	Code	Comment	Comment %	Blank	Total
java	2737	227551	102476	31.1%	61095	391122
xml	1814	87627	5470	5.9%	3134	96231
xmlschema	418	69194	8743	11.2%	753	78690
javascript	423	41392	9386	18.5%	5096	55874
html	429	19675	580	2.9%	2025	22280
css	86	8697	619	6.6%	1951	11267
sql	10	7289	246	3.3%	168	7703
xslt	47	3092	636	17.1%	569	4297
php	2	600	110	15.5%	102	812
bat	11	575	22	3.7%	166	763
scala	5	552	35	6.0%	99	686
python	29	544	786	59.1%	419	1749
shell	18	502	95	15.9%	99	696
make	6	356	24	6.3%	78	458
Total	6036	467707	129228	21.6%	75759	672694

Cloc gives similar numbers off by a few thousand and separated by csh vs bash. Ohcount could do with some more shell script discrimination.

Bringing back an old favourite - the Butteruino

After I had done some damage with AVR's bought from Dick Smith in 2002 on a breadboard, I moved onto the butterfly. I did a mini project measuring waterflow using thermistors for GroGuard. Then they lay disused in boxes until recently. I ordered a carrier board from Ecross and put together a handy proto-typing platform, except the serial port on the butterfly I have seems to be obstinately broken.



This did not stop me from getting some software together for loading code onto it. Since everybody has caught the Arduino fever. I decided to go with Wiring and the Arduino IDE to develop for the butterfly, fortunately the Butteruino project has already done a lot of the hardwork for me, I just upgraded the changes to v22 of the Arduino IDE release. The patch should go live on SVN soon. Till then if you have a butterfly and want to test it with the Arduino IDE get it here.

Now the assembled carrier will lie in boxes till I work up the courage to order some more old butterfly's from Digikey.
In spite of all their peripherals and fancy demo code, the butterflies suffer from the weirdness of the joystick, lack of FTDI for direct USB connection and overall fragility. Still they are excellent for long-term, low-power data logging applications, may be coupled with an Xbee. I believe given slightly different design choices, a better IDE than AvrStudio, no funny press the reset button and pray code uploads, the butterfly could have beaten the Arduino back then. I will look into making the flow sensor again once I get my hands on a working butterfly and develop some more Eagle skills.

Making wheel under the cameras - Beagle has 4 wheels

All the work with stereo vision and kinect hackery is of not much use without obstacles to sense and avoid. Hence the beagleboard is now mounted on 4 DC motors driven by the DFRobot Arduino Romeo board. The robot base is quite sturdy aluminum and has plenty of pre-drilled holes, but to mount the beagle I had to acquire my first rotary tool set.



ASUS is coming out with the PC version (Xtion PRO) of Kinect soon, which will be smaller than the original and will not have the RGB camera, it will also be purely USB powered. Perfect opportunity to use it along with the stereo vision cameras to increase the depth of field and build a platform which can use vision indoor and outdoors and in complete darkness, by combining the sensors.

Next comes getting data from the camera out wirelessly with sufficient bandwidth. I have the Beagle working with 3G modems, but this was going to be a first with a Ralink 2870 (which I got el-cheapo at DealExtreme) . With Koen's advice I decided to use the rt3070sta, the modinfo makes is obvious that this is going to work with the card:


root@beagleboard:~# modinfo rt3070sta
filename:       /lib/modules/2.6.32/kernel/drivers/net/wireless/rt3070sta.ko
version:        2.3.0.4
license:        GPL
description:    RT2870 Wireless Lan Linux Driver
author:         Paul Lin


It does not like pumping dhcp at boot time so I have to go the long way and pull it up with ifup later on. The usual gstreamer tricks work fine for sending the camera streams around:


In the beagle - gst-launch v4l2src device=/dev/video1 ! queue ! videorate ! video/x-raw-yuv,framerate=15/1 ! videoscale ! video/x-raw-yuv,width=320,height=240 !  ffmpegcolorspace ! jpegenc ! udpsink host=x.x.x.x port=xxxx

In the laptop - usr/bin/gst-launch udpsrc port=xxxx ! jpegdec ! autovideosink

Next comes pan-tilt and actual object avoidance and object following. Pass the dog a ball.

Expanding the Arm-AVR menagerie (Training Pandas)

I have recently acquired some more hardware - a Tincan tools Trainer for xM and a Pandaboard. Sometimes I feel embedded should start with an 'A' - for ARM and Arduino.

Word of caution in the new kernels the serial ports on the xM appear at /dev/ttyO1. Look in /proc/tty/drivers - OMAP-SERIAL  /dev/ttyO     253 0-3 serial.

You will need this little tid-bit to program the trainer. The programmer seemed to be atleast respond to avrdude when fresh out of packaging:

avrdude: stk500_getsync(): not in sync: resp=0x20avrdude -V -F -c stk500v1 -p m328p -P /dev/ttyO1 -b 57600 -U flash:w:blink.hex

avrdude: stk500_getsync(): not in sync: resp=0x20

It was not working anyway, but can I make it fail with 0x20 predictably ? So I got fancy and forwarded the serial port over the network. The beagle acts as a serial port server and the host machine acts as the client. This is plain tcp, but you can build an ssh-tunnel or whatever. The serial port is redirected with interceptty. The same failure was observed.



So I forked out some cash and got myself an ISP cable. This let me load the arduino bootloader using avrdude and the instructions here. After the first bootload, avrdude and avrgal can both push blink.hex to the arduino bootloader, but fail to perform further writes due to the lack of the software reset (stty -F /dev/ttyO1 hupcl). So I resorted to how the arduino visual IDE does things and eventually I used the Arduino.mk in /usr/share/arduino and built up a blink.hex which was significantly larger than the pure Makefile based one. This one supports loading and reloading via avrgal. The first attempt made an arduino bootsector virus as av500 put it.

Now that all is hunky-dory in the tincan + beagle-xM world, it is time to attach a few peripherals to the arduino and watch it squirm.