Shallow Hal - calibrating complex wide angle lenses

I have spent a couple of weeks now chasing down various methods to get a COTS camera calibrated to sub-pixel accuracy. We are effectively measuring the terrain hidden in the lens and the CCD, while assuming the target being imaged is as perfect as we could get it. There are a few ways of modeling the imperfection:

1. Classic Brown's polynomial model with radial and tangential components assuming a symmetric lens
2. More realistic 2D spline based models which can be assymmetric and most likely fit the lens better.

The spline based solution has more parameters and is messier to solve for, but I started by image processing career filling holes in 3D models of teeth with thin-plate-splines (TPS). So they come naturally to me. Here is the methodology used to estimate a B-spline to fit the distortion, you can do the same with more complex splines (Cubic splines) as well. As always a set of targets (equally spaced solid circles) is used as a target.

The lens distortion manifests itself by the variation in the distance between the dots and via the distortion of the dots into ellipses. Multiple views of the coplanar dots helps establish greater sampling of the distortion space. We always depend on our perception being the only reality, without taking time to calibrate the lenses of prejudice through which we view the world. Fit all the changes from the flatworld view by accounting for the curvature using splines and you turn a hippo in to Gwyneth Paltrow.

New year in life and new job

A new year in life started a couple of days ago. Sharing the birthday with Einstein brings some disadvantages and advantages at the same time. One of the advantages is the constant nagging urge to learn more about the universe and see it the way the my illustrious birthdaymate did. The disadvantage is realising that if I were ever to invent a time machine and move in this extra dimension he conjured up, I will have to keep looking for clues that I have left myself - damn you predestination paradox.

As part of my continuous Brownian motion through life, I started a new job. No organisation (organism) is cool without a scientific name or acronym, the previous one I worked for was a mouthful - CSIRO, the current one is shorter, just AMX (Aerometrex). Doing multiview geometry mapping and point cloud collection. Calibrating lots of cameras in Agisoft, Imageiron, PhotoModeler etc. The approach here is very pragmatic, we will go with whatever is available off the shelf to create the product and my job is to develop an efficient production chain using the right mix of automation and human intervention.

The first order of business was setting up the development environment including - Python 2.7 64bit goodies and implementing Python image calibration (with some changes to account for the new cv2 API which makes numpy arrays and OpenCV images identical). I read through Zhang's core paper on simple flat checkerboard based camera calibration, implemented in OpenCV to estimate 2-3 radial distortion parameters and 2 tangential parameters (Brown's model), as well as the X and Y focal lengths and principal point (which can be different if the lens has astigmatism). Staring at these calibration targets for a while tends to give you optical illusions as the eye and the brain aim to iteratively approach a calibrated view of reality.



OpenCV needs to be told how many corners to expect, so a simple histogram equalisation and mean transition count is required on the checkerboard. Then simply populate the camera ( remember the corner element is 1) and distortions matrix and undistort. Writing calibrations and undistorts with higher order polynomials and even piecewise linear functions will be required for wide angle lenses. Otherwise I am also looking at the 3 rotations degrees of freedom we have in spaceland and their methods of representation via Rotation matrixes, Quaternions and Euler angles. Is there rotational degree of freedom in space-time land ?

Beagle has 3 eyes - Kinect + Beagleboard

People have hooked up the Kinect to a few embedded platforms - Intel Atom and Gumstix Overo have come up. It was time for the other darling of open development, the BeagleBoard to talk to the Kinect. I have done stereo on the beagleboard before. The kinect makes things a lot easier.



First the usual step of getting libfreenect going, I ended up using unstable since it includes the OpenCV bindings. The sync API is much more stable than the callbacks and has fewer frame drops. Since I don't have a spare keyboard/mouse set for the Beagle and Synergy refused to co-operate, I did a demo frame viewer run with OpenCV rendering the frames. I had to upgrade opencv, ffmpeg, libusb, cmake and bunch of things on Angstrom, then replace CMake opencv detection with pkg-config to get everything working.


Meanwhile there have been some major changes in the OpenKinect world with PrimeSense releasing the OpenNI framework. This releases the Microsoft stigma from the Kinect and encourages more open source development of applications. You can grab the OpenNI code from Github and build on your platform a from the manufacturer kinect driver. No stigma about a hacked Kinect (or shall we say PrimeSensor) any more either, it is officially open.

Getting Data from Kinect to OpenCV

A fair bit of work has already been done in simple template based object identification with the Kinect and even Skeleton tracking. I finally got my hands on one today from JB Hi-Fi, and paid a few bucks extra for it. Harvey Norman finally got them in stock at the end of the day. Terrible logistics or favouritism to employees, or something in-between. Anyway I can now get a 2nd one from the pre-order. Apparently a single USB 2.0 bus is insufficient to handle the stream from 2 Kinects. I did some simple experiments with OpenCV, just adding some hue mapping and swapping the RGB channels, tomorrow we are having the 12Hr Hackathon at Adelaide Hackerspace, might get to make a lantern controller then.

HgSubversion with Ossim-opencv

I have come to really like the flexibility of making changes and saving them locally with Mercurial , before pushing them off to a repository online. It lets me work on a project independently while saving my work as well as staying upto date with core progress. Ossim-opencv is one of the projects I did some minor contributions to a while back. I wanted to get back into it, but following the incident with losing my last google account I no longer had commit access. So I checked out the SVN with HgSubversion and started making changes and commiting them locally. Finally the organizer switched to my new ID and I tried a push. Things fell apart at this point. I had committed things with my local username, and I usually use my googlecode authentication to push since these are separate things. In subversion however these need to be the same and I was stuck with my changes. Happily Mercurial lets me shelve, make patches, push to another mercurial repository (sandbox) and do a lot of fancy things. I decided to take the convert route and changed name of the committer to match my ID.

After that I pushed it back into SVN. While doing this I of course validated the results. Here is the plugin loaded into my latest compile of Imagelinker showing OpenCV based Laplacian filter.

The beagle has 2 eyes - OpenCV Stereo on BeagleBoard

I have the ambitious aim of allowing my Quadcopter to see in stereo. So recent tests are being conducted of a suitability of on board stereo processing via OpenCV. So far the unoptimized version of OpenCV makes this really slow and impossible to do in realtime. I just got the last day of author registration in for IGARSS as well, I hope it was on time.

Some USB devices arrived from deal extreme - USB GPS running at  1Hz / 38400 baud and a Ralink Wireless-N dongle which does not have prebuilt drivers in Linux or atleast not one that autoloads with the USB device. I will investigate the wireless later but for now it frees up a slot to try out stereo - so here is the rig.

The gstreamer capture and opencv stereo_match both work as expected and it takes about 300ms to grab frames and 12300 milliseconds to form the disparity map with default settings. Much can be improved in the opencv front to reduce the time required for processing. As a comparison my laptop using the same set of tools - Gstreamer for windows with the ksvideosrc as frame source and OpenCV 2.1. The frame capture speed was 250ms and the disparity map was produced in 300ms.

So a realtime stereo vision based quadcopter may not be possible, but data capture and indoor map building with post processing is definitely feasible. I got a quote for the Hokuyo laser scanner, though significantly more expensive than the stereo rig ( few hundred thousand yen) it has larger field of view and more reliable point cloud production independent of features on the target surface.

EarthHour - working together and version control

Yesterday I had the pleasure of observing EarthHour at Adelaide HackerSpace. We managed to make a few volts from various magnet and coils of wire contraptions. We have indeed come a long way in understanding electromagnetic induction since Faraday threw the magnet out of his wire coil in frustration.

The first set up used a coil and few magnets along the spokes of a bike. That produced way too few volts and there was talk about the joule thief. I had brought along one of my brushless DC motors and with 6-diodes forming a 3-phase bridge you could run it as a generator.

Working together with other people is always fun and I strongly believe that the group is much more than the sum of the individuals forming it (in signal processing speak - there are cross-terms). The tools needed to work in different groups can be different. Hence I have ended up learning a fair few languages and currently a fair few version control systems.

Recently I set up Bazaar for the Lyx-Outline project on window. TortoiseBzr came to the party, but it did not solve the Launchpad authentication issue. I had to use Putty tools (PuttyGen and Pageant) to generate and lodge the ssh keys so that I could perform pushes to the repository. Hopefully this work will encourage me to write my thesis nicely formatted in Lyx.

Lastly the vision project I started in hackerspace with my cheap webcams to detect depth has progressed slightly and acquired some robot os camera calibration code and a checkerboard. May be my quadcopter will be able perceive depth and fly by vision and landmarks much like a pigeon.

On the sports track - divers in OpenCV

Video tracking can be quite challenging. I compared 3 algorithms based on OpenCV for a tracking solution:

1. Camshift - originally created for face tracking, it locks onto a blob with similar colour. In this case the colours are too similar with the background resulting is appalling performance from camshift.
2. Optical Flow - LkDemo shows this technique in use. It works pretty well till an obstruction passes in front of the tracked object clearing all flow points. The flow points then need to be reinitialized.
   
   
   
3. Template Tracking - Works much better than the other two. However does not regain lock after obstruction.