Saturday, March 31, 2012

Image processing with cat brains - Gabor Filters

When building panoramas or capturing depth using multiview geometry or structure from motion techniques, temporal decorrelation often makes things harder. One of the confusing elements typically is cloud. Hugin had a nice addition during one of the previous GSoC's to identify match points found on clouds - Celeste. So I decided to attach it to the SFM solution while I figured out how the cloudy_photos option in Bundler actually worked.

So this is how I initially came across the fabled Gabor filter. The Hugin cloud detection implementation classifies matchpoints using the local texture estimated with -- a system similar to that in the mammalian visual cortex. Here is a nice OpenCV Gabor filter which can be conveniently glued together in Python to the osm-bundler scripts.  The cat brain research and the biological basis of the Gabor filter is quite interesting :

In 1946, Gabor suggested representing signals over space and time called Information diagrams. He showed that a Gaussian occupies minimal area in such diagrams.  Time and Frequency analysis are the two extremes of such an analysis.  Gabor filters  became popular in vision partly because J.G Daugman (1980, ‘88, ‘90) showed that the receptive fields of most orientation receptive neurons in the (cat’s) brain looked very much like Gabor functions.
As with Gabor filters, the brain often makes use of over complete, non-orthogonal functions.

- Stanforn Robotics.

I tend to get excited whenever a new library sprouts Python bindings. For point cloud processing, a lot of effort is going into the Point Cloud Library (PCL), it is nice to see Python bindings among the projects for GSoC 2012. There is a pass through filter in PCL which can be used perform simple colour-range based filters on the bundle adjustment outputs. The statistical outlier detection can then be used to clean up any straggling garbage points.

Sunday, March 25, 2012

Shallow Hal - calibrating complex wide angle lenses

I have spent a couple of weeks now chasing down various methods to get a COTS camera calibrated to sub-pixel accuracy. We are effectively measuring the terrain hidden in the lens and the CCD, while assuming the target being imaged is as perfect as we could get it. There are a few ways of modeling the imperfection:

  1. Classic Brown's polynomial model with radial and tangential components assuming a symmetric lens
  2. More realistic 2D spline based models which can be assymmetric and most likely fit the lens better.
Vertical distortion
The spline based solution has more parameters and is messier to solve for, but I started by image processing career filling holes in 3D models of teeth with thin-plate-splines (TPS). So they come naturally to me. Here is the methodology used to estimate a B-spline to fit the distortion, you can do the same with more complex splines (Cubic splines) as well. As always a set of targets (equally spaced solid circles) is used as a target.
Transform to estimate chromatic aberration
The lens distortion manifests itself by the variation in the distance between the dots and via the distortion of the dots into ellipses. Multiple views of the coplanar dots helps establish greater sampling of the distortion space. We always depend on our perception being the only reality, without taking time to calibrate the lenses of prejudice through which we view the world. Fit all the changes from the flatworld view by accounting for the curvature using splines and you turn a hippo in to Gwyneth Paltrow.

Saturday, March 24, 2012

Walking to the beach - 30seconds of observation

Back in St. David's Junior School in Kenya, Mr. Mbao(wood in swahili), our deputy head master once said - "Tisham do you have a photographic memory ?". It did not quite make sense back in year 6. Since then I have heard it repeated a few time. Then I started watching movies (watched Temple Grandin on my flight to Vancouver), reading up on cognition and working in the image processing field. I have come to analyse my own cognitive system and I suppose I am a highly visual person. I convert words into visualisations and concepts become related to reality rather than being strictly tied to language.

It is very difficult to filter from memory all the previous instances of the same object whenever I see a new instance of it. I decided to do the video adventure project yesterday and went on a 10km or so hike to Henley Beach and back. I have travelled on this road numerous times before on a bike when I was living in Mile End back in 2002-03. The previous trips play back in my head like a movie as I walk along the familiar road looking for things to film.

There are several clips, I am fairly sure which one I will put in as a submission (one featuring my shoes, some rocks and water). Still the activities of the arthropods is always interesting. We visit supermarkets like the ants visit a dropped chip, oblivious of where the food actually came from, pay for it with money we think we worked hard for (which in reality other people are working harder for and not getting), and drag it home using the leverage provides by our exoskeletons (read cars). In the ant society only the queen has the rights to reproduce, the sisterhood of working ants are simply born to their fate. After all is said and done, in bee and ant society the workers still retain their queen making rights - yay epigenetics.

A lot of things happen in the universe in 30seconds, it is hard to have a photographic memory of all of it. I seem to live in relative time, the frame rate of my memory increases when my emotions are high and retention is near perfect, with detail I can recall in retrospect which is not obvious at the moment of capture. The key to remembering more of the life I am living seems to be to infuse happiness and sadness into every moment.

Tuesday, March 20, 2012

Clockwork Universe - Certainty, Repeatability,Chaos and GSoC

Whenever someone says that something is random, I always ask - sampling which distribution ? This is faced with stares of ridicule because people always come to hasty generalisations about the randomness of inidividual events without studying the whole system. The above statement is a generalisation on my part and I should spend more time socialising, rather than blog writing.

In nature every individual is random/unique, yet patterns repeat since all the individuals are part of the same distribution. After the development of Newtonian/classical mechanics clockwork universe ideas were rampant, till Poincare pointed out that we cannot actually predict how the clock functions, since our knowledge of the fundamental parameters forming its machinery, namely e and π is uncertain, as is the knowledge of the exact position of the hands of the said clock.


Small uncertainities in the knowledge of initial conditions also rapidly devolve into large uncertainities in the knowledge of the evolved condition, while staying in scope of a larger pattern. All this musing was brought about while watching Hugo.


Some systems are however easier to predict in a short term. It is time again for the annual Google Summer of Code. I have signed up with Osgeo to be a mentor, mostly for Ossim and Opticks. If some other Image processing or raster serving based project pops up I will see how I can fit it in. Till then it remains to be seen whether last year's wonderful GSoC experience repeats itself this leap year (and supposedly the universe ending year too).

Saturday, March 17, 2012

New year in life and new job

A new year in life started a couple of days ago. Sharing the birthday with Einstein brings some disadvantages and advantages at the same time. One of the advantages is the constant nagging urge to learn more about the universe and see it the way the my illustrious birthdaymate did. The disadvantage is realising that if I were ever to invent a time machine and move in this extra dimension he conjured up, I will have to keep looking for clues that I have left myself - damn you predestination paradox.

As part of my continuous Brownian motion through life, I started a new job. No organisation (organism) is cool without a scientific name or acronym, the previous one I worked for was a mouthful - CSIRO, the current one is shorter, just AMX (Aerometrex). Doing multiview geometry mapping and point cloud collection. Calibrating lots of cameras in Agisoft, Imageiron, PhotoModeler etc. The approach here is very pragmatic, we will go with whatever is available off the shelf to create the product and my job is to develop an efficient production chain using the right mix of automation and human intervention.

The first order of business was setting up the development environment including - Python 2.7 64bit goodies and implementing Python image calibration (with some changes to account for the new cv2 API which makes numpy arrays and OpenCV images identical). I read through Zhang's core paper on simple flat checkerboard based camera calibration, implemented in OpenCV to estimate 2-3 radial distortion parameters and 2 tangential parameters (Brown's model), as well as the X and Y focal lengths and principal point (which can be different if the lens has astigmatism). Staring at these calibration targets for a while tends to give you optical illusions as the eye and the brain aim to iteratively approach a calibrated view of reality.



OpenCV needs to be told how many corners to expect, so a simple histogram equalisation and mean transition count is required on the checkerboard. Then simply populate the camera ( remember the corner element is 1) and distortions matrix and undistort. Writing calibrations and undistorts with higher order polynomials and even piecewise linear functions will be required for wide angle lenses. Otherwise I am also looking at the 3 rotations degrees of freedom we have in spaceland and their methods of representation via Rotation matrixes, Quaternions and Euler angles. Is there rotational degree of freedom in space-time land ?

Tuesday, March 6, 2012

Traveller's tales - waiting at airports

Kolkata(way out) - Waiting alone in the upstairs lounge of the rather ramshackle Netaji Subhas Airport. Calcutta seems to have gone to the dogs since the center of power shifted from here to New Delhi in 1912, or may be it was when Netaji's army got trounced (according to General Slim of the "Forgotten 14th army") in the Burma campaign. The viewing gallery upstairs is rather empty and desolate, there is vomit on the floor, I remember coming here nearly 20 years ago to see my father off to Kenya. The will to do something on their own among Bengalis seems to have been broken when they saw their hero fail at liberating India, or it might have been the last 35 years of communist rule which has exposed how the state functions to the people. The Catch-22 of modern economy is that the state puts the currency in place, yet it is supposed to let the market control it. In Bengal people simply aspire to join the state in controlling the market and take home a fixed salary every month, they endlessly sit for the exams to land them government jobs or serve as party cadres in hope of a reward when their candidate gets elected.


Kathmandu(way in) - Paid the price for being unprepared with a visa by jumping through Visa-on-Arrival hoops. Some enterprising locals charged me NPR 400 for a couple of photos printed from a Canon Selphy CP800 (I had left mine in Kolkata). Met some interesting people in the queue, a Baptist volunteer looking after the welfare of Nepalese women repatriated from India after working in the prostitution industry down there. Seems like exoticism is in demand everywhere. And a mother-daughter duo from Brazil who graciously lent me a few US dollars in exchange for NPR's and a sketch.

Kathmandu(way out) - Flight to Kolkata was canceled due to Bandh (Strike), there would be no people to pick up in Kolkata. So they put me on a flight via Delhi which would arrive late at night and depart the next day. While waiting I met a Dutch lady doing strange no gaps/black areas crossword, need to get one of them in English or design one for myself. To pass the time I drew some people listlessly waiting for the often delayed flights, out of the window was the unwritten rule for asking people before drawing them. Met Mr. Morning Agarwal and discussed demand of steel and coal in India and the mining business, apparently his family is from a mining background. Much has changed since wildcatting early days of mining in India, but as is the case elsewhere the wealth and assets from the early days persists. He was interested in Aerial survey and selecting right areas for ground survey since permits are expensive and requires lots of palm greasing.

Delhi - We are greeted by an Air India employee gathering all the passengers going to Kolkata in a pile to ease hopping the barriers which are in place to prevent rather irregular use of the transfer facilities we were about to make. The mudras in the entrance hall and concave and convex mirrors, made an impression about the culture India wants to portray to the rest of the world. Altering the Air India bookings and getting past airport security without boarding passes produced jokes from the IT folks. Mr. Zeus Beater was holding the magic Manifest with 9 names on it. I chatted with someone attending weddings in India as part of a business deal. Here you do not get married to your wife, the marriage is a social and economical transaction. Words floated about Kidd street being a bad area. I tried to explain how aerial survey can help locate aerials which allows Airtel to stay on air, I think I tangled it up as people joked about bad reception in the terminal.


Kolkata(way in) - We are back to base, strolling out of the airport is easy. Everyone from Kathmandu got their luggage together. I walked outside and waited for my new acquaintance to arrive. He rode with us to his large property near Ballygunge. Few acres of land in a prime location in Kolkata, guards at the gate and rows of rented shops outside - the tenants have long stopped paying rent. The reality of holding land and economics in the city.

Thursday, March 1, 2012

In and out of Magical Nepal - Poonhill Trek

Nepal is a Magical place, a great place to visit and just walk around aimlessly. There is always a great panorama at the top of every hill you climb and at the bottom of every river valley and the shores of lake Phewa.

Yet those born here are eager to get out. Kathmandu and elsewhere is full of language schools for guides and those who want to pursue education in - Korea, UK, USA, Australia, Japan, India and anywhere but Nepal. By my estimates there are 4 ways out : Matrimony, Magnanimity, Military and Money.

The last is brought in my the steady streams of tourists taking Visas-on-Arrival at the airport (they only accept foreign currency) or paying in dollars at the trekking agencies for hikes, helicopter rides or a splash down the Trishuli in a raft. The government money concentrates in the hands of the Chhetris (who hold political power) and the tourist money in the hands of the Bahuns (who hold economic power). This sends their sons and daughters to the language schools and eventually abroad. There is nothing wrong with this, just how things operate. I went to Calcutta Medical College with the daughter of former Interior Minister Khum Bahadur Khadka - Irena Khadka.


The hills people particularly the Gurung, who live in the Poon hill - Ghandruk - Ghorepani area which I trekked in get recruited by armies worldwide particularly the British. A lot of the infrastructure around this area is funded by them. The electricity(solar and hydro) and water in these remote areas much more consistent than in the towns - Pokhara and Kathmandu. The bridges are solid and a picture of a retired military patriarch adorns every tea-house. In such a tea house I met Jimmy Woods, a wild and weird man from Banff, Canada who has spent the last 30 years learning traditional Nepalese skills and teaching them to Commando units. He speaks nearly fluent Nepalese and spends half his time in Canada and the other half in Nepal. The martial history here runs deep and lets the people leap beyond the borders of their country.

The other source of foreign currency is donations, some to individual children being sponsored - like the kid I met on top of Swyambhunath, who said his Canadian sponsors had promised to take him to Canada if he aced his SLC (school leaving certificate) and insisted that I buy him some English books. The other lot of donations go to orphanages which tend to sprout up everywhere and attract foreign donations and mostly act as grand swindles at the expense of the kids. Luke, a fellow traveler, tried unsuccessfully to locate the orphanage his parents support. Anyway some magnanimity serves to pull the Nepalese out of Nepal.

The first option I am all too familiar with, as the never ending phone calls my parent keeps receiving after putting up my profile in a local website, of what nature I have already gone into before. When all is considered Nepal is a beautiful place and everybody wants to go there, but the Nepalese want to have a go on the outside. Grass is always greener on the other side, the entire economic system functions on people having different value systems. May be we should sue CocaCola for not providing adequate recycling and causing the accumulation of garbage everywhere including the Annapurna conservation area. The Nepalese are used to biodegradable packaging and are having a hard time adapting to rubbish that does not just rot away in the rain, instead stands around in brightly coloured heaps.

The Himalayas are a rare place and have enormous value in their grandness and breathtaking beauty, they will keep existing for a while and watch the flow of people in and out of Nepal.