Friday, December 13, 2013

A few days in Samoa - Reefs, Thieves and Blue waters

We went to Samoa (the Western one not the American one) for a few days to experience sun, surf and the typical tropical paradise things. However paradises are very fragile, snorkeling is difficult since reefs are dying after a tropical cyclone, which flattened some of the best resorts.
Beach Fales
Beach Fales along the Eastern shores of Upolu

Churches are everywhere, our resort was right next to a very large Methodist compound. The biggest building in Apia is probably the L.D.S. church with a shiny top. In spite of all the religious zeal the commandments are not obeyed, we had been warned about theft at the resort by TripAdvisor and it did happen. Some money disappears from our resort room, the police got called in to take statements from the staff. Only fair given Australian Federal Police is shoring up the policing in Samoa.
To-Suo Swimming hole
The To-Suo water hole at low tide

We drove around for a couple of days, getting througly lost and finding some wonderful places. Samoans only started driving on the left side of the road recently and about 50% of the cars are still left-hand drive, sharing the road with them is dangerous at best. I went off road for a bit and managed to get stuck in mud in pouring rain. A local thankfully came by and helped us extricate the car with some well placed planks.

Eventually we got to To-suo after repeated searching at opposite ends of Upolu. I had a stroll through the Livingstone museum and took photos of falls at sunset. Ate greasy food at the local market, stared at huge taros and rode the local bus (one of which was swept of a ford by a river the day after we left). I was a fun holiday overall.
Sopoaga Falls
Falls near Falefa

Saturday, November 30, 2013

Reflashing BFL Jalapeno with UDOO

I funded the UDOO kickstarter a while back and received it a few weeks ago. The bot I was planning to put it on already has a controller, so the UDOO now lives a sad life as a BTC mining hub.
JTAG Port Pinout - Jalapeno has silk screen for 1 and 9

Testing the GPIOs on the UDOO manually
I have a couple of Jalapenos which I overpaid for on eBay and are all sold out due to Black Friday, but the prices are going up on BTC and I figured I could make them slightly more desirable for resale by performing a firmware upgrade. There are guides for performing the upgrade using the AVR-Dragon JTAG and the Raspberry Pi. Since the UDOO claims to be 4-Pi's in one it should be more than capable of running urJTAG using the GPIO cable and reflashing. The compilation took a couple of attempts due to the lack of back-up battery on the UDOO, every time I rebooted the time had an epoch reset and make got stuck in an eternal loop, eventually it settled down. You can also apt-get install urjtag, but that version does not have the GPIO cable.
UDOO GPIO pins used

On the UDOO we have a very large selection of GPIO's to choose from, for convenience I selected 4-near the edge - 88,56,105,89 and 144 for reset. Then used the Pandaboard GPIO toggling guide to get into the /sys/class/gpio file structure and quickly test these pins with an LED. The UDOO GPIO's are at 3.3v so it is safe to use them with the Jalapeno. I also colour coded the cables for future reference. Here is the table of the colours and pins on both the UDOO and the Jalapeno. The reset pin can be toggled using the LED testing method above to reset the board if you get stuck, simply output "low" then "high" to GPIO 144.
Cabling between Jalapeno and UDOO
Function Colour Jalapeno  GPIO
RESET Grey 6 144
TCK Yellow 1 89
TDO Purple 3 104
TMS Orange 5 56
TDI Blue 9 88
GND Black 2 GND
Once everything is connected fire up the Jalapeno and reflash it using the JTAG command chain below. I would really like to gather some statistics on the results of the detect command to see which CPU people have.
urJTAG reflashing of Jalapeno AVR
That's it, a great use for the many UDOO GPIO's go crazy and hook up 10 Jalapenos at once and run a reflashing service or send it to me for a reflash. If you found this useful BTC donations are welcome - 1JGoD1muFP5RiGPUWF12ncYz2WreYk35Rh .

Saturday, November 23, 2013

All forms of LTC mining - CPU and GPU (OpenCL/CUDA)

CPU mining is definitely deprecated. You never get the bang for power, unless you have machines with 32 cores and free power. Well at least for BTC. For LTC cpu mining (very slowly) is still feasible. Quite good if you have one of those 32 core - 128GB RAM beasts lying idle, or you are Amazon or Google. Just fire up pooler's minerd and take up a bunch of CPU's. A little bit of math shows EC2 CPU power will not be worth the investment, but with LTC prices heading north of $10 theres is no telling what people will try - including writing minerd into viruses. Amazon might stop EC2 service and start CPU/GPU miners itself.

CGMiner 3.7.2 with scrypt support can also get a fairly good rate on AMD Radeon cards with better OpenCL support. The ROI on these cards is definitely better than any of the miners in hand. Meanwhile people are pay AUD700 for 5GH BFL Jalapenos on ebay and joining the BTC bandwagon. Fun times ahead.

I am enjoying myself trading LTC/BTC and getting cheap thrills on penny stocks. Much better than when my horse (Verema) came last at the Melbourne cup, stumbled fell and had to be put down. I still won for last place. Even if like the horse, this vaporware currencies stumble fall and have to be put down they will leave behind legends and fiat currency millionaires.

Monday, November 18, 2013

Big Mole hill to fill holes - Meshlab

For a few days I was battling holes in a very dense mesh, a taste of which you can find here - Aero3Dpro. Meshlab is pretty handy for working with these, but there was a UI bug in edit_holes plugin preventing sorting of the holes to identify and squash the biggest ones. So in addition to filing a bug I decided to look into it myself.

There was a small run into the Qt change of ownership dilemma, tracking down the tools required some digging. Eventually I tracked down a working source set, had to compile it myself since the x64 binary I was depending on was built without QtScript support which Meshlab uses. By the end of all the trial and error I had 3 versions of Qt kicking around in my system - 4.8.0, 4.8.1 and 4.8.3. At least they all work for various projects.

Time to start writing my own meshlab plugins such as polygon based selection as featured in 3D Studio Max (called Fence Selection) or Cloud Compare(called Manual Segmentation). Also the Edit holes tool can be modified to automatically select larger holes to fill. Fixing up the textures to vertex colour transfer, since only the first texture is sampled -
path = path.left(std::max(path.lastIndexOf('\\'),path.lastIndexOf('/'))+1)
.append(srcMesh->cm.textures[0].c_str());

Also linking up with libLAS to allow LAS format saving with offsets and scale avoiding quantisation errors. More meshlab development to come.

Coming late to the party - Coin mining

Deep in the days of custom built ASIC miner boxes, mining with USB miners and hubs is considered a waste of electricity. Unless of course you are the government and own Three Gorges, still large miners are more efficient. Still I needed to get started and picked up a bunch of now collectors item - Icarus USB miners of eBay. Cgminer makes these really easy to use, with proper modules (in linux) or usb driver substituion with Zadig.

Next comes the fun part, choosing a pool to mine at. All say they are fair and there are so many pay-out algorithms, worth a PhD in economics by themselves for rewarding people for work done. Eligius has the CPPSRB, Bitminter uses PPLNS with donations. Some support all SHA256(SHA256) based coins, both BTC and NMC. Miners practice aggresive pool hopping (and currency hopping) to pick up the one with best luck. Some pools require registration, other just the payout address. All fertile ground for bots to play at.

After playing with BTC for a while I looked into LTC. Since it is still mined on the GPU (with scrypt CGMiner 3.7.2) and has less aggresive speculation. It might also just become irrelevant, but not being mainstream has some allure. Indeed these are the days of panning gold in early California or even closer to home at Eureka. Individuals issuing currency causes some disruption to monetary policy and governance. May be there will a tax put on selling mining ASICs, similar to a mining licence. Overall it is an intersting economic experiment.

Thursday, October 10, 2013

Short Holiday in Hobart - Meeting David Walsh

It has slightly more than a year since I left Hobart. I did make some very good friends, back here and I wanted to come back and catch up with them. Especially the people at MONA and fellow Salsa dancers, as well as the motley crew at the Prince of Wales. So when I was invited to visit MONA and catch up with David Walsh, I jumped at the opportunity.
While living in Hobart I used to visit the museum every Sunday. Funny how only religions are associated with regular activities, I must have signed myself upto the religion of feasting sense on the weird.

I went through the museum and noted all the changes since I was last there. Listened to the chatter from visitors and made some notes which came back to bite me when one of David's guest picked up my notebook.

David has made his made his fortune gambling. He is not the James Bond sort of poker player, rather he is a mathematician and sort of a programmer. I have some theories about why we persist in gambling in spite of the apparent losses to individuals.

Gambling exists to redistribute wealth, preventing it from accumulating over generations in one family causing resentment and decimation of the family through revolt and aggression from the rest of the society. Though decimation of powerful families does happen in spite of the gambling habits as shown by the Russian revolution or even more recently the Nepalese royal family. The best way to stave of socialism and revolts is for the wealthy to voluntarily dispense of their wealth in gambles. Just means of justifying wealth accumulated by David, under the nose of ATO. Apparently David used to work for the ATO before he declared himself dead and quit.


The conversation drifted to light rails for Hobart and the need for personal ownership of assets i.e. cars vs public infrastructure such as rails. This debate has been raging on even recently in the roads vs rails talk. A psychologist was around outlining a Jungian experiments in a Victorian mansion, people participating dreamt of fires. There were some stories about table tennis diplomacy and frame dragging, and lots of conversations with constant backup from wikipaedia.

Dinner on MONA
I was invited to dinner at David's place with take-away from his restaurant. David and his wife are pescetarian. Dinner was followed by a David Bowie variety hour on Youtube and hanging out with the MONA music co-ordinator who has encyclopaedic knowledge of the songs. I finally slunk away rather dazed by the days events.

Tuesday, September 10, 2013

JPEG2000 Encoding with CUDA

For the last few days we have been struggling to efficiently write out very large JPEG-2000 files. In all their wisdom Erdas decided to supply only a 32-bit version of the ECW compressor. It seems to have trouble allocating memory for large JPEG-2000 files, is single threaded and dead-slow. Somewhere in the Ermapper install directory there is also the Kakadu core dll. So I decided to have a look at other methods of creating JPEG-2000, possibly faster and more scalable options. Kakadu seems to only supply an SDK and we can test the speed of compression with the SDK. GDAL also supports building against the Kakadu SDK and would be handy for creating the large mosaics we have been struggling with. It seems with modern hardware that is not the fastest option.

After some trial with Kakadu and a bit of reading I came across a few CUDA based implementations - CUJ2K and JPEG2K.  I decided to port the most recently maintained CUDA based JPEG-2000 implementation, claiming to be the fastest on commodity hardware to windows. The CMake based build system makes things easier. The code however was written for a POSIX based systems so I was missing a bunch of headers. I had to hack away at it and move variables around to cope with lack of C99 compliance in the MSVC C-Compiler or I am forced to used the /TP flag to upgrade the C-files to C++ files and allow variable declaration anywhere. Lots of moving variables around later, everything worked fine.

The compression library is built against FreeImage which functions as an I/O driver for the encoder/decoder. However only writing to the J2K profile is supported, a wrapper for the JP2 mode will be required to add-in all the projection metadata GIS Rasters typically require. To make this library more useful for GIS purposes we will need to add an I/O handler for GDAL instead of FreeImage to allow faster compression/decompression.

I have tested the JP2ECW, JP2OpenJPEG and JP2Kakadu drivers on a small file (26MB in .bmp format, since the Kakadu demo binary does not support LZW .tif I have lying around). The times averaged over a few runs on an i7-3930K @ 3.2GHz are shown below:
  • LibECW - 0.8s
  • Kakadu (1 Thread) - 1.5s
  • Kakadu (8 Threads) - 0.5s
  • OpenJPEG - 2s
Compared to these the CUDA library on the GTX580 on the same machine runs at 1.2s (with compression taking 0.75s, rest is I/O).  The logical next step is hooking up the compressor to GDAL for data-block I/O and building a JP2 wrapper in addition to the J2K profile which it currently uses. The Affero GPL licence also needs to be sorted out.


For those interested in the windows build, here is the SVN Diff against rev 84 and the pre-built release binaries against CUDA 5.5.



Thursday, July 4, 2013

A simple ray tracer in Python - with CGKit and RTree

First of all I am over-reaching by calling this a ray-tracer, it is more appropriately called a "Splatter". I am simply projecting all the top-triangles in an OBJ model onto a raster for True Ortho generation purposes. It was a fun project for playing with Kd-trees and line-triangle intersections. The OBJ models are geo-referenced so it does not make sense to take them into a purely CG/Game oriented ray-tracer like Blender and lose the geographic context, as well as fiddle around with setting up orthographic cameras and render resolution to match ground-sampling distance. Doing it through a proper ray-tracer typically also requires setting up appropriate lighting, while I just want to sample the pre-exisiting texture in the model.
Direct vertical projection render from script
Properly lit render using Terragen

As usual a bit of googling located the python batteries that will perform the intersection as well as parse the obj mesh I have at hand. CGKit fits the bill perfectly both in terms of parsing as well as ray-triangle intersection.

However testing every triangle for intersection for every grid-cell quickly becomes horridly slow and begs for a hierarchial index to speed up the look-up and only process the triangles in the path of the ray. Here I picked up RTree to build a 3D index , a kD-Tree of Bounding volume hierarchy to speed up intersection testing.

After building the tree I went back to only testing relevant triangles in CGKit and dug through the parsed OBJ data structure to extract the texture co-ordinates of the intersected face. The barycentric description of the intersection point is sufficient to sample the texture for the right pixel colour and transfer it to the raster grid. For those interested in the implementation, the code follows, improvement suggestions are welcome. Multiprocessing is thrown in for tile-by-tile processing.

The point of the ray-tracer was to generate true-orthos for GIS use, as a side effect of implementing the ray tracer we can now generate true-orthos from both the top and the bottom of the model. The main article about true-orthos can be found here.

Wednesday, July 3, 2013

Making 64-bit installers with CMake and NSIS

I was getting tired of my 64-bit programs getting installed by NSIS into the x86 folder under windows, just because the NSIS installer itself is 32-bit.

I started piecing the solution together from this bug report for CMake. NSIS has been ported to 64bit but the inclusion in CMake is only available in the latest release. So I had to get NSIS 64,  and latest version of CMake 2.8.11 and then make some modifications to CPack to enable the new 64-bit installer.


Change installer type to NSIS64: SET(CPACK_GENERATOR NSIS64) and re-package your application. Voila you get correct program file target location and here is a screenshot of the running application just for fun.
Some products such as PCL do a platform detection while building to suggest an install folder, using the snippet below. This method will also work if you don't feel like installing NSIS64.

if(CMAKE_CL_64)
    set(CPACK_NSIS_INSTALL_ROOT "$PROGRAMFILES64")
    set(win_system_name win64)
 else(CMAKE_CL_64)
    set(CPACK_NSIS_INSTALL_ROOT "$PROGRAMFILES32")
    set(win_system_name win32)
 endif(CMAKE_CL_64)

Thursday, May 16, 2013

Medieval Fair 2013 - Making rings for chainmail

I have dressed up as Robin Hood and gone to the Medieval fair in Gumeracha sporting a bow. This year I decided to sport my camera with the classic 20mm pancake instead. The first thing I walked into was a sword holding competition. The challenge was to hold a 15 kilo sword with the arm extended horizontally for as long as possible. Plenty of fitness instructors and heavy lifters had a go at it, the record while I watched was around 3 minutes. I think I am going to practice with a cricket bat for next year.
Feats of Strength
Being a builder builds muscles
There were also jugglers and irish dancers, and to my dismay people with huge amounts of camera gear shadowing them. I felt really unprofessional with my dinky m43, then again this is the age of mobile photography and the battle between processing vs gear is on. It is now even being used as a marketing point by those Sony NEX ads. Granted a micro-four thirds sensor is smaller than the APS-C, still the weight advantages and flexibility bonuses do apply.
Juggler
Juggler practicing
There were a few knights in chainmain and I walked into a conversation involving effort and time that goes into making the armor. Eventually the talk drifted to cheap Indian labour working for $1 a day so that people can have fun the fair, at that point people glanced at me and stopped talking. I should start tinkering with fencing wire to keep the stereotype going. Some things cannot be mechanised yet, clothing be it out of fibres or metals is one of them. The economics of hobbies is also interesting.
Period Piece
Riveted chainmail
The melee of kids with pugil sticks reminded me of a Schlock Mercenary episode. It will be good to get some excercise done while in costume.
Pugil sticks
Pugilists not boxing
There were also Middle eastern belly dancers and scribes doing calligraphy while complaining that the Belly dancers are not Medieval. Though the ululation accompanying the dancing is fairly universal.
Belly dancer
Belly dancer and musician

Finally I managed to put the 850mm IR filter to good use and capture a panorama including the little lake at the Viking encampment. Without special IR modifications to the camera I had to use a pretty high ISO (1600) and a 2 second exposure. The processing through RawTherapee worked fine using some IR profiles I found floating around in Github, Hugin did a fair job of stitching as well.
Viking Camp in Infra Red
850mm IR filter with unmodified camera panorama

Monday, May 13, 2013

NASA SpaceApps Adelaide 2013

A few weeks ago NASA hosted its second annual space apps challenge. I did not manage to participate on the first day and only managed to struggle in on the second day to witness the finale. The Adelaide Hackerspace provided the venue this year and Simon has very nice write-up on their blog.
There were a few more entries from around Australia including a slinky based greenhouse suitable for use on the moon and a meteor tracker integrated with Google sky. It was a couple of days of pizza and coffee, ending with beers by the time we finished.
Everybody went home pretty tired, but happy. I will definitely try to put in a couple of days of participation next year. Meanwhile feel free to use Mearth and look up all those cold place in Russia, Scandanavia and Hokkaido which have Mars like temperatures. Whoever is considering applying for Mars-One should visit one of these places and film on location.
Space Apps Coding room

Saturday, May 11, 2013

Accurate positioning with UBlox Neo-6P and STM32

The Ublox Neo-6P provides raw clock phases and ephemeris suitable for use in post-processing based accurate positioning, or even RTK solutions using RTKLib.

STM32 with Eclipse was fairly messy to set up and depending on which version of GCC and CoreSupport libraries you use their are a few bugs to iron out. Used Code-Sourcery Lite toolchain, but compiled version did not behave as original firmware did. There are Eclipse based STM32 development commercial support from Atollic, and there were hacks to get the debugger Atollic packages working for ST-Link using the free options, but in the recent incarnation they have locked it down to TrueStudio only. I had to resort to OpenOCD for my debugging needs, it works like a charm with standard Eclipse settings, but I managed to nuke my STLink driver in the process and had to work hard to get it back.

Importing to GrafNav is fairly straight forward via the UBX converter, but it gives no indication on how it converts events, and you need sufficiently long logs to pick up any ephemeris. UBlox Receiver protocol is a must read for configuring the NEO-6P properly and monitoring the logs to ensure GPS lock. I am looking into the STM32 demonstration samples to add some more functionality to the project, including external interrupts from a camera for event marking and a GPS lock acquired status LED.

Thursday, May 9, 2013

Adelaide Mini Maker Faire, Zoo and Indian Mela

I have been involved with the Adelaide Hackerspace on and off. Lately my attendance has dropped a lot due to work commitments and a casual addiction to photography. A few photography related hacks are stewing in my brain, so I might go back there. Anyway a few weeks back I went to the Adelaide Mini Makerfaire and it was great opportunity to see all the different things going on in Adelaide, from Dalek making, Metal working to 3D printing and Electric cars. I had a great time catching up with people and learning about new groups. The airstream guys had some hacked wireless AP's on display, including the Minitar on which I cut my teeth in cross-compiling and writing kernel modules to interact with the wifi-stack. There are a lot more photos.

Maker Faire
I had planned a busy day, so I headed off to the zoo to attend the local photography group session and check out some pandas. It was my first time at the Adelaide zoo, so it took a while to get oriented. We covered a fair bit of ground in the hour and a half before closing. Managed a few shots of hippos, birds, otters and mountain goats. Lots of other animals which hardly stood still enough to compensate for my nascent photography skills.
Flamingo
We were trudging and hobbling back to the car when we noticed the bright lights and steady stream of people heading to Elder park. The local indian community was throwing their annual mela. So we headed there for a meal and to feed our cameras with some colour. The movement and low light made photography pretty difficult from the back of the crowd. It was a great day overall.

Thursday, April 4, 2013

Building Liblas for Win x64 with GDAL and Geotiff

Some things are obvious to even the most oblivious, how to get liblas from OSGeo4W is one of them. However you quickly run into trouble as images, dem files and las files get bigger and the 32 bit osgeo builds start causing issues. There have been some discussions regarding a 64 bit version of Osgeo4W but getting builders for all the packages probably hasn't been achieved yet. I will see if I can step in and fill my drive with source trees and wield my MSVC 2010 license for good rather than evil. It is now up on the OSGeo GSOC ideas page, may be someone will take it up and make GIS toolchain deployment easier.
Liblas install for OSGeo4W set-up
Anyway here is how to built Liblas, assuming you are reasonably familiar with downloading sources and clicking a few buttons in CMake GUI. Grab LibLAS 1.7.0 sources. Since we want to add image support, grab GDAL 64 bit builds with headers from GISInternals. Build Geotiff 64 bit by hand using the handy CMake system, pointing it to proj and tiff obtained from the GDAL build above. Grab Boost from the handy distribution in PCL, you can get it somewhere else but PCL was my last port of call for boost. The final Liblas config page in CMake should look something like below.
CMake Liblas config
Change CMakeLists.txt for LibLAS to include the boost chrono library, there are linker errors otherwise.

find_package(Boost 1.38 COMPONENTS program_options thread chrono REQUIRED)

MSVC 2010 Liblas project
Configure and build away. I use MSVC 2010 Professional, running the bigfile_test after building has finished creates a 4GB+ dummy .las file and is nice for checking the x64 version. Put all the dependency DLL's in place after the build has finished using the handy Dependency walker.
Dependency walker check for libraries
Most windows users do not have a compiler installed and the inability to get a GIS toolchain on windows which can handle large datasets beyond 32bit restrictions is a major stumbling block in the wider adoption of the OSGeo toolchain. You can grab my Liblas binaries here and if you are interested student looking to package for 64bit let  OSGeo know.

Thursday, February 28, 2013

Where am I .... all the time

Okay lets start by clearly stating the futility of position, every defintion of position requires a reference frame. It would be pretty messy to define where I am relative to the centre of our galaxy at all times, the super-massive blackhole makes measuring time there pretty messy as well. I could define my position better in ECEF or ENU or in most cases Platte Carre. I set out to build a project which could define where I am at all times for the posterity and essentially keep a track of my spime. My spime is the only thing I have absolute natural rights to, everything else can be taken away and be subject to argument with sufficient legal juggling. Come to think of it even the personal spime is not inviolate, meh reading to much Hannu Rajaniemi. Android My Tracks is pretty good, but a phone sometimes feels like too many eggs in one basket, I don't want to leave it lying around in the car dash gathering sunlight.

The project is mainly based on a Seeeduino stalker board with a convenient Bee socket. I plugged the UBlox Neo-6M based GPS Bee there. Data logs go to a 2GB SD card and Lipo power is backed by a 1w solar cell. The GPS constantly spits out NMEA strings which get logged to the SD card as long as power and space is available. A log with 2 days worth of data took up 45Mb, so I can hold about 3 months of data. Unfortunately the 2000mAh battery died after 2 days of continous use, with some solar recharge while in use. Since the battery death, in order to prevent melt down in harsh sunlight and continuous use, I have added a USB charger option as a stop gap. This should hold the fort till I plug in the quartz/heat powered charger for use while hiking and the mini windmill for use while wind surfing. Eventually the SD card will blow off into star dust after I have had my fun and extracted and time and location of said fun, but hey SD cards are more expensive per-ounce than gold.

Monday, February 18, 2013

Baby food and Poker machines - Snapping stuff in MongKok

The biggest story making headlines while I killed time between city walks in HongKong was one featuring "Baby Formula", including an event just a few steps from me at the MongKok East station. Someone taking lights over was cornered and searched for baby food by citizens and angry mothers.
Australia is one of the wealthiest nations in the world in per-capita terms and most of our wealth is locked up in the Australian dream - house. In contrast it is quite difficult for young people to own a house in HongKong, so the consumption tends to be high in other areas. People eat out more often, since the living areas are small and can be a hassle cooking.
 People invest more in personal appearance, buying jewellery, clothes and watches. Admittedly I am very poor at brand recognition, but the greek links to Doxa made me remember that particular brand. Personal investment is also apparent in all the rush for cosmetics and offers for private tuition. Image and wealth you can carry about on you is everything here. Just like the pokie joints pretending to be video game parlours.

There is a tension between pro and anti-Falun gung groups. The posters for both factions are displayed side-by-side near the star ferry terminal, right next to the anti-communist party booth. The Harbour city mall with its clientele of mostly mainland buyers is nearby. The rebelliousness may be filtering back along with the baby food.