Monday, September 20, 2010

Processing TerraSAR-X in Python and R

Finally some 12GB of TerraSAR-X Quad-pol data finished downloading. I have essentially 3 PolInSAR capable sets (if the notorious X-band coherence holds up). I can process them in RAT but that requires converting them to floating point and more bloat, the TSX data is in a CInt32 form. Solution was to use GDAL's TerraSAR-X Cosar support and read subsets directly into Python. I took this opportunity to do some nice unit testing in Python. The result was the simple monstrosity here. It extends tsx_dual class I had implemented before.
R von Mises

Then I moved onto some phase difference statistics and encountered for the n-th time the nicely non-Gaussian von Mises distribution. Trying to use R Circular Statistics from Python became the bane of my existence tonight. RPy2 current code is unsupported on windows so I am lacking package import. At least I managed to hack it into compiling with the following tricks:
  1. Copied Rinterface.h from R source distro
  2. Hacked Rinterface.h to remove uintptr_t typedef
  3. Hacked na_values.c to remove dynamic allocations (compile time non-constants)
  4. Copied R dll's from "bin" to "lib" in R install
This makes rpy2 build and install, but I am left with - Assertion failed: PyType_IsSubtype(type, &PyLong_Type), file rpy\rinterface\/na_values.c, line 166  - at runtime obviously due to my hacks. Well I am missing this code block in my hand hacked things:

/* on some platforms these are not compile-time constants, so we must fill them at runtime */
+  NAInteger_Type.tp_base = &PyInt_Type;
+  NALogical_Type.tp_base = &PyInt_Type;
+  NAReal_Type.tp_base = &PyFloat_Type;
+  NACharacter_Type.tp_base = &PyString_Type;


Now to figure out where to put it and von Mises are in Python.

PS: So it is "Talk like a Pirate Day" so "R - ARRR in".

3 comments:

ecollage said...

Could you do a post on how exactly you got Rpy2 installed on win7? Have been tearing my hair out for >2 days with no luck. Am trying with rpy2-2.2.1 on win7 32 bit, Python 2.7 and R 2.13.1.

I replaced the missing Rinterface.h file into the R binary distro INCLUDE folder. I then run:

> python setup.py build install

…and it chokes at:


File “setup.py”, line 152, in finalize_options
config += get_rconfig(r_home, about)
File “setup.py”, line 312, in get_rconfig
rc = RConfig.from_string(rconfig, allow_empty = allow_empty)
File “setup.py”, line 272, in from_string
+ ‘\nin string\n’ + string)
ValueError: Invalid substring

in string

I tried to run the command directly in a python shell, using:

rexec = os.path.join(R_HOME, ‘bin’, ‘R’)
cmd = ‘”‘ + rexec + ‘” CMD –cppflags’
rp = os.popen(cmd)

…and the response for all –ldflags’, ‘–cppflags’, ‘LAPACK_LIBS’, ‘BLAS_LIBS’ (called on line 150 of setup.py) are empty.

HELP!!

ecollage said...

Could you do a post on how exactly you got Rpy2 installed on win7? Have been tearing my hair out for >2 days with no luck. Am trying with rpy2-2.2.1 on win7 32 bit, Python 2.7 and R 2.13.1. I moved the files across from \bin\i386 to \bin like you mention. I then run:

> python setup.py build install

…and it chokes at:


File “setup.py”, line 152, in finalize_options
config += get_rconfig(r_home, about)
File “setup.py”, line 312, in get_rconfig
rc = RConfig.from_string(rconfig, allow_empty = allow_empty)
File “setup.py”, line 272, in from_string
+ ‘\nin string\n’ + string)
ValueError: Invalid substring

in string

I tried to run the command directly in a python shell, using:

rexec = os.path.join(R_HOME, ‘bin’, ‘R’)
cmd = ‘”‘ + rexec + ‘” CMD –cppflags’
rp = os.popen(cmd)

…and the response for all –ldflags’, ‘–cppflags’, ‘LAPACK_LIBS’, ‘BLAS_LIBS’ (called on line 150 of setup.py) are empty.

HELP!!

Jiri Pik said...

These problems are easy. You can simply comment out the code for get_rconfig and replace them with


config = RConfig()

#
# for about in ('--ldflags', '--cppflags',
# 'LAPACK_LIBS', 'BLAS_LIBS'):
# config += get_rconfig(r_home, about)

print(config.__repr__())

self.include_dirs.extend(['c:\\Progra~1\\R\\R-2.13.1\\include\\'])
self.libraries.extend(['c:\\Progra~1\\R\\R-2.13.1\\library\\'])
self.library_dirs.extend(['c:\\Progra~1\\R\\R-2.13.1\\library\\'])

you will get further on, up to the C++ problems.

I will post the procedure at
http://blog.jiripik.com/archives/2011/08/22/how-to-get-rpy2-work-in-windows/

once finished