Monday, September 20, 2010

Processing TerraSAR-X in Python and R

Finally some 12GB of TerraSAR-X Quad-pol data finished downloading. I have essentially 3 PolInSAR capable sets (if the notorious X-band coherence holds up). I can process them in RAT but that requires converting them to floating point and more bloat, the TSX data is in a CInt32 form. Solution was to use GDAL's TerraSAR-X Cosar support and read subsets directly into Python. I took this opportunity to do some nice unit testing in Python. The result was the simple monstrosity here. It extends tsx_dual class I had implemented before.
R von Mises

Then I moved onto some phase difference statistics and encountered for the n-th time the nicely non-Gaussian von Mises distribution. Trying to use R Circular Statistics from Python became the bane of my existence tonight. RPy2 current code is unsupported on windows so I am lacking package import. At least I managed to hack it into compiling with the following tricks:
  1. Copied Rinterface.h from R source distro
  2. Hacked Rinterface.h to remove uintptr_t typedef
  3. Hacked na_values.c to remove dynamic allocations (compile time non-constants)
  4. Copied R dll's from "bin" to "lib" in R install
This makes rpy2 build and install, but I am left with - Assertion failed: PyType_IsSubtype(type, &PyLong_Type), file rpy\rinterface\/na_values.c, line 166  - at runtime obviously due to my hacks. Well I am missing this code block in my hand hacked things:

/* on some platforms these are not compile-time constants, so we must fill them at runtime */
+  NAInteger_Type.tp_base = &PyInt_Type;
+  NALogical_Type.tp_base = &PyInt_Type;
+  NAReal_Type.tp_base = &PyFloat_Type;
+  NACharacter_Type.tp_base = &PyString_Type;

Now to figure out where to put it and von Mises are in Python.

PS: So it is "Talk like a Pirate Day" so "R - ARRR in".
at

3 comments:

  1. ecollageJune 29, 2011 at 5:27 AM

    Could you do a post on how exactly you got Rpy2 installed on win7? Have been tearing my hair out for >2 days with no luck. Am trying with rpy2-2.2.1 on win7 32 bit, Python 2.7 and R 2.13.1.

    I replaced the missing Rinterface.h file into the R binary distro INCLUDE folder. I then run:

    > python setup.py build install

    …and it chokes at:

    …
    File “setup.py”, line 152, in finalize_options
    config += get_rconfig(r_home, about)
    File “setup.py”, line 312, in get_rconfig
    rc = RConfig.from_string(rconfig, allow_empty = allow_empty)
    File “setup.py”, line 272, in from_string
    + ‘\nin string\n’ + string)
    ValueError: Invalid substring

    in string

    I tried to run the command directly in a python shell, using:

    rexec = os.path.join(R_HOME, ‘bin’, ‘R’)
    cmd = ‘”‘ + rexec + ‘” CMD –cppflags’
    rp = os.popen(cmd)

    …and the response for all –ldflags’, ‘–cppflags’, ‘LAPACK_LIBS’, ‘BLAS_LIBS’ (called on line 150 of setup.py) are empty.

    HELP!!

    ReplyDelete
  2. ecollageJune 29, 2011 at 5:27 AM

    Could you do a post on how exactly you got Rpy2 installed on win7? Have been tearing my hair out for >2 days with no luck. Am trying with rpy2-2.2.1 on win7 32 bit, Python 2.7 and R 2.13.1. I moved the files across from \bin\i386 to \bin like you mention. I then run:

    > python setup.py build install

    …and it chokes at:

    …
    File “setup.py”, line 152, in finalize_options
    config += get_rconfig(r_home, about)
    File “setup.py”, line 312, in get_rconfig
    rc = RConfig.from_string(rconfig, allow_empty = allow_empty)
    File “setup.py”, line 272, in from_string
    + ‘\nin string\n’ + string)
    ValueError: Invalid substring

    in string

    I tried to run the command directly in a python shell, using:

    rexec = os.path.join(R_HOME, ‘bin’, ‘R’)
    cmd = ‘”‘ + rexec + ‘” CMD –cppflags’
    rp = os.popen(cmd)

    …and the response for all –ldflags’, ‘–cppflags’, ‘LAPACK_LIBS’, ‘BLAS_LIBS’ (called on line 150 of setup.py) are empty.

    HELP!!

    ReplyDelete
  3. Jiri PikAugust 22, 2011 at 10:38 AM

    These problems are easy. You can simply comment out the code for get_rconfig and replace them with


    config = RConfig()

    #
    # for about in ('--ldflags', '--cppflags',
    # 'LAPACK_LIBS', 'BLAS_LIBS'):
    # config += get_rconfig(r_home, about)

    print(config.__repr__())

    self.include_dirs.extend(['c:\\Progra~1\\R\\R-2.13.1\\include\\'])
    self.libraries.extend(['c:\\Progra~1\\R\\R-2.13.1\\library\\'])
    self.library_dirs.extend(['c:\\Progra~1\\R\\R-2.13.1\\library\\'])

    you will get further on, up to the C++ problems.

    I will post the procedure at
    http://blog.jiripik.com/archives/2011/08/22/how-to-get-rpy2-work-in-windows/

    once finished

    ReplyDelete