Sunday, September 16, 2018

PyconAU 2018 - Keep Notes

This is a raw dump of notes I took on keep while attending PyconAU. Each to be fully expanded to an article on its own. Suitably redacted to keep my intrusive thoughts to myself. 

Documentation and Technical Writing
  • Persuasive, Referential, Literary, Expressive. Writing pull requests. Foucault on discourse.
  • Think goals, Think audience, Write (structure, conventions). Active vs Passive (active is shorter). Bullets.
  • Plain English, easy words. Edit. Find good writing.
API Design
  • Forestry Data. User interface for developers. Access control , pagination. API versioning. Break in controlled fashion. Deprecation schedule.
  • Accept versioning header. URL Path versions. Less emotional versions. Small changes version transformer. Django middleware.
  • Auth- Token. OAuth.
  • Standardized (familiarity, avoid pitfalls, standards win) - jsonapi.org (next,.previous), graphql, Basecamp. Swagger.
  • Larger data sets out of band
  • Documentation. Selling page. Dev account, demo key, root url. Encodings, formats
  • OpenAPI, Swagger, tools, docs/code, reduce friction, automation.
Working with Spatial data in Python
  • Routing. Isochrones, Latte line.
  • Db spatial extension. Point in polygon.
  • Basemaps, JS library.
  • Geomesa.
Multi-lingual Python
  • i18n, l10n, language_code, locale, po (files translation). Language session choice flow. Url-conf.
  • trans and trans block tag. Every file must have the i18n tag. Use lazy in models.py.
  • makemessages in all depencies. Zh_cn.
  • Model fields to enter data. Keep tables in _zh_cn. Transifix.
Web without Javascript
  • Intercooler.js. CSOF… use web 1.0 submissions in a 2.0 way.
  • Large data. Out of-band.
Database optimization
  • Write amplification. Partial index. Hash, GIN(generalised inverse index) , BRIN (min/max)
Design for non-designers
  • Choose few fronts. Use different spacing. Curated fonts. Beautiful Web Type. Fontpair.co . Google Fonts. Making things clean (whitespace). Data analytics. Less is more. Unsplash, Photopin, Fiverr. Importance of imitation.
Designing environments with Docker
  • Front end builder. Task queue/Celery. Postgres. Docker compose .
Hypothesis to replace auto-manual tests
  • Hypothesis for testing by schema. Always true unless there is a bug. Fuzz the code with variety of input. Faker. Django integration push model and clean up. Quick check.
  • Generators.
  • Use lib2to3 parse through Python code and refactor / reformat. Bowler, safe refactoring for Python. Reuse refactoring code.
Tom Eastman - Keynote
  • koordinates, geospatial data. Effective learning for programmers. Effortful retrieval.
  • Fixed vs Growth mindset.
Lightning talks
  • DRF model pusher. Ably, Django Channels.
  • Claire Krause, GA satellite imagery analysis
  • Wrapt, allows introspection of wrapped functions. Context manager.
Many Python Web Frameworks
  • Cherrypy, Class based views. Falcon API first framework , fast, returns json for microservices. Hug with syntactic sugar. CCBV. Tornado, Async. Sanic.
Women in Python
  • Jason Fagon.
  • Katherine Johnson.
  • Carol Willing (Jupyter)
Celery and alternatives
  • Celery message formats, task chaining. Disable pre-fetch, long running. Own task queue. RQ (redis queue). Huey. TaskTiger(distributed locking). Dask (pydata), scheduler distributes pandas and numpy dataframes. Task chaining, celery import chain. RQ failed queue.
Trouble with sensor data
  • Kafka stream processing. React and Socket.io. HTML5 gyroscope API.
Pub conversations
  • RNN (Fast.ai), pre-trained model zoo.

Wednesday, August 29, 2018

Using AWS Athena in Anger

The journey down this particular rabbit hole started with this challenge "Create Read-replicas for MSSQL 2017 database hosted in RDS". Hosting services on AWS typically reduces IT load and makes focus on product development easier, however there are certain edge cases where AWS falls down and requires special intervention.

The above challenge turned out to be one of them. RDS supports seamless read-replicas on a lot of other types of databases , however MSSQL is not one of them. The recommended approach for creating read-replicas for MSSQL is to enable CDC and do so using DMS (Database Migration Service). This is similar to what the RDS does internally for multi-AZ redundancy. We hit another edge-case here, DMS replication support with CDC does not extend to the 2017 version.

We did manage to run DMS successfully from MSSQL 2017 to S3 as sort of a disaster recovery measure. Then I was stuck with the conundrum of what to do with the CSV table dumps this got me, at the AWS Summit in Cape Town one of the solution specialists introduced me to Athena and the adventure began.

Athena is essentially a managed Apache Hive based map-reduce setup which can churn through webserver logs or in this case CSV's to give me back the database I was replicating. The replica is not perfect, it is missing stored procedures and views. These need to be redone in HiveQL.

All tables can be mapped using corresponding schema and DDL to Athena tables, thus replicating fully the database without foreign keys and other niceties. AWS now has cross region S3 replication , this coupled with Athena can give a very Rube-Goldberg cross-region database replication.

Athena is also particularly useful with scrapers where data is being collected from 3rd-party sites and consolidated into S3 tables. It can seamlessly blast through millions of rows of scraped data for goodies in seconds.

This post is a WIP and will be enriched with Athena goodies over time.

Sunday, July 15, 2018

Preserving pins, auto-transmit RS485 with Lora32_Wifi

This is a story of the always helpful NE555 (in this case its 3.3v cousin the TLC555). I have been designing more and more complex DIN rail energy monitors. The last one is 3-unit DIN rail oeuvre with the following features:
  • Lora, Wifi and Bluetooth thanks to Lora32_Wifi
  • 128x64 SSD1306 Oled screen
  • Three ATM90E36 3-phase energy monitor IC's
  • 3.3V differential line driver for RS485 bus driver
  • Some GPIO's for triggering relays or receiving inputs.
With all these features consuming pins on the ESP32 module I eventually ran out of pins to attach to the RTS line for the RS485 line driver. I could have added many more pins by using an I2C ioexpander and solved my issues, however the RS485 bus on this thing is half-duplex, so there is a much easier way to do this which does not involve any code.

The trick is to use the NE555 in one shot mode, trigger it for a transmit, then let it fall-back to receive after a set period of time. The NE555 is useful in so many ways.

Thursday, December 14, 2017

Sense Energy monitor teardown - sampling in MHz

Recently I obtained a Sense Energy monitor via US from Margaret of BitKnitting. She is doing a very interesting neighbourhood energy efficiency project. As usual I could not contain my curiosity and opened it up to have a look. I will start off with an analogy - the closest bit of open-source kit that I have to do half the amount of analog functions as the Sense is the PRUDAQ on the BeagleBone wifi as pictured below.
Beaglebone Wireless + PRUDAQ
The case is very well built from high density plastic, however it is clipped together neatly and easy to take off without permanent damage. The wifi antenna is screwed on with a neat water sealed connector. The power/voltage and current sampler connectors are Molex Microfit and Nanofit respectively. I rather liked the Molex Nanofit and adopted it for my own DIN Rail sampler board.
Enclosure and main boards
Inside this very nice exterior are a couple of very dense sandwiched PCB's. One is Green, almost a generic power supply board, but with additional function to sample the split phase AC in the North American grid. The other is a black board where the magic happens. It is essentially a beaglebone wireless with a custom energy monitor system put together using high-speed ADC's and a CPLD
Green Power board, Black Digital+Analog board

Main PMIC, same one is used on the BeagleBone Black
The image below shows the subsection that performs energy measurements, it is composed of x2 high sample rate ADC's (14bit 2MSPS) and an Altera Max V 64pin CPLD. I assume the cells inside the CPLD perform the reading from the ADC's and multiplication/accumulation to compute energy usage, then transmit computation results to the TI CPU.

Custom energy monitor with high-speed (2MSPS) ADC's and Altera CPLD

Wifi Module
Switcher IC's on power supply board

The main CPU is a Linux compatible AM3352 CPU from TI. It does not have a PRU co-processor like the one used on the BeagleBone, hence the need for a CPLD to perform some of the real-time computation required.
TI Arm CPU and RAM/Flash

Overall the Sense energy monitor hardware is a great design, though a bit specific to the USA market. I am looking forward to a release in the Australian market. The purchase to study one was certainly worth it. Now off to install my own linux build on it and send the data elsewhere.

Thursday, October 12, 2017

Simulating 3-phase AC for Energy Monitor testing

Finding 3-phase is difficult, convincing the owner of the said supply to test some home made hardware is even more so. After building a 3-phase energy monitor my testing options for it appeared very limited. So I set about making my own low-cost 3-phase energy monitor calibration system. Here are some build photos.
Initial testing with DSO scope
Arduino code and Sine-wave table
The main protagonists of this build are the Teensy 3.2 and MCP4728 4-channel I2C DAC. The Teensy pushes an integer valued sine-wave array into the DAC which is then RC-filtered and amplified via some opamps.

With target energy monitor
Eventually I worked up a budget to buy myself a 4-channel Rigol scope from Emona. This showed the nice equal amplitude and properly phased 50-Hz waveform the DAC was producing.


3-phase voltages on oscilloscope
There is more of a backstory and details about this project in the DIYODE Magazine article. I have tested this a fair bit now as well with ATM90E36 Energy Monitor IC and will look at adding power opamps and transferring to a PCB. Below is a video of a 3-phase energy monitor test rig using synthesized waveform vs 12V AC-AC transformer.

Sunday, August 13, 2017

PyconAu 2017 - Melbourne

I recently attended my first PyconAu - a cauldron of programming goodness. It was held in cloudy and grey Melbourne at the convention center. The exact same venue I attended the Nvidia conference a while ago. It was three days of information flow and meeting very interesting people. I jotted down some quick notes in my Keep and I will flesh them out here for my own reference.

These were the sessions where I had more detailed notes and some photos:

GnuRadio Python bindings and simulator. SoapySDR. HackRF simplex hardware including a freaky demo of using playback attack to unlock cars by copying car remotes.
An update on the state of MicroPython. A live demo with I2C read from gyroscope controlling a servo. New stuff like help('modules'), upip install, VFS , assembler, Zephyr RTOS etc. Really got into this one. The idea of having line by line firmware building convenience and a REPL on an embedded device is oh so appealing.
The Murchison Widefield Array(MWA) and sampling the universe at 655MSPS - Rocket IO to Xilinix. FPGA Fourier variants and finaly 24 of 256 spectral channels trickling back to Perth at 10Gbps. The processing electronics is double Faraday caged to prevent RF feeding back into the telescope.
The messaging of data from IoT devices while minimizing data usage and allowing convenience and transfer of state. Various contenders are JSON, Avro, Protobuf and MQTT over Websockets or simple HTTP, with advantages in HTTP v2 due to header compression via HPACK.

Presentation featuring GUI's for IoT devices and a glowing orb seen in a famous Trump photograph. With interesting details about Progressive Web Apps and Javascript based IoT controls.
Another very very impressive demo came from Russell showing Python running on 6 different platforms on a few minutes. After that I should really checkout our BeeWare. Will have to write a full length article on it as well with a proper usage tutorial to earn these great looking challenge coins.

During the breaks people were leaving notes seeking jobs, offering jobs or setting a topic for a lightning talk. The looking for employees board was slightly daunting but I put up a brave little ad featuring Aerometrex anyway. I also gave a brave little talk about my LSTM and energy monitoring time series adventures.
The last day had this fun little guy controlled via Python and a Jupyter Notebook. On my list of toys to get to play with my son.
I also enjoyed the end of the day talk from Xavier on scientific data cleaning using Pandas was great as well. I quickly put it to use to clean my Tindie buyers list to send out a MailChimp Promo, not very scientific, but useful when you are running a crowd funding campaign. After that it was a couple of days of sprints and getting ATM90E26 to play nice with ESP32 and Micropython, that is another blog post altogether.

Thursday, June 29, 2017

3G Energy Monitor - WattWatchers teardown

Tear-downs are always done in a spirit of learning and admiration. Having built some hobby electronics I really appreciate what goes into laying out a mixed signal PCB as well as one that has high and low-voltages on the same board and is subject to regulatory compliance both for electrical characteristics and RF emissions.

I was handed some detailed WattWatchers images by a client and I did some mark-up of what goes into them based on previous experience of how energy monitors are put together. It is a 2-board design with an SMD PCI connector joining the boards. The PCI connector passes the power and current-voltage signal up to a vertical board.

I shall call the bottom board power and sampling  board and the side-board metering, CPU and network board. The first image below shows one side of both boards. The power supply board features a hefty cap-drop supply with assortment of fuses and TVS, as well as the aforementioned PCI connector. The vertical board features a large 3G module and associated Telstra sim slot, I assume the SIM is linked to a custom M2M plan which costs a few dollars a month.

The other side of the vertical board features the most interesting goodies. On the left half it has a low-noise / high power voltage regulator from ST (presumably needed to feed the hungry 3G modules), as well as a low-power MSP430 MCU with a couple of large old-school tantalum caps. The right side of this board (separated by a row of what is most likely transient suppressors or isolation amplifiers) are a couple of Analog devices 3-phase energy monitor IC's. The isolation section indicates the presence of 9 analog channels which corresponds to 3-phases and 6-CT's. The 2 ADE7880's share a single clock crystal presumably buffered. There are also a bunch of passives to perform 50Hz filtering.

The other side of the bottom board also featured in this image contains the voltage divider to sample the AC-phases and some current-multiplier/burden resistors to convert the current measured by the CT's into a voltage value. There is also another LDO on this board to presumably power one side of the isolation amplifiers, nothing too high power. This LDO must be at the end of the cap-drop system so the neutral or phase on it is at high voltage and generous dob of insulating snot is providing extra protection. The insulating snot can also be seen at the top-left corner of this board where the phase and neutral(?) or 2 phases are too close together for comfort/compliance.

WattWatchers has done a great job packing a lot of features into an M2 DIN rail enclosure without resorting to many off the shelf modules and pre-built components except the 3G module. I am hoping to re-use some of the concepts here in my ATM90E36 based design.