Saturday, April 4, 2020

Distributed Locking from Python while running on AWS

In the day and age of eventually consistent web-scale applications the concept of locking may seem very archaic. However in some instances attempting to obtain a lock and failing to do so within a limited window can prevent dogpile effects for expensive server side operations or prevent over-write of already executing long running tasks such as ETL processes.
I have used 3-basic approaches to create distributed locks on AWS with the help of built-in services and accessed them via Python which is what I build most of my sofware in.

File locks upgraded to EFS

File based locks in UNIX file-systems are very common. They are typically created using the flock command, avalaible in Python under os-specific flock API. Also checkout the platform independent filelock. This is well and good for a VM or single application instance. For distributed locking, we will need EFS as the filesystem on which these locks are held, Linux-Kernel and NFS will use byte-range locks to help simulate locally attached file system type locks. However if the client loses connectivity the NFS lock-state cannot be determined, better run that EFS with enough replicas to ensure connectivity.
File locking this way is very useful if we are using EFS for holding large file and processing data anyway.

Redis locks upgraded to ElastiCache

Another popular pattern for holding locks in Python is using Redis. This can be upgraded in the cloud-hosted scenario to Redis-Elasticache, This pairs well with the redis-lock library.
Using redis requires a bit of setup and is subject to similar network vagaries and EFS. It makes sense when using Redis already as an in-memory cache for accelration or as a broker/results mechanism for Celery. Having data encrypted at rest and transit may require running an Stunnel Proxy.

An AWS only Method - DynamoDB

A while ago AWS published an article for creating and holding locks on DynamoDB using a Java lock client. This client creates the lock and holds it live using heart-beats while the relevant code section executes. Since then it has been ported to Python and I am maintaining my own fork.
It works well and helps scale-out singleton processes run as Lambdas to multiple lambdas in a serverless fashion, with a given lambda quickly skipping over a task another lambda is holding a lock on. I have also used it on EC2 based stuff where I was already using DynamoDB for other purposes. This is possibly the easiest and cheapest method for achieving distributed locking. Locally testing this technique is also quite easy using local-dynamodb in a docker container.
Feel free to ping me other distributed locking solutions that work well on AWS and I will try them out.

Friday, January 24, 2020

Testing the OrangeCrab r0.1

After hassling Greg Davill for a while on twitter and admiring his OrangeCrab hardware I managed to catch up with him in person in Adelaide. I have been away in Nairobi till October last year, then I spent a brief few days in Adelaide before coming over to Canberra to take up a position in Geoscience Australia. The new gig is much less time commitment than the start-up world and hopefully will allow more time for blogging and board bring-ups like this one.

I caught up with Greg at a Japanese restaurant in Rundle mall and was treated to his now trademark led cube and led icosahedron. They are insanely detailed pieces of work and deserve staring at. However I am most grateful for the care package he left me, an OrangeCrab v0.1. This is an ECP5 board in feather form factor with an ADC built in to respect the Analog In pins on the feather. My aim for this board is to host some energy monitoring code on the FPGA with a fast 4mbps or so ADC and perform power/energy calculation on some parts of LUT's/DSP and have a softcpu push data out.
Greg also left me a home-made FTDI based board to use a JTAG programmer. The whole setup requires 3 USB cables:

  •  To plugin and power the FPGA board (eventually it should be alos programmable via this port) 
  • To attach a USB-Serial converter and watch the console when the gateware comes up 
  • To program the board over JTAG using an FTDI chip

Getting firmware compiled these days is getting easier, but Greg had done his initial testing with Lattice Diamond. I managed to installed it in WSL and promptly ran into a tonne of issues. The weirdest being close coupling to bash, Ubuntu actually uses dash as its default shell. You can get a Diamond licence and help support integration of diamond in litex-buildenv.
I was about to give up then Greg got it working with the opensource toolchain and [NextPNR-ECP5. I had by then setup litex-buildenv to support the orangecrab. So getting some gateware on was relatively easy. Then I got stuck on RAM timing bug in Litex till a few hours ago when I tested out some new gateware.

Also Checkout out Greg's foboot fork and help make programming over the USB possible and reduce 1 USB cable. More work on this including getting MicroWatt running coming soon.

Sunday, October 6, 2019

Around Kenya in 4 days and a year ( part 2)

The New Year adventure continued past the sad mercury laden mines of Migori to the Tanzania border , Isebania in particular. We met up with professor Sangai Mohochi and had a brief gander into Tanzania ( Serere) for a beer. Later we had some Orokore Beer made from millet, we sat around a car tyre sipping still fermenting beer from straws.

We took our leave from the professor and had a long drive out to Rusinga island. A beautiful place without much of the tourist trappings lakeside places have. The Suba culture is being revived with festivals. We woke up early and did a trip around Rusinga Island and visited Tom Mboya's Mausoleum. One of my high school's illustrious alumni. 

After another long day of driving and some fiddling with charging phones directly from a car battery ( another story on how many engineers does it take to charge a phone), we ended up in Naiberi River Campsite. It is more like a glamping spot and ideal for a quiet New Year's celebration around a fire in the main hall.

A late lunch/early dinner at Rift Valley Lodge Golf Club. We went for a post dinner walk on the greens and ran into herds of zebras and antelopes. No wonder this golf club is classed as one of the best in the Africa. All the excitement was followed by an uneventful drive back to Nairobi.

We arrived tired but energized the beauty and possibilities in Kenya. The roads need to be built, electricity needs to be channeled to houses, there is a lot to do. We look forward to hanging around and getting it done. Eric is heading back to his PhD at Harvard, Gichini will be doing something about the weather prediction and I have helped put Lori Systems on a solid technical footing and change the Logistics landscape.

Wednesday, August 14, 2019

Open-source Sustainability (The tale of 2 package managers)

Last weekend I had the priviledge to attend the 10th PyconAU and listen to some amazing speakers. I went with my I will write markdown on the fly and make a blog-post at the end of the day mindset. Even though I did write a lot of markdown on the fly, I haven't gathered the courage to push these unedited notes into a public post. Excellent examples of live-blogging from conferences here.
What did happen was that the niggling doubts I had around how open-source works in the real world outside of just the code crystallized. This was as a result of 2 very good talks , one about how the PyPi project works and another around Open-source sustainability beyond money.

For the last year I have been writing and reviewing a lot of React Frontend, Python backend (Flask/Django) and Notebooks code. Both frameworks are super easy to buy batteries for where the included ones are running out of juice. Simply via pip install and npm install you can climb onto the shoulders of giants who are library maintainers and the life-blood of lean start-ups everywhere. However the maintainer burnout is a thing and start-ups when building their stack should be highly cognizant of this. Package repository burnout is also a thing. In my time in the software industry I have seen Maven repositories disappear. More recently NPM go through an identity crisis and the left-pad incident.

The PyPi talk gave me great background on a tool I use every single-hour without too much thought. It takes some dedicated volunteers to keep the dream alive. Who according to Dustin Ingram are :

  • Unemployed and bored and poor (but super talented) 
  • Paid for by their employer (thanks employers who support FOSS) 
  • Not getting enough sleep (or in my case time with the family)

Vicky's talk covers another aspect. Developers and maintainers need more than money to keep going, they need back-up. The community need insurance against the bus-factor and burn-out. I have been guilty of this myself, putting a few dollars behind features I would like to see in BountySource instead of diving in. This has become more so as I have progressed in my career and become increasingly time-poor. Talking about this would anyone at VSCode like to claim the few dollars we put here ?

I love the longevity and discipline of project warehouse and will find some time to contribute to it. I also look forward to a similar alternative to npm, rather than a caching proxy with community behind it.

Wednesday, June 26, 2019

Sunbeam Coffee Maker Teardown

The bad Kenyan grid has struck again. There is very little built in surge protection in the system, so all appliances plugged to the wall end up with in-house surge protectors. This time the victim was my beloved coffee machine which had served us well for more than a year.

So I decided to take it home and spend some quality time with Pascal tearing it down. He is always curious about stuff and what is inside, so the Coffee machine is a complex beast capable of keeping him and myself occupied for hours on end. First we took of the top and looked at the heater and the electromechanical assembly controlling the hot-water head and frother was exposed. Also was exposed the heating tank with dual heating elements and a thermistor (I am assuming a thermistor rather than a thermocouple).

When we took the back plate off we found the main controller board. There is to obvious transformer in the set-up so I assume to whole power supply is cap-drop. There is a giant heat-sink attached to a 3-pin device which I assume is a triac or SCR for controlling the heating elements.

The bottom-left corner has a water pump with its own switching mechanism driven by relatively thin signal-wires from the main board.

There are some more interesting details on the main processor board including thick high current traces driven by the silicon switch with exposed copper and extra solder on top to increase thickness and provide a lower current path.

The main CPU is TQFP packaged micro-controller which is covered by the conformal coating the entire PCB is covered in. A macro image on twitter helped establish the lineage of the processor. It is a chinese 8051 variant with the datasheet here. Any help in translating it will be highly appreciated. But most of the schematic is legible.

Next up will be powering on the processor with a suitable 3.3v supply and examining the analog and digital buses. As well as powering on the whole thing to see why it turns of immediately on start. If the machine is unfixable I am planning to scavenge the thermistors, heating elements etc. to build a reflow mechanism.

All the water circulation mechanism is really interesting as well and can possibly be shoe-horned into a water cooler for a 3D printing system.

Saturday, June 22, 2019

Nairobi 2nd OSHW meetup

After a many month hiatus we finally put together the 2nd Nairobi Open-source hardware meetup. This time we were at the Tav Irish Pub near Nairobi Garage on Waiyaki way. The pub was chosen due to proximility to Nairobi garage and possible use of that venue. As it turned out none of us were memebers paying the $60 per-month fee so we ended up co-opting the pub in Chicago 3H fashion.
PCB Lineup
PCB Lineup
I had collected PCB's over the last few month from Aisler, OSHPark and PCBWay all shipped to Kenya from Germany, US and China respectively. So we lined up to try various soldering techniques on 0603 Jellybeans and some 20pin TSSOP IC's.
Soldering demo
Soldering Demo
The techniques we we tried were the tack, finish and reflow one using the iron and an experimental one with paste and dry-iron I had picked up recently. I didn't have a temperature profile controller for the dry-iron method. So we were just going to wing it by the eye.
Part Placement
Part Placement
We pasted up the boards using screwdrivers and tweezers. Started the reflow till the parts settled. Then simply turned off the iron. I am sourcing a K-Type thermocouple and SSR from KTechnics to get a better handle on this for next time. However the dry-iron does reach sufficient temperature for leaded solder reflow.
Dry iron reflow
Dry iron reflow
By the end of the session we had assembled x2 CS5464 Breakouts. Next step from here will be suitable MicroPython and Arduino drivers to ensure these things actually work. We had a lot of fun over the meetup and I hope to continue having these sessions. Next time one of the participants has promised to bring his Weighbridge Automation and IoT set-up.
Scavenging Coffee Machine
Scavenging a Coffee Machine
Following the meetup I have been hunting for accessories to make the dry-iron reflow less of a fire hazard and more controllable. (Un)fortunately the coffee machine at work has been fried by the spikes prevalent in the grid in Kenya (especially during the rainy season), so I have a spare K-type thermocouple I can work into the build. I have also co-opted a tissue holder we had on our dining table to stabilize the inverted iron during reflow.
A stand of opportunity
Stand of opportunity
Onwards to the next Meetup.

KiCon 2019 - Chicago in a weekend

Getting away from a fast paced start-up to attend conferences is a hard ask. However I managed to get away for a few days to attend the inaugural KiCad conference in Chicago a few weeks ago.
The pace has been such that only after about a month I am getting a chance to write it all up. The first night I managed to attend the Hardware Happy Hour at the Ballast Point. I got to see some amazing stuff including pneumatics for an autoplaying piano using layers of PCB.
The huge selection of beers and the chicago dogs were definitely unexpected bonuses.
Tour of MHub assembly area with pick and place machine and large Molex sign. Also got to see Chris Gammell's desk and pick up some Amazon and Mouser shopping to bring back to Nairobi.
I could definitely use the pick-n-place machine in my house and I hope Chris makes use of it whenever he can.
I also got to speak to some companies I have worked with remotely but not met the principles and representatives in person, including CrowdSupply, Aisler, SnapEDA, Hologram and of course DigiKey. It was great to speak to the KiCAD dev team as well, perhaps I can contribute on the MSVC windows native build. I also managed to score credits freebies from all the sources.
Here are pure notes to self on various topics that I picked up on 2-days of super varied and dense conferencing. Some are commands I would like to run some, are techniques I learnt and some are pure stories I picked up.

Schematics are a drug, use code instead

  • pip install skidl

Auto-routers are evil but useful

  • Triangles (Topological/ Using TIN mesh)
  • Maze solving
  • Shape based ( Rectangles)
  • Channel Based
  • FreeRouting
  • Adaptive heuristics using deep learning
  • Linear increase in PCB complexity

High-frequency simulation (in GhZ range) for PCB's

  • OpenEMS ( MEEP)
  • Wilkinson filter
  • Important in Radar Design

Prototyping in a few hours

  • Isolation milling
  • Midwest Circuit Technologies
  • Bantam tools. Binary save.
  • 1/32" end mill 0.8mm tool
  • 10 mill trace

Rendering photorealistic PCB's

  • Render layers to image texture
  • Use Blender cycles based render with bump maps for silk-screen and traces

Contributing to KiCAD

  • Development in LaunchPad + Github
  • Starter bugs are available

More HF PCB's

  • 30MHz to 3Ghz, with narrow bins
  • 5.6GSPS digitizer
  • Tuned lines. Right angles create problems.
  • Use footprint, Arc or Trace around edge cuts.
I woke up pretty early on Saturday morning and walked around and almost hopped on a fire-engine tour. Always wanted to do that. Then I checked the radar, another thing missing in Nairobi, and a huge snowstorm was inbound. Snowstorms at this time in Chicago are atypical, but hey everyone stays in for the coference. By the time the conference finished and I headed home from the after-party everything was covered in snow.
The last day (Sunday) was spent taking an Uber Pool , a novelty for me, to a Vietnamese grocery and spending around 2 hours getting additional exotic food-supplies to bring back to Nairobi
I saw a Harvey milk tail fin and took to the skies back to hop across the Atlantic and Mediterrnean to Nairobi. Flying 40 hrs to conference for 48 hours and shop for 6 is a real pain. Will have to bring the KiCAD movement back home to Nairobi and pour more effort into the local hardware scene.
So long Chicago, see you next conference.