Redhatter (VK4MSL)

Solar Cluster: Debugging an ATTiny24A, programmer upgrade

So, debugging the ATTiny24A, one big problem I’ve got is understanding what the ADC is seeing in each channel. There’s no serial output, no LCD, just a handful of LEDs and a PWM output. Not good enough.

The ICSP header though, necessarily exposes the pins needed to do SPI and I²C. Could that do? I’d need something to do the transfers with.

The programmer I’ve used to date has been a Olimex STK500v2 clone (the tiny one built into a DB25 backshell), which works well, but it has one nit: I haven’t figured out a way to do raw SPI transfers with it. It might be possible, I’m not sure.

I immediately thought of the Raspberry Pi. The other option I had close on hand was a Freetronics LeoStick. One I’d have to write programming firmware for — which may be worth doing some day. The other, I can just install from repositories. But how does one interface the two?

Adafruit have this tutorial on doing exactly that. HOWEVER, they wire the Pi straight up to the AVR. Fine if they’re both 3.3V, but trouble if the AVR is running at 5V like mine. I’d expect this to release magic smoke!

So, a level shifter is needed. I happened to have a Freetronics one laying around which gave me 4 channels, good enough. I just had to figure out what pins to use. For reasons unexplained, Adafruit seem to pick weird and wonderful pins that are not close together. Another guide, suggested using the standard SPI pins. I more or less went this route, but used GPIO channel 22 instead for reset, so I could use the one female header to connect to them all.

The connector was a spare that came with the LeoStick: they come with two 13-pin ones. I cut it with a hacksaw to give me two 3-pin headers and a 6-pin header. The 3-pin headers were glued together to give me a 2×3 pin header, and the other was soldered to the level converter. Two pins had to be swapped, annoyingly, but otherwise wiring it up was straightforward.

I just ran some off-cut CAT5e cable to the ICSP connector, keeping the lead length short so as to prevent clock skew.

The configuration file for AVRDude looks like this:

# Linux GPIO configuration for avrdude.
# Change the lines below to the GPIO pins connected to the AVR.
programmer
  id    = "pi";
  desc  = "Use the Linux sysfs interface to bitbang GPIO lines";
  type  = "linuxgpio";
  reset = 22;
  sck   = 11;
  mosi  = 10;
  miso  = 9;
;

I can flash my ATTiny24A from the Pi now with the following command:

$ sudo avrdude -p t24 -c pi …arguments…

So with that done, I should be able to use a simple Python script to read and write bytes via bit-banged SPI via the ICSP header, and implement some firmware to react via SPI.

Solar Cluster: More MOSFET fun

So I’ve managed to get the board up and going, sort-of. I’m developing the firmware, getting acquainted with the ATTiny24A’s hardware.

The logic is that it’ll be sensing the battery voltage and two inputs from mains and solar, and so when it goes into charge mode, it picks one of the two sources and starts pumping current. Simple enough.

Except testing it has proven “fun”. I hooked everything up, using a power supply with a diode to stand in for the battery. I noticed the ADCs were seeing a voltage on their inputs. How? Why? Of course, the answer was plain to see in the datasheet if I bothered to look!

That little body diode, was of course, passing the current from my “battery” back to the outside world, and that was messing with measurements.

Great. So I’ll be needing a series diode to cram in there somewhere, and the MOSFET is expected to switch up to 30A, so the diode needs to handle that too. The challenge, is there isn’t much room for a heatsink.

Actually, the MOSFETs can do over 70A, so I’ll aim for a diode that can do about 60A, with a view that it won’t be stressed doing 30A even without the heatsink. The Vishay VS-60EPU02PBF is looking like a good option, although expensive.

One annoyance is there doesn’t seem to be a diode that has the cathode connected to the tab of a TO-220, as then I’d just solder the MOSFETs and diodes back-to-back and clamp a heatsink to the pair of them.

I guess for now I can try a few experiments to get acquainted with how it’ll all work, perhaps de-solder the tabs of the MOSFETs (again) and perhaps put a small 3A diode in as a stand-in for testing so I can at least get the firmware written.

Solar Cluster: Lessons learned

  1. Don’t rely on the internal pull-up in the MCU for the nRESET pin. It might work for no-connect scenarios, but it’s not going to win a war against a Olimex programmer dongle that’s decided to lean on the pin a bit too hard.
  2. The linker needs to know what MCU it is too.

I basically found I had the ATTiny24A resetting repeatedly when the programmer was connected. Set a LED to stay on, it’d blink. The cause was the programmer was interacting with the MCU via the reset pin when connected. The solution was a resistor between 5V and reset, I stuck a 47k across the relevant pins of the ICSP header. (Annoyingly, they’re at opposite corners!)

The other blooper was having me scratching my head every time I tried to define an interrupt, the MCU would just sit there. Even if the ISR had nothing in it, and sei was never called, it’d still sit there dumb.

My Makefile at this point looked like this:

CFLAGS = -Os -g -mmcu=attiny24a -Wall -Werror
CPPFLAGS = -DF_CPU=1000000UL
CROSS_COMPILE ?= avr-
CC = $(CROSS_COMPILE)gcc
OBJCOPY = $(CROSS_COMPILE)objcopy
OBJDUMP = $(CROSS_COMPILE)objdump
SIZE = $(CROSS_COMPILE)size
PROG_ARGS ?=-c stk500v2 -P /dev/ttyACM0
PROG_DEV ?= t24

.PHONY: clean all

all: powerctl.ihex

clean:
	-rm *.ihex *.elf *.o

%.ihex: %.elf
	$(OBJCOPY) -j .text -j .data -O ihex $^ $@

%.elf:
	$(CC) -o $@ $^
	$(SIZE) -d $@
	$(OBJDUMP) -xdS $@

powerctl.elf: powerctl.o
test.elf: test.o

%.pgm: %.ihex
	avrdude $(PROG_ARGS) -p $(PROG_DEV) -U flash:w:$^:i

The call to objdump was to try and figure out what was going wrong. Note that I don’t pass -mmcu to the final link. The linker needs to know what MCU it is just as much as the compiler does. That was my error. Having done this, I now have working interrupts.

Solar Cluster: Charge Controller description

So, I’ve built the controller. The design was pretty simple. Using an ATTiny24A, I’d monitor the voltages of the battery and two power inputs, and code would decide which input to use, if any. It also could use the in-built temperature sensor to control cooling fans. This is the schematic I knocked up this morning.

The values of most resistors are not critical. I found I needed 1kOhm resistors into the bases of the transistors as the MCU was not happy driving them directly. The transistors I’m using are BC547Bs controlling AUIRF4905 MOSFETs.

The only components that are critical are the voltage dividers on the ADC inputs. I’ll be using the built-in 1.1V reference in the MCU as that’s what’s needed for the temperature sensor anyway.

This was a bit of an exercise in reviving old brain cells as it’s been some time since I’ve done a proper PCB myself. This is a one-off prototype with mostly larger components, so no point in getting boards fabricated. I did it the old fashioned way, using a dalo pen then etching in a bath of Ferric Chloride.

That gives you an idea of what the board looked like prior to population. The underside was covered with tape to prevent it from being etched. It took a while, and I think I could have upped the concentration of the solution a bit, since it did leave some tracks un-etched.

Perhaps my solution is getting a little old too… the logo on the bottle really dates it. I found I had to attack the gaps between some tracks with a knife since the etchant didn’t quite get it all.

There are no tracks on the bottom, it’s just one piece of un-etched copper, to act as a ground plane. I guess the construction style is a cross between Manhattan and groundplane (dead-bug) construction. The constructed board looks like this.

I’m not sure what all the LEDs will be doing at this point. Three share pins with the ICSP header, which means they flash as the board is being programmed… useful for troubleshooting ICSP issues. The IC socket is a cheap 14-pin one, I just bent the pins to mount it flush to the board. The 10uF tantalum on the output of the 5V PSU is possibly a 10V one. Where the electrolytic is, is where I had the 330uF tantalum mounted, and it went bang when I gave it 12V.

I tried the following program on the board which just steps through all the LEDs and MOSFETs:

/* board.h */
/* LEDs */
#define LED_U1_BIT		(1 << 7)
#define LED_MOSI_BIT		(1 << 6)
#define LED_MISO_BIT		(1 << 5)
#define LED_SCK_BIT		(1 << 4)
#define LED_U0_BIT		(1 << 3)
#define LED_PORT		PORTA
/* MOSFETs */
#define FET_MAINS		(1 << 0)
#define FET_SOLAR		(1 << 1)
#define FET_FAN			(1 << 2)
#define FET_PORT		PORTB
/* test.c */
#include <avr/interrupt.h>
#include <util/delay.h>
#include <stdint.h>
#include "board.h"
uint8_t heartbeat = 10;
int main(void) {
	DDRA = LED_U1_BIT | LED_MOSI_BIT | LED_MISO_BIT
		| LED_SCK_BIT | LED_U0_BIT;
	DDRB = FET_MAINS | FET_SOLAR | FET_FAN;
	PORTA = 0;
	PORTB = 0;
	/* Test sequence */
	while (1) {
		PORTA = LED_U0_BIT;	_delay_ms(1000);
		PORTA = LED_U1_BIT;	_delay_ms(1000);
		PORTA = LED_MOSI_BIT;	_delay_ms(1000);
		PORTA = LED_MISO_BIT;	_delay_ms(1000);
		PORTA = LED_SCK_BIT;	_delay_ms(1000);
		PORTA = 0;
		PORTB = FET_MAINS;	_delay_ms(1000);
		PORTB = FET_SOLAR;	_delay_ms(1000);
		PORTB = FET_FAN;	_delay_ms(1000);
		PORTB = 0;
	}
	return 0;
}

That seems to prove the hardware is alive, and now I just have to get the software working. Now to try out the toolchain I built!

 

Solar Cluster: Working around build errors in crossdev

I’ve been doing further work on my charge controller. Last weekend, I managed to fix the issues that were causing the MOSFETs to not work, and the “faulty” MOSFET turned out to be fine: the fault was a glitch with my home-fabricated PCB. (No, commercial PCB makers, this is not an invitation for you to advertise as you have done on other projects! The job is done.)

In the midst of doing this, I was also having problems getting the toolchain to generate code correctly, the code would fail to run if I had an interrupt service routine defined, even if the interrupts were not enabled it still failed to do anything. Something in the interrupt vector table was off.

So back to crossdev to see if I can build a newer toolchain that works. crossdev -t avr would get as far as building the full C/C++ compiler, then crap out with a missing ldscript. (I don’t have the log handy to show you the exact error, but it takes place in a ./configure script, so you see “C compiler cannot generate executables” or some such like that, and will see the ldscript error in the offending config.log.)

The kludge?

# cd /usr/avr/lib
# ln -s ../../lib64/binutils/avr/${BINUTILS_VERSION}/ldscripts/

Substitute ${BINUTILS_VERSION} with the active version. Voila, having done that kludge, I now have this:

$ avr-gcc --version
avr-gcc (Gentoo 5.3.0 p1.1, pie-0.6.5) 5.3.0
Copyright (C) 2015 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Not sure if it works or not, but I’m documenting the above as a note to myself next time I hit this error.

Solar Cluster: ATTiny24A charge controller: Fail so far

Well, last weekend I had finally acquired the bits after some delays getting everything together. This included some ATTiny24As, some P-channel MOSFETs and some 5V DC-DC PSUs.

Last weekend I got around to building the PCB, and yes, I’m not as handy with a dalo pen as I used to be. That, and the ferric chloride I was using had seen better days (the bottle has “Dick Smith Electronics” printed on it — enough said).

So I spent much time last weekend finding shorts between tracks and attacking those with a sharp knife. Last Sunday I managed to get something built, but not tested.

Today, I got around to testing it. At first I plugged it into a bank of 6 AA cells, and got 0.8V across the power input. WTF? Okay, maybe the DC-DC converter needs a little more current. So I wire up a cable loom with a 5A blade fuse and 30A Andersen connector. Plug everything together: *BOOM*, a tantalum capacitor blows up!

The tantalum was a 330µF that was scavenged from an old computer motherboard, probably only rated for 10V, and when you over-voltage a tantalum, they do throw a tantrum!

Lesson learned: don’t use those scavenged parts for 12V! I swap that out for an electrolytic (rated at 16V).

The reason for my high voltage drop though? A faulty MOSFET. I found that by de-soldering the tab on both input MOSFETs until the problem disappeared. Then pressing down the faulty one caused the short to re-appear. Strange, as the MOSFET has never seen use.

I proceeded minus a MOSFET for a bit, see if I could get some code into the MCU. I was able to program that okay, but then fun came when I tried to use the timer interrupt: including the timer ISR would cause the MCU to not boot. It’d just sit there.

The problem disappeared if I compiled my code without optimisation, or at -O1, but would return at -O2 or -Os (I was using -Os). So something in my toolchain was broken for the ATTiny24A.

Whilst waiting for a toolchain re-build, I decided to tackle the faulty MOSFET. I had one spare, so I carefully soldered that into place, only for the original issue to re-appear, so now I’m completely lost.

I guess next weekend, I’ll take a closer look and the cause will become obvious, but right now I’m more confused than a moth in a light shop!

Improved Helmets: Case Study: Phillip Hughes

It appears that Cricket Australia might release the findings of a safety review into the death of Phillip Hughes. This could be an interesting report to read when it comes out, in that it hints at what parts of the head are particularly vulnerable to hits from hard objects.

If you consider a cricket helmet for a moment, and contrast that to the coverage area of a bicycle helmet. Both are designed with the assumption of a collision with an object coming from in front, and so the back of the neck was not an area given much thought.

Motorcycle helmets do a lot better here in almost all cases except the “half helmets”: tick-a-box helmets if ever I saw one! More recent designs come down much lower on the head. Interestingly, professional cricket did originally start with motorcycle helmets, but found them too heavy.

Within the construction of even a motorcycle helmet however, it’s interesting to note where the foam exists: and it’s thickest around the top of the head. The sides where one’s ears would be, the foam gets thinner. Possibly a compromise for acoustic reasons, but is it significant?

Bicycle helmets however, with the exception of MTB type ones, don’t seem to cover this very well from what I’ve seen. Perhaps the back and sides needs a bit more attention. I guess I’ll have to have a gander at the report when it is released.

Solar cluster: Software stack beginning to take shape.

So, after putting aside the charge controller for now, I’ve taken some time to see if I can get the software side of things into shape.

In the midst of my development, I found a small wiring fault that was responsible for blowing a couple of fuses. A small nick in the sheath of the positive wire in a power cable was letting the crimp part of a DC barrel connector contact +12V. A tweak of that crimp and things are back to normal. I’ve swapped all the 10A fuses for 5A ones, since the regulators are only rated at 7.5A.

The VLANs are assigned now, and I have bonding going between the two pairs of Ethernet devices. In spite of the switch only supporting 4 LAGs, it seems fine with me doing LACP on effectively 10 LAGs. I’ll see how it goes.

The switch has 5 ports spare after plugging in all 5 nodes and a 16-port switch for the IPMI subnet. One will be used for a management interface so I can plug a laptop in, and the others will be paired with LACP for linking to my two existing Cisco SG200-8s.

One of the goals of this project is to try and push the performance of Ceph. In the office, we tried bare Ceph, and found that, while it’s fine for sequential I/O, it suffers a bit with random read/writes, and Windows-based HyperV images like to do a lot of random reads/writes.

Putting FlashCache in the mix really helped, but I note now, it’s no longer maintained. EnhanceIO had only just forked when I tried FlashCache, now it seems that’s the official successor.

There are two alternatives to FlashCache/EnhanceIO: bcache and dm-cache.

I’ll rule out bcache now as it requires the backing image be “formatted” for use. In other words, the backing image is not a raw image, but some proprietary (to bcache) format. This isn’t unworkable, but it raises concerns with me about portability: if I migrate a VM, do I need to migrate its cache too, or is it sufficient to cleanly shut down and detach the bcache device before re-assembling it on the new host?

By contrast, dm-cache and EnhanceIO/FlashCache work with raw backing images, making them much more attractive. Flush the cache before migration or use writethru mode, and all should be fine. dm-cache does however require a separate metadata device: messy, but not unworkable. We can provision the cache-related devices we need using LVM2, and use the kernel-mode Rados block device as our backing image.

So I think my caching subsystem is a two-horse race: dm-cache or EnhanceIO. I guess we’ll give them a try and see how they go.

For those following along at home, if you’re running kernel >4.3, you might want use this fork of EnhanceIO due to changes in the kernel block I/O layer.

To manage the OpenNebula master node, I’ve installed corosync/pacemaker. Normally these are used with DR:BD, however I figure Ceph can fulfil that role. The concepts are similar: it’s a shared block device. I’m not sure if it’ll be LXC, Docker or a VM at this point that “contains” the server, but whatever it is, it should be possible for it to have its root FS and data on Ceph.

I’m leaning towards LXC for this. Time for some more experimentation.

Improved Helmets: Project background

Well, looks like this project is very much thrust into the spotlight having been covered in Hacklet 105 . Mine’s probably the least technical of the lot, it’s definitely worth having a look at what the others are doing, as there’s some really innovative ideas there. Many thanks to @Mike Szczys and @Adam Fabio for the shout-out. 🙂

One thing I haven’t done with this project yet, is to actually post the background of why I’ve started this. A big part of this was I wanted to get permission from the family of a work colleague of mine so that I could mention him by name, but at this stage, permission has not been given, so I have to keep things anonymous.

On the 12th of February, a colleague of mine was cycling to work over the Go Between Bridge here in Brisbane when he lost control on a bend as the bridge joins the Bicentennial Bikeway. This is an off-road, dedicated cycleway, so no cars, and supposedly no pedestrians, however many seem to not understand what a sign with a bicycle symbol and the letters O, N, L, Y mean. (I usually ride past and comment: “Funny bike you’re riding!”. Since this accident though, I intend to be a lot more assertive.)

(Above: the crash scene. That blood smear is still visible on the path today.)

I’m no crash investigator, but I did study physics, and I cycle as my sole means of transport myself, having no driver’s license. (And no interest in getting one either.) I’m familiar with what that bridge is like to cycle over, having done it many times shortly after it opened when I worked at West End.

Looking at the scene though, it was apparent to me that my colleague was going much faster than was sensible for that stretch of road, and something caused him to lose control just prior to the bend.

The resulting impact with the railing was devastating: in addition to a few broken bones elsewhere in the body, he suffered skull fractures, and what I understand now to be a Coup-Contrecoup injury to the brain.

I remember that morning arriving at work early (we both were early birds, and had he not crashed, he would have beaten me that morning), sitting down at my desk and preparing to do battle with U-Boot and an industrial PC, when at 6:34AM, the office phone rings. It was then I learned that my colleague was in a serious condition in hospital, and I then found myself frantically looking for contact details for his wife. (Which were nowhere to be found.)

We later learned he’d never regain consciousness, having lost all executive function in the brain. The only bits that worked, were the bits responsible for low-level muscle control. From bright mind, to persistent vegetative state. He passed away about a fortnight after his accident.

During his brief time in ICU, we were told by one of the people there that these sorts of injuries were common in bicycle and motorcycle accidents. That worried me.

That tells me that perhaps, something is wrong with these blocks of foam we insist on strapping to our heads, and that we’ve missed something. This is one of the first goals I’d like to pinpoint, but so far, has been the most difficult: trying to get hold of data that would statistically prove or disprove how “common” these injuries are.

There’s no point in protecting the skull itself if the brain is to get shaken around to the point that the person winds up with total mental incapacitation.

Research seems to suggest that helmets have had a big hand in reducing the incidents of these injuries, but the fact that it’s still “common”, seems to suggest there’s lots more work to be done.

The standards are focussed on linear acceleration, and single impacts at no more than about 20km/hr. Is that sufficient? I regularly find myself doing 40, and I’m no speed demon. (Hell, I’ve accidentally found myself doing 71km/hr once!) I think it’s time the standards were revised. The question is: how?

My colleague was a key member of our team, and one of the brighter minds I know. While he shouldn’t have taken that bend at such speed and expect to get away with it, he did not deserve to die. I can’t save him, but perhaps I can help save someone else. That’s what this project is about.

Solar cluster: Trying out the analogue controller: FAIL

Well, not sure what went wrong, but the controller I built on Monday evening, dead-bug style, is one big fail.

There’s no output from the LM311s, even after adding pull-ups, they still don’t seem to respond to the battery voltage falling below the threshold. Add to that, a faulty IRF540N MOSFET (drain-source resistance of ~40Ω), and you’ve got all the makings of things going wrong.

So time for a U-turn, after deciding against doing a microcontroller-based solution before on the grounds I had the parts on hand to do an analogue comparator solution, I’ve decided I’ll do it with a ATTiny24A after all. I can get these for about $12 for a pack of 5 from a local supplier.

I also have placed on order, two 5V switchmode PSU modules and four P-channel MOSFETs: we’ll drop the relay as well and make it all solid-state.

The MCU doesn’t have to do much, just take an ADC reading every 100msec of the battery voltage, compare it to a threshold then either turn on or turn off the power.

The MCU has up to 6 ADC channels, embeds a small temperature sensor, has one PWM channel and a number of GPIOs. Reserving the reset and SPI lines for ISP work, that gives us 3 digital outputs and one PWM for controlling things and 4 ADC channels.

I can use the PWM channel to drive a MOSFET for the fans, one of the outputs to drive NPN transistors for controlling the ACPI power buttons on the nodes, and two MOSFETs for the mains and solar inputs. 3 ADCs can monitor the battery, mains and solar inputs, so decisions can be made on whether to switch between solar/mains or to turn off all inputs and let the battery drain for a bit.

The internal temperature sensor can be used for fan control. The internal 8MHz oscillator will be “good enough” I think. It mainly needs to tell the difference between hot and cold. If things are >25°C, then we should run the fans, the hotter it is, the faster they should run.

This isn’t rocket-science, and should be achievable via a simple while loop in C.