Public Syndication

Solar Cluster: BCDC1225 switching between solar and mains not reliable

So yeah, it seems history repeats itself.  The Redarc BCDC1225 is not reliable in switching between solar inputs and 12V input derived from the mains.

At least this morning’s wake-up call was a little later in the morning:

From: ipmi@hydrogen.ipmi.lan
To: stuartl@longlandclan.id.au
Subject: IPMI hydrogen.ipmi.lan
Message-Id: <20171023194305.72ECB200C625@atomos.longlandclan.id.au>
Date: Tue, 24 Oct 2017 05:43:05 +1000 (EST)

Incoming alert
IP : xxx.xxx.xxx.xxx
Hostname: hydrogen.ipmi.lan
SEL_TIME:"1970/01/27 02:03:00" 
SENSOR_NUMBER:"30"
SENSOR_TYPE:"Voltage          "
SENSOR_ID:"12V             " 
EVENT_DESCRIPTION:"Lower Critical going low                                         "
EVENT_DIRECTION:"Assertion  "
EVENT SEVERITY:"non-critical"

We’re now rigging up the Xantrex charger that I was using in early testing and will probably use that for mains. I have a box wired up with a mains SSR for switching power to it.  I think that’ll be the long-term plan and the Redarc charger will be retired from service, perhaps we might use it in some non-critical portable station.

Solar Cluster: Solar Testing

So I’ve now had the solar panels up for a month now… and so far, we’ve had a run of very overcast or wet days.

Figures… and we thought this was the “sunshine state”?

I still haven’t done the automatic switching, so right now the mains power supply powers the relay that switches solar to mains.  Thus the only time my cluster runs from solar is when either I switch off the mains power supply manually, or if there’s a power interruption.

The latter has not yet happened… mains electricity supply here is pretty good in this part of Brisbane, the only time I recall losing it for an extended period of time was back in 2008, and that was pretty exceptional circumstances that caused it.

That said, the political football of energy costs is being kicked around, and you can bet they’ll screw something up, even if for now we are better off this side of the Tweed river.

A few weeks back, with predictions of a sunny day, I tried switching off the mains PSU in the early morning and letting the system run off the solar.  I don’t have any battery voltage logging or current logging as yet, but the system went fine during the day.  That evening, I turned the mains back on… but the charger, a Redarc BCDC1225, seemingly didn’t get that memo.  It merrily let both batteries drain out completely.

The IPMI BMCs complained bitterly about the sinking 12V rail at about 2AM when I was sound asleep.  Luckily, I was due to get up at 4AM that day.  When I tried checking a few things on the Internet, I first noticed I didn’t have a link to the Internet.  Look up at the switch in my room and saw the link LED for the cluster was out.

At that point, some choice words were quietly muttered, and I wandered downstairs with multimeter in hand to investigate.  The batteries had been drained to 4.5V!!!

I immediately performed some load-shedding (ripped out all the nodes’ power leads) and power-cycled the mains PSU.  That woke the charger up from its slumber, and after about 30 seconds, there was enough power to bring the two Ethernet switches in the rack online.  I let the voltage rise a little more, then gradually started re-connecting power to the nodes, each one coming up as it was plugged in.

The virtual machine instances I had running outside OpenNebula came up just fine without any interaction from me, but  it seems OpenNebula didn’t see it fit to re-start the VMs it was responsible for.  Not sure if that is a misconfiguration, or if I need to look at an alternate solution.

Truth be told, I’m not a fan of libvirt either… overly complicated for starting QEMU VMs.  I might DIY a solution here as there’s lots of things that QEMU can do which libvirt ignores or makes more difficult than it should be.

Anyway… since that fateful night, I have on two occasions run the cluster from solar without incident.  On the off-chance though, I have an alternate charger which I might install at some point.  The downside is it doesn’t boost the 12V input like the other one, so I’d be back to using that Xantrex charger to charge from mains power.

Already, I’m thinking about the criteria for selecting a power source.  It would appear there are a few approaches I can take, I can either purely look at the voltages seen at the solar input and on the battery, or I can look at current flow.

Voltage wise, I tried measuring the solar panel output whilst running the cluster today.  In broad daylight, I get 19V off the panels, and at dusk it’s about 16V.

Judging from that, having the solar “turn on” at 18V and “turn off” at 15V seems logical.  Using the comparator approach, I’d need to set a reference of 16.5V and tweak the hysteresis to give me a ±3V swing.

However, this ignores how much energy is actually being produced from solar in relation to how much is being consumed.  It is possible for a day to start off sunny, then for the weather to cloud over.  Solar voltage in that case might be sitting at the 16V mentioned.

If the current is too low though, the cluster will drain more power out than is going in, and this will result in the exact conditions I had a few weeks ago: a flat battery bank.  Thus I’m thinking of incorporating current shunts both on the “input” to the battery bank, and to the “output”.  If output is greater than input, we need mains power.

There’s plenty of literature about interfacing to current shunts.  I’ll have to do some research, but immediately I’m thinking an op-amp running from the battery configured as a non-inverting DC gain block with the inputs going to either side of the current shunt.

Combining the approaches is attractive.  So turn on when solar exceeds 18V, turn off when battery output current exceeds battery input current.  A dual op-amp, a dual comparator, two current shunts, a R-S flip-flop and a P-MOSFET for switching the relay, and no hysteresis calculations needed.

Drones

So, over the last few years, computing power has gotten us to the point where remotely operated aerial vehicles are not only a thing, but are cheap and widely available.

There are of course, lots of good points about these toys, lots of tasks in which they can be useful.  No, I don’t think Amazon Prime is one of them.

They come with their risks though, and there’s a big list of do’s and don’ts regarding their use.  For recreational use, CASA for example, have this list of rules.  This includes amongst other things, staying below 120m altitude, and 30m away from any person.

For a building, that might as well be 30m from the top of the roof, as you cannot tell if there are people within that building, or where in that building those people reside, or from what entrance they may exit.

I in principle have no problem with people playing around with them.  I draw the line where such vehicles enter a person’s property.

The laws are rather lax about what is considered trespass with regards to such vehicles.  The no-brainer is if the vehicle enters any building or lands (controlled or otherwise) on any surface within the property.  A big reason for this is that the legal system often trails technological advancement.

This does not mean it is valid to fly over someone’s property.  For one thing, you had better ensure there is absolutely no chance that your device might malfunction and cause damage or injury to any person or possession on that property.

Moreover, without speaking to the owner of said property, you make it impossible for that person to take any kind of preventative action that might reduce the risk of malfunction, or alert you to any risks posed on the property.

In my case, I operate an amateur radio station.  My transmitting equipment is capable of 100W transmit power between 1.8MHz and 54MHz, 50W transmit power between 144MHz and 148MHz, and 20W transmit power between 420MHz and 450MHz, using FM, SSB, AM and CW, and digital modes built on these analogue modulation schemes.

Most of my antennas are dipoles, so 2.2dBi, I do have some higher-gain whips, and of course, may choose to use yagis or even dish antennas.  The stations that I might choose to work are mostly terrestrial in nature, however, airborne stations such as satellites, or indeed bouncing off objects such as the Moon, are also possibilities.

Beyond the paperwork that was submitted when applying for my radio license (which for this callsign, was filed about 9 years ago now, or for my original callsign was filed back in December 2007), there is no paperwork required to be submitted or filled out prior to me commencing transmissions.  Not to the ACMA, not to CASA, not to registered drone operators in the local area, not anybody.

While I’ve successfully operated this station with no complaints from my neighbours for nearly 10 years… it is worth pointing out that the said neighbours are a good distance away from my transmitting equipment.  Far enough away that the electromagnetic fields generated are sufficiently diminished to pose no danger to themselves or their property.

Any drone that enters the property, is at risk of malfunction if it strays too close to transmitting antennas.  If you think I will cease activity because you are in the area, think again.  There is no expectation on my part that I should alter my activities due to the presence of a drone.  It is highly probable that, whilst being inside, I am completely unaware of your device’s presence.  I cannot, and will not, take responsibility for your device’s electromagnetic immunity, or lack thereof.

In the event that it does malfunction though… it will be deemed to have trespassed if it falls within the property, and may be confiscated.  If it causes damage to any person or possession within the property, it will be confiscated, and the owner will be expected to pay damages prior to the device’s return.

In short, until such time as the laws are clarified on the matter, I implore all operators of these devices, to not fly over any property without the express permission of the owner of that property.  At least then, we can all be on the same page, we can avoid problems, and make the operation safer for all.

Solar Cluster: Solar Panel Installation

So… there came a weekend where two of us were free, and we had the bits organised, we could install the panels themselves.

We mounted two rails to the metal roof, then one by one, I’d terminate a cable with the solar connectors, I’d pass the panel up where my father would mount it to the rails, then the cable would be passed up, connected to the panel, then the unterminated end tossed over the gutter.

Once we were certain of cable length, I’d cut it to length (a fun job cutting a live cable), then the process would repeat.

We started about 8AM and we’re now pretty much finished the actual panel installation. We need to get some conduit to better protect the cable, and once the sun is down, I might look at terminating the other ends of the cables via 10A fuses.

This is the installation on the roof as it is now.

There’s space for one more panel, which would give me 480W. There’s also the option of buying more rails and mounting those… plenty of space up there.

DIY DC “power wall” is an option, certainly a 12V feed in the kitchen would be nice for powering the slow cooker and in major weather events, the 12V fridge/freezer.

The cables just run over the edge of the roof, and will terminate under the roof on the back deck.

I’m thinking the fuse box will be about head height, and there’ll be an isolation switch for the 12V feed going (via 8GA cable) downstairs to where the cluster lives.

As it happens, we did a pretty good job estimating the length of cable needed.

The plan is, we’ll get some conduit to run that cable in, as having it run bare across a hot tin roof is not good for its longevity. One evening, I’ll terminate those cables and wire up the fuse box.

I’ve got to think about how I’ll mount the isolation switch, I’m thinking a separate smaller box might be the go there. After that, then I need to work on the automatic switching.

Solar Cluster: Adding Solar

So we’ve got a free weekend where there’ll be two of us to do a solar installation… thus the parts have now been ordered for that installation.

First priority will be to get the panels onto the roof and bring the feed back to where the cluster lives.  The power will come from 3 12V 120W solar panels that will be mounted on the roof over the back deck.  Theoretically these can push about 7A of current with a voltage of 17.6V.

We’ve got similar panels to these on the roof of a caravan, those ones give us about 6A of current when there’s bright sunlight.  The cluster when going flat-chat needs about 10A to run, so with three panels in broad daylight, we should be able to run the cluster and provide about 8A to top batteries up with.

We’ll be running individual feeds of 8-gauge DC cable from each panel down to a fused junction box under the roof on the back deck.  From there, it’ll be 6-gauge DC cable down to the cluster’s charge controller.

Now, we have a relay that switches between mains-sourced DC and the solar, and right now it’s hard-wired to be on when the mains supply is switched on.

I’m thinking that the simplest solution for now will be to use a comparator with some hysteresis.  That is, an analogue circuit.  When the solar voltage is greater than the switchmode DC power supply, we use solar.  We’ll need the hysteresis to ensure the relay doesn’t chatter when the solar voltage gets near the threshold.

The other factor here is that the solar voltage may get as high as 22V or so, thus resistor dividers will be needed both sides to ensure the inputs to the comparator are within safe limits.

The current consumption of this will be minimal, so a LM7809 will probably do the trick for DC power regulation to power the LM311.  If I divide all inputs by 3, 22V becomes ~7.3V, giving us plenty of head room.

I can then use the built-in NPN to drive a P-channel MOSFET that controls the relay.  The relay would connect between MOSFET drain and 0V, with the MOSFET source connecting to the switchmode PSU (this is where the relay connects now).

The solar controller also connects its control line to the MOSFET drain.  To it, the MOSFET represents the ignition switch on a vehicle, starting the engine would connect 12V to the relay and the solar controller control input, connecting the controller’s DC input to the vehicle battery and telling the controller to boost this voltage up for battery charging purposes.

By hooking it up in this manner, and tuning the hysteresis on the comparator, we should be able to handle automatic switch-over between mains power and solar with the minimum of components.

Same Sex Marriage Postal Vote: Thwarting the peepers

So it seems that the Same Sex Marriage postal votes are finally being sent around.  This is good news in a way: we get to have a say in the matter and hopefully put the matter to bed one way or the other.

No more umming and arring, which I’m frankly sick and tired of, as I feel there are more pressing needs.  Yes, it’s important, but we have two nuclear armed crazy-haired nutters at opposite sides of the Pacific ready to light the planet up like a neon light!

I’m in support of the legislation changing by the way.  I think same-sex couples are entitled to the same rights, and it wasn’t that long ago that marriage was restricted to those not just of the opposite sex, but also had to be of the same “race” and religion.

To quote a song by John Williamson: “They’d chain you up to a boab tree, for kissing an Aborigine!”

So to my way of thinking, society changes.  What was taboo yesterday, we don’t think twice about today.  An Anglican family sending their children to a Catholic school would be heresy years ago… but for my sister and I, that is exactly what happened.  The world doesn’t seem to have imploded as a result.

The status quo regarding marriage is a hang-over from when the Church was the only place where you could get married, and ruled with far greater weight than today.  This is no longer the case, thus it no longer makes sense to hang onto this concept.

Anyway… my opinions on this are beside the point.  In spite of the good intentions, it looks as if the postal vote envelopes overlook one serious flaw: with sufficient light they are see through!

So my proposal: Put a thin piece of card in with the postal vote to block the light.  Not thick enough that it might cause the envelope to jam or interfere with sorting equipment, just opaque enough to prevent the contents being visible.  A small piece of black paper would likely do the job nicely.

Sure the ABS will have a little bit more paper to dispose of, but then at least, our votes are secure and people can’t “manipulate” the vote by snooping on sealed envelopes and discard the ones that disagree with their opinions.  At least then we won’t be wasting $122M.

Splitting an mbox on OpenBSD

I have a virtual machine that I set up as a secondary DNS server which runs OpenBSD 6.1.  Today logging into it, I noticed system messages were piling up in /var/mail because I hadn’t configured the mail server to deliver those messages.  Setting up OpenSMTPD was no trouble, but then I had the old mail (thankfully not much) that was still to be delivered.

There are a couple of solutions out there, written in Perl, Python and PHP (urgh!).  I don’t have Python on this box, and the Perl one didn’t seem to work with the mailbox.  So I cooked up my own:

#!/bin/sh

for file in "$@"; do
        grep -n '^From ' ${file} | {
                prev=1
                while read line; do
                        cur=$( echo "${line}" | cut -f 1 -d: )
                        if [ "${prev}" != "${cur}" ]; then
                                sed -ne "${prev},$(( ${cur} - 1 )) p" ${file} > ${prev}.eml
                        fi
                        prev=${cur}
                done
        }
done

If there’s a line in your email body starting with “From “, it may get confused, but it was good enough for the messages that OpenBSD’s daemons send me. I was then able to pipe these individually into sendmail -t to send them on their way.

Configuring Telstra NextG with nmcli on Debian

This is a quick brain-dump, as doing a quick Google search did not help, taking me to a mailing list thread I had posted about 2.5 years ago.  I swear there’s a song in that… something about the dreaded Google Echo.

Anyway, unlike that last occasion where the modem wasn’t even seen at all (and no, I didn’t solve it, we stuffed a 3G dongle in the case in the end), this time around, ModemManager sees it.  It just so happens that nmtui doesn’t do wireless broadband. These were the magic commands.

root@wsg-74fe481fe117:~# nmcli connection edit type gsm con-name telstra-nextg

===| nmcli interactive connection editor |===

Adding a new 'gsm' connection

Type 'help' or '?' for available commands.
Type 'describe [.]' for detailed property description.

You may edit the following settings: connection, gsm, serial, ipv4
nmcli>

From here, we need to set the APN, telstra.internet.

nmcli> set gsm.apn telstra.internet

Having done that, we give the configuration a last check before saving it:

nmcli> print all
… lots of settings …
nmcli> save persistent
Saving the connection with 'autoconnect=yes'. That might result in an immediate activation of the connection.
Do you still want to save? (yes/no) [yes] (enter)
Connection 'telstra-nextg' (57c78d91-4a66-475b-8843-2cba590fbcfd) successfully saved.
nmcli> quit

Solar Cluster: OpenNebula, DNS shennanigans and network documentation

OpenNebula is running now… I ended up re-loading my VM with Ubuntu Linux and throwing OpenNebula on that.  That works… and I can debug the issue with Gentoo later.

I still have to figure out corosync/heartbeat for two VMs, the one running OpenNebula, and the core router.  For now, the VMs are only set up to run on one node, but I can configure them on the other too… it’s then a matter of configuring libvirt to not start the instances at boot, and setting up the Linux-HA tools to figure out which node gets to fire up which VM.

The VM hosts are still running Gentoo however, and so far I’ve managed to get them to behave with OpenNebula.  A big part was disabling the authentication in libvirt, otherwise polkit generally made a mess of things from OpenNebula’s point of view.

That, and firewalld had to be told to open up ports for VNC/spice… I allocated 5900-6900… I doubt I’ll have that many VMs.

Last weekend I replaced the border router… previously this was a function of my aging web server, but now I have an ex-RAAF-base Advantech UNO-1150G industrial PC which is performing the routing function.  I tried to set it up with Gentoo, and while it worked, I found it wasn’t particularly stable due to limited memory (it only has 256MB RAM).  In the end, I managed to get OpenBSD 6.1/i386 running sweetly, so for now, it’s staying that way.

While the AMD Geode LX800 is no speed demon, a nice feature of this machine is it’s happy with any voltage between 9 and 32V.

The border router was also given the responsibility of managing the domain: I did this by installing ISC BIND9 from ports and copying across the config from Linux.  This seemed to be working, and so I left it.  Big mistake, turns out bind9 didn’t think it was authoritative, and so refused to handle AXFRs with my slaves.

I was using two different slave DNS providers, puck.nether.net and Roller Network, both at the time of subscription being freebies.  Turns out, when your DNS goes offline, puck.nether.net responds by disabling your domain then emailing you about it.  I received that email Friday morning… and so I wound up in a mad rush trying to figure out why BIND9 didn’t consider itself authoritative.

Since I was in a rush, I decided to tell the border router to just port-forward to the old server, which got things going until I could look into it properly.  It took a bit of tinkering with pf.conf, but eventually got that going, and the crisis was averted.  Re-enabling the domains on puck.nether.net worked, and they stayed enabled.

It was at that time I discovered that Roller Network had decided to make their slave DNS a paid offering.  Fair enough, these things do cost money… At first I thought, well, I’ll just pay for an account with them, until I realised their personal plans were US$5/month.  My workplace uses Vultr for hosting instances of their WideSky platform for customers… and aside from the odd hiccup, they’ve been fine.  US$5/month VPS which can run almost anything trumps US$5/month that only does secondary DNS, so out came the debit card for a new instance in their Sydney data centre.

Later I might use it to act as a caching front-end and as a secondary mail exchanger… but for now, it’s a DIY secondary DNS.  I used their ISO library to install an OpenBSD 6.1 server, and managed to nut out nsd to act as a secondary name server.

Getting that going this morning, I was able to figure out my DNS woes on the border router and got that running, so after removing the port forward entries, I was able to trigger my secondary DNS at Vultr to re-transfer the domain and debug it until I got it working.

With most of the physical stuff worked out, it was time to turn my attention to getting virtual instances working.  Up until now, everything running on the VM was through hand-crafted VMs using libvirt directly.  This is painful and tedious… but for whatever reason, OpenNebula was not successfully deploying VMs.  It’d get part way, then barf trying to set up 802.1Q network interfaces.

In the end, I knew OpenNebula worked fine with bridges that were already defined… but I didn’t want to have to hand-configure each VLAN… so I turned to another automation tool in my toolkit… Ansible:

- hosts: compute
  tasks:
  - name: Configure networking
    template: src=compute-net.j2 dest=/etc/conf.d/net
# …
- hosts: compute
  tasks:
# …
  - name: Add symbolic links (instance VLAN interfaces)
    file: src=net.lo dest=/etc/init.d/net.bond0.{{item}} state=link
    with_sequence: start=128 end=193
  - name: Add symbolic links (instance VLAN bridges)
    file: src=net.lo dest=/etc/init.d/net.vlan{{item}} state=link
    with_sequence: start=128 end=193
# …
  - name: Make services start at boot (instance VLAN bridges)
    command: rc-update add net.vlan{{item}} default
    with_sequence: start=128 end=193 

That’s a snippet of the playbook… and it basically creates symbolic links from Gentoo’s net.lo for all the VLAN ports and bridges, then sets them up to start at boot.

In the compute-net.j2 file referenced above, I put in the following to enumerate all the configuration bits.

# Instance VLANs
{% for vlan in range(128,193) %}
config_vlan{{vlan}}="null"
config_bond0_{{vlan}}="null"
rc_net_vlan{{vlan}}_need="net.bond0.{{vlan}}"
{% endfor %}
# …
vlans_bond0="5 8 10{% for vlan in range(128,193) %} {{vlan}} {% endfor %}248 249 250 251 252"
vlans_bond1="253"
# …
# Instance VLANs
{% for vlan in range(128,193) %}
bridge_vlan{{vlan}}="bond0.{{vlan}}"
{% endfor %} 

The start and end ranges are a little off, but it saved a lot of work.

This naturally took a while for OpenRC to bring up… but it worked. Going back to OpenNebula, I told it what bridges to use, and before long I had my first instance… an OpenBSD router to link my personal VLAN to the DMZ.

I spent a bit of time re-working my routing tables after that… in fact, my network is getting big enough now I have to write some details down.  I spent a few hours documenting the effort:

That’s page 1 of about 15… yes my hand is sore… but at least now should I get run over by a bus, others have a fighting chance doing anything with the network without my technical input.