September 2018

Solar Cluster: Thank goodness for good monitoring

A few months back now, I had the misfortune of overshooting my Internet quota, and winding up with a AU$380 bill for the month (and that was capped… in truth it was more like AU$3000).  In fact, it happened a couple of times until I finally nailed down the cause.

Part of it was NTP traffic (seems lots of cowboys write SNTP clients now and point them at pool.ntp.org), some was the #Hackaday.io Spambot Hunter Project and related activity.  In short, I invested some money into upping the quota, and some time into better monitoring.

I wanted to do the monitoring anyway to keep an eye on operations, as well as things like the solar panel voltages, etc.  Since I got it in place, I’ve been able to get much faster notifications of when things go awry.  Much sooner than the 120% quota usage alarm that Internode sends you.

I’m glad I did that now, last night I left a few tabs open on the Hackaday.io site.  I noticed this evening they were still trying to load something and got suspicious… then I saw this:

Double checking, sure enough, something on one of those pages made Chromium get its knickers into a twist, and chew through all that data.

It took me a bit of tinkering to get the right query to extract the above chart.  Essentially there was a sustained 1.5MB/sec download for over 21 hours which would account for the 113.1GB that Internode recorded.

It’s a bit co-incidental that the usage dropped the moment I re-started Chromium.  Not sure why it was continually re-loading pages, but never mind.

The above data is collected using a combination of collectd and InfluxDB, with Grafana doing the dashboarding and alarms, and a small Perl script pulling the usage data off Internode’s API.

Sunsetting the yi.org domains

Update: Real life intervened and I didn’t get around to reconfiguring Postfix like I said I would. I am doing that now, any longlandclan.yi.org or vk4msl.yi.org will start bouncing now.

Originally, when I started down the path of running my own server, I was an unemployed student, so the servers were hand-me-down second hand affairs and the domains I used were freebie ones.  I started out with no-ip.com, and when they changed their policies, I switched to yi.org (in 2007).

yi.org have been fantastic.  Not only is the domain short, but they also allow many record types including TXT (needed for SPF rules), AAAA (IPv6), MX (for mail servers) and NS, yes they’ll even let you delegate a subdomain to DNS servers of your choosing.

That said, they did have a spot of unreliability a few years back (around 2015).  Given that I now have an income of my own, it no longer made sense to just go for free services, so I bought a couple of id.au domains.  My email client was configured so that in the event someone sent me an email to the old address, they would see the following in my reply:

Reply-To: user@longlandclan.id.au
Subject: Re: …
References: <…>
To: …
From: "Stuart Longland (OLD ADDRESS see reply-to)"
 <user@longlandclan.yi.org>

I’ve been doing this for a few years now.  The yi.org domains now only receive mail that comes into two categories:

  • people who still use the old email address not realising I’ve changed
  • spammers that have harvested the old email address

From October November this year, I’ll be bouncing emails sent to the old domain, with a link to this page.  From November this year In 2019, the MX records for the yi.org domains will disappear.  In short I will not be receiving email from any of the old yi.org addresses from November 2018!

If you see yi.org in an email address for one of us, now is the time to replace that with id.au.  If you see one ending in hopto.org, then you are really out of date.

Please note, I might be able to give you instructions on how to update the email address in your client, but I’ll assume your client works exactly the same as mine does to the pixel, my instructions will be something along the lines of “Go to the Tools menu, click Address Book, type ‘yi.org’ into Criteria, press ENTER, then for each contact you see with my old address in it, double-click on it, change the address and click OK”.

I provide 0 support for email clients I don’t use: I’ll assume you know how to use your system or know how to research the problem, it’s not up to me to teach you as life’s too short.

Solar Cluster: Return of the power controller

Well, I’ve been tossing up how to control the mains charger for a while now.

When I first started the project, my thinking was to use an old Xantrex charger we had kicking around, and just electrically disconnect it from the batteries when we wanted to use the solar power.  I designed a 4-layer PCB which sported a ATTiny24A microcontroller, MOSFETs (which I messed up) and some LEDs.

This was going to be a combined fan controller and power management unit.  It had the ability (theoretically) to choose a supply based on the input voltage, and to switch if needed between supplies.

It didn’t work out, the charger got really confused by the behaviour of the controller.  I was looking to re-instate it using the Redarc solar controller, but I never got there.  In the end, it was found that the Redarc controller had problems switching sources and would do nothing whilst the batteries went flat.

We’ve now replaced both ends of the system.  The solar controller is a Powertech MP3735 and integrates over-discharge protection.  The mains charger is now a MeanWell HEP-600C-12 (which has not missed a beat since the day it was put in).

Unlike my earlier set-up, this actually has a 5V logic signal to disable it, and my earlier controller could theoretically generate that directly.

Looking at the PCB of my earlier power controller attempt, it looks like this could still work.

Above is the PCB artwork.  I’ve coloured in the sections and faded out the parts I can omit.

In green up the top-left we have the mains control/monitoring circuitry.  We no longer see the mains voltage, so no point in monitoring it, so we can drop the resistor divider that fed the ADC.  This also means we no longer need the input socket P2.

Q2 and Q7 were the footprints of the two P-channel MOSFETs.  We don’t need the MOSFETs themselves, but the signals we need can be found on pin 1 of Q2.  This is actually the open-drain output of Q1, which we may be able to hook directly to the REMOTE+ pin on the charger.  A pull-up between there and the charger’s 5V rail, and we should be in business.

In yellow, bottom left is the solar monitoring interface.  This is still useful, but we won’t be connecting solar to the battery ourselves, so we just keep the monitoring parts.  The LED can stay as an indicator to show when solar is “good enough”.

In purple, occupying most of the board, is the controller itself.  It stays for obvious reasons.

In red, is the fan control circuitry.  No reason why this can’t stay.

In blue is the circuitry for monitoring the battery voltage.  Again, this stays.

The main advantage of doing this is I already have the boards, and a number of microcontrollers already present.  There’s a board with all except the big MOSFETs populated: with the MOSFETs replaced by 3-pin KK sockets.

How would the logic work?  Much the same as the analogue version I was pondering.

  • If battery voltage is low, OR, the sun has set, enable the mains charger.

What concerned me about an analogue solution was what would happen once the charger got to the constant-voltage stage.  We want to give it a bit of time to keep the battery topped up.  Thus it makes sense to shut down the charger after a fixed delay.

This is easy to do in a microcontroller.  Not hard with analogue electronics either, it’s fundamentally just a one-shot, but doing it with an MCU is a single-chip solution.  I can make the delay as long as I like.  So likely once the battery is “up to voltage”, I can let it float there for an hour, or until sunrise if it’s at night.