computing

Telstra: another mob that didn’t get the RFC5233 memo

So this evening, I got a bit of marketing from Telstra. This was to an email address I had used to register the SIM card that I’m trying out in the Kite. I naturally followed the same approach I have with other such suppliers as an anti-phishing tactic.

The email is not unsolicited, but it is a commercial email nonetheless. I figured I’d just quietly opt-out, no need to make a fuss. The email itself was legitimate, so no concern about boobytrapped unsubscribe links. Naturally, I copied the address from their email and paste it into the form on their webpage. I get told this:

Errm, excuse me? That is the email address that I wish to unsubscribe, and if it were invalid, I would not be trying to unsubscribe because I would not have gotten the email in the first place!

Okay, so I’ll need to go through a human to get this resolved, what joy. Navigate the labyrinth that is the Telstra support site (they really don’t want you to be able to make complaints), and I get to a complaints form. First thing I note, they forgot to close an <a> tag (end of line 154)…

<p>If you require immediate assistance with a complaint, <b>Consumer customers</b> can call us anytime on 132200 and say "complaint".<br><br>
If you are a <b>Business customer</b> and require immediate assistance with a complaint, you can call us anytime on 132000 and say "complaint".</p>
<b>Enterprise and Government customers:</b> please go to your specialised contact page <a href="https://www.telstra.com.au/business-enterprise/contact-us/make-a-complaint" target="_self">here</a>.
&nbsp;
<p>Further information on how we handle complaints can be found in our <a href="https://www.telstra.com.au/content/dam/tcom/personal/help/pdf/telstra-complaint-handling-process.pdf">complaints handling process document (PDF).</p></pre>
</div>
<div id="surveyMainDiv" class="main-background">
<div class="place-holder-div" id="surveyMainDivBannerDiv"></div>
<div id="surveyContentDiv" class="content-background">

As a result, Firefox thinks everything to the end of the form, is part of the link! They also close a tag that isn’t open: <pre>.

UPDATE 2018-07-07: This has now been fixed.

Right, so there’s two things. I persevere with the form, resorting to keyboard shortcuts since clicking on any form element brings up that PDF.

Happy that I’ve covered what I wanted to say, I hit the submit. Only to find out the same person who designed the last form, must have designed this one too.

Great, so that’s now three things to complain about.

What really saddens me with Telstra is that today their management tell us they “aspire to be a technology company”. The fact that years ago, Telecom Australia was very much a respected member of the ITU meant it pretty much was a technology company… and the fact they can’t get something as basic as email address validation or a simple web form right, really does show how far they have fallen.

I fully expect this will go back-and-forth while they ask for my browser details (irrelevant, this is broken HTML at their end), my OS (again irrelevant), and then the claim that: “Ohh, we don’t support that!” Which will hold about as much water as a tissue paper G-string.


So, an update. I had a reply back, basically they stated a few things:

  1. they claim to not have seen any marketing emails for the past two months sent to me. (how hard did they look?)
  2. they claim to have taken my name off the list (we’ll see)

They make no comment about fixing the forms. The complaints form now has its closing </a> tag back, so clicking on form elements no longer causes it to pop up with a PDF download. Great, 1 problem of 3 fixed.

I finally had a moment to reply, and did so. In their email, they give an address to send the reply to (because we’re to cool to set the Reply-To header or use the correct From address):

I got back an immediate response:

Delivery has failed to these recipients or distribution lists:

ComplaintResolutionCentre@team.telstra.com
The recipient’s e-mail address was not found in the recipient’s e-mail system. Microsoft Exchange will not try to redeliver this message for you. Please check the e-mail address and try resending this message, or provide the following diagnostic text to your system administrator.


Sent by Microsoft Exchange Server 2007

Diagnostic information for administrators:

Generating server: srv.dir.telstra.com

ComplaintResolutionCentre@team.telstra.com
#550 5.1.1 RESOLVER.ADR.RecipNotFound; not found ##

Original message headers:

Received: from ipani.tcif.telstra.com.au (10.97.216.198) by
 ties-smtp.in.telstra.com.au (172.49.40.197) with Microsoft SMTP Server id
 8.3.485.1; Sat, 7 Jul 2018 17:58:02 +1000
Received: from ipocni.tcif.telstra.com.au ([10.97.216.53])  by
 ipbani.tcif.telstra.com.au with ESMTP; 07 Jul 2018 17:58:02 +1000
X-IronPort-Anti-Spam-Filtered: true
X-IronPort-Anti-Spam-Result: =?us-ascii?q?A0GkBACJcUBb/+KwZZaFN5wRlRWBaTKBT?=
 =?us-ascii?q?YYSBgMCAgKGSwtCJwE8FYEggwqqCQUOgmyEHYUAgStDAWaJaIMgSYRqCAUFAQs?=
 =?us-ascii?q?IB1eCWYo0hF4Pg1eBKA6YUIQOgmt2imKIYIUYPYIxoUUCDRsDggU?=
X-IPAS-Result: =?us-ascii?q?A0GkBACJcUBb/+KwZZaFN5wRlRWBaTKBTYYSBgMCAgKGSwt?=
 =?us-ascii?q?CJwE8FYEggwqqCQUOgmyEHYUAgStDAWaJaIMgSYRqCAUFAQsIB1eCWYo0hF4Pg?=
 =?us-ascii?q?1eBKA6YUIQOgmt2imKIYIUYPYIxoUUCDRsDggU?=
X-IronPort-AV: E=Sophos;i="5.51,320,1526306400"; 
   d="png'150?scan'150,208,217,150";a="119258049"
X-Amp-Result: UNKNOWN
X-Amp-Original-Verdict: FILE UNKNOWN
X-Amp-File-Uploaded: False
X-SBRS: None
Received: from eth2015.qld.adsl.internode.on.net (HELO
 mail.longlandclan.id.au) ([150.101.176.226])  by ipxcno.tcif.telstra.com.au
 with ESMTP; 07 Jul 2018 17:57:59 +1000
Received: from [IPv6:2001:44b8:21ac:7053:a64e:31ff:fe53:99cc] (unknown
 [IPv6:2001:44b8:21ac:7053:a64e:31ff:fe53:99cc])	by mail.longlandclan.id.au
 (Postfix) with ESMTPSA id C159B51F720	for
 <ComplaintResolutionCentre@team.telstra.com>; Sat,  7 Jul 2018 17:57:56 +1000
 (EST)
Subject: [SR 1-1580842703975] Re: Follow Up-Your complaint with Telstra
References: <1e3d0bcc-a187-42cb-ac52-1e1ef0f4673b@wsmsg3704.srv.dir.telstra.com>
To: <ComplaintResolutionCentre@team.telstra.com>
From: Stuart Longland <me@mydomain.org>
Openpgp: id=77102FB21549FFDE5E13B83A0C7F53F4F359B8EF;
 url=https://stuartl.longlandclan.id.au/key.asc
Message-ID: <b5da1c9c-bc3d-8b2f-0f56-55361dc16503@longlandclan.id.au>
Date: Sat, 7 Jul 2018 17:57:51 +1000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.7.0
MIME-Version: 1.0
In-Reply-To: <1e3d0bcc-a187-42cb-ac52-1e1ef0f4673b@wsmsg3704.srv.dir.telstra.com>
Content-Type: multipart/mixed;
	boundary="------------37DC9E91B74192D682B54693"
Content-Language: en-GB
Return-Path: me@mydomain.org
Reporting-MTA: dns;srv.dir.telstra.com
Received-From-MTA: dns;ipani.tcif.telstra.com.au
Arrival-Date: Sat, 7 Jul 2018 07:58:02 +0000

Final-Recipient: rfc822;ComplaintResolutionCentre@team.telstra.com
Action: failed
Status: 5.1.1
Diagnostic-Code: smtp;550 5.1.1 RESOLVER.ADR.RecipNotFound; not found

Oops… so there’s another complaint:

I note there’s another address (with an ‘s’ on the end) in the footer of the email, and so I have sent them the following:

Hi,
It's taken a little while to get back to you on this as I've been flat
out, but here goes.

On 07/07/18 17:20, Telstra_Notifications wrote:
> Your complaint with Telstra
>
> Reference no: SR x-xxxxxxxxxxxxx
>
> Dear Mr Longland,
>
> Thank you for getting in touch with us on 28 June 2018 about a
> complaint relating to your Telstra account number xxxx xxxxx xxxx.
>
> I’m sorry that you’ve experienced an issue with your service, but
> I'm pleased to offer you the following resolution.

To be clear, the issue is not with the mobile service itself, that's
been fine for the purpose I've used it. The issue is in the marketing
that came with it, that was unwanted.

> You were concerned that:
>
> * You’d like to be removed from Telstra’s marketing list

Yes, this is correct. It might be polite to ask people when they sign
up whether they want to be on this marketing list or not.

In my case, the service is temporary: I have the loan of a prototype
mobile phone: iSquare Mobility Kite v1.

http://www.kiteboard.io/ is the device being trialled.

The manufacturer has loaned it so that I can trial the device on the
Australian mobile networks, and see how it performs in weak-signal
conditions. I have loan of it possibly for another month or so at most.

(So far, it performs *MUCH* better than the ZTE T83 I use, and holds its
own against the ZTE T84 which uses the same chipset as the Kite.)

I'd have used my own SIM card, but my card is too big to fit in this
phone (mine is a miniature SIM, this phone requires a micro-SIM), and
given its temporary custody, it made no sense to get my existing Telstra
service moved to a new SIM.

Thus for this purpose, I just activated a pre-paid service to be able to
try the device out. I also have a service activated with Optus as it's
a dual-SIM device.

Once iSquare Mobility ask for the return of the device, naturally I'll
have little use for the two pre-paid SIM cards that are presently in it,
and won't have any interest of any offers from Telstra (or Optus).

I have an old 3G phone I can possibly use up the remaining credit of the
Telstra SIM in, otherwise I'll just use my current phone service which
I've had since 2001.

> * Telstra should fix broken complaints form
>
> I've confirmed that:
>
> * We have checked your account and found no marketing emails sent to
> you for the past two months

Allow me to present exhibit A; sent Thu, 28 Jun 2018 00:39:53 -0700.
This is attached.

I'm a little surprised your list management software had trouble finding
it, unless of course, you didn't read the complaint message carefully to
see the address my account was *actually* registered under.

I see you don't mention the issues with the form. One issue makes the
form damn-near unusable for anyone due to malformed HTML causing the
entire form to act as a hyperlink to the complaints information PDF.

The other, prevented me from self-unsubscribing and was the reason for
the complaint in the first place.

Don't worry, the world already knows:
https://vk4msl.com/2018/06/28/telstra-another-mob-that-didnt-get-the-rfc5233-memo/

I see the missed tag on the complaint form has now been corrected.

The original issue that started this, so far has not been corrected.
I've attached screenshots for your reference.

> We know you've been put out by this matter so we'd like to fix things
> by:
>
> * Confirming the medium of marketing (SMS, Email, phone call, MMS,
> face to face marketing, etc) and date you received it

This is email marketing. There have not been any other forms of marketing.

> * Removing your name and details from Telstra’s marketing list.
> Please be advised that this is only applicable for Telstra marketing
> calls.

Yep, I understand this. This is a silent number, and a temporary one at
that. By Christmas time, this service will be no-more, as it will be
surplus to requirements.

> If you’d like to talk more about this or accept this offer, please
> contact me on 1800 241 787* PIN 5172 or email
> ComplaintResolutionCentre@team.telstra.com quoting your Telstra
> reference SR x-xxxxxxxxxxxxx number. I'm available Tuesday-Saturday,
> 9am-5pm (AEST).

For reference, ComplaintResolutionCentre@team.telstra.com bounces. I've
attached the bounce message I received, and have also submitted it as SR
x-xxxxxxxxxxxxx just in case this email doesn't get through.

So that's now 4 issues in total, with 1 resolved so far.

If you could fix up the broken email validation on the opt-out form and
complaints form, and fix the broken email address in your reply messages
then that will resolve the remaining issues.

Thanks in advance.
Regards,
--
Stuart Longland (aka Redhatter, VK4MSL)

I haven't lost my mind...
...it's backed up on a tape somewhere.

Solar Cluster: NTP time synchronisation

For years up until last month, I ran a public NTP server which was on pool.ntp.org.  This mostly sat at Stratum II and was synchronised from random stratum I servers.

Last month, I had to knock that on the head because of a sudden spike in Internet traffic.  Through tcpdump on the border router, I identified the first culprit: NTP client traffic.  For whatever reason, I was getting a lot of traffic.  Dumping packets to a file over a 15 minute period and analysing showed about 90% of the traffic was NTP.

For about 3 weeks straight, I had an effectively constant 4~5Mbps incoming stream … a pelican flew into my inbox, carrying a AU$300 bill for the nearly 800GB consumed in May.  (My plan was originally 250GB.  Many thanks for Internode there: they capped the overusage charge at AU$300 and provided an instant upgrade of the plan to 500GB/month.)

The other culprit I tracked down to HackChat’s websockets, which was a big contributor to June’s data usage.  (Thank-you OpenBSD pflow and nfdump!)

So right now, I’m looking at whether I re-join the NTP server pool.  I have the monitoring in place to keep track of data usage now.  For sure the experience has cost me over $400, but that’s still cheaper than university studies … and they didn’t teach me this stuff anyway!

One thought is that I could do away with any external NTP server altogether by using local sources to synchronise time.  With a long-wire antenna, I could sync to WWV.  Usually I can pick it up on 10MHz or 15MHz, but it’s weak, and I’d have to rig up the receiver, the antenna and a suitable decoder.

Then there’s the problem of lightning strikes, I already have a Yaesu FT-897D in the junk box thanks to Thor’s past efforts.

The more practical approach would appear to be using GPS time sync.  You can do it with any NMEA-compatible module, but you get far better results using one that supports PPS.  Some even have a 10PPS option, but I’m not sure if kpps supports that.

As it happens, I could have got a GPS module in the TS-7670.  Here’s where it is on the schematic:

and here are the connections to its UART (bottom two):

and the PPS pin (LCD_D16):

The footprints are there on the board, and I can buy the module.  No problems.  Buying the TS-7670 with the GPS option would have meant buying the top-of-the-line “development” model which would have included WiFi (not needed), an extra 128MB RAM (nice to have, but not essential) and an extra CAN port (not needed).

The other option is to go something else, such as this module.  Either way, I need to watch logic levels, Freescale like their 1.8V, although it looks like the I/O pins can be switched between 1.8V and 3.3V from the GPIO registers (chapter 9 of the datasheet).

Doing this, would allow me to run Stratum 1 within the network, and perhaps to IPv6 NTP clients.  IPv4 clients might get Stratum 2, we’ll see how I feel about it later.

NBN Wireless vs gamers: raw download speeds misses the point

Recently, a stoush erupted between NBN chief executive Bill Morrow and the gaming community over whether “gamers” were “causing” the congestion issues experienced on fixed-wireless broadband links.

The ABC published this chart, comparing the average transfer rate, of various games, to the average transfer rate seen watching various movies.  It’s an interesting chart, but I think it completely misses the point.

One thing that raw download speeds miss, is latency.

Multimedia is hard real-time, however unless you’re doing a two-way video or voice call, a few seconds of latency is not going to bother you. Your playback device can buffer several seconds worth of movie to feed to your video and sound devices and keep their buffers fed. No problem.

If those buffers aren’t kept topped up, you get break-up in your audio and the video “freezes” momentarily, loosing the illusion of animation. So long as the data is received over the Internet link, passed to the decoder to be converted to raw video frames and audio samples, and stuffed into the relevant buffers in time, it all runs smoothly. Pre-recorded material makes this dead easy (by comparison). Uni-directional live streams are a bit more tricky, but again you can put up with quite a bit of latency.

Radio stations often have about 300-500ms of latency … just listen to the echo effect when a caller rings up with a radio on in the background, if it were truly live, it would howl like a PA microphone!

It’s two-way traffic that’s the challenge.

Imagine if, when typing an email… it was 5 seconds before the letters you just typed showed up. Or if you moved the mouse, it took 3 seconds before it registered that you had moved. If someone were just observing the screen (unaware of when the keystrokes/mouse clicks had been entered), they’d think the user was drunk!

And yes, I have personally experienced such links… type something, then go wait 30 seconds before hitting the ENTER key, or if you spot a mistake, count up the number of backspaces or cursor movements you need to type, then wait for the cursor to reach that spot before you make your correction. It’s frustrating!

Now consider online gaming, where reaction time requirements are akin to driving a race car. One false move, and suddenly your opposition has shot you, or they’ve successfully dodged your virtual bullet.

Carrier pigeons carrying MicroSD cards (which reach 128GB capacity these days) could actually outperform NBN in many places for raw data throughput. However, if the results from the Bergen Linux User’s Group experiments are anything to go by, you can expect a latency measured in hours. (Their ping log shows the round-trip-time to be about 53 minutes in the best case.)

The movie stream will be sending many large packets at a mostly regular rate. The video game will be sending lots of tiny packets that Must Be Delivered Right Now!

I think it naïve to directly compare the two in the manner these graphs simply due to the nature of the types of traffic involved. Video/VoIP calling would be a better metric, since a 100ms delay in a telephone conversation will have both parties verbally tripping over each other.

Tele-medicine is touted as one of the up-and-comming technologies, but for a surgeon to remotely operate on a patient, they need that robotic arm to respond right now, not in 30 seconds time.  It may not be a lot of data to say “rotate 2°”, or “move forward 500µm”, but it needs to get there quickly, and the feedback from said movement arrive back quickly if the patient is going to live.

The sooner we stop ignoring this elephant in the room, the better off we’ll all be.

Mitigating BREACH: a “what if”

So, recently there was a task at my work to review enabling gzip compression on our nginx HTTP servers to compress the traffic.

Now, in principle it seemed like a good idea, but having been exposed to the security world a little bit, I was familiar with some of the issues with this, notably, CRIME, BEAST and BREACH.  Of these, only BREACH is unmitigated at the browser end.

The suggested mitigations, in order of effectiveness are:

  1. Disabling HTTP compression
  2. Separating secrets from user input
  3. Randomizing secrets per request
  4. Masking secrets (effectively randomizing by XORing with a random secret per request)
  5. Protecting vulnerable pages with CSRF
  6. Length hiding (by adding random number of bytes to the responses)
  7. Rate-limiting the requests

Now, we’ve effectively being doing (1) by default… but (2), (3) and (4) make me wonder how protocols like OAuth2 are supposed to work.  That got me thinking about a little toy I was given for attending the 2011 linux.conf.au… it’s a YubiKey, one of the early model ones.  The way it operates is that Yubico’s servers, and your key, share a secret AES key (I think it’s AES-128), some static data, and a counter.  Each time you generate a one-time pad with the key, it increments its counter, encrypts the value with the static data, then encodes the output as a hexdump using a keyboard-agnostic encoding scheme to be “typed” into the computer.

Yubico receive this token, decrypt it, then compare the counter value.  If it checks out, and is greater than the existing counter value at their end, they accept it, and store that new counter value.

The same made me wonder if that could work for requests from a browser… that is, you agree on a shared secret over HTTPS, or using Diffie Hellman.  You synchronise counters (either using your new shared secret, or over HTTPS at the same time as you make the shared key), then from there on, each request to your API made by the browser, is then accompanied by a one-time pad, generated by encrypting the counter value and the static data and sending that in the HTTP headers.

There are a few libraries that do AES in the browser, such as JSAES (GPLv3) and aes-js (MIT).

This is going to be expensive to do, so a compromise might be to use this every N requests, where N is small enough that BREACH doesn’t have a sufficient number of requests from which it can derive a secret.  By the time it figures out that secret, the token is expired.  Or they could be bulk-generated at the browser end in the background so there’s a ready supply.

I haven’t gone through the full in’s and out’s of this, and I’m no security expert, but that’s just some initial thinking.

A few gripes with AliExpress

So, recently I bit the bullet and decided to sign up for an account with AliExpress.

So far, what I’ve bought from there has been clothing (unbranded stuff, not counterfeit) … while there’s some very cheap electronics there, I’m leery about the quality of some of it, preferring instead to spend a little more to buy through a more reliable supplier.

Basically, it’s a supplier of last resort, if I can’t buy something anywhere else, I’ll look here.

So far the experience has been okay.  The sellers so far have been genuine, while the slow boat from China takes a while, it’s not that big a deal.

That said, it would appear the people who actually develop its back-end are a little clueless where it comes to matters on the Internet.

Naïve email address validation rules

Yes, they’re far from the first culprits, but it would seem perfectly compliant email addresses, such as foo+bar@gmail.com, are rejected as “invalid”.

News to you AliExpress, and to anyone else, You Can Put Plus Signs In Your Email Address!

Lots of SMTP servers and webmail providers support it, to quote Wikipedia:

Addresses of this form, using various separators between the base name and the tag, are supported by several email services, including Runbox (plus), Gmail (plus),[11] Yahoo! Mail Plus (hyphen),[12] Apple’s iCloud (plus), Outlook.com (plus),[13] ProtonMail (plus),[14] FastMail (plus and Subdomain Addressing),[15] MMDF (equals), Qmail and Courier Mail Server (hyphen).[16][17] Postfix allows configuring an arbitrary separator from the legal character set.[18]

You’ll note the ones that use other characters (e.g. MMDF, Yahoo, Qmail and Courier) are in the minority.  Postfix will let you pick nearly anything (within reason), all the others use the plus symbol.

Doing this means instead of using my regular email address, I can use user+secret@example.com — if I see a spoof email pretending to be from you sent to user@example.com, I know it is fake.  On the other hand, if I see someone else use user+secret@example.com, I know they got that email address from you.

Email validation is actually a lot more complex than most people realise… it’s gotten simpler with the advent of SMTP, but years ago …server1!server2!server3!me was legitimate in the days of UUCP.  During the transition, server1!server2!server3!user@somesmtpserver.example.com was not unheard of either.  Or maybe user%innnerhost@outerhost.net?  Again, within standards.

Protocol-relative URIs don’t work outside web browsers

This, I’ve reported to them before, but basically the crux of the issue is their message notification emails.  The following is a screenshot of an actual email received from AliExpress.

Now, it would not matter what the email client was.  In this case, it’s Thunderbird, but the same problem would exist for Eudora, Outlook, Windows Mail, Apple Mail, The Bat!, Pegasus Mail … or any other email client you care to name.  If it runs outside the browser, that URI is invalid.  Protocol-relative means you use the same protocol as the page the hyperlink exists on.

In this case, the “protocol” used to retrieve that “page” was imap; imap://msg.aliexpress.com is wrong.  So is pop3://msg.aliexpress.com.  The only place I see this working, is on webmail sites.

Clearly, someone needs a clue-by-four to realise that not everybody uses a web browser to browse email.

Weak password requirements

When I signed up, boy where they fussy about the password.  My standard passwords are gibberish with punctuation… something AliExpress did not like.  They do not allow anything except digits and letters, and you must choose between 6 and 20 characters.  Not even XKCD standards work here!

Again, they aren’t the only ones… Suncorp are another mob that come to mind (in fact, they’re even more “strict”, they only allow 8… this is for their Internet banking… in 2018).  Thankfully the one bank account I have Internet banking on, is a no-fee account that has bugger all cash in it… the one with my savings in it is a passbook account, and completely separate.  (To their credit though, they do allow + in an email address.  They at least got that right.)

I can understand the field having some limit… you don’t want to receive two blu-ray discs worth of “password” every time a user authenticates themselves… but geez… would it kill you to allow 50 characters?  Does your salted hashing algorithm (you are using salted hashes aren’t you?) really care what characters you use?  Should you be using it if it does?  Once hashed, the output is going to be a fixed width, ideal for a database, and Bobby Tables is going to be hard pushed to pick a password that will hash to “‘; drop table users; –“.

By requiting these silly rules, they’ve actually forced me to use a weaker password.  The passwords I would have used on each site, had I been given the opportunity to pick my own, would have featured a much richer choice of characters, and thus been harder to break.  Instead, you’ve hobbled your own security.  Go team!

Reporting website issues is more difficult than it needs to be

Reporting a website issue is neigh on impossible.  Hence the reason for this post.  Plenty is there if I want to pick a fight with a seller (I don’t), or if I think there’s an intellectual property issue (this isn’t).  I eventually did find a form, and maybe they’ll do something about it, but I’m not holding my breath.

Forget to whitelist a script, and you get sworn at, in Mandarin

This is a matter of “unhappy code paths” not receiving the attention that they need.  In fact, there are a few places where they haven’t really debugged their l10n support properly and so the untranslated Alibaba pops up.

Yeah, the way China is going with global domination, we might some day find ourselves having to brush up on our Mandarin, and maybe Cantonese too… but that day is not today.

Anyway, I think that more or less settles it for now.  I’ll probably find more to groan about, but I do need to get some sleep tonight and go to work tomorrow.

Solar Cluster: New Router

So before going on the trip, I noticed the router I was using would occasionally drop off the network.  The switch still reported the link as being up, but the router would not respond to pings from the internal network.  If I SSHed into it from outside the network, and tried pinging internal IPs, it failed to ping them.

Something was up.  After much debugging (and some arguments about upgrades), it was decided that the hardware was flakey.  In that discussion, it was recommended that I have a look at PC Engines’ APU2 single board computer.

This is the only x86 computer I have seen with schematics and CoreBoot out-of-the-box, and it happens there’s a local supplier of them.  For sure, this machine is overkill for the job, but it ticks nearly all the boxes.

The only one it didn’t tick was being able to run directly from the battery.  As it happens, the unit only draws about 1.5A, and so a LM1085-12 LDO which can be sourced locally did the trick.  I basically put 100µF capacitors on the input and output, bolted it to a small heatsink and threw it all into a salvaged case.

After hooking it up to a bench supply (disconnected from the APU2) and winding the voltage right up to the PSUs maximum, and observing that the voltage stayed at 12V, I decided to hook it up and see how it went.  I plugged in my null modem cable, and sure enough, I was staring at CoreBoot.

I PXE-booted OpenBSD 6.3 and installed that onto the SD card, this was fairly painless and before long, the machine was booting on its own. I copied across the configuration settings from the old one, set up sniproxy, and I was in business, it was time to issue a `shutdown -p now` to both machines and for them to swap places.

Of course, a nicety of this box is there’s three Ethernet ports, so room for a move to another Internet connection, such as the HFC we’re supposed to be getting in this part of Brisbane (sadly, no thin pieces of glass for us), so in theory, I can run both in parallel and migrate between them.

A review of the iSquare Mobility Kite v1

Recently, a new project sprang up on the Hackaday.io site; it was for the KiteBoard, an open-source cellular development platform.  In a nutshell, this is a single-board-computer that embeds a full mobile system-on-chip and runs the Android operating system.  The project is seeking crowd funding for the second version of this platform.

With it, you can build smartphones (of course), tablets, tele-presence robots, or really, any project which can benefit from a beefy CPU with a built-in cellular modem.  It comes as a kit, which you then assemble yourself.  The level of difficulty in assembly is no greater than that of assembling a desktop PC: the circuit boards are pre-populated, you just need to connect them together.  In this version, some soldering of pushbuttons and wires is needed: all through-hole components.  No reflow ovens or solder paste is necessary here, an 8-year-old could do it.

The break-out board for the CPU card features in addition to connections for all the usual cellular phone signals (e.g. earpiece, microphone, button inputs) a GPIO header that follows the de-facto standard “Raspberry Pi” interface, allowing many Raspberry Pi “hats” to plug directly into this board.

That lends itself greatly to expandability.  Want a eInk or OLED notification display on the back?  A scrolling LED display?  A piano?  A games console?  Knock yourself out!  You, are the designer, you decide.  There are lots of options.

I for one, would consider an amateur radio transceiver, an external antenna socket and a beefier battery.  Presently, I get around with the ZTE T83 (“Telstra Dave”), which works okay, but as it runs an old version of Android (4.1), running newer applications on it is a problem.  I believe it could run something newer, but ZTE believe that their job was finished in 2013 when the first one rolled off the production line.

The box did not include a copy of the kernel sources or any link to where that could be obtained.  (GNU GPL v2 section 2b?  What’s that?)

The successor, the T84 is a little better, in fact it has pretty much the same hardware that’s in Kite, but it struggles in rural areas.  On a recent trip into the Snowy Mountains, my phone would be working fine, when my father’s T84 would report “no service available”.  Clearly, someone at Telstra/ZTE screwed up the firmware on it, and so it fails to switch networks correctly.  Without the sources, we are unable to fix that.  Even something as simple as replacing a battery is neigh on impossible, they’re built like bombs: not designed to be taken apart.

I have no desire to spend money on a company that puts out poorly supported rubbish running pirated operating system kernels.  The story is similar elsewhere, and most devices while better in specs and operating system, lack the external antenna connection that I desire in a phone.

Kite represents a breath of fresh air in that regard.  It is to smart phones, what the Raspberry Pi is to single board computers in general.  It’s not only designed to be taken apart, it’s shipped to you as parts.  Apparently with Kite v2, there’ll be schematics available, so you’ll be able to look-up the datasheets of respective components and be able to make informed decisions about part substitutions.  All antenna connections are socketed, so you can substitute at will.

While the OS isn’t going to be as open as one might like (mobile chipset manufacturers like their black boxes), it’s a BIG step in the right direction.  There’s more scope for supporting this platform long-term, than contemporary ones.

As far as actually using Kite, Shree Kumar was generous enough to organise the loan of a Kite for me to test with the Australian networks.  The phone takes up to two micro-SIMs (about 15mm×12mm); one on the daughter card (this is SIM 1) and one on the CPU card (SIM 2).

For the sake of testing, I figured I’d try it out with the two major networks, Telstra and Optus.  As it happens, my Telstra SIM is too big (they call it a “full-size” SIM now; I remember full-size SIMs being credit-card sized), so rather than chopping up my existing SIM or getting it transferred, I bought and activated a prepaid service.  I also bought a SIM for Optus.  I bought $10 credit for each.

As it happens, the Optus one came with data, the Telstra did not.  No big deal in this case.  The phone does have a limitation in that it will talk to one 3G/4G network and one GSM (2G) network at a time.  Given both networks I chose have abandoned 2G, that pretty much means the dual-SIM functionality on this model is severely hobbled.  That said, either SIM can operate in 3G mode, and so it’s simple enough to switch one SIM into 2G mode then activate the other in 3G/4G mode.  So far, the Kite has spent most of its time on Optus.

Evidently Vodaphone still have a 2G network… at least the Kite does see one 2G cell operated by them.  Long term, this is a problem that all dual-SIM phone chipset makers will have to deal with, a future Kite may well be able to do 3G simultaneously on both SIMs, but for me, this is not a show-stopper.

I’ve put together this review of the Kite.  It’s rare for me to be in front of a camera instead of behind it, and yes, the editing is very rough.  If there is time (there won’t be this weekend) I hope to take the phone out to a rural area and try it out with the more distant networks, but so far it seems happy enough to switch to 3G when I get home, and use 4G when I’m at work, so this I see as a promising sign.

The KickStarter is lagging behind quite a way in the funding goal, but alternate options are being considered for getting this project off-the-ground.  Here’s hoping that the project does get up, and that we get to see Kite v2 being developed and made for real, as I think the mobile phone industry really does need a viable open competitor.

Solar Cluster: arm-unknown-linux-musleabi… saga part IV

So, at long last, I finally saw this in my chroot‘s /var/log/emerge.log:

1524887925: Started emerge on: Apr 28, 2018 03:58:45
1524887926:  *** emerge --oneshot sys-devel/gcc::musl
1524888211:  >>> emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1524888212:  === (1 of 1) Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1524888307:  === (1 of 1) Compiling/Packaging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472690:  === (1 of 1) Merging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472838:  >>> AUTOCLEAN: sys-devel/gcc:7.3.0
1525473358:  === (1 of 1) Post-Build Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525473358:  ::: completed emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1525473360:  *** Finished. Cleaning up...
1525473373:  *** exiting successfully.

That’s 6 days, 18 hours and 32 minutes, of solid compiling. BUT WE GOT THERE!

What’s left? This:

Calculating dependencies... done!
[ebuild     U  ] sys-libs/musl-1.1.19 [1.1.18]
[binary   R    ] sys-libs/zlib-1.2.11-r1
[binary   R    ] app-arch/xz-utils-5.2.3
[ebuild     U  ] sys-libs/ncurses-6.1-r2 [6.0-r1]
[binary   R    ] sys-libs/readline-7.0_p3
[binary   R    ] virtual/libintl-0-r2
[binary   R    ] dev-lang/python-exec-2.4.5
[binary   R    ] virtual/libiconv-0-r2
[binary   R    ] sys-apps/gentoo-functions-0.12
[binary   R    ] dev-libs/libpcre-8.41-r1
[binary   R    ] sys-apps/sed-4.2.2
[binary   R    ] app-arch/bzip2-1.0.6-r8
[binary   R    ] dev-libs/gmp-6.1.2
[binary   R    ] app-shells/bash-4.4_p12
[binary   R    ] sys-apps/file-5.32
[binary   R    ] sys-devel/gnuconfig-20170101
[binary   R    ] dev-libs/mpfr-3.1.6
[binary   R    ] app-misc/c_rehash-1.7-r1
[binary   R    ] app-misc/mime-types-9
[binary   R    ] app-arch/tar-1.29-r3
[binary   R    ] app-arch/gzip-1.8
[binary   R    ] dev-libs/mpc-1.0.3
[binary   R    ] sys-devel/gcc-config-1.8-r1
[binary   R    ] app-misc/editor-wrapper-4
[binary   R    ] sys-apps/less-529
[binary   R    ] sys-apps/debianutils-4.8.3
[binary   R    ] net-libs/libmnl-1.0.4
[binary   R    ] sys-libs/libseccomp-2.3.2
[binary   R    ] dev-libs/popt-1.16-r2
[binary   R    ] sys-libs/e2fsprogs-libs-1.43.6
[binary   R    ] sys-devel/binutils-config-5-r4
[binary   R    ] dev-libs/libffi-3.2.1
[binary   R    ] virtual/libffi-3.0.13-r1
[binary   R    ] sys-apps/sysvinit-2.88-r9
[binary   R    ] sys-apps/opentmpfiles-0.1.3
[binary   R    ] virtual/tmpfiles-0
[binary   R    ] app-text/manpager-1
[binary   R    ] sys-libs/cracklib-2.9.6-r1
[binary   R    ] sys-apps/install-xattr-0.5
[binary   R    ] app-editors/nano-2.8.7
[binary   R    ] app-portage/elt-patches-20170815
[binary   R    ] sys-devel/m4-1.4.17
[binary   R    ] app-arch/unzip-6.0_p21-r2
[binary   R    ] sys-devel/autoconf-wrapper-13
[binary   R    ] sys-devel/bison-3.0.4-r1
[binary   R    ] sys-devel/flex-2.6.4-r1
[binary   R    ] dev-libs/libltdl-2.4.6
[binary   R    ] sys-devel/automake-wrapper-10
[binary   R    ] app-text/sgml-common-0.6.3-r6
[binary   R    ] dev-libs/libgpg-error-1.27-r1
[ebuild  N     ] dev-lang/perl-5.24.3-r1  USE="-berkdb -debug -doc -gdbm -ithreads"
[ebuild  N     ] sys-kernel/linux-headers-4.13  USE="-headers-only"
[ebuild  N     ] virtual/perl-Data-Dumper-2.160.0-r1
[ebuild  N     ] virtual/perl-Test-Harness-3.360.100_rc-r3
[ebuild  N     ] perl-core/File-Temp-0.230.400-r1
[ebuild  N     ] virtual/perl-File-Temp-0.230.400-r5
[ebuild  N     ] perl-core/File-Path-2.130.0
[ebuild  N     ] virtual/perl-File-Path-2.130.0
[binary   R    ] virtual/os-headers-0
[ebuild  N     ] sys-devel/autoconf-2.69-r4  USE="-emacs"
[ebuild  N     ] sys-apps/attr-2.4.47-r2  USE="-nls -static-libs"
[ebuild   R    ] sys-apps/coreutils-8.28-r1
[ebuild     U  ] app-admin/eselect-1.4.12 [1.4.8]
[ebuild     U  ] app-eselect/eselect-python-20171204 [20160516]
[ebuild     U  ] sys-devel/patch-2.7.6-r1 [2.7.5]
[ebuild  N     ] sys-apps/shadow-4.5  USE="cracklib xattr -acl -audit -nls -pam (-selinux) -skey"
[binary   R    ] virtual/shadow-0
[ebuild  N     ] virtual/perl-ExtUtils-MakeMaker-7.100.200_rc-r4
[ebuild  N     ] sys-libs/libcap-2.24-r2  USE="-pam -static-libs"
[ebuild  N     ] dev-perl/Text-Unidecode-1.270.0
[ebuild  N     ] dev-perl/libintl-perl-1.240.0-r2
[ebuild  N     ] sys-apps/help2man-1.47.4  USE="-nls"
[ebuild  N     ] sys-devel/automake-1.15.1-r2  USE="{-test}"
[ebuild  N     ] sys-devel/libtool-2.4.6-r3  USE="-vanilla"
[ebuild  N     ] dev-libs/expat-2.2.5  USE="unicode -examples -static-libs"
[ebuild   R    ] sys-process/psmisc-22.21-r3
[ebuild  N     ] sys-libs/gdbm-1.13-r2  USE="readline -berkdb -exporter -nls -static-libs"
[ebuild  N     ] sys-apps/groff-1.22.2  USE="-X -examples" L10N="-ja"
[ebuild  N     ] dev-libs/libelf-0.8.13-r2  USE="-debug -nls"
[ebuild  N     ] virtual/libelf-2
[ebuild  N     ] dev-libs/libgcrypt-1.8.1  USE="-doc -static-libs"
[ebuild  N     ] dev-perl/XML-Parser-2.440.0
[ebuild  N     ] virtual/perl-File-Spec-3.630.100_rc-r4
[ebuild  N     ] dev-perl/Unicode-EastAsianWidth-1.330.0-r1
[ebuild  N     ] sys-apps/texinfo-6.3  USE="-nls -static"
[ebuild  N     ] dev-libs/iniparser-3.1-r1  USE="-doc -examples -static-libs"
[ebuild  N     ] app-portage/portage-utils-0.64  USE="-nls -static"
[ebuild  N     ] dev-libs/openssl-1.0.2o  USE="asm sslv3 tls-heartbeat zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -static-libs {-test} -vanilla"
[binary  N     ] dev-lang/python-2.7.14-r1  USE="ipv6 ncurses readline ssl (threads) (wide-unicode) xml (-berkdb) -build -doc -examples -gdbm -hardened -libressl -sqlite -tk -wininst"
[binary  N     ] sys-apps/openrc-0.34.11  USE="ncurses netifrc unicode -audit -debug -newnet -pam (-prefix) (-selinux) -static-libs" 
[ebuild  N     ] net-misc/netifrc-0.5.1
[binary   R    ] sys-apps/grep-3.0
[binary   R    ] sys-apps/findutils-4.6.0-r1
[binary   R    ] sys-apps/kbd-2.0.4
[ebuild  N     ] sys-apps/busybox-1.28.0  USE="ipv6 static -debug -livecd -make-symlinks -math -mdev -pam -savedconfig (-selinux) -sep-usr -syslog (-systemd)"
[binary   R    ] virtual/service-manager-0
[binary   R    ] sys-devel/binutils-2.29.1-r1
[ebuild  N     ] sys-apps/net-tools-1.60_p20161110235919  USE="arp hostname ipv6 -nis -nls -plipconfig (-selinux) -slattach -static" 
[binary   R    ] sys-apps/gawk-4.1.4
[binary   R    ] virtual/editor-0
[binary   R    ] sys-devel/make-4.2.1
[binary   R    ] sys-process/procps-3.3.12-r1
[binary   R    ] virtual/dev-manager-0-r1
[binary   R    ] sys-apps/which-2.21
[ebuild  N     ] net-misc/iputils-20171016_pre  USE="arping filecaps ipv6 openssl ssl -SECURITY_HAZARD -caps -clockdiff -doc -gcrypt
 (-idn) -libressl -nettle -rarpd -rdisc -static -tftpd -tracepath -traceroute"
[binary   R    ] virtual/pager-0
[binary   R    ] sys-apps/diffutils-3.5
[binary   R    ] sys-apps/baselayout-2.4.1-r2
[binary   R    ] virtual/libc-1
[binary   R   ~] sys-devel/gcc-7.3.0
[binary   R    ] virtual/pkgconfig-0-r1
[ebuild  N     ] dev-lang/python-3.5.5  USE="ipv6 ncurses readline ssl (threads) xml -build -examples -gdbm -hardened -libressl -sqlite {-test} -tk -wininst"
[ebuild  N     ] app-misc/ca-certificates-20170717.3.36.1  USE="-cacert -insecure_certs"
[ebuild  N     ] sys-apps/util-linux-2.30.2-r1  USE="cramfs ncurses readline suid unicode -build -caps -fdformat -kill -nls -pam -python (-selinux) -slang -static-libs (-systemd) {-test} -tty-helpers -udev" PYTHON_SINGLE_TARGET="python3_5 -python2_7 -python3_4 -python3_6" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild     U  ] app-misc/pax-utils-1.2.3 [1.1.7]
[ebuild     U  ] sys-apps/sandbox-2.13 [2.10-r4]
[ebuild     U  ] net-misc/rsync-3.1.3 [3.1.2-r2]
[ebuild  N     ] net-firewall/iptables-1.6.1-r3  USE="ipv6 -conntrack -netlink -nftables -pcap -static-libs"
[ebuild     U  ] dev-libs/libpipeline-1.4.2 [1.4.0]
[ebuild  N     ] sys-apps/man-db-2.7.6.1-r2  USE="gdbm manpager zlib -berkdb -nls (-selinux) -static-libs"
[ebuild     U  ] sys-apps/kmod-24 [23] PYTHON_TARGETS="-python3_6%"
[ebuild  N     ] dev-python/pyblake2-1.1.0  PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] net-misc/openssh-7.5_p1-r4  USE="hpn pie ssl -X -X509 -audit -bindist -debug -kerberos -ldap -ldns -libedit -libressl -livecd -pam -sctp (-selinux) -skey -ssh1 -static {-test}"
[ebuild  N     ] dev-util/gtk-doc-am-1.25-r1
[ebuild  N     ] dev-libs/libxml2-2.9.7  USE="ipv6 readline -debug -examples -icu -lzma -python -static-libs {-test}" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild  N     ] sys-devel/gettext-0.19.8.1  USE="cxx ncurses openmp -acl -cvs -doc -emacs -git -java (-nls) -static-libs"
[ebuild  N     ] app-text/build-docbook-catalog-1.19.1
[ebuild  N     ] dev-libs/libxslt-1.1.30-r2  USE="crypt -debug -examples -python -static-libs" PYTHON_TARGETS="python2_7"
[ebuild  N     ] app-text/docbook-xsl-stylesheets-1.79.1-r2  USE="-ruby"
[ebuild  N     ] app-text/docbook-xml-dtd-4.1.2-r6
[ebuild  N     ] dev-util/intltool-0.51.0-r2
[ebuild  N     ] dev-libs/glib-2.52.3  USE="mime xattr -dbus -debug (-fam) (-selinux) -static-libs -systemtap {-test} -utils" PYTHON_TARGETS="python2_7"
[ebuild  N     ] x11-misc/shared-mime-info-1.9  USE="{-test}"
[ebuild  N     ] dev-python/setuptools-36.7.2  USE="{-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/certifi-2017.4.17  PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/pyxattr-0.5.5  USE="-doc {-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4"
[ebuild  N     ] sys-apps/portage-2.3.24-r1  USE="(ipc) native-extensions xattr -build -doc -epydoc -gentoo-dev (-rsync-verify) (-selinux)" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] app-admin/perl-cleaner-2.25
[binary   R    ] virtual/man-0-r1
[binary   R    ] virtual/modutils-0
[ebuild  N     ] sys-fs/e2fsprogs-1.43.6  USE="-fuse (-nls) -static-libs"
[ebuild     U  ] virtual/package-manager-1 [0]
[ebuild  N     ] sys-apps/iproute2-4.14.1-r2  USE="iptables ipv6 -atm -berkdb -minimal (-selinux)"
[binary   R    ] virtual/ssh-0
[ebuild  N     ] net-misc/wget-1.19.1-r2  USE="ipv6 pcre ssl zlib -debug -gnutls -idn -libressl -nls -ntlm -static {-test} -uuid"
[ebuild   R    ] dev-util/pkgconfig-0.29.2  USE="-internal-glib*"

!!! The following binary packages have been ignored due to non matching USE:

    =dev-util/pkgconfig-0.29.2 internal-glib
    =sys-apps/attr-2.4.47-r2 nls
    =sys-apps/man-db-2.7.6.1-r2 nls
    =dev-libs/libelf-0.8.13-r2 nls
    =sys-apps/shadow-4.5 -linguas_cs -linguas_da -linguas_de -linguas_es -linguas_fi -linguas_fr -linguas_hu -linguas_id -linguas_it -linguas_ja -linguas_ko -linguas_pl -linguas_pt_BR -linguas_ru -linguas_sv -linguas_tr -linguas_zh_CN -linguas_zh_TW nls

NOTE: The --binpkg-respect-use=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-respect-use=y will silence this warning.

I think that’s broken the back of the job.  Of course when I come to running Catalyst, I’ll have to do it all over again, but at least now the environment is clean.

Solar Cluster: Next steps, better control of the charger

So, a few weeks ago I installed a new battery charger, and tweaked it so that the solar did most of the leg work during the day, and the charger kept the batteries topped up at night.

I also discussed the addition of a new industrial PC to perform routing and system monitoring functions… which was to run Gentoo Linux/musl. For now, that little PC is still running Debian Stretch, but for 45 days, it was rock solid. The addition of this box, and taking on the role of router to the management network meant I could finally achieve one of my long-term goals for the project: decommissioning the old server.

The old server is still set up with all my data and software… but now the back-up cron job calls /sbin/poweroff when it’s done, and the BIOS is set to wake the machine up in the evening ready to receive a back-up late at night.

In its place, a virtual machine clone of the box, handles my email and all the old functions of that server. This was all done just prior to my father and I leaving for a 3 week holiday in the Snowy Mountains.

I did have a couple of hiccups with Ceph OSDs crashing … but basically re-starting the daemons (done remotely whilst travelling through Cowra) got everything back up. A bit of placement group cleaning, and everything was back online again. I had another similar hiccup coming out of Maitland, but once again, re-starting the daemons fixed it. No idea why it crashed, that’s something I’ll have to investigate.

Other than that, the cluster itself has run well.

One thing that did momentarily kill the industrial PC though: I wandered down to the rack with a small bus-powered 2.5″ HDD with the intent of re-starting my Gentoo builds. This HDD had the same content as the 3.5″ HDD I had plugged in before. I figured being bus powered, I would not be dependent on mains, and it could just chug away to its heart’s content.

No such luck, the moment I plugged that drive in, the little machine took great umbrage to the spinning rust now vacuuming the electrons away from its core functions, and shut down abruptly. I’ve now brought my 3.5″ drive and dock down, plugged that into the wall, and have my builds resuming. If power goes off, hopefully the machine either handles the loss of swap gracefully. If it does crash, the watchdog will take care of it.

Thus, I have the little TS-7670 first attempting a build of gcc, to see how we go. Finger’s crossed our power should remain up. There was at least one outage in the time we were away, but hopefully we should get though this next build!

The next step I think should be to add some control of the mains charger to allow the batteries to be boosted to full charge overnight. The thinking is a simple diode-OR arrangement. Many comparators such as the LM393 have an open-collector output, which gives us this for free.

The theory is this.

The battery bank powers a simple circuit which runs of a 5V regulator. That regulator powers a dual comparator IC and provides a reference voltage. The comparator draws bugger all power, so I’m happy to use a linear PSU here. It’s mainly there as a voltage reference.

Precision isn’t really the aim here, so adjustable pots will make life easier.

The voltages from the battery bank and the solar panel are fed through voltage dividers to bring the voltages down to below 5V, then those voltages are individually fed into separate pots that control the hysteresis. I can adjust all points of the system.

The idea is that should the batteries get too low, or the sun go down, one or the other (or both) comparators will go low and pull down on R2. If the batteries are high and the sun is up, nothing pulls on R2 so the REMOTE+ pin on the HEP-600C-12 is allowed to float to +5V, turning off the mains charger.

The advantage of this is there’s no programming of a microcontroller, it’s just analogue electronics. The LM393s are pretty hardy things, the datasheet says they’ll run at 36V and can accept a maximum voltage of VCC-1.5V; so if I run at 5V, 3.5V is my recommended maximum. The adjustment pots should let me set a threshold voltage that avoids going above this.

I mainly need 5V for the HEP-600C-12, and for providing that stable known voltage reference. The LM78C05 should be fine for this.

Once I’ve done that, I should be able to wind that charger back up to its factory setting of 14.4V, which will mean that overnight the batteries will be charged back to full charge.