Public Syndication

Toy Synthesizer: Never forget thy series resistor!

So, I’ve gotten back to this project having spent a lot of my time on work, the Yarraman to Wulkuraka bike ride and the charging controller #Solar-powered cloud computing — just to name 3 things vying for my attention.

In the test board, I had wired up some LEDs for debugging, dead-bugged 0805s, which were hooked between the output of the octal latch and 0V.  I omitted the series resistor, as I presumed that, given the output was PWMed with a maximum duty cycle of ⅛, the LEDs shouldn’t burn out.

Turns out I had forgotten a property that all diodes exhibit, that is the desire to clamp the voltage across them.  Today I was testing the board, and wondering why some channels were dim, others didn’t work at all, but one worked so much better.  Did I accidentally put the wrong current limiting resistor in series with the drain?  No, all checked out as about 12 ohms.

I put a program on the MCU that just turned a channel on when the button was pressed.  No music, no fancy PWM stuff, just turn on a LED when the corresponding button was pressed.  Measuring the gate voltage showed about 2V.

Even with the PWM output forced low, the output was still 2V.  Moreover, I was using my new bench supply, and with nothing running, the circuit was drawing ~300mA!  Why?

Turns out, the LEDs I had dead-bugged in, were trying, and succeeding, in clamping the output voltage.  2V was just barely enough to trigger the output MOSFETs, but clearly this was borderline as some worked better than others.  I was likely in the linear region.

Snip out the common connection for the LEDs to 0V, and the problems disappeared.  I’ve dead-bugged a 1kOhm resistor in series with the lot, and that’s got my debug LEDs back and working again.  The MOSFET outputs now work properly.

The bigger chunkier MOSFETs I bought by mistake could have worked just fine: maybe I was just driving them wrong!

Two prospects have crossed my mind:

  • Getting the MOSFET board made professionally
  • Getting a board that combines all components onto one PCB made professionally

The version that is shown was really designed for the home PCB maker to be able to produce.  The traces are wide and the board is fundamentally single-sided: when etching, you just etch one side of the board and leave the other side unetched.  When drilling the holes, you just countersink the holes a bit on all pins not connected to 0V.

A smaller board with everything in one would be worth making now that I’ve proven the concept.  Not sure there’s a good reason to go to SMT at this stage: I still want to make assembly simple.  The thinking is the all-in-one would have some headers so you can conceivably break things out for other projects and just omit parts as required.

This could theoretically be entered into the #The 2018 Hackaday Prize as part of the musical instrument contest, as that’s what it is: it’s a musical instrument for the severely physically handicapped.  There is a video of a slightly earlier prototype in this post .

Code wise, I’ve done little.  The basic functionality is there, it makes noises, it flashes LEDs, that’s about what it needs to do for now.  I did have to increase the start-up delay so that the buttons were detected properly, as without this, if I used my bench-top supply, it would fail to see any inputs.  People aren’t going to notice 100ms boot-up delay vs 1ms, but it makes a difference if the power supply is a little slow.

Eurocrats and governance over-reach

Politicians and bureaucrats, aren’t they wonderful?  They create some of the laws that are the cornerstone of our civilisation.  We gain much stability in the world from their work.

Many are often well versed in law, and how the legal systems of the world, work.  They believe that their laws are above all overs.

So much so, they’ll even try to legislate the ratio of a circle’s circumference from its diameter.  Thankfully back then, others had better common sense.

They legislated for websites to display a banner on their pages that people have to click, telling the user that the website uses cookies for XYZ purpose.  Now, I have never set foot in Europe, I really don’t have any desire to leave Australia for that matter.  I am not a European citizen.  I do not use a VPN for accessing foreign websites: they see my Australian IP address.

In spite of this, now every website insists on pestering me about a law that is not in force here.  You know what?  You can disable cookies.  It is a feature of web browsers.  Even NCSA Mosaic, Netscape Navigator and the first versions of Internet Explorer (which were dead ringers for NCSA’s browser by the way), had this feature.  I’m talking mid-90s era browsers … and every descendent thereon.

It’d be far more effective for the browser to ask if XYZ site was allowed to set a cookie, but no, let’s foist this burden onto the website owner.  I don’t doubt people abuse this feature for various nefarious purposes, but a solution this is not!

It gets better though.  To quote the EFF (Today, Europe Lost The Internet. Now, We Fight Back):

Today, in a vote that split almost every major EU party, Members of the European Parliament adopted every terrible proposal in the new Copyright Directive and rejected every good one, setting the stage for mass, automated surveillance and arbitrary censorship of the internet: text messages like tweets and Facebook updates; photos; videos; audio; software code — any and all media that can be copyrighted.

Three proposals passed the European Parliament, each of them catastrophic for free expression, privacy, and the arts:

1. Article 13: the Copyright Filters. All but the smallest platforms will have to defensively adopt copyright filters that examine everything you post and censor anything judged to be a copyright infringement.

Yep, this is basically much like China’s Great Firewall, just outsourced.

It actually has me thinking about whether it is possible to detect if a given HTTP client is from the EU, and respond back with a HTTP error 451, because doing business in the EU is just too dangerous legally.

Solar Cluster: Power controller installed

Well, I finally got around to installing that power controller.

Yes, the top of that rack is getting to be a pigsty. Even the controller isn’t my best work:

You can see above I’ve just tacked wires onto the points I need and brought those out to a terminal strip.  There’s provision there for some PWM-controlled fans, but right now this is unused.  I’ve omitted the parts not required for the application.  If this works out, I might consider doing another board, this time better dedicated to the task at hand.

With that controller in place, I’ve now wound the charger back up.  In fact I made a whoopsie at first: I forgot that the Vout pot on the HEF-600C-12 sets the float voltage and wound that right up to 14.4V which meant a boost voltage of 15V!

Thankfully I looked over at the volt meter on the solar controller and realised my mistake quick.  15 seconds won’t hurt anything, but it is now set at 13.6V.  You don’t even see it on the 40-sample average.  The controller should let the mains charger sit there for an hour before it reconsiders the need for mains.

I think my next step … there’s a yard that could do with a hair cut… I’ll drag the mower out and chase that around the yard for a bit.  Then we’ll see what it looks like.


Okay, back from a little mowing… and sure enough, the controller is mostly doing the right thing.  I think I’ll need to tweak some set-points, maybe set the solar threshold lower.  Thankfully the “inhibit” LED is just an indication that it considers the solar voltage low, the solar is going to be on no matter what.

Yes, that SSR is massive for the job. It’s what I had on hand at the time. I’ll probably replace it with something small, maybe a reed relay since they’re cheap.

Right at this point, I have the SSR’s inputs connected between the solar V+ and the BC-547B on the board, so when the sun *does* go down, the mains power will be turned back on no matter what the controller thinks.

A close-up look of the status LEDs shows what mode we’re in:

We’ll ignore the temperature ones. Ultimately they indicate the state of the fan controller, and in this state, it’d be running the fans, as indicated by the Fan PWM LED to the left. Temperature is measured by the sensor in the ATTiny24A on-board, so not highly accurate.

The other mystery “LED” is the shiny surface on the BC-547B to the left of the two source status LEDs.

Here, I suspect the sun ducked behind a cloud so the voltage dropped, hence both “inhibit” LEDs came on. Earlier, “Float” was lit (you can sort-of make it out in the previous photo), the charger was actually actively trying to charge the battery, but to the controller it looked to be done. It left it go for the hour as programmed, then turned off the mains charger to let the solar panel take over.

The idea is that during the day, if it gets low, give it a boost from mains, then go back to solar. We only want to rely on mains at night.

Now, it should stay in that state until tonight, when the lack of sun should bring the mains charger online (by sheer fact that the solar panels power the “coil” of the relay).


So, I saw that, and had a look… sure enough, the controller is still asserting that the mains charger should be off. I think I need to bump the battery thresholds up a bit, although that’s still safely above the danger zone, it’s lower than I’m comfortable with.

Right up until 5:58PM, it seems the MCU just held on, thinking the battery voltage was “good enough”, so no need for a charge yet. I might want to drop the solar threshold down some so it doesn’t “flap” when broken cloud passes over, then raise the minimum battery threshold a bit.

Even now, the thing that’s turning the mains charger on is the fact that the 1.5V coming off the panels is not sufficient for activating the solid-state relay I’m using. I’m thankful I wired the SSR to Vsolar and made the MCU output open-collector. This is a useful little safety feature, making it impossible for the MCU to latch-up and hold the mains charger off, as the sun will eventually set, and that will force the mains charger to turn on like it did tonight.

Solar Cluster: Thank goodness for good monitoring

A few months back now, I had the misfortune of overshooting my Internet quota, and winding up with a AU$380 bill for the month (and that was capped… in truth it was more like AU$3000).  In fact, it happened a couple of times until I finally nailed down the cause.

Part of it was NTP traffic (seems lots of cowboys write SNTP clients now and point them at pool.ntp.org), some was the #Hackaday.io Spambot Hunter Project and related activity.  In short, I invested some money into upping the quota, and some time into better monitoring.

I wanted to do the monitoring anyway to keep an eye on operations, as well as things like the solar panel voltages, etc.  Since I got it in place, I’ve been able to get much faster notifications of when things go awry.  Much sooner than the 120% quota usage alarm that Internode sends you.

I’m glad I did that now, last night I left a few tabs open on the Hackaday.io site.  I noticed this evening they were still trying to load something and got suspicious… then I saw this:

Double checking, sure enough, something on one of those pages made Chromium get its knickers into a twist, and chew through all that data.

It took me a bit of tinkering to get the right query to extract the above chart.  Essentially there was a sustained 1.5MB/sec download for over 21 hours which would account for the 113.1GB that Internode recorded.

It’s a bit co-incidental that the usage dropped the moment I re-started Chromium.  Not sure why it was continually re-loading pages, but never mind.

The above data is collected using a combination of collectd and InfluxDB, with Grafana doing the dashboarding and alarms, and a small Perl script pulling the usage data off Internode’s API.

Sunsetting the yi.org domains

Update: Real life intervened and I didn’t get around to reconfiguring Postfix like I said I would. I am doing that now, any longlandclan.yi.org or vk4msl.yi.org will start bouncing now.

Originally, when I started down the path of running my own server, I was an unemployed student, so the servers were hand-me-down second hand affairs and the domains I used were freebie ones.  I started out with no-ip.com, and when they changed their policies, I switched to yi.org (in 2007).

yi.org have been fantastic.  Not only is the domain short, but they also allow many record types including TXT (needed for SPF rules), AAAA (IPv6), MX (for mail servers) and NS, yes they’ll even let you delegate a subdomain to DNS servers of your choosing.

That said, they did have a spot of unreliability a few years back (around 2015).  Given that I now have an income of my own, it no longer made sense to just go for free services, so I bought a couple of id.au domains.  My email client was configured so that in the event someone sent me an email to the old address, they would see the following in my reply:

Reply-To: user@longlandclan.id.au
Subject: Re: …
References: <…>
To: …
From: "Stuart Longland (OLD ADDRESS see reply-to)"
 <user@longlandclan.yi.org>

I’ve been doing this for a few years now.  The yi.org domains now only receive mail that comes into two categories:

  • people who still use the old email address not realising I’ve changed
  • spammers that have harvested the old email address

From October November this year, I’ll be bouncing emails sent to the old domain, with a link to this page.  From November this year In 2019, the MX records for the yi.org domains will disappear.  In short I will not be receiving email from any of the old yi.org addresses from November 2018!

If you see yi.org in an email address for one of us, now is the time to replace that with id.au.  If you see one ending in hopto.org, then you are really out of date.

Please note, I might be able to give you instructions on how to update the email address in your client, but I’ll assume your client works exactly the same as mine does to the pixel, my instructions will be something along the lines of “Go to the Tools menu, click Address Book, type ‘yi.org’ into Criteria, press ENTER, then for each contact you see with my old address in it, double-click on it, change the address and click OK”.

I provide 0 support for email clients I don’t use: I’ll assume you know how to use your system or know how to research the problem, it’s not up to me to teach you as life’s too short.

Solar Cluster: Return of the power controller

Well, I’ve been tossing up how to control the mains charger for a while now.

When I first started the project, my thinking was to use an old Xantrex charger we had kicking around, and just electrically disconnect it from the batteries when we wanted to use the solar power.  I designed a 4-layer PCB which sported a ATTiny24A microcontroller, MOSFETs (which I messed up) and some LEDs.

This was going to be a combined fan controller and power management unit.  It had the ability (theoretically) to choose a supply based on the input voltage, and to switch if needed between supplies.

It didn’t work out, the charger got really confused by the behaviour of the controller.  I was looking to re-instate it using the Redarc solar controller, but I never got there.  In the end, it was found that the Redarc controller had problems switching sources and would do nothing whilst the batteries went flat.

We’ve now replaced both ends of the system.  The solar controller is a Powertech MP3735 and integrates over-discharge protection.  The mains charger is now a MeanWell HEP-600C-12 (which has not missed a beat since the day it was put in).

Unlike my earlier set-up, this actually has a 5V logic signal to disable it, and my earlier controller could theoretically generate that directly.

Looking at the PCB of my earlier power controller attempt, it looks like this could still work.

Above is the PCB artwork.  I’ve coloured in the sections and faded out the parts I can omit.

In green up the top-left we have the mains control/monitoring circuitry.  We no longer see the mains voltage, so no point in monitoring it, so we can drop the resistor divider that fed the ADC.  This also means we no longer need the input socket P2.

Q2 and Q7 were the footprints of the two P-channel MOSFETs.  We don’t need the MOSFETs themselves, but the signals we need can be found on pin 1 of Q2.  This is actually the open-drain output of Q1, which we may be able to hook directly to the REMOTE+ pin on the charger.  A pull-up between there and the charger’s 5V rail, and we should be in business.

In yellow, bottom left is the solar monitoring interface.  This is still useful, but we won’t be connecting solar to the battery ourselves, so we just keep the monitoring parts.  The LED can stay as an indicator to show when solar is “good enough”.

In purple, occupying most of the board, is the controller itself.  It stays for obvious reasons.

In red, is the fan control circuitry.  No reason why this can’t stay.

In blue is the circuitry for monitoring the battery voltage.  Again, this stays.

The main advantage of doing this is I already have the boards, and a number of microcontrollers already present.  There’s a board with all except the big MOSFETs populated: with the MOSFETs replaced by 3-pin KK sockets.

How would the logic work?  Much the same as the analogue version I was pondering.

  • If battery voltage is low, OR, the sun has set, enable the mains charger.

What concerned me about an analogue solution was what would happen once the charger got to the constant-voltage stage.  We want to give it a bit of time to keep the battery topped up.  Thus it makes sense to shut down the charger after a fixed delay.

This is easy to do in a microcontroller.  Not hard with analogue electronics either, it’s fundamentally just a one-shot, but doing it with an MCU is a single-chip solution.  I can make the delay as long as I like.  So likely once the battery is “up to voltage”, I can let it float there for an hour, or until sunrise if it’s at night.

Solar Cluster: Kernel driver now up on Github

So, I’m happy enough with the driver now that I’ll collapse down the commits and throw it up onto the Github repository.  I might take another look at kernel 4.18, but for now, you’ll find them on the ts7670-4.14.67 branch.

Two things I observe about this voltage monitor:

  1. The voltage output is not what you’d call, accurate.  I think it’s only a 10-bit ADC, which is still plenty good enough for this application, but the reading I think is “high” by about 50mV.
  2. There’s significant noise on the reading, with noticeable quantisation steps.

Owing to these, and to thwart the possibility of using this data in side-channel attacks using power analysis, I’ve put a 40-sample moving-average filter on the “public” data.

Never the less, it’s a handy party trick, and not one I expected these devices to be able to do.  My workplace manages a big fleet of these single-board computers in the residential towers at Barangaroo where they spend all day polling Modbus and M-Bus meters.  In the event we’re at all suspicious about DC power supplies though, it’s a simple matter to load this kernel tree (they already run U-Boot) and configure collectd (which is also installed).

I also tried briefly switching off the mains power to see that I was indeed reading the battery voltage and not just a random number that looked like the voltage.  That yielded an interesting effect:

You can see where I switched the mains supply off, and back on again.  From about 8:19PM the battery voltage predictably fell until about 8:28PM where it was at around 12.6V.

Then it did something strange, it rose about 100mV before settling at 12.7V.  I suspect if I kept it off all night it’d steadily decrease: the sun has long set.  I’ve turned the mains charger back on now, as you can see by the step-rise shortly after 8:44PM.

The bands on the above chart are the alert zones.  I’ll get an email if the battery voltage strays outside of that safe region of 12-14.6V.  Below 12V, and I run the risk of deep-cycling the batteries.  Above 14.6V, and I’ll cook them!

The IPMI BMCs on the nodes already sent me angry emails when the battery got low, so in that sense, Grafana duplicates that, but does so with pretty charts.  The BMCs don’t see when the battery gets too high though, for the simple matter that what they see is regulated by LDOs.

Solar Cluster: Getting the battery voltage into Grafana

I’ve succeeded in getting a working battery monitor kernel module. This is basically taking the application note by Technologic Systems and spinning that into a power supply class driver that reports the voltage via sysfs.

As it happens, the battery module in collectd does not see this as a “battery”, something I’ll look at later. For now the exec plug-in works well enough. This feeds through eventually to an InfluxDB database with Grafana sitting on top.

https://netmon.longlandclan.id.au/d/IyZP-V2mk/battery-voltage?orgId=1

Solar Cluster: Kernel driver debugging

So, I successfully last night, parted the core bits out of ts_wdt.c and make ts-mcu-core.c.  This is a multi-function device, and serves to provide a shared channel for the two drivers that’ll use it.

Tonight, I took a stab at writing the PSU part of it.  Suffice to say, I’ve got work to do:

[  158.712960] Unable to handle kernel NULL pointer dereference at virtual address 00000005
[  158.721328] pgd = c3854000
[  158.724089] [00000005] *pgd=4384f831, *pte=00000000, *ppte=00000000
[  158.730629] Internal error: Oops: 1 [#3] ARM
[  158.734947] Modules linked in: 8021q garp mrp stp llc nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_tcpudp nf_conntrack_ipv6 nf_defrag_ipv6 xt_conntrack nf_conntrack ip6table_filter ip6_tables x_tables flexcan can_dev
[  158.755812] CPU: 0 PID: 2059 Comm: cat Tainted: G      D         4.14.67-vrt-ts7670+ #3
[  158.763840] Hardware name: Freescale MXS (Device Tree)
[  158.769008] task: c68f3a20 task.stack: c3846000
[  158.773598] PC is at ts_mcu_transfer+0x1c/0x48
[  158.778073] LR is at 0x3
[  158.780630] pc : []    lr : [<00000003>]    psr: 60000013
[  158.786918] sp : c3847e44  ip : 00000000  fp : 014000c0
[  158.792165] r10: c5035000  r9 : c5305900  r8 : c777b428
[  158.797412] r7 : c0a7fa80  r6 : c777b400  r5 : c5035000  r4 : c3847e6c
[  158.803961] r3 : c3847e58  r2 : 00000001  r1 : c3847e4c  r0 : c07b8c68
[  158.810512] Flags: nZCv  IRQs on  FIQs on  Mode SVC_32  ISA ARM  Segment none
[  158.817671] Control: 0005317f  Table: 43854000  DAC: 00000051
[  158.823440] Process cat (pid: 2059, stack limit = 0xc3846190)
[  158.829212] Stack: (0xc3847e44 to 0xc3848000)
[  158.833611] 7e40:          c05bd150 00000001 00010000 00000004 c3847e48 00000150 c5035000
[  158.841833] 7e60: c777b420 c05bcaa4 c4f70c60 c777e070 c0a7fa80 c05bca20 00000fff c07ad4b0
[  158.850051] 7e80: c777b428 c04e46dc c4f52980 00001000 00000fff c01b1994 c4f52980 c4f70c60
[  158.858268] 7ea0: c3847ec8 ffffe000 00000000 c3847f88 00000001 c0164eac c3847fb0 c4f529b0
[  158.866487] 7ec0: 00020000 b6e3d000 00000000 00000000 c4f73f70 00000800 00000000 c01b0f60
[  158.874703] 7ee0: 00020000 c4f70c60 ffffe000 c3847f88 00000000 00000000 00000000 c013eb84
[  158.882918] 7f00: 000291ac 00000000 00000000 c0009344 00000077 b6e3c000 00000022 00000022
[  158.891135] 7f20: c686bdc0 c0117838 000b6e3c c3847f80 00022000 c686be14 b6e3c000 00000000
[  158.899354] 7f40: 00000000 00022000 b6e3d000 00020000 c4f70c60 ffffe000 c3847f88 c013ed0c
[  158.907571] 7f60: 00000022 00000000 000b6e3c c4f70c60 c4f70c60 b6e3d000 00020000 c000a9e4
[  158.915786] 7f80: c3846000 c013f2d8 00000000 00000000 00000000 00000000 00000000 00000000
[  158.924002] 7fa0: 00000003 c000a820 00000000 00000000 00000003 b6e3d000 00020000 00000000
[  158.932217] 7fc0: 00000000 00000000 00000000 00000003 00020000 00000000 00000001 00000000
[  158.940434] 7fe0: be8a62c0 be8a62ac b6eb77c4 b6eb6b9c 60000010 00000003 00000000 00000000
[  158.948691] [] (ts_mcu_transfer) from [] (ts_psu_get_prop+0x38/0xb0)
[  158.956847] [] (ts_psu_get_prop) from [] (power_supply_show_property+0x84/0x220)
[  158.966036] [] (power_supply_show_property) from [] (dev_attr_show+0x1c/0x48)
[  158.974974] [] (dev_attr_show) from [] (sysfs_kf_seq_show+0x84/0xf0)
[  158.983129] [] (sysfs_kf_seq_show) from [] (seq_read+0xcc/0x4f4)
[  158.990930] [] (seq_read) from [] (__vfs_read+0x1c/0x11c)
[  158.998117] [] (__vfs_read) from [] (vfs_read+0x88/0x158)
[  159.005304] [] (vfs_read) from [] (SyS_read+0x3c/0x90)
[  159.012232] [] (SyS_read) from [] (ret_fast_syscall+0x0/0x28)
[  159.019766] Code: e52de004 e281300c e590e004 e25cc001 (e1dee0b2) 
[  159.026278] ---[ end trace 2807dc313991fd87 ]---

The good news is the machine didn’t crash.c

Solar Cluster: Battery voltage kernel module

So, I had a brief look after getting kernel 4.18.5 booting… sure enough the problem was I had forgotten the watchdog, although I did see btrfs trigger a deadlock warning, so I may not be out of the woods yet.  I’ve posted the relevant kernel output to the linux-btrfs list.

Anyway, as it happens, that watchdog driver looks like it’ll need some re-factoring as a multi-function device.  At the moment, ts-wdt.c claims it based on this binding.

If I try to add a second driver, they’ll clash, and I expect the same if I try to access it via userspace.  So the sensible thing to do here, is to add a ts-companion.c MFD driver here, then re-factor ts-wdt.c to use it.  From there, I can write a ts-psu.c module which will go right here.

I think I’ll definitely be digging into those older sources to remind myself how that all worked.