solar-cluster

Solar Cluster: Battery monitor PC now running Gentoo

So, after some argument, and a bit of sitting on a concrete floor with the netbook, I managed to get Gentoo loaded onto the TS-7670.  Right now it’s running off the MicroSD card, I’ll get things right, then shift it across to eMMC.

ts7670 ~ # emerge --info
Portage 2.3.40 (python 3.5.5-final-0, default/linux/musl/arm/armv7a, gcc-6.4.0, musl-1.1.19, 4.14.15-vrt-ts7670-00031-g1a006273f907-dirty armv5tejl)
=================================================================
System uname: Linux-4.14.15-vrt-ts7670-00031-g1a006273f907-dirty-armv5tejl-ARM926EJ-S_rev_5_-v5l-with-gentoo-2.4.1
KiB Mem:      111532 total,     13136 free
KiB Swap:    4194300 total,   4191228 free
Timestamp of repository gentoo: Fri, 17 Aug 2018 16:45:01 +0000
Head commit of repository gentoo: 563622899f514c21f5b7808cb50f6e88dbd7d7de
sh bash 4.4_p12
ld GNU ld (Gentoo 2.30 p2) 2.30.0
app-shells/bash:          4.4_p12::gentoo
dev-lang/perl:            5.24.3-r1::gentoo
dev-lang/python:          2.7.14-r1::gentoo, 3.5.5::gentoo
dev-util/pkgconfig:       0.29.2::gentoo
sys-apps/baselayout:      2.4.1-r2::gentoo
sys-apps/openrc:          0.34.11::gentoo
sys-apps/sandbox:         2.13::musl
sys-devel/autoconf:       2.69-r4::gentoo
sys-devel/automake:       1.15.1-r2::gentoo
sys-devel/binutils:       2.30-r2::gentoo
sys-devel/gcc:            6.4.0-r1::musl
sys-devel/gcc-config:     1.8-r1::gentoo
sys-devel/libtool:        2.4.6-r3::gentoo
sys-devel/make:           4.2.1::gentoo
sys-kernel/linux-headers: 4.13::musl (virtual/os-headers)
sys-libs/musl:            1.1.19::gentoo
Repositories:

gentoo
    location: /usr/portage
    sync-type: rsync
    sync-uri: rsync://virtatomos.longlandclan.id.au/gentoo-portage
    priority: -1000
    sync-rsync-verify-jobs: 1
    sync-rsync-extra-opts: 
    sync-rsync-verify-metamanifest: yes
    sync-rsync-verify-max-age: 24

ACCEPT_KEYWORDS="arm"
ACCEPT_LICENSE="* -@EULA"
CBUILD="arm-unknown-linux-musleabi"
CFLAGS="-Os -pipe -march=armv5te -mtune=arm926ej-s -mfloat-abi=soft"
CHOST="arm-unknown-linux-musleabi"
CONFIG_PROTECT="/etc /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/env.d /etc/gconf /etc/gentoo-release /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-Os -pipe -march=armv5te -mtune=arm926ej-s -mfloat-abi=soft"
DISTDIR="/home/portage/distfiles"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
FEATURES="assume-digests binpkg-logs config-protect-if-modified distlocks ebuild-locks fixlafiles merge-sync multilib-strict news parallel-fetch preserve-libs protect-owned sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=armv7-a -mfpu=vfpv3-d16 -mfloat-abi=hard"
GENTOO_MIRRORS=" http://virtatomos.longlandclan.id.au/portage http://mirror.internode.on.net/pub/gentoo http://ftp.swin.edu.au/gentoo http://mirror.aarnet.edu.au/pub/gentoo"
INSTALL_MASK="charset.alias"
LANG="en_AU.UTF-8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
PKGDIR="/usr/portage/packages"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="arm bindist cli crypt cxx dri fortran iconv ipv6 modules ncurses nls nptl openmp pam pcre readline seccomp ssl tcpd unicode xattr zlib" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon plan sheets stage words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" ELIBC="musl" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="libinput keyboard mouse" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php5-6 php7-0" POSTGRES_TARGETS="postgres9_5 postgres10" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby23" USERLAND="GNU" VIDEO_CARDS="dummy fbdev v4l" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, EMERGE_DEFAULT_OPTS, LC_ALL, LINGUAS, MAKEOPTS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS

I still have to update the kernel.  I actually did get kernel 4.18 to boot, but I forgot to add in support for the watchdog, so U-Boot tickled it, then the watchdog got hungry and kicked the reset half way through the boot sequence.

Rolling back to my older 4.14 kernel works.  I’ll try again with 4.18.5 in a moment.  Failing that, I have also brought the 4.14 patches up to 4.14.69 which is the latest LTS release of the kernel.

I’ve started looking at the power supply sysfs device class, with a view to exposing the supply voltage via sysfs.  The thinking here is that collectd supports reading this via the “battery” module (and realistically, it is a battery that is being measured: two 105Ah AGMs).

Worst case is I do something a little proprietary and deal with it in user space.  I’ll have to dig up the Linux kernel tree I did for Jacques Electronics all those years ago, as that had some examples of interfacing sysfs to a Cypress PSOC device that was acting as an I²C slave.  Rather than using an off-the-shelf solution, they programmed up a MCU that did power management, touchscreen sensing, keypad sensing, RGB LED control and others, all in one chip.  (Fun to try and interface that to the Linux kernel.)

Technologic Systems appear to have done something similar.  The device ID 0x78 implies a 10-bit device, but I think they’re just squatting on that 7-bit address.  They hail 0x78 then read out 4 bytes, which the last two bytes are the supply voltage ADC readings.  They do their own byte swapping before scaling the value to get mV.

Solar Cluster: Gentoo Linux MUSL for ARMv5 finally built

It’s taken several months and had a few false starts, but at long last I have some stage tarballs for Gentoo Linux MUSL for ARMv5 processors.  I’m not the only one wanting such a port, even looking for my earlier thread on the matter, I stumbled on this post.  (Google translate is hopeless with Russian, but I can get the gist of what’s being said.)

This was natively built on the TS-7670 using an external hard drive connected over USB 2.0 as both swap and the chroot.  It took two passes to clean everything up and get something that’s in a “release-able” state.

I think my next steps now will be:

  • Build an updated kernel … I might see if I can expose that I²C register via a sysfs file or something that collectd can pick up whilst I’m at it.  I have the kernel sources and bootloader sources.
  • Prepare the 32GB MicroSD card I bought a few weeks back with the needed partitions and load Gentoo onto that.
  • Install the MicroSD card and boot off it.
  • Back up the eMMC
  • Re-format the eMMC and copy the MicroSD card to it.

It’s supposed to be wet this weekend, so it sounds like a good project for indoors.

Solar Cluster: Measuring the battery voltage

So, I was just updating the project details for this project, and I happened to see this blog post about reading the DC voltage input on the TS-7670v2.

I haven’t yet gotten around to finishing the power meters that I was building which would otherwise be reading these values directly, but they were basically going to connect via Modbus to the TS-7670v2 anyway.  One of its roles, aside from routing between the physical management network (IPMI and switch console access), was to monitor the battery.

I will have to explore this.  Collectd doesn’t have a general-purpose I²C module, but it does have one for barometer modules, so with a bit of work, I could make one to measure the voltage input which would tell me what the battery is doing.

Solar Cluster: NTP time synchronisation

For years up until last month, I ran a public NTP server which was on pool.ntp.org.  This mostly sat at Stratum II and was synchronised from random stratum I servers.

Last month, I had to knock that on the head because of a sudden spike in Internet traffic.  Through tcpdump on the border router, I identified the first culprit: NTP client traffic.  For whatever reason, I was getting a lot of traffic.  Dumping packets to a file over a 15 minute period and analysing showed about 90% of the traffic was NTP.

For about 3 weeks straight, I had an effectively constant 4~5Mbps incoming stream … a pelican flew into my inbox, carrying a AU$300 bill for the nearly 800GB consumed in May.  (My plan was originally 250GB.  Many thanks for Internode there: they capped the overusage charge at AU$300 and provided an instant upgrade of the plan to 500GB/month.)

The other culprit I tracked down to HackChat’s websockets, which was a big contributor to June’s data usage.  (Thank-you OpenBSD pflow and nfdump!)

So right now, I’m looking at whether I re-join the NTP server pool.  I have the monitoring in place to keep track of data usage now.  For sure the experience has cost me over $400, but that’s still cheaper than university studies … and they didn’t teach me this stuff anyway!

One thought is that I could do away with any external NTP server altogether by using local sources to synchronise time.  With a long-wire antenna, I could sync to WWV.  Usually I can pick it up on 10MHz or 15MHz, but it’s weak, and I’d have to rig up the receiver, the antenna and a suitable decoder.

Then there’s the problem of lightning strikes, I already have a Yaesu FT-897D in the junk box thanks to Thor’s past efforts.

The more practical approach would appear to be using GPS time sync.  You can do it with any NMEA-compatible module, but you get far better results using one that supports PPS.  Some even have a 10PPS option, but I’m not sure if kpps supports that.

As it happens, I could have got a GPS module in the TS-7670.  Here’s where it is on the schematic:

and here are the connections to its UART (bottom two):

and the PPS pin (LCD_D16):

The footprints are there on the board, and I can buy the module.  No problems.  Buying the TS-7670 with the GPS option would have meant buying the top-of-the-line “development” model which would have included WiFi (not needed), an extra 128MB RAM (nice to have, but not essential) and an extra CAN port (not needed).

The other option is to go something else, such as this module.  Either way, I need to watch logic levels, Freescale like their 1.8V, although it looks like the I/O pins can be switched between 1.8V and 3.3V from the GPIO registers (chapter 9 of the datasheet).

Doing this, would allow me to run Stratum 1 within the network, and perhaps to IPv6 NTP clients.  IPv4 clients might get Stratum 2, we’ll see how I feel about it later.

Solar Cluster: New Router

So before going on the trip, I noticed the router I was using would occasionally drop off the network.  The switch still reported the link as being up, but the router would not respond to pings from the internal network.  If I SSHed into it from outside the network, and tried pinging internal IPs, it failed to ping them.

Something was up.  After much debugging (and some arguments about upgrades), it was decided that the hardware was flakey.  In that discussion, it was recommended that I have a look at PC Engines’ APU2 single board computer.

This is the only x86 computer I have seen with schematics and CoreBoot out-of-the-box, and it happens there’s a local supplier of them.  For sure, this machine is overkill for the job, but it ticks nearly all the boxes.

The only one it didn’t tick was being able to run directly from the battery.  As it happens, the unit only draws about 1.5A, and so a LM1085-12 LDO which can be sourced locally did the trick.  I basically put 100µF capacitors on the input and output, bolted it to a small heatsink and threw it all into a salvaged case.

After hooking it up to a bench supply (disconnected from the APU2) and winding the voltage right up to the PSUs maximum, and observing that the voltage stayed at 12V, I decided to hook it up and see how it went.  I plugged in my null modem cable, and sure enough, I was staring at CoreBoot.

I PXE-booted OpenBSD 6.3 and installed that onto the SD card, this was fairly painless and before long, the machine was booting on its own. I copied across the configuration settings from the old one, set up sniproxy, and I was in business, it was time to issue a `shutdown -p now` to both machines and for them to swap places.

Of course, a nicety of this box is there’s three Ethernet ports, so room for a move to another Internet connection, such as the HFC we’re supposed to be getting in this part of Brisbane (sadly, no thin pieces of glass for us), so in theory, I can run both in parallel and migrate between them.

Solar Cluster: arm-unknown-linux-musleabi… saga part IV

So, at long last, I finally saw this in my chroot‘s /var/log/emerge.log:

1524887925: Started emerge on: Apr 28, 2018 03:58:45
1524887926:  *** emerge --oneshot sys-devel/gcc::musl
1524888211:  >>> emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1524888212:  === (1 of 1) Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1524888307:  === (1 of 1) Compiling/Packaging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472690:  === (1 of 1) Merging (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525472838:  >>> AUTOCLEAN: sys-devel/gcc:7.3.0
1525473358:  === (1 of 1) Post-Build Cleaning (sys-devel/gcc-7.3.0::/root/musl/sys-devel/gcc/gcc-7.3.0.ebuild)
1525473358:  ::: completed emerge (1 of 1) sys-devel/gcc-7.3.0 to /
1525473360:  *** Finished. Cleaning up...
1525473373:  *** exiting successfully.

That’s 6 days, 18 hours and 32 minutes, of solid compiling. BUT WE GOT THERE!

What’s left? This:

Calculating dependencies... done!
[ebuild     U  ] sys-libs/musl-1.1.19 [1.1.18]
[binary   R    ] sys-libs/zlib-1.2.11-r1
[binary   R    ] app-arch/xz-utils-5.2.3
[ebuild     U  ] sys-libs/ncurses-6.1-r2 [6.0-r1]
[binary   R    ] sys-libs/readline-7.0_p3
[binary   R    ] virtual/libintl-0-r2
[binary   R    ] dev-lang/python-exec-2.4.5
[binary   R    ] virtual/libiconv-0-r2
[binary   R    ] sys-apps/gentoo-functions-0.12
[binary   R    ] dev-libs/libpcre-8.41-r1
[binary   R    ] sys-apps/sed-4.2.2
[binary   R    ] app-arch/bzip2-1.0.6-r8
[binary   R    ] dev-libs/gmp-6.1.2
[binary   R    ] app-shells/bash-4.4_p12
[binary   R    ] sys-apps/file-5.32
[binary   R    ] sys-devel/gnuconfig-20170101
[binary   R    ] dev-libs/mpfr-3.1.6
[binary   R    ] app-misc/c_rehash-1.7-r1
[binary   R    ] app-misc/mime-types-9
[binary   R    ] app-arch/tar-1.29-r3
[binary   R    ] app-arch/gzip-1.8
[binary   R    ] dev-libs/mpc-1.0.3
[binary   R    ] sys-devel/gcc-config-1.8-r1
[binary   R    ] app-misc/editor-wrapper-4
[binary   R    ] sys-apps/less-529
[binary   R    ] sys-apps/debianutils-4.8.3
[binary   R    ] net-libs/libmnl-1.0.4
[binary   R    ] sys-libs/libseccomp-2.3.2
[binary   R    ] dev-libs/popt-1.16-r2
[binary   R    ] sys-libs/e2fsprogs-libs-1.43.6
[binary   R    ] sys-devel/binutils-config-5-r4
[binary   R    ] dev-libs/libffi-3.2.1
[binary   R    ] virtual/libffi-3.0.13-r1
[binary   R    ] sys-apps/sysvinit-2.88-r9
[binary   R    ] sys-apps/opentmpfiles-0.1.3
[binary   R    ] virtual/tmpfiles-0
[binary   R    ] app-text/manpager-1
[binary   R    ] sys-libs/cracklib-2.9.6-r1
[binary   R    ] sys-apps/install-xattr-0.5
[binary   R    ] app-editors/nano-2.8.7
[binary   R    ] app-portage/elt-patches-20170815
[binary   R    ] sys-devel/m4-1.4.17
[binary   R    ] app-arch/unzip-6.0_p21-r2
[binary   R    ] sys-devel/autoconf-wrapper-13
[binary   R    ] sys-devel/bison-3.0.4-r1
[binary   R    ] sys-devel/flex-2.6.4-r1
[binary   R    ] dev-libs/libltdl-2.4.6
[binary   R    ] sys-devel/automake-wrapper-10
[binary   R    ] app-text/sgml-common-0.6.3-r6
[binary   R    ] dev-libs/libgpg-error-1.27-r1
[ebuild  N     ] dev-lang/perl-5.24.3-r1  USE="-berkdb -debug -doc -gdbm -ithreads"
[ebuild  N     ] sys-kernel/linux-headers-4.13  USE="-headers-only"
[ebuild  N     ] virtual/perl-Data-Dumper-2.160.0-r1
[ebuild  N     ] virtual/perl-Test-Harness-3.360.100_rc-r3
[ebuild  N     ] perl-core/File-Temp-0.230.400-r1
[ebuild  N     ] virtual/perl-File-Temp-0.230.400-r5
[ebuild  N     ] perl-core/File-Path-2.130.0
[ebuild  N     ] virtual/perl-File-Path-2.130.0
[binary   R    ] virtual/os-headers-0
[ebuild  N     ] sys-devel/autoconf-2.69-r4  USE="-emacs"
[ebuild  N     ] sys-apps/attr-2.4.47-r2  USE="-nls -static-libs"
[ebuild   R    ] sys-apps/coreutils-8.28-r1
[ebuild     U  ] app-admin/eselect-1.4.12 [1.4.8]
[ebuild     U  ] app-eselect/eselect-python-20171204 [20160516]
[ebuild     U  ] sys-devel/patch-2.7.6-r1 [2.7.5]
[ebuild  N     ] sys-apps/shadow-4.5  USE="cracklib xattr -acl -audit -nls -pam (-selinux) -skey"
[binary   R    ] virtual/shadow-0
[ebuild  N     ] virtual/perl-ExtUtils-MakeMaker-7.100.200_rc-r4
[ebuild  N     ] sys-libs/libcap-2.24-r2  USE="-pam -static-libs"
[ebuild  N     ] dev-perl/Text-Unidecode-1.270.0
[ebuild  N     ] dev-perl/libintl-perl-1.240.0-r2
[ebuild  N     ] sys-apps/help2man-1.47.4  USE="-nls"
[ebuild  N     ] sys-devel/automake-1.15.1-r2  USE="{-test}"
[ebuild  N     ] sys-devel/libtool-2.4.6-r3  USE="-vanilla"
[ebuild  N     ] dev-libs/expat-2.2.5  USE="unicode -examples -static-libs"
[ebuild   R    ] sys-process/psmisc-22.21-r3
[ebuild  N     ] sys-libs/gdbm-1.13-r2  USE="readline -berkdb -exporter -nls -static-libs"
[ebuild  N     ] sys-apps/groff-1.22.2  USE="-X -examples" L10N="-ja"
[ebuild  N     ] dev-libs/libelf-0.8.13-r2  USE="-debug -nls"
[ebuild  N     ] virtual/libelf-2
[ebuild  N     ] dev-libs/libgcrypt-1.8.1  USE="-doc -static-libs"
[ebuild  N     ] dev-perl/XML-Parser-2.440.0
[ebuild  N     ] virtual/perl-File-Spec-3.630.100_rc-r4
[ebuild  N     ] dev-perl/Unicode-EastAsianWidth-1.330.0-r1
[ebuild  N     ] sys-apps/texinfo-6.3  USE="-nls -static"
[ebuild  N     ] dev-libs/iniparser-3.1-r1  USE="-doc -examples -static-libs"
[ebuild  N     ] app-portage/portage-utils-0.64  USE="-nls -static"
[ebuild  N     ] dev-libs/openssl-1.0.2o  USE="asm sslv3 tls-heartbeat zlib -bindist -gmp -kerberos -rfc3779 -sctp -sslv2 -static-libs {-test} -vanilla"
[binary  N     ] dev-lang/python-2.7.14-r1  USE="ipv6 ncurses readline ssl (threads) (wide-unicode) xml (-berkdb) -build -doc -examples -gdbm -hardened -libressl -sqlite -tk -wininst"
[binary  N     ] sys-apps/openrc-0.34.11  USE="ncurses netifrc unicode -audit -debug -newnet -pam (-prefix) (-selinux) -static-libs" 
[ebuild  N     ] net-misc/netifrc-0.5.1
[binary   R    ] sys-apps/grep-3.0
[binary   R    ] sys-apps/findutils-4.6.0-r1
[binary   R    ] sys-apps/kbd-2.0.4
[ebuild  N     ] sys-apps/busybox-1.28.0  USE="ipv6 static -debug -livecd -make-symlinks -math -mdev -pam -savedconfig (-selinux) -sep-usr -syslog (-systemd)"
[binary   R    ] virtual/service-manager-0
[binary   R    ] sys-devel/binutils-2.29.1-r1
[ebuild  N     ] sys-apps/net-tools-1.60_p20161110235919  USE="arp hostname ipv6 -nis -nls -plipconfig (-selinux) -slattach -static" 
[binary   R    ] sys-apps/gawk-4.1.4
[binary   R    ] virtual/editor-0
[binary   R    ] sys-devel/make-4.2.1
[binary   R    ] sys-process/procps-3.3.12-r1
[binary   R    ] virtual/dev-manager-0-r1
[binary   R    ] sys-apps/which-2.21
[ebuild  N     ] net-misc/iputils-20171016_pre  USE="arping filecaps ipv6 openssl ssl -SECURITY_HAZARD -caps -clockdiff -doc -gcrypt
 (-idn) -libressl -nettle -rarpd -rdisc -static -tftpd -tracepath -traceroute"
[binary   R    ] virtual/pager-0
[binary   R    ] sys-apps/diffutils-3.5
[binary   R    ] sys-apps/baselayout-2.4.1-r2
[binary   R    ] virtual/libc-1
[binary   R   ~] sys-devel/gcc-7.3.0
[binary   R    ] virtual/pkgconfig-0-r1
[ebuild  N     ] dev-lang/python-3.5.5  USE="ipv6 ncurses readline ssl (threads) xml -build -examples -gdbm -hardened -libressl -sqlite {-test} -tk -wininst"
[ebuild  N     ] app-misc/ca-certificates-20170717.3.36.1  USE="-cacert -insecure_certs"
[ebuild  N     ] sys-apps/util-linux-2.30.2-r1  USE="cramfs ncurses readline suid unicode -build -caps -fdformat -kill -nls -pam -python (-selinux) -slang -static-libs (-systemd) {-test} -tty-helpers -udev" PYTHON_SINGLE_TARGET="python3_5 -python2_7 -python3_4 -python3_6" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild     U  ] app-misc/pax-utils-1.2.3 [1.1.7]
[ebuild     U  ] sys-apps/sandbox-2.13 [2.10-r4]
[ebuild     U  ] net-misc/rsync-3.1.3 [3.1.2-r2]
[ebuild  N     ] net-firewall/iptables-1.6.1-r3  USE="ipv6 -conntrack -netlink -nftables -pcap -static-libs"
[ebuild     U  ] dev-libs/libpipeline-1.4.2 [1.4.0]
[ebuild  N     ] sys-apps/man-db-2.7.6.1-r2  USE="gdbm manpager zlib -berkdb -nls (-selinux) -static-libs"
[ebuild     U  ] sys-apps/kmod-24 [23] PYTHON_TARGETS="-python3_6%"
[ebuild  N     ] dev-python/pyblake2-1.1.0  PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] net-misc/openssh-7.5_p1-r4  USE="hpn pie ssl -X -X509 -audit -bindist -debug -kerberos -ldap -ldns -libedit -libressl -livecd -pam -sctp (-selinux) -skey -ssh1 -static {-test}"
[ebuild  N     ] dev-util/gtk-doc-am-1.25-r1
[ebuild  N     ] dev-libs/libxml2-2.9.7  USE="ipv6 readline -debug -examples -icu -lzma -python -static-libs {-test}" PYTHON_TARGETS="python2_7 python3_5 -python3_4 -python3_6"
[ebuild  N     ] sys-devel/gettext-0.19.8.1  USE="cxx ncurses openmp -acl -cvs -doc -emacs -git -java (-nls) -static-libs"
[ebuild  N     ] app-text/build-docbook-catalog-1.19.1
[ebuild  N     ] dev-libs/libxslt-1.1.30-r2  USE="crypt -debug -examples -python -static-libs" PYTHON_TARGETS="python2_7"
[ebuild  N     ] app-text/docbook-xsl-stylesheets-1.79.1-r2  USE="-ruby"
[ebuild  N     ] app-text/docbook-xml-dtd-4.1.2-r6
[ebuild  N     ] dev-util/intltool-0.51.0-r2
[ebuild  N     ] dev-libs/glib-2.52.3  USE="mime xattr -dbus -debug (-fam) (-selinux) -static-libs -systemtap {-test} -utils" PYTHON_TARGETS="python2_7"
[ebuild  N     ] x11-misc/shared-mime-info-1.9  USE="{-test}"
[ebuild  N     ] dev-python/setuptools-36.7.2  USE="{-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/certifi-2017.4.17  PYTHON_TARGETS="python2_7 python3_5 (-pypy) (-pypy3) -python3_4 -python3_6"
[ebuild  N     ] dev-python/pyxattr-0.5.5  USE="-doc {-test}" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4"
[ebuild  N     ] sys-apps/portage-2.3.24-r1  USE="(ipc) native-extensions xattr -build -doc -epydoc -gentoo-dev (-rsync-verify) (-selinux)" PYTHON_TARGETS="python2_7 python3_5 (-pypy) -python3_4 -python3_6"
[ebuild  N     ] app-admin/perl-cleaner-2.25
[binary   R    ] virtual/man-0-r1
[binary   R    ] virtual/modutils-0
[ebuild  N     ] sys-fs/e2fsprogs-1.43.6  USE="-fuse (-nls) -static-libs"
[ebuild     U  ] virtual/package-manager-1 [0]
[ebuild  N     ] sys-apps/iproute2-4.14.1-r2  USE="iptables ipv6 -atm -berkdb -minimal (-selinux)"
[binary   R    ] virtual/ssh-0
[ebuild  N     ] net-misc/wget-1.19.1-r2  USE="ipv6 pcre ssl zlib -debug -gnutls -idn -libressl -nls -ntlm -static {-test} -uuid"
[ebuild   R    ] dev-util/pkgconfig-0.29.2  USE="-internal-glib*"

!!! The following binary packages have been ignored due to non matching USE:

    =dev-util/pkgconfig-0.29.2 internal-glib
    =sys-apps/attr-2.4.47-r2 nls
    =sys-apps/man-db-2.7.6.1-r2 nls
    =dev-libs/libelf-0.8.13-r2 nls
    =sys-apps/shadow-4.5 -linguas_cs -linguas_da -linguas_de -linguas_es -linguas_fi -linguas_fr -linguas_hu -linguas_id -linguas_it -linguas_ja -linguas_ko -linguas_pl -linguas_pt_BR -linguas_ru -linguas_sv -linguas_tr -linguas_zh_CN -linguas_zh_TW nls

NOTE: The --binpkg-respect-use=n option will prevent emerge
      from ignoring these binary packages if possible.
      Using --binpkg-respect-use=y will silence this warning.

I think that’s broken the back of the job.  Of course when I come to running Catalyst, I’ll have to do it all over again, but at least now the environment is clean.

Solar Cluster: Next steps, better control of the charger

So, a few weeks ago I installed a new battery charger, and tweaked it so that the solar did most of the leg work during the day, and the charger kept the batteries topped up at night.

I also discussed the addition of a new industrial PC to perform routing and system monitoring functions… which was to run Gentoo Linux/musl. For now, that little PC is still running Debian Stretch, but for 45 days, it was rock solid. The addition of this box, and taking on the role of router to the management network meant I could finally achieve one of my long-term goals for the project: decommissioning the old server.

The old server is still set up with all my data and software… but now the back-up cron job calls /sbin/poweroff when it’s done, and the BIOS is set to wake the machine up in the evening ready to receive a back-up late at night.

In its place, a virtual machine clone of the box, handles my email and all the old functions of that server. This was all done just prior to my father and I leaving for a 3 week holiday in the Snowy Mountains.

I did have a couple of hiccups with Ceph OSDs crashing … but basically re-starting the daemons (done remotely whilst travelling through Cowra) got everything back up. A bit of placement group cleaning, and everything was back online again. I had another similar hiccup coming out of Maitland, but once again, re-starting the daemons fixed it. No idea why it crashed, that’s something I’ll have to investigate.

Other than that, the cluster itself has run well.

One thing that did momentarily kill the industrial PC though: I wandered down to the rack with a small bus-powered 2.5″ HDD with the intent of re-starting my Gentoo builds. This HDD had the same content as the 3.5″ HDD I had plugged in before. I figured being bus powered, I would not be dependent on mains, and it could just chug away to its heart’s content.

No such luck, the moment I plugged that drive in, the little machine took great umbrage to the spinning rust now vacuuming the electrons away from its core functions, and shut down abruptly. I’ve now brought my 3.5″ drive and dock down, plugged that into the wall, and have my builds resuming. If power goes off, hopefully the machine either handles the loss of swap gracefully. If it does crash, the watchdog will take care of it.

Thus, I have the little TS-7670 first attempting a build of gcc, to see how we go. Finger’s crossed our power should remain up. There was at least one outage in the time we were away, but hopefully we should get though this next build!

The next step I think should be to add some control of the mains charger to allow the batteries to be boosted to full charge overnight. The thinking is a simple diode-OR arrangement. Many comparators such as the LM393 have an open-collector output, which gives us this for free.

The theory is this.

The battery bank powers a simple circuit which runs of a 5V regulator. That regulator powers a dual comparator IC and provides a reference voltage. The comparator draws bugger all power, so I’m happy to use a linear PSU here. It’s mainly there as a voltage reference.

Precision isn’t really the aim here, so adjustable pots will make life easier.

The voltages from the battery bank and the solar panel are fed through voltage dividers to bring the voltages down to below 5V, then those voltages are individually fed into separate pots that control the hysteresis. I can adjust all points of the system.

The idea is that should the batteries get too low, or the sun go down, one or the other (or both) comparators will go low and pull down on R2. If the batteries are high and the sun is up, nothing pulls on R2 so the REMOTE+ pin on the HEP-600C-12 is allowed to float to +5V, turning off the mains charger.

The advantage of this is there’s no programming of a microcontroller, it’s just analogue electronics. The LM393s are pretty hardy things, the datasheet says they’ll run at 36V and can accept a maximum voltage of VCC-1.5V; so if I run at 5V, 3.5V is my recommended maximum. The adjustment pots should let me set a threshold voltage that avoids going above this.

I mainly need 5V for the HEP-600C-12, and for providing that stable known voltage reference. The LM78C05 should be fine for this.

Once I’ve done that, I should be able to wind that charger back up to its factory setting of 14.4V, which will mean that overnight the batteries will be charged back to full charge.

Solar Cluster: Fine Tuning

So yesterday I wound back the mains charger so that the solar would take on the load during the day.  Seems I wound it back a bit far, and the mains charger did almost no work overnight, leaving the battery somewhere around 11.8V.

That’s a wee bit low for my comfort.  Yes, they are deep cycle AGMs, but I’d rather not get that low.

Thus, I wound it up a bit, float at 12.8V, so Vboost at 13.6V.  That looks to be the sweet spot.  Now that the sun is up, I’m getting nice healthy amps of current down the wire from the roof:

The cluster is drawing about 8A, so that’s the cluster powered, and about 6A going to the batteries. It intermittently peaks about 15A or so.

I also found myself fine tuning the Ethernet settings on the border router. For some reason, its Realtek RTL8139 was happy to talk to the Cisco SG-200-08 it was connected to before, but didn’t quite get along with the Linksys LGS326-AU. I’ve told the switch to force 100Mbps full-duplex MDIX (evidently, it’s a cross-over cable), and so far, that seems to have settled things down.

Solar Cluster: Coaxing two supplies to get along

So last post, I mentioned about the installation of the new battery charger, which is fed from 240V mains. Over the last few days this charger has held the batteries at a rock-solid 14.4V. Not once did the batteries drop below that voltage setpoint.

So good in fact, the solar charger does no work at all.

By the way, this is what the install looks like. I promised pictures last post.

That’s the DC end … and the nasty AC end is all sealed up…

I will eventually move this to a spot on the back of the rack, but it can sit here for now.

Ultimately, the proper fix to this will be to have the mains-powered charger power off when the sun is up. On the DC output connector, the two rightmost screw terminals go to an opto-isolator that, when powered, shuts off the charger, putting it into stand-by mode. This was one of the reasons I bought this particular unit. The other was the wide range of voltage adjustment.

The question is when to turn on, and when to go to stand-by. Basically if the following expression is true, then turn off the mains:

(V_{batt} > 12.8) \\wedge (V_{solar} > 15)

We do not want solar if the battery is very low, as there’s a possibility that the solar output will not be sufficient.  Likewise, if the sun’s out, we need the mains to keep the battery topped up.

The solar output is nearly always above 15V when the sun is up, so there’s our first clue.  We can safely get to 12.8V before things start going pear shaped on the cluster, so we can use that as our low-voltage safety net.  If both of these conditions are met, then it’s safe to turn off the mains power and rely on solar only.

We need a +5V signal when both these conditions are met.  This very much sounds like the job of a dual-comparator with diode-OR outputs pulling on a 5V pull-up.  Maybe a wee bit of hysteresis on those to prevent flapping, and we should be good.

Unfortunately, to do that, I need to unscrew terminals to feed some wires in.  I don’t feel like doing that just now… we’re packing up to go away for a while, and I think this sort of job can wait until we return.

In the meantime, I’ve done something of a hack.  I mentioned the PSU is adjustable.  I wound Vfloat back to 12V… thus Vboost has gone to 12.8V.  Right now, the mains PSU is showing a green LED, meaning it is in floating mode.

We have good sun right now, and the solar controller is currently boosting the battery.  When the battery gets low, the charging circuitry of the mains PSU should kick in, and bring the battery voltage up, holding it at 12.8V until the sun comes up.  I’ll leave it for now and see how this hack goes.

On other news… I might need to re-consider my NTP server arrangements.  I’m not sure if it’s a quirk of OpenBSD, or of the network here, but it seems OpenNTPD struggles to keep good time.  Never tried using the Advantech PC as a NTP server until now, and I’m also experimenting with using my VPS at Vultr as a NTP server.

http://www.pool.ntp.org/user/Redhatter

Both are drifting like crazy.  I have a GPS module lying around that I might consider hooking up to the TS-7670… perhaps make it a Stratum 1 NTP server on the NTP server pool, then the Advantech can sync to that.

This won’t help the VPS though, and I’m at a loss to explain why a Geode LX800 running on an ADSL link in my laundry, outperforms a VPS in a nicely climate-controlled data centre with gigabit Internet.

But at least now that’s one less job for my aging server.  I’ve also moved mail server duties off the old box onto a VM, so I’ll be looking at the BIOS settings there to see if I can get the box to wake up some time in the evening, let cron run the back-up jobs, then power the whole lot back down again, save some juice.