arm

Solar Cluster: TS-7670 showed up today

So, I now have my little battery monitoring computer.  Shipping wound up being a little more than I was expecting… about US$80… but never mind.  It’s here, arrived safely:

HTLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLLFLC
>> TS-BOOTROM - built Jan 26 2017 12:29:21
>> Copyright (c) 2013, Technologic Systems
LLCLLLLLLLFLCLLJUncompressing Linux... done, booting the kernel.
/ts/fastboot file present.  Booting to initramfs instead
Booted from eMMC in 3.15s
Initramfs Web Interface: http://ts7670-498476.local
Total RAM: 128MB
# exit
INIT: version 2.88 booting
[info] Using makefile-style concurrent boot in runlevel S.
[ ok ] Starting the hotplug events dispatcher: udevd.
[ ok ] Synthesizing the initial hotplug events...done.
[ ok ] Waiting for /dev to be fully populated...done.
[ ok ] Activating swap...done.
[....] Checking root file system...fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
/dev/mmcblk2p2: clean, 48540/117600 files, 282972/469760 blocks
done.
[ ok ] Cleaning up temporary files... /tmp /lib/init/rw.
…
ts7670-498476 login: root
Linux ts7670-498476 2.6.35.3-571-gcca29a0+ #1 PREEMPT Mon Nov 27 11:05:10 PST 2017 armv5tejl
TS Root Image 2017-11-27

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.

root@ts7670-498476:~# 

The on-board 2GB eMMC has a version of Debian Wheezy on it.  That’ll be going very soon.  For now, all I’ve done is pop the cover, shove a 8GB MicroSD card into one of the on-board slots, wired up a 12V power brick temporarily to the unit, hooked a USB cable into the console port (/dev/ttyAMA0 is wired up to an on-board CP2103 USB-serial chip) and verified that it is alive.

Next step will be to bootstrap Gentoo.  I could use standard ARMv5 stages, or I can build my own, which I might do.  I’ve done this before for mips64el n64 using glibc.  Modern glibc is a goliath on a machine with 128MB RAM though, so I’ll be looking at either µClibc/µClibc-ng or musl… most likely the latter.

That said, 20 years ago, we had the same computing power in a desktop. 🙂

I have a few options for interfacing to the power meters…

  • I²C, SPI, a number of GPIOs and a spare UART on a 2.54mm header inside the case.
  • Another spare UART on the footprint for the GPS module (which my unit does not have)
  • Two RS-232 serial ports with RTS/CTS control lines, exposed via RJ-45 jacks
  • Two CANbus ports on a single RJ-45 jack
  • RS-485 on a port marked “Modbus”

In theory, I could just skip the LPC810s and hook this up directly to the INA219Bs.  I’d have to double check what the TTL voltage is… Freescale love their 1.8V logic… but shifting that up to 3.3V or 5V is not hard.  The run is a little longer than I’m comfortable running I²C though.

The LPC810s don’t feature CANbus, so I think my original plan of doing Modbus is going to be the winner.  I can either do a single-ended UART using a resistor/diode in parallel to link RX and TX to the one UART line, or use RS-485.

I’m leaning towards the latter, if I decide to buy a little mains energy meter to monitor power, I can use the same RS-485 link to poll that.  I have some RS-485 transceivers coming for that.

For now though, I’ll at least get Debian Stretch going… this should not be difficult, as I’ll just use the images I’ve built for work to get things going.  I’m downloading a Jessie image now:

root@ts7670-498476:~# curl https://bne.vrt.com.au/technologicsys/ts7670d-jessie-4.4.1-20160226.dd.xz | xzcat | dd of=/dev/mmcblk0 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0  113M    0  544k    0     0   114k      0  0:16:48  0:00:04  0:16:44  116k

Once that is done, I can reboot, re-format the eMMC and get debootstrap going.  I might even publish an updated image while I’m at it.

Solar Cluster: arm-unknown-linux-musleabi… saga part II

So, last time I was trying to get Gentoo’s portage to cross-build gcc so that I’d have a C/C++ compiler in my ARMv5 musl environment.

It is literally the last piece of the puzzle.  Once compiled, that is the last step I need before I can throw the shiny new environment onto an ARMv5 VM (or real ARMv5 CPU), do an emerge -e world on it then tar the lot up and throw it at Catalyst.

Building an entire OS on a 454MHz ARMv5 machine with 128MB RAM does not faze me one bit… I used to do it regularly on a (Gateway-branded) Cobalt Qube II server appliance, which sports a 250MHz QED RM5231 and 128MB RAM.  The other compile workhorse I used in those days was an SGI O2; 300MHz RM5200, again 128MB RAM.

Yes, Linux and its userland has bulked up a bit in the last 10 years, but not so much so that a build on these is impossible.

Certainly, native building is easier than cross-compiling.  Cross-compilers have always been a voodoo art for me.  Getting one that will build a Linux kernel or U-Boot, usually isn’t too hard… but get userland involved and it gets complex.  Throw in C++ and complexity skyrockets!

I’m taking OpenADK for a spin now, and in concept, it’s exactly what I remember buildroot used to be.  It’s a tool for generating a fully fledged embedded Linux system with a wide package selection including development tools.  I also find that you have to hold your tongue just right to get stuff to compile.

Selecting a generic arm926ej-s; it succeeded to build a x86-64 hosted cross-toolchain once, but then silently refused to build anything else.  I told it instead to build for a Versatile PB with an arm926ej-s CPU… it failed to build the cross-toolchain, even though it pretty much is the exact same target.

A make cleandirs later, and it happily started building everything, but then hiccupped on permissions, so against my better judgement, I’m running it now with sudo, and things are progressing.  With some luck, I should have something that will give me a working native gcc/g++ for musl on ARMv5.

Solar Cluster: 2 days and counting on solar…

So, I’m home now for the Christmas break… and the fan in my power supply decided it would take a Christmas break itself.

The power supply was purchased brand new in June… it still works as a power supply, but with the fan seized up, it represents an overheating risk.  Unfortunately, the only real options I have are the Xantrex charger, which cooked my last batteries, or a 12V 20A linear PSU I normally use for my radio station.  20A is just a touch light-on, given the DC-DC converter draws 25A.  It’ll be fine to provide a top-up, but I wouldn’t want to use it for charging up flat batteries.

Now, I can replace the faulty fan.  However, that PSU is under warranty still, so I figure, back it goes!

In the meantime, an experiment.  What happens if I just turn the mains off and rely on the batteries?  Well, so far, so good.  Saturday afternoon, the batteries were fully charged, I unplugged the mains supply.  Battery voltage around 13.8V.

Sunday morning, battery was down to 12.1V, with about 1A coming in off the panels around 7AM (so 6A being drained from batteries by the cluster).

By 10AM, the solar panels were in full swing, and a good 15A was being pumped in, with the cluster drawing no more than 8A.  The batteries finished the day around 13.1V.

This morning, batteries were slightly lower at 11.9V.   Just checking now, I’m seeing over 16A flowing in from the panels, and the battery is at 13.2V.

I’m in the process of building some power meters based on NXP LPC810s and TI INA219Bs.  I’m at two minds what to use to poll them, whether I use a Raspberry Pi I have spare and buy a case, PSU and some sort of serial interface for it… or whether I purchase a small industrial PC for the job.

The Technologic Systems TS-7670 is one that I am considering, given they’ll work over a wide range of voltages and temperatures, they have plenty of UARTs including RS-485 and RS-232, and while they ship with an old Linux kernel, yours truly has ported both U-Boot and the mainline Linux kernel.  Yes, it’s ARMv5, but it doesn’t need to be a speed demon to capture lots of data, and they work just fine for Barangaroo where they poll Modbus (via pymodbus) and M-bus (via python-mbus).

Solar Cluster: Solar controller replaced, upgrading RAM and VM management

So, this morning I decided to shut the whole lot down and switch to the new solar controller.  There’s some clean-up work to be done, but for now, it’ll do.  The new controller is a Powertech MP3735.  Supposedly this one can deliver 30A, and has programmable float and bulk charge voltages.  A nice feature is that it’ll disconnect the load when it drops below 11V, so finding the batteries at 6V should be a thing of the past!  We’ll see how it goes.

I also put in two current shunts, one on the feed into/out of the battery, and one to the load.  Nothing is connected to monitor these as yet, but some research suggested that while in theory it is just an op-amp needed, that op-amp has to deal with microvolt differences and noise.

There are instrumentation amplifiers designed for that, and a handy little package is TI’s INA219B.  This incorporates aforementioned amplifier, but also adds to that an ADC with an I²C interface.  Downside is that I’ll need an MCU to poll it, upside is that by placing the ADC and instrumentation amp in one package, it should cut down noise, further reduced if I mount the chip on a board bolted to the current shunt concerned.  The ADC measures bus voltage and temperature as well.  Getting this to work shouldn’t be hard.  (Yes, famous last words I know.)

A few days ago, I also placed an order for some more RAM for the two compute nodes.  I had thought 8GB would be enough, and in a way it is, except I’ve found some software really doesn’t work properly unless it has 2GB RAM available (Gitea being one, although it is otherwise a fantastic Git repository manager).  By bumping both these nodes to 32GB each (4×8GB) I can be less frugal about memory allocations.

I can in theory go to 16GB modules in these boxes, but those were hideously expensive last time I looked, and had to be imported.  My debit card maxes out at $AU999.99, and there’s GST payable on anything higher anyway, so there goes that idea.  64GB would be nice, but 32GB should be enough.

The fun bit though, Kingston no longer make DDR3 ECC SO-DIMMs.  The mob I bought the last lot though informed me that the product is no longer available, after I had sent them the B-Pay payment.  Ahh well, I’ve tossed the question back asking what do they have available that is compatible.

Searching for ECC SODIMMs is fun, because the search engines will see ECC and find ECC DIMMs (i.e. full-size).  When looking at one of these ECC SODIMM unicorns, they’ll even suggest the full-size version as similar.  I’d love to see the salespeople try to fit the suggested full-size DIMM into the SODIMM socket and make it work!

The other thing that happens is the search engine sees ECC and see that that’s a sub-string of non-ECC.  Errm, yeah, if I meant non-ECC, I’d have said so, and I wouldn’t have put ECC there.

Crucial and Micron both make it though, here’s hoping mixing and matching RAM from different suppliers in the same bank won’t cause grief, otherwise the other option is I pull the Kingston sticks out and completely replace them.

The other thing I’m looking at is an alternative to OpenNebula.  Something that isn’t a pain in the arse to deploy (like OpenStack is, been there, done that), that is decentralised, and will handle KVM with a Ceph back-end.

A nice bonus would be being able to handle cross-architecture QEMU VMs, in particular for ARM and MIPS targets.  This is something that libvirt-based solutions do not do well.

I’m starting to think about ways I can DIY that solution.  Blockchain was briefly looked at, and ruled out on the basis that while it’d be good for an audit log, there’s no easy way to index it: reading current values would mean a full-scan of the blockchain, so not a solution on its own.

CephFS is stable now, but I’m not sure how file locking works on it.  Then there’s object storage itself, librados.  I’m not sure if there’s a database engine that can interface to that, or maybe to Amazon S3 storage (radosgw can emulate that), that’ll be the next step.  Lots to think about.