Computing

Software development ethics

I’ve been doing a lot of thinking about my choices of software of late in the face of “the sloppening”. LLMs have infiltrated many open-source projects, some of these being ones I’ve used for decades.

This has put me between a rock and a hard place. On the one hand, I wish to support ethical choices in technology. LLMs are not ethical. On the other, security vulnerabilities will drag me down the road of LLM use whether I want it or not: we can’t simply ask attackers “please don’t use LLMs” — if that worked then our prison systems should be empty.

My objections to LLM use

I covered this in detail in my last post, but it boils down to this:

  1. LLM output, whilst itself not directly applicable to copyright (as it is not the work of a human), can sometimes contain the copyrighted work of other humans (e.g. like this) in violation of the license agreements that those copyrighted works are distributed. There exists a legal risk that users of “derivative works” of copyrighted material may be found liable for unauthorised use of this copyrighted material.I don’t want that risk!
  2. Current generation LLMs are trained on material that is merely scraped off the Internet using poorly coded and highly inefficient scraper bots, then labelled with the aide of humans — often slave labour in developing countries. — I do not wish to support the slave trade!
  3. Whilst the effort right now with LLMs is to achieve “human-level” intelligence, the ability of current technology to effectively simulate a neural network with the complexity and density of a human brain is severely lacking, requiring vast data centres that consume exorbitant amounts of electrical energy and producing vast amounts of thermal energy that must then be radiated away, represents a threat to our climate ecosystem that cannot be justified at any cost. — I value the natural environment that our society exists in!
  4. Development of these tools is progressing at such a rapid pace, that current generation equipment depreciates at a rapid rate, requiring frequent replacement. Semiconductor production is necessarily a resource intensive industry and the frequent manufacture–commissioning–decommissioning cycle is producing massive amounts of e-waste whilst putting heavy demand on these facilities, further compounding the already poor environmental footprint.
  5. The output of these tools is often wrong, requiring the user to send the work back to be re-worked. Wasted time and effort. Computers in the 1950s might’ve been wrong occasionally, but it was generally traced down to failure of hardware components, the mathematical concepts underpinning this were sound, better manufacturing and components fixed this. “Hallucination” (really confabulation) is presently believed to be unfixable.
  6. It is said that relying on these tools has been shown to reduce one’s cognitive abilities, increasing dependence on these tools for functional work.
  7. Financially, the companies running these systems are a house of cards on the verge of collapse. No LLM vendor right now is making a profit. When the real bills finally arrive, will you be able to afford them? If they go bust, can your work continue without?

My legal objection to LLM use

The first of these is a legal argument, and really it’s going to take time, but eventually there will be a test case that will confirm whether my suspicions are founded. I am not a lawyer, so take what you see there with a grain of salt, but to be a successful software developer, one must have at least some basic understanding of the legal principles at play here, otherwise you find yourself in court (or in gaol) instead of in front of the computer.

I don’t have the money to fend off a potential case of copyright infringement, nor do I want to spend time having to mop up after someone is found guilty of such infringement. Code from a LLM, even if it looks novel and new to us because we’ve never seen it before, existed somewhere, and few have the resources to properly conduct a check to see where it came from.

Ethical objections

Points 2-4 point out the ethical issues, notably sourcing of training material, slavery and environmental concerns.

I fully expect that as technology gets better, we may see hardware that is better able to simulate the biological systems these tools try to emulate. Greater density, lower power consumption, less thermal waste output. The Manchester Baby was an early implementation of the Von-Neumann computer architecture that underpins the basic principles of every personal computer today… it needed 3.5kW to run this early 32-bit computer which had a mere 128 bytes (in today’s measurement; 1024 bits) of memory and ran at around 1100 instructions per second. There are ARM Cortex M0+ cores today (also 32-bits) with thousands of times more memory, clock speeds in the hundreds of megahertz and run full speed with less than 1W of power.

That part will get better. Even if they do fix that problem though, they still likely will be enslaving people to do the labelling skulduggery and using pirated material to boot. They need to solve this problem too, but so far, their efforts have been on trying to be made exempt from those rules.

Practical concerns

The financial situation suggests prices will skyrocket once funding dries up, and when that happens, people using these tools will face a choice: hike their fees to cover the increased costs, or abandon the tools en masse and return to the “old way” of doing things.

I also question the logic of feeding tokens into a machine and pushing a button, hoping that this time, we’ll hit the jackpot and get useful output. If I wanted to earn an income that way, I’d visit the local poker machine.

My actions going forward

There are two classes of software I need to consider, the software packages I simply use, and those which I actually contribute to. In both cases, some have viable replacements that I can switch to, some may have viable replacements in the future, some I still need to keep using, some I can abandon.

Linux kernel

Dealing with this is a tough nut. The core kernel project is now using LLMs as part of their core workflow. Linus Torvalds basically told us to fork it or leave if we don’t like it. I considered both:

“Fork off”, use something else

This isn’t a full list, but I looked at several options:

  • NetBSD looks like a good option for a lot of my servers. It is open-source, POSIX-compatible and does many of the things I need of a server. At the moment this blog runs on AlpineLinux, but moving it over to NetBSD is very doable.

    As a desktop I gave it a try, hardware support being its big Achilles heel — it can’t tap into the hardware support Linux has without tainting the kernel with GPLv2-licensed code, so drivers must be written from first principles. Support for interfaces like Bluetooth are really lacking (mind you, it at least supports it unlike OpenBSD). It feels a lot like Linux circa early 2000s. It’s a good base we could build from, but lots of work is needed.

    For non-x86 platforms, the situation is quite spotty. My TS-7670s basically would require me to dig through the Freescale i.MX28 user manual and port the kernel to this new (to NetBSD) SoC, before then porting the TS-7670 rev D board. My Raspberry Pis are harder still, because much of the documentation for these is under a NDA from Broadcom. A Linux fork is really my best option for these.

    While the kernel is free of slop, there are some packages shipped in the base OS that have some slop in them (e.g. Postfix). Not a lot we can do there.
  • OpenBSD is also an option on the servers, and I already do run some with OpenBSD. tmux is vibe-coded now, but that’s not a core component of the OS (despite being shipped in base) and can be safely ignored — I use GNU screen anyway.
  • I can rule out FreeBSD as they now embrace the slop. Might as well stay with Linux in that case.
  • Similarly, I can strike out 4.4BSD fork DragonFlyBSD as they embrace slop too.
  • RedoxOS has a no-AI policy, but is a long way from being usable as a desktop as it lacks support for common hardware such as WiFi and is still in an early development phase. Worth looking at later, but not viable yet.
  • HaikuOS has come a long way. It’s a re-implementation of BeOS, and like RedoxOS, has a no-AI policy. Many common software packages run on it. I played with it in a VM and it actually did feel like a nice desktop OS. There is an ARM64 port of it, but when I tried that on my Raspberry Pi 400, it refused to boot. So not ready to displace Linux yet, but looking good.
  • Illumos is a fork of OpenSolaris, so basically a genetic Unix rooted in BSD and Sun ancestry. Has an anti-AI policy, supports a lot of software, but hardware support is lacking.
  • 9Front looks interesting, but very esoteric. It is a continuation of Bell Labs Plan9. It too has a no-AI policy, but hardware and software support is probably not going to make it a practical choice for me.
  • GNU Hurd doesn’t seem to have moved far from where it was in the 90s when Linux first started and displaced it as “the” GNU kernel. It is clean of slop as far as I can tell, and it is even supported by Debian and Gentoo, Hardware support is lacking, but being a GPLv2 kernel, we could possibly port Linux drivers and features over to it — this would be a big job though.

I might’ve missed something, but that looks like a pretty extensive list. Of that list, NetBSD seems to be the best approach, failing that maybe getting behind HaikuOS (for a desktop) or filling the gaps in Hurd (if we want to re-purpose GPLv2 code).

“Fork it” approach

Under the terms of the GPLv2, we’re allowed to do this. The “Linux™” name is trademarked, and while it’s tempting to mention an alternate name just now, I know the moment such a name is mentioned, it’ll be off to the races to the nearest domain registrar to snag the .org domain for it ahead of the squatters. Linus himself already knows about people squatting names and profiting from it, which is why he now holds the trademark in the first place.

This is not something I can do alone. Arguably, I don’t feel I’m a skilled enough coder to do this either. I know enough C code to be dangerous, but this is playing with stuff deep in a system where I have never needed to stray before. That said, in the aim of “getting the ball rolling”, I made a start. I also made a map of where known slop is in the kernel so in the meantime, I can avoid touching it.

An uncomfortable truth with this: we will get security vulnerability reports from people that used LLMs to discover them even if we ask them not to use such tools.

I don’t want to condone such use in any way, but nonetheless, a security bug is a security bug. I don’t think it’s unreasonable to push back on submitters and ask for a human-written-and-verified proof-of-concept, but people will be feeding this code to LLMs to find holes whether we like it or not. Such a fork should aim to patch vulnerabilities with slop-free commits, but refusing to fix it at all because a LLM was involved isn’t a viable option as the bug can still be exploited.

Another uncomfortable truth, people will sometimes neglect to disclose LLM use, either accidentally or deliberately. Where we have obvious and clear evidence, we should reject such commits. Otherwise I’m willing to operate on the principle of “innocent until found guilty”, as its nearly impossible to tell just scrutinising a commit in isolation.

That said, I feel like maintaining a long-term support fork that just focusses on keeping the security updates maintained will give those of us with existing systems a viable platform that will allow us to run our existing software stacks. This won’t be “big and professional” like Linux (yeah, I know), but if a few of us band together and pitch in, it’ll give many of us what we need in the immediate term whilst the alternatives get up to speed.

Linux distributions

I already use Gentoo as my desktop OS of choice, so there I’m clear. I have a few machines that run other OSes, notably a couple of Ubuntu and Debian servers, my Ceph storage cluster all run Debian, my Mastodon instance is Debian. I have a tablet running Debian 13, and Raspberry Pi OS on a few Raspberry Pis.

Most of my servers run AlpineLinux, I do not know what their policy is, but so far I’ve not seen any slop creeping in.

I have run Gentoo on servers before, but I think if a re-load is on the cards, I’ll be looking at NetBSD first. Failing that, it may be a case of I figure out a build environment that can keep grinding away at security updates for my fleet so I can perform updates in a timely manner.

Desktop environment

My current desktop is FVWM. This is a X11 window manager, and while FVWM itself is slop-free, the next release of X.org server is not. I am watching CoW very closely, and have given it a try from time to time. Wayland itself seems to be slop-free, but it is lacking some features of X11, notably accessibility is a big issue.

For my tablet, a decent on-screen keyboard is a must, right now I use Onboard which I’ve found to be the best option (although I note now, they have jumped in bed with Claude as of June this year), but this only works on X11. Things I need: basically look at any laptop keyboard — letters and digits, with symbols in their usual places, all modifier keys, function keys and navigation keys. Just because I’m using a tablet does not mean I won’t be interacting with pointer-naïve terminal applications, possibly via SSH… and sometimes interacting with a text field is easier done with on-screen arrow keys than with touch, not even Apple or Google can make pure-touch work.

I need to figure out a screen locking solution, I use XScreensaver at present which is a good option, but it depends on features that are X11-only. It also does not work with on-screen keyboards. I theoretically could fork it, but given recent experience, I’m not sure that my derivative work would be welcome. Better approach might be to deal with a screen locker that maybe isn’t as pretty, but achieves the necessary job: locking the workstation securely on a Wayland desktop. For the tablet, it’ll need on-screen keyboard support — I need to be able to unlock without a USB keyboard plugged in.

I have not seen a Wayland launcher that I like, so I’m quietly working on my own based on ideas I came up with when I last pondered desktop environments.

Other applications, I’ve previously used a lot from the KDE desktop, but that is something I’m starting to review as they embrace the slop. I historically have disliked GTK+ for its insistence on spewing error logs to stderr, but nonetheless, as at this time, this is the only main-stream widget set now that remains slop-free with Qt, wxWidgets and FLTK all embracing slop.

So where I had a Qt preference years ago, I’m now moving to GTK+: it appears the UI toolkit might be finally over, with the opposition choosing self-sacrifice and GTK+ winning by default.

Bluetooth stack

Right now I use BlueZ, and it is useful as a means of moving files to/from the mobile phone as well as a key part in managing the audio links to headsets. If I switched OSes, I’d be looking at something with a similar feature-set.

Sadly it embraces the slop now. Thankfully I haven’t needed to dig into its code… I can put up with it for now, but a more ethical option would be welcome.

Audio subsystem

Right now I use pipewire for managing audio as it is a very usable blend of features from PulseAudio and JACK. I like the ability to filter audio through applications where needed and would like to keep this feature. Its integration into the BlueZ Bluetooth stack is good too.

I think JACK is cross-platform, but PipeWire is likely Linux-only, and it recently has embraced the slop. I know OpenBSD has its own called sndiod, this probably can compete with PulseAudio but not with JACK/Pipewire.

For now unless I need to actually debug it, I can “put up” with it.

Web Browser

I dislike Chromium because of its multi-day long builds from source code, it’s a very time consuming package to compile from source. As a Gentoo user, that is the default way it is shipped, as source code you compile yourself. I also dislike the idea of an Internet web browser mono-culture. We had this 25 years ago with IE and it was hell.

The only thing in Chromium’s favour is that it is open-source, so we can port it and build it ourselves, but that’s about where the advantages end. The principle maintainer, Google, are a major LLM vendor, and very much want to encourage people to use their services including Gemeni. I haven’t checked, but I can’t imagine Google not using LLMs to code their browser engine.

I’m a Netscape user from way back: in 1997 it was the only viable option for getting on the Internet with Linux and so that’s what I got to know. When Mozilla formed from Netscape’s ashes, I switched over to using the full Mozilla suite (which lives on as Seamonkey) and when Firefox came, I switched to that. Building Firefox from source can be done in a couple of hours on even reasonably modest hardware, vs about 5 days for my teenage laptop to build Chromium.

Mozilla have since come out as pro-LLM, a tone-deaf stance if ever I heard one, so much they not only use LLMs to code, they also integrate LLM features in the browser.

That said, they do include an OFF switch for the LLM features, and that I do respect. I’d have preferred if they made it an extension you chose to install if you wanted it, but they at least let me turn it off. This is akin to having a gun in a safe at home but not possessing any ammunition for it. I’d rather not have the gun in the house at all, but at least I’m not forced to have it loaded.

Servo Browser is one to watch in this space. It’s a ground-up implementation of a new HTML renderer to compete with Blink/Webkit/KHTML and Netscape Gecko. It’s a long way from daily driver status, but well worth getting behind, and they are making good progress. I wish them well and continue to follow from a distance. I’ll probably be testing some of my web-based projects with this engine going forward.

I’ve tried building Waterfox, but so far haven’t cracked it… so for now I stick with Firefox ESR. Last time I had to delve into its source code, it was trying to produce a compromise patch that would enable Firefox to build on more common MIPS platforms (e.g. SGI and Lemote) whilst retaining the compatibility for the PlayStation II that the Mozilla developers wanted to keep. Firefox is partially coded in Rust now so this is a moot point: Rust doesn’t work on MIPS.

As such, I haven’t needed to mess with its code in a long time. So short term, I’ll stay where I am, and start looking at getting behind Servo as the replacement.

Office Suite

Text documents I tend to do with LaTeX. This so far remains slop-free, but they’ve recently adopted a policy that permits LLM-generated commits so it won’t remain that way for long.

For spreadsheeting, gnumeric is “good enough”, and in fact its Engineering notation I find is a killer feature. It remains slop-free.

I do have LibreOffice installed for more complex tasks and interfacing with Microsofties. They’ve said they won’t add LLM features to LibreOffice (a stance I appreciate), but they have allowed its use with writing its code. Again, I don’t need to debug it, so I’ll deal with that later.

EDA

I don’t just write code, I design circuits too. I’ve bounced between a few, but the two I’ve used in the open-source world are KiCAD and GEDA. KiCAD is by far the better supported package. Sadly contains a lot of slop in it now, but once again, so long as I don’t interact with its code I’m not tainting myself.

There are alternative toolchains too, maybe the answer isn’t a “graphical” schematic capture, but to do something more like is done for FPGAs, a hardware-description language that translates to a netlist?

Graphics

Right now I use three tools principally: GIMP, Krita and Inkscape. GIMP recently started accepting slop (just the one commit) as have Inkscape (two commits). Krita remains clean (for now).

I mostly still use GIMP v2 because of my use of the XSane plug-in which has not yet been ported to GIMP v3. Inkscape taking this slop on is annoying, but so far I’ve not needed to debug it, so I can possibly put up with it for now until I can jump ship to an alternative.

Audio editing

I normally use Audacity for basic needs, I should give Tenacity a look.

For more advanced tasks I’ve used Ardour but given I’ve not gone deep into learning Ardour, switching to something like LMMS isn’t a big issue.

Video editing

Historically I’ve used Kdenlive, so far it remains clean. ffmpeg though is a mess.

Text editing

My bread-and-butter, editing plain text files is how I’ve been earning a living for the past 18 years or so of professional life. My go-to until recently was gvim, but as they now embrace the slop, I switched over to hard fork gevi. An easy switch because EVi at this time, is practically identical to Vim, all my plug-ins and configs JustWork™.

This blog

This blog currently uses WordPress 7.1, which includes their Gutenberg block editor. I did look at alternatives some time back, I use this WordPress instance as an ActivityPub node, and that’s not something that’s well supported in the other platforms.

Migrating to something else is also not so easy… thankfully I host it myself so I have control of my data either way.

My social media instance

I run my own Mastodon server, have done for a couple of years now. There is some (two commits) slop there, but generally things have remained clean. The modifications I’ve made were done long before this slop appeared, and I’ve merely been using git stash save / update / git stash pop to move the patch between releases.

Migrating to something else is not exactly trivial, even changing hostnames isn’t easy.

Writing my own software

So I write a lot of my own software. When I needed an embedded AX.25 stack for a project for Brisbane WICEN, I wrote aioax25. When I needed a SSTV encoder after finding pySSTV wasn’t performant-enough, I wrote libsstvenc. I’m in the process of writing my on launcher.

You’ll see lots of examples on my Github and Codeberg repositories.

Going forward, I’ll probably start looking more carefully at my dependencies. Notably, I do not wish to force a hard-dependency on a LLM-infected project unless there is no viable alternative.

aioax25 is actually tested on Python 3.9 and up, and I’ll keep supporting this as long as I practically can. The code is written with Python 3.5 in mind, so it should still work there too even though I don’t regularly test it on that. This in theory opens the door to use on other Python-compatible language interpreters like Omegathon (untested). It also has no hard dependency on an OS: it should in theory work just fine on NetBSD/OpenBSD as it does on Linux.

libsstvenc is straight C and doesn’t itself have any dependencies, the example programs support libgd which is clean of slop for now. GNU GCC have stated they won’t allow LLM-generated commits for the compiler itself, although there are a couple of commits already for their Fortran compiler, their C/C++ compiler remains clean for now. If I remain conservative with the version of C standard, this allows use of any pre-slop C compiler that meets the spec.

Checkpoint Reporter is written in pure Java using the OpenJDK v8 SDK. I can confirm it runs fine on NetBSD, and so far, big LLM boosters Oracle have requested that people not use LLMs for contributing to OpenJDK. This position I do welcome, but it does make me raise an eyebrow given how pro-LLM, if it’s supposedly so good, why not? Their lawyers must know something they’re not willing to share. I might have to review its use of Maven, this was a default option when creating a project in Netbeans, it may be that good ol’e Ant might be the better option. I’ll have a look.

My biggest gripe with Java remains their reluctance to consider serial devices: all solutions I can find require JNI (with bindings compiled for each desired platform, so much for write-once-run-everywhere, Ugh!). Highly annoying.

The launcher I’m writing, right now I’m trying gtkmm and friends as an alternative to Qt, which was my previous go-to UI toolkit. Again, if I stick to older C++ standards (gtkmm requires C++11, C++20 is widely supported too), people can choose to use an older release of GCC or LLVM that’s pre-slop, or in theory, any other compiler that supports that revision of C++. gtkmm itself too, is clean, and if I support as old a release as practical, that’ll enable the launcher’s use on many pre-slop versions of Linux distributions and other Unix-like OSes. I use C++ here because it allows the tools use on platforms where more recent languages such as Rust and Zig are not available.

I might have a closer look at Rust, I’ve done some hello-world level stuff with it. The memory-management approach is an interesting one, and while their policy allows “LLM ingestion” (meaning they let people feed its code to a LLM), they’ve stated that any contributions must be human-written (not just human-reviewed).

Zig may also be worth a look, it is manual memory management like C/C++, but it doesn’t have some of C’s flaws, and they have an anti-AI policy. Right now they use the LLVM back-end (which is itself, quite heavily LLM-tainted) but I hear they’re working on an alternative, so maybe in time they may sever that dependency (at least on common platforms).

eC is another I might look at. Rather than implementing the full compiler, theirs is effectively a transpiler to C. So it can sit atop gcc, clang or anything else.

I have written Go code in the past, but won’t be doing so from here on in. Especially as Go as a language isn’t anything special memory-management wise: it uses garbage-collection the same as Java and C#.

I mentioned Omegathon which is a Python 3.9 fork, Tauthon forks Python 2.7 and adds some interesting features from Python 3. Given I usually enjoy working with Python, these might be worth looking at closer.

For JavaScript, I lately have been using ReactJS because of my need to understand it at work, but I really haven’t enjoyed dealing with ReactJS and so will look at other options. The elephant in the room here is tools like webpack that are LLM-infested. Avoiding slop in this ecosystem is not an easy or trivial task, especially as there isn’t a mainstream open-source browser that is clean.

In short, while slop is invading a lot of projects, there looks to be some scope to dodge the slop when creating new software, and there are things I can do to ensure existing software I write can be used in a mostly slop-free environment if that is something people want. I aim to try to do this where practical.

My work situation in the age of “AI”

So its been a few years since I put my opinion piece on generative AI.

I continue to do work that is free from the outputs of large language models and still hold the view that these statistical models, whilst most definitely artificial are not intelligent. It looks intelligent because the viewer has never seen this content before, and is interpreting it from the view point of wanting it to be intelligent. The output would have existed somewhere. And in there, lies the danger from a legal standpoint: if that somewhere was a place the author very much wanted to stay private, there’s potential for copyright infringement and misappropriation of others’ intellectual property. Amnesty International calls it unlawful by design.

Much of this material is scraped with no regard to resource limits at the source, sometimes causing denial-of-service for legitimate users. In one case, Anthropic’s ClaudeBot found a mirror of the Linux kernel tree on infrastructure I run, and decided to scrape that. It hit the VM so hard, it triggered a high-temperature alarm on the server concerned! Now there are two points I’d make here:

  1. There copies of this same source code on “faster” servers than mine, like kernel.org itself or any of its mirrors.
  2. They could just use git clone to grab their own copy of the code, and scrape that instead of asking my server to spend CPU cycles fetching the source, the HTML pretty-printing the source, only for them to extract the source code they came to scrape.

The latter point shows a total disregard for doing things efficiently. The former point shows just how little intelligence these systems actually have… a few pages ought to have been enough to say “I’ve seen this before, nothing new here, move on”… but no, they kept hammering.

Much of this raw material then has to be categorised and sorted before it can be fed into the training algorithms. More often than not, slave labour does this. This isn’t talked about much, but cheap labour from second/third world countries is the principle source for this labour.

The larger-scale systems continue to have an appalling environmental record, both in terms of their running costs, but also in their construction. Companies want to build “AI” data centres here in Australia… gigawatt data centres. South Australia’s peak demand hit 3.274GW in 2025, and there are proposals to build data centres that draw a third of this peak, all the time. It’s good that the Australian Government demand that such facilities build “new renewable generation“, however that consumed energy must go somewhere, and it’ll be into the atmosphere around the facility. The environmental impacts will likely be enormous.

This is just running costs, there is also hardware replacement costs, because this stuff works hard, wears out quickly, and is obsoleted even faster. Cory Doctorow points out that this equipment is allegedly replaced every 2 years, with the new replacements requiring a full datacentre refit as the new parts are not backward compatible with old installations. Semiconductor production requires a lot of water, a lot of very pure water, and high energy consumption. This is hardly environmentally friendly!

As such, I have big ethical objections to using this stuff. I also have big legal objections to working with the output of these systems. Using a library that is coded using these tools, sometimes you can be lucky and things work… but if you ever have to debug that code or write a pull request, you’ve then got to dig into the slop. Get your hands dirty with it… figure out exactly what it’s doing, perhaps what you or it is doing wrong. You become tainted by that slop.

I raised these issues with my workplace last year… their response was two fold:

  1. they put out an “AI policy” dictating how it should be used
  2. they subscribed to Anthropic’s Claude service

Now my work colleagues, instead of writing code, are cosplaying as IT managers micromanaging virtual junior developers. There is slop everywhere. Not only did management pretend they could “policy” their way out of a situation that even the High Court of Australia has not yet decided on… but they’ve actively encouraged everyone to join everyone on the slop train, because otherwise they’ll be “left behind”.

Funnily enough, our very own code-of-conduct has words about this:

Protect the environment by making environmentally friendly choices

It would seem they do not honour their own code-of-conduct. A facility that needs energy equivalent to a third of an entire state’s peak demand doesn’t sound like an “environmentally friendly” choice to me.

And while I do value co-workers contributions, the ones they themselves produce… that value is not required to extend to contributions made by external parties. Anthropic Claude is not a human, and as such in Australian law, cannot be held accountable for its actions or enter into contracts. Therefore it cannot be considered an “employee” or co-worker. I will not value its input here.

Even if LLMs did not have these issues, there is also the matter of cost. My workplace frequently cried poor when I asked for some equipment to do our jobs properly. This meant we were guessing with high-stakes projects, and this stress was causing me burn-out. Automated systems badgering me to submit timesheets (I always did, the “reminder” often arrived 3 days after I had submitted completed timesheets), or enable unwanted LLM features further intensified this burn-out.

The reality of my work had changed too… I’m mostly a firmware developer these days as JavaScript’s idiosyncrasies drive me up the wall, and the project I do this for, is a mature product. There isn’t that much work to do, just occasional maintenance of the code base. It didn’t justify 5 days a week, and I was stressing out trying to figure out how to fill-out a 38 hour week. So I dropped back to 3 days part-time… 22½ hours a week.

That worked well, there was a project to port this firmware over to a new hardware platform, that kept me busy… until late last year. I started my own little project to make a test rig for this firmware project, so we could have automated testing of firmware images — something I’ve wanted to do since about 2019. I got a prototype going, and had boards on order for me to build: the order got cancelled, because they were “not going ahead” with that project.

So that killed the only thing that has been keeping me busy the past few months. I’ve been thumb-twiddling the past few weeks since returning from long-service leave. I feel my head is on the chopping block, and I’m just waiting for the executioner to swing the axe. My thought is the sooner this happens, the better!

I have big concerns about the financial viability of my workplace if they start building their whole workflow on these LLMs. As Cory points out in that video, the financial situation of these tools is not great. Anthropic for example, is estimated to have earned US$22 billion, but have spent US$331.1 billion… a loss of US$309.1 billion… in the time they’ve been operating. Nikkei Asia believes may of these leading LLM providers are hiding as much as US$1.65 trillion in debt. Oracle has apparently over-invested so hard in “AI” that S&P downrated their credit rating to “just above junk”.

This is a financial house-of-cards waiting to be blown apart by the slightest puff of breeze. The financiers are waking up to this reality, and are now starting to ask hard questions of company boards. Much of LLM subscriptions are subsidised by venture capital, and when this funding is pulled, LLM subscription costs will skyrocket. Businesses that are dependent on these tools will be sent to the wall with crippling bills if they don’t reign in their usage.

Back at the tail end of 2010, I was hired as a contractor for the organisation that would, in 2022, itself be acquired by a larger organisation. I was a contractor until they hired me full-time-permanent in 2013. I had offers from more prestigious places, but this was a “devil I know” situation, and at the time, I was comfortable where I was.

A fatal accident would forever change the culture of the place, but we largely stayed “the same” up until 2020. Just prior to that fateful day, a colleague and I were working on a re-vamp of the office network. A project spearheadded and lead by the person who sadly lost their life that day. I poured my heart and soul into that project. When we did switch over, we had something that would be the envy of organisations today: all our data was hosted in-house, email, calendars… working in any standards-compliant client. A network where the client OS largely didn’t matter, you could use the tools that suited what you did.

After that event, I wanted to quit, but I kept going because at that point, projects that we collaborated on now had a bus-factor of 1… me. So I stayed, and kept going. A couple of years later, the firmware project that would become basically my sole work started… and that, as frustrating as it was at times… was engaging and rewarding. That too, I poured my heart and soul into.

2020 hit, and lo and behold, we’re locked down. This was a big shift, and it got frustrating at times. But I stuck with it. Sale of the business was also a big topic… with various organisations looking at acquiring us. I had read the horror stories of what can happen when a little firm gets swallowed up, but I was assured “nothing would change”. When the purchase finally did happen in 2022, I was apprehensive, but I made a point of “giving it a go” first. After all, maybe this would be different?

One of their first acts was to rip out the infrastructure I had poured my heart and soul into setting up… in favour of Office365. When this was announced, I couldn’t help this was merely being done for the sake of corporate fashion, much the same reason LLMs are being adopted today.

I thought, okay, Microsoft has changed their stance, maybe they’ve improved and their mail system isn’t the standards pariah it was in the past… wrong… we wound up with a mail system that couldn’t even deliver a email without subtly molesting it (enough to make PGP signatures fail). Later they’d manage to break sending on my account, so I couldn’t send an email from my work account, to anyone. Then they broke inbound email so I couldn’t email people at work from my home email address. I reported this, but yeah… years later, the problem persists. I’m fed up with it. I can talk to other Microsoft Office365 users from my home email server, just not my workplace. I can only see this as a deliberate decision they made to configure their server this way.

In 2023, my 10 years full-time service came up. I was starting to show signs of burn-out, but I figured if I quit at this point I’d never reach this milestone. It was a struggle to finally make it to that point, but I finally got there. A small amount of stress relief. With this, came some long-service leave. I also had a stack of annual leave I hadn’t yet used. After this milestone, the decision to drop to part-time permanent was made in an effort to recoup some work-life balance.

Now in 2026, I think my time there is done. The writing had been on the wall for some years. Spoken words might say one thing, but it was clear the direction was more towards front-end applications, which I can do, but I find the most frustrating. The incursion of LLMs into my work was the final straw. A year ago I delivered an ultimatum, it’s me or the LLM Late last year I warned that these tools would ultimately make my position untenable for legal and ethical reasons.

Looking around the office in mid 2026, hearing the chatter… I see they’ve chosen the LLM: I am done. Obsolete, and ready for the scrap heap.

Work-wise, I’m an oddball: while I have a preference for back-end work, I am a full-stack developer with talents that extend right down to circuit design. An “overqualified rejection” magnet. I also am a Linux specialist that’s starting to dabble in BSD and has absolutely NO interest in Microsoft, not even their online services. You can see my CV here.

That said, I’m not in a hurry and do not have a position lined up. I expect to take a break for the first bit of 2027 so I can process what was the last 16 years of my life… and figure out where I can go from here. Long-term employment might be my future, but we’ll see. I get the feeling yanking cables through conduits is going to be a better bet than software in 2027/2028 because it’s very hard to get a LLM to do that.

Batch-downloading artifacts from Atlassian Bamboo

Atlassian Bamboo is a CI/CD server intended for large organisations to run internal builds. One thing that the API designers forgot with this system was a way to enumerate and download artifacts from the CI server. Not easily anyway.

The Bamboo server has a reasonable REST API that does about 60% of the work for us, it’ll let us query what the last successful build was, and give us the metadata details of that, as JSON, which we can pick through to get out what we need: mostly this is just the build number.

With that, we can hit up the artifacts endpoint and download the rest, but there’s two flies in the ointment:

  1. While the REST API supports Bearer authentication using a personal access token, the artifacts endpoint does not: one must use HTTP basic authentication
  2. To download directory trees, one must scrape the HTML and walk the tree

Thankfully (2) can be achieved using wget --mirror.

# Bamboo server
: ${BAMBOO_URI:=https://bamboo.example.net}

# Output directory
: ${OUTPUT_DIR:=/tmp/output}

# Plan keys (with project key prefix)
: ${PLAN_KEYS:=PK-PROJ01 PK-PROJ02}

if [ -z "${BAMBOO_USER}" ]; then
	read -p "Bamboo username: " BAMBOO_USER
fi
if [ -z "${BAMBOO_PASSWORD}" ]; then
	read -s -p "Bamboo password: " BAMBOO_PASSWORD
fi

bamboo_wget() {
	wget 	--http-user "${BAMBOO_USER}" \
		--http-password "${BAMBOO_PASSWORD}" \
		"${@}"
}

# Download the latest artifacts from the following projects
for plankey in ${PLAN_KEYS}; do
	bamboo_wget	--header "Accept: application/json" \
			-O "${OUTPUT_DIR}/${plankey}-result.json" \
		${BAMBOO_URI}/rest/api/latest/result/${plankey}?buildstate=Successful\&os_authType=basic

	buildno=$( jq -r .results.result[0].buildNumber \
			"${OUTPUT_DIR}/${plankey}-result.json" )

	bamboo_wget -P "${OUTPUT_DIR}/${plankey}" --mirror -nd -np \
		${BAMBOO_URI}/artifact/${plankey}/shared/build-${buildno}/?os_authType=basic
done

# ${OUTPUT_DIR} will accumulate some cruft from `wget --mirror` but all the files
# should be there.

Hopefully it won’t be much longer and I won’t have to deal with this house of horrors much more, but for others that are stuck with it, here’s something that may help.

Using image resources with Maven Java projects in Netbeans

Last Easter, I was running a checkpoint at Imbil as I’ve done before… operating a checkpoint at Derrier Hill Grid with horses passing through from three different events simultaneously, coming from two different directions, and getting more confused than a moth in a light shop. At that time I thought it’d be really handy to have a software program that could “sort ’em all out”. I punch in the competitor numbers, it tells me what division they’re in and records the time… I then assign the check-points and update the paperwork.

We have such a program, a VisualBASIC 6 application written by one of the other amateurs, however I use Linux. My current tablet, a Panasonic FZ-G1 Mk1, won’t run any supported version of Windows well (Windows 10 on 4GB RAM is agonisingly slow… and it goes out of support in October anyway), but otherwise would be an ideal workhorse for this, if I could write a program.

So I rolled up my sleeves, and wrote a checkpoint reporting application. Java was used because then it can be used on Windows too, as well as any Linux distribution with OpenJDK’s JRE. I wanted a single “distribution package” that would run on any system with the appropriate runtime, that way I wouldn’t need a build environment for each OS/architecture I wanted to support.

One thing that troubled me through this process… was getting image resources working. I used the Netbeans IDE to try and make it easier for others to contribute later on if desired: it has a GUI form builder that can help with all the GUI creation boilerplate, and helps keep the project structure more-or-less resembling a “standard” structure. (This is something that Python’s tkinter seriously lacks: a RAD tool for producing the UIs. The author of the aforementioned VB6 software calls it “T-stinker”, and I find it hard to disagree!)

Netbeans defaults to using the Maven build system for Java projects, although Ant and Gradle are both supported as well. (Not sure which one of the three is “preferred”, I know Android often use Gradle… thoughts Java people?) It also supports adding bitmap resources to a project for things like icons. I used some icons from the GTK+ v3 (LGPLv2) and Gnome Adwaita Legacy (CCBYSA3) projects.

The problem I faced, was actually using them in the UI. I was getting a NullPointerException every time I tried setting one, and Netbeans’ documentation was no help at all. It just wasn’t finding the .png files no matter what I did:

2025-06-15T06:32:54.461Z [FINEST] com.vk4msl.checkpointreporter.ui.ReporterForm: Choose nav tree node test
Exception in thread "AWT-EventQueue-0" java.lang.NullPointerException
        at javax.swing.ImageIcon.<init>(ImageIcon.java:217)
        at com.vk4msl.checkpointreporter.ui.event.EventPanel.initComponents(EventPanel.java:239)
        at com.vk4msl.checkpointreporter.ui.event.EventPanel.<init>(EventPanel.java:63)
        at com.vk4msl.checkpointreporter.ui.ReporterForm.showEvent(ReporterForm.java:895)
        at com.vk4msl.checkpointreporter.CheckpointReporter.showEntity(CheckpointReporter.java:532)
        at com.vk4msl.checkpointreporter.ui.ReporterForm.navTreeValueChanged(ReporterForm.java:480)
        at com.vk4msl.checkpointreporter.ui.ReporterForm.access$100(ReporterForm.java:70)
        at com.vk4msl.checkpointreporter.ui.ReporterForm$2.valueChanged(ReporterForm.java:182)

Maybe it’s my search skills, or the degradation of search, but I could not put my finger on why it kept failing… the file was where it should be, the path in the code was correct according to the docs, why was it failing?

Turns out, when Maven does a build, it builds all the objects in a target/classes directory. When Netbeans runs your project, it does so out of that directory. Maven did not bother to copy the .png files across, because Netbeans never told it to.

I needed the following bit of code in my pom.xml file:

       <resources>
         <resource>
           <targetPath>com/vk4msl/checkpointreporter/ui/components/icons</targetPath>
           <directory>${project.basedir}/src/main/java/com/vk4msl/checkpointreporter/ui/components/icons</directory>
           <includes>
             <include>**/README.md</include>
             <include>**/*.png</include>
           </includes>
         </resource>
       </resources>

That tells Maven to pick up those .png files (in the com.vk4msl.checkpointreporter.ui.components.icons package) and put them, along with the README.md, in the staging directory for the application. Then Java would be able to find those resources, and they’d be in the .jar file in the right place.

Other suggestions have been to move the project to using Ant (which was the old way Java projects were built, but seems to be out of favour now?)… not sure if Gradle has this problem… maybe some people more familiar with Java’s build systems can comment. This is probably the most serious Java stuff I’ve done in the last 20 years.

I used Java because it produced a single platform-independent binary that could run anywhere with the appropriate runtime, and featured a runtime that had everything I needed in a format that was easy to pick back up. C# I’ve used for command-line applications at university, but I’ve never done anything with Windows Forms, so I’d have to learn that from scratch as well as wrestling MSBuild (yuck!). Python almost was it, but as I say, dealing with tkinter and trying to map that to all the TK docs out there that assume you’re using TCL, made it a nightmare to use. I didn’t want to bring in third-party libraries like Qt or wxWidgets as that’d complicate deployment, and other options like C++, Rust and Go all produce native binaries, meaning I’d have to compile for each platform. (Or force people to build it themselves.)

Java did the job nicely. Not the prettiest application, but the end result is I have a basic Java program, using the classical Swing UI interface that should be a big help at Southbrook later this month. I’ll probably build on this further, but this should go a big way to scratching the itch I had.

Blog news: user registrations now disabled

Today, I noticed I had an extremely large number of “new” users, about 206 accounts in total created, with an odd username pattern:

www.XXXXX.blogspot.YY - Z.ZZZ BINANCE

XXXXX was a randomly chosen set of letters, YY was one of many TLDs that Blogspot uses, and Z.ZZZ was some kind of price value. Clearly, spammers have found a new way to send spam: using the registration email that WordPress uses to confirm your email address is valid.

This blog has always allowed comments in one form or another. Originally I allowed anonymous user comments and ping-backs, but when those got abused, I disabled them, requiring that a user be registered with the site to comment.

That has been fine up until today. Some “BINANCE” arsehole decided that the username field was a perfect way to spray shite from one end of the Internet to the other. In total, 230 accounts were created in the past few hours, mostly to gmail.com email addresses.

Maybe the username field can be stripped on such emails so the only thing they can supply is the email address (basically making sending mass spam this way very difficult; they’d have to “encode” it in a sub-address … not all providers support this and they can do it different ways).

We’ll see what damage that’s done to the sender score on this site. I have a second route I can use for sending outbound email that’s got a clean reputation so not all is lost.

Comments for the time being will now be exclusively through ActivityPub.

lvmcache fun and games

This blog has never been on what I’d call, a high-performance server. In fact, things are a little on the slow side. I try to be frugal with my system resource allocation, with the assumption that my little site does not get a lot of traffic (much less since it’s no longer syndicated on Gentoo Planet). However, I think I managed to get the performance up a notch…

The site runs on my solar powered server cluster, with a couple of Ceph RBDs, one for the root OS and one for the data (MariaDB / www root / /home). The VM runs AlpineLinux. The VM host was over-provisioned with a larger SSD than required, allowing me to dedicate some space for local cache.

I had thought I could set something up that would organise the cache on the VM host, and abstract it from the VM, but so far, I’ve not gotten around to doing that. (I did have something sort-of working in OpenNebula with flashcache at work, but it was flaky.)

In libvirt, I provisioned a new RBD to serve as the backing store (thus keeping a pristine copy to roll back to should things go pear shaped), and a new LVM volume for the cache. For the time being, I moved the existing volume to be the last device. So I had:

  • /dev/vda: OS
  • /dev/vdb: Data volume
  • /dev/vdc: Cache volume
  • /dev/vdd: temporary Old /dev/vdb for data migration

Failed approaches

Firstly, what didn’t work for me, was bcachefs and bcache.

bcachefs

bcachefs wanted to fight me every step of the way, making formatting the volumes difficult with sketchy documentation (especially as I wanted a write-through cache to facilitate VM migration).

bcachefs format gives some very cryptic error messages, and has a somewhat quirky argument syntax for formatting. The command I figured out through trial-and-error was this:

bcachefs format \
    --replicas=1 \
    --durability=0 /dev/vdc1 \
    --durability=1 /dev/vdb1 \
    --foreground_target /dev/vdc1 \
    --promote_target /dev/vdc1 \
    --background_target /dev/vdb1 \
    --metadata_target /dev/vdc1

The problem was convincing mount to actually mount it. I was supposed to specify every device, but each time it flatly refused, no matter what order I used, it told me “no such device”.

bcache

This is the underlying caching logic that bcachefs was built on, so I figured I’d try that. This worked better, but I found AlpineLinux had no real knowledge of bcache, and thus did not provide any means for me to bring up /dev/bcache0 before localmount mounted it.

I could have written a OpenRC init script to do this, but I wasn’t certain about this path, so decided to put the idea aside.

Winning approach: lvmcache

Luckily lvm2 has a built-in method: lvmcache. After installing the lvm2 package in AlpineLinux, I blatted the partition tables on my two virtual disks, formatted them as LVM physical volumes, and added them to a volume group.

~ # pvcreate /dev/vdb /dev/vdc
  Physical volume "/dev/vdb" successfully created.
  Physical volume "/dev/vdc" successfully created.
~ # vgcreate data /dev/vdb
  Volume group "data" successfully created
~ # vgextend data /dev/vdc 
  Volume group "data" successfully extended

Now to create the logical volumes, first… I created the volumes themselves. This wound up being a little tricky because I wanted to use all the available space on each volume… I had tried specifying -L ${SZ}G but this ignored the fact that LVM uses a bit of header space on each physical volume. It complained, but in doing so, told me the size in extents that was available, so I was able to use -l ${SZ} to specify that number of extents:

~ # lvcreate --size 8G --name datavol data /dev/vdb
 Insufficient free space: 2048 extents needed, but only 2047 available
~ # lvcreate -l 2047 --name datavol data /dev/vdb  
 Logical volume "datavol" created.
~ # lvcreate -n cachevol -l 4095 data /dev/vdc
 Volume group "data" has insufficient free space (1023 extents): 4095 required.
~ # lvcreate -n cachevol -l 1023 data /dev/vdc
 Logical volume "cachevol" created.

Now I had two separate LVM volumes, one on each physical device. Now to link them:

~ # lvconvert --type cache --cachevol cachevol data/datavol
Erase all existing data on data/cachevol? [y/n]: y
 Logical volume data/datavol is now cached.

Great, except I forgot to specify the write mode. Turns out, this is a lvchange away:

~ # lvchange --cachemode writethrough /dev/data/datavol  
 Logical volume data/datavol changed.

I could now format /dev/data/datavol with a filesystem, and migrate the data across. rsync here we come. An update to /etc/fstab and we were in business.

So far, things seem to be more snappy, so we’ll keep an eye on things. It’s survived a couple of reboots, the question is what happens when I boost a post on Mastodon, does all the ActivityPub instances out there cause problems? Guess I’ll find out in a moment.

New laptop: StarBook Mk VI

I rarely replace computers… I’ll replace something when it is no longer able to perform its usual duty, or if I feel it might decide to abruptly resign anyway. For the last 10 years, I’ve been running a Panasonic CF-53 MkII as my workhorse, and it continues to be a reliable machine.

I just replaced the battery in it, so I now have two batteries, the original which now has about 1.5-2 hours of capacity, and a new one which gives me about 6 hours. A nice thing about that particular machine is it still implements legacy interfaces like RS-232 and Cardbus/PCMCIA. I’ve upgraded the internal storage to a 2TB SSD and replaced the DVD burner with a Blu-ray burner. There is one thing though it does lack which didn’t matter much prior to 2020: an internal microphone. I can plug a headset in, and that works okay for joining in on work meetings, but if there’s a group of us, that doesn’t work so well.

The machine is also a hefty lump to lug around due to being a “semi-rugged”. There’s also no webcam, not a deal breaker, but again, a reflection of how we communicate in 2023 vs what was typical in 2013.

Given I figured it “didn’t owe me anything”… it was time to look at a replacement and get that up and running before the old faithful decided to quit working and leave me stranded. I wanted something designed for open-source software ground-up this time around. The Panasonic worked great for that because it was quite conservative on specs — despite being purchased new in 2013, it sported an Intel IvyBridge-class Core i5, whereas the latest and greatest was the Haswell generation. Linux worked well, and still does, but it did so because of conservatism rather than explicit design.

Enter the StarBook Mk VI. This machine was built for Linux first and foremost. Windows is an option, that you pay extra for on this system. You also can choose your preferred CPU option, and even choose your preferred boot firmware, with AMI UEFI and coreboot (*Intel models only for now) available.

Figuring, I’ll probably be using this for the better part of 10 years from now… I aimed for the stars:

  • CPU: AMD Ryzen 7 5800U 8-core CPU with hyperthreading
  • RAM: 64GiB DDR4
  • SSD: 1.8TB NVMe
  • Boot firmware: coreboot
  • OS: Ubuntu 22.04 LTS (used to test the machine then install Gentoo)
  • Keyboard Layout: US
  • Power adapter: AU with 2m USB-C cable
         -/oyddmdhs+:.                stuartl@vk4msl-sb 
     -odNMMMMMMMMNNmhy+-`             ----------------- 
   -yNMMMMMMMMMMMNNNmmdhy+-           OS: Gentoo Linux x86_64 
 `omMMMMMMMMMMMMNmdmmmmddhhy/`        Host: StarBook Version 1.0 
 omMMMMMMMMMMMNhhyyyohmdddhhhdo`      Kernel: 6.5.7-vk4msl-sb-… 
.ydMMMMMMMMMMdhs++so/smdddhhhhdm+`    Uptime: 1 hour, 15 mins 
 oyhdmNMMMMMMMNdyooydmddddhhhhyhNd.   Packages: 2497 (emerge) 
  :oyhhdNNMMMMMMMNNNmmdddhhhhhyymMh   Shell: bash 5.1.16 
    .:+sydNMMMMMNNNmmmdddhhhhhhmMmy   Resolution: 1920x1080 
       /mMMMMMMNNNmmmdddhhhhhmMNhs:   WM: fvwm3 
    `oNMMMMMMMNNNmmmddddhhdmMNhs+`    Theme: Adwaita [GTK2/3] 
  `sNMMMMMMMMNNNmmmdddddmNMmhs/.      Icons: oxygen [GTK2/3] 
 /NMMMMMMMMNNNNmmmdddmNMNdso:`        Terminal: konsole 
+MMMMMMMNNNNNmmmmdmNMNdso/-           Terminal Font: Terminus (TTF) 16 
yMMNNNNNNNmmmmmNNMmhs+/-`             CPU: AMD Ryzen 7 5800U (16) @ 4.507GHz 
/hMMNNNNNNNNMNdhs++/-`                GPU: AMD ATI Radeon Vega Series / Radeon Vega Mobile Series 
`/ohdmmddhys+++/:.`                   Memory: 4685MiB / 63703MiB 
  `-//////:--.

First impressions

The machine arrived on Thursday, and I’ve spent much of the last few days setting it up. I first checked it out with the stock Ubuntu install: the machine boots up into an installer of sorts, which is good as it means you set up the user account yourself — there’s no credentials loose in the box. Downside is you don’t get to pick the partition layout.

The machine, despite being ordered with coreboot boot firmware, actually arrived with AMI boot firmware instead. Apparently the port of coreboot for AMD systems is still under active development, and I’m told there will be a guide published describing the procedure for installing coreboot. Minor irritation, I was looking forward to trying out coreboot on this machine — but not a show-stopper… I look forward to trying the guide when it becomes available.

The machine itself felt quite zippy… but then again, when you’re used to a ~12-year-old CPU, 8GB RAM and a 2TB SATA-II SSD for storage, it isn’t much of a surprise that the performance would be a big jump.

Installing Gentoo

After trying the machine out, I booted up a SysRescueCD USB stick and used gparted to shove-over the Ubuntu install into the last 32GiB of the partition, then proceeded to create a set of partitions for Gentoo’s root, a 80GiB swap partition (seems a lot, but it’s 64GiB for suspend-to-disk plus 16GiB for contingencies) some space for a /home partition, some LVM space for VMs, and my Ubuntu install right at the end.

I booted back into Ubuntu, and used it as my environment for bootstrapping Gentoo, that way I could experience how the machine behaved under a heavy load. Firefox was, not bad, under the circumstances. My only gripe being the tug-o-war between Ubuntu insisting that I use their Snap package, and me preferring a native install due to the former’s inability to respect my existing profile settings. This is a weekly battle I have with the office workstation.

In discussing with Starlabs Systems, they mentioned two possible gremlins to watch out for, WiFi (important since this machine has no Ethernet) and the touch pad.

I used a self-built Gentoo stage 3, unwittingly I used one built against the still-experimental 23.0 profiles, which meant it used a merged /usr base layout… but we’ll see how that goes anyway… since it’s the direction that Debian and others are going anyway. So far, the only issue has been the inability to install openrc and minicom together since both install a runscript binary in the same place.

Once I had enough installed to be able to boot the Gentoo install, including building a kernel, I got the boot-loader installed, re-configured UEFI to boot that in preference to Ubuntu, then booted the new OS.

First boot under Gentoo

OS boot-up was near instantaneous. I’m used to about 10-15 seconds spent, but this took no time at all.

WiFi worked out-of-the-box with kernel 6.5.7, but the touch pad was not detected. Actually, under X11 the keyboard was unresponsive too because I forgot to install the various drivers for X.org. Oops! I sorted out the drivers easy enough, but the touch pad was still an issue.

Troubleshooting the touch pad

To get the touch pad working, I ended up taking the Ubuntu kernel config, setting NVMe and btrfs to being built-in, and re-built the whole thing again… took a long time, but success, I had the touch pad working.

The tricky bit is the touch pad is a I²C device connected via the AMD chipset, and described in the ACPI. Not quite sure how this will work under coreboot, but we’ll cross that bridge later. I spent a little time today refining the kernel down a little from the everything kernel that Ubuntu use… to something a little more specific. Notably, things you can’t directly plug into this machine (like ISA/PCI/PCIe cards, CardBus/PCMCIA, etc) or interfaces the machine did not have (e.g. floppy drive, 8250 serial), I took out. Things that could conceivably be plugged in like USB devices were left in.

It took several tries, but I got something that’s workable for this hardware in the end.

Final kernel configuration

The end result is this kernel config. Intel StarBook users might be better off starting with the Ubuntu kernel config like I did and pare it back, but that file may still give you some clues.

Thoughts

Whilst compiling, this machine does not muck around… being a 8-core SMT machine it actually builds things quite rapidly, although on this occasion I gave the machine a helping hand on some bigger packages like Chromium by using a pre-built binary built for my other machines.

Everything I use seems to work just fine under Gentoo… and of course, having copied my /home from the Panasonic (you never realise how much crap you’ve got until you move house!), it was just a little tweaking of settings to suit the newer software installed.

I’m yet to try it out a full day running on the battery to see how that fares. Going flat-chat doing builds it only lasted about 2 hours, but that’s to be expected when you’ve got the machine under a heavy load.

Zoom sees the webcam and can pick up the microphone just fine. I expect Slack will too, but I’ll find that out when I return to work (ugh!) in a fortnight.

My only gripe right now is that my right pinkie finger keeps finding the SysRq/PrintScreen button when moving around with the arrow keys… been used to that arrow cluster being on the far-right of the keyboard not one row back like this one. Other than that, the keyboard seems reasonable for typing on. The touch pad not being recessed sometimes picks up stray movements when typing, but one can disable/enable it pretty easily via Fn+F10 (yes, I have Fn-lock enabled). The keyboard backlight is a nice touch too.

The lack of an Ethernet port is my other gripe, but not hard to work-around, I have a USB-C “dock” that I bought to use with my tablet that gives me 3×USB-3A, full-size SD, microSD, 2×HDMI, Ethernet and audio out and pass-through USB-C for charging. The Ethernet port on that works and the laptop happily charges through it, so that works well enough.

The power supply for this thing is tiny, 65W with USB-A and USB-C ports. I also tried charging this laptop with a conventional USB-A charger but it did not want to know (the PSU probably doesn’t do USB PD). Should be possible to find a 12V-powered USB-C charger that will work though.

The Toughbook will likely be my go-to on camping trips and WICEN events, despite being a heavier and bigger unit, as usually I’m not lugging the thing around, it’s better ruggedised for outdoor activities, and it’s also looks about 10 years older than it really is, so not attractive to steal.

Leave at last

So… I’ve been busy at work lately, and that for the last few months has been my primary focus. A big reason why I’ve been keeping my head low is because a few years ago, it was pointed out that I had been physically with the company I’m working for for about 10 years.

Here in Australia, that milestone grants long-service leave; a bonus 8⅔ weeks of leave. This is something that’s automatic for full-time employees of a company, and harks back to the days of when people used to travel to Australia from England by ship to work, this gave that person an opportunity to travel back and visit family “back home”.

But at the time, I wasn’t there yet! See, for the first few years I was a contractor, so not a full-time employee. I didn’t become full-time until 2013, meaning my 10 years would not tick up until this year.

While the milestone is 99% symbolic, the thing is at my age (nearing 40), I’m unlikely to ever see that milestone come up again. If I did something that blew it or put it in jeopardy in any way, it’d be up in smoke.

There are some select cases where such leave may be granted early (between 7-10 years):

  • if the person dies, suffers total physical disability or serious illness
  • the person’s position becomes untenable
  • the person’s domestic situation forces them to leave (e.g. dropping out of work to become a carer for a family member)
  • the employer dismisses the person for reasons other than that person’s performance, conduct or capacity
  • unfair dismissal

I thought, it was worth sticking it out… after 10 years, it’s a done deal, the person is entitled to the full amount. If they booted me out after that, they’d still have to pay out that, plus the holiday leave (which I still have lots because I haven’t taken much since 2018).

Employment plans

Right now, I’m not going anywhere, I’ve got nowhere to go anyway. While doing work on things like electricity billing brings me no joy whatsoever (“bill shock as-a-service” is what it feels like), it pays the bills, and I’m not quite at the point where I can safely let it all go and coast into an early retirement.

Work has actually changed a lot in the past few years. Years ago, we did a lot of Python work, I also did some work in C. Today, it’s lots of JavaScript, which is an idiosyncratic language to say the least, and don’t get me started on the moving target that is UI frameworks for web pages!

Dealing with the disaster that is Office365 (which is threatening to invade even more into my work) doesn’t make this any easier, but sadly that piece of malware has infected most organisations now. (Just do a dig -t MX <employer-domain>, or just look at the headers of an email from their employees, many show Office365 today). I’ve so far dodged Microsoft Teams which I now flatly refuse to use as I do not consent to my likeness/voice being used in AI models and Microsoft isn’t being open about how it uses AI.

Most people my age get shepherded into management positions, really not my scene. In a new job I’d be competing with 20-somethings that are more familiar with the current software stacks. Non-technical jobs exist, but many assume you own a motor vehicle and possess the requisite license to operate it.

This pretty much means I’m unemployable if I leave or are booted out, so whatever I have in the bank balance needs to make it through until my time on this planet is done.

Thus, I must stick it out a bit longer… I might not get the 15-year bonus (4⅓ weeks), but at least I can’t lose what I have now. If excrement does meet a rotary cooling device though, simulations suggest with some creative accounting, I may just scrape through. I don’t plan on setting up a donations page and talking to Centrelink is a waste of time, I’ll die a pauper before they answer the phone.

Plans for this month

So I have holiday leave off until November. Unlike previous times I’ve taken big amounts off, I won’t be travelling far this time around. Instead, it’s a project month.

Financial work

I need to plan ahead for the possibility that I wind up in long-term unemployment. I don’t expect to live long (the planet cannot sustain everyone living to >100 years), but I do need to be around to finalise the estates of my parents and see my cat out.

That suggests I need to keep the lights on for another 20~30 years. Presently my annual expenditure has been around the $30k mark, but much of that is discretionary (most of it has been on music), and I could possibly reduce that to around the $10k mark.

I have some shares, but need to expand on this further. David Rowe posted some ideas in a two part series which provides some food for thought here.

At the moment, I’m nowhere near that 10% yield figure mentioned…that post was written in 2015 and lot has changed in 8 years. Interest rates are currently at ~5% for term deposits.

I do plan to start one though all the same. After Suncorp closed both The Gap and Ashgrove branches (forcing me all the way to Michelton), I set up an account at BOQ who have branches in both Ashgrove and The Gap… so I can do a term deposit with either, and they’re both offering a 5% 12-month term deposit.

I have a year’s worth sitting at BOQ in an interest bearing account… so that’s money that’s readily accessible. The remainder I have, I plan to split — some going into the aforementioned term deposit, the other will go into that interest bearing account in case I decide to buy more shares.

That should start building the reserves up.

Hardware refurbishment and replacement

Some of my equipment is getting a bit long in the tooth. The old desktop I bought back in 2010 is playing silly-buggers at the moment, and even the laptop I’m typing this on is nearing 10 years old. I have one desktop which used to be my office workstation before the pandemic, so between it and the old desktop, I have decent processing capacity.

The server rack needs work though. One compute node is down, and I’m actually running out of space. I also need to greatly expand the battery bank. I bought a full-height open-frame rack to replace the old one, and was gifted a new solar controller, so some time during this break, I’ll be assembling that, moving the old servers into it… and getting the replacement compute node up and running.

Software updates

I’ve been doing this to critical servers… I recently replaced the mail server with a new VM instance which made the maintenance work-load a lot lower… but there’s still some machines that need my attention.

I’m already working on getting my Mastodon instance up to release 4.2.0 (I bumped it to 4.1.9 to at least get a security patch off my back), there are a couple of OpenBSD routers that need updates and some similar remedial work.

Projects

Already mentioned is the server cluster (hardware and software), but there are some other projects that need my attention.

  • setuptools-pyproject-migration is a project that David Zaslavsky and I have been working on that is intended to help projects migrate from the old setup.py scripts in Python projects to the new pyproject.toml structure. Work has kept me busy, but the project is nearly ready for the first release. I need to help finish up the bits that are missing, and get that out there.
  • aioax25 could use some love, connected mode nearly works, plus it could do with a modernisation.
  • Brisbane WICEN‘s RFID tracking project is something I have not posted much about, but nonetheless got a lot of attention at the Tom Quilty this year, this needs further work.

Self-Training

Some things I’d like to try and get my head around, if possible…

  • Work uses NodeJS for a lot of things, but we’re butting up against its limits a lot. We use a lot of projects that are written in GoLang (e.g. InfluxDB, Grafana, Terraform, Vault), and while I did manage to hack some features into s3sync needed for work, I should get to know GoLang properly.
  • Rust interests me a lot. I should at least have a closer look at this and learn a little. It has been getting a mention around the office in the context of writing NodeJS extensions. Definitely worth looking into further.
  • I need to properly get to understand OAuth2, as I don’t think I completely understand it as it stands now. I’m not sure I’m doing it “right”.
  • COSE would have applications in both the WideSky Hub (end-to-end encryption) and in Brisbane WICEN’s RFID tracking system (digital signatures).

Physical exercise

I have not been out on the bike for some time, and it shows! I need to get out more. I intend to do quite a bit of that over the next few weeks.

Maybe I might do the odd over-nighter, but we’ll see.

Generating ball tickets/programmes using LaTeX

My father does a lot of Scottish Country dancing, he was a treasurer for the Clan MacKenzie association for quite a while, and a president there for about 10 years too. He was given a task for making some ball tickets, but each one being uniquely numbered.

After hearing him swear at LibreOffice for a bit, then at Avery’s label making software, I decided to take matters into my own hands.

First step was to come up with a template. The programs were to be A6-size booklets; made up of A5 pages folded in half. For ease of manufacture, they would be printed two to a page on A4 pages.

The first step was to come up with the template that would serve as the outer and inner pages. The outer page would have a placeholder that we’d substitute.

The outer pages of the programme/ticket booklet… yes there is a typo in the last line of the “back” page.
\documentclass[a5paper,landscape,16pt]{minimal}
\usepackage{multicol}
\setlength{\columnsep}{0cm}
\usepackage[top=1cm, left=0cm, right=0cm, bottom=1cm]{geometry}
\linespread{2}
\begin{document}
\begin{multicols}{2}[]

\vspace*{1cm}

\begin{center}
\begin{em}
We thank you for your company today\linebreak
and helping to celebrate 50 years of friendship\linebreak
fun and learning in the Redlands.
\end{em}
\end{center}

\begin{center}
\begin{em}
May the road rise to greet you,\linebreak
may the wind always be at your back,\linebreak
may the sun shine warm upon your face,\linebreak
the rains fall soft upon your fields\linebreak
and until we meet again,\linebreak
may God gold you in the palm of his hand.
\end{em}
\end{center}

\vspace*{1cm}

\columnbreak
\begin{center}
\begin{em}
\textbf{CLEVELAND SCOTTISH COUNTRY DANCERS\linebreak
50th GOLD n' TARTAN ANNIVERSARY TEA DANCE}\linebreak
\linebreak
1973 - 2023\linebreak
Saturday 20th May 2023\linebreak
1.00pm for 1.30pm - 5pm\linebreak
Redlands Memorial Hall\linebreak
South Street\linebreak
Cleveland\linebreak
\end{em}
\end{center}

\begin{center}
\begin{em}
Live Music by Emma Nixon \& Iain Mckenzie\linebreak
Black Bear Duo
\end{em}
\end{center}

\vspace{1cm}

\begin{center}
\begin{em}
Cost \$25 per person, non-dancer \$15\linebreak
\textbf{Ticket No \${NUM}}
\end{em}
\end{center}
\end{multicols}
\end{document}

The inner pages were the same for all booklets, so we just came up with one file that was used for all. I won’t put the code here, but suffice to say, it was similar to the above.

The inner pages, no placeholders needed here.

So we had two files; ticket-outer.tex and ticket-inner.tex. What next? Well, we needed to make 100 versions of ticket-outer.tex, each with a different number substituted for $NUM, and rendered as PDF. Similarly, we needed the inner pages rendered as a PDF (which we can do just once, since they’re all the same).

#!/bin/bash
NUM_TICKETS=100

set -ex

pdflatex ticket-inner.tex
for n in $( seq 1 ${NUM_TICKETS} ); do
	sed -e 's:\\\${NUM}:'${n}':' \
            < ticket-outer.tex \
            > ticket-outer-${n}.tex
	pdflatex ticket-outer-${n}.tex
done

This gives us a single ticket-outer.pdf, and 100 different ticket-inner-NN.pdf files that look like this:

A ticket outer pages document with substituted placeholder

Now, we just need to put everything together. The final document should have no margins, and should just import the relevant PDF files in-place. So naturally, we just script it; this time stepping every 2 tickets, so we can assemble the A4 PDF document with our A5 tickets: outer pages of the odd-numbered ticket, outer pages of the even-numbered ticket, followed by two copies of the inner pages. Repeat for all tickets. We also need to ensure that initial paragraph lines are not indented, so setting \parindent solves that.

This is the rest of my quick-and-dirty shell script:

cat > tickets.tex <<EOF
\documentclass[a4paper]{minimal}
\usepackage[top=0cm, left=0cm, right=0cm, bottom=0cm]{geometry}
\usepackage{pdfpages}
\setlength{\parindent}{0pt}
\begin{document}
EOF
for n in $( seq 1 2 ${NUM_TICKETS} ); do
	m=$(( ${n} + 1 ))
	cat >> tickets.tex <<EOF
\includegraphics[width=21cm]{ticket-outer-${n}.pdf}
\includegraphics[width=21cm]{ticket-outer-${m}.pdf}
\includegraphics[width=21cm]{ticket-inner.pdf}
\includegraphics[width=21cm]{ticket-inner.pdf}
EOF
done
cat >> tickets.tex <<EOF
\end{document}
EOF
pdflatex tickets.tex

The result is a 100-page PDF, which when printed double-sided, will yield a stack of tickets that are uniquely numbered and serve as programmes.

A crude attempt at memory management

The other day I had a bit of a challenge to deal with. My workplace makes embedded data collection devices which are built around the Texas Instruments CC2538 SoC (internal photos visible here) and run OpenThread. To date, everything we’ve made has been an externally-powered device, running off either DC power (9-30V) or mains (120/240V 50/60Hz AC). CC2592 range extender support was added to OpenThread for this device.

The CC2538, although very light on RAM (32KiB), gets the job done with some constraints. Necessity threw us a curve-ball the other day, we wanted a device that ran off a battery. That meant going into sleep mode periodically, deep sleep! The CC2538 has a number of operating modes:

  1. running mode (pretty much everything turned on)
  2. light sleep mode (clocks, CPU and power stays on, but we pause a few peripherals)
  3. deep sleep mode — this comes in four flavours
    • PM0: Much like light-sleep, but we’ve got the option to pause clocks to more peripherals
    • PM1: PM0, plus we halt the main system clock (32MHz crystal or 16MHz RC), halting the CPU
    • PM2: PM1 plus we power down the bottom 16KiB of RAM and some other internal peripherals
    • PM3: PM2 plus we turn off the 32kHz crystal used by the sleep timer and watchdog.

We wanted PM2, which meant while we could use the bottom 16KiB of RAM during run-time, the moment we went to sleep, we had to forget about whatever was kept in that bottom 16KiB RAM — since without power it would lose its state anyway.

The challenge

Managing RAM in a device like this is always a challenge. malloc() is generally frowned upon, however in some cases it’s a necessary evil. OpenThread internally uses mbedTLS and that, relies on having a heap. It can use one implemented by OpenThread, or one provided by you. Our code also uses malloc for some things, notably short-term tasks like downloading a new configuration file or for buffering serial traffic.

The big challenge is that OpenThread itself uses a little over 9KiB RAM. We have a 4KiB stack. We’ve got under 3KiB left. That’s bare-bones OpenThread. If you want JOINER support, for joining a mesh network, that pulls in DTLS, which by default, will tell OpenThread to static-allocate a 6KiB buffer.

9KiB becomes about 15KiB; plus the stack, that’s 19KiB. This is bigger than 16KiB — the linker gives up.

Using heap memory

There is a work-around that gets things linking; you can build OpenThread with the option OPENTHREAD_CONFIG_HEAP_EXTERNAL_ENABLE — if you set this to 1, OpenThread forgoes its own heap and just uses malloc / free instead, implemented by your toolchain.

OpenThread builds and links in 16KiB RAM, hooray… but then you try joining, and; NoBufs is the response. We’re out of RAM. Moving things to the heap just kicked the can down the road, we still need that 6KiB, but we only have under 3KiB to give it. Not enough.

We have a problem in that, the toolchain we use, is built on newlib, and while it implements malloc / free / realloc; it does so with a primitive called _sbrk(). We define a pointer initialised up the top of our .bss, and whenever malloc needs more memory for the heap, it calls _sbrk(N); we grab the value of our pointer, add N to it, and return the old value. Easy.

Except… we don’t just have one memory pool now, we have two. One of which, we cannot use all the time. OpenThread, via mbedTLS also winds up calling on malloc() very early in the initialisation (as early as the otInstanceInitSingle() call to initialise OpenThread). We need that block of RAM to wind up in the upper 16KiB that stays powered on — so we can’t start at address 0x2000:0000 and just skip over .data/.bss when we run out.

malloc() will also get mighty confused if we suddenly hand it an address that’s lower than the one we handed out previously. We can’t go backwards.

I looked at replacing malloc() with a dual-pool-aware version, but newlib is hard-coded in a few places to use its own malloc() and not a third-party one. picolibc might let us swap it out, but getting that integrated looked like a lot of work.

So we’re stuck with newlib‘s malloc() for better or worse.

The hybrid approach

One option, we can’t control what malloc the newlib functions use. So use newlib‘s malloc with _sbrk() to manage the upper heap. Wrap that malloc with our own creation that we pass to OpenThread: we implement otPlatCAlloc and otPlatFree — which are essentially, calloc and free wrappers.

The strategy is simple; first try the normal calloc, if that returns NULL, then use our own.

Re-purposing an existing allocator

The first rule of software engineering, don’t write code you don’t have to. So naturally I went looking for options.

Page upon page of “No man don’t do it!!!”

jemalloc looked promising at first, it is the FreeBSD malloc(), but that there, lies a problem — it’s a pretty complicated piece of code aimed at x86 computers with megabytes of RAM minimum. It used uint64_ts in a lot of places and seemed like it would have a pretty high overhead on a little CC2538.

I tried avr-libc‘s malloc — it’s far simpler, and actually is a free-list implementation like newlib‘s version, but there is a snag. See, AVR microcontrollers are 8-bit beasts, they don’t care about memory alignment. But the Cortex M3 does! avrlibc_malloc did its job, handed back a pointer, but then I wound up in a HARDFAULT condition because mbedTLS tried to access a 32-bit word that was offset by a few bytes.

A simple memory allocator

The approach I took was a crude one. I would allocate memory in fixed-sized “blocks”. I first ran the OpenThread code under a debugger and set a break-point on malloc to see what sizes it was asking for — mostly blocks around the 128 byte mark, sometimes bigger, sometimes smaller. 64-byte blocks would work pretty well, although for initial testing, I went the lazy route and used 8-byte blocks: uint64_ts.

In my .bss, I made an array of uint8_ts; size equal to the number of 8-byte blocks in the lower heap divided by 4. This would be my usage bitmap — each block was allocated two bits, which I accessed using bit-banding: one bit I called used, and that simply reported the block was being used. The second was called chained, and that indicated that the data stored in this block spilled over to the next block.

To malloc some memory, I’d simply look for a string of free blocks big enough. When it came to freeing memory, I simply started at the block referenced, and cleared bits until I got to a block whose chained bit was already cleared. Because I was using 8-byte blocks, everything was guaranteed to be aligned.

8-byte blocks in 16KiB (2048 blocks) wound up with 512 bytes of usage data. As I say, using 64-byte blocks would be better (only 256 blocks, which fits in 64 bytes), but this was a quick test. The other trick would be to use the very first few blocks to store that bitmap (for 64-byte blocks, we only need to reserve the first block).

The scheme is somewhat inspired by the buddy allocator scheme, but simpler.

Bit banding was simple enough; I defined my struct for accessing the bits:

struct lowheap_usage_t {
        uint32_t used;
        uint32_t chained;
};

and in my code, I used a C macro to do the arithmetic:

#define LOWHEAP_USAGE                                                   \
        ((struct lowheap_usage_t*)(((((uint32_t)&lowheap_usage_bytes)   \
                                     - 0x20000000)                      \
                                    * 32)                               \
                                   + 0x22000000))

The magic numbers here are:

  • 0x20000000: the start of SRAM on the CC2538
  • 0x22000000: the start of the SRAM bit-band region
  • 32: the width of each word in the CC2538

Then, in my malloc, I could simply call…

struct lowheap_usage_t* usage = LOWHEAP_USAGE;

…and treat usage like an array; where element 0 was the usage data for the very first block down the bottom of SRAM.

To implement a memory allocator, I needed five routines:

  • one that scanned through, and told me where the first free block was after a given block number (returning the block number) — static uint16_t lowheap_first_free(uint16_t block)
  • one that, given the start of a run of free blocks, told me how many blocks following it were free — static uint16_t lowheap_chunk_free_length(uint16_t block, uint16_t required)
  • one that, given the start of a run of chained used blocks, told me how many blocks were chained together — static uint16_t lowheap_chunk_used_length(uint16_t block)
  • one that, given a block number and count, would claim that number of blocks starting at the given starting point — static void lowheap_chunk_claim(uint16_t block, uint16_t length)
  • one that, given a starting block, would clear the used bit for that block, and if chained was set; clear it and repeat the step on the following block (and keep going until all blocks were freed) — static void lowheap_chunk_release(uint16_t block)

From here, implementing calloc was simple:

  1. first, try the newlib calloc and see if that succeeded. Return the pointer we’re given if it’s not NULL.
  2. if we’re still looking for memory, round up the memory requirement to the block size.
  3. initialise our starting block number (start_nr) by calling lowheap_first_free(0) to find the first block; then in a loop:
    • find the size of the free block (chunk_len) by calling lowheap_chunk_free_length(start_nr, required_blocks).
    • If the returned size is big enough, break out of the loop.
    • If not big enough, increment start_nr by the return value from lowheap_chunk_used_length(start_nr + chunk_len) to advance it past the too-small free block and the following used chunk.
    • Stop iterating of start_nr is equal to or greater than the total number of blocks in the heap.
  4. If start_nr winds up being past the end of the heap, fail with errno = ENOMEM and return NULL.
  5. Otherwise, we’re safe, call lowheap_chunk_claim(start_nr, required_blocks); to reserve our space, zero out the actual blocks allocated, then return the address of the first block cast to void*.

Implementing free was not a challenge either: either the pointer was above our heap, in which case we simply passed the pointer to newlib‘s free — or if it was in our heap space, we did some arithmetic to figure out which block that address was in, and passed that to lowheap_chunk_release().

I won’t publish the code because I didn’t get it working properly in the end, but I figured I’d put the notes here on how I put it together to re-visit in the future. Maybe the thoughts might inspire someone else. 🙂