Overview

In 2014, my workplace began investigating virtual machine infrastructure to use for both internal projects and for hosting customer sites. Ultimately, they decided to abandon this in favour of using existing cloud providers (originally Vultr, later Amazon AWS), my research was brought home and I built my own system from scratch to investigate the idea further.

Details

Cluster specifications

Many of the nodes follow a “periodic table” naming convention, as it started with Intel Atoms.

• hydrogen, carbon: Storage nodes
  • Motherboard: Supermicro A1SAi-2750F
  • CPU: 8-core Intel Atom C2750 at 2.4GHz
  • RAM: 32GB ECC DDR3
  • Data storage:
    • 2× Samsung 860EVO/870EVO 2TB SSDs
  • Operating system storage:
    • 1× 120GB M.2 SSD (USB enclosure)
  • OS: Debian Linux 12
• boron, helium, nitrogen, oxygen: Storage nodes
  • Various generation Intel NUCs
  • RAM: 8-16GB DDR4
  • Data storage: Samsung 860EVO/870EVO 2TB SSD
  • Operating system storage: Western Digital 120GB M.2 SSD
  • OS: Debian Linux 12
• lithium: Compute node
  • Motherboard: Supermicro A2SDi-16C-HLN4F
  • CPU: 16-core Intel Atom C3955 at 2.4GHz
  • RAM: 32GB ECC DDR4
  • Disk: Western Digital Green 240GB
  • OS: AlpineLinux 3.23
• beryllium, magnesium: Compute nodes
  • Motherboard: Supermicro A1SAi-2750F
  • CPU: 8-core Intel Atom C2750 at 2.4GHz
  • RAM: 32GB ECC DDR3
  • Disk: Western Digital Green 240GB
  • OS: AlpineLinux 3.23
• sodium: Compute node
  • MSI MS-B0A81
  • RAM: 64GB DDR4
  • OS Disk: Western Digital Green 240GB
  • Data Disk: WD Green SN3000 1TB
  • OS: AlpineLinux 3.23
• Virtualisation Stack: KVM through libvirt, managed with virt-manager
• Network fabric: Netgear GS748T
• Solar input: 3×350W 12V panels
• Battery bank: 2×600Ah 12V LiFePO₄
• Monitoring node
  • Technologic Systems TS-7670v2 revision D
  • CPU: Freescale^WNXP i.MX286 at 454MHz ARMv5 (single-core)
  • RAM: 128MB
  • SSD: On-board 2GB eMMC
  • OS: Gentoo/Linux (musl C library)
• NBN Internet router
  • PC Engines APU3
  • CPU: AMD GX-412TC quad-core 1GHz
  • RAM: 4GB DDR3
  • Disk: 256GB mSATA SSD
  • OS: OpenBSD 7.8

What is this cluster doing?

Since late June 2016, it has been in production running a number of websites:

• My blog
• Brisbane Area WICEN Group‘s OwnCloud site and other miscellaneous sites (e.g. their calendar)
• My photo gallery

…among lots of ad-hoc projects.

It also has been used on a few occasions to run test instances at my workplace, in one case, providing about 5 virtual machines to try out Kubernetes, and in another, spinning up a test instance of WideSky, because our usual hosting provider (Vultr) was full (specifically their Sydney data centre).  For that reason, this equipment appears on my tax records.

In March 2018, I officially decommissioned the old Intel Atom D525 server that had been running much of my infrastructure to date, doing a physical-to-virtual migration of the old server onto a VM.  The old box was re-configured to just power on at 9PM so that its cron jobs could do a back-up of the real instances, then shut down.  This machine has since been reloaded, still performs the same function but now the OS is stripped down to the bare essentials.  (Thank-you Gentoo/musl.)

I may yet convert it to run off 12V with the cluster too as the PSU fan is making noises, we’ll see.

Can it run entirely off solar?

In good weather, yes.  If there’s good sunlight during the day.  An extra battery and another panel would help here, and I’m considering doing exactly that.

For now though, it runs both mains and solar, which already has reduced our power bill.

If doing it again, what would I do different?

• The switch: the Linksys can’t do LACP with more than 4 LAGs, whereas the Netgear one can do the required number of LAGs.
• At the time that Supermicro board was one of the best options available, but buying DDR3 ECC SO-DIMMs is a pain.  There are newer ones now that aside from having more cores (up to 16!), take full-size DDR4 ECC DIMMs which are easier to come by.
• Getting ATX DC-DC PSUs that can tolerate up to 16V natively.  (I think mini-box were out-of-stock of the other models, hence I took the ones I have and used LDOs to hack around the problem.)

Project Logs

See the Solar-powered Cloud Computing category.