Computing

Solar cluster: Charge controllers

Well, having gotten the output of the battery sorted out, now it’s time to turn my attention to the input side, namely managing the battery voltage and two possible charge sources.

Now, I have a second-hand Xantrex 20A charger kicking around that I plan to use for when the sun isn’t around and my battery is getting low. When the sun’s out though, I plan to let that charge the battery. I could do this with a small MCU, and I did briefly consider whether I used an ATTiny24A to do it, or one of my spare ATMega8Ls.

I have a beefy 30A relay that can connect and disconnect the charger as needed, it’s a matter of having a controller that decides when it’s needed. I’m not looking for PWM control, the charger will do that itself.

There are two thresholds I want to consider:

  • The low threshold: about 11.5V or so.
  • The high threshold: about 14V.

We want to not let the battery get much below 11.5V as the regulators on the compute nodes will drop up to 700mV and the IPMI BMCs will start to get grumpy. Likewise, they complain when they see more than 13.5V. The regulators should look after it, but let’s not stress them too hard.

I could use a single comparator with hysteresis to do the above, by selecting a reference voltage mid-way between 11.5 and 14V, and setting resistors to set the threshold gap. I’ve decided to just use two comparators, so I can use a LM393, or I have a LM339 kicking around. I also dug around in the junk box and found a stack of MM74C76s, some MM74C221Ns.

Some tinkering with a breadboard, and I came up with this:

Now, the beauty of this set-up, is that I’m using half of each IC, so I effectively have two independent controllers on the one board. Thresholds can be tweaked on each one so that one charger starts sooner than the other, maybe I kick the solar in when battery drops below 12V and let it go to 14V, the mains charger kicks in when we get to 11.5V and stops when we reach 13V.

I haven’t decided on a regulator, yes I could use a LM78C05, the low-power version of the LM7805, as the power drain of this is going to be tiny and headroom enormous for 5V. There are probably better options, I’ll have to shop around, although for a quick prototype, I might just use the LM78C05s since they’re on hand.

Solar cluster: 12V regulator installation

Well, I finally got busy with the soldering iron again. This time, installing the regulators in the cluster nodes and in the 26-port switch.

I had a puzzle as to where to put the regulator, I didn’t want it exposed, as they’re a static-sensitive device, so better to keep them enclosed. It needed somewhere where the air would be flowing, and looking around, I found the perfect spot, just in behind the CPU heatsink. There’s a small gap where the air will be flowing past to cool the CPU, and it’s sufficiently near the ATX PSU to feed the power cabling past.

I found I was able to tap M3 threads into the tops of the heatsinks and fix them to the “front” of the case near where the DIN rail brackets fit in. So from the outside, it looks all neat and tidy.

After installing those, I turned my attention to the switch. Now I had an educated guess that the switch would be stepping down from 12V, so being close to that was not so critical, however going above it would stretch the friendship.

Rather than feeding it 13.1V like the compute nodes, I decided I’d find some alternate resistor values that’d be closer to 12V. Those wound up being R1=3.3kΩ and R2=390Ω, which gave about 11.8V. Close enough. It was then a matter of polarity. The wiring inside this switch uses a non-standard colour code, and as I suspected, the conductors are just paralleled, it’s the one feed of 12V.

Probing with a multimeter revealed the pin pairs were shorted, and removing the PSU confirmed this. I pulled out the switch mainboard and probed around the electrolytics which had their negative sides marked. Sure enough, it’s the Australian Olympic team colours that give away the 0V side.

I’ve shown the original colour code here as coloured dots, but essentially, green and yellow are the 0V side, and red and black are the +12V side. So I had everything necessary. I grabbed a bit of scrap PCB, used the old PSU as a template for drilling out the holes, used a hacksaw to divide the PCB surface up then dead-bugged the rest. To position the heatsink, I drilled a 3mm hole in the bottom of the case and screwed a 10mm M3 stand-off there. Yes, this means there’s an annoying lump on the bottom, I should use a countersunk M3 screw, I’ll fix that later if it bothers me, I’ll be rack-mounting it anyway.

On the input to the regulator, I have a 330uF electrolytic capacitor and 100nF monolithic capacitor in parallel, on the output, it’s a 470uF and a 100nF. A third 100nF hooks the adjust pin to 0V to reduce noise. I de-soldered the original PSUs socket and used that on the new board. It fits beautiful. 100-240V? Not any more Linksys.

So now, the whole lot runs off a single 12V battery supply. The remainder of this project is the charging of that battery and the software configuration of the cluster.

At present, the whole cluster’s doing an `emerge @system`, with distcc running, and drawing about 7.5A with the battery sitting at 12.74V (~95W). Edit: Now that they’ve properly fired up, I’m seeing a drain of 10.3A (126W). Looks that’s going to be the “worst case scenario”.

Solar cluster: Trying out the MIC29712s

I figured, rather than letting these loose directly on the nodes themselves, I’d give them a try with a throw-away dummy load. For this, I grabbed an old Philips car cassette player that’s probably older than I am and hooked that up. I shoved some old cassette in.

The datasheet for the regulators defines the output voltage as: V_{OUT}=1.240 \big({R_1 \over R_2} + 1\big)

Playing with some numbers, I came out with R1 being a 2.7kΩ and 560Ω resistors in series, and R2 being a 330Ω. So I scratched around for those resistors, grabbed one of the MIC29172s and hooked it all up for a test.

The battery here is not the one I’ll use in the cluster ultimately, I have a 100Ah AGM cell battery for that. The charger seen there is what will be used, initially as a sole power source, then in combination with solar when I get the panels. It’s capable of producing 20A of current, and runs off mains power.

This is the power drain from the battery, with the charger turned on.

Not as much as I thought it’d be, but still a moderate amount.

This is what the output side of the regulator looked like:

So from 14.8V down to 13.1V. It also showed 13.1V when I had the charger unplugged, so it’s doing its job pretty well I think. That’s a drop of 1.7V, so dissipating about 600mW. Efficiency is therefore about 93%, not bad for linear regulators.

Solar cluster: Measuring for a cabinet

One elephant in the room, is how I’m going to store the system whilst in operation.

The obvious solution is some sort of metal cabinet with provision for 19″ rack mounting and DIN rail equipment. Question is, how big?

A big consideration here is thermal matters. When going flat out, there will be 100W-150W worth of thermal energy being dissipated in there. So room for convection currents is a must!

Some decent fans on the top to suck the hot air out would also be a good idea. Blowing up so that dust doesn’t get sucked down into the works.

I figured I’d sit everything sort-of in situ. I figured out that the DIN rail mounts don’t have to go on the bottom, with these cases, if you remove the front panel there’s four holes for mounting those same DIN rail mounts on the front. So that’s what I’ve done. I’ve now got a DIN rail spare for future expansion.

If I try to pack everything up as densely as possible (not wise), this is what it looks like:

There’s room there for possibly one more node to squeeze in there. I’d think that’d be pushing it however. 5 is probably a good number, meaning we can space the units out a bit to allow them to draw air in via the gaps.

On top of the units I have my two switches. The old Netcomm 24-port switch was retired from our network when a lightning strike to a neighbour’s tree an 8-port switch, my Yaesu FT-897D radio transceiver, some ports on a wireless 3G router/switch, and an ADSL router out. It also did damage some ports on the big Netcomm switch, so in short, I know it has issues.

Replacing its 3.3V PSU with one that steps down from 12V would cost me the price of a 16-port 10/100Mbps switch brand new.

When we replaced the switch (paid for by insurance) we decided to buy a 8-port and 16-port switch. The 16-port switch, retired due to an upgrade to gigabit, is sitting on top, and takes 12V 1A input. It’ll be perfect for the IPMI VLAN, where speed is not important. It also accepts the DC plugs I bought by mistake.

The 8-port one takes 7.5V 1A, so a little less convenient for this task, I’d need to make a DC-DC converter for it. Maybe later if this works.

So considering a cabinet for this, we have:

  • 5 nodes measuring 190mm in height: ~5 RU
  • A 24-port switch: 1 RU
  • A 16-port switch: 1 RU
  • Some power distribution electronics: 3RU

Yes, the battery and its charger is external to the cabinet.

Judging from this, the cabinet probably needs to be a 10RU or 12RU cabinet to give us space for mounting everything cleanly and to ensure good ventilation. Using 8-port IPMI switches and 24+2-port comms switches, that leaves us with sufficient port space for the 5 nodes and gives us one port left for a small in-chassis monitoring device and 4 ports left on the main switch for an uplink trunk.

You could conceptually then consider these as homogeneous building blocks for larger networks, using Ceph’s CRUSH maps to ensure copies get distributed amongst these “cabinets”.

Solar cluster: Alternate solution to the PicoPSU

So, I’ve been doing a bit of research about how I can stabilise the battery voltage which will drift between around 11V and 14.6V. It’s a deep-cycle type battery, so it’s actually capable of going down to 10V, but I really don’t want to push it that far.

Once I get below 12V, that’s the time to signal to the VM hosts to start hibernating the VMs and preparing for a blackout, until such time as the voltage picks back up again.

The rise above 13.5V is a challenge due to the PicoPSU limitations. @Vlad Conut rightly pointed out that the M3-ATX-HV PSUs sold by the same company would have been a better choice. For about $20 more, or an extra $100 for the cluster, I’d have something that’d work 6-30V. I’d still have to solve the problem with the switch, but it’d just be that one device, not 6.

Maybe it was because they were out of stock that I went the PicoPSU route, I also wasn’t sure about power demands, I knew the CPU needed 20W, but wasn’t sure about everything else. So I over-dimensioned everything. Hindsight is 20:20.

One option I considered was a regulator in front of each node. I had mentioned the LM7812 because I knew of it. I also knew it was a crap choice for this task, the 1.5V drop, with a 5A load would result in about 7.5W dissipated thermally. So 20W would jump to nearly 28W — not good.

That of course assumes a 7812 would handle 5A, which it won’t.

LDOs were the way to go for anything linear, otherwise I was looking at a switchmode power supply. The LM2576 has similar requirements to the LM7812, but is much more efficient being a buck converter. If 1.5V was fine, I’d be looking for a 5A-capable equivalent.

The other option would be to have one single power supply regulate power for all nodes. I mentioned in my previous log about the Redarc DC-DC power supply, and that is certainly still worthy of consideration. It is a single point of failure however, but then again, Redarc aren’t known for making crap, and the unit has a 2 year warranty anyway. I’d have downtime, but would not lose data if this went down.

@K.C. Lee pointed me to one LDO that would be a good candidate though, and is cheap enough to be worth the experiment: the Micrel MIC29750. 7.5A current, and comes in an adjustable form up to 16V. I’d imagine if I set this near 13.5V, it’d dissipate maybe 2.5W max at 5A, or 1W at 2A. Much better.

Not as good as Redarc’s solution of course, and that’s still an option, but cheap enough to try out.

Solar cluster: An off-the-shelf workaround for the PicoPSU

Well, I’ve been researching the problem. I have a battery that could be floating anywhere from 10V to 14.6V, depending on the input from the charger.

I have a computer PSU, that is not happy with voltages outside of 10.5V—13.5V.

What are my options?

  • Linear regulator: the standard ones have a 1.5V drop across them, which at the full rated input current of the PSU, 8A is 12W. Per node.
  • Low drop-out regulators get a little lower, but I’d still lose a few watts per node.
  • Buck converters can do better, but a lot still need at least 1V difference.

So I really need to boost it first, then regulate down. One thing I was not looking forward to, was designing then winding the transformer/inductor needed. An off-the-shelf solution therefore seems attractive, even if I miss out on kudos points for a DIY solution.

Redarc make a couple, and this unit looks like it’ll do exactly what is needed. Not cheap, but it seems comparable to what I’ve seen elsewhere. I’ll have a look and see what else there is, but this might be the most time-economical way to solve the problem and the efficiency is pretty good.

Solar cluster: PicoPSU limitations

Of course, there’s always something there to throw a spanner in the works, and for me, it’s the PicoPSU.

It seems to work great, however, there is an Achilles heel with these things: they have a fairly narrow band of tolerable voltages they’ll operate at. Namely 10.5—13.5V.

Now, 10.5V is fair enough, but 13.5V? Typical lead acid batteries are 13.8V nominal voltage, and will get to 14.5V when charging. So I need some preregulator that will handle when the voltage is up around 13.5V or above, and drop it down just a little, passing through up to 2A (5A to be safe).

It still has to be stable when the current changes, “turned off” on these computers means a drain of about 200mA for the IPMI. So our operating parameters are summed up as 10.5—13.5V and 200mA—5A.

It needs to continue operating when the battery gets to ~11.5V.

So what are my options?

  • LM2576 simple switcher? 12V in will produce 10.5V out.
  • LM7812 has the same problem, and will chew more power.

A series regulator built on a zener/NPN might work. The voltage drop across the NPN ordinarily is going to be fairly low, however there will still be a drop of about 0.7V or so. That’s possibly “good enough”, since at 11.5V input, we should still see about 10.8V out which is within range.

Two diodes in series, with a relay to short them out when the voltage drops below 12V would work too. That’d need a comparator and voltage reference to drive the relay. It’s a cheap solution too.

Another prospect is a beefy DC-DC converter on the battery, so we don’t actually care what the battery voltage is, we boost it say to 15V then regulate it back down to 12V. A 30A-capable flyback or boost-buck converter would do it. This is more complex, and much more expensive to do off-the-shelf, so I think that’d be a method of last resort.

Solar cluster: Handling the solar input

Seems one of the past projects, the #ARDUINO MPPT SOLAR CHARGE CONTROLLER might be worth taking a close look at.

That could sit between my relay-based circuit and the solar panel to up the efficiency a little bit. Previously, I’ve used a LM2576 buck converter and that has done okay, but needs constant tweaking. This alternative should be a bit more automatic in its operation.

Solar cluster: Charge controller ponderings

I’ve been giving some thought to how to manage charging of the battery.

There’ll be two charge sources essentially, one will be mains power via a conventional battery charger, the other will be solar. Both are current-limited and will be below 24 volts.

The battery will not respond straight away to a step change in applied voltage, rather current will start flowing in and the battery voltage will begin to rise. There’ll be a delay in the voltage being applied and the observed battery voltage reaching that point. The only thing we really need to watch is that the current doesn’t exceed the inrush capabilities of the battery.

What we want to avoid is that, as we get close to charge, we do not “chatter” on the upper set-point. When the upper set point is reached, we should back off some minimum delay, before continuing to charge.

This is a possible implementation of that idea.

We consider 3 voltages:

Variable Notes
V_{CL} “Low” control voltage set-point. Charger should turn on near this point. Set by a potentiometer on a regulated supply.
V_{CH} “High” control voltage set-point. Voltages above this point are considered harmful and we should cut power when this voltage is reached. Set by a potentiometer on a regulated supply.
V_{CB} Battery control voltage. This is proportional to the battery voltage, set by a resistor divider or potentiometer across the battery.

We have two comparators that tell us when the battery is below the high set-point, telling us we’re “safe” to begin charging, and when the battery is below the low set-point, telling us the battery is “low” and needs charging. We use an SR latch to achieve this. The “safe” is inverted, and helps drive the “reset” side of the SR latch, the “low” signal drives the “set” side.

If the voltage gets above our high set point at any time, we must stop charging immediately. So we use an AND-gate to ensure that the power is shut off as soon as possible. We feed this into the SR latch reset pin via a NOT gate to the SR latch to reset the state.

If the voltage gets low, depending on our set-point and rate of discharge, we can tolerate some small delay in getting going. We ensure it is safe to do so by ANDing “LOW” with “SAFE”, then delay the rising edge pulse by a few seconds, possibly with an RC circuit or 555 timer, before passing that into the SR-latch.

The AND logic also prevents us from asserting “SET” and “RESET” on the SR latch simultaneously.

The graph I’ve drawn is a bit of an exaggeration, hopefully the discharge curves won’t be that steep. It’s possible I could do away with an op-amp and use hysteresis, however I feel using two means I can control the two set-points independently.

Two of these, and tweak the delays a bit so that the solar comes on first, and we should be able to run both in parallel to charge the same battery.

Solar cluster: Modding the switch for 12V operation

Probably going to be easier than expected. I popped open the cover to see whether it was like the much older Netcomm switch we have.

Sure enough, Linksys do it the same way in the LGS326AU:

This PSU is an open-frame PSU made by Asian Power Devices, Inc. Model NW-20A12-BAAB. Not sure if there’s someone here who knows more about this particular PSU.

It appears it’s the one 12V feed split in two. Red/Black are negative, green/yellow are positive. I’ll double check this. I think with a nice big inductor/capacitor as an in-line filter, this can be hooked straight up to the 12V line, perhaps with a small amount of zener overvoltage protection. It appears that the voltage is further rectified on the switch mainboard: