Sungrow SG5K-D and HomeAssistant

In December 2021 we decided to install Solar Panels on our house (thanks to Quest Energy in Perth for doing a great job), choosing JA Solar Solar Panels and a Sungrow SG5K-D inverter. Unfortunately there isn't an official Sungrow Inverter integration for HomeAssistant, but it is possible to feed solar data to HomeAssistant by sticking together a few components.

A high-level diagram showing components and data flows is given below:

 

1: SolarIoT connects periodically to Sungrow Inverter using ModBus over TCP and periodically reads the inverter's read registers and holding registers.

2: SolarIoT sends data to pvoutput.org.

3: SolarIoT sends data to a local MQTT broker on a specified topic.

4: HomeAssistant MQTT integration subscribes to the specified topic and makes the Solar Inverter data available via HomeAssistant Sensors.

This setup lets me use HomeAssistant to track my Solar production and energy consumption over the day:

Details of how to configure all the components are below:

Sungrow Inverter

I have a SG5K-D with a wifi module that sends data to iSolarCloud (and Solar Analytics) periodically. It's also possible to connect in to the wifi module and pull down data using ModBus over TCP.

SolarIoT

SolarIoT connects to a ModBus capable inverter and reads registers and then sends the data to other systems. This works well for me as I want to send my inverter data to pvoutput.org, but also want to publish the data to other local systems that will use the solar inverter data. To do the latter I've configured my instance of SolarIoT to publish the data to a local MQTT broker.

I had problems with SolarIoT with my Sungrow SG5K-D where connections would succeed most of the time, but once in a while the inverter would stop responding to ModBus handshakes. I found that I could work around this by doing a TCP connect to the inverter on the ModBus power, waiting a couple of seconds and then disconnecting and I would need to do this whenever the inverter ModBus comms went into a stuck state. I've forked the SolarIoT source here and added a feature to automate the resetting dummy connection when SolarIoT has issues communicating with the inverter (add --sungrow-reset to the command line options to use it).

I also expanded the ModBus map for the SG5K-D so it includes the "daily export energy" register which is needed for HomeAssistant's energy dashboard.

There's also a fix for pvoutput.org with the SG5K-D: the "power meter" value for the SG5K-D measures house power consumption, not the self-consumption of PV energy so this value must not be sent otherwise the data in pvoutput.org will be incorrect.

I'm running SolarIoT on one of my Linux VMs. I'll deploy it into a container eventually, I haven't gotten around to setting it up yet :)

MQTT Broker

I run a MQTT broker (Mosquitto) to manage publishing and subscribing data from various IoT sensors (e.g. Sungrow inverter, Xiaomi / Aqara Zigbee sensors, TP Link Kasa smart switches with power monitoring, OpenSprinkler and a custom battery voltage sensor I built for my powered home sliding gate). SolarIoT publishes payloads to a topic "inverter/stats".

HomeAssistant

HomeAssistant has an MQTT integration but in this case manual configuration in the config files are needed because we need to specify the device class, state class and unit of measurement fields so the energy dashboard can use the sensors. My configuration for this is below (from sensors.yaml): 

- platform: mqtt

  state_topic: "inverter/stats"

  name: "Inverter - Daily Power Yield"

  value_template: '{{ value_json["daily_power_yield"] }}'

  unit_of_measurement: Wh

  device_class: energy

  state_class: total_increasing

- platform: mqtt

  state_topic: "inverter/stats"

  name: "Inverter - Daily Export Energy"

  value_template: '{{ value_json["daily_export_energy"] }}'

  unit_of_measurement: kWh

  device_class: energy

  state_class: total_increasing

- platform: mqtt

  state_topic: "inverter/stats"

  name: "Inverter - Daily Purchased Energy"

  value_template: '{{ value_json["daily_purchased_energy"] }}'

  unit_of_measurement: kWh

  device_class: energy

  state_class: total_increasing

Just a reminder that my configuration.yaml has the following in it to read sensor config from sensors.yaml:

# manually configured sensors

sensor: !include sensors.yaml

And you also need to make sure the recorder integration has been set up to save sensor data, for example mine is below:

recorder:

  purge_keep_days: 28

  db_url: !secret URL_MariaDB

Nb. I don't have any sensors listed in my recorder section as I'm using the implicit behaviour to record all sensors.

Once the MQTT sensors have been configured and their entities are showing values from the inverter, the Energy dashboard can be set up with the MQTT sensors that we set up previously:

Once this is all set up, you'll be able to use HomeAssistant's energy dashboard (it will look like this once it has collected enough data to show a whole day's energy production and consumption):

Linux software mirrored raid - swap out drives

I have a home server running CentOS that acts as a VM host and file server. It has a simple raid 1 (mirrored) configuration for the file system which holds some VM images and other important data. It's pretty handy to have this at home as I can just spin up a VM (Windows or Linux) and try out things without having to affect my desktop machine.

The data on the RAID sans the VMs is also backed up to CrashPlan's cloud. The RAID is not my data backup :)

A couple of weeks ago I noticed that the hard drives in the raid had been powered on for > 27000 hours (more than 3 years of runtime). These hard drives are just consumer level SATA drives, so it was time to swap them out for some newer ones. Here's the process that I followed:

• Buy a new 3TB WD Red drive
• Plug it into the home server
• Apply high LCC fix
• Partition the new drive
• Add the new drive to the RAID set
• Resync to the new drive
• Buy a second new 3TB WD Red drive
• Remove both old drives from the RAID set
• Plug the second new drive into the home server, remove the two old drives
• Apply high LCC fix again
• Partition the second new drive
• Add the second new drive to the RAID set
• Grow the RAID set array size to the full size of the new disks
• Grow the filesystem to the full size of the new RAID set
• Wipe data on both old drives

Buy a new 3TB WD Red drive

I chose to use WD Reds because they're designed for a NAS environment (not too different to my usage of the drives) - the drives are meant to be switched on all the time. They also have a higher MTBF.

Plug it into the home server

I plugged the drive into one of the 6G SATA ports on my server's motherboard and booted up the home server.

Apply high LCC fix
There's a known issue with some models of WD Reds where their load cycle count increases because the drive spins down too frequently. WD has a fix for this available here. This fix needs to be applied while the disk is not in use, so I decided to do it before partitioning. I downloaded the 64 bit linux build (wd5741x64) and applied it to the drive. The tool reported that no firmware upgrade was necessary.

Partition the drive

Booting into Linux, the drive wasn't initialised so I used parted (available from the EPEL repo) to write a GPT signature to the drive, create a partition and allocate all space to it, and then mark it as a raid partition:

# parted -a optimal /dev/sda
GNU Parted 2.1
Using /dev/sda
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]?
Start? 2048s
End? -1 
(parted) set 1 raid on

Run an alignment check to make sure the partition starts on a physical sector boundary:

(parted) align-check
alignment type(min/opt)  optimal/[minimal]? optimal
Partition number? 1
(parted) 

It seems a bit strange that the align-check command doesn't return anything if the alignment check succeeded, but it looks like that's intended behaviour - I tried creating a partition on a non-aligned sector boundary later on and it returned a warning message.

Now check that everything looks okay:

(parted) print
Model: ATA WDC WD30EFRX-68E (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  3001GB  3001GB                     raid

(parted) quit

Add the drive to the RAID set
My raid set is /dev/md0 so I ran:

mdadm --add /dev/md0 /dev/sda1

Mdadm adds the drive as a hot spare, so now you need to tell it to use the new drive as an active drive:

mdadm --grow --raid-devices=3 /dev/md0

Resync to the new drive
This took approximately 5 hours. You can do a cat /proc/mdstat to see progress (the text below is just an example):

[root@nyamo crypt]# cat /proc/mdstat
Personalities : [raid1]
md0 : active raid1 sdb1[2] sda1[3]
      2930133512 blocks super 1.2 [2/2] [UU]
      [================>....]  resync = 84.7% (2482257528/2930133512) finish=138.7min speed=53785K/sec

unused devices: <none>

Buy a new 3TB WD Red drive
A week later I went to the local computer store and bought a second drive - the idea is that you get the same model of drive, but from a different batch. This decreases the likelihood of both drives simultaneously failing since the drives will be sufficiently different.

Remove both old drives from the RAID set
I reduced the raid size back to 2 and took the old drive which was no longer an active device out of the RAID set:

# mdadm --grow /dev/md0 --raid-devices=2
# mdadm --remove /dev/md0 /dev/sdb1

Plug the second new drive into the home server, remove the two old drives
I powered off my home server, removed the two old hard drives and plugged into the second new WD Red into another 6G SATA port.

Apply high LCC fix again
After booting up the home server again,  I repeated the same steps as outlined previously, and the tool reported that no firmware upgrade was necessary.

Partition the second new drive
Same as before, but on a different drive:

# parted -a optimal /dev/sdb
GNU Parted 2.1
Using /dev/sdb
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) mklabel gpt
(parted) mkpart
Partition type?  primary/extended? primary
File system type?  [ext2]?
Start? 2048s
End? -1 
(parted) set 1 raid on

(parted) align-check
alignment type(min/opt)  optimal/[minimal]? optimal
Partition number? 1
(parted) print
Model: ATA WDC WD30EFRX-68E (scsi)
Disk /dev/sda: 3001GB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  3001GB  3001GB                     raid
(parted) quit

Add the second new drive to the RAID set.
The RAID set has started up and reports that there's one active device and the RAID is degraded. Adding the new drive to the RAID set triggers a resync:

# mdadm --add /dev/md0 /dev/sdb1

Which then takes hours to complete. Doing a cat /proc/mdstat shows progress:

# cat /proc/mdstat

Personalities : [raid1]
md0 : active raid1 sdb1[2] sda1[3]
      2930133512 blocks super 1.2 [2/2] [UU]
      [=================>...]  resync = 85.4% (2502889848/2930133512) finish=148.3min speed=47998K/sec

unused devices: <none>

Grow the RAID set array size to the full size of the new disks.
My old drives were 2TB drives and the new ones are 3TB, so to use all the available space the RAID set needs to be grown (I ran this using screen and detached as it takes some time):

# mdadm --grow /dev/md0 --size=max

Grow the filesystem to the full size of the new RAID set
Now the file system needs to be grown to use all of the available space in the RAID set (again using screen and it takes longer than the previous command):

# resize2fs /dev/md0

Wipe data on both old drives
This is on my todo list :)

Files from the future and make

Ran into a problem today because I set the clock wrongly on a linux machine, built something using make, then realised that the clock was wrong. Now after setting the clock correctly, make starts throwing errors all over the place saying Clock Skew Detected: your build may be incomplete.

If you're sure that it's because of your mistake with setting the clock incorrectly, you can fix this by "touch"-ing all the files which are future dated. To do this we create a dummy file with the current date, ask find to show all files newer than this dummy file, and touch them:

touch dummy
find . -newer dummy | xargs touch
rm dummy