Since its inception, Domoticz has tried to reconcile many different goals:

• the availability of different plug-ins and hardware
• ideas for user automation
• scripting (LUA and more recently Python) and if, then, else mechanisms
• integration of off-the-shelf third-party solutions
• integration of open-source libraries
• support for various hardware platforms, including SBCs, which often have their own hardware and performance limitations

As a result, a both fresh and advanced user finds himself in a situation where the system – ceases to work at all, or requires frequent restarts, or worse causes quick degradation of the main storage device – which is often (micro) SD card.

Is there a simple and effective solution to increase Domoticz’s availability on Raspberry Pi / Orange Pi?

Sure there is! But first – before you begin – make a copy of your (micro)SD working used in domoticz instance on your Raspberry Pi/Orange Pi.
CAUTION: We will perform operations that may lead to the loss of all data – so make sure you have a copy!

What will you need?

Software:

• monit
• rsync

Hardware:

• harddrive either classic HDD or better SSD, small is OK from 32GB up to 128GB
• heatsinks for Raspberry Pi/Orange Pi
• USB 2.0 or 3.0 to SATA adapter – so we can connect the harddrive via USB port

  Let’s do it!

  The plan is very simple – connect the hard disk, move data and check if we have heat sinks. SSD type disk is recommended due to long life, reliability and low power consumption. Even the older designs are fine or MLC/TLC flash type based- will serve us for a very long time. We will transfer the following resources to the disk:

/root – this is one of the domoticz possible location
/home – old style installation folder for domoticz
/var – fast changing files, databases, possibly influxdb
While switching to a solid disk (SSD) – we cannot neglect the microSD card – avoid any no-name cards, and remember to verify the named ones by registering on the manufacturer’s website. The “endurance” cards can be also used, but usually their price is close to the price of the SSD.
The risk of the damage to the micro SD will be significantly reduced after the switch, especially the unfamous problem of the physical damage to the card’s cells when power cycling the computer
Don’t forget to put the heat sinks on the CPU, memory and controller – RPi 3 gets quite hot!

Hardware – the SSD

The SSD needs to have:

SATA interface
5V only voltage
power (mA), the best if it reaches a 50% of the max. 800mA of the Raspberry Pi3’s USB port, enable max_current=1 for older models or Zero
You will need to purchase a USB-SATA adapter/bridge for 2.5″ SATA disks, to be able to connect a SSD with a SATA interface, with the integrated, USB-sourced power. Adapters with one USB 3.0 connector (blue), compatible with USB 2.0 and based on the JMicron chipset are recommended. I’m using today (3 years) an old one – without trim/ATA support.
Please test the connection of the disk to an already working Raspberry Pi, and during the restart – please note the behavior of the Pi’s red LED – it cannot go off, neither a lightning bolt icon might appear in the case you’re uning monitor via HDMI / CVSB.
Issue the “dmesg” command after connecting new SSD, output should be similar:

[    5.649443] usb 1-1.3: new high-speed USB device number 6 using dwc_otg
[    5.782360] usb 1-1.3: New USB device found, idVendor=05e3, idProduct=0718
[    5.786326] usb 1-1.3: New USB device strings: Mfr=0, Product=1, SerialNumber=2
[    5.790444] usb 1-1.3: Product: USB Storage
[    5.794449] usb 1-1.3: SerialNumber: 000000000033
[    5.806179] usb-storage 1-1.3:1.0: USB Mass Storage device detected
[    5.814256] scsi host0: usb-storage 1-1.3:1.0
[    5.922077] gpiomem-bcm2835 3f200000.gpiomem: Initialised: Registers at 0x3f200000
[    6.362614] EXT4-fs (mmcblk0p2): re-mounted. Opts: (null)
[    6.902236] scsi 0:0:0:0: Direct-Access     INTEL SS DSA2CT040G3      0016 PQ: 0 ANSI: 4
[    6.917156] sd 0:0:0:0: [sda] 78165356 512-byte logical blocks: (40.0 GB/37.3 GiB)
[    6.933290] sd 0:0:0:0: [sda] Write Protect is off
[    6.933314] sd 0:0:0:0: [sda] Mode Sense: 03 00 00 00
[    6.934475] sd 0:0:0:0: [sda] No Caching mode page found
[    6.934491] sd 0:0:0:0: [sda] Assuming drive cache: write through
[    6.941523]  sda: sda1
[    6.944631] sd 0:0:0:0: Attached scsi generic sg0 type 0
[    6.949765] sd 0:0:0:0: [sda] Attached SCSI disk

This gives us the following information: this is the only external drive, system knows it as /dev/sda, first partition is /dev/sda1. Next up is the task of formatting that partition (deletes all data!) with the ext4 file system (side note: B-tree File System or Butter FS is also good choice, but we talk stable here).

First – let’s install new software:

apt update
apt upgrade
apt install rsync monit

Then issue the mkfs command on your Pi (make sure it’s the right Pi, the right partition – as the process deletes all data).

mkfs.ext4 /dev/sda1

Now, lets mount the new drive to check if it works:

mkdir -p /mnt/ssd
mount /dev/sda1 /mnt/ssd

The dmesg should show us:

[    7.744129] EXT4-fs (sda1): mounting with "discard" option, but the device does not support discard
[    7.754021] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: discard,commit=600,errors=remount-ro

Now update the /etc/fstab with the following from the top:

/dev/sda1       /mnt/ssd        ext4    discard,noatime,commit=600,errors=remount-ro    0       1
/mnt/ssd/var    /var/           none    bind    0       1
/mnt/ssd/home   /home           none    bind    0       1
/mnt/ssd/opt    /opt            none    bind    0       1

It’s time to turn off all services – except sshd – so we can tranfer all the files to SSD. It sould be done on other computer with system not working but the risk is rather low.

sudo su -
systemctl stop domoticz.sh
systemctl stop influx.service
systemctl stop grafana.service
systemctl stop cron.service
systemctl stop rsyslog.service
...

Copy the /home /root i /var to SSD.

sudo rsync -avz /home /root /var /mnt/ssd/

Check:

ls -alR /mnt/ssd

This should give us a long list of files.
Now restart your Pi and after booting up – check:

sudo mount | grep sda1

Expected result:

/dev/sda1 on /mnt/ssd type ext4 (rw,noatime,discard,errors=remount-ro,commit=600,data=ordered)
/dev/sda1 on /root type ext4 (rw,noatime,discard,errors=remount-ro,commit=600,data=ordered)
/dev/sda1 on /home type ext4 (rw,noatime,discard,errors=remount-ro,commit=600,data=ordered)
/dev/sda1 on /var/ type ext4 (rw,noatime,discard,errors=remount-ro,commit=600,data=ordered)

Stable software – monit

Domoticz is quite a complex software, often incorporates code of different quality and development dynamics. As a result, adding even an innocent sensor or function – may cause destabilization – even if using official stable version. A good remedy for the “Domoticz offline” syndrome is monit – a piece of software that can be used to monitor domoticz’s operation (of course not only domoticz), restarting it after crash while having a nice web interface.

Since we installed monit in previous step – let’s edit the /etc/monit/monitrc – find this section and modify it – the PASSWORD and “httpd port” value

## Monit has an embedded HTTP interface which can be used to view status of 
## services monitored and manage services from a web interface. The HTTP 
## interface is also required if you want to issue Monit commands from the
## command line, such as 'monit status' or 'monit restart service' The reason
## for this is that the Monit client uses the HTTP interface to send these
## commands to a running Monit daemon. See the Monit Wiki if you want to 
## enable SSL for the web server. 
#
 set httpd port 8888 and
    use address localhost  # only accept connection from localhost
    allow localhost        # allow localhost to connect to the server and
    allow admin:TAJNEHASLO # require user 'admin' with password 'monit'
    use address 0.0.0.0    # only accept connection from localhost (comment to connect from other hosts)
    allow 0.0.0.0/0.0.0.0     
    allow @monit           # allow users of group 'monit' to connect (rw)
    allow @users readonly  # allow users of group 'users' to connect readonly

Add the domoticz section (assuming it’s there using port 8080, and has no username/password for localhost):

check process domoticz with pidfile /var/run/domoticz.pid
  start program = "/etc/init.d/domoticz.sh start"
  stop  program = "/etc/init.d/domoticz.sh stop"
  if failed
     url http://127.0.0.1:8080/json.htm?type=command¶m=getversion
         and content = '"status" : "OK"'
     for 2 cycles
     then restart
  if 5 restarts within 5 cycles then exec "/sbin/reboot"

Start monit:

sudo systemctl enable monit.service
sudo systemctl restart monit.service

Check the monit’s web interface – you should get similar results:

Some final thoughts on the first part: The best way is to have trim support – but most of the adapter/bridges doesn’t pass the ATA/trim command, which means that it’s good to do manual fstrim – at the time of regular micro SD card backup – by means of connecting the SSD to other computer.

Don’t forget this is not 100% proof solution, it’s a step to extend life and make the system more stable. The next part will add the ongoing and needed backup to external filesystem (NFS), and proper installation of external power (UPS).