Backing up users’ home directories in a Linux installation that uses systemd

This is to explain how I configured Lubuntu 17.10 on my family’s PC (single seat; multiple user accounts) to backup all users’ files to a permanently-connected external USB HDD. I wanted a basic solution that would backup automatically every user’s home directory at shutdown but not when rebooting, and would do this whichever user happens to be using the PC. We have no need to archive older versions of the same file, so overwriting a file on the external USB HDD with a newer version of that file from the PC’s HDD is fine. However, we do not want files on the external USB HDD to be deleted if the corresponding files have been deleted from the PC’s HDD. Additionally, I want a time-stamped record of the backup process to be logged to a file in my home directory so I can check periodically what has been happening. As explained in my previous post, two files are used to achieve all these things:

• a Bash script to a) determine whether the system is being rebooted, b) remount the external USB HDD on a suitable mount point, c) copy the files to the external USB HDD, and d) write to the log file;
• a systemd unit file to launch the Bash script.

The two files are described below.

By the way, I also tried using systemd-inhibit in this case, and it did not work. Looking at the manual page for systemd-inhibit, it could be used to execute a program (the example given is ‘systemd-inhibit wodim foobar.iso‘), however I found that systemd-inhibit did not prevent systemd from shutting down the machine before the backup Bash script had run to completion.

Bash script

I created a file /usr/local/sbin/backup_home_directories.sh owned by the root user, with the following permissions and contents:

fitzcarraldo@aspirexc600:~$ ls -la /usr/local/sbin/backup_home_directories.sh
-rwxr-xr-x 1 root root 2570 Jan  8 20:37 /usr/local/sbin/backup_home_directories.sh

#!/bin/bash

# This script backs up to an external USB HDD (NTFS) labelled "FREECOM HDD" the contents of the home directories
# of the users of this Lubuntu 17.10 installation if the system is shutting down but not rebooting.
# It is launched by a systemd service /etc/systemd/system/backup-to-usb-hdd.service.

# Find out if the system is rebooting (as opposed to being shut down):
REBOOT=$( systemctl list-jobs | egrep -q 'reboot.target.*start' && echo "rebooting" || echo "not_rebooting" )
if [ $REBOOT = "not_rebooting" ]; then
# Only execute the following steps if the system is shutting down but not rebooting:
    # Clean up if the backup did not complete last time:
    umount /media/usbhdd 2>/dev/null # Make sure you enter this line correctly.
    rm -rf /media/usbhdd/* # Make sure you enter this line correctly.
    # Unmount the external USB HDD if mounted by udisks2 with the logged-in username in the path:
    umount /media/*/FREECOM\ HDD 2>/dev/null
    # Find out the USB HDD device:
    DEVICE=$( blkid | grep "FREECOM\ HDD" | cut -d ":" -f1 )
    # Create a suitable mount point if it does not already exist, and mount the device on it: 
    mkdir /media/usbhdd 2>/dev/null
    mount -t ntfs-3g $DEVICE /media/usbhdd 2>/dev/null
    # Create the backup directory on the USB HDD if it does not already exist:
    mkdir /media/usbhdd/Lubuntu_home_folders_backup 2>/dev/null
    # Backup recursively all the home directories of all the users, and add a time-stamped summary to the log file: 
    echo "********** Backing up Acer Aspire XC600 users' home directories **********" >> /home/fitzcarraldo/backup.log
    date >> /home/fitzcarraldo/backup.log
    # Log username of user shutting down the PC (may not be this user if Switch User was used):
    echo -ne "User who shutdown PC (may not be this user if Switch User has been used): " >> /home/fitzcarraldo/backup.log
    last | cut -d " " -f1 | head -1 >> /home/fitzcarraldo/backup.log
    sleep 2s
    # To backup the directories and files I prefer to use the following cp command rather than rsync:
    cp --recursive --update --preserve=all --no-dereference --force /home/ /media/usbhdd/Lubuntu_home_folders_backup 2>> /home/fitzcarraldo/backup.log
    echo "Copying completed" >> /home/fitzcarraldo/backup.log
    date >> /home/fitzcarraldo/backup.log
    echo "********** Backup completed **********" >> /home/fitzcarraldo/backup.log
    cp /home/fitzcarraldo/backup.log /media/usbhdd/Lubuntu_home_folders_backup/home/fitzcarraldo/
    # Unmount the USB HDD so that udisks2 can subsequently re-mount it with the user's username in the path:  
    umount /media/usbhdd
fi
exit 0

The Bash script unmounts the external USB HDD and mounts it to /media/usbhdd before performing a backup. This is done because the Udisks daemon (udisksd) automounts the HDD at login to a mountpoint /media/<username>/FREECOM HDD  (e.g. ‘/media/fitzcarraldo/FREECOM HDD‘ when I log in, ‘/media/claudia/FREECOM HDD‘ when user claudia logs in, and so on).

systemd unit file

I created a file /etc/systemd/system/backup-to-usb-hdd.service owned by the root user, with the following permissions and contents:

fitzcarraldo@aspirexc600:~$ ls -la /etc/systemd/system/backup-to-usb-hdd.service 
-rw-r--r-- 1 root root 274 Jan  8 17:51 /etc/systemd/system/backup-to-usb-hdd.service

[Unit]
Description=Backup home directories of all users to USB HDD
DefaultDependencies=no
Before=shutdown.target halt.target
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/backup_home_directories.sh

[Install]
WantedBy=halt.target shutdown.target

Then I enabled the unit file as follows:

fitzcarraldo@aspirexc600:~$ sudo systemctl enable backup-to-usb-hdd.service
Created symlink /etc/systemd/system/halt.target.wants/backup-to-usb-hdd.service → /etc/systemd/system/backup-to-usb-hdd.service.
Created symlink /etc/systemd/system/shutdown.target.wants/backup-to-usb-hdd.service → /etc/systemd/system/backup-to-usb-hdd.service.
fitzcarraldo@aspirexc600:~$

Reboot (or shutdown and restart) to launch it.

The log file

The log file /home/fitzcarraldo/backup.log looked like the following after shutting down a couple of times:

********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********

Whenever the system is rebooted by any user, nothing is appended to the log file. As desired, entries are only appended to the log file when the system is shutdown by any user.

Manually initiated backups

An added benefit is that any user in the sudo group can run the Bash script from the command line at any time to backup without having to shutdown the PC, as illustrated below:

claudia@aspirexc600:~$ cat /home/fitzcarraldo/backup.log 
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********
claudia@aspirexc600:~$ date && sudo backup_home_directories.sh
Tue  9 Jan 01:57:50 GMT 2018
[sudo] password for claudia:
claudia@aspirexc600:~$ cat /home/fitzcarraldo/backup.log 
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Tue  9 Jan 01:57:54 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): claudia
Copying completed
Tue  9 Jan 01:58:10 GMT 2018
********** Backup completed **********
claudia@aspirexc600:~$

Running a shell script at shutdown only (not at reboot) – a comparison between OpenRC and systemd

Gentoo Linux on my laptops uses OpenRC with SysVinit, whereas Lubuntu 17.10 on my family’s PC uses systemd. I have had to configure both Linux distributions to run a backup job at shutdown, so I thought it would be interesting to summarise the two approaches.

OpenRC

Create a Bash script /etc/local.d/10-run_on_shutdown.stop with the following contents:

#!/bin/bash
if [ `who -r | awk '{print $2}'` = "0" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

From now on the script will run to completion when you shutdown the machine, but not when you reboot it.

systemd

1. Create a Bash script /usr/local/sbin/run_on_shutdown.sh with the following contents:

#!/bin/bash
REBOOT=$( systemctl list-jobs | egrep -q 'reboot.target.*start' && echo "rebooting" || echo "not_rebooting" )
if [ $REBOOT = "not_rebooting" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

2. Create a unit file /etc/systemd/system/run_on_shutdown.service with the following contents:

[Unit]
Description=Run a Bash script at shutdown
DefaultDependencies=no
Before=shutdown.target halt.target
# If your script requires any mounted directories, add them below: 
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/run_on_shutdown.sh

[Install]
WantedBy=halt.target shutdown.target

I have assumed the Bash script is to backup /home, so change the directory list in ‘RequiresMountsFor=‘ if the script necessitates that other directories are still mounted – see ‘man systemd.unit‘ for details.

3. Enable the unit file and start the service as follows:

user $ sudo systemctl enable run_on_shutdown.service
user $ sudo reboot

From now on, the script will run to completion when you shutdown the machine, but not when you reboot it.

Notes on the systemd solution:

There are plenty of posts on the Web suggesting how to run a script at shutdown in installations that use systemd, but the approaches either do not work in my case or they do not discriminate between shutting down and rebooting. For example, if you are wondering why I do not use ‘Conflicts=reboot.target‘ in the unit file, I found it does not prevent the Bash script from being launched when the system is rebooted (see the entry for ‘Conflicts=‘ in ‘man systemd.unit‘ for the functionality). If you are wondering why I do not simply place a script in /lib/systemd/system-shutdown/, it is because scripts in that directory are launched late in the shutdown process, after file systems have been mounted read-only. And if you are wondering why the Bash script tests if the system is rebooting (as distinct from shutting down) rather than the unit file being configured to do that, it is because the systemd shutdown target is active in any type of system termination, be it rebooting or halting, and it therefore does not help in discriminating between the termination types (see ‘man systemd.special‘). I tried several approaches in the unit file to see if it could be made to launch a Bash script when shutting down but not when rebooting, but the only solution that works for me in the Lubuntu 17.10 installation on my family’s PC uses the two files listed above working in conjunction with each other.

Furthermore, of all the unit files I found on the Web that actually make systemd launch a Bash script when a user initiates shutdown, only one of them prevents systemd from shutting down the machine before a time-consuming script has run to completion (for example a script to backup to an external HDD). I also tried using systemd-inhibit in this case, and it did not work. Looking at the manual page for systemd-inhibit, it could be used to execute a program (the example given is ‘systemd-inhibit wodim foobar.iso‘), however I found that systemd-inhibit did not prevent systemd from shutting down the machine before a Bash script to backup a single-seat, multi-user installation had run to completion.

Some background reading

1. Easy Systemd Startup and Shutdown Scripts for Ubuntu 16.04 and Mint 18
2. Stack Exchange Super User : How do I run a script before everything else on shutdown with systemd?
3. Stack Overflow : how can a systemd controlled service distinguish between shutdown and reboot?
4. Unix & Linux Stack Exchange : Systemd : How to execute script at shutdown only (not at reboot)

Bye bye Windows 10, and good riddance

Up until a couple of days ago my family’s PC, an Acer Aspire XC600 tower purchased in early 2014, had Microsoft Windows 10 Home (64-bit) installed. Because of a problem updating Windows 10 which finally rendered the PC unbootable and the OS unrecoverable, I installed Lubuntu 17.10 (64-bit). It is performing very well and my family are finding it easy to use. Although I had no intention of installing Linux on this machine before the problem updating Windows arose, I’m now glad to be rid of Windows on this machine, as Windows has been a pain to use and maintain.

The Windows update saga

When I bought the Aspire XC600 in February 2014 it came with Windows 8 pre-installed, and I immediately upgraded it to Windows 8.1. I say ‘immediately’, but it actually took me three days to get Windows Update to install it properly; the first attempts resulted in what looked like Windows 8.1 but turned out to be incomplete installations, and several times I had to roll back to a Restore Point and try to update again.

I upgraded the machine to Windows 10 Home when Microsoft offered it free-of-charge to current users of Windows 8.1 and Windows Update informed me the update was available to install. The early Windows 10 Home was buggy, but various updates by Microsoft eventually got it to a reasonably stable state by the time the so-called ‘Anniversary Update’ (Windows 10 Version 1607) was released in 2016. I again had to struggle for several days before I managed to update Windows 10 Home to Version 1607.

In April 2017 Microsoft released the ‘Creators Update’, and in October 2017 the ‘Fall Creators Update’. However, no matter what I did it was simply impossible to upgrade Window 10 Home Version 1607 on the Aspire XC600 to either of those 2017 updates. There are hundreds if not thousands of posts on the Web regarding problems installing these updates on various PC models from various manufacturers, with similar or even identical symptoms to those I was seeing. In my case the update process froze at 33%, 75% and 83%, despite Microsoft’s update utility informing me that the CPU, RAM size and HDD free space were valid for these updates. Furthermore, I only tried to update once Windows Update had informed me the updates were available to install. I should also point out that I regularly made sure the OS had all other updates installed.

I lost count of the number of times and hours spent trying to update to the Creators Update and the Fall Creators Update. Each time I had a go at updating, after two consecutive attempts Windows 10 Home would give up and, when I eventually cycled the mains power in order to exit the frozen state, would roll back to Version 1607. However, during my latest attempt a couple of days ago, Windows 10 Home would no longer complete booting, instead popping up a window informing me the machine needed to be rebooted to complete the installation process. Every time I clicked ‘OK’ in the window, the machine would reboot and the same window popped up again. So I dug out the Windows 10 Home Recovery Disk (actually a USB pendrive) I had carefully created as soon as I had upgraded the installation from Windows 8.1 to Windows 10 in November 2016. (That pendrive had previously been the Windows 8.1 Recovery Disk that I created as soon as I upgraded the installation to Windows 8.1 in February 2014.) But, no matter what I did, the Recovery Disk would only re-install Windows 8, even though the time-date stamp of the files on the pendrive corresponded to the date on which I created the Windows 10 Recovery Disk. And, strangely, there were three so-called Recovery Partitions on the HDD.

Several attempts to re-install using the Recovery Disk had the same outcome, so I decided to install a couple of Linux binary distributions in succession, both of which worked fine and definitely removed all traces of Windows from the HDD, including the three Recovery Partitions (I checked using GParted to make sure). Then I tried again to re-install Windows 10 Home from the Recovery Disk, but it still created three Recovery Partitions and still installed Windows 8.

Clearly it was not going to be possible to re-install Windows 10 Home using the Recovery Disk, so I instead used Windows Update in Windows 8 to update the installation to Windows 8.1, a process that took several hours and reboots. Once Windows 8.1 was installed, I tried to upgrade to Window 10, first using Windows Update and, when that told me there were no updates, by using the Recovery Disk. Neither approach was successful, so I was stuck with a working, fully-updated Windows 8.1. The trouble was, Windows 8.1 is no longer supported by Microsoft (‘Mainstream Support End Date’ is 9 January 2018). Not to mention that Windows 8.1 is even worse than Windows 10.

The move to Linux

At this point I’d had more than enough of Microsoft Windows. Therefore I used my laptop to download the ISO for Lubuntu 17.10 and create a LivePendrive, and I installed Lubuntu on the Aspire XC600. Although I use a source-based Linux distribution on two laptops, for ease and speed of installation and maintenance I opted to install a binary-based distribution on the family PC. I chose Lubuntu specifically because it uses the LXDE desktop environment, which is closer in look and feel to classic Windows than e.g. the Unity or GNOME desktop environments in Ubuntu, and is not as ‘CPU-hungry’ as KDE. I found that Lubuntu worked extremely well out-of-the-box, including scanning and printing using my Canon MP510 MFP. I used the GUI Software utility (‘System Tools’ > ‘Software’ from the LXDE application menu) to uninstall AbiWord and Gnumeric and install the LibreOffice suite. I added user accounts for the members of my family (‘System Tools’ > ‘Users and Groups’). Since the machines on my home network use SMB to share files, I installed samba and sambaclient and edited the smb.conf file via the command line, and browsing SMB shares worked first time. We have a decent family PC again.

There was not much more for me to do to make the installation behave exactly how I wanted it to:

• I configured the installation so that each user’s avatar appears on the login screen (LightDM GTK Greeter).
• I have an external USB HDD permanently connected to the PC so that users’ files can be backed up. I configured the installation to unmount automatically this external USB HDD when any user logs out. The USB HDD is automatically mounted anyway when another user logs in, and, by unmounting it automatically at logout, the next user can access the USB HDD properly via the GUI File Manager (the USB drive is mounted as /media/<username>/FREECOM HDD).
• I installed Language Support so that I can switch to some other languages I use, and I configured LXDE so I can click on an icon on the panel (or use a keyboard shortcut) to switch between the associated keyboard layouts.
• I installed the anti-virus utility ClamAV, the ClamAV daemon and the ClamTk GUI front-end, and configured the installation to scan automatically any files downloaded to each user’s ~/Downloads directory, and to quarantine infected files and notify the user via a pop-up window and log file.
• I configured the installation to create a network route when I log in, so that I can access in a Web browser the GoAccess dashboard for database reports produced by my network server.
• I configured the installation to backup the files in each user’s ~/home directory to an external USB HDD at shutdown (impossible in Windows 10 Home — see my comments further on).
• I installed Skype Preview for Linux, which worked out-of-the-box with a GUCEE HD92 HD 720p USB Webcam with built-in microphone.

I intend to explain in future posts how I implemented each of the above.

Backing up users’ files at shutdown

Windows XP and Vista on my family’s previous PCs were able to run a batch file (BACKUP.BAT) automatically at shutdown to backup the users’ files to an external USB HDD (and, crucially, to wait until the batch job was completed before powering down the PC). To achieve this I used the utility Xecutor by Xpertdesign Software, which enabled users to use the normal Windows method of shutting down yet allowed the batch file to run to completion. However, such utilities do not work in Windows 8 and onwards. A kludge that is often suggested is to add an extra button on the Desktop or Taskbar to run the backup commands then shutdown the machine afterwards, but I did not want to do that because there is no guarantee my family would click on it rather than shutting down Windows the normal way.

Another method of configuring Windows to run a batch file at shutdown is to use the GPE (Group Policy Editor) a.k.a. GPOE (Group Policy Object Editor) to configure the Registry. However, Windows 10 Home does not include the GPE, so I was unable to use the GPE to configure Windows 10 Home to run a batch file to backup users’ files to an external USB HDD at shutdown. (Actually, as Windows 8/8.1/10 makes it almost impossible to interrupt the shutdown process once the user has initiated shutdown, I wonder if a backup batch file would actually run to completion if the GPE were used in an edition of Windows that provides it, such as Windows 10 Enterprise.) It is possible to configure the Task Scheduler in Windows 10 Home to run a batch file at shutdown, but it is impossible to pause the shutdown process to allow the backup batch file to run to completion. Believe me, I tried everything, and it is impossible to backup automatically all users’ files for multiple user accounts at shutdown with Windows 10 Home (even though it was possible in Windows XP). So I had to resort to a kludge recommended by Microsoft, which is to configure the Task Scheduler to run the batch file at startup instead of shutdown. Clearly this is less safe than backing up before shutting down the PC.

Actually, it is possible to install/enable the GPE in Windows 10 Home — there are many Web sites explaining how to do this — but Microsoft has restricted many GPOs (Group Policy Objects) in Windows 10 Home, and therefore adding a GPO using the GPE or by editing the Registry directly in Windows 10 Home will have no effect. Even if you enable the GPE in Windows 10 Home, the policies will not work until you buy a licence for the Windows 10 Pro or Enterprise editions. In summary, in Windows 10 Home it is a waste of time either installing/enabling the GPE or editing the Registry directly.

However, now that Lubuntu 17.10 is installed I was able to configure it to run a Bash script automatically to backup all the users’ files before the machine actually actually shuts down or reboots. In a future post I’ll explain how I achieved that.

Summary

In my opinion Microsoft jumped the shark a long time ago. I had plenty of trouble with Windows Vista (to the extent I had to ditch it in the end), but Windows 7 was not bad (although on a couple of occasions I had a big scare with ‘Windows Backup and Restore’ that necessitated restoring the MBR via the command line). Windows 8 and 8.1 were awful, and Windows 10 is not much better in my opinion. Furthermore, I think it is very bad form for Microsoft to release updates to Windows 10 that cannot be installed on a machine that is only four years old and still has a reasonable specification: 64-bit Intel Pentium G2030 @ 3.00GHz, 4GB DDR3 RAM (upgradable to 8GB), Intel HD Graphics (Xeon E3-1200 v2/3rd Gen), and 1TB 7200RPM HDD. I’m now glad Windows 10 is history on this PC and I’m typing this in a Linux installation.