Backing up users’ home directories in a Linux installation that uses systemd

This is to explain how I configured Lubuntu 17.10 on my family’s PC (single seat; multiple user accounts) to backup all users’ files to a permanently-connected external USB HDD. I wanted a basic solution that would backup automatically every user’s home directory at shutdown but not when rebooting, and would do this whichever user happens to be using the PC. We have no need to archive older versions of the same file, so overwriting a file on the external USB HDD with a newer version of that file from the PC’s HDD is fine. However, we do not want files on the external USB HDD to be deleted if the corresponding files have been deleted from the PC’s HDD. Additionally, I want a time-stamped record of the backup process to be logged to a file in my home directory so I can check periodically what has been happening. As explained in my previous post, two files are used to achieve all these things:

  • a Bash script to a) determine whether the system is being rebooted, b) remount the external USB HDD on a suitable mount point, c) copy the files to the external USB HDD, and d) write to the log file;
  • a systemd unit file to launch the Bash script.

The two files are described below.

Bash script

I created a file /usr/local/sbin/backup_home_directories.sh owned by the root user, with the following permissions and contents:

fitzcarraldo@aspirexc600:~$ ls -la /usr/local/sbin/backup_home_directories.sh
-rwxr-xr-x 1 root root 1449 Jan  8 20:37 /usr/local/sbin/backup_home_directories.sh
#!/bin/bash

# This script backs up to an external USB HDD (NTFS) labelled "FREECOM HDD" the contents of the home directories
# of the users of this Lubuntu 17.10 installation if the system is shutting down but not rebooting.
# It is launched by a systemd service /etc/systemd/system/backup-to-usb-hdd.service.

# Find out if the system is rebooting (as opposed to being shut down):
REBOOT=$( systemctl list-jobs | egrep -q 'reboot.target.*start' && echo "rebooting" || echo "not_rebooting" )
if [ $REBOOT = "not_rebooting" ]; then
# Only execute the following steps if the system is shutting down but not rebooting:
    # Clean up if the backup did not complete last time:
    umount /media/usbhdd 2>/dev/null # Make sure you enter this line correctly.
    rm -rf /media/usbhdd/* # Make sure you enter this line correctly.
    # Unmount the external USB HDD if mounted by udisks2 with the logged-in username in the path:
    umount /media/*/FREECOM\ HDD 2>/dev/null
    # Find out the USB HDD device:
    DEVICE=$( blkid | grep "FREECOM\ HDD" | cut -d ":" -f1 )
    # Create a suitable mount point if it does not already exist, and mount the device on it: 
    mkdir /media/usbhdd 2>/dev/null
    mount $DEVICE /media/usbhdd 2>/dev/null
    # Create the backup directory on the USB HDD if it does not already exist:
    mkdir /media/usbhdd/Lubuntu_home_folders_backup 2>/dev/null
    # Backup recursively all the home directories of all the users, and add a time-stamped summary to the log file: 
    echo "********** Backing up Acer Aspire XC600 users' home directories **********" >> /home/fitzcarraldo/backup.log
    date >> /home/fitzcarraldo/backup.log
    # Log username of user shutting down the PC (may not be this user if Switch User was used):
    echo -ne "User who shutdown PC (may not be this user if Switch User has been used): " >> /home/fitzcarraldo/backup.log
    last | cut -d " " -f1 | head -1 >> /home/fitzcarraldo/backup.log
    sleep 2s
    # To backup the directories and files I prefer to use the following cp command rather than rsync:
    cp --recursive --update --preserve=all --no-dereference --force /home/ /media/usbhdd/Lubuntu_home_folders_backup 2>> /home/fitzcarraldo/backup.log
    echo "Copying completed" >> /home/fitzcarraldo/backup.log
    date >> /home/fitzcarraldo/backup.log
    echo "********** Backup completed **********" >> /home/fitzcarraldo/backup.log
    cp /home/fitzcarraldo/backup.log /media/usbhdd/Lubuntu_home_folders_backup/home/fitzcarraldo/
    # Unmount the USB HDD so that udisks2 can subsequently re-mount it with the user's username in the path:  
    umount /media/usbhdd
fi
exit

The Bash script unmounts the external USB HDD and mounts it to /media/usbhdd before performing a backup. This is done because the Udisks daemon (udisksd) automounts the HDD at login to a mountpoint /home/<username>/<label>  (e.g. ‘/home/fitzcarraldo/FREECOM HDD‘ when I log in, ‘/home/claudia/FREECOM HDD‘ when user claudia logs in, and so on).

systemd unit file

I created a file /etc/systemd/system/backup-to-usb-hdd.service owned by the root user, with the following permissions and contents:

fitzcarraldo@aspirexc600:~$ ls -la /etc/systemd/system/backup-to-usb-hdd.service 
-rw-r--r-- 1 root root 269 Jan  8 17:51 /etc/systemd/system/backup-to-usb-hdd.service
[Unit]
Description=Backup home directories of all users to USB HDD
Before=shutdown.target
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=/usr/local/sbin/backup_home_directories.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

Then I enabled the unit file as follows:

fitzcarraldo@aspirexc600:~$ sudo systemctl enable backup-to-usb-hdd.service

You can either reboot (or shutdown and restart) to launch it, or launch it immediately using the following command:

fitzcarraldo@aspirexc600:~$ sudo systemctl start backup-to-usb-hdd.service

The log file

The log file /home/fitzcarraldo/backup.log looked like the following after shutting down a couple of times:

********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********

Whenever the system is rebooted by any user, nothing is appended to the log file. As desired, entries are only appended to the log file when the system is shutdown by any user.

Manually initiated backups

An added benefit is that any user in the sudo group can run the Bash script from the command line at any time to backup without having to shutdown the PC, as illustrated below:

claudia@aspirexc600:~$ cat /home/fitzcarraldo/backup.log 
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********
claudia@aspirexc600:~$ date && sudo backup_home_directories.sh
Tue  9 Jan 01:57:50 GMT 2018
[sudo] password for claudia:
claudia@aspirexc600:~$ cat /home/fitzcarraldo/backup.log 
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:07:05 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:07:32 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Mon  8 Jan 21:15:31 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): fitzcarraldo
Copying completed
Mon  8 Jan 21:15:48 GMT 2018
********** Backup completed **********
********** Backing up Acer Aspire XC600 users' home directories **********
Tue  9 Jan 01:57:54 GMT 2018
User who shutdown PC (may not be this user if Switch User has been used): claudia
Copying completed
Tue  9 Jan 01:58:10 GMT 2018
********** Backup completed **********
claudia@aspirexc600:~$
Advertisements

Running a shell script at shutdown only (not at reboot) – a comparison between OpenRC and systemd

Gentoo Linux on my laptops uses OpenRC with SysVinit, whereas Lubuntu 17.10 on my family’s PC uses systemd. I have had to configure both Linux distributions to run a backup job at shutdown, so I thought it would be interesting to summarise the two approaches.

OpenRC

Create a Bash script /etc/local.d/10-run_on_shutdown.stop with the following contents:

#!/bin/bash
if [ `who -r | awk '{print $2}'` = "0" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

From now on the script will run to completion when you shutdown the machine, but not when you reboot it.

systemd

1. Create a Bash script /usr/local/sbin/run_on_shutdown.sh with the following contents:

#!/bin/bash
REBOOT=$( systemctl list-jobs | egrep -q 'reboot.target.*start' && echo "rebooting" || echo "not_rebooting" )
if [ $REBOOT = "not_rebooting" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

2. Create a unit file /etc/systemd/system/run_on_shutdown.service with the following contents:

[Unit]
Description=Run a script at shutdown but not at reboot
Before=shutdown.target
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStart=/bin/true
ExecStop=/usr/local/sbin/run_on_shutdown.sh
RemainAfterExit=true

[Install]
WantedBy=multi-user.target

I have assumed the Bash script is to backup /home, so change the directory list in ‘RequiresMountsFor=‘ if the script necessitates that other directories are still mounted – see ‘man systemd.unit‘ for details.

3. Enable the unit file and start the service as follows:

user $ sudo systemctl enable run_on_shutdown.service
user $ sudo systemctl start run_on_shutdown.service

From now on, the script will run to completion when you shutdown the machine, but not when you reboot it.

A note on the systemd solution:

There are plenty of posts on the Web suggesting how to run a script at shutdown in installations that use systemd, but the approaches either do not work in my case or they do not discriminate between shutting down and rebooting. For example, if you are wondering why I do not use ‘Conflicts=reboot.target‘ in the unit file, I found it does not prevent the Bash script from being launched when the system is rebooted (see the entry for ‘Conflicts=‘ in ‘man systemd.unit‘ for the functionality). If you are wondering why I do not simply place a script in /lib/systemd/system-shutdown/, it is because scripts in that directory are launched late in the shutdown process, after file systems have been mounted read-only. And if you are wondering why the Bash script tests if the system is rebooting (as distinct from shutting down) rather than the unit file being configured to do that, it is because the systemd shutdown target is active in any type of system termination, be it rebooting or halting, and it therefore does not help in discriminating between the termination types (see ‘man systemd.special‘). I tried several approaches in the unit file to see if it could be made to launch a Bash script when shutting down but not when rebooting, but the only solution that works for me in the Lubuntu 17.10 installation on my family’s PC uses the two files listed above working in conjunction with each other.

Some background reading

  1. Stack Exchange Super User : How do I run a script before everything else on shutdown with systemd?
  2. Stack Overflow : how can a systemd controlled service distinguish between shutdown and reboot?
  3. Unix & Linux Stack Exchange : Systemd : How to execute script at shutdown only (not at reboot)

Bye bye Windows 10, and good riddance

Up until a couple of days ago my family’s PC, an Acer Aspire XC600 tower purchased in early 2014, had Microsoft Windows 10 Home (64-bit) installed. Because of a problem updating Windows 10 which finally rendered the PC unbootable and the OS unrecoverable, I installed Lubuntu 17.10 (64-bit). It is performing very well and my family are finding it easy to use. Although I had no intention of installing Linux on this machine before the problem updating Windows arose, I’m now glad to be rid of Windows on this machine, as Windows has been a pain to use and maintain.

The Windows update saga

When I bought the Aspire XC600 in February 2014 it came with Windows 8 pre-installed, and I immediately upgraded it to Windows 8.1. I say ‘immediately’, but it actually took me three days to get Windows Update to install it properly; the first attempts resulted in what looked like Windows 8.1 but turned out to be incomplete installations, and several times I had to roll back to a Restore Point and try to update again.

I upgraded the machine to Windows 10 Home when Microsoft offered it free-of-charge to current users of Windows 8.1 and Windows Update informed me the update was available to install. The early Windows 10 Home was buggy, but various updates by Microsoft eventually got it to a reasonably stable state by the time the so-called ‘Anniversary Update’ (Windows 10 Version 1607) was released in 2016. I again had to struggle for several days before I managed to update Windows 10 Home to Version 1607.

In April 2017 Microsoft released the ‘Creators Update’, and in October 2017 the ‘Fall Creators Update’. However, no matter what I did it was simply impossible to upgrade Window 10 Home Version 1607 on the Aspire XC600 to either of those 2017 updates. There are hundreds if not thousands of posts on the Web regarding problems installing these updates on various PC models from various manufacturers, with similar or even identical symptoms to those I was seeing. In my case the update process froze at 33%, 75% and 83%, despite Microsoft’s update utility informing me that the CPU, RAM size and HDD free space were valid for these updates. Furthermore, I only tried to update once Windows Update had informed me the updates were available to install. I should also point out that I regularly made sure the OS had all other updates installed.

I lost count of the number of times and hours spent trying to update to the Creators Update and the Fall Creators Update. Each time I had a go at updating, after two consecutive attempts Windows 10 Home would give up and, when I eventually cycled the mains power in order to exit the frozen state, would roll back to Version 1607. However, during my latest attempt a couple of days ago, Windows 10 Home would no longer complete booting, instead popping up a window informing me the machine needed to be rebooted to complete the installation process. Every time I clicked ‘OK’ in the window, the machine would reboot and the same window popped up again. So I dug out the Windows 10 Home Recovery Disk (actually a USB pendrive) I had carefully created as soon as I had upgraded the installation from Windows 8.1 to Windows 10 in November 2016. (That pendrive had previously been the Windows 8.1 Recovery Disk that I created as soon as I upgraded the installation to Windows 8.1 in February 2014.) But, no matter what I did, the Recovery Disk would only re-install Windows 8, even though the time-date stamp of the files on the pendrive corresponded to the date on which I created the Windows 10 Recovery Disk. And, strangely, there were three so-called Recovery Partitions on the HDD.

Several attempts to re-install using the Recovery Disk had the same outcome, so I decided to install a couple of Linux binary distributions in succession, both of which worked fine and definitely removed all traces of Windows from the HDD, including the three Recovery Partitions (I checked using GParted to make sure). Then I tried again to re-install Windows 10 Home from the Recovery Disk, but it still created three Recovery Partitions and still installed Windows 8.

Clearly it was not going to be possible to re-install Windows 10 Home using the Recovery Disk, so I instead used Windows Update in Windows 8 to update the installation to Windows 8.1, a process that took several hours and reboots. Once Windows 8.1 was installed, I tried to upgrade to Window 10, first using Windows Update and, when that told me there were no updates, by using the Recovery Disk. Neither approach was successful, so I was stuck with a working, fully-updated Windows 8.1. The trouble was, Windows 8.1 is no longer supported by Microsoft (‘Mainstream Support End Date’ is 9 January 2018). Not to mention that Windows 8.1 is even worse than Windows 10.

The move to Linux

At this point I’d had more than enough of Microsoft Windows. Therefore I used my laptop to download the ISO for Lubuntu 17.10 and create a LivePendrive, and I installed Lubuntu on the Aspire XC600. Although I use a source-based Linux distribution on two laptops, for ease and speed of installation and maintenance I opted to install a binary-based distribution on the family PC. I chose Lubuntu specifically because it uses the LXDE desktop environment, which is closer in look and feel to classic Windows than e.g. the Unity or GNOME desktop environments in Ubuntu, and is not as ‘CPU-hungry’ as KDE. I found that Lubuntu worked extremely well out-of-the-box, including scanning and printing using my Canon MP510 MFP. I used the GUI Software utility (‘System Tools’ > ‘Software’ from the LXDE application menu) to uninstall AbiWord and Gnumeric and install the LibreOffice suite. I added user accounts for the members of my family (‘System Tools’ > ‘Users and Groups’). Since the machines on my home network use SMB to share files, I installed samba and sambaclient and edited the smb.conf file via the command line, and browsing SMB shares worked first time. We have a decent family PC again.

There was not much more for me to do to make the installation behave exactly how I wanted it to:

  • I configured the installation so that each user’s avatar appears on the login screen (LightDM GTK Greeter).
  • I have an external USB HDD permanently connected to the PC so that users’ files can be backed up. I configured the installation to unmount automatically this external USB HDD when any user logs out. The USB HDD is automatically mounted anyway when another user logs in, and, by unmounting it automatically at logout, the next user can access the USB HDD properly via the GUI File Manager (the USB drive is mounted as /media/<username>/FREECOM HDD).
  • I installed Language Support so that I can switch to some other languages I use, and I configured LXDE so I can click on an icon on the panel (or use a keyboard shortcut) to switch between the associated keyboard layouts.
  • I installed the anti-virus utility ClamAV, the ClamAV daemon and the ClamTk GUI front-end, and configured the installation to scan automatically any files downloaded to each user’s ~/Downloads directory, and to quarantine infected files and notify the user via a pop-up window and log file.
  • I configured the installation to create a network route when I log in, so that I can access in a Web browser the GoAccess dashboard for database reports produced by my network server.
  • I configured the installation to backup the files in each user’s ~/home directory to an external USB HDD at shutdown (impossible in Windows 10 Home — see my comments further on).
  • I installed Skype Preview for Linux, which worked out-of-the-box with a GUCEE HD92 HD 720p USB Webcam with built-in microphone.

I intend to explain in future posts how I implemented each of the above.

Backing up users’ files at shutdown

Windows XP and Vista on my family’s previous PCs were able to run a batch file (BACKUP.BAT) automatically at shutdown to backup the users’ files to an external USB HDD (and, crucially, to wait until the batch job was completed before powering down the PC). To achieve this I used the utility Xecutor by Xpertdesign Software, which enabled users to use the normal Windows method of shutting down yet allowed the batch file to run to completion. However, such utilities do not work in Windows 8 and onwards. A kludge that is often suggested is to add an extra button on the Desktop or Taskbar to run the backup commands then shutdown the machine afterwards, but I did not want to do that because there is no guarantee my family would click on it rather than shutting down Windows the normal way.

Another method of configuring Windows to run a batch file at shutdown is to use the GPE (Group Policy Editor) a.k.a. GPOE (Group Policy Object Editor) to configure the Registry. However, Windows 10 Home does not include the GPE, so I was unable to use the GPE to configure Windows 10 Home to run a batch file to backup users’ files to an external USB HDD at shutdown. (Actually, as Windows 8/8.1/10 makes it almost impossible to interrupt the shutdown process once the user has initiated shutdown, I wonder if a backup batch file would actually run to completion if the GPE were used in an edition of Windows that provides it, such as Windows 10 Enterprise.) It is possible to configure the Task Scheduler in Windows 10 Home to run a batch file at shutdown, but it is impossible to pause the shutdown process to allow the backup batch file to run to completion. Believe me, I tried everything, and it is impossible to backup automatically all users’ files for multiple user accounts at shutdown with Windows 10 Home (even though it was possible in Windows XP). So I had to resort to a kludge recommended by Microsoft, which is to configure the Task Scheduler to run the batch file at startup instead of shutdown. Clearly this is less safe than backing up before shutting down the PC.

Actually, it is possible to install/enable the GPE in Windows 10 Home — there are many Web sites explaining how to do this — but Microsoft has restricted many GPOs (Group Policy Objects) in Windows 10 Home, and therefore adding a GPO using the GPE or by editing the Registry directly in Windows 10 Home will have no effect. Even if you enable the GPE in Windows 10 Home, the policies will not work until you buy a licence for the Windows 10 Pro or Enterprise editions. In summary, in Windows 10 Home it is a waste of time either installing/enabling the GPE or editing the Registry directly.

However, now that Lubuntu 17.10 is installed I was able to configure it to run a Bash script automatically to backup all the users’ files before the machine actually actually shuts down or reboots. In a future post I’ll explain how I achieved that.

Summary

In my opinion Microsoft jumped the shark a long time ago. I had plenty of trouble with Windows Vista (to the extent I had to ditch it in the end), but Windows 7 was not bad (although on a couple of occasions I had a big scare with ‘Windows Backup and Restore’ that necessitated restoring the MBR via the command line). Windows 8 and 8.1 were awful, and Windows 10 is not much better in my opinion. Furthermore, I think it is very bad form for Microsoft to release updates to Windows 10 that cannot be installed on a machine that is only four years old and still has a reasonable specification: 64-bit Intel Pentium G2030 @ 3.00GHz, 4GB DDR3 RAM (upgradable to 8GB), Intel HD Graphics (Xeon E3-1200 v2/3rd Gen), and 1TB 7200RPM HDD. I’m now glad Windows 10 is history on this PC and I’m typing this in a Linux installation.

Moving to the slotted WINE package system in Gentoo Linux

Earlier this year the Gentoo Linux developers responsible for maintaining the package app-emulation/wine decided to split it and slot it so that different versions of WINE can be installed and co-exist simultaneously:

clevow230ss fitzcarraldo # eselect news read 34
2017-04-10-split-and-slotted-wine
  Title                     app-emulation/wine split and slotting
  Author                    NP-Hardass 
  Posted                    2017-04-10
  Revision                  1

Starting with Wine 2.0, Wine in Gentoo is transitioning away from its
traditional packaging and toward a new, split and slotted, Wine.

As many Wine users know, there are often regressions or an application
works better on one version of wine than another.  Going forward, 
packaging in Gentoo will allow simultaneous installation of multiple
versions of Wine.

Additionally, to expedite vanilla releases as well as permit multiple
configurations for each Wine installation, the major patchsets have
been split out into separate packages.

Going forward, app-emulation/wine will transition to:
app-emulation/wine-vanilla: upstream Wine with no external patchsets
             (like if the old packaging forced USE="-staging -d3d9")
app-emulation/wine-staging: Wine with Wine-Staging's patchset
             (like if the old packaging forced USE="+staging -d3d9")
app-emulation/wine-d3d9: Wine with Ixit's Gallium Nine patchset
             (like if the old packaging forced USE="-staging +d3d9")
app-emulation/wine-any: Wine with any of the patchsets or flags
             (exactly like the old packaging regarding USE flags)

wine-any exists to allow the user to build any combination that they'd
like (like the old packaging).  This means the user could use wine-any
to use both Wine-Staging and Gallium Nine.  Alternatively, the user
could use wine-any to try out another configuration from other
packages.  For example, the user could build wine-vanilla without
PulseAudio, and could build wine-any with PulseAudio.  The sky is the
limit on how a user may choose to use app-emulation/wine-any.

Users may opt for any specific package, or may emerge virtual/wine,
which is provided for dependency resolution.
Maintainers: Please note, app-emulation/wine will be dropped, so
please use virtual/wine going forward.

Users may call each version specifically, or may call a symlink based
on their installed patchset, for example wine-2.1, wine-staging-2.2,
or wine-d3d9.

Symlinks for wine are managed with app-eselect/eselect-wine.
# eselect wine set wine-vanilla-2.0
/usr/bin/wine -> /usr/bin/wine-vanilla-2.0
# eselect wine set --staging wine-staging-2.4
/usr/bin/wine-staging -> /usr/bin/wine-staging-2.4

Earlier this year the Gentoo Linux Forums thread ‘wine: questions on recent changes‘ discussed the new system and how to use it. Several users, myself included, posted questions in that thread asking how to go about the change. I received differing advice and remained uncertain about what to do. At the time, the slotted packages for the new system had not yet been unmasked, so I left my installation as it was and decided to put off making the change for as long as possible. Today when I synchronised my installation with the Portage tree there was a Gentoo Linux news item ‘2017-11-21 Old Wine versions moving to wine-overlay’. So the time had come for me to make the switch from app-emulation/wine-2.3 to the new slotted WINE package system. Here is what I did…

Previous situation

I had been using WINE Staging:

clevow230ss fitzcarraldo # eix -I wine
[?] app-emulation/wine
     Available versions:  [M]2.0^t [M](~)2.1^t [M](~)2.2^t [M](~)2.3^t [M]**9999^t {+X +alsa capi cups custom-cflags d3d9 dos +fontconfig +gecko gphoto2 gsm gstreamer +jpeg +lcms ldap +mono mp3 ncurses netapi nls odbc openal opencl +opengl osmesa oss pcap +perl pipelight +png prelink pulseaudio +realtime +run-exes s3tc samba scanner selinux +ssl staging test themes +threads +truetype udev +udisks v4l vaapi +xcomposite xinerama +xml ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="(+)32 (+)64 x32" ELIBC="glibc" KERNEL="FreeBSD" LINGUAS="ar bg ca cs da de el en en_US eo es fa fi fr he hi hr hu it ja ko lt ml nb_NO nl or pa pl pt_BR pt_PT rm ro ru sk sl sr_RS@cyrillic sr_RS@latin sv te th tr uk wa zh_CN zh_TW"}
     Installed versions:  2.3^t(19:16:31 20/05/17)(X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32" ELIBC="glibc" KERNEL="-FreeBSD" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW")
     Homepage:            https://www.winehq.org/
     Description:         Free implementation of Windows(tm) on Unix

[I] app-emulation/winetricks
     Available versions:  20170823^t **99999999^t {gtk kde rar}
     Installed versions:  20170823^t(02:50:56 06/09/17)(gtk kde -rar)
     Homepage:            https://github.com/Winetricks/winetricks https://wiki.winehq.org/Winetricks
     Description:         Easy way to install DLLs needed to work around problems in Wine

Found 2 matches

The file /etc/portage/package.accept_keywords/wine in my installation included the following line:

app-emulation/wine ~amd64

The file /etc/portage/package.use/wine in my installation included the following line:

app-emulation/wine -ldap -pipelight staging abi_x86_32

New situation

The file /etc/portage/package.accept_keywords/wine in my installation now includes the lines listed below. The reason why I had to include app-emulation/wine-vanilla is explained further on.

virtual/wine ~amd64
app-emulation/wine-staging ~amd64
app-emulation/wine-vanilla ~amd64

The file /etc/portage/package.use/wine in my installation now includes the lines listed below. The reason why I had to include app-emulation/wine-vanilla is explained further on.

virtual/wine -d3d9 staging abi_x86_32
app-emulation/wine-staging X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
app-emulation/wine-vanilla X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32

(Note that e.g. the pipelight USE flag does not actually exist in the package app-emulation/wine-vanilla, but it does no harm to include “-pipelight” in the line for app-emulation/wine-vanilla in the file package.use.)

After adding the above-mentioned lines to the two files, I then had to uninstall the masked package app-emulation/wine-2.3 and the package app-emulation/winetricks-20170823 that depended on it:

clevow230ss fitzcarraldo # emerge -aC app-emulation/wine app-emulation/winetricks

Then I had to install the package virtual/wine, which pulled in the package app-emulation/wine-staging as I wanted, but also pulled in the package app-emulation/vanilla:

clevow230ss fitzcarraldo # emerge -a virtual/wine

Then I had to select wine-staging:

clevow230ss fitzcarraldo # eselect wine list
clevow230ss fitzcarraldo # eselect wine set

Finally, I had to install winetricks, which would now use the selected slotted package:

clevow230ss fitzcarraldo # emerge winetricks

How I arrived at the contents of package.accept_keywords and package.use

Note that, before I added the line for app-emulation/wine-vanilla to /etc/portage/package.use/wine and /etc/portage/package.accept_keywords/wine, this is what Portage wanted to install:

clevow230ss fitzcarraldo # emerge -p virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild   R    ] net-nds/openldap-2.4.44  ABI_X86="32*" 
[ebuild  N     ] app-emulation/wine-vanilla-2.0.2  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms ldap mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by app-emulation/wine-vanilla-2.0.2::gentoo[ldap]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=net-nds/openldap-2.4.44 abi_x86_32
# required by app-emulation/wine-vanilla-2.0.2::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32

!!! The following installed packages are masked:
- dev-qt/qtwebkit-4.8.7::gentoo (masked by: package.mask)
/usr/portage/profiles/package.mask:
# Andreas Sturmlechner  (16 Nov 2017)
# Qt4WebKit is ancient and full of security holes.
# Masked for removal in 30 days. Bug #620684

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

clevow230ss fitzcarraldo #

After I added a line for app-emulation/wine-vanilla to /etc/portage/package.use/wine and /etc/portage/package.accept_keywords/wine, this is what Portage wanted to do:

clevow230ss fitzcarraldo # emerge -p virtual/wine                        

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by app-emulation/wine-vanilla-2.20::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32
clevow230ss fitzcarraldo #

So I added the above-mentioned USE change for app-emulation/wine-gecko to /etc/portage/package.use/wine, and then Portage wanted to do the following:

clevow230ss fitzcarraldo # emerge -p virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 
clevow230ss fitzcarraldo #

I was happy with that, so I went ahead without the --pretend option:

clevow230ss fitzcarraldo # emerge -a virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

Would you like to merge these packages? [Yes/No] Yes
>>> Verifying ebuild manifests
>>> Running pre-merge checks for app-emulation/wine-vanilla-2.20
>>> Running pre-merge checks for app-emulation/wine-staging-2.19
>>> Emerging (1 of 6) app-emulation/wine-gecko-2.47-r1::gentoo
>>> Emerging (2 of 6) app-eselect/eselect-wine-1.2.2::gentoo
>>> Installing (1 of 6) app-emulation/wine-gecko-2.47-r1::gentoo
>>> Installing (2 of 6) app-eselect/eselect-wine-1.2.2::gentoo
>>> Emerging (3 of 6) app-emulation/wine-desktop-common-20150204::gentoo
>>> Installing (3 of 6) app-emulation/wine-desktop-common-20150204::gentoo
>>> Emerging (4 of 6) app-emulation/wine-vanilla-2.20::gentoo
>>> Emerging (5 of 6) app-emulation/wine-staging-2.19::gentoo
>>> Installing (4 of 6) app-emulation/wine-vanilla-2.20::gentoo
>>> Installing (5 of 6) app-emulation/wine-staging-2.19::gentoo
>>> Emerging (6 of 6) virtual/wine-0-r6::gentoo
>>> Installing (6 of 6) virtual/wine-0-r6::gentoo
>>> Recording virtual/wine in "world" favorites file...
>>> Jobs: 6 of 6 complete                           Load avg: 6.80, 7.34, 7.06
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19
  [2]   wine-vanilla-2.20 *
clevow230ss fitzcarraldo # eselect wine set 1
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19 *
  [2]   wine-vanilla-2.20
clevow230ss fitzcarraldo # emerge app-emulation/winetricks
Calculating dependencies... done!
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) app-emulation/winetricks-20170823::gentoo
>>> Installing (1 of 1) app-emulation/winetricks-20170823::gentoo
>>> Recording app-emulation/winetricks in "world" favorites file...
>>> Jobs: 1 of 1 complete                           Load avg: 0.51, 2.78, 5.09
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.
clevow230ss fitzcarraldo #

Summary

So, in summary, this is what I did in order to install and use app-emulation/wine-staging:

1. Add the following lines to /etc/portage/package.accept_keywords/wine:

virtual/wine ~amd64
app-emulation/wine-staging ~amd64
app-emulation/wine-vanilla ~amd64

2. Add the following lines to /etc/portage/package.use/wine:

virtual/wine -d3d9 staging abi_x86_32
app-emulation/wine-staging X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
app-emulation/wine-vanilla X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
# required by app-emulation/wine-vanilla-2.20::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32

3. Uninstall the old, un-slotted WINE packages:

root # emerge -aC app-emulation/wine app-emulation/winetricks

4. Merge the new virtual WINE package:

root # emerge -a virtual/wine

5. Select the slotted WINE package I wish to use (WINE Staging):

root # eselect wine list
root # eselect wine set wine-staging-2.19

6. Merge the winetricks package, which will now recognise the slotted WINE package selected:

root # emerge app-emulation/winetricks

After completing the above process, I was still able to launch as before the various Microsoft Windows applications installed on my laptop under WINE.

UPDATE (December 1, 2017): As explained by user Chiltoo in a new post in the Gentoo Linux Forums thread mentioned earlier in the above post, a bug in Portage Version 2.3.13-r1 results in app-emulation/wine-vanilla being installed unnecessarily, as borne out in my post above.

The work-around he proposed does indeed work for me. Below are the commands I have just used for the work-around (I have left the new entries I made in package.accept_keywords and package.use in those files):

clevow230ss fitzcarraldo # emerge -aC app-emulation/wine-vanilla app-emulation/wine-staging virtual/wine app-emulation/wine-desktop-common app-emulation/wine-gecko app-eselect/eselect-wine app-emulation/winetricks
clevow230ss fitzcarraldo # emerge -1 app-emulation/wine-staging && emerge virtual/wine && emerge winetricks

The outcome of the above two commands in my case is:

clevow230ss fitzcarraldo # eix --installed --compact wine
[I] app-emulation/wine-desktop-common (20150204@01/12/17): Various desktop menu items and icons for wine
[I] app-emulation/wine-gecko (2.47-r1(2.47)@01/12/17): A Mozilla Gecko based version of Internet Explorer for Wine
[I] app-emulation/wine-staging (2.19(2.19)@01/12/17): Free implementation of Windows(tm) on Unix, with Wine-Staging patchset
[I] app-emulation/winetricks (20170823@01/12/17): Easy way to install DLLs needed to work around problems in Wine
[I] app-eselect/eselect-wine (1.2.2@01/12/17): Manage active wine version
[I] virtual/wine (0-r6@01/12/17): Virtual for Wine that supports multiple variants and slotting
Found 6 matches
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19 *
clevow230ss fitzcarraldo #

The superfluous app-emulation/wine-vanilla is no longer installed, which is a cleaner outcome.

Prevent Linux firewalls interfering with Samba commands in a home network that uses broadcast NetBIOS name resolution

Or “How come devices in a home network can browse SMB shares but Linux Samba commands and Windows nbtstat commands do not work properly?”

Introduction

In a previous post I explained how it is possible to browse SMB shares when using broadcast NetBIOS name resolution in a home network consisting of machines running Linux, Windows and other operating systems. Browsing SMB/Samba shares will work as expected, but Samba commands such as ‘smbtree‘, ‘smbclient‘ and ‘nmblookup‘ will not work properly if the Linux machines use a firewall that has not been configured for broadcast NetBIOS name resolution. This post is to explain how to do that.

If broadcast NetBIOS name resolution is being used and none of the Linux machines has a firewall enabled, or if their firewalls have been correctly configured, the output of e.g. the ‘smbtree‘ command on one of those machines would look something like the example below.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
                \\TUTANKHAMUN\Samsung_Xpress_C460FW     Samsung Xpress C460FW
                \\TUTANKHAMUN\Canon_MP560_Printer       Canon PIXMA MP560
                \\TUTANKHAMUN\Canon_MP510_Printer       Canon PIXMA MP510
                \\TUTANKHAMUN\Virtual_PDF_Printer       Virtual PDF Printer
                \\TUTANKHAMUN\IPC$              IPC Service (Samba 4.2.11)
                \\TUTANKHAMUN\Public
                \\TUTANKHAMUN\anne-share
                \\TUTANKHAMUN\print$
                \\TUTANKHAMUN\netlogon          Network Logon Service
        \\BTHUB5                        BT Home Hub 5.0A File Server
                \\BTHUB5\IPC$                   IPC Service (BT Home Hub 5.0A File Server)
        \\THUTMOSEIII                   Windows 10 computer

If Linux firewalls have not been correctly configured, the output would be missing some information about other machines in the network. For example, compare the output above with the output below from the same network, this time with the Linux firewalls configured using typical rules for Samba specified in Web articles, blog posts and forums.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
        \\BTHUB5                        BT Home Hub 5.0A File Server
        \\THUTMOSEIII                   Windows 10 computer

To avoid this problem you need to add a further Linux firewall rule to the set of rules usually used for Samba. Below I first list the usual firewall rules for Samba, then I give the additional rule necessary if using broadcast NetBIOS name resolution. In each case I give the applicable rules for a pure IPTABLES firewall and for UFW (Uncomplicated Firewall). The rules listed here assume the IP address range of the home network is 192.168.1.0/24, so change the range to suit the specific network.

Firewall rules typically specified for machines using Samba

IPTABLES

The rules listed below assume the machine uses interface eth0, so change the interface to suit the specific machine.

# NetBIOS Name Service (name resolution)
iptables -A INPUT -i eth0 -p udp --dport 137 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Datagram Service (BROWSER service)
iptables -A INPUT -i eth0 -p udp --dport 138 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Session Service (data transfer legacy SMB/NetBIOS/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 139 -s 192.168.1.0/24 -j ACCEPT

# Microsoft Directory Service (data transfer SMB/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 445 -s 192.168.1.0/24 -j ACCEPT

UFW

In some Linux distributions the ufw application allows a single command to add Samba support, such as:

user $ sudo ufw allow Samba

or

user $ sudo ufw allow CIFS

These ‘application profiles’ are specified in files in the directory /etc/ufw/applications.d/, so you could add application profiles or modify existing ones if you wish. In one of my installations the file /etc/ufw/applications.d/ufw-fileserver includes the following application profile for Samba, for example:

[CIFS]
title=SMB/CIFS server
description=SMB/CIFS server
ports=137,138/udp|139,445/tcp

If such an application profile does not exist in your installation, typical Samba rules can be added in UFW using the following two commands:

user $ sudo ufw allow from 192.168.1.0/24 to any port 137,138 proto udp
user $ sudo ufw allow from 192.168.1.0/24 to any port 139,445 proto tcp

The correct addition of the rules can be checked using the following command:

user $ sudo ufw status verbose
Password:
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
137,138/udp (CIFS)         ALLOW IN    192.168.1.0/24
139,445/tcp (CIFS)         ALLOW IN    192.168.1.0/24

The extra rule required when using broadcast NetBIOS name resolution

The reason why an extra rule is required when using broadcast NetBIOS name resolution is because UFW (which is based on IPTABLES) is ‘stateful’, as is a purely IPTABLES firewall (unless explicitly configured not to be stateful). The firewall does not consider packets it receives in response to its broadcast to be ESTABLISHED or RELATED, and therefore drops those packets. So, despite the IPTABLES and UFW rules listed above including a rule to accept incoming UDP packets on Port 137, any UDP packets received on Port 137 that do not constitute a one-to-one, two-way communication flow are dropped by the firewall. The extra rule below overrules this and makes the firewall accept packets coming from other devices’ Port 137 in response to broadcast NetBIOS Name Service packets. To do this, the extra rule uses a CT (Connection Tracking) helper named ‘netbios-ns‘ (obviously meaning ‘NetBIOS Name Service’). In order to use this rule the kernel must have been configured to use the IPTABLES ‘raw‘ table and to use CT (see the section ‘Kernel configuration’ further on).

IPTABLES

# All NetBIOS clients must have the netbios-ns helper enabled for broadcast name resolution to work
iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns

By the way, in addition to flushing the usual tables, flush the ‘raw‘ table too when you restart the firewall:

iptables -t raw -F OUTPUT

UFW

Add the following lines to the end of the file /etc/ufw/before.rules

# The following is needed to enable Samba commands to
# work properly for broadcast NetBIOS name resolution
#
# raw table rules
*raw
:OUTPUT ACCEPT [0:0]
-F OUTPUT
-A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns
COMMIT

Note that the output of the command ‘ufw status verbose‘ will not include the above rule. This is not a bug.

Kernel configuration

If you are using a binary-based distribution such as Ubuntu Linux, the kernel will probably have been configured to include the needed modules (CONFIG_IP_NF_RAW=m, CONFIG_IP6_NF_RAW=m and CONFIG_NETFILTER_XT_TARGET_CT=m), and the installation configured to load the modules automatically. However, if you are using a source-based distribution such as Gentoo Linux make sure the kernel configuration includes these three options before you build the kernel, and also add the module names ‘iptable_raw‘ and ‘xt_CT‘ to the module list in the file /etc/conf.d/modules as shown in the example below, so that the modules are loaded at boot:

modules="r8169 nvidia agpgart fuse bnep rfcomm hidp uvcvideo cifs mmc_block rtsx_pci snd-seq-midi vboxdrv vboxnetadp vboxnetflt iptable_raw xt_CT"

You can use the following two commands to check if the two modules are loaded:

user $ sudo lsmod | grep iptable_raw
user $ sudo lsmod | grep xt_CT

How to check the additional rule is active

You can use the command below whether you are using pure IPTABLES or UFW.

user $ sudo iptables -nvL -t raw
Password: 
Chain PREROUTING (policy ACCEPT 2613 packets, 1115K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 2773 packets, 475K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   16  1248 CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:137 CT helper netbios-ns

The packet and byte counts will increase whenever you use a Samba command.

Bibliography

  1. The netfilter.org "iptables" project
  2. Iptables Tutorial
  3. Introduction to IPTables
  4. Gentoo Wiki : iptables
  5. Arch Linux Wiki : Samba : "Browsing" network fails with "Failed to retrieve share list from server"
  6. Ubuntu : Manpage : ufw-framework
  7. Gentoo Wiki : UFW

Dealing with the kernel image and kernel source code in Sabayon Linux

Some Sabayon Linux users do not know the particulars of the Linux kernel installation or how to upgrade the kernel image. A recent post by a new user in the Sabayon Linux Forums illustrates this (translated from the German):

I’m just setting up my first sabayon box.

In the manual you should see which kernel is selected with:

eselect kernel list

But this is empty for me:

(not found)

When installing one or other packages I also already have a corresponding error message. What am I doing wrong?

The command ‘eselect kernel list‘ lists the versions of kernel source code currently installed (directories such as /usr/src/linux-4.8.17-sabayon/, /usr/src/linux-4.9.38-sabayon/ or whatever) and indicates to which kernel source code directory the symlink /usr/src/linux/ is currently pointing (see Kernel/Configuration – Set symlink in the Gentoo Linux Wiki).

My guess is that this user wanted to rebuild the kernel image. Anyway, the situation in Sabayon Linux he described occurs as a result of either not having installed the kernel source code or not having installed the version of kernel source code corresponding to the version of the installed kernel image.

Before I go into detail on how to do the above, it is essential to understand the basics of the Entropy package management system and how to maintain a sane installation in Sabayon Linux. You should regularly: a) update the local database containing a list of available packages and their versions; b) upgrade installed packages to their latest available version; c) check and fix dependencies and libraries; d) remove any old versions of installed packages:

root # equo update
root # equo upgrade
root # equo conf update
root # equo deptest
root # equo libtest
root # equo upgrade --purge
root # equo cleanup

Do not blindly enter the above commands if you do not know what they do; first read the explanations in the list of equo functions in the Sabayon Linux Wiki.

I have found that sometimes the command ‘equo upgrade --purge‘ does not report all the packages that have to be removed manually. I therefore repeat the command a few times until no more packages are listed for removal.

Right, now let’s look at why the aforementioned Sabayon Linux user was in trouble…

If you happen to have the directory /boot on a separate partition to / (root) and you specified in the file /etc/fstab that /boot should not be mounted automatically when the machine boots, do not forget to mount the boot partition first.

I would see the same message as the aforementioned Sabayon Linux user if the kernel image and source code were not set up correctly in my Sabayon Linux installation:

sabayon fitzcarraldo # eselect kernel list
Available kernel symlink targets:
  (none found)
sabayon fitzcarraldo #

Obviously a kernel image must be present, so first let’s check which version of the kernel image package is installed:

sabayon fitzcarraldo # equo search --installed linux-sabayon
╠  @@ Searching...
╠      @@ Package: sys-kernel/linux-sabayon-4.8.17 branch: 5, [__system__]
╠          Installed:     version: 4.8.17 ~ tag: NoTag ~ revision: 0
╠          Slot:          4.8
╠          Homepage:      https://github.com/Sabayon/kernel
╠          Description:   Official Sabayon Linux Standard
╠                         kernel image
╠          License:       GPL-2 freedist
╠   Keywords:  linux-sabayon
╠   Found:     1 entry
sabayon fitzcarraldo #

As you can see above, Version 4.8.17 of the kernel image package was installed in my case.

Note that the kernel source code is not installed by default in Sabayon Linux, so my guess is that the aforementioned user didn’t have the kernel source code installed. Here’s how to find out which version of the kernel source code package is installed, if any:

sabayon fitzcarraldo # equo search --installed sabayon-sources
╠  @@ Searching...
╠      @@ Package: sys-kernel/sabayon-sources-4.11.10 branch: 5, [__system__]
╠          Installed:     version: 4.11.10 ~ tag: NoTag ~ revision: 0
╠          Slot:          4.11
╠          Homepage:      https://github.com/Sabayon/kernel
╠          Description:   Official Sabayon Linux Standard
╠                         kernel sources
╠          License:       GPL-2 freedist
╠   Keywords:  sabayon-sources
╠   Found:     1 entry
sabayon fitzcarraldo #

As you can see above, in my case I had previously installed Version 4.11.10 of the kernel source code, and it didn’t correspond to the kernel image in use.

If you want to rebuild the kernel, you also need to have the linux kernel headers installed, and those are not installed by default in Sabayon Linux, so let’s check if that package is already installed while we’re at it:

sabayon fitzcarraldo # equo search --installed linux-headers
╠  @@ Searching...
╠      @@ Package: sys-kernel/linux-headers-4.4 branch: 5, [__system__]
╠          Installed:     version: 4.4 ~ tag: NoTag ~ revision: 0
╠          Slot:          0
╠          Homepage:      https://www.kernel.org/
╠                         https://www.gentoo.org/
╠          Description:   Linux system headers
╠          License:       GPL-2
╠   Keywords:  linux-headers
╠   Found:     1 entry
sabayon fitzcarraldo #

As you can see above, in my case I had previously installed Version 4.4 of the linux kernel headers. However, if you don’t already have the package installed, install it:

sabayon fitzcarraldo # equo install linux-headers
╠  @@ Calculating dependencies...
╠  ## [R] [sabayon-weekly] sys-kernel/linux-headers-4.4|0   [4.4|0]
╠  @@ Packages needing to be installed/updated/downgraded: 1
╠  @@ Packages needing to be removed: 0
╠  @@ Download size: 937.6kB
╠  @@ Freed disk space: 0.0b
╠  @@ You need at least: 1.9MB of free space
╠  ::: >>>  (1/1) 1 package
╠    ## Downloading: 1 package
╠    ## ( mirror #1 ) [sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2] @ http://na.mirror.garr.it
╠   ## Aggregated download: 1 item
╠    # [1] na.mirror.garr.it => sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2
╠    ## Checking package checksum...
╠       : [sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2] GPG validated
╠       : SHA1 disabled
╠       : [sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2] SHA256 validated
╠       : SHA512 disabled
╠    ## ( mirror #1 ) [sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2] success @ http://na.mirror.garr.it
╠    ##  Aggregated transfer rate: 1.3MB/second
╠  +++ >>>  (1/1) sys-kernel/linux-headers-4.4
╠    ## Unpacking: sys-kernel:linux-headers-4.4.150ec398796671a9b475328e5ae3f180b9b096c3~0.tbz2
╠    ## Package phase: setup
╠    ## Package phase: preinstall
╠    ## Installing package: sys-kernel/linux-headers-4.4
╠    ## [Linux system headers]
╠    ## Updating installed packages repository: sys-kernel/linux-headers-4.4
╠    ## Cleaning previously installed application data.
╠    ## Package phase: postremove
╠    ## Package phase: postinstall
╠    ## Cleaning: sys-kernel/linux-headers-4.4
╠  @@ Installation complete.
╠  @@ No configuration files to update.
sabayon fitzcarraldo #

To make sure you have the latest versions of the packages sys-kernel/linux-sabayon (the kernel image) and sys-kernel/sabayon-sources (the kernel source code), install the packages as follows:

sabayon fitzcarraldo # equo install linux-sabayon sabayon-sources
╠  @@ Calculating dependencies...
╠  ## [N] [sabayon-weekly] sys-kernel/linux-sabayon-4.11.10|0
╠  ## [R] [sabayon-weekly] sys-kernel/sabayon-sources-4.11.10|0   [4.11.10|0]
╠  @@ Packages needing to be installed/updated/downgraded: 2
╠  @@ Packages needing to be removed: 0
╠  @@ Download size: 198.9MB
╠  @@ Used disk space: 211.3MB
╠  @@ You need at least: 609.1MB of free space
╠  ::: >>>  (1/1) 2 packages
╠    ## Downloading: 2 packages
╠    ## ( mirror #1 ) [sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2] @ http://na.mirror.garr.it
╠    ## ( mirror #1 ) [sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2] @ http://na.mirror.garr.it
╠   ## Aggregated download: 2 items
╠    # [1] na.mirror.garr.it => sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2
╠    # [2] na.mirror.garr.it => sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2
╠    ## Checking package checksum...
╠       : [sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2] GPG validated
╠       : SHA1 disabled
╠       : [sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2] SHA256 validated
╠       : SHA512 disabled
╠    ## Checking package checksum...
╠       : [sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2] GPG validated
╠       : SHA1 disabled
╠       : [sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2] SHA256 validated
╠       : SHA512 disabled
╠    ## ( mirror #1 ) [sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2] success @ http://na.mirror.garr.it
╠    ## ( mirror #1 ) [sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2] success @ http://na.mirror.garr.it
╠    ##  Aggregated transfer rate: 3.7MB/second
╠  +++ >>>  (1/2) sys-kernel/linux-sabayon-4.11.10
╠    ## Unpacking: sys-kernel:linux-sabayon-4.11.10.1f22c24cc709872f47d62d3c9e44af879fe18888~0.tbz2
╠    ## Package phase: setup
 * To avoid automounting and auto(un)installing with /boot,
 * just export the DONT_MOUNT_BOOT variable.

 * Your boot partition was detected as being mounted at /boot.
 * Files will be installed there for linux-sabayon to function correctly.
 * Preparing kernel and its modules
╠    ## Package phase: preinstall

 * Your boot partition was detected as being mounted at /boot.
 * Files will be installed there for linux-sabayon to function correctly.
╠    ## Installing package: sys-kernel/linux-sabayon-4.11.10
╠    ## [Official Sabayon Linux Standard kernel image]
╠    ## Updating installed packages repository: sys-kernel/linux-sabayon-4.11.10
╠    ## Package phase: postinstall
 * Removing extents option for ext4 drives from /etc/fstab
Generating grub configuration file ...
Found background: /boot/grub/default-splash.png
Found linux image: /boot/kernel-genkernel-x86_64-4.11.0-sabayon
Found initrd image: /boot/initramfs-genkernel-x86_64-4.11.0-sabayon
Found linux image: /boot/kernel-genkernel-x86_64-4.8.0-sabayon
Found initrd image: /boot/initramfs-genkernel-x86_64-4.8.0-sabayon
done

 * You are currently booting with kernel:
 * kernel-genkernel-x86_64-4.8.0-sabayon
 *
 * Use 'eselect bzimage' in order to switch between the available ones


 * If you are upgrading from a previous kernel, you may be interested
 * in the following document:
 *   - General upgrade guide: https://wiki.gentoo.org/wiki/Kernel/Upgrade

 * Updating module dependencies for 4.11.0-sabayon ...
depmod: WARNING: Ignored deprecated option -r                                                                                                                                                                                               [ ok ]
 * Please report kernel bugs at:
 * http://bugs.sabayon.org
 * The source code of this kernel is located at
 * =sys-kernel/sabayon-sources-4.11.10.
 * Sabayon Linux recommends that portage users install
 * sys-kernel/sabayon-sources-4.11.10 if you want
 * to build any packages that install kernel modules
 * (such as ati-drivers, nvidia-drivers, virtualbox, etc...).
╠    ## Cleaning: sys-kernel/linux-sabayon-4.11.10
╠  +++ >>>  (2/2) sys-kernel/sabayon-sources-4.11.10
╠    ## Unpacking: sys-kernel:sabayon-sources-4.11.10.bd94789f59b1e0b33118e122a23a0164c7205b8a~0.tbz2
╠    ## Package phase: setup
 * To avoid automounting and auto(un)installing with /boot,
 * just export the DONT_MOUNT_BOOT variable.

 * Your boot partition was detected as being mounted at /boot.
 * Files will be installed there for sabayon-sources to function correctly.
 * Preparing kernel and its modules
╠    ## Package phase: preinstall
╠    ## Installing package: sys-kernel/sabayon-sources-4.11.10
╠    ## [Official Sabayon Linux Standard kernel sources]
╠    ## Updating installed packages repository: sys-kernel/sabayon-sources-4.11.10
╠    ## Package phase: preremove
╠    ## Cleaning previously installed application data.
╠    ## Package phase: postremove
╠    ## Package phase: postinstall

 * If you are upgrading from a previous kernel, you may be interested
 * in the following document:
 *   - General upgrade guide: https://wiki.gentoo.org/wiki/Kernel/Upgrade

╠    ## Cleaning: sys-kernel/sabayon-sources-4.11.10
╠  @@ Installation complete.
╠  @@ No configuration files to update.
sabayon fitzcarraldo #

Now upgrade the OS installation to the latest kernel image and reboot the machine so that the new kernel image is loaded:

sabayon fitzcarraldo # kernel-switcher switch linux-sabayon-4.11.10
sabayon fitzcarraldo # reboot

Check that the new kernel version is running:

sabayon fitzcarraldo # uname -a
Linux sabayon.local 4.11.0-sabayon #1 SMP Sat Jul 15 09:33:23 UTC 2017 x86_64 Intel(R) Pentium(R) CPU G2030 @ 3.00GHz GenuineIntel GNU/Linux
sabayon fitzcarraldo #

The minor in the version number reported by the uname command does not necessarily correspond to the minor in the version number of the kernel image package. This is not an error: see Misunderstandings about the kernel in the Sabayon Wiki regarding this apparent discrepancy.

The symlink to the kernel source code will now exist:

sabayon fitzcarraldo # eselect kernel list
Available kernel symlink targets:
  [1]   linux-4.11.0-sabayon *
sabayon fitzcarraldo #

If everything works correctly, you can uninstall the earlier version(s) of the kernel image package:

sabayon fitzcarraldo # equo search --installed linux-sabayon
sabayon fitzcarraldo # equo remove linux-sabayon-4.8.17

In my case an earlier version of the kernel source code package was not installed, but if one had been then I would have uninstalled that package too:

sabayon fitzcarraldo # equo search --installed sabayon-sources
sabayon fitzcarraldo # equo remove sabayon-sources-4.8.17

You can also remove old kernel image files manually from the /boot directory if they were not removed automatically:

sabayon fitzcarraldo # ls /boot
sabayon fitzcarraldo # rm -i /boot/*genkernel-x86_64-4.8.0-sabayon

Regenerate the file grub.cfg so that the GRUB 2 menu at boot only lists the kernel images actually present in the /boot directory:

sabayon fitzcarraldo # grub-mkconfig -o /boot/grub/grub.cfg

If you’re running Sabayon Linux inside a VirtualBox virtual machine, you’ll also need to install in the guest installation the correct version of the VirtualBox Guest Additions for the new version of the kernel you just installed in the guest installation. Uninstall the old version if it has not already been uninstalled automatically, then search for the version that corresponds to the kernel you installed in the guest installation:

sabayon fitzcarraldo # equo remove virtualbox-guest-additions
sabayon fitzcarraldo # equo search virtualbox-guest-additions
sabayon fitzcarraldo # equo install virtualbox-guest-additions-5.1.22#4.11.0-sabayon

xdotool comes to the rescue

In a previous post I explained how I implemented a method for adding my current location and the local time to my e-mail signature wherever I happen to be in the World, irrespective of the time on the laptop’s hardware clock and system clock. In that post I described how I created a keyboard shortcut using the Linux application AutoKey. Unfortunately AutoKey has not been updated for several years and no longer works properly in KDE Plasma 5 on my laptops. Therefore I decided to replace it with a KDE keyboard shortcut, and this is to explain how I did it.

First create a custom shortcut in KDE:

  1. ‘System Settings’ > ‘Shortcuts’ > ‘Custom Shortcuts’
  2. ‘Edit’ > ‘New’ > ‘Global Shortcut’ > ‘Command/URL’, and name the New Action ‘Insert current time’
  3. On the Comment pane for ‘Insert current time’, add the comment ‘Insert current time at specified location’ (without the quotes)
  4. On the Trigger pane, configure the shortcut to be Ctrl+Alt+Space
  5. On the Action pane, enter the Command/URL as ‘/home/fitzcarraldo/timezone_signature_GeoNames.sh‘ (without the quotes)
  6. Click ‘Apply’

Next modify the Bash script timezone_signature_GeoNames.sh so that it contains the following (obviously change the username and path to suit):

#!/bin/bash

place=$(kdialog --title "Current Location" --inputbox "Enter your location:")

placetime=$(perl /home/fitzcarraldo/now1.pl $place)

# xdotool does not output a space in a string, so we have to extract each field from the string
# and print each field individually, separated by a space character.

city=$(echo $placetime | awk -F "|" '{print $1}')
country=$(echo $placetime | awk -F "|" '{print $2}' | sed 's/[)(]//g')
region=$(echo $placetime | awk -F "|" '{print $4}')

datetime=$(/usr/bin/zdump $region | awk -F " " '{print $2" "$3" "$4" "$5" "$6" "$7}')
dayofweek=$(echo $datetime | awk -F " " '{print $1}')
month=$(echo $datetime | awk -F " " '{print $2}')
day=$(echo $datetime | awk -F " " '{print $3}')
time=$(echo $datetime | awk -F " " '{print $4}')
year=$(echo $datetime | awk -F " " '{print $5}')
timezone=$(echo $datetime | awk -F " " '{print $6}')

activewindow=$(xdotool getactivewindow)

xdotool type --window $activewindow "Sent from:"
for oneword in $city; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow comma
for oneword in $country; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow Return
xdotool type --window $activewindow "Local time now: "
xdotool type --window $activewindow $dayofweek
xdotool type --window $activewindow " "
xdotool type --window $activewindow $month
xdotool type --window $activewindow " "
xdotool type --window $activewindow $day
xdotool type --window $activewindow " "
xdotool type --window $activewindow $time
xdotool type --window $activewindow " "
xdotool type --window $activewindow $year
xdotool type --window $activewindow " "
if [ ${timezone:0:1} = "-" ]; then
    timezone="UTC-"${timezone#*-}
elif [ ${timezone:0:1} = "+" ]; then
    timezone="UTC+"${timezone#*+}
fi
xdotool type --window $activewindow $timezone
xdotool type --window $activewindow " "
xdotool key --window $activewindow Return
xdotool key --window $activewindow Return
echo

The Perl script now1.pl is listed in my my earlier post. Notice that the script timezone_signature_GeoNames.sh in my earlier post was much simpler. This was because the AutoKey shortcut took care of sending the text to the currently active window. Without AutoKey, I now had to do this myself in the script timezone_signature_GeoNames.sh, and the command xdotool came to the rescue. The developer explains what xdotool does as follows:

This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11’s XTEST extension and other Xlib functions.

Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

So I installed xdotool via the Gentoo package manager:

# emerge xdotool
# eix xdotool
[I] x11-misc/xdotool
     Available versions:  3.20150503.1-r1^t ~3.20160805.1^t {examples}
     Installed versions:  3.20150503.1-r1^t(22:51:30 02/04/17)(-examples)
     Homepage:            http://www.semicomplete.com/projects/xdotool/
     Description:         Simulate keyboard input and mouse activity, move and resize windows

Anyway, my Bash script using xdotool works a treat with Thunderbird (and KWrite, LibreOffice Writer, etc.). I used to experience a problem with certain characters, for example a colon was printed as a semi-colon (see the xdotool bug report xdotool writes the wrong case #121), but that no longer happens in my current KDE Plasma 5 installation:

Sent from: Galeão International Airport, Brazil
Local time now: Thu Jul 6 15:11:40 2017 UTC-03

What a useful tool xdotool is!

Stuttering audio in Linux: PulseAudio strikes again

I unmasked PulseAudio 10.0 back in January 2017 and installed it in my Gentoo Stable amd64 installation, and everything worked fine… until a couple of days ago, when the audio in streaming YouTube videos started to stutter every so often. It sounded rather like a scratched LP jumping. At first I thought the problem lay with Firefox, but the stuttering audio also occurred in Chrome. Then I wondered if my Internet connection was to blame; perhaps the ISP’s service had deteriorated. But a Windows 10 machine on my home network didn’t suffer from the problem, so that seemed to rule out the Internet connection. I tested the broadband throughput, and it was circa 32 Mbps, actually a little higher than the last time I tested it last year.

Now, Gentoo is a rolling distribution and I update my laptops regularly, but I couldn’t think what had been upgraded in the last couple of months that could be causing the problem. Although PulseAudio had not been upgraded since January, I began to wonder if PulseAudio could be involved, as my audio woes in the past have usually been due to PulseAudio.

I have always had PulseAudio installed with USE=”-realtime”:

user $ eix -I pulseaudio
[I] media-sound/pulseaudio
     Available versions:  10.0 {+X +alsa +alsa-plugin +asyncns bluetooth +caps dbus doc equalizer +gdbm +glib gnome gtk ipv6 jack libressl libsamplerate lirc native-headset neon ofono-headset +orc oss qt4 realtime selinux sox ssl system-wide systemd tcpd test +udev +webrtc-aec zeroconf ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  10.0(16:07:53 19/04/17)(X alsa alsa-plugin asyncns bluetooth caps dbus gdbm glib gnome gtk ipv6 jack orc qt4 ssl tcpd udev webrtc-aec zeroconf -doc -equalizer -libressl -libsamplerate -lirc -native-headset -neon -ofono-headset -oss -realtime -selinux -sox -system-wide -systemd -test ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://www.pulseaudio.org/
     Description:         A networked sound server with an advanced plugin system

but I wondered if PulseAudio’s real-time scheduling was somehow the cause of the problem, so I edited /etc/pulse/daemon.pa and added ‘realtime-scheduling = no‘ (I assume the default is ‘yes‘, as it was commented as such in the file):

; realtime-scheduling = yes
realtime-scheduling = no

Problem solved. PulseAudio is indeed a demon. 😡

Using the ClamAV daemon to scan files placed in my Downloads directory in Gentoo Linux

In a previous post I explained how to automatically detect files placed in my Downloads directory in Linux and scan them for viruses. The method I described in that post used clamscan, the command-line anti-virus scanner of ClamAV. Now, in addition ClamAV has a daemon (a program that runs continuously in the background), clamdscan, that you can enable. So I decided to switch to using clamdscan, as its response to downloaded files is much faster because the process waiting for new files to appear in ~/Downloads/ does not have to load clamscan from disk each time a new file arrives. Anyway, if you want to monitor a download directory in Gentoo Linux (running OpenRC) by using the ClamAV daemon — which will also download virus signature database updates automatically — then the procedure to set this up is given below.

1. Install clamav if it is not installed already:

root # emerge clamav

2. Add the service to the default runlevel:

root # rc-update add clamd default

The daemon will be launched automatically next time the computer boots.

3. The first download of the virus database has to be done manually:

root # freshclam

4. Start the daemon now:

root # rc-service clamd start

5. Create the Bash script ~/monitorDownloadsGUI with the following contents:

#!/bin/bash

DIR=$HOME/Downloads

# Get rid of old log file, if any
rm $HOME/virus-scan.log 2> /dev/null

IFS=$(echo -en "\n\b")

# Optionally, you can use shopt to avoid creating two processes due to the pipe
shopt -s lastpipe
inotifywait --quiet --monitor --event close_write,moved_to --recursive --format '%w%f' $DIR | while read FILE
# Added '--recursive' so that a directory copied into $DIR also triggers clamscan/clamdscan, although downloads
# from the Web would just be files, not directories.
do
     # Have to check file length is nonzero otherwise commands may be repeated
     if [ -s $FILE ]; then
          # Replace 'date >' with 'date >>' if you want to keep log file entries for previous scans.
          date > $HOME/virus-scan.log
          clamdscan --move=$HOME/virus-quarantine $FILE >> $HOME/virus-scan.log
          kdialog --title "Virus scan of $FILE" --msgbox "$(cat $HOME/virus-scan.log)"
     fi
done

Make it executable:

user $ chmod +x ~/monitorDownloadsGUI

6. Create the directory ~/virus-quarantine/ to store infected files pending investigation/deletion:

user $ mkdir ~/virus-quarantine

7. Install kdialog if it is not already installed:

root # emerge kdialog

8. Use ‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’ to add the script ~/monitorDownloadsGUI to the list of script files that are automatically started each time you log in to KDE.

9. Log out then back in again, and you should see that everything is running as expected:

user $ rc-status | grep clam
 clamd                                                             [  started  ]

user $ ps -ef | grep clam | grep -v grep
clamav    1920     1  0 01:48 ?        00:00:00 /usr/sbin/clamd
clamav    1929     1  0 01:48 ?        00:00:00 /usr/bin/freshclam -d

user $ ps -ef | grep GUI | grep -v grep
fitzcarraldo      9143  8971  0 13:56 ?        00:00:00 /bin/bash /home/fitzcarraldo/.config/autostart-scripts/monitorDownloadsGUI.sh

10. To test, surf to http://www.eicar.org/85-0-Download.html and download one of the EICAR test files into your ~/Downloads/ directory. You should see a pop-up KDialog window with a message similar to the following:

Virus scan of /home/fitzcarraldo/Downloads/eicarcom2.zip — KDialog

Mon 27 Feb 14:05:26 GMT 2017
/home/fitzcarraldo/Downloads/eicarcom2.zip: Eicar-Test-Signature FOUND
/home/fitzcarraldo/Downloads/eicarcom2.zip: moved to ‘/home/fitzcarraldo/virus-quarantine/eicarcom2.zip’

———– SCAN SUMMARY ———–
Infected files: 1
Time: 0.001 sec (0 m 0 s)

Note that the above-mentioned pop-up window may be preceded by one or more pop-up windows with an error message. I’m using the Chrome browser at the moment, but you may get a similar message if you are using another browser. Here is an example:

Virus scan of /home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm — KDialog ?

Mon 27 Feb 14:16:30 GMT 2017
/home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm: Access denied. ERROR

———– SCAN SUMMARY ———–
Infected files: 0
Total errors: 1
Time: 0.000 sec (0 m 0 s)

Read the error message and click ‘OK’, as this is not an actual problem; it is inotifywait detecting temporary files in the ~/Downloads/ directory during the download process. With larger files sometimes several such messages are displayed, presumably because the file being downloaded is being opened and closed more than once during the downloading process. This issue does not occur if you copy or move a file into ~/Downloads/ from another directory in your installation; try it and see for yourself. Then you only get the one pop-up window with the scan result for the file you put in ~/Downloads/.

Also have a look in ~/virus-quarantine/ and you will see the EICAR test file in that directory. You can delete it if you want (it is not infected with a real virus, so does no harm).

In future be sure to read the messages in the pop-up windows before clicking ‘OK’, as they will inform you that an infected file has been moved to the quarantine directory.

That’s all there is to it. Very simple, and quite handy if you want to check quickly that files you download don’t have a malware payload. Just make sure you download all files into ~/Downloads/ or they will not be checked automatically. Also, if you are given e.g. a USB pen drive with a file on it, you can copy the file to ~/Downloads/ if you want it to be scanned for malware.

Partitioning hard disk drives for BIOS-MBR, BIOS-GPT and UEFI-GPT in Linux

Introduction

This post was prompted by recent threads in the Gentoo Linux Forums such as the following:

  • partition MBR/GPT fdisk question‘ in which two people asked how to partition a HDD using GPT. One has a computer that only has BIOS firmware; the other has a computer with firmware that supports UEFI and BIOS (user-selectable) and who apparently wanted to boot using BIOS.
  • [SOLVED] installing on UEFI system‘ in which someone asked how to partition a HDD for a computer with UEFI firmware.

These Gentoo Linux users were confused by the instructions in the Gentoo Installation Guide – Preparing the disks. In my opinion the Gentoo Installation Guide is confusing. You do not need a BIOS boot partition (Code EF02) if you want to use a GPT-partitioned HDD with a UEFI computer; you only need a BIOS boot partition on a GPT-partitioned HDD if you want to use it with a BIOS computer. You do not need an ESP (EFI System Partition) if you want to use a GPT-partitioned HDD with a BIOS computer; you only need an ESP (Code EF00) — which must be formatted as FAT32 — if you want to use UEFI.

Coincidentally, recently an Ubuntu user also told me he finds GPT partitioning for UEFI confusing.

So, in the hope of helping people having trouble understanding how to partition a HDD for Linux, I decided to post some information on the two firmware designs and the two partitioning designs, plus list a few (of the many) partitioning schemes that can be used. The purpose of this post is to provide an overview of the designs and possible schemes, not to explain how to use the various partitioning tools or to list the precise steps you must follow. I have tried to avoid going into fine detail (for example the oft-quoted limit of 2TiB for MBR-partitioned HDDs is contigent on 512B sectors).

Firstly, let’s get the terminology straight: you should say ‘UEFI firmware’, not ‘UEFI BIOS’. The latter is an oxymoron. I know that some computer manufacturers and third-party firmware providers use the term ‘UEFI BIOS’, but it is incorrect. The ‘BIOS’ (a.k.a. ‘PC BIOS’) and the UEFI are two different designs of firmware used during the booting process. BIOS firmware and MBR partitioning are older (‘legacy’) designs; UEFI firmware and GPT partitioning are the latest designs, intended to replace BIOS and MBR. See the end of this post for links to articles regarding BIOS, MBR, UEFI and GPT that I recommend you also read.

I personally have not come across computers that support solely UEFI; all the computers I have used allow the user to configure the firmware at boot via the Setup menu to use either UEFI or BIOS. Some UEFI firmware manufacturers show the BIOS support option in the Setup menu as ‘Compatibility Support Module’ (CSM) or ‘Legacy Mode’.

‘Secure Boot’ is one of the touted UEFI features not present in BIOS. Linux expert Roderick W. Smith explains Secure Boot in his article ‘Linux on UEFI: A Quick Installation Guide‘:

One optional feature of UEFI deserves mention: Secure Boot. This feature is designed to minimize the risk of a computer becoming infected with a boot kit, which is a type of malware that infects the computer’s boot loader. Boot kits can be particularly difficult to detect and remove, which makes blocking them a priority. Microsoft requires that all desktop and laptop computers that bear a Windows 8 logo ship with Secure Boot enabled. This type of configuration complicates Linux installation, although some distributions handle this problem better than do others. Do not confuse Secure Boot with EFI or UEFI, though; it’s possible for an EFI computer to not support Secure Boot, and it’s possible to disable Secure Boot even on x86-64 EFI computers that support it. Microsoft requires that users can disable Secure Boot for Windows 8 certification on x86 and x86-64 computers; however, this requirement is reversed for ARM computers—such computers that ship with Windows 8 must not permit the user to disable Secure Boot. Fortunately, ARM-based Windows 8 computers are currently rare. I recommend avoiding them.

GPT-partitioned HDDs have become the norm these days for two principal reasons:

  • the MBR design limits the amount of disk space accessible to a maximum of 2TiB but HDDs larger than 2TiB are now common;
  • Microsoft has standardised on UEFI, and UEFI will not boot an MBR-partitioned HDD (Section 5.2.1 of Version 2.6 of the UEFI Specification specifies that UEFI firmware shall not execute the boot code in an MBR located at the first logical block of a disk with the MBR disk layout).

The ‘Protective MBR’ at the beginning of every GPT-partitioned disk (in the same location on the disk as a legacy MBR would be) is designed to prevent MBR-based disk utilities misrecognising and possibly overwriting GPT-partitioned disks. A fake (i.e. it does not really exist) single partition called the ‘GPT Protective Partition’ (Code EE00) is specified in the Protective MBR to occupy as much of the drive as can be represented in an MBR, namely a maximum of 2TiB in the case of a disk with 512B sectors. Operating systems and tools not designed for GPT disks will read the Protective MBR and detect that the disk contains a single partition of unknown type and with no empty space, and will refuse to modify the disk unless the user deletes this partition. This design (i.e. the Protective MBR and GPT Protective Partition) was devised in order to minimise the possibility of a) legacy software accidentally overwriting a GPT-partitioned HDD; b) GPT-aware software accidentally overwriting an MBR-partitioned HDD (the absence of a partition of type EEh defined in the Protective MBR would indicate to GPT-aware operating systems and tools that the HDD is not GPT-partitioned).

You can use an MBR-partitioned HDD with BIOS; you can use a GPT-partitioned HDD with BIOS; you can use a GPT-partitioned HDD with UEFI; you cannot use an MBR-partitioned HDD with UEFI. If the firmware in your computer has an option to select BIOS mode (some firmware manufacturers refer to this as ‘Compatibility Support Module’ or ‘Legacy Mode’) instead of UEFI and you want to use an MBR on the HDD, you will have to use BIOS. In summary, for Linux your options are BIOS-MBR, BIOS-GPT or UEFI-GPT. I will discuss these three options and provide some possible partitioning schemes in each case.

The layout of a GPT-partitioned HDD is as follows:

The beginning of a GPT-partitioned disk
Protective MBR 512B
Primary GPT Header 512B
Primary Partition Table entries Up to 16KiB (maximum of 128 partitions)

Therefore the unpartitioned space at the beginning of the HDD should be at least 17KiB.

Unlike in the MBR design, the end of a GPT-partitioned disk stores a backup of the partition table:

The end of a GPT-partitioned disk
Secondary Partition Table entries Up to 16KiB (maximum of 128 partitions)
Secondary GPT Header 512B

(The Secondary GPT Header really does come after the Secondary Partition Table entries.)

Therefore the upartitioned space at end of a GPT-partitioned disk should be at least 16.5KiB.

The partitions themselves are located between the Primary GPT and the Secondary GPT, i.e. between the above two tables.

The remainder of a GPT-partitioned disk between the above two areas can contain up to 128 partitions. One partition — normally the first partition is used — must be the ‘ESP’ (EFI System Partition), which should be formatted as FAT32 and have the ‘esp’ and ‘boot’ flags set. The partition code must be EF02. The minimum possible size of a FAT32 partition is 33,549,824B (~32MiB). The required size of the ESP will depend on what is stored in it (for example the now-orphaned ELILO boot loader stored the Linux kernel images in it, whereas the GRUB boot loader does not). The Ubuntu Community Help Wiki specifies a minimum of 100MiB and recommends 200MiB. The Gentoo Installation Guide recommends 128MB.

The versions of GParted and KDE Partition Manager I have used to partition HDDs seem to want to reserve at least 2048 512B sectors of empty space before the first partition (sometimes these two utilities force me to have 4096 512B sectors of empty space before the first partition), so I suggest leaving at least 1MiB of unpartitioned space at the beginning and end of the HDD if you want to partition a disk for GPT using those tools. Actually, for peace of mind you may as well leave 1MiB of empty space at the beginning and end of the disk whatever GUI tools or console commands (parted, fdisk, gdisk, etc.) you use to partition the disk.

By the way, if you wanted to use MBR instead of GPT, the MBR (512B) plus the GRUB embedding area after the MBR (~31KiB) means that 1MiB would also be more than enough for an MBR HDD (see GNU GRUB Manual – 3.4 BIOS installation).

Thus my suggestions for HDD partitioning would be as shown below. Note that the order of the partitions I have adopted below is not obligatory; you can change the order if you wish. I prefer to specify precise sizes for the swap and root partitions, and put /home on the last partition so it can occupy the remainder of the disk, whatever size that may be.

UEFI-GPT

Option 1

This is my preferred option because I can edit /etc/fstab and specify that /boot must not be mounted at boot, thus reducing the possibility of the files in /boot getting corrupted.

UEFI-GPT Option 1
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 512 MiB
File system: FAT32
Flags: boot & esp
Code: EF00
Label: ESP
Mount point: /boot/efi
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda3
Size: 512 MiB
File system: ext2
Flags: None
Code: 8300
Label: BOOT
Mount point: /boot Therefore /boot/grub/ will be on this partition if you use GRUB.
/dev/sda4
Size: e.g. 64 GiB (128 GiB if the drive is big)
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root)
/dev/sda5
Size: 1.00 MiB less than the remaining disk space
File system: ext4
Flags: None
Code: 8300
Label: HOME
Mount point: /home
1.00 MiB of empty space at the end of the disk.

* You will often see recommendations to make the size of the swap partition the same as the size of the RAM if you want to be able to put the computer into hibernation. In fact, the Linux kernel is normally configured to compress the contents of the RAM image for hibernation, and I personally have seen the disk image of 4GB of RAM compressed to 23% of that size. Nevertheless, if you have a large HDD you may as well just take the easy route and allocate the size of the swap partition to be the same as the RAM size, even if all of the swap partition will never be used in practice. That way you are guaranteed to be able to put the computer into hibernation.

Below is an example of the above partitioning scheme on a virtual UEFI machine with a 64GiB virtual GPT-partitioned HDD:

root # gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 134217728 sectors, 64.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 54B3C38F-1C55-4A19-9BAA-499C4D0D8DD0
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 134217694
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   EF00
   2         1050624         5244927   2.0 GiB     8200
   3         5244928         6293503   512.0 MiB   8300
   4         6293504        72353791   31.5 GiB    8300
   5        72353792       134215679   29.5 GiB    8300

root # fdisk -l /dev/sda
Disk /dev/sda: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 54B3C38F-1C55-4A19-9BAA-499C4D0D8DD0

Device        Start       End  Sectors  Size Type
/dev/sda1      2048   1050623  1048576  512M EFI System
/dev/sda2   1050624   5244927  4194304    2G Linux swap
/dev/sda3   5244928   6293503  1048576  512M Linux filesystem
/dev/sda4   6293504  72353791 66060288 31.5G Linux filesystem
/dev/sda5  72353792 134215679 61861888 29.5G Linux filesystem

root # parted /dev/sda print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  538MB   537MB   fat32                 boot, esp
 2      538MB   2685MB  2147MB  linux-swap(v1)
 3      2685MB  3222MB  537MB   ext2
 4      3222MB  37.0GB  33.8GB  ext4
 5      37.0GB  68.7GB  31.7GB  ext4

root # blkid
/dev/sda4: LABEL="ROOT" UUID="174ac3e8-f105-4606-bed1-7a1aa22c3631" TYPE="ext4" PARTUUID="01d9c139-fe70-415a-abc6-2351fad33384"
/dev/sda1: UUID="B4C1-7EA5" TYPE="vfat" PARTUUID="d941f728-c386-4f4c-b0c3-aa76f4290774"
/dev/sda2: LABEL="SWAP" UUID="e3ddf9b5-2ae3-4469-a121-0a1a78aa6702" TYPE="swap" PARTUUID="a4daec88-da44-4ae3-8119-01cc81325f03"
/dev/sda3: LABEL="BOOT" UUID="1e24ea9d-5358-4e9b-8667-d7a42e7b6ad7" TYPE="ext2" PARTUUID="b5369ce3-4b44-4d19-be6f-1d226dc71cb3"
/dev/sda5: LABEL="HOME" UUID="87f6a0af-dbed-4587-b810-efca8f269618" TYPE="ext4" PARTUUID="19fd7d00-2d89-4653-af03-e81618a3b70d"

Option 2

You could use this scheme if you are not interested in having /boot on its own partition.

UEFI-GPT Option 2
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 512 MiB
File system: FAT32
Flags: boot & esp
Code: EF00
Label: ESP
Mount point: /boot/efi
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda3
Size: e.g. 64 GiB (128 GiB if the drive is big)
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root) Therefore /boot (and /boot/grub/) will on this partition too.
/dev/sda4
Size: 1.00 MiB less than the remaining space on the disk
File system: ext4
Flags: None
Code: 8300
Label: HOME
Mount point: /home
1.00 MiB of empty space at the end of the disk.

* See my earlier note regarding hibernation.

Below is an example of the above scheme on a virtual UEFI machine with a 64GiB virtual GPT-partitioned HDD:

root # gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 134217728 sectors, 64.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 54B3C38F-1C55-4A19-9BAA-499C4D0D8DD0
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 134217694
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   EF00
   2         1050624         5244927   2.0 GiB     8200
   3         5244928        72353791   32.0 GiB    8300
   4        72353792       134215679   29.5 GiB    8300

root # fdisk -l /dev/sda
Disk /dev/sda: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 54B3C38F-1C55-4A19-9BAA-499C4D0D8DD0

Device        Start       End  Sectors  Size Type
/dev/sda1      2048   1050623  1048576  512M EFI System
/dev/sda2   1050624   5244927  4194304    2G Linux swap
/dev/sda3   5244928  72353791 67108864   32G Linux filesystem
/dev/sda4  72353792 134215679 61861888 29.5G Linux filesystem

root # parted /dev/sda print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system     Name  Flags
 1      1049kB  538MB   537MB   fat32                 boot, esp
 2      538MB   2685MB  2147MB  linux-swap(v1)
 3      2685MB  37.0GB  34.4GB  ext4
 4      37.0GB  68.7GB  31.7GB  ext4


root # blkid
/dev/sda3: LABEL="ROOT" UUID="fdf2b11a-8c6b-4bb3-9534-477c3ed49d95" TYPE="ext4" PARTUUID="f393129f-ab32-40fb-bf78-3aead3dd4af0"
/dev/sda1: UUID="C024-8A30" TYPE="vfat" PARTUUID="d941f728-c386-4f4c-b0c3-aa76f4290774"
/dev/sda2: LABEL="SWAP" UUID="1f752a05-a1fb-4c5f-ab2e-079715207b4d" TYPE="swap" PARTUUID="a4daec88-da44-4ae3-8119-01cc81325f03"
/dev/sda4: LABEL="HOME" UUID="041e4ab2-d54c-4092-b445-779997ac09ce" TYPE="ext4" PARTUUID="7e1b8dc0-2f38-4260-95af-fbb80bb72156"

Option 3

You could use this scheme if you are not interested in having /boot and /home on their own partitions.

UEFI-GPT Option 3
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 512 MiB
File system: FAT32
Flags: boot & esp
Code: EF00
Label: ESP
Mount point: /boot/efi
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda3
Size: 1.00 MiB less than the remaining space on the disk
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root) Therefore /boot (and /boot/grub/) and /home will on this partition too.
1.00 MiB of empty space at the end of the disk.

* See my earlier note regarding hibernation.

BIOS-GPT

If you have partitioned a large drive correctly then a computer with BIOS firmware will be able to access partitions larger than 2TiB. One of my computers has BIOS firmware only but can access its 3TiB GPT-partitioned HDDs.

Option 1

This is my preferred option because I can edit /etc/fstab and specify that /boot must not be mounted at boot, thus reducing the possibility of the files in /boot getting corrupted.

BIOS-GPT Option 1
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1 This is the BIOS boot partition
Size: 1.00 MiB
File system: Unformatted
Flags: bios_grub
Code: EF02
Label: Not applicable
Mount point: Not applicable
/dev/sda2
Size: 512 MiB
File system: ext2
Flags: None
Code: 8300
Label: BOOT
Mount point: /boot
/dev/sda3
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda4
Size: e.g. 64 GiB (128 GiB if the drive is big)
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root) Therefore /boot and (/boot/grub/) will on this partition too.
/dev/sda5
Size: 1.00 MiB less than the remaining space on the disk
File system: ext4
Flags: None
Code: 8300
Label: HOME
Mount point: /home
1.00 MiB of empty space at the end of the disk.

* See my earlier note regarding hibernation.

Option 2

You could use this scheme if you are not interested in having /boot on its own partition.

BIOS-GPT Option 2
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1 This is the BIOS boot partition
Size: 1.00 MiB
File system: Unformatted
Flags: bios_grub
Code: EF02
Label: Not applicable
Mount point: Not applicable
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda3
Size: e.g. 64 GiB (128 GiB if the drive is big)
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root) Therefore /boot (and /boot/grub/) will on this partition too.
/dev/sda4
Size: 1.00 MiB less than the remaining space on the disk
File system: ext4
Flags: None
Code: 8300
Label: HOME
Mount point: /home
1.00 MiB of empty space at the end of the disk.

* See my earlier note regarding hibernation.

Option 3

You could use this scheme if you are not interested in having /boot and /home on their own partitions.

BIOS-GPT Option 3
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1 This is the BIOS boot partition
Size: 1.00 MiB
File system: Unformatted
Flags: bios_grub
Code: EF02
Label: Not applicable
Mount point: Not applicable
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
File system: linux-swap
Flags: None
Code: 8200
Label: SWAP
Mount point: None
/dev/sda3
Size: 1.00 MiB less than the remaining space on the disk
File system: ext4
Flags: None
Code: 8300
Label: ROOT
Mount point: / (root) Therefore /boot and /home will on this partition too.
1.00 MiB of empty space at the end of the disk.

* See my earlier note regarding hibernation.

BIOS-MBR

Option 1

This is my preferred option because I can edit /etc/fstab and specify that /boot must not be mounted at boot, thus reducing the possibility of the files in /boot getting corrupted.

BIOS-MBR Option 1
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 512 MiB
Type: Primary
File system: ext2
Flags: boot
Label: BOOT
Mount point: /boot
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
Type: Primary
File system: linux-swap
Flags: None
Label: SWAP
Mount point: None
/dev/sda3
Size: e.g. 64 GiB (128 GiB if the drive is big)
File system: ext4
Flags: None
Label: ROOT
Mount point: / (root)
/dev/sda4
Size: remaining space on the disk
Type: Primary
File system: ext4
Flags: None
Label: HOME
Mount point: /home

* See my earlier note regarding hibernation.

Option 2

You could use this scheme if you are not interested in having /boot on its own partition.

BIOS-MBR Option 2
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 16 GiB for a computer with 16 GiB of RAM*
Type: Primary
File system: linux-swap
Flags: None
Label: SWAP
Mount point: None
/dev/sda2
Size: e.g. 64 GiB (128 GiB if the drive is big)
Type: Primary
File system: ext4
Flags: boot
Label: ROOT
Mount point: / (root) Therefore /boot will be on this partition too.
/dev/sda3
Size: remaining space on the disk
Type: Primary
File system: ext4
Flags: None
Label: HOME
Mount point: /home

* See my earlier note regarding hibernation.

Option 3

You could use this scheme if you are not interested in having /boot and /home on their own partitions.

BIOS-MBR Option 3
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 16 GiB for a computer with 16 GiB of RAM*
Type: Primary
File system: linux-swap
Flags: None
Label: SWAP
Mount point: None
/dev/sda2
Size: remaining space on the disk
Type: Primary
File system: ext4
Flags: boot
Label: ROOT
Mount point: / (root) Therefore /boot and /home will be on this partition too.

* See my earlier note regarding hibernation.

Option 4

If you want to have more than four partitions — let’s say you wanted to have a separate NTFS partition, for example — you would need to use an Extended Partition.

BIOS-MBR Option 4
1.00 MiB of empty space at the beginning of the disk.
/dev/sda1
Size: 512 MiB
Type: Primary
File system: ext2
Flags: boot
Label: BOOT
Mount point: /boot
/dev/sda2
Size: 16 GiB for a computer with 16 GiB of RAM*
Type: Primary
File system: linux-swap
Flags: None
Label: SWAP
Mount point: None
/dev/sda3
Size: Remainder of disk
Type: Extended
File system: Not applicable
Flags: None
Label: Not applicable
Mount point: Not applicable
/dev/sda4
Will not exist
/dev/sda5
Size: e.g. 128GiB
Type: Logical
File system: ext4
Flags: None
Label: ROOT
Mount point: / (root)
/dev/sda6
Size: e.g. 256GiB
Type: Logical
File system: ext4
Flags: None
Label: HOME
Mount point: /home
/dev/sda7
Size: remaining space on the disk
Type: Logical
File system: NTFS
Flags: None
Label: NTFS
Mount point: /media/NTFS

* See my earlier note regarding hibernation.

Below is an example of the above scheme (this happens to be the scheme on my main laptop):

root # fdisk -l /dev/sda
Disk /dev/sda: 698.7 GiB, 750156374016 bytes, 1465149168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x291ba0e7

Device     Boot     Start        End    Sectors   Size Id Type
/dev/sda1            2048     264191     262144   128M 83 Linux
/dev/sda2          264192   33822719   33558528    16G 82 Linux swap / Solaris
/dev/sda3        33822720 1465147391 1431324672 682.5G  5 Extended
/dev/sda5        33824768  302260223  268435456   128G 83 Linux
/dev/sda6       302262272  839133183  536870912   256G 83 Linux
/dev/sda7       839135232 1465147391  626012160 298.5G  7 HPFS/NTFS/exFAT

root # lsblk -o NAME,TYPE,SIZE,FSTYPE,MOUNTPOINT,LABEL,PARTFLAGS /dev/sda
NAME   TYPE   SIZE FSTYPE  MOUNTPOINT  LABEL PARTFLAGS
sda    disk 698.7G                           
├─sda1 part   128M ext2    /boot       BOOT  
├─sda2 part    16G swap    [SWAP]      SWAP  
├─sda5 part   128G ext4    /           ROOT  
├─sda6 part   256G ext4    /home       HOME  
└─sda7 part 298.5G ntfs-3g /media/NTFS NTFS

Notice that the boot flag is not set. Nevertheless the laptop boots fine.

Command-line tools

Below are examples of command-line utilities parted, gdisk and fdisk examining a GPT-partitioned HDD and an MBR-partitioned HDD. The original fdisk utility predates the invention of GPT, but the latest versions of fdisk understand the GPT design (you can check this by using the command ‘man fdisk‘). Personally, to partition GPT HDDs from the command line I would use parted or gdisk before fdisk, and to partition MBR HDDs from the command line I would use parted or fdisk before gdisk. Mind you, I like an easy life and so I tend to use the GUI tools GParted or KDE Partition Manager to partition and format an HDD for Linux.

GPT-partitioned HDD

root # parted /dev/sda print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name       Flags
 1      2097kB  539MB   537MB   fat32                      boot, esp
 2      539MB   1076MB  537MB   ext2            /boot
 3      1076MB  3223MB  2147MB  linux-swap(v1)  linuxswap
 4      3223MB  37.6GB  34.4GB  ext4            /
 5      37.6GB  68.7GB  31.1GB  ext4            /home

root # gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 134217728 sectors, 64.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 9807AF0F-8BD5-4727-A3CD-9995B2705732
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 134217694
Partitions will be aligned on 2048-sector boundaries
Total free space is 8125 sectors (4.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            4096         1052671   512.0 MiB   EF00  
   2         1052672         2101247   512.0 MiB   8300  /boot                                                                    
   3         2101248         6295551   2.0 GiB     8200  linuxswap                                                                
   4         6295552        73404415   32.0 GiB    8300  /                                                                        
   5        73404416       134213631   29.0 GiB    8300  /home                                                                    
root # fdisk -l /dev/sda                                                                                           
Disk /dev/sda: 64 GiB, 68719476736 bytes, 134217728 sectors                                                                       
Units: sectors of 1 * 512 = 512 bytes                                                                                             
Sector size (logical/physical): 512 bytes / 512 bytes                                                                             
I/O size (minimum/optimal): 512 bytes / 512 bytes                                                                                 
Disklabel type: gpt                                                                                                               
Disk identifier: 9807AF0F-8BD5-4727-A3CD-9995B2705732                                                                             
                                                                                                                                  
Device        Start       End  Sectors  Size Type                                                                                 
/dev/sda1      4096   1052671  1048576  512M EFI System
/dev/sda2   1052672   2101247  1048576  512M Linux filesystem
/dev/sda3   2101248   6295551  4194304    2G Linux swap
/dev/sda4   6295552  73404415 67108864   32G Linux filesystem
/dev/sda5  73404416 134213631 60809216   29G Linux filesystem

MBR-partitioned HDD

root # parted /dev/sda print
Model: ATA VBOX HARDDISK (scsi)
Disk /dev/sda: 68.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Disk Flags: 

Number  Start   End     Size    Type     File system     Flags
 1      1049kB  538MB   537MB   primary  ext2            boot
 2      538MB   2685MB  2147MB  primary  linux-swap(v1)
 3      2685MB  37.0GB  34.4GB  primary  ext4
 4      37.0GB  68.7GB  31.7GB  primary  ext4

root # gdisk -l /dev/sda
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. 
***************************************************************


Warning! Secondary partition table overlaps the last partition by
33 blocks!
You will need to delete this partition or resize it in another utility.
Disk /dev/sda: 134217728 sectors, 64.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): F6E9E53E-33BE-44CB-BEFC-D93E03B79B84                                                   
Partition table holds up to 128 entries                                                                        
First usable sector is 34, last usable sector is 134217694                                                     
Partitions will be aligned on 2048-sector boundaries                                                           
Total free space is 2014 sectors (1007.0 KiB)                                                                  

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   8300  Linux filesystem
   2         1050624         5244927   2.0 GiB     8200  Linux swap
   3         5244928        72353791   32.0 GiB    8300  Linux filesystem
   4        72353792       134217727   29.5 GiB    8300  Linux filesystem
root # fdisk -l /dev/sda
Disk /dev/sda: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7b9f1623

Device     Boot    Start       End  Sectors  Size Id Type
/dev/sda1  *        2048   1050623  1048576  512M 83 Linux
/dev/sda2        1050624   5244927  4194304    2G 82 Linux swap / Solaris
/dev/sda3        5244928  72353791 67108864   32G 83 Linux
/dev/sda4       72353792 134217727 61863936 29.5G 83 Linux

Notice the gdisk warning that the last partition would not be viable on a GPT-partitioned HDD because there would not be space for the Secondary GPT. However, as this is an MBR-partitioned HDD for use with a BIOS firmware computer, that does not apply.

A warning note if installing Ubuntu on a UEFI computer

When I installed Ubuntu Desktop 16.04.1 in a virtual machine I encountered a bug (in VirtualBox? in the Ubuntu Installer?) which results in subsequent reboots leaving the virtual machine running the UEFI Shell instead of launching GRUB and booting Ubuntu. To fix that, I did the following as soon as Ubuntu booted from the virtual HDD for the first time after I had clicked ‘Restart Now’ in the ‘Installation Complete’ window:

user $ sudo su
root # mount /dev/sda1 /mnt
root # cd /mnt
root # echo "\EFI\ubuntu\grubx64.efi" > startup.nsh
root # cd
root # umount /dev/sda1
root # exit

The partition /dev/sda1 being the FAT32 ESP in this case.

This creates the file /boot/efi/startup.nsh required by the UEFI Shell. Then you can reboot and all should work as intended, i.e. the UEFI machine boots to the UEFI Shell which, after a few seconds, launches GRUB.

UPDATE (16 February 2017): A downside to the above approach is the 5-second timeout in the UEFI Shell until startup.nsh is launched. Below is an alternative that avoids having to wait for the 5-second countdown in the UEFI Shell until startup.nsh is launched. The downside is that you must remember to repeat this procedure if a new version of GRUB is installed and there is a new version of /EFI/ubuntu/grubx64.efi.

user $ sudo su
root # mount /dev/sda1 /mnt
root # cd /mnt/EFI/
root # ls
ubuntu
root # ls ubuntu
fw  fwupx64.efi  grub.cfg  grubx64.efi  MokManager.efi  shimx64.efi
root # mkdir BOOT
root # cp ubuntu/grubx64.efi BOOT/BOOTX64.EFI
root # cd
root # umount /dev/sda1
root # exit

See the Arch Linux Wiki article VirtualBox – 2.1 Installation in EFI mode.

A warning note if installing Gentoo on a UEFI computer

To date, the Gentoo Minimal Installation CD does not support UEFI, and therefore I recommend that you use SystemRescueCd instead. It is a Gentoo-based LiveCD with several tools useful for installing Gentoo, including a Web browser so you can access the on-line Gentoo documentation during installation. You can follow the Gentoo Installation Guide verbatim. SystemRescueCd also has a Desktop Environment with various GUI utilities, so you could partition the HDD using GParted instead of the command-line utilities if you wish. SystemRescueCd also comes with various wireless network card drivers and a network manager, so there is a good chance you will be able to connect easily to a wireless network if you prefer (I once used SystemRescueCd to install Gentoo on a laptop using Wi-Fi alone, as a wired connection was not convenient at the time).

How can I find out in Linux if a computer has booted using UEFI?

Check if the RAM-based directory /sys/firmware/efi/ exists. If it does not exist, the computer did not boot using UEFI.

Further Reading

  1. Wikipedia – BIOS
  2. Wikipedia – Master boot record
  3. UEFI Specification Version 2.6, January 2016
  4. Wikipedia – Unified Extensible Firmware Interface
  5. Wikipedia – GUID Partition Table
    Also, the diagram GUID Partition Table Scheme in that article is quite helpful in understanding the layout of a GPT-partitioned HDD.
  6. Wikipedia – BIOS boot partition
    Also, the diagram GNU GRUB 2 in that article is quite helpful in understanding: a) MBR partitioning; b) how GRUB 2 fits on an MBR-partitioned HDD; c) GPT partitioning; d) how GRUB 2 fits on a GPT-partitioned HDD; e) where the BIOS boot partition fits on a GPT-partitioned HDD and how it is used by GRUB 2.
  7. GNU GRUB Manual – 3.4 BIOS installation
  8. Ubuntu Documentation – UEFI
  9. Gentoo Installation Guide – Preparing the disks
  10. Ubuntu Bug Reports – Bug No. 811485 – Ubuntu partman-efi package – EFI SYSTEM PARTITION should be atleast 100 MiB size and formatted as FAT32, not FAT16
  11. Roderick W. Smith’s Web Page