Automatically clearing the /usr/tmp/portage directory in Gentoo Linux

Gentoo Linux has been in use for nine years on one of my old laptops. A couple of days ago I performed the usual rolling update of the installation, but the latest version of a large package that normally takes several hours to compile failed to compile due to a lack of disk space. Sure enough, the command ‘df -h‘ showed me that the root partition was full. After a little digging I discovered that the directory /usr/tmp/portage/ contained a whopping 30GB of directories and files.

Portage uses the directory /usr/tmp/portage/ as a temporary store for the package source code when merging a package. The temporary files are not deleted if a merge fails, but the emerge command should delete them on the next merge of that package. On the other hand the ebuild command does not delete the temporary files, although normally you only use the ebuild command if you are creating a manifest.

Anyway, in the nine years that Gentoo Linux has been installed on the laptop I had never bothered to check that /usr/tmp/portage/ was actually empty, and its contents had slowly increased. The cure to my immediate problem was simply to empty the directory:

root # rm -rf /usr/tmp/portage/*

I doubt the laptop would still be working by the time /usr/tmp/portage would become that full again, but the situation got me thinking: What if I were to create a script to delete the temporary directories and files in /usr/tmp/portage/ at shutdown?

Gentoo Linux on this laptop uses OpenRC so I simply created a file /etc/local.d/99delete_tmp_files_from_failed_merges.stop containing the following code:

#!/bin/bash
# If root partition is more than 90% full, delete any temporary directories and
# files that were left in /var/tmp/portage/ instead of being deleted.
#
# The root partition is on /dev/sda6 and the emerge command must not be running.
#
if [ `pgrep -c emerge` -eq 0 ] && [ `df | awk '/sda6/ {print $5}' | awk -F% '{print $1}'` -gt 90 ]; then
    rm -rf /usr/tmp/portage/*
fi

I made the script executable:

root # chmod +x /etc/local.d/99delete_tmp_files_from_failed_merges.stop

Now, if the root partition is more than 90% full when I shut down the laptop, the script will automatically empty that directory. One less thing to think about.

Advertisements

HEIC image files in Linux

I was at an event recently where the attendees were asked to upload their camera and smartphone photos and videos to a shared Google Drive folder. Some of the uploaded photo files have a .HEIC (High Efficiency Image Container) extension, which I had not come across before. I have since learnt that these HEIC files were produced by iPhones running iOS 11, encoded using the HEIF (High Efficiency Image File) format. Apparently the HEIF format is superior to the JPEG format in a number of ways (see the links at the end of my post, especially the image examples given by Nokia), although it is subject to patents and therefore I believe there are certain constraints to coding image files in HEIC format. Anyway, I’ll leave you to read the fine print. My interest was simply because I wanted to be able to download the above-mentioned photo files and view them all in the file managers and image-viewing applications in Linux and Android on my various devices.

Now, I can browse and view the above-mentioned shared HEIC images in Google Drive in the Firefox and Chrome browsers in Linux, although an ownCloud site viewed using the same browsers displays the HEIC files as grey icons that can only be downloaded, not opened and viewed in the browser. I also found that Cirrus, the Android app for ownCloud that I use on my Galaxy Note 8 phone, cannot display HEIC photos either.

I downloaded the HEIC files to a machine running Lubuntu 18.04 and to a machine running Gentoo Linux. The file manager PCManFM in Lubuntu 18.04 displays grey icons rather than thumbnails for these HEIC files, and KDE’s Dolphin 18.08.3 file manager in Gentoo Linux displays green image icons rather than thumbnails for them. As far as Linux image viewers go, in Lubuntu 18.04 I find that GPicView 0.2.5 and Geeqie 1.4 cannot display HEIC images, and in Gentoo Linux KDE I find that GQview 2.1.5-r1, Okular 18.08.3 and Gwenview 18.08.3 cannot display HEIC images. So I set about converting all the HEIC files to JPG files. I managed to do this but needed to use a range of tools, as illustrated by a couple of examples below for Lubuntu 18.04 and Gentoo Linux. This post might seem long-winded but perhaps may be of help to Linux users coming across .HEIC files for the first time.

From the .HEIC files I had downloaded I picked one at random to try and convert to a JPG file: IMG_3706.HEIC. Its EXIF data confirms it is an HEIC file:

user $ exiftool IMG_3706.HEIC | grep "File Type"
File Type                       : HEIC
File Type Extension             : heic
user $ exiftool IMG_3706.HEIC | grep "Camera Model"
Camera Model Name               : iPhone 7 Plus

Several of the files with the .HEIC suffix that I downloaded were not real HEIC files according to their EXIF data:

user $ exiftool IMG_9474.HEIC | grep "File Type"
File Type                       : JPEG
File Type Extension             : jpg
user $ exiftool IMG_9474.HEIC | grep "Camera Model"
Camera Model Name               : iPhone 8

Those files were apparently treated as JPEG files by the tools I mention below, so I have omitted the results for those ‘false’ HEIC files.

Lubuntu 18.04

1. I installed the libheif example tools:

user $ sudo apt install libheif-examples

2. I used the heif-info command to check the file:

user $ heif-info IMG_3706.HEIC 
image: 3024x4032 (id=49), primary
  thumbnail: 240x320
  alpha channel: no
  depth channel: no

3. I tried to convert the file using the heif-convert command:

user $ heif-convert IMG_3706.HEIC IMG_3706.jpg
File contains 1 images
Written to IMG_3706.jpg

4. Apparently Imagemagick >=7.0.7-22 compiled with --with-libheif is supposed to be able to convert HEIC files to JPG. Anyway, I tried to convert the file using the current version of Imagemagick in Lubuntu 18.04 (the current package version is 8:6.9.7.4+dfsg-16ubuntu6.4):

user $ convert IMG_3706.HEIC IMG_3706a.jpg
convert-im6.q16: no decode delegate for this image format `HEIC' @ error/constitute.c/ReadImage/504.
convert-im6.q16: no images defined `IMG_3706a.jpg' @ error/convert.c/ConvertImageCommand/3258.

5. Apparently the GIMP >=2.10.2 supports HEIF by using heif-gimp-plugin. Anyway, I tried to open the file with the current version of the GIMP in Lubuntu 18.04 (the current package version is 2.8.22-1). The GIMP launches and pops-up a window with the title ‘GIMP Message’ containing the following message and an ‘OK’ button:

GIMP Message
Opening /home/fitzcarraldo/IMG_3706.HEIC’ failed: Unknown file type

6. I used the online tool ‘libheif decoder demo’ (https://strukturag.github.io/libheif/) in a browser window. This can load the file IMG_3706.HEIC (‘Browse…’ button) and convert it (‘Save image…’ button) to IMG_3706.jpeg.

Gentoo Linux with KDE 5

1. I installed the libheif example tools implicitly by re-merging Imagemagick with USE="heif", which installs libheif.

root # cat /etc/portage/package.use/imagemagick 
media-gfx/imagemagick heif
root # emerge imagemagick

2. I used the heif-info command to check the file:

user $ heif-info IMG_3706.HEIC  
image: 3024x4032 (id=49), primary
  thumbnail: 240x320
  alpha channel: no
  depth channel: no

3. I tried to convert the file using the heif-convert command:

user $ heif-convert IMG_3706.HEIC IMG_3706.jpg
File contains 1 images
Written to IMG_3706.jpg

4. I tried to convert the file using Imagemagick >=7.0.7-22 compiled with --with-libheif (Imagemagick merged with USE="heif"):

user $ convert IMG_3706.HEIC IMG_3706a.jpg
user $

So Imagemagick 7.0.8.16 in Gentoo has no trouble with the file IMG_3706.HEIC.

5. I tried to open the file with the GIMP >=2.10.2, which supports HEIF using heif-gimp-plugin (GIMP >=2.10.6-r1 with USE="heif" in the case of Gentoo Linux)

First I re-merged the GIMP with the heif USE flag:

root # cat /etc/portage/package.accept_keywords/gimp
=media-gfx/gimp-2.10.8-r1 ~amd64
# required by media-gfx/gimp-2.10.8-r1::gentoo
=media-libs/libmypaint-1.3.0 ~amd64
# required by media-gfx/gimp-2.10.8-r1::gentoo
=media-gfx/mypaint-brushes-1.3.0-r1 ~amd64
# required by media-gfx/gimp-2.10.8-r1::gentoo
=media-libs/gegl-0.4.12 ~amd64
# required by media-gfx/gimp-2.10.8-r1::gentoo
=media-libs/babl-0.1.60 ~amd64
root # cat /etc/portage/package.use/gimp
media-gfx/gimp heif
root # emerge -1vp gimp

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ~] media-gfx/mypaint-brushes-1.3.0-r1:1.0::gentoo  2,390 KiB
[ebuild     U ~] media-libs/babl-0.1.60::gentoo [0.1.38::gentoo] USE="(-altivec)" CPU_FLAGS_X86="mmx sse sse2 sse3%* sse4_1 -f16c" 670 KiB
[ebuild  N     ] media-libs/gexiv2-0.10.8::gentoo  USE="-introspection -python -static-libs -test -vala" PYTHON_TARGETS="python2_7 python3_6 -python3_4 -python3_5" 620 KiB
[ebuild  NS   ~] media-libs/gegl-0.4.12:0.4::gentoo [0.2.0-r5:0::gentoo] USE="cairo ffmpeg introspection lcms openexr sdl svg tiff v4l -debug -jpeg2k -lensfun -libav -raw -test -umfpack -vala -webp" CPU_FLAGS_X86="mmx sse" 6,900 KiB
[ebuild  NS    ] media-libs/gegl-0.3.26:0.3::gentoo [0.2.0-r5:0::gentoo] USE="cairo ffmpeg introspection lcms openexr sdl svg tiff v4l -debug -jpeg2k -lensfun -raw -test -umfpack -vala -webp" CPU_FLAGS_X86="mmx sse" 6,378 KiB
[ebuild  N    ~] media-libs/libmypaint-1.3.0::gentoo  USE="gegl nls openmp -introspection" 428 KiB
[ebuild     U ~] media-gfx/gimp-2.10.8-r1:2::gentoo [2.8.22-r1:2::gentoo] USE="alsa heif%* mng openexr%* udev wmf -aalib (-altivec) (-aqua) -debug -doc -gnome -jpeg2k -postscript -python -smp -test -unwind% -vector-icons% -webp% -xpm (-bzip2%*) (-curl%) (-dbus%*) (-exif%*) (-jpeg%*) (-lcms%*) (-pdf%*) (-png%*) (-svg%*) (-tiff%*)" CPU_FLAGS_X86="mmx sse" PYTHON_TARGETS="python2_7" 31,206 KiB

Total: 7 packages (2 upgrades, 3 new, 2 in new slots), Size of downloads: 48,591 KiB

I then launched the GIMP and successfully opened the file IMG_3706.HEIC, and I was able to export it as IMG_3706.jpg.

6. As would be expected, the online tool ‘libheif decoder demo’ (https://strukturag.github.io/libheif/) behaves exactly the same in Gentoo Linux as it does in Lubuntu 18.04 (see earlier).

Summary

So there you have it; if the Linux file manager and/or image viewing applications you use cannot already handle HEIC files, the tools in Linux that I found may work are as follows:

  • heif-convert (from the package libheif-examples in Ubuntu/Lubuntu, or from from the package libheif in Gentoo).
  • Imagemagick (not every version).
  • The GIMP (not every version).
  • the online tool ‘libheif decoder demo’ (https://strukturag.github.io/libheif/).

I have not tried the copyright open-source code from Nokia (see link under Further Reading below), qt-heif-image-plugin and tifig (not in active development). If you have had success using another tool to convert HEIC files, please post a comment below for the benefit of other users, giving the name of the tool, the package name and version, and the Linux distribution (including release number, if not a rolling distribution).

Further reading

  1. Wikipedia – High Efficiency Image File Format
  2. Lifewire – What Are HEIF and HEIC, and Why Is Apple Using Them?
  3. Nokia – High Efficiency Image File Format (HEIF)
  4. libheif – a ISO/IEC 23008-12:2017 HEIF file format decoder and encoder
  5. askubuntu – Any app on Ubuntu to open HEIF (.heic, High Efficiency Image File Format) pictures?

A brief discussion about package installation times in Gentoo Linux

I thought that perhaps users of binary-based Linux distributions who are contemplating trying out the source-based distribution Gentoo Linux might be interested to know a bit about package installation times in contrast to binary distributions. I am not going to go into great detail here; this is just to give interested people a quick idea of possible package installation times in Gentoo Linux.

The package manager of a binary-based distribution such as Ubuntu downloads and installs ‘binary’ packages, i.e. packages containing pre-built executables. On the other hand, Gentoo’s package manager Portage downloads source-code packages and builds the binaries (executables) on your machine. Nevertheless, a small number of Portage packages contain binaries rather than source code, either because the source code could take many hours to build on older hardware or because the source code is simply not available in the public domain. An example of the first scenario is Firefox, which is available in Gentoo both as the source code package www-client/firefox and as the binary package www-client/firefox-bin so that the user can choose which to install (‘merge’, in Gentoo parlance). An example of the second scenario is TeamViewer, which is only available as the binary package net-misc/teamviewer because TeamViewer is closed-source software (i.e. the company that develops TeamViewer does not release its source code).

Clearly, installing an application from a package containing source code that has to be compiled and linked on your machine will take longer than installing the application from a package containing the executable binary that someone else has already built. The time to install from source-code packages of course depends on the precise package (some can take only a minute or so to install), the power of the machine on which the package is being installed, and various other factors.

I currently have two laptops running Gentoo Linux: one is over eight years old with an early Intel Core i7-720QM (throttled to 933 MHz because of the small PSU); the other is three and a half years old and has a newer Intel Core i7-4810MQ (2.8 GHz). Although the older laptop is slow by today’s standards, the newer laptop is still reasonably powerful. Even so, those few large source-code packages such as Firefox, Chromium and LibreOffice can take quite a long time to install, as illustrated below for Firefox by using the Gentoo utility genlop to find out the time it took to install:

clevow230ss /home/fitzcarraldo # genlop -t firefox | tail -n 3
     Wed Sep 26 19:53:47 2018 >>> www-client/firefox-62.0.2
       merge time: 1 hour, 4 minutes and 22 seconds.

On the other hand, installing TeamViewer took no time at all:

clevow230ss /home/fitzcarraldo # # genlop -t teamviewer | tail -n 3
     Mon Oct  1 21:39:29 2018 >>> net-misc/teamviewer-13.2.13582
       merge time: 16 seconds.

And youtube-dl also installed quickly:

clevow230ss /home/fitzcarraldo # genlop -t youtube-dl | tail -n 3
     Mon Oct  1 21:42:26 2018 >>> net-misc/youtube-dl-2018.09.26
       merge time: 53 seconds.

I also used to run Gentoo Linux on a 1998 Gateway Solo 9300 laptop (Intel Mobile Pentium III Coppermine 800 MHz). Neither its CPU nor its 288 MB RAM could cope with building large source code packages such as Firefox and LibreOffice, so on that machine I installed www-client/firefox-bin and app-office/libreoffice-bin instead.

Anyway, now let’s have a quick look at a typical package upgrade session, which I aim to perform at least once a week. On the above-mentioned newer laptop today I upgraded all installed packages that had newer versions available in the so-called Portage ‘tree’. This amounted to installing (‘merging’) the following fifteen packages:

  1. net-analyzer/traceroute-2.1.0
  2. media-libs/exiftool-11.03
  3. dev-libs/libtasn1-4.13
  4. x11-misc/shared-mime-info-1.10
  5. x11-libs/libxcb-1.13.1
  6. media-libs/x265-2.8
  7. dev-libs/redland-1.0.17-r1
  8. media-libs/mesa-18.1.9
  9. dev-libs/dbus-glib-0.110
  10. sys-power/upower-0.99.8
  11. app-text/libspectre-0.2.8
  12. app-crypt/pinentry-1.1.0-r2
  13. net-im/pidgin-2.13.0
  14. app-emulation/wine-vanilla-3.17
  15. media-video/handbrake-1.1.1

It took just under thirty-four minutes to install the above fifteen packages on my newer laptop. Below is what the upgrade process looked like in a terminal window. I have used the package manager’s long command options here, but normally I would use the single-letter shortcuts. As the focus of this post is on package installation time, I will not bother to show the command I used before this to synchronise (update) package information on the laptop with the remote package servers.

clevow230ss /home/fitzcarraldo # emerge --update --verbose --deep --newuse --with-bdeps=y --keep-going @world

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild     U  ] net-analyzer/traceroute-2.1.0::gentoo [2.0.20::gentoo] USE="-static" 70 KiB
[ebuild     U  ] media-libs/exiftool-11.03::gentoo [10.95::gentoo] USE="-doc" 4,311 KiB
[ebuild   R    ] dev-libs/libtasn1-4.13:0/6::gentoo  USE="-doc -static-libs -test% -valgrind" ABI_X86="32 (64) (-x32)" 0 KiB
[ebuild     U  ] x11-misc/shared-mime-info-1.10::gentoo [1.9::gentoo] USE="-test" 603 KiB
[ebuild     U  ] x11-libs/libxcb-1.13.1:0/1.12::gentoo [1.13:0/1.12::gentoo] USE="xkb -doc (-selinux) -static-libs -test" ABI_X86="32 (64) (-x32)" 495 KiB
[ebuild  r  U  ] media-libs/x265-2.8:0/160::gentoo [2.6:0/146::gentoo] USE="10bit 12bit -numa -pic (-power8) -test" ABI_X86="(64) -32 (-x32)" 1,348 KiB
[ebuild     U  ] dev-libs/redland-1.0.17-r1::gentoo [1.0.16::gentoo] USE="berkdb -iodbc -mysql -odbc -postgres -sqlite -static-libs (-xml%*)" 1,584 KiB
[ebuild     U  ] media-libs/mesa-18.1.9::gentoo [18.1.6::gentoo] USE="classic dri3 egl gallium gbm llvm nptl wayland -bindist -d3d9 -debug -gles1 -gles2 -opencl -openmax -osmesa -pax_kernel -pic (-selinux) -unwind -vaapi -valgrind -vdpau -vulkan -xa -xvmc" ABI_X86="32 (64) (-x32)" VIDEO_CARDS="i915 i965 intel (-freedreno) (-imx) -nouveau -r100 -r200 -r300 -r600 -radeon -radeonsi (-vc4) -virgl (-vivante) -vmware" 10,885 KiB
[ebuild     U  ] dev-libs/dbus-glib-0.110::gentoo [0.108::gentoo] USE="-debug -static-libs -test" ABI_X86="(64) -32 (-x32)" 817 KiB
[ebuild     U  ] sys-power/upower-0.99.8:0/3::gentoo [0.99.5:0/3::gentoo] USE="introspection -doc -ios (-selinux)" 439 KiB
[ebuild     U  ] app-text/libspectre-0.2.8::gentoo [0.2.7::gentoo] USE="-debug -doc -static-libs" 412 KiB
[ebuild     U  ] app-crypt/pinentry-1.1.0-r2::gentoo [1.0.0-r2::gentoo] USE="gnome-keyring gtk ncurses qt5 -caps -emacs -fltk% -static" 457 KiB
[ebuild     U  ] net-im/pidgin-2.13.0:0/2::gentoo [2.12.0:0/2::gentoo] USE="dbus gnutls gstreamer gtk ncurses networkmanager nls spell xscreensaver zeroconf (-aqua) -debug -doc -eds -gadu -groupwise -idn -meanwhile -perl -pie -prediction -python -sasl -silc -tcl -tk -zephyr" PYTHON_TARGETS="python2_7" 8,784 KiB
[ebuild  NS   ~] app-emulation/wine-vanilla-3.17:3.17::gentoo [3.16:3.16::gentoo] USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner sdl ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gssapi -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) -test -vkd3d -vulkan -xinerama" ABI_X86="32 64 (-x32)" 20,955 KiB
[ebuild  rR   ~] media-video/handbrake-1.1.1::gentoo  USE="fdk gstreamer gtk x265 -libav -libav-aac" 0 KiB

Total: 15 packages (12 upgrades, 1 in new slot, 2 reinstalls), Size of downloads: 51,156 KiB

The following packages are causing rebuilds:

  (media-libs/x265-2.8:0/160::gentoo, ebuild scheduled for merge) causes rebuilds for:
    (media-video/handbrake-1.1.1:0/0::gentoo, ebuild scheduled for merge)

>>> Verifying ebuild manifests
>>> Running pre-merge checks for app-emulation/wine-vanilla-3.17
>>> Emerging (1 of 15) net-analyzer/traceroute-2.1.0::gentoo
>>> Emerging (2 of 15) media-libs/exiftool-11.03::gentoo
>>> Emerging (3 of 15) dev-libs/libtasn1-4.13::gentoo
>>> Installing (1 of 15) net-analyzer/traceroute-2.1.0::gentoo
>>> Installing (2 of 15) media-libs/exiftool-11.03::gentoo
>>> Installing (3 of 15) dev-libs/libtasn1-4.13::gentoo
>>> Emerging (4 of 15) x11-misc/shared-mime-info-1.10::gentoo
>>> Installing (4 of 15) x11-misc/shared-mime-info-1.10::gentoo
>>> Emerging (5 of 15) x11-libs/libxcb-1.13.1::gentoo
>>> Installing (5 of 15) x11-libs/libxcb-1.13.1::gentoo
>>> Emerging (6 of 15) media-libs/x265-2.8::gentoo
>>> Emerging (7 of 15) dev-libs/redland-1.0.17-r1::gentoo
>>> Emerging (8 of 15) media-libs/mesa-18.1.9::gentoo
>>> Installing (7 of 15) dev-libs/redland-1.0.17-r1::gentoo
>>> Installing (6 of 15) media-libs/x265-2.8::gentoo
>>> Installing (8 of 15) media-libs/mesa-18.1.9::gentoo
>>> Emerging (9 of 15) dev-libs/dbus-glib-0.110::gentoo
>>> Installing (9 of 15) dev-libs/dbus-glib-0.110::gentoo
>>> Emerging (10 of 15) sys-power/upower-0.99.8::gentoo
>>> Emerging (11 of 15) app-text/libspectre-0.2.8::gentoo
>>> Emerging (12 of 15) app-crypt/pinentry-1.1.0-r2::gentoo
>>> Installing (10 of 15) sys-power/upower-0.99.8::gentoo
>>> Installing (11 of 15) app-text/libspectre-0.2.8::gentoo
>>> Installing (12 of 15) app-crypt/pinentry-1.1.0-r2::gentoo
>>> Emerging (13 of 15) net-im/pidgin-2.13.0::gentoo
>>> Emerging (14 of 15) app-emulation/wine-vanilla-3.17::gentoo
>>> Emerging (15 of 15) media-video/handbrake-1.1.1::gentoo
>>> Installing (15 of 15) media-video/handbrake-1.1.1::gentoo
>>> Installing (13 of 15) net-im/pidgin-2.13.0::gentoo
>>> Installing (14 of 15) app-emulation/wine-vanilla-3.17::gentoo
>>> Jobs: 15 of 15 complete                         Load avg: 2.94, 5.84, 6.01
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * Regenerating GNU info directory index...
 * Processed 129 info files.
 * After world updates, it is important to remove obsolete packages with
 * emerge --depclean. Refer to `man emerge` for more information.
 

The genlop tool in Gentoo can be used to check chronologically which packages were installed and removed. For example:

clevow230ss /home/fitzcarraldo # genlop -l | grep "Mon Oct  1"
     Mon Oct  1 18:16:19 2018 >>> net-analyzer/traceroute-2.1.0
     Mon Oct  1 18:16:27 2018 >>> media-libs/exiftool-11.03
     Mon Oct  1 18:16:49 2018 >>> dev-libs/libtasn1-4.13
     Mon Oct  1 18:17:02 2018 >>> x11-misc/shared-mime-info-1.10
     Mon Oct  1 18:17:49 2018 >>> x11-libs/libxcb-1.13.1
     Mon Oct  1 18:19:11 2018 >>> dev-libs/redland-1.0.17-r1
     Mon Oct  1 18:28:07 2018 >>> media-libs/x265-2.8
     Mon Oct  1 18:28:14 2018 >>> media-libs/mesa-18.1.9
     Mon Oct  1 18:28:30 2018 >>> dev-libs/dbus-glib-0.110
     Mon Oct  1 18:28:52 2018 >>> sys-power/upower-0.99.8
     Mon Oct  1 18:28:57 2018 >>> app-text/libspectre-0.2.8
     Mon Oct  1 18:29:03 2018 >>> app-crypt/pinentry-1.1.0-r2
     Mon Oct  1 18:30:02 2018 >>> media-video/handbrake-1.1.1
     Mon Oct  1 18:30:34 2018 >>> net-im/pidgin-2.13.0
     Mon Oct  1 18:47:45 2018 >>> app-emulation/wine-vanilla-3.17
clevow230ss /home/fitzcarraldo #

When packages are being upgraded, the package manager will inform you if a package has an update to the associated application’s configuration file. Even if the package manager does not indicate that any configuration files need to be updated, it does no harm to check anyway:

clevow230ss /home/fitzcarraldo # etc-update
Scanning Configuration files...
Exiting: Nothing left to do; exiting. :)

To remove packages that are no longer required (either because the package is no longer a dependency or because a newer version of a slotted* package has been installed), the ‘depclean‘ option is used. Below is what happened when I used that option to clean up after the above-mentioned fifteen packages were upgraded/re-installed.

* Some Gentoo packages are ‘slotted’, i.e. more than one version of the package can be installed and used concurrently. Examples include WINE, Python and kernel sources.

clevow230ss /home/fitzcarraldo # emerge --ask --depclean

 * Always study the list of packages to be cleaned for any obvious
 * mistakes. Packages that are part of the world set will always
 * be kept.  They can be manually added to this set with
 * `emerge --noreplace `.  Packages that are listed in
 * package.provided (see portage(5)) will be removed by
 * depclean, even if they are part of the world set.
 * 
 * As a safety measure, depclean will not remove any packages
 * unless *all* required dependencies have been resolved.  As a
 * consequence of this, it often becomes necessary to run 
 * `emerge --update --newuse --deep @world` prior to depclean.

Calculating dependencies... done!
>>> Calculating removal order...

>>> These are the packages that would be unmerged:                                                                                                                                           

 app-emulation/wine-vanilla
    selected: 3.16 
   protected: none 
     omitted: 3.17 

 dev-python/pyliblzma
    selected: 0.5.3-r1 
   protected: none 
     omitted: none 

 dev-python/twisted-web
    selected: 15.2.1 
   protected: none 
     omitted: none 

 dev-python/psutil
    selected: 5.4.3 
   protected: none 
     omitted: none 

 dev-python/twisted-core
    selected: 15.2.1 
   protected: none 
     omitted: none 

 dev-python/service_identity
    selected: 16.0.0 
   protected: none 
     omitted: none 

 dev-python/pyasn1-modules
    selected: 0.0.8 
   protected: none 
     omitted: none 

 dev-python/attrs
    selected: 16.3.0-r1 
   protected: none 
     omitted: none 

 dev-python/zope-interface
    selected: 4.4.3 
   protected: none 
     omitted: none 

 dev-python/namespace-zope
    selected: 1 
   protected: none 
     omitted: none 

All selected packages: =dev-python/pyliblzma-0.5.3-r1 =dev-python/namespace-zope-1 =dev-python/zope-interface-4.4.3 =dev-python/pyasn1-modules-0.0.8 =app-emulation/wine-vanilla-3.16 =dev-python/psutil-5.4.3 =dev-python/twisted-web-15.2.1 =dev-python/twisted-core-15.2.1 =dev-python/service_identity-16.0.0 =dev-python/attrs-16.3.0-r1

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Would you like to unmerge these packages? [Yes/No] Yes
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging (1 of 10) app-emulation/wine-vanilla-3.16...
>>> Unmerging (2 of 10) dev-python/pyliblzma-0.5.3-r1...
>>> Unmerging (3 of 10) dev-python/twisted-web-15.2.1...
>>> Unmerging (4 of 10) dev-python/psutil-5.4.3...
>>> Unmerging (5 of 10) dev-python/twisted-core-15.2.1...
>>> Unmerging (6 of 10) dev-python/service_identity-16.0.0...
>>> Unmerging (7 of 10) dev-python/pyasn1-modules-0.0.8...
>>> Unmerging (8 of 10) dev-python/attrs-16.3.0-r1...
>>> Unmerging (9 of 10) dev-python/zope-interface-4.4.3...
>>> Unmerging (10 of 10) dev-python/namespace-zope-1...
Packages installed:   1714
Packages in world:    198
Packages in system:   43
Required packages:    1714
Number removed:       10

 * GNU info directory index is up-to-date.

As you can see above, the package manager found a number of installed packages that were no longer required, either because a new version of a slotted package had been installed (notice wine-vanilla) or because they are no longer dependencies. As I did not want to keep both wine-3.16 and wine-3.17 installed simultaneously, I simply replied ‘Yes’. By the way, the depclean option can also be used to remove (uninstall) older versions of an individual slotted package:

clevow230ss /home/fitzcarraldo # emerge --ask --depclean wine-vanilla

It can also be used to remove an individual package if it is not a dependency. For example, the following command does not allow me to remove the installed package vlc because other installed packages depend on it:

clevow230ss /home/fitzcarraldo # emerge --ask --depclean vlc

Calculating dependencies... done!
>>> No packages selected for removal by depclean
>>> To see reverse dependencies, use --verbose
Packages installed:   1714
Packages in world:    198
Packages in system:   43
Required packages:    1714
Number removed:       0

Whereas the following command would allow me to remove the installed package winetricks because no other installed package depends on it:

clevow230ss /home/fitzcarraldo # emerge --ask --depclean winetricks

Calculating dependencies... done!
>>> Calculating removal order...

>>> These are the packages that would be unmerged:                                                                                                                                           

 app-emulation/winetricks
    selected: 20180815 
   protected: none 
     omitted: none 

All selected packages: =app-emulation/winetricks-20180815

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Would you like to unmerge these packages? [Yes/No] No

Quitting.

Packages installed:   1714
Packages in world:    198
Packages in system:   43
Required packages:    1713
Number to remove:     1

Anyway, I hope this post has given potential adopters a rough idea of installation times in Gentoo Linux, even if it is far from thorough.

Trouble again with PulseAudio and Thunderbird sound notifications

In an earlier post I described how I fixed a scratchy-sounding sound file which the Thunderbird e-mail client plays when a new message arrives. Well, the problem started again recently, but this time the contents of /etc/pulse/daemon.conf looked OK to me. Furthermore, the sound file sounds fine when played using following commands:

aplay ~/Music/wav/E-mail_notifications/halmsg.wav
paplay ~/Music/wav/E-mail_notifications/halmsg.wav
mplayer ~/Music/wav/E-mail_notifications/halmsg.wav
cvlc ~/Music/wav/E-mail_notifications/halmsg.wav

Now, Thunderbird uses libcanberra to play sounds, so I began to wonder if the problem lay with libcanberra. As it happens, libcanberra is maintained by the same person who invented PulseAudio. However, I notice from the libcanberra Git repository that its source code has not been changed since 2012.

My Gentoo Linux installation had libcanberra installed with support for both ALSA and PulseAudio:

root # eix -I libcanberra
[I] media-libs/libcanberra
     Available versions:  0.30-r5 {alsa gnome gstreamer +gtk +gtk3 oss pulseaudio +sound tdb udev ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  0.30-r5(08:27:41 18/05/18)(alsa gtk gtk3 pulseaudio sound udev -gnome -gstreamer -oss -tdb ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://git.0pointer.net/libcanberra.git/
     Description:         Portable sound event library

So, even though my installation uses PulseAudio, I decided to try and re-install libcanberra without PulseAudio support, only ALSA support:

root # USE="-pulseaudio" emerge -1v libcanberra
root # eix -I libcanberra
[I] media-libs/libcanberra
     Available versions:  0.30-r5 {alsa gnome gstreamer +gtk +gtk3 oss pulseaudio +sound tdb udev ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  0.30-r5(15:47:14 26/05/18)(alsa gtk gtk3 sound udev -gnome -gstreamer -oss -pulseaudio -tdb ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://git.0pointer.net/libcanberra.git/
     Description:         Portable sound event library

Lo and behold, Thunderbird (libcanberra) plays the sound file correctly now. So I have added the following line to my file /etc/portage/package.use/thunderbird in order to make the change permanent:

media-libs/libcanberra -pulseaudio

PulseAudio ūüôĄ

Gentoo Linux: A work-around to be able to Resume from Suspend to RAM when using the NVIDIA closed-source driver

My Clevo W230SS laptop has NVIDIA Optimus graphics hardware (NVIDIA GPU plus Intel IGP). I do not use Bumblebee, preferring to switch between the Intel video driver and the NVIDIA closed-source driver myself (see Switching between Intel and NVIDIA graphics processors on a laptop with NVIDIA Optimus hardware running Gentoo Linux). The laptop can suspend to RAM and resume perfectly when using the Intel video driver (but see Stopping my laptop spontaneously resuming immediately after Suspend to RAM, which is applicable whatever the GPU or IGP).

In order to be able to resume properly from Suspend-to-RAM when using the NVIDIA driver, the laptop needs to disable compositing before suspending, then re-enable compositing after resuming. For how I achieve that, see under Problem 2 in the third link above. If this is not done, the graphics on the Desktop are corrupted after resuming.

However, recently when using the NVIDIA driver and KDE Plasma 5 (I am currently using nvidia-drivers-387.22 and plasma-meta-5.11.5), when resuming from suspension the monitor would briefly display the LightDM wallpaper (I use different wallpapers for the display manager and the lock screen, so I know it was not the KDE lock screen) followed by a blank screen with a mouse pointer (which I could move normally). More recently, in between displaying the display manager’s wallpaper and the blank screen, the monitor would briefly display an earlier image of the Desktop just before the laptop suspended.

Now, I could simply leave the laptop configured to use the Intel driver. However, sometimes I need to use a CAD application and the performance is better when using the NVIDIA GPU.

There are umpteen posts on the Web about this problem, and the root cause seems to be the closed-source NVIDIA driver. I have seen the KDE lock screen mentioned in some posts as the culprit, so I disabled the lock screen (‘System Settings’ > ‘Desktop Behaviour’ > ‘Screen Locking’) but that did not solve the problem.

I put up with this for several weeks in the hope that the next release of the NVIDIA driver would fix the problem. If I suspended to RAM while the laptop was using the NVIDIA driver, I was able to resume and get to a working Desktop –¬†albeit without the open windows and applications that had been running before suspending¬†– by pressing Ctrl+Alt+F1 to get to TTY1, logging in as the root user and entering the command ‘/etc/init.d/xdm restart‘. However, the final straw was in a meeting a couple of weeks ago when I wanted to resume the laptop and show a worksheet to someone. The laptop monitor of course displayed a blank screen with a mouse pointer, and it took me a couple of minutes to restart the display manager, login to KDE Plasma 5 and open the spreadsheet again. So this week I decided to look into the problem to see if I could at least find a work-around that would enable the laptop to resume without needing to restart X Windows and login to Plasma 5 each time.

I created a Bash script in /etc/pm/sleep.d/ to unload the NVIDIA modules before suspending to RAM and to re-load them when resuming, but that did not solve the problem either.

I switched the rendering background from OpenGL 2.0 to OpenGL 3.1 (‘System Settings’ > ‘Display and Monitor’ > ‘Compositor’), but that did not work either. I switched the rendering backend to XRender, and that did enable the laptop to resume from suspend successfully with the NVIDIA driver, but I do not want to use that work-around. Firstly, with software rendering there is a performance hit, and, secondly, there was no KDE Desktop Cube when using XRender instead of OpenGL. I use the Desktop Cube when working, as I often have a lot of windows open on each virtual desktop (cube side), and I find it easier to use the cube than a flat UI.

Eventually I found that, after resuming, if I pressed Ctrl+Alt+F1 to get to a virtual console, logged into my user account, entered the command ‘DISPLAY=:0 /usr/bin/kwin_x11 --resume‘ and then pressed Ctrl+Alt+F7 to get back to TTY7, my Desktop would appear on TTY7. Even so, I noticed on TTY1 that the following error messages were displayed when I ran that command:

kwin_core: OpenGL 2 compositing setup failed
kwin_core: Failed to initialize compositing, compositing disabled

Anyway, the Plasma 5 Desktop was displayed on TTY7, and with the windows that were open when I suspended the laptop, so restarting KWin would at least be a viable work-around until NVIDIA fix their video driver.

I incoporated the command in my script /etc/pm/sleep.d/02-toggle-compositing like so:

#!/bin/sh
#
# Turn off compositing on hibernate or suspend
# Turn on compositing on thaw or resume

username=fitzcarraldo
userhome=/home/$username
export XAUTHORITY="$userhome/.Xauthority"
export DISPLAY=":0"

case "$1" in
     suspend|hibernate)
          su $username -c "qdbus org.kde.KWin /Compositor suspend" &
     ;;
     resume|thaw)
          su $username -c "qdbus org.kde.KWin /Compositor resume" &
          su $username -c "/usr/bin/kwin_x11 --replace" &
     ;;
     *)
          exit $NA
     ;;
esac

It is an ugly hack, but at least now the laptop can resume properly from Suspend-to-RAM while the NVIDIA driver is being used.

Perhaps Linus Torvalds was correct. I will try to avoid NVIDIA hardware when I replace my current laptop.

Running a shell script at shutdown only (not at reboot) – a comparison between OpenRC and systemd

Gentoo Linux on my laptops uses OpenRC with SysVinit, whereas Lubuntu 17.10 on my family’s PC uses systemd. I have had to configure both Linux distributions to run a backup job at shutdown, so I thought it would be interesting to summarise the two approaches.

OpenRC

Create a Bash script /etc/local.d/10-run_on_shutdown.stop with the following contents:

#!/bin/bash
if [ `who -r | awk '{print $2}'` = "0" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

From now on the script will run to completion when you shutdown the machine, but not when you reboot it.

systemd

1. Create a Bash script /usr/local/sbin/run_on_shutdown.sh with the following contents:

#!/bin/bash
REBOOT=$( systemctl list-jobs | egrep -q 'reboot.target.*start' && echo "rebooting" || echo "not_rebooting" )
if [ $REBOOT = "not_rebooting" ]; then
    ########################################################################
    # Put Bash commands here to be executed on shutdown but not on reboot. #
    # For example, backup home directories to an external USB HDD.         #
    ########################################################################
fi

2. Create a unit file /etc/systemd/system/run_on_shutdown.service with the following contents:

[Unit]
Description=Run a Bash script at shutdown
DefaultDependencies=no
Before=shutdown.target halt.target
# If your script requires any mounted directories, add them below: 
RequiresMountsFor=/home

[Service]
Type=oneshot
ExecStart=/usr/local/sbin/run_on_shutdown.sh

[Install]
WantedBy=halt.target shutdown.target

I have assumed the Bash script is to backup /home, so change the directory list in ‘RequiresMountsFor=‘ if the script necessitates that other directories are still mounted – see ‘man¬†systemd.unit‘ for details.

3. Enable the unit file and start the service as follows:

user $ sudo systemctl enable run_on_shutdown.service
user $ sudo reboot

From now on, the script will run to completion when you shutdown the machine, but not when you reboot it.

Notes on the systemd solution:

There are plenty of posts on the Web suggesting how to run a script at shutdown in installations that use systemd, but the approaches either do not work in my case or they do not discriminate between shutting down and rebooting. For example, if you are wondering why I do not use ‘Conflicts=reboot.target‘ in the unit file, I found it does not prevent the Bash script from being launched when the system is rebooted (see the entry for ‘Conflicts=‘ in ‘man¬†systemd.unit‘ for the functionality). If you are wondering why I do not simply place a script in /lib/systemd/system-shutdown/, it is because scripts in that directory are launched late in the shutdown process, after file systems have been mounted read-only. And if you are wondering why the Bash script tests if the system is rebooting (as distinct from shutting down) rather than the unit file being configured to do that, it is because the systemd shutdown target is active in any type of system termination, be it rebooting or halting, and it therefore does not help in discriminating between the termination types (see ‘man¬†systemd.special‘). I tried several approaches in the unit file to see if it could be made to launch a Bash script when shutting down but not when rebooting, but the only solution that works for me in the Lubuntu 17.10 installation on my family’s PC uses the two files listed above working in conjunction with each other.

Furthermore, of all the unit files I found on the Web that actually make systemd launch a Bash script when a user initiates shutdown, only one of them prevents systemd from shutting down the machine before a time-consuming script has run to completion (for example a script to backup to an external HDD). I also tried using systemd-inhibit in this case, and it did not work. Looking at the manual page for systemd-inhibit, it could be used to execute a program (the example given is ‘systemd-inhibit wodim foobar.iso‘), however I found that systemd-inhibit did not prevent systemd from shutting down the machine before a Bash script to backup a single-seat, multi-user installation had run to completion.

Some background reading

  1. Easy Systemd Startup and Shutdown Scripts for Ubuntu 16.04 and Mint 18
  2. Stack Exchange Super User : How do I run a script before everything else on shutdown with systemd?
  3. Stack Overflow : how can a systemd controlled service distinguish between shutdown and reboot?
  4. Unix & Linux Stack Exchange : Systemd : How to execute script at shutdown only (not at reboot)

Moving to the slotted WINE package system in Gentoo Linux

Earlier this year the Gentoo Linux developers responsible for maintaining the package app-emulation/wine decided to split it and slot it so that different versions of WINE can be installed and co-exist simultaneously:

clevow230ss fitzcarraldo # eselect news read 34
2017-04-10-split-and-slotted-wine
  Title                     app-emulation/wine split and slotting
  Author                    NP-Hardass 
  Posted                    2017-04-10
  Revision                  1

Starting with Wine 2.0, Wine in Gentoo is transitioning away from its
traditional packaging and toward a new, split and slotted, Wine.

As many Wine users know, there are often regressions or an application
works better on one version of wine than another.  Going forward, 
packaging in Gentoo will allow simultaneous installation of multiple
versions of Wine.

Additionally, to expedite vanilla releases as well as permit multiple
configurations for each Wine installation, the major patchsets have
been split out into separate packages.

Going forward, app-emulation/wine will transition to:
app-emulation/wine-vanilla: upstream Wine with no external patchsets
             (like if the old packaging forced USE="-staging -d3d9")
app-emulation/wine-staging: Wine with Wine-Staging's patchset
             (like if the old packaging forced USE="+staging -d3d9")
app-emulation/wine-d3d9: Wine with Ixit's Gallium Nine patchset
             (like if the old packaging forced USE="-staging +d3d9")
app-emulation/wine-any: Wine with any of the patchsets or flags
             (exactly like the old packaging regarding USE flags)

wine-any exists to allow the user to build any combination that they'd
like (like the old packaging).  This means the user could use wine-any
to use both Wine-Staging and Gallium Nine.  Alternatively, the user
could use wine-any to try out another configuration from other
packages.  For example, the user could build wine-vanilla without
PulseAudio, and could build wine-any with PulseAudio.  The sky is the
limit on how a user may choose to use app-emulation/wine-any.

Users may opt for any specific package, or may emerge virtual/wine,
which is provided for dependency resolution.
Maintainers: Please note, app-emulation/wine will be dropped, so
please use virtual/wine going forward.

Users may call each version specifically, or may call a symlink based
on their installed patchset, for example wine-2.1, wine-staging-2.2,
or wine-d3d9.

Symlinks for wine are managed with app-eselect/eselect-wine.
# eselect wine set wine-vanilla-2.0
/usr/bin/wine -> /usr/bin/wine-vanilla-2.0
# eselect wine set --staging wine-staging-2.4
/usr/bin/wine-staging -> /usr/bin/wine-staging-2.4

Earlier this year the Gentoo Linux Forums thread ‘wine: questions on recent changes‘ discussed the new system and how to use it. Several users, myself included, posted questions in that thread asking how to go about the change. I received differing advice and remained uncertain about what to do. At the time, the slotted packages for the new system had not yet been unmasked, so I left my installation as it was and decided to put off making the change for as long as possible. Today when I synchronised my installation with the Portage tree there was a Gentoo Linux news item ‘2017-11-21 Old Wine versions moving to wine-overlay’. So the time had come for me to make the switch from app-emulation/wine-2.3 to the new slotted WINE package system. Here is what I did…

Previous situation

I had been using WINE Staging:

clevow230ss fitzcarraldo # eix -I wine
[?] app-emulation/wine
     Available versions:  [M]2.0^t [M](~)2.1^t [M](~)2.2^t [M](~)2.3^t [M]**9999^t {+X +alsa capi cups custom-cflags d3d9 dos +fontconfig +gecko gphoto2 gsm gstreamer +jpeg +lcms ldap +mono mp3 ncurses netapi nls odbc openal opencl +opengl osmesa oss pcap +perl pipelight +png prelink pulseaudio +realtime +run-exes s3tc samba scanner selinux +ssl staging test themes +threads +truetype udev +udisks v4l vaapi +xcomposite xinerama +xml ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="(+)32 (+)64 x32" ELIBC="glibc" KERNEL="FreeBSD" LINGUAS="ar bg ca cs da de el en en_US eo es fa fi fr he hi hr hu it ja ko lt ml nb_NO nl or pa pl pt_BR pt_PT rm ro ru sk sl sr_RS@cyrillic sr_RS@latin sv te th tr uk wa zh_CN zh_TW"}
     Installed versions:  2.3^t(19:16:31 20/05/17)(X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32" ELIBC="glibc" KERNEL="-FreeBSD" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW")
     Homepage:            https://www.winehq.org/
     Description:         Free implementation of Windows(tm) on Unix

[I] app-emulation/winetricks
     Available versions:  20170823^t **99999999^t {gtk kde rar}
     Installed versions:  20170823^t(02:50:56 06/09/17)(gtk kde -rar)
     Homepage:            https://github.com/Winetricks/winetricks https://wiki.winehq.org/Winetricks
     Description:         Easy way to install DLLs needed to work around problems in Wine

Found 2 matches

The file /etc/portage/package.accept_keywords/wine in my installation included the following line:

app-emulation/wine ~amd64

The file /etc/portage/package.use/wine in my installation included the following line:

app-emulation/wine -ldap -pipelight staging abi_x86_32

New situation

The file /etc/portage/package.accept_keywords/wine in my installation now includes the lines listed below. The reason why I had to include app-emulation/wine-vanilla is explained further on.

virtual/wine ~amd64
app-emulation/wine-staging ~amd64
app-emulation/wine-vanilla ~amd64

The file /etc/portage/package.use/wine in my installation now includes the lines listed below. The reason why I had to include app-emulation/wine-vanilla is explained further on.

virtual/wine -d3d9 staging abi_x86_32
app-emulation/wine-staging X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
app-emulation/wine-vanilla X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32

(Note that e.g. the pipelight USE flag does not actually exist in the package app-emulation/wine-vanilla, but it does no harm to include “-pipelight” in the line for app-emulation/wine-vanilla in the file package.use.)

After adding the above-mentioned lines to the two files, I then had to uninstall the masked package app-emulation/wine-2.3 and the package app-emulation/winetricks-20170823 that depended on it:

clevow230ss fitzcarraldo # emerge -aC app-emulation/wine app-emulation/winetricks

Then I had to install the package virtual/wine, which pulled in the package app-emulation/wine-staging as I wanted, but also pulled in the package app-emulation/vanilla:

clevow230ss fitzcarraldo # emerge -a virtual/wine

Then I had to select wine-staging:

clevow230ss fitzcarraldo # eselect wine list
clevow230ss fitzcarraldo # eselect wine set

Finally, I had to install winetricks, which would now use the selected slotted package:

clevow230ss fitzcarraldo # emerge winetricks

How I arrived at the contents of package.accept_keywords and package.use

Note that, before I added the line for app-emulation/wine-vanilla to /etc/portage/package.use/wine and /etc/portage/package.accept_keywords/wine, this is what Portage wanted to install:

clevow230ss fitzcarraldo # emerge -p virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild   R    ] net-nds/openldap-2.4.44  ABI_X86="32*" 
[ebuild  N     ] app-emulation/wine-vanilla-2.0.2  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms ldap mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by app-emulation/wine-vanilla-2.0.2::gentoo[ldap]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=net-nds/openldap-2.4.44 abi_x86_32
# required by app-emulation/wine-vanilla-2.0.2::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32

!!! The following installed packages are masked:
- dev-qt/qtwebkit-4.8.7::gentoo (masked by: package.mask)
/usr/portage/profiles/package.mask:
# Andreas Sturmlechner  (16 Nov 2017)
# Qt4WebKit is ancient and full of security holes.
# Masked for removal in 30 days. Bug #620684

For more information, see the MASKED PACKAGES section in the emerge
man page or refer to the Gentoo Handbook.

clevow230ss fitzcarraldo #

After I added a line for app-emulation/wine-vanilla to /etc/portage/package.use/wine and /etc/portage/package.accept_keywords/wine, this is what Portage wanted to do:

clevow230ss fitzcarraldo # emerge -p virtual/wine                        

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

The following USE changes are necessary to proceed:
 (see "package.use" in the portage(5) man page for more details)
# required by app-emulation/wine-vanilla-2.20::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
# required by virtual/wine (argument)
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32
clevow230ss fitzcarraldo #

So I added the above-mentioned USE change for app-emulation/wine-gecko to /etc/portage/package.use/wine, and then Portage wanted to do the following:

clevow230ss fitzcarraldo # emerge -p virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 
clevow230ss fitzcarraldo #

I was happy with that, so I went ahead without the --pretend option:

clevow230ss fitzcarraldo # emerge -a virtual/wine

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N     ] app-emulation/wine-gecko-2.47-r1  ABI_X86="32 (64)" 
[ebuild  N     ] app-eselect/eselect-wine-1.2.2 
[ebuild  N     ] app-emulation/wine-desktop-common-20150204 
[ebuild  N    ~] app-emulation/wine-vanilla-2.20  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -kerberos -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -prelink -samba (-selinux) {-test} -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] app-emulation/wine-staging-2.19  USE="X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl (staging) threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba (-selinux) {-test} -themes -vaapi -xinerama" ABI_X86="32 64 (-x32)" LINGUAS="en pt_BR -ar -bg -ca -cs -da -de -el -en_US -eo -es -fa -fi -fr -he -hi -hr -hu -it -ja -ko -lt -ml -nb_NO -nl -or -pa -pl -pt_PT -rm -ro -ru -sk -sl -sr_RS@cyrillic -sr_RS@latin -sv -te -th -tr -uk -wa -zh_CN -zh_TW" 
[ebuild  N    ~] virtual/wine-0-r6  USE="staging -d3d9" ABI_X86="32 64" 

Would you like to merge these packages? [Yes/No] Yes
>>> Verifying ebuild manifests
>>> Running pre-merge checks for app-emulation/wine-vanilla-2.20
>>> Running pre-merge checks for app-emulation/wine-staging-2.19
>>> Emerging (1 of 6) app-emulation/wine-gecko-2.47-r1::gentoo
>>> Emerging (2 of 6) app-eselect/eselect-wine-1.2.2::gentoo
>>> Installing (1 of 6) app-emulation/wine-gecko-2.47-r1::gentoo
>>> Installing (2 of 6) app-eselect/eselect-wine-1.2.2::gentoo
>>> Emerging (3 of 6) app-emulation/wine-desktop-common-20150204::gentoo
>>> Installing (3 of 6) app-emulation/wine-desktop-common-20150204::gentoo
>>> Emerging (4 of 6) app-emulation/wine-vanilla-2.20::gentoo
>>> Emerging (5 of 6) app-emulation/wine-staging-2.19::gentoo
>>> Installing (4 of 6) app-emulation/wine-vanilla-2.20::gentoo
>>> Installing (5 of 6) app-emulation/wine-staging-2.19::gentoo
>>> Emerging (6 of 6) virtual/wine-0-r6::gentoo
>>> Installing (6 of 6) virtual/wine-0-r6::gentoo
>>> Recording virtual/wine in "world" favorites file...
>>> Jobs: 6 of 6 complete                           Load avg: 6.80, 7.34, 7.06
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19
  [2]   wine-vanilla-2.20 *
clevow230ss fitzcarraldo # eselect wine set 1
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19 *
  [2]   wine-vanilla-2.20
clevow230ss fitzcarraldo # emerge app-emulation/winetricks
Calculating dependencies... done!
>>> Verifying ebuild manifests
>>> Emerging (1 of 1) app-emulation/winetricks-20170823::gentoo
>>> Installing (1 of 1) app-emulation/winetricks-20170823::gentoo
>>> Recording app-emulation/winetricks in "world" favorites file...
>>> Jobs: 1 of 1 complete                           Load avg: 0.51, 2.78, 5.09
>>> Auto-cleaning packages...

>>> No outdated packages were found on your system.

 * GNU info directory index is up-to-date.
clevow230ss fitzcarraldo #

Summary

So, in summary, this is what I did in order to install and use app-emulation/wine-staging:

1. Add the following lines to /etc/portage/package.accept_keywords/wine:

virtual/wine ~amd64
app-emulation/wine-staging ~amd64
app-emulation/wine-vanilla ~amd64

2. Add the following lines to /etc/portage/package.use/wine:

virtual/wine -d3d9 staging abi_x86_32
app-emulation/wine-staging X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
app-emulation/wine-vanilla X alsa cups fontconfig gecko gphoto2 gsm jpeg lcms mp3 ncurses nls openal opengl perl png pulseaudio realtime run-exes scanner ssl staging threads truetype udev udisks v4l xcomposite xml -capi -custom-cflags -d3d9 -dos -gstreamer -ldap -mono -netapi -odbc -opencl -osmesa -oss -pcap -pipelight -prelink -s3tc -samba -selinux -test -themes -vaapi -xinerama abi_x86_32
# required by app-emulation/wine-vanilla-2.20::gentoo[gecko]
# required by virtual/wine-0-r6::gentoo
>=app-emulation/wine-gecko-2.47-r1 abi_x86_32

3. Uninstall the old, un-slotted WINE packages:

root # emerge -aC app-emulation/wine app-emulation/winetricks

4. Merge the new virtual WINE package:

root # emerge -a virtual/wine

5. Select the slotted WINE package I wish to use (WINE Staging):

root # eselect wine list
root # eselect wine set wine-staging-2.19

6. Merge the winetricks package, which will now recognise the slotted WINE package selected:

root # emerge app-emulation/winetricks

After completing the above process, I was still able to launch as before the various Microsoft Windows applications installed on my laptop under WINE.

UPDATE (December 1, 2017): As explained by user Chiltoo in a new post in the Gentoo Linux Forums thread mentioned earlier in the above post, a bug in Portage Version 2.3.13-r1 results in app-emulation/wine-vanilla being installed unnecessarily, as borne out in my post above.

The work-around he proposed does indeed work for me. Below are the commands I have just used for the work-around (I have left the new entries I made in package.accept_keywords and package.use in those files):

clevow230ss fitzcarraldo # emerge -aC app-emulation/wine-vanilla app-emulation/wine-staging virtual/wine app-emulation/wine-desktop-common app-emulation/wine-gecko app-eselect/eselect-wine app-emulation/winetricks
clevow230ss fitzcarraldo # emerge -1 app-emulation/wine-staging && emerge virtual/wine && emerge winetricks

The outcome of the above two commands in my case is:

clevow230ss fitzcarraldo # eix --installed --compact wine
[I] app-emulation/wine-desktop-common (20150204@01/12/17): Various desktop menu items and icons for wine
[I] app-emulation/wine-gecko (2.47-r1(2.47)@01/12/17): A Mozilla Gecko based version of Internet Explorer for Wine
[I] app-emulation/wine-staging (2.19(2.19)@01/12/17): Free implementation of Windows(tm) on Unix, with Wine-Staging patchset
[I] app-emulation/winetricks (20170823@01/12/17): Easy way to install DLLs needed to work around problems in Wine
[I] app-eselect/eselect-wine (1.2.2@01/12/17): Manage active wine version
[I] virtual/wine (0-r6@01/12/17): Virtual for Wine that supports multiple variants and slotting
Found 6 matches
clevow230ss fitzcarraldo # eselect wine list
Available wine versions:
  [1]   wine-staging-2.19 *
clevow230ss fitzcarraldo #

The superfluous app-emulation/wine-vanilla is no longer installed, which is a cleaner outcome.

Prevent Linux firewalls interfering with Samba commands in a home network that uses broadcast NetBIOS name resolution

Or “How come devices in a home network can browse SMB shares but Linux Samba commands and Windows nbtstat commands do not work properly?”

Introduction

In a previous post I explained how it is possible to browse SMB shares when using broadcast NetBIOS name resolution in a home network consisting of machines running Linux, Windows and other operating systems. Browsing SMB/Samba shares will work as expected, but Samba commands such as ‘smbtree‘, ‘smbclient‘ and ‘nmblookup‘ will not work properly if the Linux machines use a firewall that has not been configured for broadcast NetBIOS name resolution. This post is to explain how to do that.

If broadcast NetBIOS name resolution is being used and none of the Linux machines has a firewall enabled, or if their firewalls have been correctly configured, the output of e.g. the ‘smbtree‘ command on one of those machines would look something like the example below.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
                \\TUTANKHAMUN\Samsung_Xpress_C460FW     Samsung Xpress C460FW
                \\TUTANKHAMUN\Canon_MP560_Printer       Canon PIXMA MP560
                \\TUTANKHAMUN\Canon_MP510_Printer       Canon PIXMA MP510
                \\TUTANKHAMUN\Virtual_PDF_Printer       Virtual PDF Printer
                \\TUTANKHAMUN\IPC$              IPC Service (Samba 4.2.11)
                \\TUTANKHAMUN\Public
                \\TUTANKHAMUN\anne-share
                \\TUTANKHAMUN\print$
                \\TUTANKHAMUN\netlogon          Network Logon Service
        \\BTHUB5                        BT Home Hub 5.0A File Server
                \\BTHUB5\IPC$                   IPC Service (BT Home Hub 5.0A File Server)
        \\THUTMOSEIII                   Windows 10 computer

If Linux firewalls have not been correctly configured, the output would be missing some information about other machines in the network. For example, compare the output above with the output below from the same network, this time with the Linux firewalls configured using typical rules for Samba specified in Web articles, blog posts and forums.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
        \\BTHUB5                        BT Home Hub 5.0A File Server
        \\THUTMOSEIII                   Windows 10 computer

To avoid this problem you need to add a further Linux firewall rule to the set of rules usually used for Samba. Below I first list the usual firewall rules for Samba, then I give the additional rule necessary if using broadcast NetBIOS name resolution. In each case I give the applicable rules for a pure IPTABLES firewall and for UFW (Uncomplicated Firewall). The rules listed here assume the IP address range of the home network is 192.168.1.0/24, so change the range to suit the specific network.

Firewall rules typically specified for machines using Samba

IPTABLES

The rules listed below assume the machine uses interface eth0, so change the interface to suit the specific machine.

# NetBIOS Name Service (name resolution)
iptables -A INPUT -i eth0 -p udp --dport 137 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Datagram Service (BROWSER service)
iptables -A INPUT -i eth0 -p udp --dport 138 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Session Service (data transfer legacy SMB/NetBIOS/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 139 -s 192.168.1.0/24 -j ACCEPT

# Microsoft Directory Service (data transfer SMB/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 445 -s 192.168.1.0/24 -j ACCEPT

UFW

In some Linux distributions the ufw application allows a single command to add Samba support, such as:

user $ sudo ufw allow Samba

or

user $ sudo ufw allow CIFS

These ‘application profiles’ are specified in files in the directory /etc/ufw/applications.d/, so you could add application profiles or modify existing ones if you wish. In one of my installations the file /etc/ufw/applications.d/ufw-fileserver includes the following application profile for Samba, for example:

[CIFS]
title=SMB/CIFS server
description=SMB/CIFS server
ports=137,138/udp|139,445/tcp

If such an application profile does not exist in your installation, typical Samba rules can be added in UFW using the following two commands:

user $ sudo ufw allow from 192.168.1.0/24 to any port 137,138 proto udp
user $ sudo ufw allow from 192.168.1.0/24 to any port 139,445 proto tcp

The correct addition of the rules can be checked using the following command:

user $ sudo ufw status verbose
Password:
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
137,138/udp (CIFS)         ALLOW IN    192.168.1.0/24
139,445/tcp (CIFS)         ALLOW IN    192.168.1.0/24

The extra rule required when using broadcast NetBIOS name resolution

The reason why an extra rule is required when using broadcast NetBIOS name resolution is because UFW (which is based on IPTABLES) is ‘stateful’, as is a purely IPTABLES firewall (unless explicitly configured not to be stateful). The firewall does not consider packets it receives in response to its broadcast to be ESTABLISHED or RELATED, and therefore drops those packets. So, despite the IPTABLES and UFW rules listed above including a rule to accept incoming UDP packets on Port 137, any UDP packets received on Port 137 that do not constitute a one-to-one, two-way communication flow are dropped by the firewall. The extra rule below overrules this and makes the firewall accept packets coming from other devices’ Port 137 in response to broadcast NetBIOS Name Service packets. To do this, the extra rule uses a CT (Connection Tracking) helper named ‘netbios-ns‘ (obviously meaning ‘NetBIOS Name Service’). In order to use this rule the kernel must have been configured to use the IPTABLES ‘raw‘ table and to use CT (see the section ‘Kernel configuration’ further on).

IPTABLES

# All NetBIOS clients must have the netbios-ns helper enabled for broadcast name resolution to work
iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns

By the way, in addition to flushing the usual tables, flush the ‘raw‘ table too when you restart the firewall:

iptables -t raw -F OUTPUT

UFW

Add the following lines to the end of the file /etc/ufw/before.rules

# The following is needed to enable Samba commands to
# work properly for broadcast NetBIOS name resolution
#
# raw table rules
*raw
:OUTPUT ACCEPT [0:0]
-F OUTPUT
-A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns
COMMIT

Note that the output of the command ‘ufw status verbose‘ will not include the above rule. This is not a bug.

Kernel configuration

If you are using a binary-based distribution such as Ubuntu Linux, the kernel will probably have been configured to include the needed modules (CONFIG_IP_NF_RAW=m, CONFIG_IP6_NF_RAW=m and CONFIG_NETFILTER_XT_TARGET_CT=m), and the installation configured to load the modules automatically. However, if you are using a source-based distribution such as Gentoo Linux make sure the kernel configuration includes these three options before you build the kernel, and also add the module names ‘iptable_raw‘ and ‘xt_CT‘ to the module list in the file /etc/conf.d/modules as shown in the example below, so that the modules are loaded at boot:

modules="r8169 nvidia agpgart fuse bnep rfcomm hidp uvcvideo cifs mmc_block rtsx_pci snd-seq-midi vboxdrv vboxnetadp vboxnetflt iptable_raw xt_CT"

You can use the following two commands to check if the two modules are loaded:

user $ sudo lsmod | grep iptable_raw
user $ sudo lsmod | grep xt_CT

How to check the additional rule is active

You can use the command below whether you are using pure IPTABLES or UFW.

user $ sudo iptables -nvL -t raw
Password: 
Chain PREROUTING (policy ACCEPT 2613 packets, 1115K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 2773 packets, 475K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   16  1248 CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:137 CT helper netbios-ns

The packet and byte counts will increase whenever you use a Samba command.

Bibliography

  1. The netfilter.org "iptables" project
  2. Iptables Tutorial
  3. Introduction to IPTables
  4. Gentoo Wiki : iptables
  5. Arch Linux Wiki : Samba : "Browsing" network fails with "Failed to retrieve share list from server"
  6. Ubuntu : Manpage : ufw-framework
  7. Gentoo Wiki : UFW

xdotool comes to the rescue

In a previous post I explained how I implemented a method for adding my current location and the local time to my e-mail signature wherever I happen to be in the World, irrespective of the time on the laptop’s hardware clock and system clock. In that post I described how I created a keyboard shortcut using the Linux application AutoKey. Unfortunately AutoKey has not been updated for several years and no longer works properly in KDE Plasma 5 on my laptops. Therefore I decided to replace it with a KDE keyboard shortcut, and this is to explain how I did it.

First create a custom shortcut in KDE:

  1. ‘System Settings’ > ‘Shortcuts’ > ‘Custom Shortcuts’
  2. ‘Edit’ > ‘New’ > ‘Global Shortcut’ > ‘Command/URL’, and name the New Action ‘Insert current time’
  3. On the Comment pane for ‘Insert current time’, add the comment ‘Insert current time at specified location’ (without the quotes)
  4. On the Trigger pane, configure the shortcut to be Ctrl+Alt+Space
  5. On the Action pane, enter the Command/URL as ‘/home/fitzcarraldo/timezone_signature_GeoNames.sh‘ (without the quotes)
  6. Click ‘Apply’

Next modify the Bash script timezone_signature_GeoNames.sh so that it contains the following (obviously change the username and path to suit):

#!/bin/bash

place=$(kdialog --title "Current Location" --inputbox "Enter your location:")

placetime=$(perl /home/fitzcarraldo/now1.pl $place)

# xdotool does not output a space in a string, so we have to extract each field from the string
# and print each field individually, separated by a space character.

city=$(echo $placetime | awk -F "|" '{print $1}')
country=$(echo $placetime | awk -F "|" '{print $2}' | sed 's/[)(]//g')
region=$(echo $placetime | awk -F "|" '{print $4}')

datetime=$(/usr/bin/zdump $region | awk -F " " '{print $2" "$3" "$4" "$5" "$6" "$7}')
dayofweek=$(echo $datetime | awk -F " " '{print $1}')
month=$(echo $datetime | awk -F " " '{print $2}')
day=$(echo $datetime | awk -F " " '{print $3}')
time=$(echo $datetime | awk -F " " '{print $4}')
year=$(echo $datetime | awk -F " " '{print $5}')
timezone=$(echo $datetime | awk -F " " '{print $6}')

activewindow=$(xdotool getactivewindow)

xdotool type --window $activewindow "Sent from:"
for oneword in $city; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow comma
for oneword in $country; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow Return
xdotool type --window $activewindow "Local time now: "
xdotool type --window $activewindow $dayofweek
xdotool type --window $activewindow " "
xdotool type --window $activewindow $month
xdotool type --window $activewindow " "
xdotool type --window $activewindow $day
xdotool type --window $activewindow " "
xdotool type --window $activewindow $time
xdotool type --window $activewindow " "
xdotool type --window $activewindow $year
xdotool type --window $activewindow " "
if [ ${timezone:0:1} = "-" ]; then
    timezone="UTC-"${timezone#*-}
elif [ ${timezone:0:1} = "+" ]; then
    timezone="UTC+"${timezone#*+}
fi
xdotool type --window $activewindow $timezone
xdotool type --window $activewindow " "
xdotool key --window $activewindow Return
xdotool key --window $activewindow Return
echo

The Perl script now1.pl is listed in my my earlier post. Notice that the script timezone_signature_GeoNames.sh in my earlier post was much simpler. This was because the AutoKey shortcut took care of sending the text to the currently active window. Without AutoKey, I now had to do this myself in the script timezone_signature_GeoNames.sh, and the command xdotool came to the rescue. The developer explains what xdotool does as follows:

This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11’s XTEST extension and other Xlib functions.

Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

So I installed xdotool via the Gentoo package manager:

# emerge xdotool
# eix xdotool
[I] x11-misc/xdotool
     Available versions:  3.20150503.1-r1^t ~3.20160805.1^t {examples}
     Installed versions:  3.20150503.1-r1^t(22:51:30 02/04/17)(-examples)
     Homepage:            http://www.semicomplete.com/projects/xdotool/
     Description:         Simulate keyboard input and mouse activity, move and resize windows

Anyway, my Bash script using xdotool works a treat with Thunderbird (and KWrite, LibreOffice Writer, etc.). I used to experience a problem with certain characters, for example a colon was printed as a semi-colon (see the xdotool bug report xdotool writes the wrong case #121), but that no longer happens in my current KDE Plasma 5 installation:

Sent from: Gale√£o International Airport, Brazil
Local time now: Thu Jul 6 15:11:40 2017 UTC-03

What a useful tool xdotool is!

Stuttering audio in Linux: PulseAudio strikes again

I unmasked PulseAudio 10.0 back in January 2017 and installed it in my Gentoo Stable amd64 installation, and everything worked fine… until a couple of days ago, when the audio in streaming YouTube videos started to stutter every so often. It sounded rather like a scratched LP jumping. At first I thought the problem lay with Firefox, but the stuttering audio also occurred in Chrome. Then I wondered if my Internet connection was to blame; perhaps the ISP’s service had deteriorated. But a Windows 10 machine on my home network didn’t suffer from the problem, so that seemed to rule out the Internet connection. I tested the broadband throughput, and it was circa 32 Mbps, actually a little higher than the last time I tested it last year.

Now, Gentoo is a rolling distribution and I update my laptops regularly, but I couldn’t think what had been upgraded in the last couple of months that could be causing the problem. Although PulseAudio had not been upgraded since January, I began to wonder if PulseAudio could be involved, as my audio woes in the past have usually been due to PulseAudio.

I have always had PulseAudio installed with USE=”-realtime”:

user $ eix -I pulseaudio
[I] media-sound/pulseaudio
     Available versions:  10.0 {+X +alsa +alsa-plugin +asyncns bluetooth +caps dbus doc equalizer +gdbm +glib gnome gtk ipv6 jack libressl libsamplerate lirc native-headset neon ofono-headset +orc oss qt4 realtime selinux sox ssl system-wide systemd tcpd test +udev +webrtc-aec zeroconf ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  10.0(16:07:53 19/04/17)(X alsa alsa-plugin asyncns bluetooth caps dbus gdbm glib gnome gtk ipv6 jack orc qt4 ssl tcpd udev webrtc-aec zeroconf -doc -equalizer -libressl -libsamplerate -lirc -native-headset -neon -ofono-headset -oss -realtime -selinux -sox -system-wide -systemd -test ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://www.pulseaudio.org/
     Description:         A networked sound server with an advanced plugin system

but I wondered if PulseAudio’s real-time scheduling was somehow the cause of the problem, so I edited /etc/pulse/daemon.pa and added ‘realtime-scheduling = no‘ (I assume the default is ‘yes‘, as it was commented as such in the file):

; realtime-scheduling = yes
realtime-scheduling = no

Problem solved. PulseAudio is indeed a demon. ūüė°