Installing and using NeXT OPENSTEP in VirtualBox for Linux


Introduction and some history

My first micro computer was an Apple II+, which I used extensively both for work and leisure. In fact I liked it so much that I bought a //e when Apple Computer, Inc. released that model. I was not tempted by the Apple /// or Lisa when they were released, although I did quite fancy the IIGS but could not justify buying one. The //c was a nice portable, and a family member bought one on my recommendation. I was not at all tempted by the first Macintosh and subsequent models using the so-called Classic Mac OS, but I drooled when Steve Jobs founded NeXT, Inc. in 1985 and launched the magnesium-cased NeXT workstations: the cube-shaped NeXT Computer (Motorola 68030 CPU) in 1989 and in 1990 the second generation (Motorola 68040 CPU) NeXTcube and the NeXTstation (commonly referred to as ‘the slab’) running the NEXTSTEP operating system. The hardware build quality and aesthetic were fabulous, and the machines and NEXTSTEP were way ahead of their time. NEXTSTEP, which was built around Unix and therefore fully multi-tasking, looked amazing when compared to the competition and its performance was superior. Drooling was all I could do, though, because the price of any NeXT machine was totally out of my league.

OPENSTEP 4.2 Desktop in a VirtualBox VM

OPENSTEP 4.2 Desktop in a VirtualBox VM.

By the way, Tim Berners-Lee invented HTTP, HTML and the first HTML browser using NEXTSTEP on a NeXTcube at CERN: see The Science Museum, London – The World Wide Web: A global information space.

Following Apple’s acquisition in 1997 of NeXT, which by then was only a software company (NeXT Software, Inc.), Apple developed Mac OS X based on OPENSTEP (the successor to NEXTSTEP). Even today some of the features in macOS are the same as in NEXTSTEP and OPENSTEP: NeXTSTEP vs Mac OS X – System Demo and Comparison. The final release of NEXTSTEP was NEXTSTEP 3.3, succeeded by OPENSTEP, the final release of which was OPENSTEP 4.2. OPENSTEP was effectively NEXTSTEP 4.

So, even though the NeXT company only sold around 50,000 machines during its relatively short existence as a manufacturer between 1988 and 1993, its impact on modern computing has been significant. Below are a few links to interesting videos about the company and some of its products. You’ll find plenty more videos about NeXT on YouTube.

You can still find the occasional second-hand NeXT computer on eBay, but they are either incomplete or very expensive. As I write this there is a complete and pristine-looking NeXTcube system, including (non-working) NeXT laser printer, in Portugal listed on eBay at US$35,000 plus US$750 shipping! So I will never get to play with a real NeXT computer. But, thanks to VirtualBox, I can at least install the i386 release of OPENSTEP 4.2 in a VM (virtual machine) to try it out for fun. I decided to install the OS and the type of applications I would typically use (assuming I could find packages on the Web, that is). I wanted to find out how usable the OS was, how good the applications were, and whether I could access Unix easily from the GUI. As NeXT hardware and software are obsolete I had to spend a lot of time searching the Web for applications that would actually install and work. Some applications work in both NEXTSTEP and OPENSTEP, but plenty of applications have different packages for the two versions of the OS, which made my searches more complicated. Some OPENSTEP packages are so-called ‘fat binaries’ containing executables for some or all the different CPU types that OPENSTEP supported, and I found a few such packages on the Web. I wanted to install and try to use at least a Web browser, a word processor, a spreadsheet, an mp3 player and a video player. I also wanted to see if I could access files on a server on my home network using Samba.

There are quite a few tutorials and videos available on the Web explaining how to install OPENSTEP in a VM, but I did not find any on installing applications in OPENSTEP. Also, many of the OS installation tutorials I found are incomplete, for example not covering either audio or networking. I am not going to give a step-by-step explanation here of how I installed the OS and the applications, but I will explain what I installed, how I rated it, and any other information I found interesting or useful. Hopefully the tips I provide will be of some help if you fancy installing the OS and any applications yourself. I should also mention that you will have an advantage if you are a Unix and/or Linux user and are au fait with using the command line. OPENSTEP 4.2 provides the C Shell (csh). I did come across a package for the Bourne Again Shell (bash), but have not tried to install it. Sometimes I had to resort to the Unix command line to change ownership or permissions of a file and to move applications to folders owned by the root user. The pwd, cd, ls, su, cp, mv, chmod and chown commands came in handy a few times. By the way, unlike Linux the ls -la command does not display the group to which a file belongs, only its owner; you need to use the command ls -lag to show both. Also, the chown command accepts the notation owner.group but not owner:group when changing attributes.

Installation of OPENSTEP/Mach 4.2 for Intel i386 in VirtualBox

‘Mach’ refers to the Mach kernel, a microkernel developed at Carnegie Mellon University. OPENSTEP was available for Motorola 68k, Intel i386 and Sun SPARC CPUs. VirtualBox supports both 32-bit and 64-bit Intel CPUs, so the 32-bit OS can be installed in a VirtualBox 32-bit VM. NEXTSTEP also supported Hewlett-Packard’s PA-RISC CPU, but NeXT dropped support for that CPU in OPENSTEP.

Regarding the spelling of the two OSs, apparently the APIs are spelt ‘NeXTStep’ and ‘OpenStep’, and the OSs are spelt ‘NEXTSTEP’ and ‘OPENSTEP’. Confusing, or what? It’s no wonder these are used interchangeably all over the Web.

I found a reasonable tutorial on the installation of OPENSTEP 4.2, including links to download the image files of the CDROM and floppy disks required. Unlike many tutorials on the Web, it also explains how to get network access working, and I was able to ping other nodes on my home network and the Internet once I had completed the tutorial: ‘Installing NextStep OS (OPENSTEP) in VirtualBox‘. There were only one or two minor differences between the tutorial and what I saw on screen, and installation in VirtualBox for Linux was essentially painless. One of the packages that has to be installed (OS42MachUserPatch4.pkg) includes a Y2K patch for the OS. The tutorial tells you to use the command line to install that package, and I followed the instructions in the tutorial but, having now learned how to install packages via the OPENSTEP GUI by selecting a package and then ‘Services’ > ‘Open Sesame’ > ‘Open As Root’ > ‘Login’ to launch the Installer, I could have used only the GUI instead of the command line to install OS42MachUserPatch4.pkg (which I have checked). No matter, though, because using the OPENSTEP command line in Terminal.app is a good learning exercise. The tutorial does not mention some other things I had to configure in VirtualBox. To get audio working I had to select ‘SoundBlaster 16’ for the Audio Controller, install a driver in OPENSTEP and reboot the VM (see details further on), and under ‘Network’ in VirtualBox Manager I had to select ‘Bridged Adapter, PCnet-PCI II (Am79C970A)’ with ‘Promiscuous Model: Allow All’. I also enabled ‘Serial Ports’ and disabled ‘USB Controller’ (USB had not yet been invented back then!).

The OS installer installs US English support and offers the option of installing support for any of five other languages too: Swedish, Spanish, Italian, German and French. I unticked all those and completed the installation. Later I decided it might be useful to have support for those additional languages, and I found it very easy to install them retrospectively: I simply loaded the OPENSTEP-Install-4.2.iso file into the VM’s ‘optical drive’, browsed the CDROM’s contents, selected Upgrader.app and then ‘Workspace’ > ‘File’ > ‘Open as Folder’. I found the language packages (SwedishEssentials.pkg etc.) in the folder ‘NextCD’ > ‘Packages’. I could then select each language package and use ‘Services’ > ‘Open Sesame’ and so on to install it, as explained earlier.

To get sound working in OPENSTEP running in VirtualBox the procedure given in a 2009 tutorial ‘Installation of OPENSTEP 4.2 in VMware 3.0 and VirtualBox‘ miraculously still worked for me:

Audio: Alejandro Diaz Infante (aka astroboy) managed to make the OPENSTEP Sound Blaster driver work under VMWare and VirtualBox.
The solution: use the drivers created by University of Glasgow (Thanks, developer(s) of them, wherever you are, for drivers you never imagined would be so useful in the future).

  1. Download SBSoundMidi.I.b.tar.gz and SBMixer.I.tar.gz
  2. Install SBSoundMidi driver for either Vibra16Cpnp or AWE32pnp. Both work great! (I use the default irq and io, but the second DMA I put it on 7, ’cause it was the detected one when used VMWare to test Windoze. Anyway, I didn’t detect any failure when using the second DMA in its default of 5, so I guess it could be up to you. In VirtualBox I didn’t change any default setting, just select the driver “SoundBlaster 16” in VirtualBox audio setting before installing.
  3. Install SBMixer to have better control of your sound card.

That’s it. Put those audio CD’s and multimedia apps back!

After copying SBSoundMidi.I.b.tar.gz to OPENSTEP I double-clicked on it to unpack it, and then double-clicked on SBSoundMidi.config to install the SoundBlaster 16 drivers. I then navigated to ‘openstep’ > ‘NextAdmin’ > ‘Configure.app’, selected the loudspeaker icon and specified the driver ‘SBSoundMidi driver for SoundBlaster AWE32 PnP (v3.38)‘.

SBMixer works, and OPENSTEP’s Sound Inspector can play .snd files without having to install additional software, although I found that some .snd files would not play completely. TheNeXTSong.snd (16-bit Linear format) which I downloaded from one of the OPENSTEP software repositories on the Web (see links at the end of this post) plays perfectly (and is amusing), but the shorter Welcome-to-the-NeXT-world.snd (8-bit muLaw format) stalls. I did manage to install a couple of audio players (see further down).

The only minor problem that occurs every time you login if the floppy disk drive is empty is a pop-up window with the message ‘The floppy disk is unreadable’. You can just click on ‘Eject’ but, to stop this happening, you can change the boot order in VirtualBox Manager and load one of the OPENSTEP floppy disk image files in the VM’s floppy disk drive (‘Settings…’ > ‘Storage’ > ‘Floppy Drive’ in VirtualBox Manager). Actually, I copied Driver_Floppy.img to Work_Floppy.img, loaded the latter in the VM’s floppy disk drive and I changed the Boot Order from ‘Floppy’|’Optical’|’Hard Disk’ to ‘Hard Disk’|’Optical’|’Floppy’ (‘Settings…’ > ‘System’ > ‘Motherboard’ > ‘Boot Order’ in the VirtualBox Manager). Furthermore, although not essential, I selected Work_Floppy in File Viewer, then in the Workspace menu I selected ‘Disk’ > ‘Initialize…’ and initialised (formatted) the floppy disk. Its icon disappears momentarily from File Viewer, then reappears after it has been formatted.

The command ifconfig on my VM host computer running Lubuntu 18.04 tells me that the IP address of the host machine is 192.168.1.74 (I had previous configured my router to always assign this address to this machine), the netmask is 255.255.255.0 and the broadcast IP address is 192.168.1.255. My router’s Management page in a Web browser has the DHCP network range configured as 192.168.1.64 – 192.168.1.253, so I decided the OPENSTEP VM would have a static IP address of 192.168.1.63. The router’s Management page also told me that the ISP’s Primary DNS IP address is 81.139.57.100 and the Secondary DNS IP address is 81.139.56.100. Therefore, in accordance with the OPENSTEP installation tutorial I followed, I edited the file /etc/hostconfig in OPENSTEP to have the following shell variables:

# /etc/hostconfig
#
# This file sets up shell variables used by the various rc scripts to
# configure the host.  Edit this file instead of rc.boot.
#
# Warning:  This is sourced by /bin/sh.  Make sure there are no spaces
#           on either side of the "=".
#
# There are some special keywords used by rc boot and the programs it
# calls:
#
#       -AUTOMATIC-     Configure automatically
#       -YES-           Turn a feature on
#       -NO-            Leave a feature off or do not configure
#
HOSTNAME=openstep
INETADDR=192.168.1.63
ROUTER=192.168.1.254
IPNETMASK=255.255.255.0
IPBROADCAST=192.168.1.255
YPDOMAIN=-NO-
NETMASTER=-NO-
TIME=-AUTOMATIC-

I also created the file /etc/resolv.conf as specified in the tutorial, containing the following two lines with the ISP’s nameserver IP addresses I found from my router:

nameserver 81.139.57.100
nameserver 81.139.56.100

It was not specified in the tutorial, but to get NFS working later I found it was necessary to edit the file /etc/hosts to comment out the list of IP addresses and to add the hostname I had chosen (openstep) for the OPENSTEP VM plus the IP address (192.168.1.74) and hostname (aspirexc600) of the VM host machine running Lubuntu 18.04:

#
# NOTE: This file is never consulted if NetInfo or Yellow Pages is running.
#
#
# To do anything on the network, you need to assign an address to your
# machine.  This default host table will get you started.  "myhost"
# can be used for the first machine on the network, and client[1-8]
# can be used for subsequent machines.  You must make sure that no two
# machines have the same address.  If you need to add more machines
# just keep adding entries.  Each digit in the four digit number must
# be between 1 and 254 inclusive.
#
#192.42.172.1	myhost
#192.42.172.2	client1
#192.42.172.3	client2
#192.42.172.4	client3
#192.42.172.5	client4
#192.42.172.6	client5
#192.42.172.7	client6
#192.42.172.8	client7
#192.42.172.9	client8
#
# This is the reserved address for the loopback interface.  Don't muck
# with it.
#
127.0.0.1       localhost       openstep
192.168.1.74    aspirexc600

While setting up networking in the VM I also temporarily disabled the firewall in the VM host to make sure the VM host was not interfering in any way with the network connection of the VM, then enabled it again once I was happy it was not causing any problems. Later, when I configured the VM host as an NFS server and the VM as an NFS client, I had to create the appropriate rules for NFS in the VM host’s firewall (see further down).

You will see NetInfo mentioned in the OPENSTEP networking apps. You should ignore NetInfo unless you are going to network a cluster of machines running NEXTSTEP/OPENSTEP, as it is an obsolete NeXT networking system configuration database and we don’t want to use it.

Installation of utilities and applications

After installing the OS neither the ‘me’ account nor the root account are password protected. You can use the OS like this if you wish, but I set up a password for the ‘me’ account by navigating to ‘openstep’ > ‘NextApps’ > Preferences.app and clicking on the padlock icon. Then I logged out and logged in to the root account and did the same to set up a password for the root user. If you want to save a bit of time during installation of applications, you could do this after installing all the packages.

OPENSTEP comes with quite a few utilities, such as Terminal.app, TextEdit.app, Draw.app, Sound.app (possibly useful if your host computer has a microphone socket and you enabled audio input in VirtualBox Manager), PhotoAlbum.app, CDPlayer.app, Webster.app (yes, a full dictionary), Librarian.app, PrintManager.app, Grab.app (to grab snapshots of all or parts of the screen and save them to .tiff files), Preview.app (an image file viewer), Mail.app, and others. You can try these and they are reasonably intuitive so I won’t dwell on them here, instead concentrating on how I installed third-party apps and utilities.

I had to trawl the Web to find packages and applications suitable for OPENSTEP/Mach 4.2 for i386. I find the filenames of the files stored on these Web sites confusing. I think.s‘ in the filename of a compressed file means it contains source code, and ‘.b‘ means it contains binary code, i.e. executable. However, some filenames have ‘.bs‘ but only contain source code, so I could be wrong. Also, I’m not sure what the letters ‘N‘, ‘I‘, ‘H‘ and ‘S‘ represent in these filenames; NeXT (Motorola 68k), Intel, Hewlett-Packard PA-RISC and SPARC, presumably? Some OPENSTEP packages are called ‘fat binaries’ as they contain binaries for several or all the supported CPU types, thus enabling the package to be installed in OPENSTEP on different hardware. So my guess about the letters in the filenames could be correct.

Without a Web browser in OPENSTEP, the easiest way to copy files to the OPENSTEP VM initially is to use the Linux mkisofs command to create an ISO file and then to load it into the VM’s optical drive. For example, let’s say I want to copy the file OpenUp-1.01.tar to the VM, I would type the following on the host machine:

$ mkdir ~/ToCopy
$ cp ~/Downloads/OpenUp-1.01.tar ~/ToCopy
$ mkisofs -o ToCopy.iso ~/ToCopy

I then use the VirtualBox Manager GUI (‘Settings’ > ‘Storage’ > ‘Choose Virtual Optical Disk File…’) to insert the ToCopy.iso file into the VM’s optical drive. OPENSTEP mounts the ‘CDROM’ automatically and it becomes visible in the OPENSTEP File Viewer window. When I click on the CDROM icon a window opens and I see it contains the file openup_1.tar which I can then drag to the Shelf or to another folder directly.

Packages for installation using the OPENSTEP Installer have a ‘.pkg‘ suffix (e.g. ParaSheet.pkg) and are actually a folder, not a file. Applications have a ‘.app‘ suffix (e.g. ParaSheet.app) and are also a folder, not a file. Some of the compressed files I found for OPENSTEP on the Web are tarballs of OPENSTEP packages (e.g. OpenWrite.2.1.8.NIHS.b.tar.gz contains OpenWrite.pkg), others are tarballs of OPENSTEP applications (e.g. mpap.1.0.m.I.b.tar.gz contains mpap.app) which require unpacking but no installation, just copying to a folder. The mkisofs command truncates filenames to the Short Filename format (a.k.a. DOS 8.3 format), so if I had any uncompressed .pkg files, .app files and indeed any other files (.pdf, .mp3 or whatever) to transfer to the VM, I compressed them first as .tar files before creating the .iso file. Even though the .tar filename is truncated to DOS 8.3 by mkisofs, the filenames of the packed files are not.

Installing a package in OPENSTEP 4.2.

a) Installing a package in OPENSTEP 4.2.

Installing a package in OPENSTEP 4.2.

b) Installing a package in OPENSTEP 4.2.

Once you get the hang of installing packages in OPENSTEP, it is actually simple. For example, to install the package ParaSheet.pkg, I drag the .tar file from the CDROM to the Shelf, and from there to the folder /me. I double-click on the .tar file which opens a window showing the ParaSheet.pkg inside. I drag that to the /me folder. Then I select the package, and select ‘Workspace’ > ‘Open Sesame’ > ‘Open As Root’ > ‘Login’ and the Installer GUI opens. I then click on ‘Set…’ to specify the folder into which I want to install the application (e.g. /LocalApps/Office, as I had created the Office folder beforehand using Terminal.app) and then ‘Install’, and the Installer takes care of the rest.

In the case of applications that are not packaged and are just .app folders, I do not need to use the Installer, I just copy the .app folder to the folder I wish (/LocalApps/, /me/LocalApps/ or just /me/).

I found that, as-installed, OPENSTEP 4.2 can unpack .tar files from the GUI but does not have a GUI app for unpacking .tar.gz files, so the first thing I did was to install the OpenUp utility: OpenUp-1.01.m.NI.b.tgz which can be found at http://www.nextcomputers.org/NeXTfiles/Software/OPENSTEP/Apps/Compression_Utilities/ and works very well. Of course, I could have instead unpacked .tar.gz files in the host machine first and copied the .tar files to OPENSTEP using the mkisofs method I explained above, which the OPENSTEP GUI can unpack when I double-click on the .tar file. But OpenUp is well worth installing. After I had installed OpenUp and the OmniWeb browser in OPENSTEP, I was also able to download .tar.gz files directly in OPENSTEP from the various file repositories on the Web (see links at the end of this post) and unpack them in OPENSTEP.

By the way, see the links at the end of this post for user documentation. The OPENSTEP GUI is intuitive but I didn’t realise I could rename files from the GUI by clicking on the filename below the icon to get a cursor and typing directly (just like macOS), and I also didn’t know that I could use the ‘shelf’ at the top of the File Viewer as a temporary place to put copies of files to copy files between folders as an alternative to opening another File Viewer window. I also wondered how to select multiple files in a window when they are not adjacent, since using the mouse to select the group of files is not feasible in that case. It turns out the you hold down the Shift key and click on each file you want to select, which is analogous to holding down the Ctrl key and clicking on each file in Linux. I also found that I can copy a file between two File Viewer windows by clicking on it and holding down the Alt Gr key then dragging across to the other window.

Installation of a Web browser

This is where things start to get trickier. Bear in mind that NEXTSTEP and OPENSTEP were created in the 1980s and 1990s when the Web was in its infancy. As I mentioned earlier, the first Web browser was written on a NeXTcube at CERN, and that machine was the first Web server in existence. The best Web browser I could find for the platform is OmniWeb 3.1 for OPENSTEP. Before installing it, you need to install Omni Frameworks 1998G2. Also, the browser does not support HTTPS, Javascript and Flash out of the box and you have to install plugins. Unfortunately the plugins for these are very flaky, so you are severely limited in which sites and pages you can browse. Note that Netscape Communications created HTTPS in 1994, Netscape Communications and Sun Microsystems released JavaScript in December 1995, and Macromedia released Flash in November 1996. I don’t know if the OmniWeb plugins for HTTPS, JavaScript and Flash for OPENSTEP that I found are the latest or best versions for this version of OmniWeb, but they are what I could find online. JavaScript in Web pages results in a lot of pop-up error messages and made opening pages even less likely to be successful, so in the OmniWeb menu I navigated to ‘Info’ > ‘User Preferences…’ > ‘JavaScript’ and unticked ‘Display panel for errors’. I also navigated to ‘Info’ > ‘Administrator Preferences’ > ‘HTTPS – SSL’ and ticked ‘Enable TLSv1’, which seemed to enable a few HTTPS Web pages to load, at least partially.

You have to install OpenSSL before installing the HTTPS plugin for OmniWeb. I installed the package OpenSSL.0.9.5a.m.NIS.b.tar.gz which I downloaded from http://www.nextcomputers.org/NeXTfiles/Software/OPENSTEP/Apps/Internet/WWW/Web%20Browsers/Omniweb/Plugins/. Then I installed the package HTTPS.1.09b.m.NIS.b.tar.gz from the same site, which installs the file (folder) HTTPS.plugin, which needs to be in the folder /LocalLibrary/Plugins/ (‘NEXTSTEP’ > ‘LocalLibrary’ > ‘Plugins’).

Then I downloaded and installed the two packages JavaScript-OWPlugin-1999-07-20-OSM-NIS.tar.gz (installs JavaScript.plugin) and Flash-OWPlugin-19990621-OSM-NIS.tar (installs Flash.plugin) which also need to be in the folder /LocalLibrary/Plugins/ (‘NEXTSTEP’ > ‘LocalLibrary’ > ‘Plugins’ in the File Viewer). I found these two packages via a BetaArchive post [offer] OmniGroup software (NeXTSTEP, OpenStep & Rhapsody), which has a link to a .rar file at http://www.mediafire.com/file/wzyon54l4dt/OmniGroup.rar/file.

Unfortunately, even with the HTTPS and JavaScript plugins installed, almost all Web pages fail to load in OmniWeb, one exception being https://www.google.com. Old HTTP Web sites do load providing they are simple, but any JavaScript seems to cause a problem.

Installation of a PDF file reader

The best PDF file reader I could find for the platform is OmniPDF 3 for OPENSTEP. If you have not already installed Omni Frameworks, you first need to install Omni Frameworks 1998G2.

Installation of an image viewer

The best (supposedly) image file viewer I could find for the platform is OmniImage 4.0 for OPENSTEP. If you have not already installed Omni Frameworks, you first need to install Omni Frameworks 1998G2. However, according to the file /OmniImage.pkg/OmniImage.info it is a beta release and, in addition to Omni Frameworks, requires ‘Omni Plugins’:

Title OmniImage 4.0 beta for OPENSTEP/Mach 4.2
Version 4.0 beta 4 (1-Oct-1998)
Description This package contains a beta version of OmniImage. This beta release only supports viewing of images, not saving them. This release will not run unless the the Omni Frameworks (version 1998G2) are installed, and will not be fully functional (e.g., images may not be rendered) unless the Omni PlugIns (version 3.0 beta 8) are also installed. This software requires OPENSTEP/Mach 4.2.

I found the file OmniPlugIns-3.0b8-OSM-NIS.pkg.tar.gz in the BetaArchive post mentioned earlier in this post. I downloaded the tarball, created an ISO file containing it, loaded the ISO file in the VM CDROM drive, unpacked the tarball to /me/OmniPlugIns.pkg and installed the package using the OPENSTEP GUI Installer using the procedure explained earlier in this post. The Omni PlugIns were installed in the folder /LocalLibrary/PlugIns/ and I then found that OmniImage can open JPG files, even a 3456×2304 pixel JPG file with the following properties (as reported by the file command in Linux):

JPEG image data, JFIF standard 1.01, resolution (DPI), density 300x300, segment length 16, Exif Standard: [TIFF image data, big-endian, direntries=4, manufacturer=Canon, model=Canon EOS 600D], baseline, precision 8, 3456x2304, frames 3

Installation of wordprocessor and spreadsheet apps

OpenWrite and ParaSheet in use

OpenWrite and ParaSheet in use.

I created the folder /LocalApps/Office/ and installed OpenWrite from OpenWrite.2.1.8.NIHS.b.tar.gz which I downloaded from Index of /OpenStep/Soft/misc/NEXTTOYOU/97.1-Fruehjahr/APPSTOYOU. If you have not already installed it, before installing these apps you need to install Omni Frameworks 1998G2.

In the folder /LocalApps/Office/ I also installed ParaSheet from ParaSheet-1.7.pkg.tar.gz which I downloaded from Index of /NeXTfiles/Software/NEXTSTEP/Apps/Lighthouse_Design/ParaSheet. If you have not already installed Omni Frameworks, before installing these apps you need to install Omni Frameworks 1998G2.

The first time you launch OpenWrite and ParaSheet you will be notified that you cannot use the application until you enter a licence key. Exit the application and use ‘Open Sesame’ (see earlier) to launch the application as root user, and then you well be able to enter the licence. You will find a list of licences for these packages on the Web page Index of /NeXTfiles/Software/NEXTSTEP/Apps/Lighthouse_Design.

Installation of audio players

mpap and MMP audio players in action

mpap and MMP audio players in action.

The only audio players I could find that actually worked (partially) in OPENSTEP are mpap 1.0 (download mpap.1.0.m.I.b.tar.gz) and MMP 2 (download mmp2.I.b.tar.gz). mpap can play some, but not all, of the mp3 files I have, whereas I could not get MMP to play mp3 files at all, although it can play .snd files. MMP can also play MIDI files, but I had to download the Timidity patches instruments.tar.gz (not so easy to find!) and follow the instructions in the MMP Info Panel in order to install the instruments patch file. It works fine! mpap cannot play an mp3 file which the files command in Linux tells me is an ‘Audio file with ID3 version 2.4.0, contains:MPEG ADTS, layer III, v2.5, 32 kbps, 11.025 kHz, Stereo’ but it can play an mp3 file which is an ‘Audio file with ID3 version 2.4.0, contains:MPEG ADTS, layer III, v1, 192 kbps, 44.1 kHz, Stereo’. mpap has a basic playlist feature, but it is not as sophisticated as any of the modern audio players.

Installation of video players

MPLAY and Movie players in action

MPLAY and Movie players in action.

This is where OPENSTEP is severely lacking in comparison to any modern OS; apparently we’re talking 5.5 or 6 frames per second and e.g. 288×224 pixels on NeXT hardware, and no sound. I only managed to find a couple of basic video players, both at Index of /OpenStep/Soft/video/apps: MPlay 3.0 (MPlay.app unpacked from MPlay.3.0.NIHS.b.tar.gz) and Movie 3.0 (Movie3.0 folder unpacked from Movie.3.0.NIHS.bs.tar.gz). MPlay is only designed to play MPEG (.mpg and .mpeg) files, which I found it can do for the old, tiny MPEG files I downloaded from Web repositories of NEXTSTEP/OPENSTEP files. I found that Movie can also only play MPEG files, despite the app’s README file stating it can play (without sound) MPEG, TIFF sequences, ‘QuickTime and other formats’. Movie comes with a couple of demo videos (no audio), the largest of which is hula_full.mpg in the mpeg1video format, consisting of 39 frames of 352×240 pixels, with a desired frame rate of 8 fps which actually plays at between 8 and 9 frames per second in OPENSTEP in the VM, i.e. it plays for around 4 to 5 seconds. In a video player in Linux on my desktop machine it plays for just over 2 seconds at 15 frames per second. These videos and players may have been state-of-the-art in the 1980s and early 1990s, but they certainly are not now!

I could not find an app package to play .avi files. The page I linked to above has a source-code tarball named VideoStreamV1.OSrc.tar.gz for an app named VideoStream, the README of which claims the app can play .avi files, but I have not found an executable package. Anyway, the README file states it cannot play videos with sound, so obviously I didn’t bother trying to install it.

Games

I am not particularly interested in computer games, but a few are installed by default with the OS: Chess.app, Billiards.app and BoinkOut.app (a clone of Breakout). More games for OPENSTEP can be found on the Web (for example at Index of /OpenStep/Soft/). The computer game Doom was originally developed in NEXTSTEP on NeXT computers, and a version for OPENSTEP can be downloaded from the Web, although I have not tried it.

File sharing

NEXTSTEP/OPENSTEP was designed to use NFS (Network File System). However I don’t use NFS in my home network; I use SMB and have a dedicated Linux SMB server which works well with all SMB clients (Linux, Windows and Android) on my home network. Unsurprisingly I could only find early versions of Samba packages for NEXTSTEP and OPENSTEP. I also came across ramba, a Unix clone of Samba later renamed to Sharity-Light. I downloaded them both and briefly tried to get OPENSTEP to connect to my network Samba server. I was unsuccessful, which does not surprise me as the version of Samba for NEXTSTEP/OPENSTEP I found is Version 2.0.7.1 from May 2000, and the obsolete version of rumba I found is Version 0.4 from February 1997. In NEXTSTEP/OPENSTEP the Samba configuration file smb.conf is located in the directory /usr/samba/lib/ rather than /etc/samba/. I did not spend much time trying to get Samba/Rumba working as I assume there would be incompatibility between the early SMB protocol used by Samba V2.0.7.1 / Rumba V0.4 with Samba V4.* running in the Linux SMB server on my network. Perhaps I could have made it work, but I decided to try to make the VM’s host computer (192.168.1.74) a NFS server to see if I could get the VM (192.168.1.63) to access it as a NFS client. The Web page OpenStep on Microsoft Windows PC Emulators states the following, which indicates that NFS works:

Device: Network
OpenStep Configuration: AMD PCnet-32 PCI Ethernet Adapter
VirtualBox Configuration: Bridged Adapter, PCnet-PCI II, Promiscuous Mode All
Observations: This works fine. Using SimpleNetworkStarter I was able to give OpenStep an IP address on my subnet, using my real router and real DNS servers. This allowed OpenStep to be ‘seen’ on the subnet. Standard networking facilities such as FTP and NFS work. It may help to run the a command such as the following from the VirtualBox installation directory, where “OpenStep” is whatever you name the virtual machine and “192.168.1.0” depends on your local subnet:

VBoxManage modifyvm OpenStep --natnet1 "192.168.1.0/24"

As I had named the VM ‘OPENSTEP4.2’ in VirtualBox Manager, I used the following command:

$ VBoxManage modifyvm OPENSTEP4.2 --natnet1 "192.168.1.0/24"

However I doubt this made any difference, because I had set the VM’s network adapter to ‘Bridged Adapter’ in the VirtualBox Manager, not ‘NAT’. I had to select ‘Bridged Adapter’ because I could not get the VM to connect to the network otherwise.

I also made sure the adapter in the VirtualBox Manager is set to ‘PCnet-PCI II (Am79C970A)’ and Promiscuous Mode is set to ‘Allow All’.

In addition to the network configuration notes in the OPENSTEP installation tutorial I mentioned earlier, for information only see the old tutorial ‘NeXTStep/OpenStep Ethernet-Based Network Configuration For Cable Modems, DSL, LANs, Etc…‘.

Anyway, below is what I did to get NFS working. The crucial thing to note is that OPENSTEP 4.2 uses NFSv2. I spent many hours unsuccessfully trying to get NFS working between the NFS server (a machine with IP address 192.168.1.74) and the NFS client (a VM with IP address 192.168.63) until I realised this. The NFS server is running Lubuntu 18.04, which uses NFSv4 by default. Therefore I had to configure the NFS server to use NFSv2 as well. Not only that, but I had to configure NFSv2 to use static ports, because the ports can change randomly in NFSv2 which would stop NFS working if there is a firewall enabled on the host machine.

In the NFS server (Lubuntu 18.04 running on a desktop machine)

N.B. My NFS server is running in Lubuntu 18.04 on a machine with an IP address of 192.168.1.74, and my NFS client is running in OPENSTEP 4.2 on a VM with IP address of 192.168.1.63. Change the IP addresses below to suit your situation.

1. Install the NFS server software

$ sudo apt-get update
$ sudo apt-get install nfs-kernel-server

2. Create a mountpoint for the NFS shared directory

$ sudo mkdir /var/nfs
$ sudo chown nobody:nogroup /var/nfs
$ sudo chmod 777 /var/nfs

3. Configure the NFS export

$ sudo nano /etc/exports

3.1 Choose which of the following types of share you want to have

3.1.1 Less secure:

/home/fitzcarraldo/nfsshare 192.168.1.63(rw,sync,no_root_squash,no_subtree_check)

If ‘no_root_squash‘ is used, remote root users are able to change any file on the shared file system and leave trojaned applications for other users to inadvertently execute.

3.1.2 More secure:

/var/nfs 192.168.1.63(rw,sync,no_subtree_check)

3.2 Update the current table of exports for the NFS server

$ sudo exportfs -a

You can check the current table settings:

$ sudo exportfs -s
/home/fitzcarraldo/nfsshare  192.168.1.63(rw,wdelay,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash)
/var/nfs  192.168.1.63(rw,wdelay,root_squash,no_subtree_check,sec=sys,rw,secure,root_squash,no_all_squash)

If you wanted to clear the table (unexport the shared directories) you would do:

$ sudo exportfs -u 192.168.1.63:/home/fitzcarraldo/nfsshare
$ sudo exportfs -u 192.168.1.63:/var/nfs
$ sudo exportfs -s
$

4. Load the NFSv2 kernel module

If lockd is built as a module (which it is in Lubuntu 18.04), create file /etc/modprobe.d/nfsv2.conf containing the following:

options lockd.nlm_udpport=4001 lockd.nlm_tcpport=4001
$ sudo modprobe nfsv2

If you want to make that permanent so it happens automatically when booting/rebooting add ‘nfsv2‘ (without the quotes) to the file /etc/modules-load.d/modules.conf (which in Lubuntu 18.04 is symlinked to /etc/modules).

5. Configure the NFS server

See ‘How can I make the nfs server support protocol version 2 in Ubuntu 17.10?‘.

Edit /etc/default/nfs-kernel-server to include NFSv2 and to specify static ports:

$ sudo nano /etc/default/nfs-kernel-server
# Number of servers to start up
RPCNFSDCOUNT=8

# Runtime priority of server (see nice(1))
RPCNFSDPRIORITY=0

# Options for rpc.mountd.
# If you have a port-based firewall, you might want to set up
# a fixed port here using the --port option. For more information, 
# see rpc.mountd(8) or http://wiki.debian.org/SecuringNFS
# To disable NFSv4 on the server, specify '--no-nfs-version 4' here
RPCMOUNTDOPTS="--manage-gids -p 32767"
# -p 32767 above added by Fitzcarraldo

# Do you want to start the svcgssd daemon? It is only required for Kerberos
# exports. Valid alternatives are "yes" and "no"; the default is "no".
NEED_SVCGSSD=""

# Options for rpc.svcgssd.
RPCSVCGSSDOPTS=""

# All options below this comment were added by Fitzcarraldo
#
# Options to pass to rpc.statd
# ex. RPCSTATDOPTS="-p 32765 -o 32766"
RPCSTATDOPTS="-p 32765 -o 32766"
#
# Options to pass to rpc.rquotad
# ex. RPCRQUOTADOPTS="-p 32764"
RPCRQUOTADOPTS="-p 32764"
#
RPCNFSDOPTS="--nfs-version 2,3,4 --debug --syslog"
#
# To confirm above mods are in effect after service restart use
#    cat /run/sysconfig/nfs-utils
#  or 
#    service nfs-kernel-server status
#

Edit /etc/default/nfs-common to specify static ports for rpc-statd:

# If you do not set values for the NEED_ options, they will be attempted
# autodetected; this should be sufficient for most people. Valid alternatives
# for the NEED_ options are "yes" and "no".


# Options for rpc.statd.
#   Should rpc.statd listen on a specific port? This is especially useful
#   when you have a port-based firewall. To use a fixed port, set this
#   this variable to a statd argument like: "--port 4000 --outgoing-port 4001".
#   For more information, see rpc.statd(8) or http://wiki.debian.org/SecuringNFS
STATDOPTS="-o 32766 -p 32765"
# -o 32766 -p 32765 above were added by Fitzcarraldo

# Do you want to start the gssd daemon? It is required for Kerberos mounts.
NEED_GSSD=

(I had to edit /etc/default/nfs-common to specify the ports for rpc-statd in STATDOPTS because specifying the ports in RPCSTATDOPTS in /etc/default/nfs-kernel-server did not make the status ports static.)

Edit /etc/sysctl.conf to add a static port mapping for lockd:

$ sudo nano /etc/sysctl.conf
[...]
# All lines below added by Fitzcarraldo
# TCP Port for lock manager
fs.nfs.nlm_tcpport = 4001
# UDP Port for lock manager
fs.nfs.nlm_udpport = 4001

Modify the lockd kernel parameters now during runtime rather than having to reboot:

$ sudo sysctl -p

Note that it is necessary to specify static ports in the configuration files so that tight rules can be added to the firewall in the NFS server.

6. Start the NFS server

Either the sysvinit way, which still works in Lubuntu 18.04:

$ sudo service nfs-kernel-server start

or the systemd way, which also works in Lubuntu 18.04:

sudo systemctl start nfs-kernel-server

If you want, you could enable the service so it starts automatically after the system is rebooted:

$ sudo systemctl enable nfs-kernel-server

7. Start the NSM (Network Status Monitor) daemon

Either the sysvinit way, which still works in Lubuntu 18.04:

$ sudo service rpc-statd start

or the systemd way, which also works in Lubuntu 18.04:

$ sudo systemctl start rpc-statd

If you want, you could enable the service so it starts automatically after the system is rebooted:

$ sudo systemctl enable rpc-statd

8. Check that NFSv2 is running and the ports are the ones specified in the config files

$ rpcinfo -p
   program vers proto   port  service
    100000    4   tcp    111  portmapper
    100000    3   tcp    111  portmapper
    100000    2   tcp    111  portmapper
    100000    4   udp    111  portmapper
    100000    3   udp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  32767  mountd
    100005    1   tcp  32767  mountd
    100005    2   udp  32767  mountd
    100005    2   tcp  32767  mountd
    100005    3   udp  32767  mountd
    100005    3   tcp  32767  mountd
    100003    2   tcp   2049  nfs
    100003    3   tcp   2049  nfs
    100003    4   tcp   2049  nfs
    100227    2   tcp   2049
    100227    3   tcp   2049
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100227    2   udp   2049
    100227    3   udp   2049
    100021    1   udp   4001  nlockmgr
    100021    3   udp   4001  nlockmgr
    100021    4   udp   4001  nlockmgr
    100021    1   tcp   4001  nlockmgr
    100021    3   tcp   4001  nlockmgr
    100021    4   tcp   4001  nlockmgr
    100024    1   udp  32765  status
    100024    1   tcp  32765  status

9. Configure the firewall in Lubuntu 18.04

I used Gufw (LXDE Menu > ‘Preferences’ > ‘Firewall Configuration’) to add the following two UFW rules:

111,2049,4001,32765:32768/udp ALLOW IN 192.168.1.0/24
111,2049,4001,32765:32768/tcp ALLOW IN 192.168.1.0/24

The above rules permit NFSv2 to function consistently because I had configured the NFS ports to be static. If I had not done that the firewall would sometimes stop NFS from working because NFSv2 ports change randomly otherwise.

In OPENSTEP running in the VM

10. Make sure basic networking has been configured

I navigated to ‘openstep’ > ‘NextAdmin’ > ‘SimpleNetworkStartup.app’ and did the following:

  • Unticked ‘Maintain the master copy of network administrative data.’
  • Selected ‘Use the network, but don’t share administrative data.’
  • Entered the Hostname ‘openstep‘ (no quotes) and IP address 192.168.1.63.
  • Clicked on ‘Network Options…’. In the window that opened I did the following:
    • Made sure router IP is set to 192.168.1.254
    • Made sure NIS Domain Name is set to ‘None’
    • Made sure Netmask is set to 255.255.255.0
    • Made sure Broadcast Address is set to 192.168.1.255
    • ‘Limit access to local NetInfo data to the local network’ is unticked.
    • Clicked on ‘Set’.
  • Clicked on ‘Configure’.

11. Create the shared NFS director[y,ies]

N.B. I could probably have created the directory /mnt/nfs/nfsshare and/or /mnt/nfs/var/nfs (whichever you chose to create — see 3.1 above) using ‘openstep’ > ‘NextAdmin’ > ‘NFSManager.app’ instead of using the command line, but I opened a Terminal window in OPENSTEP and did the following:

openstep> su
openstep:1# mkdir /mnt
openstep:2# mkdir /mnt/nfs
openstep:3# mkdir /mnt/nfs/nfsshare
openstep:4# mkdir /mnt/nfs/var
openstep:5# mkdir /mnt/nfs/var/nfs

12. Mount the NFS share(s)

openstep:6# mount 192.168.1.74:/home/fitzcarraldo/nfsshare /mnt/nfs/nfsshare
openstep:7# mount 192.168.1.74:/var/nfs /mnt/nfs/var/nfs

Use the df command to check they are mounted correctly:

openstep:8# df

13. Test the shared director[y,ies]

In Lubuntu on the machine with hostname ‘aspirexc600‘, copy a file into /var/nfs/ (or /home/fitzcarraldo/nfsshare/). You should see it appear in /mnt/nfs/var/nfs/ (or /mnt/nfs/nfsshare/) in OPENSTEP in the VM with hostname ‘openstep‘.

In OPENSTEP on the VM with hostname ‘openstep‘, copy a file into /mnt/nfs/var/nfs/ (not /mnt/nfs/nfsshare/, as that will not be allowed). You should see it appear in /var/nfs/ in Linux in the machine with hostname ‘aspirexc600‘.

In Lubuntu on the machine with hostname ‘aspirexc600‘, delete the file in /var/nfs/ and you should see it removed from /mnt/nfs/var/nfs/ in OPENSTEP on the VM with hostname ‘openstep‘.

In Lubuntu on the machine with hostname ‘aspirexc600‘, delete the file in /home/fitzcarraldo/nfsshare/ and you should see it removed from /mnt/nfs/nfsshare/ in OPENSTEP on the VM with hostname ‘openstep‘.

14. If you later want to unmount the NFS shared folder(s)

openstep:9# umount /mnt/nfs/nfsshare
openstep:10# umount /mnt/nfs/var/nfs

15. If you want OPENSTEP to mount the NFS shared folder(s) automatically when it boots

I was unable to get OPENSTEP to mount NFS shared folders automatically at boot by adding the appropriate lines in /etc/fstab, but OPENSTEP does mount them automatically if I add the mount commands to /etc/rc.local like so:

#!/bin/sh -u
#
# This script is for augmenting the standard system startup commands. It is 
# executed automatically by the system during boot up. 
#
# Copyright (C) 1993 by NeXT Computer, Inc.  All rights reserved.
#
# In its released form, this script does nothing. You may customize
# it as you wish.
#

fbshow -B -I "Starting local services" -z 92

# Read in configuration information
. /etc/hostconfig

# (echo -n 'local daemons:')                                    >/dev/console
#
# Run your own commands here
mount 192.168.1.74:/var/nfs /mnt/nfs/var/nfs
mount 192.168.1.74:/home/fitzcarraldo/nfsshare /mnt/nfs/nfsshare
#
# (echo '.')                                                    >/dev/console

File sharing: Summary

So, I managed to get NFS working, albeit not using OPENSTEP’s NFSManager.app tool. Had I known more about OPENSTEP networking I probably could have used the OPENSTEP GUI utilities to configure NFS, but at least I have proved it is possible to copy files to and from an NFS server (which happens to be the host machine of the VM) running Lubuntu 18.04 and the VM running OPENSTEP 4.2. Mind you, NFSv2 is old. NFSv4 would be the protocol to use had OPENSTEP supported it. Also, bear in mind that NFSv2 cannot encrypt the connection, so it is not secure. Another reason to have a good firewall enabled in the VirtualBox host machine and in my router too.

Conclusions

I have had fun installing and tinkering with OPENSTEP and its applications over the last few days. Getting file sharing to work was by far the most difficult part, but I got there in the end once I had discovered OPENSTEP only supports NFSv2. It is a pity OPENSTEP and the applications for it have not been developed for many years and are all obsolete. If development of OPENSTEP drivers, networking software, productivity applications and multimedia applications had continued, the OS itself would still have been perfectly usable on modern hardware, albeit not as straightforward to use as any of the main Desktop Environments in Linux. But the OS still feels quite modern; it was definitely ahead of its time. Tinkering with OPENSTEP 4.2 has given me a new respect for Steve Jobs, for the talented hardware and software engineers in the NeXT company, and indeed for Mac OS X and macOS. The choice of Unix for NEXTSTEP/OPENSTEP was truely inspired.

In this blog post I have not covered the sophisticated development tools for NEXTSTEP/OPENSTEP, which were also way ahead of their time. I’ll leave you to read the articles, documents and videos available on the Web about the development tools.

Please comment below if you notice any errors or omissions in this post, or if you know a better way of doing something in OPENSTEP, or you know of newer versions of the OPENSTEP software than the versions I have mentioned. I’d also be interested to hear from anyone who has a NeXT machine and/or is still using one; let me know what you have and how you’re using it.

Useful links

These are just a few of the many Web pages and sites I browsed when installing OPENSTEP 4.2 and looking for applications and ways to get various things to work.

Documentation

Software repositories

Sometimes differences between NEXTSTEP and OPENSTEP may mean a NETSCAPE application cannot be installed in OPENSTEP or, if it can, may not work. Furthermore, be aware that different revisions of the same application/utility exist online, so you need to try and find the latest revisions.

How to enable a Windows application in WINE to access a Samba share on a NAS (continued)

In a 2016 post ‘How to enable a Windows application in WINE to access a Samba share on a NAS‘ I explained how to mount in Linux a networked SMB shared folder so that a Windows application running via WINE could access the folder as Drive Y: in order to open and save files in it. In that blog post I also listed a couple of Bash scripts to facilitate the mounting and unmounting of the SMB share for the WINEPREFIX used for the Windows application (~/.wine-pdfxve6 in the example I gave for PDF-XChange Editor, Version 6). However, as I have several Windows applications running via WINE on my machines, and I have used a different WINEPREFIX for each of them, I wanted to be able to mount the SMB share for whichever of those applications I happen to be using at the time. Therefore I modified the original Bash scripts as shown below. The Desktop Configuration files (.desktop files) to launch the scripts are essentially the same as in my earlier blog post; I have just removed the references to the specific Windows application. The four modified files are listed below. Obviously change the username, SMB share name and SMB server name to suit your own situation.

1. Bash script ~/mount_bsfnas1_brianfolder_share.sh

#!/bin/bash
mount_share () {
    echo
    echo "Enter your Linux account password below..."
    echo
    sudo ln -s /media/bsfnas1/brianfolder ~/$PREFIX/dosdevices/y:
    sudo mount.cifs //bsfnas1/brianfolder/ -o user=brianfolder,pass=enricocaruso,uid=$(id -u),gid=$(id -g) ~/$PREFIX/dosdevices/y:
}
echo
echo "This will mount the Samba share folder brianfolder on the bsfnas1 machine."
echo
echo
echo "== Select which WINEPREFIX you wish to use =="
echo
ls ~/.wine-* | grep .wine | awk -F'/' '{print NR " " substr($4, 1, length($4)-1)}'
NUMPREFIXES=$(ls ~/.wine-* | grep .wine | wc -l)
echo
read -p "Enter number (q to abort) and press ENTER: " CHOICE
if [ "$CHOICE" != "q" ] && [ "$CHOICE" -gt 0 ] && [ "$CHOICE" -le $NUMPREFIXES ]; then
    PREFIX=$(ls ~/.wine-* | grep .wine | awk -F'/' '{print NR " " substr($4, 1, length($4)-1)}' | grep "$CHOICE " | awk -F' ' '{print $2}')
    echo
    if [ ! -e ~/$PREFIX/dosdevices/y: ]; then
        mount_share
    else
        echo -n "~/$PREFIX/dosdevices/y: already exists. Is it OK to proceed anyway (y/n)? "
        read ANSWER
        if [ $ANSWER = "y" ]; then
            rm ~/$PREFIX/dosdevices/y:
            mount_share
        fi
    fi
    echo
fi
if grep -q "/media/bsfnas1/brianfolder" /proc/mounts; then
    echo "Samba share //bsfnas1/brianfolder is mounted for WINEPREFIX ~/$PREFIX ."
else
    echo "Samba share //bsfnas1/brianfolder is not mounted."
fi
echo
echo "You may now close this window."
read ANSWER
exit

2. Bash script ~/umount_bsfnas1_brianfolder_share.sh

#!/bin/bash
echo
echo "This will unmount the Samba share folder brianfolder on the bsfnas1 machine."
echo
echo "Enter your Linux account password below..."
echo
sudo umount ~/.wine-*/dosdevices/y: 2>/dev/null
echo
if grep -q "/media/bsfnas1/brianfolder" /proc/mounts; then
  echo "Samba share //bsfnas1/brianfolder is mounted."
else
  echo "Samba share //bsfnas1/brianfolder is not mounted."
fi
echo
echo "You may now close this window."
exit

3. Desktop Configuration file ~/Desktop/mount_bsfnas1_brianfolder_share.desktop

[Desktop Entry]
Comment[en_GB]=Mount bsfnas1 brianfolder share for current WINEPREFIX
Comment=Mount bsfnas1 brianfolder share for current WINEPREFIX
Exec=sh /home/fitzcarraldo/mount_bsfnas1_brianfolder_share.sh
GenericName[en_GB]=Mount bsfnas1 brianfolder share for current WINEPREFIX
GenericName=Mount bsfnas1 brianfolder share for current WINEPREFIX
Icon=media-mount
MimeType=
Name[en_GB]=mount_bsfnas1_brianfolder_share
Name=mount_bsfnas1_brianfolder_share
Path=
StartupNotify=true
Terminal=true
TerminalOptions=\s--noclose
Type=Application
X-DBUS-ServiceName=
X-DBUS-StartupType=none
X-KDE-SubstituteUID=false
X-KDE-Username=fitzcarraldo

4. Desktop Configuration file ~/Desktop/umount_bsfnas1_brianfolder_share.desktop

[Desktop Entry]
Comment[en_GB]=Unmount bsfnas1 brianfolder share for current WINEPREFIX
Comment=Unmount bsfnas1 brianfolder share for current WINEPREFIX
Exec=sh /home/fitzcarraldo/umount_bsfnas1_brianfolder_share.sh
GenericName[en_GB]=Unmount bsfnas1 brianfolder share for current WINEPREFIX
GenericName=Unmount bsfnas1 brianfolder share for current WINEPREFIX
Icon=media-eject
MimeType=
Name[en_GB]=umount_bsfnas1_brianfolder_share
Name=umount_bsfnas1_brianfolder_share
Path=
StartupNotify=true
Terminal=true
TerminalOptions=\s--noclose
Type=Application
X-DBUS-ServiceName=
X-DBUS-StartupType=none
X-KDE-SubstituteUID=false
X-KDE-Username=fitzcarraldo

Now when I double-click on the icon to mount the SMB share for a Windows application running via WINE, a terminal window pops up displaying the WINEPREFIXs currently installed on my machine:


This will mount the Samba share folder brianfolder on the bsfnas1 machine.


== Select which WINEPREFIX you wish to use ==

1 .wine-3dimviewer
2 .wine-myphoneexplorer
3 .wine-nbtscan
4 .wine-pdfxve6
5 .wine-PortableApps
6 .wine-radiant
7 .wine-symmetry
8 .wine-visio
9 .wine-xnviewmp

Enter number (q to abort) and press ENTER: 

Let’s say I want to use the Windows application XnViewMP. I would enter ‘9’ and press ‘Enter’. The rest of the interaction should be obvious:


This will mount the Samba share folder brianfolder on the bsfnas1 machine.


== Select which WINEPREFIX you wish to use ==

1 .wine-3dimviewer
2 .wine-myphoneexplorer
3 .wine-nbtscan
4 .wine-pdfxve6
5 .wine-PortableApps
6 .wine-radiant
7 .wine-symmetry
8 .wine-visio
9 .wine-xnviewmp

Enter number (q to abort) and press ENTER: 9

~/.wine-xnviewmp/dosdevices/y: already exists. Is it OK to proceed anyway (y/n)? y

Enter your Linux account password below...

[sudo] password for fitzcarraldo: 

Samba share //bsfnas1/brianfolder is mounted for WINEPREFIX ~/.wine-xnviewmp .

You may now close this window.

Henceforth the Windows application XnViewMP will be able to access the Y: drive which is actually the SMB share //bsfnas1/brianfolder.

Once I have finished using the application, I just double-click on the the icon to unmount the SMB share, and a terminal window pops up displaying the following:


This will unmount the Samba share folder brianfolder on the bsfnas1 machine.

Enter your Linux account password below...

[sudo] password for fitzcarraldo: 

Samba share //bsfnas1/brianfolder is not mounted.

You may now close this window.

Once I have entered my Linux password for the local machine, the script will unmount the SMB share and the terminal window will close automatically if you have configured the Desktop Configuration file by right-clicking on the icon and unticking ‘Do not close when command exits’ in KDE, ‘Keep terminal window open after command execution’ in LXDE, or similar in other desktop environments.

Note: If you use Microsoft Office via WINE, you also might be interested in a comment on my earlier blog post about a Microsoft Office problem in saving files to a remote SMB share.

Firewall zones (profiles) in Linux, and how to switch them automatically if you use UFW

Firstly, a note on terminology: UFW (Uncomplicated Firewall) and its two GUI front-ends Gufw and UFW Frontends use the term ‘application profile’ to refer to a pre-configured set of rules specified in a file. Files containing UFW application profiles are placed in the directory /etc/ufw/applications.d/. An application profile for SMB, for example, enables the root user to use the UFW command ‘ufw allow Samba‘ (‘ufw allow CIFS‘ in Gentoo Linux) rather than having to enter UFW commands specifying the precise ports and network protocols that SMB uses. However, this blog post is not about UFW’s application profiles; it is about what Gufw calls ‘profiles’ and firewalld calls ‘zones’.

In essence a profile/zone is a collection of firewall policies and rules. Both Gufw and firewalld include the concept of a ‘zone’, although Gufw uses the term ‘profile’ rather than ‘zone’. UFW Frontends does not have the concept of a ‘zone’; rules entered via UFW Frontends apply to any network to which you connect your laptop. The ability to define different zones for different networks is handy. For example, you can have certain policies and rules when your laptop is connected to your home network, and different policies and rules when your laptop is connected to the network in a café, hotel, airport or other public place.

An attractive feature of firewalld when used in conjunction with NetworkManager and KDE Plasma is that it is possible to use the desktop environment’s network management module (‘System Settings’ > ‘Connections’) to specify a particular firewalld zone for a particular network connection. For example, let’s say you used firewalld to specify certain policies and rules for a zone you named ‘office’, and you then specified in the System Settings – Connections GUI that a connection named ‘ACM’ should use the zone ‘office’. Thereafter, whenever you connect your laptop to the network named ‘ACM’, firewalld will use the policies and rules you previously configured for the zone ‘office’.

Unlike firewalld, Gufw does not have the ability to switch profiles automatically according to which network the laptop is connected. You have to select manually the profile you wish to use. You would launch Gufw prior to connecting to, for example, your office’s network, select the profile ‘Office’ (or whatever you have named it), then connect your laptop to that network.

I think many people would be satisfied with the functionality currently provided by Gufw. I could use the Gufw GUI to create Gufw profiles with names such as ‘Home’, ‘HomeDave’, ‘Public’, ‘HQoffice’, ‘USoffice’, ‘PestanaRio’ and so on, and specify the different policies and rules I want for each profile. At home I would launch Gufw on my laptop and select the Home profile then connect to my home network; in the office at work I would launch Gufw on my laptop and select the HQoffice profile then connect to the office network; at my friend Dave’s house I would launch Gufw on my laptop and select the HomeDave profile then connect to the house network; and so on. Nevertheless I do see the attraction of automated zone switching, as provided by firewalld in conjunction with NetworkManager and KDE. It would be handy if my laptop could switch automatically to the Home profile when my laptop connected to the network at my home with the name ‘BTHub5-8EUQ’, automatically switch to the HQoffice profile when my laptop connected to the network named ‘HQ-Office2’ in the office, and so on.

I use UFW on my two laptops running Gentoo Linux. The package ufw-frontends is also installed but normally I use UFW directly via the command line. However I wanted to learn about zones/profiles while using UFW, and I also wanted to see if I could automate the switching of zones without resorting to installing firewalld. NetworkManager has the ability to launch ‘hook’ scripts when certain things happen — when a network connection changes, for example — and this seemed to me to be a way of switching profiles automatically.

I had not used Gufw before, so I decided to install it. A package is available in many Linux distributions but there is no ebuild for Gufw in Gentoo’s main Portage tree and I could not find an up-to-date ebuild for it in any Portage overlays. Therefore I created the ebuild for net-firewall/gufw-19.10.0 shown below. It probably needs improving, but it does install a working Gufw in Gentoo Linux.

# Copyright 1999-2019 Gentoo Foundation
# Distributed under the terms of the GNU General Public License v2
# $Header: $

EAPI=7
PYTHON_COMPAT=( python3_{5,6,7} )
DISTUTILS_IN_SOURCE_BUILD=1

inherit distutils-r1

MY_PN="gui-ufw"
MY_PV="$(ver_cut 1-2)"

DESCRIPTION="GUI frontend for managing ufw."
HOMEPAGE="https://gufw.org/"
SRC_URI="https://launchpad.net/${MY_PN}/trunk/${MY_PV}/+download/${MY_PN}-${PV}.tar.gz"

LICENSE="GPL-3"
SLOT="0"
KEYWORDS="~amd64"
IUSE=""

DEPEND="dev-python/python-distutils-extra"
RDEPEND="net-firewall/ufw
	dev-python/netifaces
	dev-python/pygobject:3
	net-libs/webkit-gtk[introspection]
	sys-auth/elogind
	sys-auth/polkit
	x11-libs/gtk+:3[introspection]
	x11-themes/gnome-icon-theme-symbolic
"
S=${WORKDIR}/${MY_PN}-${PV}

pkg_postinst() {
	sed '/dist-packages/d' -i /usr/bin/gufw-pkexec
	sed -E '/\/share\//d' -i /usr/bin/gufw-pkexec
	local PYTHONVERSION="$(python -c 'import sys; print("{}.{}".format(sys.version_info.major, sys.version_info.minor))')"
	sed -E "s|python3\.[0-9]|python${PYTHONVERSION}|g" -i /usr/bin/gufw-pkexec
	sed -E 's|\/lib\/|\/lib64\/|g' -i /usr/bin/gufw-pkexec
}

How To Set Up a Firewall with GUFW on Linux‘ is a good tutorial on Gufw.

As I had not used Gufw previously, I had to play around with it to understand better its functional design. I found that if I configure rules directly via UFW on the command line without using Gufw, Gufw does not allow me to edit those rules (but does allow me to delete them) and those rules exist whichever Gufw profile is selected in the Gufw GUI. Gufw profiles are stored in files named ‘/etc/gufw/*.profile‘ (e.g. /etc/gufw/Home.profile) and these files will not include UFW rules entered via the command line. On the other hand, UFW rules created via the Gufw GUI apply solely to the currently-selected Gufw profile, which is what I would have expected. In other words, I can create a different set of policies and rules in each Gufw profile. Therefore I believe Gufw profiles (as distinct from UFW application profiles) are basically analogous to firewalld’s zones. It also appears to me that Gufw maintains configuration files specifying policies and rules independently of UFW, which Gufw then applies to UFW. In other words, if you are a Gufw user you should not use UFW directly to configure policies and rules, otherwise Gufw’s configuration files will not include what you did directly using UFW. To reiterate, use only Gufw or only UFW, not both.

The current Gufw profile’s name is listed in the file /etc/gufw/gufw.cfg. For example, I currently have the Home profile selected in the Gufw GUI, and the file gufw.cfg contains the following:

[GufwConfiguration]
profile = Home
windowwidth = 542
windowheight = 530
confirmdetelerule = yes

If I examine the contents of the file /etc/gufw/Home.profle I see that it contains the UFW policies and rules I specified for the Gufw Home profile:

[fwBasic]
status = enabled
incoming = deny
outgoing = allow
routed = disabled

[Rule0]
ufw_rule = 137,138/udp ALLOW IN 192.168.1.0/24
description = Samba
command = /usr/sbin/ufw allow in proto udp from 192.168.1.0/24 to any port 137,138
policy = allow
direction = in
protocol = 
from_ip = 192.168.1.0/24
from_port = 
to_ip = 
to_port = 137,138/udp
iface = 
routed = 
logging = 

[Rule1]
ufw_rule = 139,445/tcp ALLOW IN 192.168.1.0/24
description = Samba
command = /usr/sbin/ufw allow in proto tcp from 192.168.1.0/24 to any port 139,445
policy = allow
direction = in
protocol = 
from_ip = 192.168.1.0/24
from_port = 
to_ip = 
to_port = 139,445/tcp
iface = 
routed = 
logging =

I also notice that the other Gufw profiles can differ. For example, my Office.profile file contains the following:

[fwBasic]
status = enabled
incoming = deny
outgoing = allow
routed = allow

The profile name listed in gufw.cfg gets changed when the user changes the profile using the Gufw GUI. It appears to me that only at the point in time when the user selects a certain Gufw profile in the Gufw GUI does Gufw parse the applicable *.profile file and issue commands to UFW to implement the policies and rules specified in the *.profile file.

Initially I tried to automate the process of changing the Gufw profile by doing the following:

  1. I created a NetworkManager Dispatcher hook script to:

    1. detect when the laptop connects to a network;

    2. determine whether the network is at my home, at my workplace or in a public place (café, airport or wherever) by looking at the connection name;

    3. edit gufw.cfg to change the name of the Gufw profile according to the network connected.
  2. I configured KDE to launch Gufw automatically at login, hoping that would implement the Gufw profile specified in gufw.cfg.

When I connected the laptop to various networks, Gufw did indeed show the name of the profile selected by the NetworkManager Dispatcher hook script, but the associated Gufw profile’s rules had not been applied. They were only applied if I clicked on the ‘Profile’ pull-down menu in Gufw, selected a different Gufw profile, then re-selected the desired Gufw profile. Therefore driving Gufw from a NetworkManager Dispatcher hook script is not possible. This is a pity, as Gufw is an easy way to manage UFW from a GUI; it allows the user to create, delete and edit zones (Gufw profiles) and to select them manually. What Gufw doesn’t do is enable the user to associate those zones with connection names, nor trigger specific zone automatically based on the selected network connection. firewalld, on the other hand, does enable the user to do both those things.

As my attempt at automating the switching of zones in Gufw had failed, I decided to create a NetworkManager Dispatcher hook script to switch zones automatically by using UFW commands. Initially I though about creating a bespoke UFW application profile for each zone and allowing/denying those in the script, but it is actually easier to use the fundamental UFW commands in the script, especially as UFW commands are relatively easy to understand. Also, this approach means everything is in a single file, which facilitates configuration. I can simply edit the script in order to: a) add or delete a zone; b) change a zone’s name; c) change policies and rules for a zone; d) add or delete a connection; e) change the name of a connection; f) change the zone a connection uses. Granted, editing a script is not as user-friendly as using the firewalld GUI to configure a zone and then using KDE Plasma’s system settings module Connections to specify that zone for a specific connection, but my script is not particularly difficult to understand and edit. And by using such a script I can continue to use UFW rather than installing firewalld and having to learn how to use it.

My NetworkManager Dispatcher hook script /etc/NetworkManager/dispatcher.d/20_ufw-zones is listed below. In the main body of the script I define the zone I wish to use for each connection, and in the function select_zone I define the policies and rules I want each zone to use.

#!/bin/bash
INTERFACE=$1
STATUS=$2
WIRED=enp4s0f1
WIFI=wlp3s0

CT_helper_rule() {
    echo "# The following is needed to enable Samba commands to" >> /etc/ufw/before.rules
    echo "# work properly for broadcast NetBIOS name resolution" >> /etc/ufw/before.rules
    echo "#"  >> /etc/ufw/before.rules
    echo "# raw table rules" >> /etc/ufw/before.rules
    echo "*raw" >> /etc/ufw/before.rules
    echo ":OUTPUT ACCEPT [0:0]" >> /etc/ufw/before.rules
    echo "-F OUTPUT" >> /etc/ufw/before.rules
    echo "-A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns" >> /etc/ufw/before.rules
    echo "COMMIT" >> /etc/ufw/before.rules
}
 
select_zone() {
    ufw --force reset
    ufw --force enable
    ZONE=$1
    case "$ZONE" in
    'Home')
        ufw default deny incoming
        ufw default allow outgoing
        #
        # Rules for SMB
        ufw allow from 192.168.1.0/24 to any port 137,138 proto udp
        ufw allow from 192.168.1.0/24 to any port 139,445 proto tcp
        CT_helper_rule
        #
        # Rules for KDEConnect
        ufw allow from 192.168.1.0/24 to any port 1714:1764 proto udp
        ufw allow from 192.168.1.0/24 to any port 1714:1764 proto tcp
    ;;
    'Office')
        ufw default deny incoming
        ufw default allow outgoing
    ;;
    'Public')
        ufw default reject incoming
        ufw default allow outgoing
    ;;
    'JohnsHouse')
        ufw default deny incoming
        ufw default allow outgoing
        #
        # Rules for SMB
        ufw allow from 192.168.42.0/24 to any port 137,138 proto udp
        ufw allow from 192.168.42.0/24 to any port 139,445 proto tcp
        CT_helper_rule
        #
        # Rules for KDEConnect
        ufw allow from 192.168.42.0/24 to any port 1714:1764 proto udp
        ufw allow from 192.168.42.0/24 to any port 1714:1764 proto tcp
    ;;
    esac
    ufw --force reload
    rm /etc/ufw/*.rules.20* # Delete backups of *.rules files ufw makes every time it is reset
    echo -n `date +"[%F %T %Z]"` >> /var/log/ufw-zones.log
    echo " Zone $ZONE selected for connection $ACTIVE on interface $INTERFACE." >> /var/log/ufw-zones.log
}
 
# Check if either the wired or wireless interface is up
if [ "$INTERFACE" = "$WIRED" -o "$INTERFACE" = "$WIFI" ] && [ "$STATUS" = "up" ]; then
 
    # Check if a single connection is active
    if [ `nmcli c | grep -v "\-\-" | grep -v "NAME.*UUID.*TYPE.*DEVICE" | wc -l` -eq 1 ]; then
 
        # Ascertain the name of the active connection
        ACTIVE=`nmcli c | grep -v "\-\-" | grep -v "NAME.*UUID.*TYPE.*DEVICE" | awk -F' ' '{print $1}'`
 
        case "$ACTIVE" in
 
        'eth0')
            ZONE="Home"
        ;;
        'POR1-wired')
            ZONE="Office"
        ;;
        'BTHub5-8EUQ')
            ZONE="Home"
        ;;
        'BTHub5-8EUQ-5GHz')
            ZONE="Home"
        ;;
        'John1')
            ZONE="JohnsHouse"
        ;;
        'GRAND MERCURE')
            ZONE="Public"
        ;;
        *)
            # If connection name is not in above list
            ZONE="Public"
        ;;

        esac

        select_zone $ZONE
        exit $?

    fi
fi

The log file that the script uses contains a chronological record of the connections made and the zones selected:

$ cat /var/log/ufw-zones.log 
[2019-09-30 20:13:52 BST] Zone Home selected for connection eth0 on interface enp4s0f1.
[2019-10-01 22:59:18 BST] Zone Home selected for connection BTHub5-8EUQ-5GHz on interface wlp3s0.
[2019-10-02 17:59:23 EDT] Zone Public selected for connection loganwifi on interface wlp3s0.
[2019-10-03 10:12:46 EDT] Zone Office selected for connection POR1-wired on interface enp4s0f1.

Automatic backup of users’ files on a NAS device to an external USB HDD

One of my Linux machines is a 4-bay server that performs various roles, one of which is as NAS (network-attached storage) for family and visitors’ devices connected to my home network. I had configured each pair of HDDs in a RAID 1 array in order to provide some internal redundancy, but I was nervous about not having an external backup for users’ shares. Therefore I recently purchased a 6TB external USB 3.0 HDD (Western Digital Elements Desktop WDBWLG0060HBK-EESN) to connect permanently to one of the server’s USB 3.0 ports for backup purposes. I created a Bash script ~/backup_to_usbhdd.sh to perform the backup, plus a cron job to launch it automatically at 05:01 daily:

user $ sudo crontab -e
user $ sudo crontab -l | grep -v ^# | grep backup
01 05 * * * sudo /home/fitzcarraldo/backup_to_usbhdd.sh

The use of ‘sudo‘ in the crontab command may appear superfluous because the cron job was created for the root user (i.e. by using ‘sudo crontab -e‘ rather than ‘crontab -e‘). However, this is done to make cron use the root user’s environment rather than the minimal set of environment variables cron would otherwise use [1].

#!/bin/bash
#
# This script backs up to an external USB HDD (NTFS) labelled "Elements" the contents
# of the directories /nas/shares/ on my server.
# It can be launched from the server either manually using sudo or as a root-user cron
# job (Use 'sudo crontab -e' to configure the job).
#
# Clean up if the backup did not complete last time:
umount /media/usbhdd 2>/dev/null
rm -rf /media/usbhdd/*
# Unmount the external USB HDD if mounted by udisks2 with the logged-in username in the path:
umount /media/*/Elements 2>/dev/null
# Find out the USB HDD device:
DEVICE=$( blkid | grep "Elements" | cut -d ":" -f1 )
# Create a suitable mount point if it does not already exist, and mount the device on it:
mkdir /media/usbhdd 2>/dev/null
mount -t ntfs-3g $DEVICE /media/usbhdd 2>/dev/null
sleep 10s
# Create the backup directories on the USB HDD if they do not already exist:
mkdir -p /media/usbhdd/nas 2>/dev/null
# Backup recursively the directories and add a time-stamped summary to the log file:
echo "********** Backing up nas shares directory **********" >> /home/fitzcarraldo/backup_to_usbhdd.log
date >> /home/fitzcarraldo/backup_to_usbhdd.log
# Need to use rsync rather than cp, so that can rate-limit the copying to the USB HDD:
rsync --recursive --times --perms --links --protect-args --bwlimit=22500 /nas/shares /media/usbhdd/nas/ 2>> /home/fitzcarraldo/backup_to_usbhdd.log
# No --delete option is used, so that any backed-up files deleted on the server are not deleted from the USB HDD.
echo "Copying completed" >> /home/fitzcarraldo/backup_to_usbhdd.log
date >> /home/fitzcarraldo/backup_to_usbhdd.log
df -h | grep Filesystem >> /home/fitzcarraldo/backup_to_usbhdd.log
df -h | grep usbhdd >> /home/fitzcarraldo/backup_to_usbhdd.log
echo "********** Backup completed **********" >> /home/fitzcarraldo/backup_to_usbhdd.log
cp /home/fitzcarraldo/backup_to_usbhdd.log /media/usbhdd/
# Unmount the USB HDD:
umount /media/usbhdd
exit 0

The initial version of the above script used ‘cp‘ rather than ‘rsync‘, which worked fine when I launched the script manually:

user $ sudo ./backup_to_usbhdd.sh

However, the script always failed when launched as a cron job. In this case the command ‘df -h‘ showed the root directory on the server was ‘100% used’ (full). Also, the mount point directory /media/usbhdd/ had not been unmounted. The log file had twenty or so lines similar to the following, indicating the script had failed due to the root filesystem becoming full:

cp: failed to extend ‘/media/usbhdd/nas/user1/Videos/20130822_101433.mp4’: No space left on device

Apparently data was being read from the server’s HDD into the RAM buffer/cache faster than it could be written to the external HDD. The bottleneck in this case is not USB 3.0, but the USB HDD itself. The specifications for the USB HDD do not mention drive write speed, but a quick search of the Web indicated that an external USB HDD might have a write speed of around 25 to 30 MBps (Megabytes per second). I do not know why the problem happened only when the script was launched as a cron job, but I clearly needed to throttle the rate of writing to the external HDD. Unfortunately the ‘cp‘ command does not have such an option, but the ‘rsync‘ command does:

--bwlimit=RATE          limit socket I/O bandwidth

where RATE is in KiB if no units are specified. I opted to use a rate of 22500 KiB to be safe, and it is not too far below the aforementioned 25 MBps. Indeed, using this limit the script runs to completion successfully when launched by cron:

user $ cat backup_to_usbhdd.log
********** Backing up nas shares directory **********
Thu Sep 13 05:01:26 BST 2018
Copying completed
Thu Sep 13 11:41:31 BST 2018
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       5.5T  386G  5.1T   7% /media/usbhdd
********** Backup completed **********
********** Backing up nas shares directory **********
Fri Sep 14 05:01:26 BST 2018
Copying completed
Fri Sep 14 05:20:08 BST 2018
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       5.5T  403G  5.1T   8% /media/usbhdd
********** Backup completed **********
********** Backing up nas shares directory **********
Sat Sep 15 05:01:26 BST 2018
Copying completed
Sat Sep 15 05:04:58 BST 2018
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       5.5T  404G  5.1T   8% /media/usbhdd
********** Backup completed **********
********** Backing up nas shares directory **********
Sun Sep 16 05:01:26 BST 2018
Copying completed
Sun Sep 16 05:15:14 BST 2018
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       5.5T  416G  5.1T   8% /media/usbhdd
********** Backup completed **********
********** Backing up nas shares directory **********
Mon Sep 17 05:01:26 BST 2018
Copying completed
Mon Sep 17 05:04:15 BST 2018
Filesystem      Size  Used Avail Use% Mounted on
/dev/sdf1       5.5T  416G  5.1T   8% /media/usbhdd
********** Backup completed **********

Notice that the first job listed in the log file took much longer than subsequent jobs. This was because rsync had to copy every file to the external USB HDD. In subsequent runs it only had to copy new files and files that had changed since they were last copied.

The disk in the external USB HDD spins down after 10 minutes of inactivity and the drive goes into Power Saver Mode. Its LED blinks to indicate the drive is in this mode. Therefore the cron job only spins up and down the external HDD once per day.

Reference
1. Why does root cron job script need ‘sudo’ to run properly?

Installing Dropbox in Gentoo Linux following the recent restrictions introduced for Dropbox for Linux

In a 2013 post I explained how I installed Dropbox in Gentoo Linux running KDE 4. The Dropbox company has recently imposed some restrictions in the Linux client, so this is to explain what I did to get Dropbox working again in my two Gentoo Linux installations, both using the ext4 filesystem (unencrypted) and, these days, KDE Plasma 5.

Both my laptops running Gentoo Linux had a version of Dropbox installed via the Portage package manager: dropbox-45.3.88 in the case of the laptop running Gentoo amd64, and dropbox-48.3.56 in the case of the laptop running Gentoo ~amd64. Recently a Dropbox window popped up, warning me to upgrade Dropbox to the latest version within seven days otherwise the client would no longer be able to sync with the remote Dropbox server. I also received an e-mail from the Dropbox company titled ‘[Action required] We’re updating Linux system requirements‘ informing me that the only supported Linux distributions from now on would be Ubuntu 14.04 or higher and Fedora 21 or higher, and furthermore that the client will only work on an unencrypted ext4 filesystem. As both my Gentoo installations use unencrypted ext4, I was OK on that score, but I still had the problem that an up-to-date Dropbox ebuild is not available for Gentoo and the old Dropbox versions I was using no longer sync. However, I managed to install the latest version of Dropbox (currently 55.4.171) in Gentoo, and it works fine. The Dropbox client’s icon is on the KDE Plasma 5 Panel, and the local Dropbox directory is being sync’ed correctly. Below I explain what I did.

1. I selected ‘Quit Dropbox’ from the old Dropbox client’s menu, and the Dropbox icon disappeared from the Panel.

2. I removed the Dropbox daemon from the list of script files to be started at login (‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’).

3. I unmerged (uninstalled) the dropbox package:

clevow230ss /home/fitzcarraldo # emerge --ask --depclean dropbox

4. I deleted the directories ~/.dropbox and ~/.dropbox-dist but kept the directory ~/Dropbox and its contents.

fitzcarraldo@clevow230ss ~ $ rm -rf ~/.dropbox ~/.dropbox-dist

5. I followed the instructions under ‘Dropbox Headless install via command line‘ on the Dropbox Website to re-install the latest version of the daemon and client:

fitzcarraldo@clevow230ss ~ $ cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

6. I configured KDE Plasma 5 to start ~/.dropbox-dist/dropboxd at login (‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’ > ‘Add Script…’).

7. I launched ~/.dropbox-dist/dropboxd manually from a Konsole window. The Dropbox client icon appeared on the Panel and I was prompted to login to my Dropbox account via a Web browser, as per the instructions on the Dropbox Website (see link in in Step 5):

If you’re running Dropbox on your server for the first time, you’ll be asked to copy and paste a link in a working browser to create a new account or add your server to an existing account. Once you do, your Dropbox folder will be created in your home directory.

8. I logged in to my Dropbox account via the Firefox browser. As soon as I had logged in via the browser, a message appeared in the browser window informing me that “Your computer was successfully linked to your account”, and the Dropbox client icon appeared on the Panel and showed that the contents of ~/Dropbox were being synchronised.

Everything seems to be working as before. The Dropbox icon on the Panel has the same menu items it had previously. ‘Preferences…’ shows the Dropbox version as v55.4.171. I have not ticked ‘Start Dropbox on system startup’ under Dropbox Preferences because I configured automatic startup using KDE Plasma 5 ‘System Settings’ as described in Step 6 above, and the Dropbox daemon is indeed started automatically when I login.

The Dropbox Website’s instructions (see link in Step 5) also include the following:

Download this Python script to control Dropbox from the command line. For easy access, put a symlink to the script anywhere in your PATH.

I did download that Python script and made it executable:

fitzcarraldo@clevow230ss ~/Dropbox $ chmod +x dropbox.py

However the Python 3.6 interpreter in my Gentoo Linux installations report a syntax error in the script when I run it, I assume because it was written for a different version of Python:

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py 
  File "./dropbox.py", line 233
    except OSError, e:
                  ^
SyntaxError: invalid syntax

Anyway, as the Dropbox client icon is on the KDE Plasma 5 Panel and I can control Dropbox from there, I see no need for the Python script.

9. My Gentoo installations have a Bash script ~/dbox.sh that I had created to be launched by a Desktop Configuration file ~/Desktop/Dropbox.desktop with a nice icon which I double-click on if I want to relaunch the Dropbox daemon (if I previously quit Dropbox from the client’s menu, for example). I had to modify ~/dbox.sh by replacing the command ‘dbus-launch dropbox start > /dev/null‘ with the command ‘/home/fitzcarraldo/.dropbox-dist/dropboxd‘ as shown below.

dbox.sh

#!/bin/bash
notify-send 'Launching Dropbox' 'Daemon will be (re)started in 20 seconds' --icon=dialog-information
sleep 20s
ps auxww | awk '$0~/dropbox/&&$0!~/awk/{print $2}' | xargs kill
/home/fitzcarraldo/.dropbox-dist/dropboxd

Dropbox.desktop

[Desktop Entry]
Comment[en_GB]=(re)launch Dropbox daemon
Comment=(re)launch Dropbox daemon
Exec=/home/fitzcarraldo/dbox.sh
GenericName[en_GB]=Dropbox
GenericName=Dropbox
Icon=kipi-dropbox
MimeType=
Name[en_GB]=Dropbox
Name=Dropbox
Path=
StartupNotify=true
Terminal=false
TerminalOptions=
Type=Application
X-DBUS-ServiceName=
X-DBUS-StartupType=none
X-KDE-SubstituteUID=false
X-KDE-Username=fitzcarraldo

10. At the moment Dropbox is working fine again in my Gentoo installations. However, I noticed that Gentoo Linux user zsitvaij posted the following comment in a Gentoo Forums thread:

On every dropbox update, I have to remove ~/.dropbox-dist/dropbox-lnx./libdrm.so.2 to avoid having it crash on launch, works fine after until they update again.

I do not know if that will be necessary in my case, as I have not yet had to upgrade Dropbox from the Version 55.4.171 that I recently installed. When a new version of Dropbox becomes available I will update this post to confirm whether or not I had to do anything to keep Dropbox working.

Addendum (5 October 2019): With reference to my addendum of 31 August 2018, the Python script dropbox.py that can be downloaded from the Dropbox Web site has been updated and is now written in Python 3, so you can ignore my addendum of 31 August 2018.

Addendum (1 October 2018): With reference to my addendum of 2 September 2018, if you are using OpenRC it is possible to automate the deletion of the file ~/.dropbox-dist/dropbox-lnx.x86_64-/libdrm.so.2 by creating a Bash script /etc/local.d/40dropbox.start containing the following:

#!/bin/bash
if [ -e /home/fitzcarraldo/.dropbox-dist/dropbox-lnx.x86_64-*/libdrm.so.2 ]
then
    rm /home/fitzcarraldo/.dropbox-dist/dropbox-lnx.x86_64-*/libdrm.so.2
fi

Replace my username with your username, obviously. Of course the conditional test could be dispensed with and the script could just contain the shebang line and the rm line, which would still work even if the file does not exist, but it feels a bit tidier to only attempt to delete the file if it actually exists.

Addendum (2 September 2018): I have just installed Dropbox Version 56.4.94 in my Gentoo ~amd64 installation and I had to use the command shown below once in order to stop the daemon segfaulting when I entered the command ~/.dropbox-dist/dropboxd in a Konsole window:

fitzcarraldo@clevow230ss ~/Dropbox $ rm ~/.dropbox-dist/dropbox-lnx.x86_64-56.4.94/libdrm.so.2

Addendum (31 August 2018): The Python script dropbox.py that can be downloaded from the Dropbox Web site (see Step 8 above) is old, as can be seen in the comments in the header of the script:

# Dropbox frontend script
# This file is part of nautilus-dropbox 2015.10.28.

It is written in Python 2. Although I do not need to use it, I managed to get it to run in my Gentoo installations by replacing the shebang line ‘#!/usr/bin/python‘ with ‘#!/usr/bin/env python2‘. This works in my Gentoo installations because they have both Python 2.7 and Python 3.6 installed. When I now run dropbox.py I see the following:

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py 
Dropbox command-line interface

commands:

Note: use dropbox help  to view usage for a specific command.

 status       get current status of the dropboxd
 throttle     set bandwidth limits for Dropbox
 help         provide help
 stop         stop dropboxd
 running      return whether dropbox is running
 start        start dropboxd
 filestatus   get current sync status of one or more files
 ls           list directory contents with current sync status
 autostart    automatically start dropbox at login
 exclude      ignores/excludes a directory from syncing
 lansync      enables or disables LAN sync
 sharelink    get a shared link for a file in your dropbox
 proxy        set proxy settings for Dropbox

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py status
Up to date
fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py running
fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py filestatus ~/Dropbox/Getting\ Started.pdf 
/home/fitzcarraldo/Dropbox/Getting Started.pdf: up to date
fitzcarraldo@clevow230ss ~/Dropbox $

Notice that the command ./dropbox.py running does not return anything even though the daemon is definitely running, so I do not trust the script anyway.

Syncing browser bookmarks between browsers and machines in Linux

I use several computers and various browsers (predominantly Firefox, Chrome and Chromium) and was fed up with bookmarking a site on one machine and later not finding it on another machine. For quite some time I had therefore been looking for a simple way of synchronising browser bookmarks across all my machines and browsers, and I finally found one. Below I explain what I did.

I wanted to avoid storing my bookmarks on a third-party company’s server, so that ruled out tools such as Xmarks, EverSync, Google Bookmarks and the like. I wanted the bookmark database to reside on one of my own servers that is already accessible securely via the Internet. Apparently Xmarks optionally does enable you to use your own server providing you use only Firefox, but I use various browsers (Firefox is the default browser on my main laptop whereas Chrome is the default browser on my backup laptop, for example). Furthermore, I prefer to use open-source solutions whenever possible.

Although I was looking for a GUI solution, it turns out that the command-line bookmark manager Buku does a good job in a drop-down terminal such as Yakuake, Guake or Tilda. Buku is quite powerful, yet simple to use. It is certainly practical to use in a drop-down terminal (I’m currently using it with Yakuake in KDE, and with Tilda in LXDE). Not only can you click on links to open pages in the default browser, you can also easily configure your desktop environment to use a keyboard shortcut to bookmark directly from the browser window (see the instructions in the Buku Wiki for details).

Of course, if you only want to use Buku as a local bookmark manager on a machine, you can just install it and use it solely on that machine.

It is not difficult to set up a centralised Buku database that is then synchronised with any machine on which Buku is installed. If you do not have your own Cloud server (ownCloud or Nextcloud, for example), you could use Dropbox instead. The instructions are given in the Buku Wiki. Basically, I did the following to configure several machines to use Buku via the Cloud:

1. Use each browser’s bookmark manager to export the bookmarks to a file.

2. Install Buku on each machine (see ‘Installation‘ on the package’s GitHub repository page if your Linux distribution’s package manager does not offer Buku).

3. Launch Buku once on each machine to create the local database:

$ buku -p
DB file is being created at /home/fitzcarraldo/.local/share/buku/bookmarks.db.
You should encrypt it.
[ERROR] 0 records

4. On one machine, move the Buku database file (~/.local/share/buku/bookmarks.db) to a folder on the machine that is already being synced with the Cloud, then set up a symlink to it. For example:

fitzcarraldo@clevow230ss ~ $ ls -la ~/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo fitzcarraldo 51 Mar 21 13:17 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /media/NTFS/Windows/ownCloud/Bookmarks/bookmarks.db

5. Allow the Cloud client on the other machines to download the bookmarks.db file into their local Cloud sync folder, then delete the local Buku database on each machine (~/.local/share/buku/bookmarks.db) and create a symlink to the Cloud-synchronised database file. For example, in addition to the symlink shown above on the machine clevow230ss, I have the following symlinks on two other machines:

fitzcarraldo@aspirexc600:~$ ls -la ~/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo fitzcarraldo 42 Mar 21 16:05 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /home/fitzcarraldo/ownCloud/Bookmarks/bookmarks.db
fitzcarraldo@meshedgedx ~ $ ls -la /home/fitzcarraldo/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo users 42 Mar 26 19:15 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /home/fitzcarraldo/ownCloud/Bookmarks/bookmarks.db

6. Use Buku on each machine to import the browser bookmark files that you created in Step 1. See the Buku documentation for the command. You can find documentation and a demo video on the above-mentioned GitHub page. The commands ‘man buku‘ and ‘buku --help‘ also list the commands. The man(ual) page also contains several examples to help you.

7. Use Buku as normal on each machine. You will be able to search the synchronised database, add bookmarks and edit them (title, URL, comment and tags), delete bookmarks, print bookmarks, click on links to view the pages in the default browser, and so on.

Looking through a flat list of bookmarks in a terminal window to find something is not as fast as in a GUI but, overall, Buku is a decent bookmark manager and its options are easy to learn and use. Buku’s comprehensive search options of course help to find bookmarks, but it is still not quite as ergonomic as a GUI bookmark manager in my opinion. The ability to have multiple tags in Buku does help, as you can search for either any or all tags. In a browser’s bookmark manager I would copy the same bookmark into different folders if the Web page covers multiple topics.

In summary, Buku is a viable bookmark manager and I like it. It is extremely easy to configure for use with a Cloud server, and I have set it up to synchronise bookmarks on all my machines. I have already imported into Buku the 1,300+ bookmarks from the various browsers on my machines, and deleted the bookmarks in those browsers, so I am using Buku in earnest. I just kept a few of the most-used bookmarks on the browser’s Bookmarks Toolbar, but I’m using Buku on my machines for all the other bookmarks.

If I do have to use a third-party machine running Windows or Linux without Buku installed, I would not be able to access my bookmarks from my Cloud server. To partially get around that, I created a cron job for my user account on each of my machines to periodically run Buku and print the bookmarks to a text file synced on my Cloud server. That way I can at least search through the text file remotely via the Cloud’s Web browser interface (or via WebDAV or via OpenVPN) if I cannot find the Web page I want in a search engine on the third-party machine.

fitzcarraldo@clevow230ss ~ $ crontab -l | grep -v \#
6,26,46 * * * * rm /media/NTFS/Windows/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/bin/buku -p --nc > /media/NTFS/Windows/ownCloud/Bookmarks/Buku_bookmarks_backup.txt
fitzcarraldo@aspirexc600:~$ crontab -l | grep -v \#
1,21,41 * * * * rm /home/fitzcarraldo/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/local/bin/buku -p --nc > /home/fitzcarraldo/ownCloud/Bookmarks/Buku_bookmarks_backup.txt
fitzcarraldo@meshedgedx ~ $ crontab -l | grep -v \#
11,31,51 * * * * rm /home/fitzcarraldo/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/bin/buku -p --nc > /home/fitzcarraldo/ownCloud/Bookmarks/Buku_bookmarks_backup.txt

Below is a small taste of searching the bookmark database using Buku on any of my machines. Output is colour-coded (user-configurable), and links are clickable in a terminal window. You can search for any keyword(s), all keywords, sub-strings, just a tag or tags, regular expression matches, and so on. You can make titles immutable (read-only) if you want, or allow Buku to update them with the title from the Web site page. There is even a command that will check and list broken links. I will leave you to study the Buku documentation.

fitzcarraldo@aspirexc600:~$ buku -S Brazil samba
1. Kaká e Mário Monteiro são os novos carnavalescos da Imperatriz Leopoldinense [159]
   > http://www.sidneyrezende.com/editoria/carnaval
   +  Notícias sobre Carnaval 2016, escolas de samba, desfiles do Grupo Especial, Série A, ensaios técnicos, enredos, carnavalescos, bateria, mestre-sala, porta-bandeira, samba. Mangueira, Unidos da Tijuca, Vila Isabel, Beija-Flor, Grande Rio, Imperatriz, Mocidade, Portela, Salgueiro, União da Ilha, Viradouro, São Clemente, Porto da Pedra, Império da Tijuca, Império Serrano, Estácio de Sá, Caprichosos de Pilares, Tradição, Cubango, Em Cima da Hora, Inocentes de Belford Roxo, Alegria da Zona Sul, Unidos de Padre Miguel, Unidos de Bangu, Renascer de Jacarepaguá, Acadêmicos da Rocinha, Acadêmicos de Santa Cruz, Paraíso de Tuiuti, União de Jacarepaguá, União do Parque Curicica.
        
   # brazil,carnaval

2. Samba do Tuiuti 2018  Versão Acústica - YouTube [1270]
   > https://www.youtube.com/watch?v=yUxfwAzHOeY
   # brazil,carnaval,music,samba,video

buku (? for help) q

In this post I have only scratched the surface of what Buku can do. For example, a simple Buku command will encrypt (AES256) the bookmark database so you can prevent others viewing your bookmarks after you have finished searching the database, should you decide to store the database on a third-party Cloud server such as Dropbox. The search and editing tools are comprehensive yet straightforward, and you will quickly learn how to use them. I take my hat off to its developer, Arun Prakash Jana from Bangalore, India. He and the other contributors to Buku have done a great job, and I recommend you give Buku a try.

Prevent Linux firewalls interfering with Samba commands in a home network that uses broadcast NetBIOS name resolution

Or “How come devices in a home network can browse SMB shares but Linux Samba commands and Windows nbtstat commands do not work properly?”

Introduction

In a previous post I explained how it is possible to browse SMB shares when using broadcast NetBIOS name resolution in a home network consisting of machines running Linux, Windows and other operating systems. Browsing SMB/Samba shares will work as expected, but Samba commands such as ‘smbtree‘, ‘smbclient‘ and ‘nmblookup‘ will not work properly if the Linux machines use a firewall that has not been configured for broadcast NetBIOS name resolution. This post is to explain how to do that.

If broadcast NetBIOS name resolution is being used and none of the Linux machines has a firewall enabled, or if their firewalls have been correctly configured, the output of e.g. the ‘smbtree‘ command on one of those machines would look something like the example below.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
                \\TUTANKHAMUN\Samsung_Xpress_C460FW     Samsung Xpress C460FW
                \\TUTANKHAMUN\Canon_MP560_Printer       Canon PIXMA MP560
                \\TUTANKHAMUN\Canon_MP510_Printer       Canon PIXMA MP510
                \\TUTANKHAMUN\Virtual_PDF_Printer       Virtual PDF Printer
                \\TUTANKHAMUN\IPC$              IPC Service (Samba 4.2.11)
                \\TUTANKHAMUN\Public
                \\TUTANKHAMUN\anne-share
                \\TUTANKHAMUN\print$
                \\TUTANKHAMUN\netlogon          Network Logon Service
        \\BTHUB5                        BT Home Hub 5.0A File Server
                \\BTHUB5\IPC$                   IPC Service (BT Home Hub 5.0A File Server)
        \\THUTMOSEIII                   Windows 10 computer

If Linux firewalls have not been correctly configured, the output would be missing some information about other machines in the network. For example, compare the output above with the output below from the same network, this time with the Linux firewalls configured using typical rules for Samba specified in Web articles, blog posts and forums.

anne@akhanaten:~$ smbtree
Enter anne's password: 
HOME
        \\AKHANATEN                     Samba 4.3.11-Ubuntu
                \\AKHANATEN\IPC$                IPC Service (Samba 4.3.11-Ubuntu)
                \\AKHANATEN\guest               guest account
                \\AKHANATEN\matthew             matthew share
                \\AKHANATEN\marilla             marilla share
                \\AKHANATEN\anne                anne share
        \\TUTANKHAMUN                   Samba 4.5.10
        \\BTHUB5                        BT Home Hub 5.0A File Server
        \\THUTMOSEIII                   Windows 10 computer

To avoid this problem you need to add a further Linux firewall rule to the set of rules usually used for Samba. Below I first list the usual firewall rules for Samba, then I give the additional rule necessary if using broadcast NetBIOS name resolution. In each case I give the applicable rules for a pure IPTABLES firewall and for UFW (Uncomplicated Firewall). The rules listed here assume the IP address range of the home network is 192.168.1.0/24, so change the range to suit the specific network.

Firewall rules typically specified for machines using Samba

IPTABLES

The rules listed below assume the machine uses interface eth0, so change the interface to suit the specific machine.

# NetBIOS Name Service (name resolution)
iptables -A INPUT -i eth0 -p udp --dport 137 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Datagram Service (BROWSER service)
iptables -A INPUT -i eth0 -p udp --dport 138 -s 192.168.1.0/24 -j ACCEPT

# NetBIOS Session Service (data transfer legacy SMB/NetBIOS/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 139 -s 192.168.1.0/24 -j ACCEPT

# Microsoft Directory Service (data transfer SMB/TCP)
iptables -A INPUT -i eth0 -p tcp --dport 445 -s 192.168.1.0/24 -j ACCEPT

UFW

In some Linux distributions the ufw application allows a single command to add Samba support, such as:

user $ sudo ufw allow Samba

or

user $ sudo ufw allow CIFS

These ‘application profiles’ are specified in files in the directory /etc/ufw/applications.d/, so you could add application profiles or modify existing ones if you wish. In one of my installations the file /etc/ufw/applications.d/ufw-fileserver includes the following application profile for Samba, for example:

[CIFS]
title=SMB/CIFS server
description=SMB/CIFS server
ports=137,138/udp|139,445/tcp

If such an application profile does not exist in your installation, typical Samba rules can be added in UFW using the following two commands:

user $ sudo ufw allow from 192.168.1.0/24 to any port 137,138 proto udp
user $ sudo ufw allow from 192.168.1.0/24 to any port 139,445 proto tcp

The correct addition of the rules can be checked using the following command:

user $ sudo ufw status verbose
Password:
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
137,138/udp (CIFS)         ALLOW IN    192.168.1.0/24
139,445/tcp (CIFS)         ALLOW IN    192.168.1.0/24

The extra rule required when using broadcast NetBIOS name resolution

The reason why an extra rule is required when using broadcast NetBIOS name resolution is because UFW (which is based on IPTABLES) is ‘stateful’, as is a purely IPTABLES firewall (unless explicitly configured not to be stateful). The firewall does not consider packets it receives in response to its broadcast to be ESTABLISHED or RELATED, and therefore drops those packets. So, despite the IPTABLES and UFW rules listed above including a rule to accept incoming UDP packets on Port 137, any UDP packets received on Port 137 that do not constitute a one-to-one, two-way communication flow are dropped by the firewall. The extra rule below overrules this and makes the firewall accept packets coming from other devices’ Port 137 in response to broadcast NetBIOS Name Service packets. To do this, the extra rule uses a CT (Connection Tracking) helper named ‘netbios-ns‘ (obviously meaning ‘NetBIOS Name Service’). In order to use this rule the kernel must have been configured to use the IPTABLES ‘raw‘ table and to use CT (see the section ‘Kernel configuration’ further on).

IPTABLES

# All NetBIOS clients must have the netbios-ns helper enabled for broadcast name resolution to work
iptables -t raw -A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns

By the way, in addition to flushing the usual tables, flush the ‘raw‘ table too when you restart the firewall:

iptables -t raw -F OUTPUT

UFW

Add the following lines to the end of the file /etc/ufw/before.rules

# The following is needed to enable Samba commands to
# work properly for broadcast NetBIOS name resolution
#
# raw table rules
*raw
:OUTPUT ACCEPT [0:0]
-F OUTPUT
-A OUTPUT -p udp -m udp --dport 137 -j CT --helper netbios-ns
COMMIT

Note that the output of the command ‘ufw status verbose‘ will not include the above rule. This is not a bug.

Kernel configuration

If you are using a binary-based distribution such as Ubuntu Linux, the kernel will probably have been configured to include the needed modules (CONFIG_IP_NF_RAW=m, CONFIG_IP6_NF_RAW=m and CONFIG_NETFILTER_XT_TARGET_CT=m), and the installation configured to load the modules automatically. However, if you are using a source-based distribution such as Gentoo Linux make sure the kernel configuration includes these three options before you build the kernel, and also add the module names ‘iptable_raw‘ and ‘xt_CT‘ to the module list in the file /etc/conf.d/modules as shown in the example below, so that the modules are loaded at boot:

modules="r8169 nvidia agpgart fuse bnep rfcomm hidp uvcvideo cifs mmc_block rtsx_pci snd-seq-midi vboxdrv vboxnetadp vboxnetflt iptable_raw xt_CT"

You can use the following two commands to check if the two modules are loaded:

user $ sudo lsmod | grep iptable_raw
user $ sudo lsmod | grep xt_CT

How to check the additional rule is active

You can use the command below whether you are using pure IPTABLES or UFW.

user $ sudo iptables -nvL -t raw
Password: 
Chain PREROUTING (policy ACCEPT 2613 packets, 1115K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 2773 packets, 475K bytes)
 pkts bytes target     prot opt in     out     source               destination         
   16  1248 CT         udp  --  *      *       0.0.0.0/0            0.0.0.0/0            udp dpt:137 CT helper netbios-ns

The packet and byte counts will increase whenever you use a Samba command.

Bibliography

  1. The netfilter.org "iptables" project
  2. Iptables Tutorial
  3. Introduction to IPTables
  4. Gentoo Wiki : iptables
  5. Arch Linux Wiki : Samba : "Browsing" network fails with "Failed to retrieve share list from server"
  6. Ubuntu : Manpage : ufw-framework
  7. Gentoo Wiki : UFW

xdotool comes to the rescue

In a previous post I explained how I implemented a method for adding my current location and the local time to my e-mail signature wherever I happen to be in the World, irrespective of the time on the laptop’s hardware clock and system clock. In that post I described how I created a keyboard shortcut using the Linux application AutoKey. Unfortunately AutoKey has not been updated for several years and no longer works properly in KDE Plasma 5 on my laptops. Therefore I decided to replace it with a KDE keyboard shortcut, and this is to explain how I did it.

First create a custom shortcut in KDE:

  1. ‘System Settings’ > ‘Shortcuts’ > ‘Custom Shortcuts’
  2. ‘Edit’ > ‘New’ > ‘Global Shortcut’ > ‘Command/URL’, and name the New Action ‘Insert current time’
  3. On the Comment pane for ‘Insert current time’, add the comment ‘Insert current time at specified location’ (without the quotes)
  4. On the Trigger pane, configure the shortcut to be Ctrl+Alt+Space
  5. On the Action pane, enter the Command/URL as ‘/home/fitzcarraldo/timezone_signature_GeoNames.sh‘ (without the quotes)
  6. Click ‘Apply’

Next modify the Bash script timezone_signature_GeoNames.sh so that it contains the following (obviously change the username and path to suit):

#!/bin/bash

place=$(kdialog --title "Current Location" --inputbox "Enter your location:")

placetime=$(perl /home/fitzcarraldo/now1.pl $place)

# xdotool does not output a space in a string, so we have to extract each field from the string
# and print each field individually, separated by a space character.

city=$(echo $placetime | awk -F "|" '{print $1}')
country=$(echo $placetime | awk -F "|" '{print $2}' | sed 's/[)(]//g')
region=$(echo $placetime | awk -F "|" '{print $4}')

datetime=$(/usr/bin/zdump $region | awk -F " " '{print $2" "$3" "$4" "$5" "$6" "$7}')
dayofweek=$(echo $datetime | awk -F " " '{print $1}')
month=$(echo $datetime | awk -F " " '{print $2}')
day=$(echo $datetime | awk -F " " '{print $3}')
time=$(echo $datetime | awk -F " " '{print $4}')
year=$(echo $datetime | awk -F " " '{print $5}')
timezone=$(echo $datetime | awk -F " " '{print $6}')

activewindow=$(xdotool getactivewindow)

xdotool type --window $activewindow "Sent from:"
for oneword in $city; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow comma
for oneword in $country; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow Return
xdotool type --window $activewindow "Local time now: "
xdotool type --window $activewindow $dayofweek
xdotool type --window $activewindow " "
xdotool type --window $activewindow $month
xdotool type --window $activewindow " "
xdotool type --window $activewindow $day
xdotool type --window $activewindow " "
xdotool type --window $activewindow $time
xdotool type --window $activewindow " "
xdotool type --window $activewindow $year
xdotool type --window $activewindow " "
if [ ${timezone:0:1} = "-" ]; then
    timezone="UTC-"${timezone#*-}
elif [ ${timezone:0:1} = "+" ]; then
    timezone="UTC+"${timezone#*+}
fi
xdotool type --window $activewindow $timezone
xdotool type --window $activewindow " "
xdotool key --window $activewindow Return
xdotool key --window $activewindow Return
echo

The Perl script now1.pl is listed in my my earlier post. Notice that the script timezone_signature_GeoNames.sh in my earlier post was much simpler. This was because the AutoKey shortcut took care of sending the text to the currently active window. Without AutoKey, I now had to do this myself in the script timezone_signature_GeoNames.sh, and the command xdotool came to the rescue. The developer explains what xdotool does as follows:

This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11’s XTEST extension and other Xlib functions.

Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

So I installed xdotool via the Gentoo package manager:

# emerge xdotool
# eix xdotool
[I] x11-misc/xdotool
     Available versions:  3.20150503.1-r1^t ~3.20160805.1^t {examples}
     Installed versions:  3.20150503.1-r1^t(22:51:30 02/04/17)(-examples)
     Homepage:            http://www.semicomplete.com/projects/xdotool/
     Description:         Simulate keyboard input and mouse activity, move and resize windows

Anyway, my Bash script using xdotool works a treat with Thunderbird (and KWrite, LibreOffice Writer, etc.). I used to experience a problem with certain characters, for example a colon was printed as a semi-colon (see the xdotool bug report xdotool writes the wrong case #121), but that no longer happens in my current KDE Plasma 5 installation:

Sent from: Galeão International Airport, Brazil
Local time now: Thu Jul 6 15:11:40 2017 UTC-03

What a useful tool xdotool is!

Using the ClamAV daemon to scan files placed in my Downloads directory in Gentoo Linux

In a previous post I explained how to automatically detect files placed in my Downloads directory in Linux and scan them for viruses. The method I described in that post used clamscan, the command-line anti-virus scanner of ClamAV. Now, in addition ClamAV has a daemon (a program that runs continuously in the background), clamdscan, that you can enable. So I decided to switch to using clamdscan, as its response to downloaded files is much faster because the process waiting for new files to appear in ~/Downloads/ does not have to load clamscan from disk each time a new file arrives. Anyway, if you want to monitor a download directory in Gentoo Linux (running OpenRC) by using the ClamAV daemon — which will also download virus signature database updates automatically — then the procedure to set this up is given below.

1. Install clamav if it is not installed already:

root # emerge clamav

2. Add the service to the default runlevel:

root # rc-update add clamd default

The daemon will be launched automatically next time the computer boots.

3. The first download of the virus database has to be done manually:

root # freshclam

4. Start the daemon now:

root # rc-service clamd start

5. Create the Bash script ~/monitorDownloadsGUI with the following contents:

#!/bin/bash

DIR=$HOME/Downloads

# Get rid of old log file, if any
rm $HOME/virus-scan.log 2> /dev/null

IFS=$(echo -en "\n\b")

# Optionally, you can use shopt to avoid creating two processes due to the pipe
shopt -s lastpipe
inotifywait --quiet --monitor --event close_write,moved_to --recursive --format '%w%f' $DIR | while read FILE
# Added '--recursive' so that a directory copied into $DIR also triggers clamscan/clamdscan, although downloads
# from the Web would just be files, not directories.
do
     # Have to check file length is nonzero otherwise commands may be repeated
     if [ -s $FILE ]; then
          # Replace 'date >' with 'date >>' if you want to keep log file entries for previous scans.
          date > $HOME/virus-scan.log
          clamdscan --move=$HOME/virus-quarantine $FILE >> $HOME/virus-scan.log
          kdialog --title "Virus scan of $FILE" --msgbox "$(cat $HOME/virus-scan.log)"
     fi
done

Make it executable:

user $ chmod +x ~/monitorDownloadsGUI

6. Create the directory ~/virus-quarantine/ to store infected files pending investigation/deletion:

user $ mkdir ~/virus-quarantine

7. Install kdialog if it is not already installed:

root # emerge kdialog

8. Use ‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’ to add the script ~/monitorDownloadsGUI to the list of script files that are automatically started each time you log in to KDE.

9. Log out then back in again, and you should see that everything is running as expected:

user $ rc-status | grep clam
 clamd                                                             [  started  ]

user $ ps -ef | grep clam | grep -v grep
clamav    1920     1  0 01:48 ?        00:00:00 /usr/sbin/clamd
clamav    1929     1  0 01:48 ?        00:00:00 /usr/bin/freshclam -d

user $ ps -ef | grep GUI | grep -v grep
fitzcarraldo      9143  8971  0 13:56 ?        00:00:00 /bin/bash /home/fitzcarraldo/.config/autostart-scripts/monitorDownloadsGUI.sh

10. To test, surf to http://www.eicar.org/ and download one of the EICAR test files into your ~/Downloads/ directory. You should see a pop-up KDialog window with a message similar to the following:

Virus scan of /home/fitzcarraldo/Downloads/eicarcom2.zip — KDialog

Mon 27 Feb 14:05:26 GMT 2017
/home/fitzcarraldo/Downloads/eicarcom2.zip: Eicar-Test-Signature FOUND
/home/fitzcarraldo/Downloads/eicarcom2.zip: moved to ‘/home/fitzcarraldo/virus-quarantine/eicarcom2.zip’

———– SCAN SUMMARY ———–
Infected files: 1
Time: 0.001 sec (0 m 0 s)

Note that the above-mentioned pop-up window may be preceded by one or more pop-up windows with an error message. I’m using the Chrome browser at the moment, but you may get a similar message if you are using another browser. Here is an example:

Virus scan of /home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm — KDialog ?

Mon 27 Feb 14:16:30 GMT 2017
/home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm: Access denied. ERROR

———– SCAN SUMMARY ———–
Infected files: 0
Total errors: 1
Time: 0.000 sec (0 m 0 s)

Read the error message and click ‘OK’, as this is not an actual problem; it is inotifywait detecting temporary files in the ~/Downloads/ directory during the download process. With larger files sometimes several such messages are displayed, presumably because the file being downloaded is being opened and closed more than once during the downloading process. This issue does not occur if you copy or move a file into ~/Downloads/ from another directory in your installation; try it and see for yourself. Then you only get the one pop-up window with the scan result for the file you put in ~/Downloads/.

Also have a look in ~/virus-quarantine/ and you will see the EICAR test file in that directory. You can delete it if you want (it is not infected with a real virus, so does no harm).

In future be sure to read the messages in the pop-up windows before clicking ‘OK’, as they will inform you that an infected file has been moved to the quarantine directory.

That’s all there is to it. Very simple, and quite handy if you want to check quickly that files you download don’t have a malware payload. Just make sure you download all files into ~/Downloads/ or they will not be checked automatically. Also, if you are given e.g. a USB pen drive with a file on it, you can copy the file to ~/Downloads/ if you want it to be scanned for malware.

A method of ‘masking’ an OpenRC service (NetworkManager, in this case)

A Gentoo Linux user with an installation using OpenRC recently asked in the Gentoo Forums how to either a) disable NetworkManager so that it would not interfere with his netifrc configuration to give his installation a static IP address, or b) configure NetworkManager to use a static IP address (see the thread NetworkManager and static IP [SOLVED! THANKYOU]). In the end he solved the problem by uninstalling NetworkManager, the cleanest solution in his case given that his desktop machine is always in the same location and he does not need the features NetworkManager provides.

Now, although I use NetworkManager instead of netifrc, what intrigued me is that disabling the NetworkManager service using the standard command below does not stop the NetworkManager init script from running at boot-up:

root # rc-update delete NetworkManager default

Despite using the above command, following a reboot the NetworkManager service is still started and becomes active, and the NetworkManager daemon is running. Web browsers and other applications requiring network access still work. In order to stop the service running immediately so that his netifrc static IP address configuration could work, the aforementioned Gentoo user had to stop the NetworkManager service as follows:

root # /etc/init.d/NetworkManager stop

(The command ‘rc-service NetworkManager stop‘ does the same thing.)

The behaviour is the same on my laptop running Gentoo with OpenRC 0.22.4 and NetworkManager installed by networkmanager-1.4.0-r1 ebuild (with the upstream patch and necessary edit to the init script mentioned in Gentoo Bug Report No. 595806 – net-misc/networkmanager-1.4.0-r1[consolekit]: doesn’t automatically activate connections marked with "Automatically connect to this network when it’s available").

So two questions arose: What launches the NetworkManager init script when it has not been added to a runlevel? What needs to be done to stop this from happening? My curiosity was piqued.

As it happens, a somewhat similar situation exists when using systemd rather than OpenRC, as explained in Arch Linux Forums thread [SOLVED] NetworkManager auto restart even though I stop it. and Red Hat Bugzilla Report No. 815243 – Even though NetworkManager was manually stopped, it gets restarted automatically via D-Bus, although those were primarily concerned with how to prevent NetworkManager being restarted during the same session, i.e. without having rebooted.

The following systemd commands are needed to stop immediately the NetworkManager service and keep it from being restarted subsequently during the current session and after rebooting:

root # systemctl mask NetworkManager
root # systemctl stop NetworkManager
root # systemctl disable NetworkManager

Unfortunately there is no equivalent mask command for an OpenRC service. The equivalent OpenRC commands for the second and third commands above are:

root # rc-service NetworkManager stop
root # rc-update delete NetworkManager default

However, as I pointed out earlier, for some reason the latter command does not stop OpenRC running the NetworkManager init script at boot.

I wondered how I could ‘mask’ the NetworkManager service in OpenRC. I asked myself what the systemd mask command actually does. Well, it simply creates a symlink from /etc/systemd/system/NetworkManager.service to /dev/null so that there is no longer a real unit file for systemd to use, and therefore systemd can no longer launch the service. So why not do something similar in OpenRC. I hit upon the idea of telling the NetworkManager init script it needs a non-existent service in order to start, thus preventing OpenRC from starting the NetworkManager service:

root # echo 'rc_need="non-existent_service"' >> /etc/conf.d/NetworkManager # (Or just edit the file manually.)

That is all there is to it. When booting, OpenRC now displays the messages shown below:

* ERROR: NetworkManager needs service(s) non-existent_service
* ERROR: cannot start netmount as NetworkManager would not start
* ERROR: cannot start samba as NetworkManager would not start

As shown below, now the service is not started, so the NetworkManager daemon is never launched:

root # rc-status
Runlevel: default
 dbus                                                  [  started  ]
 syslog-ng                                             [  started  ]
 consolekit                                            [  started  ]
 netmount                                              [  stopped  ]
 cupsd                                                 [  started  ]
 samba                                                 [  stopped  ]
 cronie                                                [  started  ]
 clamd                                                 [  started  ]
 bluetooth                                             [  started  ]
 xdm                                                   [  started  ]
 cups-browsed                                          [  started  ]
 sshd                                                  [  started  ]
 local                                                 [  started  ]
Dynamic Runlevel: hotplugged
Dynamic Runlevel: needed/wanted
 modules-load                                          [  started  ]
 xdm-setup                                             [  started  ]
 avahi-daemon                                          [  started  ]
Dynamic Runlevel: manual
root # ps -ef | grep -v grep | grep -i network
root #

As expected, given that the netmount service and samba service depend on the NetworkManager service starting, neither of those services were able to start either.

Furthermore, because I masked the service, if I attempt to start it manually:

root # rc-service NetworkManager restart
 * ERROR: NetworkManager needs service(s) non-existent_service

To unmask the service in OpenRC, all that is needed is:

root # sed -i '/rc_need="non-existent_service"/d' /etc/conf.d/NetworkManager # (Or just edit the file manually.)

Note that, instead of “non-existent_service” I could have written “fubar”, “null” or any other string that is not the name of an actual service. But “non-existent_service” is more meaningful and less likely to confuse me when viewing system messages and contents of the service configuration file.

In summary…

Why does OpenRC run the NetworkManager service init script when it is not in any runlevel?

I have no idea!

I wondered if the D-Bus service does it. The Arch Wiki article on NetworkManager claims this is the case (see the section titled Disable NetworkManager). However, my attempts at preventing D-Bus doing anything to NetworkManager did not stop the NetworkManager init script from being run at boot. I deleted /etc/dbus-1/system.d/org.freedesktop.NetworkManager.conf and /etc/dbus-1/system.d/nm-dispatcher.conf but that did not help. Neither did creating an appropriate /etc/dbus-1/system.d/org.freedesktop.NetworkManager.conf or /etc/dbus-1/system-local.conf. There is no /usr/share/dbus-1/system-services/org.freedesktop.NetworkManager.service file in my Gentoo installation using OpenRC, but creating one did not help either. So, if you know what runs the OpenRC NetworkManager init script when it is not in any runlevel, please post a comment.

Anyway, I now know how to prevent it happening, so I have satisfied my curiosity. Below I list the commands I actually used in a Gentoo Linux installation (amd64, OpenRC) and a Sabayon Linux installation (~amd64, systemd) to check the functionality.

OpenRC

The following two (optionally three) commands are needed to stop immediately the NetworkManager service and prevent it being restarted subsequently during this session and after rebooting:

root # rc-service NetworkManager stop
root # echo 'rc_need="non-existent_service"' >> /etc/conf.d/NetworkManager
root # rc-update del NetworkManager default # (Optional.)

The following two (optionally three) commands are needed to unmask the NetworkManager service and start it immediately, and make it start automatically after rebooting:

root # sed -i '/rc_need="non-existent_service"/d' /etc/conf.d/NetworkManager
root # rc-service NetworkManager restart
root # rc-update add NetworkManager default # Only needed if I earlier deleted the service from the default runlevel.

systemd

The following three commands are needed to stop immediately the NetworkManager service and prevent it being restarted subsequently during this session and after rebooting:

root # systemctl mask NetworkManager
root # systemctl stop NetworkManager
root # systemctl disable NetworkManager

The following three systemd commands are needed to unmask the NetworkManager service and start it immediately, and also make it start automatically after rebooting:

root # systemctl unmask NetworkManager
root # systemctl enable NetworkManager
root # systemctl start NetworkManager