Installing Dropbox in Gentoo Linux following the recent restrictions introduced for Dropbox for Linux

In a 2013 post I explained how I installed Dropbox in Gentoo Linux running KDE 4. The Dropbox company has recently imposed some restrictions in the Linux client, so this is to explain what I did to get Dropbox working again in my two Gentoo Linux installations, both using the ext4 filesystem (unencrypted) and, these days, KDE Plasma 5.

Both my laptops running Gentoo Linux had a version of Dropbox installed via the Portage package manager: dropbox-45.3.88 in the case of the laptop running Gentoo amd64, and dropbox-48.3.56 in the case of the laptop running Gentoo ~amd64. Recently a Dropbox window popped up, warning me to upgrade Dropbox to the latest version within seven days otherwise the client would no longer be able to sync with the remote Dropbox server. I also received an e-mail from the Dropbox company titled ‘[Action required] We’re updating Linux system requirements‘ informing me that the only supported Linux distributions from now on would be Ubuntu 14.04 or higher and Fedora 21 or higher, and furthermore that the client will only work on an unencrypted ext4 filesystem. As both my Gentoo installations use unencrypted ext4, I was OK on that score, but I still had the problem that an up-to-date Dropbox ebuild is not available for Gentoo and the old Dropbox versions I was using no longer sync. However, I managed to install the latest version of Dropbox (currently 55.4.171) in Gentoo, and it works fine. The Dropbox client’s icon is on the KDE Plasma 5 Panel, and the local Dropbox directory is being sync’ed correctly. Below I explain what I did.

1. I selected ‘Quit Dropbox’ from the old Dropbox client’s menu, and the Dropbox icon disappeared from the Panel.

2. I removed the Dropbox daemon from the list of script files to be started at login (‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’).

3. I unmerged (uninstalled) the dropbox package:

clevow230ss /home/fitzcarraldo # emerge --ask --depclean dropbox

4. I deleted the directories ~/.dropbox and ~/.dropbox-dist but kept the directory ~/Dropbox and its contents.

fitzcarraldo@clevow230ss ~ $ rm -rf ~/.dropbox ~/.dropbox-dist

5. I followed the instructions under ‘Dropbox Headless install via command line‘ on the Dropbox Website to re-install the latest version of the daemon and client:

fitzcarraldo@clevow230ss ~ $ cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

6. I configured KDE Plasma 5 to start ~/.dropbox-dist/dropboxd at login (‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’ > ‘Add Script…’).

7. I launched ~/.dropbox-dist/dropboxd manually from a Konsole window. The Dropbox client icon appeared on the Panel and I was prompted to login to my Dropbox account via a Web browser, as per the instructions on the Dropbox Website (see link in in Step 5):

If you’re running Dropbox on your server for the first time, you’ll be asked to copy and paste a link in a working browser to create a new account or add your server to an existing account. Once you do, your Dropbox folder will be created in your home directory.

8. I logged in to my Dropbox account via the Firefox browser. As soon as I had logged in via the browser, a message appeared in the browser window informing me that “Your computer was successfully linked to your account”, and the Dropbox client icon appeared on the Panel and showed that the contents of ~/Dropbox were being synchronised.

Everything seems to be working as before. The Dropbox icon on the Panel has the same menu items it had previously. ‘Preferences…’ shows the Dropbox version as v55.4.171. I have not ticked ‘Start Dropbox on system startup’ under Dropbox Preferences because I configured automatic startup using KDE Plasma 5 ‘System Settings’ as described in Step 6 above, and the Dropbox daemon is indeed started automatically when I login.

The Dropbox Website’s instructions (see link in Step 5) also include the following:

Download this Python script to control Dropbox from the command line. For easy access, put a symlink to the script anywhere in your PATH.

I did download that Python script and made it executable:

fitzcarraldo@clevow230ss ~/Dropbox $ chmod +x dropbox.py

However the Python 3.6 interpreter in my Gentoo Linux installations report a syntax error in the script when I run it, I assume because it was written for a different version of Python:

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py 
  File "./dropbox.py", line 233
    except OSError, e:
                  ^
SyntaxError: invalid syntax

Anyway, as the Dropbox client icon is on the KDE Plasma 5 Panel and I can control Dropbox from there, I see no need for the Python script.

9. My Gentoo installations have a Bash script ~/dbox.sh that I had created to be launched by a Desktop Configuration file ~/Desktop/Dropbox.desktop with a nice icon which I double-click on if I want to relaunch the Dropbox daemon (if I previously quit Dropbox from the client’s menu, for example). I had to modify ~/dbox.sh by replacing the command ‘dbus-launch dropbox start > /dev/null‘ with the command ‘/home/fitzcarraldo/.dropbox-dist/dropboxd‘ as shown below.

dbox.sh

#!/bin/bash
notify-send 'Launching Dropbox' 'Daemon will be (re)started in 20 seconds' --icon=dialog-information
sleep 20s
ps auxww | awk '$0~/dropbox/&&$0!~/awk/{print $2}' | xargs kill
/home/fitzcarraldo/.dropbox-dist/dropboxd

Dropbox.desktop

[Desktop Entry]
Comment[en_GB]=(re)launch Dropbox daemon
Comment=(re)launch Dropbox daemon
Exec=/home/fitzcarraldo/dbox.sh
GenericName[en_GB]=Dropbox
GenericName=Dropbox
Icon=kipi-dropbox
MimeType=
Name[en_GB]=Dropbox
Name=Dropbox
Path=
StartupNotify=true
Terminal=false
TerminalOptions=
Type=Application
X-DBUS-ServiceName=
X-DBUS-StartupType=none
X-KDE-SubstituteUID=false
X-KDE-Username=fitzcarraldo

10. At the moment Dropbox is working fine again in my Gentoo installations. However, I noticed that Gentoo Linux user zsitvaij posted the following comment in a Gentoo Forums thread:

On every dropbox update, I have to remove ~/.dropbox-dist/dropbox-lnx./libdrm.so.2 to avoid having it crash on launch, works fine after until they update again.

I do not know if that will be necessary in my case, as I have not yet had to upgrade Dropbox from the Version 55.4.171 that I recently installed. When a new version of Dropbox becomes available I will update this post to confirm whether or not I had to do anything to keep Dropbox working.

Addendum (1 October 2018): With reference to my addendum of 2 September 2018, if you are using OpenRC it is possible to automate the deletion of the file ~/.dropbox-dist/dropbox-lnx.x86_64-/libdrm.so.2 by creating a Bash script /etc/local.d/40dropbox.start containing the following:

#!/bin/bash
if [ -e /home/fitzcarraldo/.dropbox-dist/dropbox-lnx.x86_64-*/libdrm.so.2 ]
then
    rm /home/fitzcarraldo/.dropbox-dist/dropbox-lnx.x86_64-*/libdrm.so.2
fi

Replace my username with your username, obviously. Of course the conditional test could be dispensed with and the script could just contain the shebang line and the rm line, which would still work even if the file does not exist, but it feels a bit tidier to only attempt to delete the file if it actually exists.

Addendum (2 September 2018):I have just installed Dropbox Version 56.4.94 in my Gentoo ~amd64 installation and I had to use the command shown below once in order to stop the daemon segfaulting when I entered the command ~/.dropbox-dist/dropboxd in a Konsole window:

fitzcarraldo@clevow230ss ~/Dropbox $ rm ~/.dropbox-dist/dropbox-lnx.x86_64-56.4.94/libdrm.so.2

Addendum (31 August 2018): The Python script dropbox.py that can be downloaded from the Dropbox Web site (see Step 8 above) is old, as can be seen in the comments in the header of the script:

# Dropbox frontend script
# This file is part of nautilus-dropbox 2015.10.28.

It is written in Python 2. Although I do not need to use it, I managed to get it to run in my Gentoo installations by replacing the shebang line ‘#!/usr/bin/python‘ with ‘#!/usr/bin/env python2‘. This works in my Gentoo installations because they have both Python 2.7 and Python 3.6 installed. When I now run dropbox.py I see the following:

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py 
Dropbox command-line interface

commands:

Note: use dropbox help  to view usage for a specific command.

 status       get current status of the dropboxd
 throttle     set bandwidth limits for Dropbox
 help         provide help
 stop         stop dropboxd
 running      return whether dropbox is running
 start        start dropboxd
 filestatus   get current sync status of one or more files
 ls           list directory contents with current sync status
 autostart    automatically start dropbox at login
 exclude      ignores/excludes a directory from syncing
 lansync      enables or disables LAN sync
 sharelink    get a shared link for a file in your dropbox
 proxy        set proxy settings for Dropbox

fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py status
Up to date
fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py running
fitzcarraldo@clevow230ss ~/Dropbox $ ./dropbox.py filestatus ~/Dropbox/Getting\ Started.pdf 
/home/fitzcarraldo/Dropbox/Getting Started.pdf: up to date
fitzcarraldo@clevow230ss ~/Dropbox $

Notice that the command ./dropbox.py running does not return anything even though the daemon is definitely running, so I do not trust the script anyway.

Advertisements

Syncing browser bookmarks between browsers and machines in Linux

I use several computers and various browsers (predominantly Firefox, Chrome and Chromium) and was fed up with bookmarking a site on one machine and later not finding it on another machine. For quite some time I had therefore been looking for a simple way of synchronising browser bookmarks across all my machines and browsers, and I finally found one. Below I explain what I did.

I wanted to avoid storing my bookmarks on a third-party company’s server, so that ruled out tools such as Xmarks, EverSync, Google Bookmarks and the like. I wanted the bookmark database to reside on one of my own servers that is already accessible securely via the Internet. Apparently Xmarks optionally does enable you to use your own server providing you use only Firefox, but I use various browsers (Firefox is the default browser on my main laptop whereas Chrome is the default browser on my backup laptop, for example). Furthermore, I prefer to use open-source solutions whenever possible.

Although I was looking for a GUI solution, it turns out that the command-line bookmark manager Buku does a good job in a drop-down terminal such as Yakuake, Guake or Tilda. Buku is quite powerful, yet simple to use. It is certainly practical to use in a drop-down terminal (I’m currently using it with Yakuake in KDE, and with Tilda in LXDE). Not only can you click on links to open pages in the default browser, you can also easily configure your desktop environment to use a keyboard shortcut to bookmark directly from the browser window (see the instructions in the Buku Wiki for details).

Of course, if you only want to use Buku as a local bookmark manager on a machine, you can just install it and use it solely on that machine.

It is not difficult to set up a centralised Buku database that is then synchronised with any machine on which Buku is installed. If you do not have your own Cloud server (ownCloud or Nextcloud, for example), you could use Dropbox instead. The instructions are given in the Buku Wiki. Basically, I did the following to configure several machines to use Buku via the Cloud:

1. Use each browser’s bookmark manager to export the bookmarks to a file.

2. Install Buku on each machine (see ‘Installation‘ on the package’s GitHub repository page if your Linux distribution’s package manager does not offer Buku).

3. Launch Buku once on each machine to create the local database:

$ buku -p
DB file is being created at /home/fitzcarraldo/.local/share/buku/bookmarks.db.
You should encrypt it.
[ERROR] 0 records

4. On one machine, move the Buku database file (~/.local/share/buku/bookmarks.db) to a folder on the machine that is already being synced with the Cloud, then set up a symlink to it. For example:

fitzcarraldo@clevow230ss ~ $ ls -la ~/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo fitzcarraldo 51 Mar 21 13:17 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /media/NTFS/Windows/ownCloud/Bookmarks/bookmarks.db

5. Allow the Cloud client on the other machines to download the bookmarks.db file into their local Cloud sync folder, then delete the local Buku database on each machine (~/.local/share/buku/bookmarks.db) and create a symlink to the Cloud-synchronised database file. For example, in addition to the symlink shown above on the machine clevow230ss, I have the following symlinks on two other machines:

fitzcarraldo@aspirexc600:~$ ls -la ~/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo fitzcarraldo 42 Mar 21 16:05 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /home/fitzcarraldo/ownCloud/Bookmarks/bookmarks.db
fitzcarraldo@meshedgedx ~ $ ls -la /home/fitzcarraldo/.local/share/buku/bookmarks.db
lrwxrwxrwx 1 fitzcarraldo users 42 Mar 26 19:15 /home/fitzcarraldo/.local/share/buku/bookmarks.db -> /home/fitzcarraldo/ownCloud/Bookmarks/bookmarks.db

6. Use Buku on each machine to import the browser bookmark files that you created in Step 1. See the Buku documentation for the command. You can find documentation and a demo video on the above-mentioned GitHub page. The commands ‘man buku‘ and ‘buku --help‘ also list the commands. The man(ual) page also contains several examples to help you.

7. Use Buku as normal on each machine. You will be able to search the synchronised database, add bookmarks and edit them (title, URL, comment and tags), delete bookmarks, print bookmarks, click on links to view the pages in the default browser, and so on.

Looking through a flat list of bookmarks in a terminal window to find something is not as fast as in a GUI but, overall, Buku is a decent bookmark manager and its options are easy to learn and use. Buku’s comprehensive search options of course help to find bookmarks, but it is still not quite as ergonomic as a GUI bookmark manager in my opinion. The ability to have multiple tags in Buku does help, as you can search for either any or all tags. In a browser’s bookmark manager I would copy the same bookmark into different folders if the Web page covers multiple topics.

In summary, Buku is a viable bookmark manager and I like it. It is extremely easy to configure for use with a Cloud server, and I have set it up to synchronise bookmarks on all my machines. I have already imported into Buku the 1,300+ bookmarks from the various browsers on my machines, and deleted the bookmarks in those browsers, so I am using Buku in earnest. I just kept a few of the most-used bookmarks on the browser’s Bookmarks Toolbar, but I’m using Buku on my machines for all the other bookmarks.

If I do have to use a third-party machine running Windows or Linux without Buku installed, I would not be able to access my bookmarks from my Cloud server. To partially get around that, I created a cron job for my user account on each of my machines to periodically run Buku and print the bookmarks to a text file synced on my Cloud server. That way I can at least search through the text file remotely via the Cloud’s Web browser interface (or via WebDAV or via OpenVPN) if I cannot find the Web page I want in a search engine on the third-party machine.

fitzcarraldo@clevow230ss ~ $ crontab -l | grep -v \#
6,26,46 * * * * rm /media/NTFS/Windows/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/bin/buku -p --nc > /media/NTFS/Windows/ownCloud/Bookmarks/Buku_bookmarks_backup.txt
fitzcarraldo@aspirexc600:~$ crontab -l | grep -v \#
1,21,41 * * * * rm /home/fitzcarraldo/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/local/bin/buku -p --nc > /home/fitzcarraldo/ownCloud/Bookmarks/Buku_bookmarks_backup.txt
fitzcarraldo@meshedgedx ~ $ crontab -l | grep -v \#
11,31,51 * * * * rm /home/fitzcarraldo/ownCloud/Bookmarks/*.txt; sleep 30s && /usr/bin/buku -p --nc > /home/fitzcarraldo/ownCloud/Bookmarks/Buku_bookmarks_backup.txt

Below is a small taste of searching the bookmark database using Buku on any of my machines. Output is colour-coded (user-configurable), and links are clickable in a terminal window. You can search for any keyword(s), all keywords, sub-strings, just a tag or tags, regular expression matches, and so on. You can make titles immutable (read-only) if you want, or allow Buku to update them with the title from the Web site page. There is even a command that will check and list broken links. I will leave you to study the Buku documentation.

fitzcarraldo@aspirexc600:~$ buku -S Brazil samba
1. Kaká e Mário Monteiro são os novos carnavalescos da Imperatriz Leopoldinense [159]
   > http://www.sidneyrezende.com/editoria/carnaval
   +  Notícias sobre Carnaval 2016, escolas de samba, desfiles do Grupo Especial, Série A, ensaios técnicos, enredos, carnavalescos, bateria, mestre-sala, porta-bandeira, samba. Mangueira, Unidos da Tijuca, Vila Isabel, Beija-Flor, Grande Rio, Imperatriz, Mocidade, Portela, Salgueiro, União da Ilha, Viradouro, São Clemente, Porto da Pedra, Império da Tijuca, Império Serrano, Estácio de Sá, Caprichosos de Pilares, Tradição, Cubango, Em Cima da Hora, Inocentes de Belford Roxo, Alegria da Zona Sul, Unidos de Padre Miguel, Unidos de Bangu, Renascer de Jacarepaguá, Acadêmicos da Rocinha, Acadêmicos de Santa Cruz, Paraíso de Tuiuti, União de Jacarepaguá, União do Parque Curicica.
        
   # brazil,carnaval

2. Samba do Tuiuti 2018  Versão Acústica - YouTube [1270]
   > https://www.youtube.com/watch?v=yUxfwAzHOeY
   # brazil,carnaval,music,samba,video

buku (? for help) q

In this post I have only scratched the surface of what Buku can do. For example, a simple Buku command will encrypt (AES256) the bookmark database so you can prevent others viewing your bookmarks after you have finished searching the database, should you decide to store the database on a third-party Cloud server such as Dropbox. The search and editing tools are comprehensive yet straightforward, and you will quickly learn how to use them. I take my hat off to its developer, Arun Prakash Jana from Bangalore, India. He and the other contributors to Buku have done a great job, and I recommend you give Buku a try.

xdotool comes to the rescue

In a previous post I explained how I implemented a method for adding my current location and the local time to my e-mail signature wherever I happen to be in the World, irrespective of the time on the laptop’s hardware clock and system clock. In that post I described how I created a keyboard shortcut using the Linux application AutoKey. Unfortunately AutoKey has not been updated for several years and no longer works properly in KDE Plasma 5 on my laptops. Therefore I decided to replace it with a KDE keyboard shortcut, and this is to explain how I did it.

First create a custom shortcut in KDE:

  1. ‘System Settings’ > ‘Shortcuts’ > ‘Custom Shortcuts’
  2. ‘Edit’ > ‘New’ > ‘Global Shortcut’ > ‘Command/URL’, and name the New Action ‘Insert current time’
  3. On the Comment pane for ‘Insert current time’, add the comment ‘Insert current time at specified location’ (without the quotes)
  4. On the Trigger pane, configure the shortcut to be Ctrl+Alt+Space
  5. On the Action pane, enter the Command/URL as ‘/home/fitzcarraldo/timezone_signature_GeoNames.sh‘ (without the quotes)
  6. Click ‘Apply’

Next modify the Bash script timezone_signature_GeoNames.sh so that it contains the following (obviously change the username and path to suit):

#!/bin/bash

place=$(kdialog --title "Current Location" --inputbox "Enter your location:")

placetime=$(perl /home/fitzcarraldo/now1.pl $place)

# xdotool does not output a space in a string, so we have to extract each field from the string
# and print each field individually, separated by a space character.

city=$(echo $placetime | awk -F "|" '{print $1}')
country=$(echo $placetime | awk -F "|" '{print $2}' | sed 's/[)(]//g')
region=$(echo $placetime | awk -F "|" '{print $4}')

datetime=$(/usr/bin/zdump $region | awk -F " " '{print $2" "$3" "$4" "$5" "$6" "$7}')
dayofweek=$(echo $datetime | awk -F " " '{print $1}')
month=$(echo $datetime | awk -F " " '{print $2}')
day=$(echo $datetime | awk -F " " '{print $3}')
time=$(echo $datetime | awk -F " " '{print $4}')
year=$(echo $datetime | awk -F " " '{print $5}')
timezone=$(echo $datetime | awk -F " " '{print $6}')

activewindow=$(xdotool getactivewindow)

xdotool type --window $activewindow "Sent from:"
for oneword in $city; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow comma
for oneword in $country; do
    xdotool key --window $activewindow space
    sleep 0.1s
    xdotool type --window $activewindow --delay 100 $oneword
done
xdotool key --window $activewindow Return
xdotool type --window $activewindow "Local time now: "
xdotool type --window $activewindow $dayofweek
xdotool type --window $activewindow " "
xdotool type --window $activewindow $month
xdotool type --window $activewindow " "
xdotool type --window $activewindow $day
xdotool type --window $activewindow " "
xdotool type --window $activewindow $time
xdotool type --window $activewindow " "
xdotool type --window $activewindow $year
xdotool type --window $activewindow " "
if [ ${timezone:0:1} = "-" ]; then
    timezone="UTC-"${timezone#*-}
elif [ ${timezone:0:1} = "+" ]; then
    timezone="UTC+"${timezone#*+}
fi
xdotool type --window $activewindow $timezone
xdotool type --window $activewindow " "
xdotool key --window $activewindow Return
xdotool key --window $activewindow Return
echo

The Perl script now1.pl is listed in my my earlier post. Notice that the script timezone_signature_GeoNames.sh in my earlier post was much simpler. This was because the AutoKey shortcut took care of sending the text to the currently active window. Without AutoKey, I now had to do this myself in the script timezone_signature_GeoNames.sh, and the command xdotool came to the rescue. The developer explains what xdotool does as follows:

This tool lets you simulate keyboard input and mouse activity, move and resize windows, etc. It does this using X11’s XTEST extension and other Xlib functions.

Additionally, you can search for windows and move, resize, hide, and modify window properties like the title. If your window manager supports it, you can use xdotool to switch desktops, move windows between desktops, and change the number of desktops.

So I installed xdotool via the Gentoo package manager:

# emerge xdotool
# eix xdotool
[I] x11-misc/xdotool
     Available versions:  3.20150503.1-r1^t ~3.20160805.1^t {examples}
     Installed versions:  3.20150503.1-r1^t(22:51:30 02/04/17)(-examples)
     Homepage:            http://www.semicomplete.com/projects/xdotool/
     Description:         Simulate keyboard input and mouse activity, move and resize windows

Anyway, my Bash script using xdotool works a treat with Thunderbird (and KWrite, LibreOffice Writer, etc.). I used to experience a problem with certain characters, for example a colon was printed as a semi-colon (see the xdotool bug report xdotool writes the wrong case #121), but that no longer happens in my current KDE Plasma 5 installation:

Sent from: Galeão International Airport, Brazil
Local time now: Thu Jul 6 15:11:40 2017 UTC-03

What a useful tool xdotool is!

Using the ClamAV daemon to scan files placed in my Downloads directory in Gentoo Linux

In a previous post I explained how to automatically detect files placed in my Downloads directory in Linux and scan them for viruses. The method I described in that post used clamscan, the command-line anti-virus scanner of ClamAV. Now, in addition ClamAV has a daemon (a program that runs continuously in the background), clamdscan, that you can enable. So I decided to switch to using clamdscan, as its response to downloaded files is much faster because the process waiting for new files to appear in ~/Downloads/ does not have to load clamscan from disk each time a new file arrives. Anyway, if you want to monitor a download directory in Gentoo Linux (running OpenRC) by using the ClamAV daemon — which will also download virus signature database updates automatically — then the procedure to set this up is given below.

1. Install clamav if it is not installed already:

root # emerge clamav

2. Add the service to the default runlevel:

root # rc-update add clamd default

The daemon will be launched automatically next time the computer boots.

3. The first download of the virus database has to be done manually:

root # freshclam

4. Start the daemon now:

root # rc-service clamd start

5. Create the Bash script ~/monitorDownloadsGUI with the following contents:

#!/bin/bash

DIR=$HOME/Downloads

# Get rid of old log file, if any
rm $HOME/virus-scan.log 2> /dev/null

IFS=$(echo -en "\n\b")

# Optionally, you can use shopt to avoid creating two processes due to the pipe
shopt -s lastpipe
inotifywait --quiet --monitor --event close_write,moved_to --recursive --format '%w%f' $DIR | while read FILE
# Added '--recursive' so that a directory copied into $DIR also triggers clamscan/clamdscan, although downloads
# from the Web would just be files, not directories.
do
     # Have to check file length is nonzero otherwise commands may be repeated
     if [ -s $FILE ]; then
          # Replace 'date >' with 'date >>' if you want to keep log file entries for previous scans.
          date > $HOME/virus-scan.log
          clamdscan --move=$HOME/virus-quarantine $FILE >> $HOME/virus-scan.log
          kdialog --title "Virus scan of $FILE" --msgbox "$(cat $HOME/virus-scan.log)"
     fi
done

Make it executable:

user $ chmod +x ~/monitorDownloadsGUI

6. Create the directory ~/virus-quarantine/ to store infected files pending investigation/deletion:

user $ mkdir ~/virus-quarantine

7. Install kdialog if it is not already installed:

root # emerge kdialog

8. Use ‘System Settings’ > ‘Startup and Shutdown’ > ‘Autostart’ to add the script ~/monitorDownloadsGUI to the list of script files that are automatically started each time you log in to KDE.

9. Log out then back in again, and you should see that everything is running as expected:

user $ rc-status | grep clam
 clamd                                                             [  started  ]

user $ ps -ef | grep clam | grep -v grep
clamav    1920     1  0 01:48 ?        00:00:00 /usr/sbin/clamd
clamav    1929     1  0 01:48 ?        00:00:00 /usr/bin/freshclam -d

user $ ps -ef | grep GUI | grep -v grep
fitzcarraldo      9143  8971  0 13:56 ?        00:00:00 /bin/bash /home/fitzcarraldo/.config/autostart-scripts/monitorDownloadsGUI.sh

10. To test, surf to http://www.eicar.org/85-0-Download.html and download one of the EICAR test files into your ~/Downloads/ directory. You should see a pop-up KDialog window with a message similar to the following:

Virus scan of /home/fitzcarraldo/Downloads/eicarcom2.zip — KDialog

Mon 27 Feb 14:05:26 GMT 2017
/home/fitzcarraldo/Downloads/eicarcom2.zip: Eicar-Test-Signature FOUND
/home/fitzcarraldo/Downloads/eicarcom2.zip: moved to ‘/home/fitzcarraldo/virus-quarantine/eicarcom2.zip’

———– SCAN SUMMARY ———–
Infected files: 1
Time: 0.001 sec (0 m 0 s)

Note that the above-mentioned pop-up window may be preceded by one or more pop-up windows with an error message. I’m using the Chrome browser at the moment, but you may get a similar message if you are using another browser. Here is an example:

Virus scan of /home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm — KDialog ?

Mon 27 Feb 14:16:30 GMT 2017
/home/fitzcarraldo/Downloads/.com.google.Chrome.Uh3oGm: Access denied. ERROR

———– SCAN SUMMARY ———–
Infected files: 0
Total errors: 1
Time: 0.000 sec (0 m 0 s)

Read the error message and click ‘OK’, as this is not an actual problem; it is inotifywait detecting temporary files in the ~/Downloads/ directory during the download process. With larger files sometimes several such messages are displayed, presumably because the file being downloaded is being opened and closed more than once during the downloading process. This issue does not occur if you copy or move a file into ~/Downloads/ from another directory in your installation; try it and see for yourself. Then you only get the one pop-up window with the scan result for the file you put in ~/Downloads/.

Also have a look in ~/virus-quarantine/ and you will see the EICAR test file in that directory. You can delete it if you want (it is not infected with a real virus, so does no harm).

In future be sure to read the messages in the pop-up windows before clicking ‘OK’, as they will inform you that an infected file has been moved to the quarantine directory.

That’s all there is to it. Very simple, and quite handy if you want to check quickly that files you download don’t have a malware payload. Just make sure you download all files into ~/Downloads/ or they will not be checked automatically. Also, if you are given e.g. a USB pen drive with a file on it, you can copy the file to ~/Downloads/ if you want it to be scanned for malware.

BT Broadband and Netflix on a smart TV: a solution at last?

The fix I described in an earlier post stopped working after a couple of weeks, and my family was again unable to access Netflix on our smart TV. Some people had reported that replacing their BT Home Hub 3 with a BT Home Hub 5 resolved the Netflix problem with their smart TVs, so I decided to buy a second-hand BT Home Hub 5 and can report that Netflix is again accessible via the Netflix app on my LG smart TV.

A post by an owner of a Sony smart TV reinforces my suspicion that the problem is due to a combination of providers:

The problem is that netflix app adds 8.8.4.4 as a DNS server. Using anything other than the homehub default gateway as the DNS server returns an error from parental controls as it uses that to apply said controls. No other app on my Sony smart TV does that and just uses what the DHCP server on the homehub hands out. The netflix app has to stop adding that backup IP address for DNS. Is it Sony setting that value or Netflix? Considering it is happening on non-Sony TVs the finger looks to be pointing at Netflix.

Anyway, I just hope the problem is finally solved in my case. The price I paid for the second-hand BT Home Hub 5 was reasonable, and it does have some advantages over the BT Home Hub 3 I was using previously, such as support for 802.11ac Wi-Fi and no need for an external DSL modem, so I am satisfied with the outcome although my opinion of BT is even lower now.

BT Broadband could be cause of Netflix app’s ui-113 message

A couple of months ago I posted regarding a problem with the Netflix app on my LG smart TV (Netflix – Not fit for purpose?). We use BT Infinity broadband with a BT HomeHub 3. The Netflix app on the TV stopped working in early January and always displayed a Netflix ‘ui-113’ message when trying to connect. As we are paying good money to Netflix, we resorted to trying various DNS server addresses posted on the Web for US Netflix, which only work for a short time. Although US Netflix has more content than UK Netflix, we want the latter as it has UK programmes not available on US Netflix. Not to mention that we live in the UK. And it seems my family is not the only one suffering: BT Broadband Users Suffer Problems with Netflix on UK Smart TVs.

Today I found a YouTube video regarding a different Netflix error code (‘nw-2-5’) but, having tried everything else, I tried the suggested fix anyway (Enable Parental Controls then Disable Parental Controls then Delete Parental Controls) and then changed the DNS server address setting in the TV back to the default 192.168.1.254 for the BT Home Hub. Although the Netflix app first displays the ‘ui-113’ or ‘nw-2-5’ messages, if I retry a few times it now manages to connect to UK Netflix. After a couple of months of hassle with Netflix, that is progress.

The fix suggested in the above-mentioned YouTube video consisted of enabling then disabling and deleting BT Parental Controls via the user’s account page on the BT Web site. Even though I had never previously enabled BT Parental Controls, I logged in to the BT Web site, enabled Parental Controls, and then disabled and deleted them as per the instructions given in the aforementioned YouTube video.

So it seems that BT is either the cause of the problem or a major contributor. I suspect the blame may lie with more than one company, though, because: a) the Netflix app’s ‘ui-113’ and ‘nw-2-5’ messages occur even after I made sure BT Parental Controls really are disabled; b) it is touch and go whether the app is successful in accessing the Netflix site at the first attempt; c) other devices accessing Netflix via my home network don’t suffer from the problem. I just hope LG, Netflix and BT are working together to solve it properly, because the current situation is completely unacceptable.

Thunderbird’s defective method of enabling anti-virus software to scan incoming POP3 e-mail messages

Thunderbird’s method of enabling anti-virus software to scan incoming e-mail messages is explained in the mozillaZine article ‘Download each e-mail to a separate file before adding to Inbox‘ and in Mozilla bug report no. 116443 (the bug report that resulted in the functionality being implemented). It is my contention that the design is deficient and is actually not a solution. In this post I explain why I believe this to be the case. Although here I will discuss Thunderbird in Linux, I believe the deficiency applies to Thunderbird in all OSs.

By default, Thunderbird inserts new incoming e-mail messages into an Inbox file. However, it is possible to configure Thunderbird to first create a temporary file containing each individual e-mail message in the /tmp directory, to enable external anti-virus software to scan each message before Thunderbird inserts it into the Inbox file. This approach only works for POP3 e-mail. The developers’ rationale for implementing this approach was to avoid the possibility of anti-virus software deleting or quarantining an entire Inbox.

In summary, if you want to scan incoming e-mails on your machine without running the risk of losing the entire Thunderbird Inbox, you must:

  1. configure Thunderbird so it creates temporary files /tmp/newmsg* (each file contains a single e-mail message containing ASCII characters);
  2. configure the anti-virus software not to scan the directory containing the Inbox;
  3. configure the anti-virus software to scan the /tmp directory.

Nevertheless, it seems Thunderbird developers would prefer you to disable local scanning of e-mail messages entirely: ‘mozillaZine – Email scanning – pros and cons‘.

Nowadays e-mail servers scan e-mail messages before you even download them. Some e-mail servers even send you an automated e-mail to inform you about an infected incoming e-mail or about an infected outgoing e-mail rejected by a receiving e-mail server. So local scanning of e-mail messages is far less important. Furthermore, I am not sure if the anti-virus software I use (ClamAV) is capable of detecting viruses in e-mail attachments encoded as ASCII characters. Anyway, purely out of curiosity I decided to investigate whether it would be possible to scan Thunderbird’s temporary files reliably.

To configure Thunderbird to create the temporary message files, it is necessary to select ‘Edit’ > ‘Preferences’ > ‘Security’ > ‘Antivirus’ and tick ‘Allow anti-virus clients to quarantine individual incoming messages’ (which sets mailnews.downloadToTempFile to true). Once that option has been selected, Thunderbird creates a temporary file /tmp/newmsg per message, which exists for a very brief (and inconstant) time before Thunderbird deletes it. When downloading several e-mail messages in very rapid succession, Thunderbird creates temporary files with different names (‘newmsg‘, ‘newmsg-1‘, ‘newmsg-2‘, ‘newmsg-3‘ and so on) to avoid overwriting messages, but usually one file named newmsg is sufficient to cater for the message download rate, as Thunderbird only keeps the temporary files for a very short time until it moves the message to the Inbox.

The problem with this approach is that Thunderbird does not provide any handshake mechanism to inform external anti-virus software that it has finished writing to a temporary file and that the file is available for scanning, nor does Thunderbird provide any handshake mechanism for external anti-virus software to inform Thunderbird when the scan of each temporary message file has finished (i.e. to tell Thunderbird that it can go ahead and delete the temporary file). In other words, the Thunderbird ‘solution’ is not a solution at all. In fact, I have found empirically that, if the anti-virus software is not fast enough, it can scan an incomplete temporary message file (i.e. the evaluation of the e-mail message would not be thorough and hence would be invalid). The Bash script below, for example, is sometimes able to scan an entire Thunderbird temporary file but at other times only manages to capture part of the file (it appears Thunderbird opens and closes the temporary file more than once) before Thunderbird deletes it:

#!/bin/bash

# This script only works with Thunderbird.
# This script only works for POP3 e-mail.
#
# You must configure Thunderbird to create temporary files /tmp/newmsg*.
# To do that, set Edit > Preferences > Security > Antivirus and tick
# 'Allow anti-virus clients to quarantine individual incoming messages'
# which sets mailnews.downloadToTempFile to true.

WORK=$HOME/clamtmp
mkdir $WORK 2> /dev/null
rm $WORK/* 2> /dev/null

counter=1

# Watch for newmsg* file(s) created by Thunderbird in /tmp
inotifywait -q -m -e close_write --format '%f' /tmp | while read FILE
do
     if [ "${FILE:0:6}" = "newmsg" ] && [ -s /tmp/$FILE ]; then
          TMPFILE=${counter}$FILE
          cp -p /tmp/$FILE $WORK/$TMPFILE
          # Do not let clamscan write temporary files to /tmp as inotifywait will detect them!
          clamscan --tempdir=$WORK $WORK/$TMPFILE
          counter=$((counter+1))
     fi
done

Below is an example of an incomplete newmsg file that the above script copied to the directory $WORK when Thunderbird downloaded an e-mail message:

From - Sun Feb 21 09:40:58 2016
X-Account-Key: account8
X-UIDL: AAAAAKlA+Ah2LghLoJE4Le5Z5U0BAI04UNOj2gdNjwPO57yvmrIAATVHik4AAA==
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:

I began to wonder if a valid scan would be possible if the script were to lock the temporary file (e.g. using the chattr +i command) until it has completed copying it. The use of the chattr command in the script means it has to be executed by the root user. When I first did that with a modified version of the script using KDialog to display the result of ClamAV’s scan, the following error message was displayed and the script aborted:

kdialog(xxxxxx)/kdeui (kdelibs): Session bus not found
To circumvent this problem try the following command (with Linux and bash)
export $(dbus-launch)

I therefore added the line ‘export $(dbus-launch)‘ to the script as follows:

#!/bin/bash

# This script must be launched by the root user.
export XAUTHORITY="/home/fitzcarraldo/.Xauthority"
export DISPLAY=":0"
export $(dbus-launch)

WORK=$HOME/clamtmp
mkdir $WORK 2> /dev/null
rm $WORK/* 2> /dev/null

inotifywait -q -m -e modify --format '%f' /tmp | while read FILE
do
     # If file name begins with "newmsg" then copy it to work directory and scan it.
     if [ "${FILE:0:6}" = "newmsg" ] && [ -f /tmp/$FILE ]; then
          chattr +i /tmp/$FILE # Stop Thunderbird opening/deleting file.
          cp -p /tmp/$FILE $WORK/
          chattr -i /tmp/$FILE # Allow Thunderbird to open/delete file.
          clamscan --tempdir=$WORK $WORK/$FILE >> $WORK/$FILE.log
          kdialog --msgbox "$(cat $WORK/$FILE.log)"
     fi
done

However, when the above script was running, Thunderbird displayed a pop-up window if it attempted to copy the temporary file to the Inbox before the script released the lock on the file:

There was an error copying the downloaded message from the temporary download file. Perhaps it contained a virus, or you are low on disk space.
From:
Subject: Test to see what happens if script locks newmsg file
Do you want to skip this message?

I clicked ‘No’ and the e-mail message in the file /tmp/newmsg was deleted without being copied to the Inbox, and the message was not deleted from the e-mail server. Subsequent attempts to re-download the message resulted in the same behaviour if the script had not finished processing the message before Thunderbird tried to move the message to the Inbox. Had I clicked ‘Yes’ I assume Thunderbird would simply have deleted the message on the mail server.

I did not bother looking into it further, but presumably the chattr command triggers inotifywait, in which case the script could cycle several times for the same file.

An approach that would probably work would be for Thunderbird to provide some sort of interlock so that it waits to delete newmsg* files until an anti-virus application gives the go-ahead.

An alternative approach would be for Thunderbird not to delete a temporary file after it writes the message to the Inbox and just leave it in the /tmp directory without overwriting it. An anti-virus application would quarantine infected temporary files and leave uninfected temporary files in /tmp, and therefore the anti-virus application would have to be written so that it deletes the temporary file once it has finished scanning it.

The second approach mentioned above would not be as good as the first approach for the following reasons:

  1. It would not stop Thunderbird adding a message to the Inbox (which would mean the user would have to delete a message manually from the Thunderbird Inbox if the virus scanner reported a message as infected).
  2. Thunderbird would have to use a different file name to any existing temporary files (at present it reuses ‘newmsg‘ if a file of that name does not already exist).
  3. The user would have to ensure the temporary files do not accumulate ad infinitum. In my case, the contents of the /tmp directory are deleted each time I reboot, but, in theory, a partition could become full if a user never switched off a machine and received a lot of e-mails.

Regarding the second reason listed above, Thunderbird already names the temporary files ‘newmsg‘, ‘newmsg-1‘, ‘newmsg-2‘ etc., so perhaps the existing Thunderbird code would automatically use a different file name if a file with the same name were still present, rather than overwriting it. If e.g. files newmsg and newmsg-2 happened to exist, I would hope Thunderbird would name the next temporary file ‘newmsg-1‘ or ‘newmsg-3‘.

I wondered if it would be possible to catch up with Thunderbird by just copying the temporary message files from the /tmp directory to another directory (see the script below), and then processing them afterwards with another script. However, even if a script just copies the temporary files to another directory without running ClamAV or KDialog, I found it is still not fast enough to catch all temporary files before Thunderbird deletes them. If Thunderbird downloads a single message and no others are waiting on the server(s) to be downloaded, it seems a script can copy the temporary messages successfully. However, if there are several messages waiting to be downloaded from the e-mail server(s) and Thunderbird downloads them in rapid succession, Thunderbird deletes some of the temporary messages before the script can copy them fully.

#!/bin/bash

WORK=$HOME/clamtmp
# Create work directory if it does not already exist
mkdir $WORK 2> /dev/null
# Delete old working files if they exist
rm $WORK/* 2> /dev/null

counter=1

inotifywait -q -m -e close_write --format '%f' /tmp | while read FILE
do
     if [ "${FILE:0:6}" = "newmsg" ] && [ -s /tmp/$FILE ]; then
          TMPFILE=${counter}$FILE
          cp -p /tmp/$FILE $WORK/$TMPFILE
          counter=$((counter+1))
     fi
done

Consider the following six messages copied to $HOME/clamtmp by the above script (the script adds the first character to the name of the copied file):

-rw-------   1 fitzcarraldo fitzcarraldo    3829 Feb 23 02:24 1newmsg
-rw-------   1 fitzcarraldo fitzcarraldo    3107 Feb 23 02:25 2newmsg
-rw-------   1 fitzcarraldo fitzcarraldo 1158576 Feb 23 02:26 3newmsg
-rw-------   1 fitzcarraldo fitzcarraldo     237 Feb 23 02:28 4newmsg
-rw-------   1 fitzcarraldo fitzcarraldo    2106 Feb 23 02:28 5newmsg-1
-rw-------   1 fitzcarraldo fitzcarraldo    3107 Feb 23 02:28 6newmsg-2

The first three messages were each the sole message on all three e-mail servers accessed, and the copied files newmsg -> 1newmsg, newmsg -> 2newmsg and newmsg -> 3newmsg were all complete messages. However, the last three messages were on three e-mail servers simultaneously waiting to be downloaded, and when I clicked on ‘Get All New Messages’ in Thunderbird, the copied files newmsg -> 4newmsg, newmsg-1 -> 5newmsg-1 and newmsg-2 -> 6newmsg-2 were downloaded by Thunderbird in rapid succession. The copy 4newmsg was incomplete, the copy 5newmsg-1 was complete and the copy 6newmsg-2 was complete. So, even with a faster script, there is no guarantee that a script can catch all the temporary message files. Therefore, as I mentioned earlier, the only way to guarantee that temporary message files are properly scanned would be to modify Thunderbird to provide either a handshake (e.g. a file lock or inter-application flag) or to leave each temporary message file on /tmp and give it a unique file name.

The downside with both the above-mentioned approaches would be that the anti-virus software developer would need to know about the method, and write the software to perform the appropriate actions. If the first approach were adopted, the anti-virus software would need to signal to Thunderbird that it had completed scanning the file (e.g. by releasing a file lock or by an inter-application message). If the second approach were adopted, the anti-virus software would need to delete the message file from /tmp once it had completed scanning the file. The second approach would be easier for a simple Bash script to use, and, had the Thunderbird source code not been so complicated, I would have had a go at patching it to leave temporary message files in the /tmp directory after Thunderbird copies their contents to the Inbox file. But, as e-mail servers already do a good job of scanning messages before Thunderbird downloads them, I will not spend more time on this. Some e-mail servers even send an e-mail to the user informing them about an infected e-mail (see examples below), so it is not worth bothering.

Example 1
Automated e-mail server message to john@smith.com warning him that the e-mail with an infected attachment he sent to dave@hotmail.com was blocked by the receiving e-mail server.

Subject: Mail delivery failed: returning message to sender
Date: Wed, 17 Feb 2016 11:30:10 +0100
From: Mail Delivery System
To: john@smith.com

This message was created automatically by mail delivery software.

A message that you sent could not be delivered to one or more of
its recipients. This is a permanent error. The following address
failed:

Reason:
virus/suspect content found

— The header of the original message is following. —

Example 2
Automated e-mail server message to john@smith.com warning him that an e-mail with an infected attachment sent to him by dave@hotmail.com was blocked by the receiving e-mail server.

Subject: VIRUS SUSPECTED: “Dave (Hotmail)”
Date: Wed, 17 Feb 2016 11:46:07 +0100 (CET)
From: Mail Delivery System
To: john@smith.com

A virus was detected in the following e-mail!

Mail details:

From: “Dave (Hotmail)”
TO: John Smith
Date: Wed, 17 Feb 2016 10:45:55 +0000
Subject: EICAR test file attachment

The concerned e-mail has been handled according to your Virus Protection Settings.

Sincerely
Your E-mail Service Provider Team

[ This is an automatically generated email, do not reply to this sender. You may find more
information in the online help of your client. ]

Other articles of interest: mozillaZine – Antivirus software.

Automatically detecting files placed in my Downloads directory in Gentoo Linux and scanning them for viruses

I have been using Linux for almost a decade and have never been unduly concerned about viruses on my machines running Linux. However, I do receive files from people who use Windows and Mac OS, and some of those files might contain Windows or Mac OS viruses, so, as a matter of courtesy and assistance to others, it would make some sense to scan those files before passing them on. Furthermore, as I use some Windows applications under WINE, it would also make sense to scan received files for Windows viruses if I am going to use those files with a Windows application running under WINE.

External files could get into my Gentoo Linux installations via pen drives, memory cards, optical discs, e-mails, my Dropbox directory and downloads from Web sites. In this post I am going to concentrate on the last of these. All the various e-mail account providers I use already scan e-mails for viruses on their e-mail servers before I even download e-mail into the e-mail client on my laptop (standard practice these days), so e-mail is not a particular worry.

I have had ClamAV and its GUI, ClamTk, installed for a long time. Whilst ClamTk can be used to schedule a daily update of virus signatures and a daily scan of one’s home directory by ClamAV, I normally run ClamTk and ClamAV ad hoc. However, I can see some benefit in launching ClamAV automatically when I download a file from the Internet, so I decided to do the following …

Automatically scan a file downloaded via a Web browser

I use Firefox to browse the Web, and had configured it to download files to the directory /home/fitzcarraldo/Downloads/. I decided to monitor automatically the Downloads directory for the addition of any file. As I use the ext4 file system, the method I opted to use is inotify, specifically the inotifywait command which is available once you install the package sys-fs/inotify-tools.

It is surprisingly easy to create a shell script to detect files downloaded into a directory. The following script, running continuously in a terminal, would detect any files created in my /home/fitzcarraldo/Downloads directory, scan the new files with ClamAV and display a report in the terminal window:

#!/bin/bash

echo
DIR=$HOME/Downloads

inotifywait -q -m -e create --format '%w%f' $DIR | while read FILE
do
     date
     echo "File $FILE has been detected. Scanning it for viruses now ..."
     clamscan $FILE
     echo
done

A usable script would need to be a bit more sophisticated than the one shown above, because an existing file in the directory could be overwritten by one with the same name, or opened and amended. Furthermore, the script above would need a permanently open terminal window. Therefore I created a script to run in the background and use a GUI dialogue tool to pop up a window with the virus scanner’s report when the script detects a new or changed file in the Downloads directory. As this laptop has KDE 4 installed I opted to use KDialog to display the pop-up window, but I could instead have used Zenity. The final script is shown below.

#!/bin/bash

DIR=$HOME/Downloads

# Get rid of old log file
rm $HOME/virus-scan.log 2> /dev/null

inotifywait -q -m -e close_write,moved_to --format '%w%f' $DIR | while read FILE
do
     # Have to check file length is nonzero otherwise commands may be repeated
     if [ -s $FILE ]; then
          date > $HOME/virus-scan.log
          clamscan $FILE >> $HOME/virus-scan.log
          kdialog --title "Virus scan of $FILE" --msgbox "$(cat $HOME/virus-scan.log)"
     fi
done

Now when I download a file in Firefox, a window pops up, displaying a message similar to the following:

Virus scan of /home/fitzcarraldo/Downloads/eicar_com.zip – KDialog

Fri 19 Feb 23:42:02 GMT 2016
/home/fitzcarraldo/Downloads/eicar_com.zip: Eicar-Test-Signature FOUND

———– SCAN SUMMARY ———–
Known viruses: 4259980
Engine version: 0.98.7
Scanned directories: 0
Scanned files: 1
Infected files: 1
Data scanned: 0.00 MB
Data read: 0.00 MB (ratio 0.00:1)
Time: 4.595 sec (0 m 4 s)

Notice in the above message that ClamAV detected a virus in a file eicar_com.zip that I downloaded from the European Expert Group for IT Security Web site (originally ‘European Institute for Computer Antivirus Research’). In fact the executable eicar.com does not contain a real virus; it was designed to contain a known signature that virus scanner creators and users can use in checking anti-virus software. You can find out more about the virus test files on the EICAR Web site.

Of course, if I use applications other than Firefox to download files, I need to make sure they download the files into the applicable directory so that the script can detect and scan the files:

fitzcarraldo@clevow230ss ~ $ cd Downloads/
fitzcarraldo@clevow230ss ~/Downloads $ youtube-dl -o Carnavalito.mp4 -f 18 https://www.youtube.com/watch?v=ZDUL3w7zFD4
ZDUL3w7zFD4: Downloading webpage
ZDUL3w7zFD4: Downloading video info webpage
ZDUL3w7zFD4: Extracting video information
ZDUL3w7zFD4: Downloading MPD manifest
[download] Destination: Carnavalito.mp4
[download] 100% of 16.61MiB in 00:05

So, now I have a shell script that pops up a window informing me whether or not any file I put in $HOME/Downloads/ contains a virus. But I would like the script to be launched automatically when I login to the Desktop Environment. Therefore, as I use KDE 4, I selected ‘System Settings’ > ‘Startup and Shutdown’ and, in the ‘Autostart’ pane, clicked on ‘Add Script…’ and entered the path to my shell script (I left ‘create as symlink’ ticked). Now, every time I use KDE, any file placed (automatically or manually) into $HOME/Downloads/ is scanned for viruses automatically and a window pops up giving the result.

As my laptop is not always connected to the Internet, I prefer to update the ClamAV virus signatures database manually, which I do either using the ClamTk GUI or via the command line using the freshclam command:

fitzcarraldo@clevow230ss ~ $ su
Password:
clevow230ss fitzcarraldo # freshclam
ClamAV update process started at Sat Feb 20 10:51:01 2016
WARNING: Your ClamAV installation is OUTDATED!
WARNING: Local version: 0.98.7 Recommended version: 0.99
DON'T PANIC! Read http://www.clamav.net/support/faq
main.cvd is up to date (version: 55, sigs: 2424225, f-level: 60, builder: neo)
Downloading daily-21375.cdiff [100%]
Downloading daily-21376.cdiff [100%]
Downloading daily-21377.cdiff [100%]
Downloading daily-21378.cdiff [100%]
Downloading daily-21379.cdiff [100%]
Downloading daily-21380.cdiff [100%]
Downloading daily-21381.cdiff [100%]
Downloading daily-21382.cdiff [100%]
Downloading daily-21383.cdiff [100%]
Downloading daily-21384.cdiff [100%]
Downloading daily-21385.cdiff [100%]
Downloading daily-21386.cdiff [100%]
Downloading daily-21387.cdiff [100%]
Downloading daily-21388.cdiff [100%]
Downloading daily-21389.cdiff [100%]
Downloading daily-21390.cdiff [100%]
Downloading daily-21391.cdiff [100%]
daily.cld updated (version: 21391, sigs: 1850214, f-level: 63, builder: neo)
bytecode.cld is up to date (version: 271, sigs: 47, f-level: 63, builder: anvilleg)
Database updated (4274486 signatures) from db.UK.clamav.net (IP: 129.67.1.218)
WARNING: Clamd was NOT notified: Can't connect to clamd through /var/run/clamav/clamd.sock: No such file or directory

Netflix – Not fit for purpose?

One of my family has a Netflix account. The account is accessible from any of the desktop and laptop computers in the house, whichever OS they are running.

Recently we bought a so-called ‘smart TV’ (an LG 40UF770V 4K Ultra HD TV, as it happens), and are pleased with it. It runs WebOS 2.0 (yay, Linux!) and the LG Content Store contains a Netflix app, which we promptly installed. The app worked perfectly for several weeks but then stopped being able to access Netflix. When the app was launched, the usual screen with the Netflix logo and the ‘Loading’ rotating indicator would appear but, after a minute or so, an error message would be displayed informing us that Netflix error ‘ui-113’ had occurred. One of the on-screen options then offered by the app was to check the network connection, which we tried, but everything was reported to be working correctly. Not to mention that all the other apps that require an Internet connection work fine. In order to watch a film using Netflix over the Christmas period we had to resort to connecting a laptop to the TV via an HDMI cable and accessing Netflix in a Web browser on the laptop. It is ridiculous to have to resort to such measures to view content on smart TVs which have Netflix apps.

I searched the Web and discovered that many, many people experience the same problem with Netflix and smart TVs. As in our case, they had no trouble accessing their Netflix account on their home network with other devices such as computers, tablets and smart phones. I came across reports by owners of smart TVs made by LG, Philips, Samsung, Sony, Toshiba, and other manufacturers. People who had contacted the relevant TV manufacturer were often told the problem is caused by Netflix, and people who had contacted Netflix were often told the problem is caused by the TV manufacturer.

Netflix has a Help page for this error message, but none of the steps Netflix listed worked, and neither did any of the remedies suggested by others on the Web (including in various YouTube videos). Resetting the TV did not solve the problem. Neither did cycling the mains power to the TV, broadband modem and router (however long the power was off). Nor did changing the TV’s setting for the IP address of the DNS server to one of the well-known public DNS servers such as Google’s. Nor did suggestions such as un-installing and re-installing the Netflix app. Nor did configuring the router to perform port forwarding for Netflix on the TV (not that this should be necessary, but I tried it anyway). Several people wrote that the parental lock in their routers caused the problem, but the parental lock is definitely not enabled in my router. I also tried to access Netflix via the TV’s Web browser; it can log-in to the Netflix account but cannot play content as it does not support the Microsoft Silverlight plug-in or HTML5 required by Netflix.

Nothing we tried solved the problem, and two weeks of this messing around was exasperating. Some people reported that changing the DNS server address in the TV to Google’s DNS servers worked, whereas others reported it didn’t. Even if some lucky person managed to get Netflix working on their smart TV using a certain procedure, other people in the same country with the same model of TV could not, even if they used the same procedure.

In addition to people in a given country trying to get the Netflix app in their smart TV to access their Netflix account in that country, I came across posts by people wanting to access Netflix in a different country (mostly people not in the USA wanting to access US Netflix because it offers a wider range of films and programmes, but also expatriates wanting to access Netflix for their home country with their home-country Netflix account). So I tried recommendations to configure the TV to use a DNS server in the US that some people in the UK had recently indicated would give the Netflix app access to US Netflix rather than UK Netflix (even though we wanted to access UK Netflix from the UK). But that didn’t work either.

However, I didn’t give up. I trawled the Web for lists of DNS servers that some people claimed would give access to Netflix in the UK. I found various Web sites listing IP addresses for DNS servers and tried several of them. Eventually I found one that actually enables the Netflix app in the TV to work, but it accesses US Netflix instead of UK Netflix. Given that the Netflix app has not worked for several weeks, I’m not complaining, but it is not what we were trying to achieve (US Netflix does not provide all the UK TV series available on UK Netflix). Furthermore, according to some of the posts I’ve read, periodically you have to change the DNS server address in the TV because Netflix stops working with the existing address.

Now, I’m a technically-oriented person and it took me several hours over a two-week period to find a solution (well, a work-around). Someone with little or no IT knowledge in the same situation would be at a complete loss as to how to get their Netfix account working. In order for streaming media services to become as ubiquitous as e.g. terrestrial TV, they have to be reliable and be accessible easily to paying customers. Use of Geolocation, GeoDNS and other complex techniques should not cause such a headache to bona fide users. Someone with a Netflix account in his/her country of residence and who simply wants to access Netflix on a smart TV should not have to jump through hoops or hit a brick wall. Clearly this is happening to many people.

On top of that, people such as myself who have to travel internationally frequently because of their work need to be certain that, if they subscribe to a streaming media service, it will work in whatever country they happen to be in at the time (except if blocked by Great Firewalls or content filters on proxy servers, of course) and not be purposely or inadvertently prevented from working by the media service provider’s network concept.

I myself had considered signing up for a Netflix account so that I could view films and TV programmes during my overseas work trips, but, after having to struggle for days to help a family member access a valid Netflix account on a smart TV in the country where the account was set up, will definitely not be giving Netflix my business. In this day and age it is ridiculous that users should have to try umpteen DNS server addresses and reset TVs, routers and modems in order to access their account with a media provider. Services such as Netflix will never have my business until their networking and DRM are sorted out properly and made to work reliably. Until Netflix changes its network delivery model, its service will remain a curate’s egg in my opinion.

Using a keyboard shortcut in Linux to add an e-mail signature giving current location and local time

In my previous post I showed how to find the current time at any town or city Worldwide from the command line in Gentoo Linux. My interest in a command to do this is not to use it on the command line per se, but to use the command in a keyboard shortcut to insert a signature at the end of my e-mails.

I have to travel internationally frequently because of my work, but I leave my laptop’s hardware clock set to UTC and the system clock set to the local time of my home town. This means that, irrespective of where I am in the World, the e-mail client (Thunderbird, in my case) uses the local time of my home town in e-mail headers and calenders. It is not practical to reconfigure Linux for each timezone I happen to be in (see my post Configuring the Linux clock), and, in any case, I want the file system’s timestamps to use one timezone only and all the timestamps in my e-mails and the e-mail client’s calender to use one timezone only, so there is less chance of me getting confused. I could have configured the installation to use UTC for the system clock, but I prefer the system clock to use the timezone of my home town. Of course, even though the system clock is always set to the timezone of my home town, on the Panel clock I select the timezone of the location where I happen to be, so that the Panel clock displays the local time in that timezone.

I wanted to be able to insert a signature at the end of each e-mail, stating my current location and the current time at that location, so that the person receiving the e-mail could tell from where in the World I sent the e-mail and the local time it was sent, as that local time could differ from the time shown in the e-mail header. For example, let us assume that Jane, who lives in the UK and whose system clock is configured for the timezone Europe/London, is making a brief visit to Perth, Australia and sends an e-mail to Dave in the UK at 06:36 on 11 October (Perth time). The e-mail below illustrates the type of signature I wanted to achieve.

Subject: Site visit
From: Jane <jane@acompany.com>
To: Dave <dave@acompany.com>
Date: Sat Oct 10 2015 23:36:40 GMT+0100 (BST)

Hello Dave,

This is to let you know that I have just arrived in Perth and will be
visiting site at 09:00 local time to speak to the client. Tomorrow p.m.
I have a meeting scheduled with our local project manager, so I would
appreciate it if you would e-mail the latest documentation to me. I will
not have spare time until I’m in my room at the hotel tonight but will
read the documents tomorrow a.m. in readiness for the meeting with
the local project manager. Thanks in advance.

Regards,
Jane
Current location: Perth (Australia)
Local time now: Sat Oct 11 06:36:31 2015 AWST

As you can see above, because the OS on Jane’s and Dave’s laptops is configured for the timezone Europe/London, the e-mail header shows the current time in the UK when the e-mail was sent, which was October 10, 23:36 British Summer Time (22:36 UTC), and the signature shows the corresponding local time in Perth, Australia, which was October 11, 06:36 Australian Western Standard Time. It becomes even more confusing if the computer of the person receiving the e-mail is configured for a third timezone. For example, let’s say Dave is based in Seattle, USA rather than the UK. His e-mail client would then display the time in that timezone when the e-mail was sent. This is usually my case, i.e. my Linux installation is configured for Timezone1 but I happen to be in Timezone2 when I send an e-mail to someone who is based in Timezone3 and whose OS is configured for that timezone.

I wanted to use a keyboard shortcut to add a signature to the end of my e-mails, as shown above. I therefore created the Bash script listed below, which I named timezone_signature_GeoNames.sh:

#!/bin/bash

location=$(kdialog --title "Current Location" --inputbox "Enter your location:")

localtime=$(perl /home/fitzcarraldo/now1.pl $location)
place=`echo $localtime | cut -d'|' -f1`
place=$place" "`echo $localtime | cut -d'|' -f2`
timezone=`echo $localtime | cut -d'|' -f4`

if [ $location != "" ]; then
  echo -n "Current location: "
  echo $place
  echo -n "Local time now:"
  /usr/sbin/zdump ${timezone} | cut -d' ' -f2-
fi
echo

Notice that the Bash script uses the GUI dialogues utility kdialog to display a pop-up window prompting me to enter the name of a town/city. As I am using KDE I opted to use a dialogues utility developed for use in KDE, but I could have used Zenity instead.

The Perl script now1.pl is a variant of the Perl script now.pl described in my previous post, modified very slightly in order to facilitate formatting of the output by the Bash script, and is listed below.

#!perl

use strict;
use warnings;

use DateTime;
use Geo::GeoNames;
use URI::Escape;
use Encode;

binmode STDOUT, ':encoding(UTF-8)';

my $city = decode("UTF-8", @ARGV ? shift : 'London');
# N.B. Replace London with your home town/city.

my $geo = Geo::GeoNames->new( username => '************' );
# N.B. Replace the asterisks with your GeoNames user name.

my $result = $geo->search(
q       => uri_escape_utf8($city),
maxRows => 1,
style   => 'FULL'
);

defined $result->[0] or die "Unrecognized city '$city'\n";

my $city_name    = $result->[0]->{name};
my $country_name = $result->[0]->{countryName};
my $time_zone    = $result->[0]->{timezone}{content};
my $time_now     = DateTime->now( time_zone => $time_zone );

#print "$city_name ($country_name) $time_now ($time_zone)\n";

print "$city_name|($country_name)|$time_now|$time_zone\n";

exit 0;

The only thing remaining was to configure a keyboard shortcut to launch the Bash script. I opted to use the key combination Ctrl-Alt-z for the shortcut. As I am using KDE I could have used KDE’s ‘System Settings’ > ‘Shortcuts and Gestures’> ‘Custom Shortcuts’ to specify the shortcut and the name of the script it launches. However, as I also use AutoKey for various shortcuts, I opted to use that instead, so I used the AutoKey GUI to create a shortcut named ‘Insert Current Time’ to use the following command:

output = system.exec_command("/home/fitzcarraldo/timezone_signature_GeoNames.sh")
keyboard.send_keys(output)

Use

I compose my e-mails as usual, and, after entering my name at the end of the e-mail, I press Ctrl-Alt-z. A window then pops up prompting me to enter my current location, which I do and then click on ‘OK’. The location and current time at that location are then added to the end of the e-mail, and it just remains for me to click on the ‘Send’ button in the e-mail client’s window. As the Perl script now1.pl uses the Internet to access the GeoNames database, my laptop must be connected to the Internet when I use the shortcut.

KDialog window that pops up when I use the keyboard shortcut

KDialog window that pops up when I use the keyboard shortcut

If the town/city name consists of more than one word (Rio de Janeiro, for example) then replace spaces with hyphens when you enter the location name in the pop-up window (Rio-de-Janeiro, for example) and then the keyboard shortcut will return the correct location and local time:

Current location: Rio de Janeiro (Brazil)
Local time now: Fri Oct 16 09:28:19 2015 BRT