Bypassing a corporate Web filter when using the command line

or ‘How to bypass a corporate Web filter and download YouTube videos via the command line’

One of the offices where I work uses a Web filter to block access to certain sites, such as YouTube. However, sometimes it is necessary to view blocked Web sites for work purposes. For example, these days a lot of companies or individuals post product reviews on YouTube that are useful for work purposes. In such cases I have used Tor to access the blocked sites in a Web browser such as Firefox, Chrome, Konqueror etc. See my post How to install and use Tor for anonymous browsing or to access country-restricted content from another country for details of how to set up and use Tor with a Web browser.

But sometimes I need to access blocked Web sites from the command line. For example, today I needed to download a YouTube video for work purposes, and I wanted to use youtube-dl to do it. The solution was simple…

First I launched vidalia and polipo as explained in the above-mentioned post on Tor, then I launched another Konsole/Terminal window and entered the commands shown below:

$ # First find out what resolutions are available for the video I want to download:
$ youtube-dl -F https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
[info] Available formats for T3Rr4CUoTSQ:
format code extension resolution note
140 m4a audio only DASH audio , audio@128k (worst)
160 mp4 192p DASH video
133 mp4 240p DASH video
134 mp4 360p DASH video
135 mp4 480p DASH video
136 mp4 720p DASH video
17 3gp 176x144
36 3gp 320x240
5 flv 400x240
43 webm 640x360
18 mp4 640x360
22 mp4 1280x720 (best)
$ # Now try to download the video at the resolution I want:
$ youtube-dl -f 22 -o Clevo_W230ST_overview.flv https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
ERROR: unable to download video data: HTTP Error 403: Forbidden

As you can see above, the corporate Web filter blocked youtube-dl from downloading the video.

So I informed the shell session about the local HTTP proxy (polipo) running on my laptop, by assigning and exporting the environment variable http_proxy using the following syntax:

export http_proxy=http://server-ip:port/

which in my case meant the following (refer to my article on Tor):

$ export http_proxy=http://127.0.0.1:8123/

and then I was able to download the video from YouTube despite the corporate Web filter:

$ youtube-dl -f 22 -o Clevo_W230ST_overview.flv https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
[download] Destination: Clevo_W230ST_overview.flv
[download] 100% of 100.23MiB in 05:50
$

Useful Reference: How To Use Proxy Server To Access Internet at Shell Prompt With http_proxy Variable

Can Linux cope with 24 Hours of Happy?

I enjoyed Pharrell Williams’ feel-good songs in ‘Despicable Me‘ and its sequel, ‘Despicable Me 2‘. ‘Happy‘, a very catchy ditty he wrote for the sequel, also features in the World’s first 24-hour-long music video, ‘24 Hours of Happy‘, shot in and around Los Angeles and released on 21 November last year. The song is played a total of 360 times over the duration of the video, each 4-minute take featuring a different person or persons dancing (improvised) along streets, in petrol stations, through Union Station, in a church, around a school, in a moving school bus, around a supermarket, in a bowling alley, and so on. Each 4-minute performance was filmed in one take using Steadicam, and the location at the end of each take segues into the next take. You see the sun rise; you see the bright sunlight of the morning and the warm sunlight of the afternoon; you see the sun set; you see the twinkling city lights at night. The concept is simple yet brilliant.

Clips from some of the takes were used to create the 4-minute official music video for ‘Happy’, so you can watch that on YouTube to get a flavour of the takes, although it does not do justice to the full video.

Williams appears in a different take every hour on the hour, and a few other takes have celebrity cameos, but the vast majority of the participants are unknown extras of all ages, races, shapes, sizes and looks. To quote Williams talking to the Los Angeles Times: “We wanted all humanity in there, not just the model-types you might expect.” Some are good dancers, others not so good. But they all have one thing in common: they’re having fun, so they look good. The joy is infectious, and I found myself watching far longer than I would have expected, having to return to the site again and again. Half the fun is watching the bystanders.

When you open the ’24 Hours of Happy’ site, the take that was in progress at the current time of day starts playing from the beginning. However, you can drag the pointer around the clock dial and watch any take from the 24-hour period. There are also buttons you can click on to jump between takes or to jump to each take featuring Williams. The yellow on-screen controls can be made to disappear by not moving the mouse pointer for 5 seconds.

Still from 24 Hours of Happy, showing on-screen controls

Still from 24 Hours of Happy, showing on-screen controls

The Web site is well-designed and fun to use, so I was not surprised it was voted ‘Site of the Month‘ and ‘Site of the Year Users’ Choice‘ by AWWWARDS, and voted ‘Site of the Month‘ and ‘Site of the Year‘ by TheFWA.

It’s impossible to list them all, but a few of my favourite takes are:

01:36  Very perky woman with ponytail.
05:28  Jogger who has to keep pulling his shorts up!
08:24  Woman on roller skates.
09:52  Very cute little girl.
09:56  Woman with some groovy moves.
10:40  Woman in Union Station. Some of the bystanders are particularly amusing.
11:16  Man with cast on foot.
11:20  Boy with an Afro.
11:36  Three groovy old ladies.
11:44  Chubby guy with style.
12:36  Woman with some groovy moves.
13:32  Dancing couple in pink.
14:20  Two cool guys in dinner jackets inside and outside Union Station.
15:00  Pharrell Williams in a church with a gospel choir.
19:04  Woman with a lizard puppet. The lizard does the lip-synching!
19:36  Guy on stilts.
23:40  Woman with LED hula hoop (love it!).

If you want to start viewing a take made at a specific time of day, you can append the time to the URL, like so:

http://24hoursofhappy.com/09h53am

Obviously I think ’24 Hours of Happy’ is fabulous, but why am I discussing it in a blog predominantly about Linux? Because Firefox 27.0.1 (32-bit) running in Windows 8.1 (64-bit) on my new Acer Aspire XC-600 micro-tower PC (dual-core Intel Pentium G2030 @ 3 GHz & 3 MB cache, 4 GB DDR3 RAM) handles ’24 Hours of Happy’ at 720p with ease, but the story is very different on my main laptop running 64-bit Gentoo Linux with KDE (quad-core Intel Core i7 720QM @ 933 MHz & 6 MB cache, 4 GB DDR3 RAM). Both machines are on my home network, connected to the Internet via high-bandwidth broadband (FTTC).

On my laptop, the latest available versions of Firefox (27.0) and Opera (12.16_p1860-r1) for Gentoo, both 64-bit, do not even complete loading the ’24 Hours of Happy’ site: the black progress bar at the bottom of the home page stops about 7/8th of the way across the page and the KDE Network Monitor widget shows there is no network activity. Clearing Firefox’s Web content cache or increasing the cache’s size to 1 GB make no difference. Konqueror 4.12.2 (configured to use the WebKit browser engine rather than the KHTML engine) loads the site and plays it quite well at 720p to start with, but eventually video becomes choppy and I notice a lot of spawned kio_http processes. The KDE Network Monitor widget shows a continuous 3600 Kib/s data stream, which does not stop when I exit Konqueror. Numerous kio_http processes are spawned and remain after I exit Konqueror, and the 3600 KiB/s activity on the network only ceases when I kill all the kio_http processes. The number of spawned kio_http processes increases if I drag the pointer around the clock to select different takes, and the page just displays ‘LOADING’ ad infinitum if I do this several times. To be fair, if I do this a lot in Firefox running in Windows 8.1, I can get Firefox to stall too. I thought I’d try a lightweight browser and installed NetSurf (3.0-r1), but that couldn’t even render the title on the home page, let alone begin to load the video.

So, does ’24 Hours of Happy’ play nicely in your Linux installation? If it does, what hardware, distribution, desktop environment, browser and quality (360p, 480p, 720p or 1080p) are you using?

Fixing a problem with received video in Skype when using the AMD Catalyst (FGLRX) driver in Linux

Some users of Skype for Linux have reported that the bottom half of the received video image is corrupted in installations that use the closed-source video driver for ATI GPUs (the AMD Catalyst proprietary Linux driver, also known as the ‘FGLRX’ driver). One user described the lower half of the video image as “covered in small coloured squares like a chequer board”.

From what I have read in a few forums, it seems the problem does not occur when the open-source Radeon driver is used. My own experience corroborates that: I use the Radeon driver on one of my laptops, and received video in Skype is fine.

My main laptop has an AMD ATI Mobility Radeon HD 5650 GPU and I am using the Catalyst driver under Gentoo Linux. In this case there was a problem with received video in most Skype sessions. Either of the following effects usually occurred:

Snapshot 1 - Extract of received video image in Skype, showing an example of the corrupted image

Snapshot 1 - Extract of received video image in Skype, showing an example of the corrupted image

Snapshot 2 - Extract of received video image in Skype, showing another example of the corrupted image

Snapshot 2 - Extract of received video image in Skype, showing another example of the corrupted image

As shown in Snapshot 1, the lower half of the received video image was covered in a grid of thin green lines with areas tinged with purple, blue or green, whereas there was no grid of lines in the upper half of the image but some areas were tinged with red or blue.

As shown in Snapshot 2, the lower half of the received video image was covered in a grid of thin red lines, with a purple tinge in some areas, whereas there was no grid of lines in the upper half of the image, which looked reasonable but had some red-, green- or blue-tinged areas.

In all cases Skype’s thumbnail of my Webcam’s video image looked fine, and the person on the other end of the call said the video image received from me looked fine too.

Because of a bug in a previous version of the Catalyst driver a few years ago — see my blog posts Playing QuickTime videos in Firefox and Chromium + XVideo bug in AMD Catalyst 11.11 and 11.12 driver and AMD Catalyst for Linux driver 12.2 fixes the XVideo bug that crashed X.Org Server 1.11.x — I happen to know that Sykpe uses X11 overlays with the XVideo extension (xv), rather than using the OpenGL renderer (gl) or X11 with the SHM extension (x11). This made me wonder whether the use of XVideo with the Catalyst driver was causing the current problem. Unlike media players such as MPlayer and VLC, it is not possible to configure Skype to use gl or x11 instead of xv, so I thought it would not be possible to test whether the use of gl or x11 instead of xv would make a difference. Until, that is, I came upon a ‘trick’ posted by openSUSE user queequeg in 2009 during the period when an earlier version of the Catalyst driver had the aforementioned bug:

Skype Video Workaround for ATI

Anybody trying to make a video call with Skype and ATI fglrx drivers has had problems due to Skype using the “xv” video mode with the driver can’t handle. For anyone interested that is affected by this, there is a workaround:

1. Run the xvinfo command and look at the number of xv sessions available. Some cards have only 1, some have as many as 4. This is the number of xv occurances that the card can do at one time.
2. “Use up” all these xv sessions by opening videos in your favorite video player making sure to use xv for the video output. The videos can then be paused.
3. Once this (or they) are open, skype can be started and will default to X11 video and work properly with video calls.

I know this is a goofy way to get around this issue, but until fglrx can handle xv or skype allows an option to choose X11 for video render, I don’t know of any other way to do it.

(From what I hear, the 11.1 fglrx drivers can handle xv, but I haven’t confirmed this.)

So I tried his work-around. I had to launch four media players in order to use all available XVideo sessions. Lo and behold, when I launched Skype and made a video call the received video image was perfect. So it appeared that the Catalyst driver is not able to handle well the XVideo output from Skype. However, playing and pausing four videos every time I want to make a video call in Skype would hardly be practical, would it? And that is not the only downside: when I maximised a Firefox window during the Skype video call, my laptop spontaneously rebooted (I assume the X.Org server crashed).

I did also wonder whether just disabling compositing would solve the problem, so I disabled KWin Desktop Effects, but that didn’t make any difference.

I had also read in several forums that enabling or disabling the TexturedVideo and/or VideoOverlay options in the xorg.conf file have an effect on the video image produced by the Catalyst driver, but I could not find a post mentioning the use of either of those options to fix the specific problem I was seeing. So I decided not to pursue the xorg.conf route.

In my searches of the Web I came across a post somewhere that mentioned using GTK+ UVC Viewer (guvcview) to adjust video properties and improve video in Skype. I thought guvcview was only for adjusting the video image from a Webcam connected to my machine, i.e. adjusting the outgoing video image, and would not have any effect on received video. Nevertheless, I decided to install and launch guvcview to see if I could adjust both incoming and outgoing video properties. To my surprise, guvcview appeared to have fixed the problem with the received video. These are the steps I followed:

  1. I launched Skype and started a video call. The received video image had a grid of thin red lines and purple/green/blue tinting (similar to Snapshot 2).
  2. I Installed guvcview using the package manager.
  3. I launched guvcview in a Konsole (terminal) window. After guvcview created the file /home/fitzcarraldo/.config/guvcview/video0 and checked various video and audio settings it exited because my Webcam was being used by Skype (‘libv4l2: error setting pixformat: Device or resource busy‘).
  4. I clicked on the Webcam icon in the Skype call window, to turn my Webcam off.
  5. I launched guvcview again. The lower half of the received video image in Skype changed from a grid of thin red lines to a continuous green-coloured band, and the upper half of the image now looked reasonable but still had some red- or blue-tinged areas (see Snapshot 3 below).
  6. Snapshot 3 - Extract of received video image in Skype after I launched guvcview again

    Snapshot 3 - Extract of received video image in Skype after I launched guvcview again

  7. On the ‘Image Controls’ tab in the ‘GUVCViewer Controls’ window I changed the video frequency from 60 Hz to 50 Hz then back to 60 Hz again. I was just tinkering, and I believe this had no bearing on the outcome.
  8. I clicked on the ‘Quit’ button in the guvcview window to terminate the application.
  9. I clicked on the Webcam icon in the Skype call window to turn on again the Webcam, and the received Skype video image changed to a perfect image (see Snapshot 4 below).
  10. Snapshot 4 - Extract of received video image in Skype after I turned on again my Webcam in Skype

    Snapshot 4 - Extract of received video image in Skype after I turned on again my Webcam in Skype

It appears that guvcview had an effect on the received video image in Skype, although, if it did, I do not understand how. To check if the fix was permanent I ended the Skype video call, signed out of Skype and quit the application, rebooted and made a new Skype video call. The received video image in Skype was again perfect. I even deleted the guvcview configuration file and repeated this check, just in case the configuration file was somehow being used even though I had not launched guvcview, but the received video in yet another Skype video call was still perfect. I also clicked on the Webcam icon in the Skype call window several times during each call in order to turn my Webcam off and on several times; the received video image of the other person remained perfect.

So there you have it: when using an AMD ATI GPU and the Catalyst driver, it seems that guvcview can be used — at least in my case — to eliminate the type of image corruption in received Skype video shown in Snapshots 1 and 2. So, if you are also using the AMD Catalyst for Linux driver and are experiencing a similar problem, try guvcview. It might just do the trick.

‘Server not found’ by browser at launch

I haven’t had any significant Linux problems or new requirements in the last few months, hence no new posts here. My last real problem was back in June 2013 when I rolled my Gentoo installation to latest using Portage and found that, whenever I launched Firefox, it displayed the ‘Server not found’ page and I had to click ‘Try Again’, and then Firefox displayed the expected Web site. From then onwards, Firefox would work as expected until I exited the application. Thunderbird was also unable to access e-mail servers on the first attempt after it was launched. The same thing happened in Sabayon Linux when I rolled to latest using Entropy a couple of days later. Anyway, here is how I fixed the problem in both distributions.

First I used Wireshark to see what was going on, and it transpired that Gentoo (and Sabayon) was sending an IPv4 request followed quickly by an IPv6 request, but the reply to the IPv6 request was being received first and was a ‘server not found’ message since my ISP does not support IPv6 and my router apparently does not handle IPv6 requests correctly. Gentoo (and Sabayon) then used an IPv4 address when I clicked ‘Try Again’ in the browser window, and thereafter Firefox always dispayed the expected Web sites.

I should point out that IPv6 is enabled in the kernels I use and I’ve never before had to disable IPv6 in Firefox (or system-wide) on the affected laptops. So why the change in functionality, I wonder?

With Wireshark capturing packets, when I launched Firefox I was seeing a server failure message indicating “AAAA” (IPv6) instead of “A” (IPv4). To stop this happening I could have chosen any one of the three following solutions:

1. I could have used about:config in Firefox (and Config Editor in Thunderbird) to change the value of network.dns.disableIPv6 to true instead of false.

2. I could have disabled IPv6 system-wide by editing /etc/modprobe.d/aliases.conf and uncommenting the line “alias net-pf-10 off“.

3. I could have forced the getaddrinfo() function in glibc to make the IPv4 and IPv6 requests sequentially rather than in parallel.

Just for the fun of it I chose the third option on a couple of my laptops, and, as they use NetworkManager, this is how I did it:

fitzcarraldo@aspire5536 ~ $ su
Password:
aspire5536 fitzcarraldo # cat /etc/resolv.conf
# Generated by resolvconf
domain home
nameserver 192.168.1.254
aspire5536 fitzcarraldo # cd /etc/NetworkManager/dispatcher.d/
aspire5536 dispatcher.d # touch 06-dhclientoptions
aspire5536 dispatcher.d # nano 06-dhclientoptions
aspire5536 dispatcher.d # cat 06-dhclientoptions
#!/bin/bash
echo "options single-request" >> /etc/resolv.conf
aspire5536 dispatcher.d # chmod +x /etc/NetworkManager/dispatcher.d/06-dhclientoptions
aspire5536 dispatcher.d # # Now I disconnect then reconnect to the network
aspire5536 dispatcher.d # cat /etc/resolv.conf
# Generated by resolvconf
domain home
nameserver 192.168.1.254
options single-request
aspire5536 dispatcher.d #

As you can see above, I added a two-line Bash script 06-dhclientoptions in the directory /etc/NetworkManager/dispatcher.d/ that appends the line “options single-request” (without the quotes) to the contents of the file /etc/resolv.conf. The addition of the line “options single-request” in resolve.conf causes the getaddrinfo() function in glibc to make the IPv4 and IPv6 requests sequentially rather than in parallel. With this change, Firefox and Thunderbird no longer have a problem accessing the Internet the first time they are launched.

From “man 5 resolv.conf” under “options”:

single-request (since glibc 2.10)
sets RES_SNGLKUP in _res.options. By default, glibc performs IPv4 and IPv6 lookups in parallel since version 2.9. Some appliance DNS servers cannot handle these queries properly and make the requests time out. This option disables the behavior and makes glibc perform the IPv6 and IPv4 requests sequentially (at the cost of some slowdown of the resolving process).

single-request-reopen (since glibc 2.9)
The resolver uses the same socket for the A and AAAA requests. Some hardware mistakenly sends back only one reply. When that happens the client system will sit and wait for the second reply. Turning this option on changes this behavior so that if two requests from the same port are not handled correctly it will close the socket and open a new one before sending the second request.

I had to use NetworkManagerDispatcher to add the line “options single-request” to /etc/resolv.conf because NetworkManager overwrites /etc/resolv.conf if you edit it manually.

UPDATE (February 4, 2014): As I have recently seen the line “options single-request” occurring more than once in the file /etc/resolv.conf I now recommend /etc/NetworkManager/dispatcher.d/06-dhclientoptions consists of the following:

#!/bin/bash
if grep -q "options single-request" /etc/resolv.conf; then
    exit
else
    echo "options single-request" >> /etc/resolv.conf
fi

Synchronise your Gentoo Linux clock with an Internet time server

There are a number of ways to synchronise Gentoo Linux with a time server on the Internet. Here I look at a few alternatives.

ntp-client

ntp-client and the NTP daemon ntpd are installed when you install the package net-misc/ntp. Although I have read on some Web sites that /etc/init.d/ntp-client should be added to the default runlevel in order to read the time from an NTP server (once-only, during start-up), this in fact does not work because usually the network connection is not up by the time the ntp-client initscript runs. Bear in mind that ntp-client does not run continuously; it syncs once with an external time server if there is a network connection, and that’s it.

NetworkManager Dispatcher

If you are using NetworkManager, an elegant solution is to use NetworkManagerDispatcher to restart ntp-client in order to resync your system clock every time a network connection comes up. This is my favoured solution for laptops; see further on for how to configure your machine to do this.

cronjob

Another way would be to create a cronjob to run periodically the ‘/etc/init.d/ntp-client restart‘ command or the ‘ntpd -q‘ command (the -q option means “set the time and quit”).

Wait a while after start up

A ‘quick-and-dirty’ method, which I have used sometimes to synchronise a laptop’s system clock every time it boots, would be to delay running ntp-client until the network is up by putting e.g. the command below in a file 10_ntp-client.start in the directory /etc/local.d/ (10 seconds is usually enough time for a wired or wireless connection to my home network to be established):

#!/bin/bash
sleep 10s && /etc/init.d/ntp-client restart

Don’t forget to make it executable:

# chmod 744 /etc/local.d/10_ntp-client.start

NTP daemon

Regarding the NTP daemon, it is possible to configure this from the command line, rather than via a Desktop Environment GUI, to run at start-up and continue running to adjust your system clock. The command:

# rc-update add ntpd default

will add the daemon’s initscript to the default runlevel so that it is launched automatically at the next startup, and the command:

# /etc/init.d/ntpd start

will start the daemon running right now.

Note that, by default, the NTP daemon won’t correct, all in one go, a time difference between your system clock and the remote NTP server if that difference is above a certain size. However, if you want to override the default behaviour, i.e. allow the NTP daemon to make a large first adjustment to the system clock, you can set the environment variable NTPD_OPTS in the file /etc/conf.d/ntpd as follows:

NTPD_OPTS="-g"
# The -g option enables ntpd to make large adjustments.

This would mean that you would not need to run ntp-client before ntpd. However, if you run ntp-client automatically — either once after start-up or periodically — then that would be good enough for the typical Desktop user, and could be an alternative to having a continuously-running NTP daemon. Nothing stops you doing both if you want, of course.

Updating the hardware clock

If you make clock_systohc="YES" in the file /etc/conf.d/hwclock then the time in the system clock will be written to the BIOS (CMOS) clock (a.k.a. hardware clock) when you shut down your PC.

How to configure NetworkManager Dispatcher to synchronise the system clock only when a network connection is made

If you’re using a machine that is permanently connected to a network, running the NTP daemon makes sense. But what if you have a machine that is not always connected to a network when it is powered up? I have a laptop and I don’t want the NTP daemon running all the time. But I would like my laptop to synchronise with an external time server once after start up when I connect to the Internet. NetworkManager has a handy tool called NetworkManager Dispatcher for doing just this.

If you have installed NetworkManager, you’ll find there is an initscript /usr/portage/net-misc/networkmanager/files/NetworkManagerDispatcher. Copy it to the directory /etc/init.d/ and give it the necessary restrictive permissions:

# cp /usr/portage/net-misc/networkmanager/files/NetworkManagerDispatcher /etc/init.d/
# chmod 744 NetworkManagerDispatcher

Then create a shell script called e.g. 99_ntp-client in the directory /etc/NetworkManager/dispatcher.d/ to be run by NetworkManagerDispatcher when a network connection is established, containing the following code:

#!/bin/bash

INTERFACE=$1 # The interface which is brought up or down
STATUS=$2 # The new state of the interface

case "$STATUS" in
    'up') # $INTERFACE is up
        echo "System time before starting ntp-client:" > /home/fitzcarraldo/ntp-client.txt
        date >> /home/fitzcarraldo/ntp-client.txt
        echo "Starting ntp-client:" >> /home/fitzcarraldo/ntp-client.txt
        rc-config restart ntp-client &>> /home/fitzcarraldo/ntp-client.txt
        echo "System time after starting ntp-client:" >> /home/fitzcarraldo/ntp-client.txt
        date >> /home/fitzcarraldo/ntp-client.txt
        ;;
    'down') # $INTERFACE is down
        # Check for active interface and down if no one active
        if [ ! `nm-tool|grep State|cut -f2 -d' '` = "connected" ]; then
                echo "Stopping ntp-client at:" > /home/fitzcarraldo/ntp-client.txt
                date >> /home/fitzcarraldo/ntp-client.txt
                rc-config stop ntp-client &>> /home/fitzcarraldo/ntp-client.txt
        fi
        ;;
esac

Make the root user the owner of the script, and only allow the root user to write to it and execute it:

# cd /etc/NetworkManager/dispatcher.d/
# chown root:root 99_ntp-client
# chmod 744 99_ntp-client

Then add NetworkManagerDispatcher to the default runlevel so that it will be launched every time you boot your machine:

# rc-update add NetworkManagerDispatcher default

As the package net-misc/ntp installs both /etc/init.d/ntpd and /etc/init.d/net-client, users could optionally add the NTP daemon ntpd to the default runlevel too if desired, which would provide continuous, incremental adjustments to the system clock once net-client has done its one-shot adjustment each time a network comes up:

# rc-update add ntpd default

But users who don’t leave their PCs on for days on end — or who use laptops — can ignore the above step and just stick with the NetworkManagerDispatcher and net-client solution, whereas users who leave their machines on for days or weeks on end can also use the NTP daemon to keep the system clock in sync in between the times when ntp-client has synchronised.

Don’t forget to delete ntp-client from the start-up level if you are using NetworkManagerDispatcher to run it:

# rc-update del ntp-client

Notice that the script /etc/NetworkManager/dispatcher.d/99_ntp-client logs some information in a text file ntp-client.txt in my home directory which I can check. Here is an example of what ntp-client.txt contains after I select a network (or it is selected automatically) following start up of my laptop:

System time before starting ntp-client:
Sun Jun 3 19:24:08 BST 2012
Starting ntp-client:
Restarting init script
* Setting clock via the NTP client 'ntpd' ...ntpd: time slew +0.067178s
[ ok ]
System time after starting ntp-client:
Sun Jun 3 19:24:17 BST 2012

As you can see above, the ntpd command was executed once by NetworkManagerDispatcher and made a small adjustment to the system time on my laptop.

Replacing ntpdate with ntpd in ntp-client

Just for the fun of it, I changed /etc/conf.d/ntp-client to use the command ntpd instead of ntpdate, even though the ntpdate command works fine. Anyway, here’s my /etc/conf.d/ntp-client file these days:

NTPCLIENT_CMD="ntpd"
NTPCLIENT_OPTS="-g -q"

I have added the -g option so that the ntpd command can make large adjustments to the system time if it is way off the actual time. This is useful at the beginning and end of Daylight Saving Time, or if you dual boot with Windows. Here is an example of the former when I powered up my laptop the morning after the clocks changed from BST to GMT at the end of Summer 2010:

$ cat /home/fitzcarraldo/ntp-client.txt
System time before starting ntp-client:
Sun Oct 31 09:37:23 GMT 2010
Starting ntp-client:
Starting init script
* Setting clock via the NTP client 'ntpd'...ntpd: time set -3600.122381s
[ ok ]
System time after starting ntp-client:
Sun Oct 31 08:37:30 GMT 2010

You can specify the NTP server or NTP server pool in the file /etc/ntp.conf, but the default server pool already specified in that file should work. Note again that, when ntpd is run with the -q option, it synchronises the system clock once and terminates, i.e. it is not running as a daemon.

How to install and use Tor for anonymous browsing or to access country-restricted content from another country

Some people want to browse the Web in complete anonymity. One tool for doing that is Tor. However, there are other reasons for using Tor. For example, when I am travelling in a country where the government blocks certain Web sites (or blocks accessing content on certain Web sites), or when I am travelling overseas and certain Web sites back home will not let me view content (e.g. TV shows), I use Tor. Note that some Web sites are now clever enough to detect that you are accessing them via a proxy and could be overseas, so even Tor will not gain you access to media on some Web sites back home. Anyway, it’s still worth trying Tor to see if it works in your case.

For an overview of the installation and configuration procedure, see Running the Tor client on Linux/BSD/Unix. Below I will explain how to install and use Tor in Gentoo Linux.

Before you use Tor, it is useful to check your current IP address. Several Web sites will tell you your current IP address; here is the site I usually use: http://whatismyipaddress.com/

1. Install Tor:

# USE="tordns" emerge -1v tor

(Actually, the more-recent versions of the tor package don’t require that USE flag but I’ve left it in as it does no harm.)

2. Install Vidalia:

# cd /usr/portage/distfiles/
# wget --no-check-certificate https://www.torproject.org/dist/vidalia/vidalia-x.y.z.tar.gz
# USE="tor" emerge -1v vidalia

Use the current version x.y.z of the vidalia package in the package manager and https://www.torproject.org/dist/vidalia/

3. Install Polipo:

# emerge -1v polipo

4. Download polipo.conf

# cd /etc/polipo
# wget --no-check-certificate https://gitweb.torproject.org/torbrowser.git/blob_plain/HEAD:/build-scripts/config/polipo.conf

Edit April 21, 2013: The above URL is now:

https://gitweb.torproject.org/torbrowser.git/blob_plain/ae4aa49ad9100a50eec049d0a419fac63a84d874:/build-scripts/config/polipo.conf

5. Edit it and change proxyPort = 8118 to proxyPort = 8123

6. Copy it to /etc/polipo/:

# cd /etc/polipo
# cp /etc/polipo/config /etc/polipo/config.bak
# cp polipo.conf config

7. Configure Firefox:

Edit > Preferences > Network > Settings

Manual proxy configuration:

HTTP Proxy: 127.0.0.1 Port: 8123
SSL Proxy: 127.0.0.1 Port 8123

SOCKS Host: 127.0.0.1 Port 9051
SOCKS v5
No Proxy for: 127.0.0.1

8. Run Vidalia and then configure it:

$ vidalia &

a) Settings > Sharing

Select ‘Run as a client only’

b) Settings > Advanced

Select ‘Use TCP connection (ControlPort)’
Address: 127.0.0.1 9051

Tor Configuration File:
/home/fitzcarraldo/.vidalia/torrc

Data Directory:
/home/fitzcarraldo/.tor

c) Click on ‘Edit current torrc’ and make it:

# This file was generated by Tor; if you edit it, comments will not be preserved
# The old torrc file was renamed to torrc.orig.1 or similar, and Tor will ignore it
ControlPort 9051
ExitNodes {gb}
Log notice stdout
SocksListenAddress 127.0.0.1
StrictNodes 1

Note that I have specified “{gb}” above so that I am perceived by Web sites to be browsing in the UK even if I am in another country. But you can use a different country code if you want Web sites to perceive you are in another country. For example, “{us}” would make it look as if you are browsing in the USA.

9. Run Polipo:

$ sudo polipo

10. Surf to http://torcheck.xenobite.eu/ to check that you are now using a Tor exit node.

11. Surf to http://whatismyipaddress.com/ to check that your IP address has changed.

Why can’t I access a specific Web site?

Is there one specific Web site that you can never access? That always causes your Web browser to stall when you click on a link to that site or enter the site’s URL in the browser’s address bar? Perhaps it’s an Internet banking site? You don’t have trouble with other sites, just this one particular site or perhaps just a couple of sites?

The chances are that this problem is caused by what is called a ‘black hole’ router somewhere between you and that site. For the technical explanation of a black hole router, see e.g. the article Black Hole (networking).

I experienced this precise problem a couple of days ago with the popular Web site IMDb. The old Gateway Solo 9300 laptop I was using has no built-in networking hardware so I use a couple of CardBus cards for wireless and wired network access. The Linksys WPC54G (EU) v7.1 wireless notebook adapter works fine but, irrespective of the Web browser, I could not access the IMDb Web site; the browser always stalled when trying to load the site. I discovered I also had the same problem with my bank’s Web site.

I could ping other sites by IP address and by domain name, but pinging the IMDb site never received a reply (and it still doesn’t, even after the fix explained below which allows browsing of the site).

I also experienced this problem several years ago under Windows XP when trying to access the Amazon UK Web site, using an MTU that I had incorrectly set to a value smaller than possible. The problem this time was due to a correctly-set MTU. In both cases, the two MTUs are smaller than a router somewhere en route to those sites wanted to handle.

BACKGROUND

Bear with me while I explain how the problem arose, as it’s relevant (and useful) information.

I had just installed Sabayon Linux on the laptop and got the wireless network connection up, and the first thing I then did was to launch Firefox. With an MTU of 1500 for wlan0, Firefox would only load Google (and that, only intermittently). All other sites would stall. So I used the following command iteratively in order to find the MTU for this hardware, which is 1464 (the maximum packet size in bytes that does not fragment + 28 bytes):

# ping -M do -s packet_size_in_bytes http://www.cisco.com

e.g.

# ping -M do -s 1472 http://www.cisco.com

I just varied the packet size in the ping command until “Frag needed and DF set” was not displayed.

I can reduce the MTU to 1464 as follows:

# ifconfig wlan0 mtu 1464

I could make this permanent by adding an entry (mtu_wlan0=”1464″) to the file /etc/conf.d/net, but I chose instead to make it permanent by adding the above ifconfig command to the file /etc/conf.d/local. This (correct) MTU works fine for all sites I have visited so far except for two, which just stall the browser: IMDb and my bank’s Web site.

I also have a Belkin F5D5010 CardBus Network Card (wired Ethernet) for the laptop. It works fine with an MTU of 1500, and there is no problem accessing the IMDb Web site in Firefox. So, apparently, the MTU is the problem when using the wireless network connection. Now let’s look at the solution…

THE SOLUTION

Well, in the case of Windows it is possible to edit the Registry to solve the problem of network ‘black holes’. But what do you do in the case of Linux? It turns out that the kernel /proc file system provides an easy way to enable and disable TCP MTU Probing by changing a value in the ‘file’ /proc/sys/net/ipv4/tcp_mtu_probing. A value of 0 = disabled; 1 = enabled when a black hole router is detected; 2 = always enabled.

This is what my laptop had in that ‘file’:

# cat /proc/sys/net/ipv4/tcp_mtu_probing
0

So all I needed to do was:

# echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

and, bingo, a browser can access IMDb and my bank’s Web site without a problem. To make it permanent, I just added the above command to the file /etc/conf.d/local so that it is executed every time I boot the laptop. Here is what my file /etc/conf.d/local looks like now:

# Here is where you can put anything you need to start
# that there is not an init script for.

local_start() {
# This is a good place to load any misc programs
# on startup (use &>/dev/null to hide output)

# START OF MY ADDITIONS
# The Linksys CardBus card does not work with a MTU greater than 1464:
ifconfig wlan0 mtu 1464
# Enable TCP MTU Probing in order to deal with black hole routers:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
# END OF MY ADDITIONS

# We should always return 0
return 0
}

local_stop() {
# This is a good place to unload any misc.
# programs you started above.

# We should always return 0
return 0
}

UPDATE (May 13, 2011): OpenRC 0.8.0 and later in Gentoo and Sabayon Linux no longer use the file /etc/conf.d/local but instead require the commands to be in files in the directory /etc/local.d/. So I created a file /etc/local.d/01network.start containing the following lines:

# The Linksys CardBus card does not work with MTU greater than 1464:
ifconfig wlan0 mtu 1464
# Enable TCP MTU probing to deal with black hole routers:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

and made it executable:

# chmod +x /etc/local.d/01network.start

UPDATE (April 11, 2012): If “echo 2″ does not solve the problem, try “echo 1″ instead. The possible values are:

0 Do not perform PLPMTUD (Packetization Layer Path MTU Discovery)
1 Perform PLPMTUD only after detecting a ‘blackhole’.
2 Always perform PLPMTUD.

Follow

Get every new post delivered to your Inbox.

Join 43 other followers