Bypassing a corporate Web filter when using the command line

or ‘How to bypass a corporate Web filter and download YouTube videos via the command line’

One of the offices where I work uses a Web filter to block access to certain sites, such as YouTube. However, sometimes it is necessary to view blocked Web sites for work purposes. For example, these days a lot of companies or individuals post product reviews on YouTube that are useful for work purposes. In such cases I have used Tor to access the blocked sites in a Web browser such as Firefox, Chrome, Konqueror etc. See my post How to install and use Tor for anonymous browsing or to access country-restricted content from another country for details of how to set up and use Tor with a Web browser.

But sometimes I need to access blocked Web sites from the command line. For example, today I needed to download a YouTube video for work purposes, and I wanted to use youtube-dl to do it. The solution was simple…

First I launched vidalia and polipo as explained in the above-mentioned post on Tor, then I launched another Konsole/Terminal window and entered the commands shown below:

$ # First find out what resolutions are available for the video I want to download:
$ youtube-dl -F https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
[info] Available formats for T3Rr4CUoTSQ:
format code extension resolution note
140 m4a audio only DASH audio , audio@128k (worst)
160 mp4 192p DASH video
133 mp4 240p DASH video
134 mp4 360p DASH video
135 mp4 480p DASH video
136 mp4 720p DASH video
17 3gp 176x144
36 3gp 320x240
5 flv 400x240
43 webm 640x360
18 mp4 640x360
22 mp4 1280x720 (best)
$ # Now try to download the video at the resolution I want:
$ youtube-dl -f 22 -o Clevo_W230ST_overview.flv https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
ERROR: unable to download video data: HTTP Error 403: Forbidden

As you can see above, the corporate Web filter blocked youtube-dl from downloading the video.

So I informed the shell session about the local HTTP proxy (polipo) running on my laptop, by assigning and exporting the environment variable http_proxy using the following syntax:

export http_proxy=http://server-ip:port/

which in my case meant the following (refer to my article on Tor):

$ export http_proxy=http://127.0.0.1:8123/

and then I was able to download the video from YouTube despite the corporate Web filter:

$ youtube-dl -f 22 -o Clevo_W230ST_overview.flv https://www.youtube.com/watch?v=T3Rr4CUoTSQ
Setting language
T3Rr4CUoTSQ: Downloading webpage
T3Rr4CUoTSQ: Downloading video info webpage
T3Rr4CUoTSQ: Extracting video information
[download] Destination: Clevo_W230ST_overview.flv
[download] 100% of 100.23MiB in 05:50
$

Useful Reference: How To Use Proxy Server To Access Internet at Shell Prompt With http_proxy Variable

ARTE Live Web videos

The Franco-German cultural Web site ARTE Live Web is an excellent resource for lovers of music (classical, jazz, alternative and World) and dance. The show videos at the site are enjoyable but unfortunately only viewable for a fixed period of time before the site removes them. Back in 2011 I wanted to download the video of a performance I’d attended and loved. I searched for a Linux tool but could not find one, then found a Windows freeware GUI tool called artepupper, and used version 0.2 to download the video. You can read about the tool on the author’s blog pages artepupper 0.1, artepupper 0.2 and artepupper 0.3. The reason for the change from 0.2 to 0.3 was that ARTE Live Web changed the format of an embedded URL inside the XML page for the video on their Web site.

Now, artepupper is based on livewebarte1.1.sh, a 2010 Bash script by Carmelo Ingrao, and a Perl script livewebarte.pl by Juan Domingo based on livewebarte1.1.sh, both of which are still available on line at Carmelo’s download page and which I only discovered after using artepupper in Windows. I tried the Perl script in Linux back in 2011, and it worked for me.

Recently I wanted to download another video from the ARTE Live Web site and found that, although artepupper 0.3 in Windows worked, the Perl script livewebarte.pl no longer worked. Presumably this was because of the aforementioned change to the embedded URL. However, I found that the Bash script livewebarte1.1.sh still worked. Actually, I had to edit the script to replace “./rtmpdump” with “rtmpdump“, as I had installed the command-line tool rtmpdump using the Linux distribution’s package manager and the executable is not stored in the same directory as the Bash script. But, apart from that, it worked.

Carmelo is a star for having deciphered how to access and download videos from the site. However, looking at his Bash script, the part where he parses a line in the XML code and extracts a string between the delimiters “MP4″ and “mp4″ is based on a hard-coded character position, which could change if the Web site’s owners change the format of the URL. So I decided to modify the Bash script to avoid using character positions to extract a string. The Bash script livewebarte1.2.sh is an updated version of Carmelo’s livewebarte1.1.sh script.

#!/bin/bash
# Script pour récupérer les vidéos FLV du site liveweb.arte.tv
# par Carmelo Ingrao <carmelo42@gmail.com> http://c.ingrao.free.fr/code/
# version 1.0
# release date 21 février 2010
## modified by Fitzcarraldo 24 April 2013
## version 1.2
## release date 24 April 2013
# licence : GPLv2
# rtmpdump compilé doit être dans le même répertoire que le script
## rtmpdump must be installed and in user's $PATH (e.g. I have it in /usr/bin/)
# utilisation du script :
#
# ./script.sh url fichier.flv
# _______________

# url --> $1
# fichier --> $2

# fichier de sortie
# on efface l'écran avant de commencer
clear

# on affiche les infos sur le script
echo "livewebarte.sh version 1.0."
echo "(c) 2010 Carmelo Ingrao; License : GPL"
echo "livewebarte.sh version 1.2."
echo "updated from version 1.1 by Fitzcarraldo on 24 April 2013; License : GPL"
echo "usage : ./livewebarte.sh url_concert_sur_liveweb.arte.tv fichier.flv"
echo "rtmpdump must be installed and in your path."

# on télécharge le code source de la page streamant le concert dans le fichier sourceconcert.html
wget $1 -O sourceconcert.html

# on récupère le numéro d'event et on le copie dans eventok.txt
#cat sourceconcert.html | grep "new LwEvent" > event.txt
#cat event.txt | cut -b "15 16 17" > eventok.txt
grep "new LwEvent" sourceconcert.html | grep -E -o -e "[0-9]+" > eventok.txt

# on prend le fichier XML d'arte et on crée l'url avec le bon numéro d'event
# xmloriginal="http://arte.vo.llnwd.net/o21/liveweb/events/event-610.xml"

# url du xml sans le numéro d'event original (pour faciliter)
xmloriginal2="http://arte.vo.llnwd.net/o21/liveweb/events/event-"

# on assigne à la variable b le contenu de eventok.txt --> numéro correct d'event
b=$(cat eventok.txt)

# on créer l'url correct du fichier XML qu'on téléchargera
xmlok=$(echo $xmloriginal2$b)
finxml=".xml"
xmlfinal=$(echo $xmlok$finxml)

# on télécharge le bon XML
wget $xmlfinal -O xmlok.xml
echo "Fichier XML téléchargé"

## I have changed the code in this part:
# on extrait le nom du fichier MP4 depuis le fichier xmlok.xml
mp4hd=$(cat xmlok.xml | grep "urlHd")
# on efface le début de l'url du MP4
# on efface le surplus à la fin du nom du MP4 et on sauve le nom dans la variable mp4hdcut2
mp4hdcut2=${mp4hd#*MP4}
mp4hdcut2=${mp4hdcut2%%mp4*}
mp4hdcut2="MP4"$mp4hdcut2"mp4"
## end of my changes to evaluate mp4hdcut2

# on lance la commande rtmpdump avec les paramètres
# rappel :
# $2 = nom du fichier de sortie
# $mp4hdcut2 = nom du fichier MP4

## Carmelo had ./rtmpdump here, but I removed the "./"
rtmpdump -r rtmp://arte.fcod.llnwd.net:1935/a2306/o25 -a a2306/o25 -f LNX 10,0,45,2 -W http://liveweb.arte.tv/flash/player.swf -t rtmp://arte.fcod.llnwd.net:1935/a2306/o25 -p http://liveweb.arte.tv/ -o $2 -y $mp4hdcut2

# on efface les fichiers crées
rm sourceconcert.html
#rm event.txt
rm eventok.txt
rm xmlok.xml

# Affichage des infos de fin
echo "________"
echo "Voilà, le téléchargement est terminé."
echo "Le fichier se trouve ici :"
echo " "
echo $2
echo " "
echo "Bon visionnage"
echo " "
echo " "
exit 0

Save it in your home directory and make it executable:

$ chmod +x livewebarte1.2.sh

Also make sure you have the package rtmpdump installed and that it is in your $PATH.

Then you can browse the ARTE Live Web site and select the performance video you wish to download. Hover the mouse pointer over the video pane and click on “INTEGRER LA VIDEO” to find the URL for that video, which will be of the form http://liveweb.arte.tv/fr/video/foo/, where “foo” is some string of characters (not literally “foo”, of course). The command to download it is then as shown below. I’ll use a file name foo.flv here, but any prefix would do:

$ ./livewebarte1.2.sh http://liveweb.arte.tv/fr/video/foo/ foo.flv

Note that it is essential to include the forward slash at the end of the URL. The file will be downloaded to your home directory and you can watch it in VLC or any other Linux media player that plays Flash video.

So there you have it; currently you can use artepupper 0.3 in Windows or livewebarte1.2.sh in Linux to download from ARTE Live Web a video of a performance you attended and loved.

How to join together several Flash video (flv) files

YouTube users will be familiar with the 10-minute limit on videos uploaded to the site. People get around the restriction by splitting longer videos into several clips. Recently I wanted to watch a video that had been split into nine clips on YouTube, but I wanted to join them together to avoid interruptions. However, it turns out that, if video clips do not all have the same resolution, bit rate and frame rate, concatenating them is more complicated than I was expecting. Below I describe the things I tried and what worked for me.

I downloaded the clips by using the script youtube-dl, which is an easy way of downloading YouTube videos without having to watch them first. For example:

youtube-dl -o brodyworld_dreamer.flv http://www.youtube.com/watch?v=ue-0UUVzUbM

The nine clips have the following characteristics:

clip1.flv 35.7 MiB (480×360, 362 average kbps, 25-26 fps according to YouTube)
clip2.flv 43.2 MiB (480×360, 460 average kbps, 25-26 fps according to YouTube)
clip3.flv 46.3 MiB (480×360, 418 average kbps, 25-26 fps according to YouTube)
clip4.flv 27.8 MiB (320×240, 238 average kbps, 25-26 fps according to YouTube)
clip5.flv 20.0 MiB (320×240, 203 average kbps, 25-26 fps according to YouTube)
clip6.flv 45.5 MiB (480×360, 414 average kbps, 25-26 fps according to YouTube)
clip7.flv 43.0 MiB (480×360, 460 average kbps, 25-26 fps according to YouTube)
clip8.flv 45.4 MiB (480×360, 495 average kbps, 25-27 fps according to YouTube)
clip9.flv 28.0 MiB (480×360, 400 average kbps, 25-26 fps according to YouTube)

Google found several sites with instructions on how to concatenate flv files. Below are a few of the suggested solutions, and my experience using them to try and concatenate these nine video clips.

GUI APPLICATIONS

Pitivi

This could import all the clips but, when I dragged them to the timeline pane at the bottom of the window, Pitivi would only recognise them as audio clips. Also, if I tried to play one of the clips in the right pane of the window, Pitivi would play only the audio track. I have found this happens with some flv files that I download from YouTube.

So, I tried another application…

Avidemux

A pop-up window appeared when I tried to open the first clip:

“H.264 detected.
If the file is using B-frames as reference it can lead to a crash or stuttering.
Avidemux can use another mode which is safe but YOU WILL LOSE FRAME ACCURACY.
Do you want to use that mode?”

I clicked on ‘Use safe mode’ and Avidemux loaded the clip successfully and could play it. A different pop-up window appeared when I then tried to open the second clip:

“Video dimensions don’t match. You cannot mix different video dimensions yet. Using partial video filter later will not work around this problem. The workaround is: 1) “Resize” /”Add Border” / “Crop” each stream to the same resolution; 2) Concatenate them together.”

Too much hassle, so on to the next applicatiion…

Kino

This looked like it might work, as it converts each flv clip to a DV file first:

“/home/fitzcarraldo/Downloads/clip1.flv” is not a DV file. Do you want to import it? Importing… (this might take a while).”

It certainly does take a while, and each clip’s .dv file was 2 GiB. I gave up after the second clip. Too long and too manual for my liking.

Time to try the command line, then…

THE COMMAND LINE

Google finds several forum threads and blogs offering command line solutions for joining several flv files together (and for joining together several avi files, or several mpeg files, and so on). I tried several of the different command strings recommended, none of which worked with the nine clips I had downloaded. Some of the commands resulted in a concatenated file that would only play the first 10 minutes (i.e. the first clip), others only the first 20 minutes (i.e. the first two clips), and others resulted in a zero-length file.

My biggest hope were the three alternatives given in the FFmpeg FAQ section 3.15 How can I join video files?, but the error messages “Estimating duration from bitrate, this may be inaccurate” and “VBV buffer size not set, muxing may fail” during processing worried me. And they are hardly quick and easy commands to enter.

FFmpeg Method 1

ffmpeg -i clip1.flv -sameq intermediate1.mpg
ffmpeg -i clip2.flv -sameq intermediate2.mpg
ffmpeg -i clip3.flv -sameq intermediate3.mpg
ffmpeg -i clip4.flv -sameq intermediate4.mpg
ffmpeg -i clip5.flv -sameq intermediate5.mpg
ffmpeg -i clip6.flv -sameq intermediate6.mpg
ffmpeg -i clip7.flv -sameq intermediate7.mpg
ffmpeg -i clip8.flv -sameq intermediate8.mpg
ffmpeg -i clip9.flv -sameq intermediate9.mpg
cat intermediate1.mpg intermediate2.mpg intermediate3.mpg intermediate4.mpg intermediate5.mpg intermediate6.mpg intermediate7.mpg intermediate8.mpg intermediate9.mpg > intermediate_all.mpg
ffmpeg -i intermediate_all.mpg -sameq film.avi

The resulting file was 124 MiB and consisted of only the first twenty minutes, i.e. the first two clips. Both VLC and SMPlayer could play it.

FFmpeg Method 2

mkfifo intermediate1.mpg
mkfifo intermediate2.mpg
mkfifo intermediate3.mpg
mkfifo intermediate4.mpg
mkfifo intermediate5.mpg
mkfifo intermediate6.mpg
mkfifo intermediate7.mpg
mkfifo intermediate8.mpg
mkfifo intermediate9.mpg
ffmpeg -i clip1.flv -sameq -y intermediate1.mpg < /dev/null &
ffmpeg -i clip2.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip3.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip4.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip5.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip6.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip7.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip8.flv -sameq -y intermediate2.mpg < /dev/null &
ffmpeg -i clip9.flv -sameq -y intermediate2.mpg < /dev/null &
cat intermediate1.mpg intermediate2.mpg intermediate3.mpg intermediate4.mpg intermediate5.mpg intermediate6.mpg intermediate7.mpg intermediate8.mpg intermediate9.mpg |\
ffmpeg -f mpeg -i - -sameq -vcodec mpeg4 -acodec libmp3lame film.avi

The resulting file was 124 MiB and again only consisted of the first twenty minutes. SMPlayer could play it (although the film’s duration was shown as 00:00:00), but VLC could not.

FFmpeg Method 3

mkfifo temp1.a
mkfifo temp1.v
mkfifo temp2.a
mkfifo temp2.v
mkfifo temp3.a
mkfifo temp3.v
mkfifo temp4.a
mkfifo temp4.v
mkfifo temp5.a
mkfifo temp5.v
mkfifo temp6.a
mkfifo temp6.v
mkfifo temp7.a
mkfifo temp7.v
mkfifo temp8.a
mkfifo temp8.v
mkfifo temp9.a
mkfifo temp9.v
mkfifo all.a
mkfifo all.v
ffmpeg -i clip1.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp1.a < /dev/null &
ffmpeg -i clip2.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp2.a < /dev/null &
ffmpeg -i clip3.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp3.a < /dev/null &
ffmpeg -i clip4.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp4.a < /dev/null &
ffmpeg -i clip5.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp5.a < /dev/null &
ffmpeg -i clip6.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp6.a < /dev/null &
ffmpeg -i clip7.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp7.a < /dev/null &
ffmpeg -i clip8.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp8.a < /dev/null &
ffmpeg -i clip9.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp9.a < /dev/null &
ffmpeg -i clip1.flv -an -f yuv4mpegpipe - > temp1.v < /dev/null &
{ ffmpeg -i clip2.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp2.v ; } &
{ ffmpeg -i clip3.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp3.v ; } &
{ ffmpeg -i clip4.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp4.v ; } &
{ ffmpeg -i clip5.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp5.v ; } &
{ ffmpeg -i clip6.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp6.v ; } &
{ ffmpeg -i clip7.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp7.v ; } &
{ ffmpeg -i clip8.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp8.v ; } &
{ ffmpeg -i clip9.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp9.v ; } &
cat temp1.a temp2.a temp3.a temp4.a temp5.a temp6.a temp7.a temp8.a temp9.a > all.a &
cat temp1.v temp2.v temp3.v temp4.v temp5.v temp6.v temp7.v temp8.v temp9.v > all.v &
ffmpeg -f u16le -acodec pcm_s16le -ac 2 -ar 44100 -i all.a \
-f yuv4mpegpipe -i all.v \
-sameq -y film.flv
rm temp[123456789].[av] all.[av]

The resulting file was 175.4 MiB. Both VLC and SMPlayer showed the correct duration of 1:27:22 but could not play further than twenty minutes.

Mencoder Method 1

A Linux forum thread suggested the following mencoder command string would do the job:

mencoder -forceidx -of lavf -oac copy -ovc copy -o film.flv clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

This failed with the error message “Audio format 0x4134504d is incompatible with ‘-oac copy’, please try ‘-oac pcm’ instead or use ‘-fafmttag’ to override it.” so I changed it to:

mencoder -forceidx -of lavf -oac copy -ovc pcm -o film.flv clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

This produced the error message “All video files must have identical fps, resolution, and codec for -ovc copy.” and the resulting film.flv was 261.2 MiB. Both VLC and SMPlayer showed the film duration as 20 minutes, and played those 20 minutes.

Mencoder Method 2

Finally I found a command that worked successfully and without hassle:

mencoder -oac pcm -ovc xvid -vf scale -zoom -xy 480 -xvidencopts bitrate=460 -o film.avi clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

I chose a bitrate of 460 kbps, as it is near the top of the range of bitrates of the nine clips, and an X-axis resolution of 480 pixels (the Y-axis resolution is scaled automatically) even though some of the clips have a smaller resolution. The resulting file is a 1.1 GiB AVI file. The image is a bit less distinct than the original clips but SMPlayer and VLC show the film’s correct duration and play it to the end. There are slight hiccups at some of the joins, but these are hardly noticeable. SMPlayer lists the following information about the video:

General
File
/home/fitzcarraldo/Downloads/film.avi
Size
1182950 KB (1155 MB)
Length
01:27:21
Demuxer
avi

Clip info
Software
MEncoder SVN-r30554-4.4.2

Video
Resolution
480 x 360
Aspect ratio
1.3333
Format
XVID
Bitrate
430 kbps
Frames per second
25.000
Selected codec
ffodivx

Initial Audio Stream
Format
1
Bitrate
1411 kbps
Rate
44100 Hz
Channels
2
Selected codec
pcm

Audio Streams
#
Language
Name
ID
0

1

So there we have it. Not perfect, but at least something that actually works, and with little effort.

If you have successfully stitched together video files each having a different resolution and/or frame rate and/or bit rate and/or codec, do add a comment and let us know the command string or GUI application that you used, and whether or not the resulting video quality was good.

Follow

Get every new post delivered to your Inbox.

Join 49 other followers