ARTE Live Web videos

The Franco-German cultural Web site ARTE Live Web is an excellent resource for lovers of music (classical, jazz, alternative and World) and dance. The show videos at the site are enjoyable but unfortunately only viewable for a fixed period of time before the site removes them. Back in 2011 I wanted to download the video of a performance I’d attended and loved. I searched for a Linux tool but could not find one, then found a Windows freeware GUI tool called artepupper, and used version 0.2 to download the video. You can read about the tool on the author’s blog pages artepupper 0.1, artepupper 0.2 and artepupper 0.3. The reason for the change from 0.2 to 0.3 was that ARTE Live Web changed the format of an embedded URL inside the XML page for the video on their Web site.

Now, artepupper is based on livewebarte1.1.sh, a 2010 Bash script by Carmelo Ingrao, and a Perl script livewebarte.pl by Juan Domingo based on livewebarte1.1.sh, both of which are still available on line at Carmelo’s download page and which I only discovered after using artepupper in Windows. I tried the Perl script in Linux back in 2011, and it worked for me.

Recently I wanted to download another video from the ARTE Live Web site and found that, although artepupper 0.3 in Windows worked, the Perl script livewebarte.pl no longer worked. Presumably this was because of the aforementioned change to the embedded URL. However, I found that the Bash script livewebarte1.1.sh still worked. Actually, I had to edit the script to replace “./rtmpdump” with “rtmpdump“, as I had installed the command-line tool rtmpdump using the Linux distribution’s package manager and the executable is not stored in the same directory as the Bash script. But, apart from that, it worked.

Carmelo is a star for having deciphered how to access and download videos from the site. However, looking at his Bash script, the part where he parses a line in the XML code and extracts a string between the delimiters “MP4″ and “mp4″ is based on a hard-coded character position, which could change if the Web site’s owners change the format of the URL. So I decided to modify the Bash script to avoid using character positions to extract a string. The Bash script livewebarte1.2.sh is an updated version of Carmelo’s livewebarte1.1.sh script.

#!/bin/bash
# Script pour récupérer les vidéos FLV du site liveweb.arte.tv
# par Carmelo Ingrao <carmelo42@gmail.com> http://c.ingrao.free.fr/code/
# version 1.0
# release date 21 février 2010
## modified by Fitzcarraldo 24 April 2013
## version 1.2
## release date 24 April 2013
# licence : GPLv2
# rtmpdump compilé doit être dans le même répertoire que le script
## rtmpdump must be installed and in user's $PATH (e.g. I have it in /usr/bin/)
# utilisation du script :
#
# ./script.sh url fichier.flv
# _______________

# url --> $1
# fichier --> $2

# fichier de sortie
# on efface l'écran avant de commencer
clear

# on affiche les infos sur le script
echo "livewebarte.sh version 1.0."
echo "(c) 2010 Carmelo Ingrao; License : GPL"
echo "livewebarte.sh version 1.2."
echo "updated from version 1.1 by Fitzcarraldo on 24 April 2013; License : GPL"
echo "usage : ./livewebarte.sh url_concert_sur_liveweb.arte.tv fichier.flv"
echo "rtmpdump must be installed and in your path."

# on télécharge le code source de la page streamant le concert dans le fichier sourceconcert.html
wget $1 -O sourceconcert.html

# on récupère le numéro d'event et on le copie dans eventok.txt
#cat sourceconcert.html | grep "new LwEvent" > event.txt
#cat event.txt | cut -b "15 16 17" > eventok.txt
grep "new LwEvent" sourceconcert.html | grep -E -o -e "[0-9]+" > eventok.txt

# on prend le fichier XML d'arte et on crée l'url avec le bon numéro d'event
# xmloriginal="http://arte.vo.llnwd.net/o21/liveweb/events/event-610.xml"

# url du xml sans le numéro d'event original (pour faciliter)
xmloriginal2="http://arte.vo.llnwd.net/o21/liveweb/events/event-"

# on assigne à la variable b le contenu de eventok.txt --> numéro correct d'event
b=$(cat eventok.txt)

# on créer l'url correct du fichier XML qu'on téléchargera
xmlok=$(echo $xmloriginal2$b)
finxml=".xml"
xmlfinal=$(echo $xmlok$finxml)

# on télécharge le bon XML
wget $xmlfinal -O xmlok.xml
echo "Fichier XML téléchargé"

## I have changed the code in this part:
# on extrait le nom du fichier MP4 depuis le fichier xmlok.xml
mp4hd=$(cat xmlok.xml | grep "urlHd")
# on efface le début de l'url du MP4
# on efface le surplus à la fin du nom du MP4 et on sauve le nom dans la variable mp4hdcut2
mp4hdcut2=${mp4hd#*MP4}
mp4hdcut2=${mp4hdcut2%%mp4*}
mp4hdcut2="MP4"$mp4hdcut2"mp4"
## end of my changes to evaluate mp4hdcut2

# on lance la commande rtmpdump avec les paramètres
# rappel :
# $2 = nom du fichier de sortie
# $mp4hdcut2 = nom du fichier MP4

## Carmelo had ./rtmpdump here, but I removed the "./"
rtmpdump -r rtmp://arte.fcod.llnwd.net:1935/a2306/o25 -a a2306/o25 -f LNX 10,0,45,2 -W http://liveweb.arte.tv/flash/player.swf -t rtmp://arte.fcod.llnwd.net:1935/a2306/o25 -p http://liveweb.arte.tv/ -o $2 -y $mp4hdcut2

# on efface les fichiers crées
rm sourceconcert.html
#rm event.txt
rm eventok.txt
rm xmlok.xml

# Affichage des infos de fin
echo "________"
echo "Voilà, le téléchargement est terminé."
echo "Le fichier se trouve ici :"
echo " "
echo $2
echo " "
echo "Bon visionnage"
echo " "
echo " "
exit 0

Save it in your home directory and make it executable:

$ chmod +x livewebarte1.2.sh

Also make sure you have the package rtmpdump installed and that it is in your $PATH.

Then you can browse the ARTE Live Web site and select the performance video you wish to download. Hover the mouse pointer over the video pane and click on “INTEGRER LA VIDEO” to find the URL for that video, which will be of the form http://liveweb.arte.tv/fr/video/foo/, where “foo” is some string of characters (not literally “foo”, of course). The command to download it is then as shown below. I’ll use a file name foo.flv here, but any prefix would do:

$ ./livewebarte1.2.sh http://liveweb.arte.tv/fr/video/foo/ foo.flv

Note that it is essential to include the forward slash at the end of the URL. The file will be downloaded to your home directory and you can watch it in VLC or any other Linux media player that plays Flash video.

So there you have it; currently you can use artepupper 0.3 in Windows or livewebarte1.2.sh in Linux to download from ARTE Live Web a video of a performance you attended and loved.

Let’s hear it for Konqueror

My browser of choice on the desktop has been Firefox for many years. Firefox uses the Gecko rendering engine. As a backup Web browser I use Konqueror but configured to use WebKit, rather than KHTML, as the rendering engine. I’ve tried Chromium, Opera, Midori, rekonq, SeaMonkey and a bunch of others, but always found them lacking in some way in comparison to Firefox (I find Opera Mobile better than Firefox for Android on my mobile phone, though).

However, Firefox sometimes lets me down. For example, some months ago I wanted to book tickets online for a concert but Firefox would not display the seat map correctly, stopping me from being able to select seats. Konqueror saved the day. And, recently, Firefox no longer displays the video component of trailers on Rotten Tomatoes; only audio works. Firefox correctly plays videos from virtually all other sites I visit (YouTube, IMDb, iTunes Trailers, Vimeo, eTelegraph etc.) so why the sudden inability to display Rotten Tomatoes trailers? Today Firefox wouldn’t play a product video on an Amazon page either. So I launched Konqueror and it can play Rotten Tomatoes trailers and the Amazon video. What gives? They are both running on the same laptop in the same OS (Gentoo Linux) and desktop environment (KDE), using the same version of Flash, the same video driver etc. The only thing I can think of is that the Firefox rendering engine Gecko is the culprit. I assume WebKit in Konqueror is more capable than Gecko, although I don’t know enough to be certain that Gecko is the cause of the problem.

Anyway, if you want to configure Konqueror to use the WebKit rendering engine instead of the KHTML rendering engine, click on ‘Settings’ on the Konqueror menu bar, select ‘Configure Konqueror…’ and click on ‘General’. You’ll see ‘Default web browser engine’ in the right pane. Select WebKit and click ‘OK’. You’ll also need to have WebKit itself installed, of course. I have the packages qt-webkit (the WebKit module for the Qt toolkit) and kwebkitpart (a WebKit KPart for Konqueror) installed.

How to join together several Flash video (flv) files

YouTube users will be familiar with the 10-minute limit on videos uploaded to the site. People get around the restriction by splitting longer videos into several clips. Recently I wanted to watch a video that had been split into nine clips on YouTube, but I wanted to join them together to avoid interruptions. However, it turns out that, if video clips do not all have the same resolution, bit rate and frame rate, concatenating them is more complicated than I was expecting. Below I describe the things I tried and what worked for me.

I downloaded the clips by using the script youtube-dl, which is an easy way of downloading YouTube videos without having to watch them first. For example:

$ youtube-dl -o brodyworld_dreamer.flv http://www.youtube.com/watch?v=ue-0UUVzUbM

The nine clips have the following characteristics:

clip1.flv 35.7 MiB (480×360, 362 average kbps, 25-26 fps according to YouTube)
clip2.flv 43.2 MiB (480×360, 460 average kbps, 25-26 fps according to YouTube)
clip3.flv 46.3 MiB (480×360, 418 average kbps, 25-26 fps according to YouTube)
clip4.flv 27.8 MiB (320×240, 238 average kbps, 25-26 fps according to YouTube)
clip5.flv 20.0 MiB (320×240, 203 average kbps, 25-26 fps according to YouTube)
clip6.flv 45.5 MiB (480×360, 414 average kbps, 25-26 fps according to YouTube)
clip7.flv 43.0 MiB (480×360, 460 average kbps, 25-26 fps according to YouTube)
clip8.flv 45.4 MiB (480×360, 495 average kbps, 25-27 fps according to YouTube)
clip9.flv 28.0 MiB (480×360, 400 average kbps, 25-26 fps according to YouTube)

Google found several sites with instructions on how to concatenate .flv files. Below are a few of the suggested solutions, and my experience using them to try and concatenate these nine video clips.

GUI APPLICATIONS

Pitivi

This could import all the clips but, when I dragged them to the timeline pane at the bottom of the window, Pitivi would only recognise them as audio clips. Also, if I tried to play one of the clips in the right pane of the window, Pitivi would play only the audio track. I have found this happens with some .flv files that I download from YouTube.

So, I tried another application…

Avidemux

A pop-up window appeared when I tried to open the first clip:

H.264 detected.
If the file is using B-frames as reference it can lead to a crash or stuttering.
Avidemux can use another mode which is safe but YOU WILL LOSE FRAME ACCURACY.
Do you want to use that mode?

I clicked on ‘Use safe mode’ and Avidemux loaded the clip successfully and could play it. A different pop-up window appeared when I then tried to open the second clip:

Video dimensions don’t match. You cannot mix different video dimensions yet. Using partial video filter later will not work around this problem. The workaround is: 1) “Resize” /”Add Border” / “Crop” each stream to the same resolution; 2) Concatenate them together.

Too much hassle, so on to the next application…

Kino

This looked like it might work, as it converts each .flv clip to a DV file first:

“/home/fitzcarraldo/Downloads/clip1.flv” is not a DV file. Do you want to import it? Importing… (this might take a while).

It certainly does take a while, and each clip’s .dv file was 2 GiB. I gave up after the second clip. Too long and too manual for my liking.

Time to try the command line, then…

THE COMMAND LINE

Google finds several forum threads and blogs offering command line solutions for joining several flv files together (and for joining together several .avi files, or several .mpeg files, and so on). I tried several of the different command strings recommended, none of which worked with the nine clips I had downloaded. Some of the commands resulted in a concatenated file that would only play the first 10 minutes (i.e. the first clip), others only the first 20 minutes (i.e. the first two clips), and others resulted in a zero-length file.

My biggest hope were the three alternatives given in the FFmpeg FAQ section 3.15 How can I join video files?, but the error messages ‘Estimating duration from bitrate, this may be inaccurate’ and ‘VBV buffer size not set, muxing may fail’ during processing worried me. And they are hardly quick and easy commands to enter.

FFmpeg Method 1

$ ffmpeg -i clip1.flv -sameq intermediate1.mpg
$ ffmpeg -i clip2.flv -sameq intermediate2.mpg
$ ffmpeg -i clip3.flv -sameq intermediate3.mpg
$ ffmpeg -i clip4.flv -sameq intermediate4.mpg
$ ffmpeg -i clip5.flv -sameq intermediate5.mpg
$ ffmpeg -i clip6.flv -sameq intermediate6.mpg
$ ffmpeg -i clip7.flv -sameq intermediate7.mpg
$ ffmpeg -i clip8.flv -sameq intermediate8.mpg
$ ffmpeg -i clip9.flv -sameq intermediate9.mpg
$ cat intermediate1.mpg intermediate2.mpg intermediate3.mpg intermediate4.mpg intermediate5.mpg intermediate6.mpg intermediate7.mpg intermediate8.mpg intermediate9.mpg > intermediate_all.mpg
$ ffmpeg -i intermediate_all.mpg -sameq film.avi

The resulting file was 124 MiB and consisted of only the first twenty minutes, i.e. the first two clips. Both VLC and SMPlayer could play it.

FFmpeg Method 2

$ mkfifo intermediate1.mpg
$ mkfifo intermediate2.mpg
$ mkfifo intermediate3.mpg
$ mkfifo intermediate4.mpg
$ mkfifo intermediate5.mpg
$ mkfifo intermediate6.mpg
$ mkfifo intermediate7.mpg
$ mkfifo intermediate8.mpg
$ mkfifo intermediate9.mpg
$ ffmpeg -i clip1.flv -sameq -y intermediate1.mpg < /dev/null &
$ ffmpeg -i clip2.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip3.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip4.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip5.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip6.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip7.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip8.flv -sameq -y intermediate2.mpg < /dev/null &
$ ffmpeg -i clip9.flv -sameq -y intermediate2.mpg < /dev/null &
$ cat intermediate1.mpg intermediate2.mpg intermediate3.mpg intermediate4.mpg intermediate5.mpg intermediate6.mpg intermediate7.mpg intermediate8.mpg intermediate9.mpg |\
ffmpeg -f mpeg -i - -sameq -vcodec mpeg4 -acodec libmp3lame film.avi

The resulting file was 124 MiB and again only consisted of the first twenty minutes. SMPlayer could play it (although the film’s duration was shown as 00:00:00), but VLC could not.

FFmpeg Method 3

$ mkfifo temp1.a
$ mkfifo temp1.v
$ mkfifo temp2.a
$ mkfifo temp2.v
$ mkfifo temp3.a
$ mkfifo temp3.v
$ mkfifo temp4.a
$ mkfifo temp4.v
$ mkfifo temp5.a
$ mkfifo temp5.v
$ mkfifo temp6.a
$ mkfifo temp6.v
$ mkfifo temp7.a
$ mkfifo temp7.v
$ mkfifo temp8.a
$ mkfifo temp8.v
$ mkfifo temp9.a
$ mkfifo temp9.v
$ mkfifo all.a
$ mkfifo all.v
$ ffmpeg -i clip1.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp1.a < /dev/null &
$ ffmpeg -i clip2.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp2.a < /dev/null &
$ ffmpeg -i clip3.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp3.a < /dev/null &
$ ffmpeg -i clip4.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp4.a < /dev/null &
$ ffmpeg -i clip5.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp5.a < /dev/null &
$ ffmpeg -i clip6.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp6.a < /dev/null &
$ ffmpeg -i clip7.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp7.a < /dev/null &
$ ffmpeg -i clip8.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp8.a < /dev/null &
$ ffmpeg -i clip9.flv -vn -f u16le -acodec pcm_s16le -ac 2 -ar 44100 - > temp9.a < /dev/null &
$ ffmpeg -i clip1.flv -an -f yuv4mpegpipe - > temp1.v < /dev/null &
$ { ffmpeg -i clip2.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp2.v ; } &
$ { ffmpeg -i clip3.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp3.v ; } &
$ { ffmpeg -i clip4.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp4.v ; } &
$ { ffmpeg -i clip5.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp5.v ; } &
$ { ffmpeg -i clip6.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp6.v ; } &
$ { ffmpeg -i clip7.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp7.v ; } &
$ { ffmpeg -i clip8.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp8.v ; } &
$ { ffmpeg -i clip9.flv -an -f yuv4mpegpipe - < /dev/null | tail -n +2 > temp9.v ; } &
$ cat temp1.a temp2.a temp3.a temp4.a temp5.a temp6.a temp7.a temp8.a temp9.a > all.a &
$ cat temp1.v temp2.v temp3.v temp4.v temp5.v temp6.v temp7.v temp8.v temp9.v > all.v &
$ ffmpeg -f u16le -acodec pcm_s16le -ac 2 -ar 44100 -i all.a \
-f yuv4mpegpipe -i all.v \
-sameq -y film.flv
$ rm temp[123456789].[av] all.[av]

The resulting file was 175.4 MiB. Both VLC and SMPlayer showed the correct duration of 1:27:22 but could not play further than twenty minutes.

Mencoder Method 1

A Linux forum thread suggested the following mencoder command string would do the job:

$ mencoder -forceidx -of lavf -oac copy -ovc copy -o film.flv clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

This failed with the error message “Audio format 0x4134504d is incompatible with ‘-oac copy’, please try ‘-oac pcm’ instead or use ‘-fafmttag’ to override it.” so I changed it to:

$ mencoder -forceidx -of lavf -oac copy -ovc pcm -o film.flv clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

This produced the error message “All video files must have identical fps, resolution, and codec for -ovc copy.” and the resulting film.flv was 261.2 MiB. Both VLC and SMPlayer showed the film duration as 20 minutes, and played those 20 minutes.

Mencoder Method 2

Finally I found a command that worked successfully and without hassle:

$ mencoder -oac pcm -ovc xvid -vf scale -zoom -xy 480 -xvidencopts bitrate=460 -o film.avi clip1.flv clip2.flv clip3.flv clip4.flv clip5.flv clip6.flv clip7.flv clip8.flv clip9.flv

I chose a bitrate of 460 kbps, as it is near the top of the range of bitrates of the nine clips, and an X-axis resolution of 480 pixels (the Y-axis resolution is scaled automatically) even though some of the clips have a smaller resolution. The resulting file is a 1.1 GiB AVI file. The image is a bit less distinct than the original clips but SMPlayer and VLC show the film’s correct duration and play it to the end. There are slight hiccups at some of the joins, but these are hardly noticeable. SMPlayer lists the following information about the video:

General
 File: /home/fitzcarraldo/Downloads/film.avi
 Size: 1182950 KB (1155 MB)
 Length: 01:27:21
 Demuxer: avi

Clip info
 Software: MEncoder SVN-r30554-4.4.2

Video
 Resolution: 480 x 360
 Aspect ratio: 1.3333
 Format: XVID
 Bitrate: 430 kbps
 Frames per second: 25.000
 Selected codec: ffodivx

Initial Audio Stream
 Format: 1
 Bitrate: 1411 kbps
 Rate: 44100 Hz
 Channels: 2
 Selected codec: pcm

Audio Streams
 #: 0
 Language: <empty>
 Name: <empty>
 ID: 1

So there we have it. Not perfect, but at least something that actually works, and with little effort.

If you have successfully stitched together video files each having a different resolution and/or frame rate and/or bit rate and/or codec, do add a comment and let us know the command string or GUI application that you used, and whether or not the resulting video quality was good.

Follow

Get every new post delivered to your Inbox.

Join 52 other followers