Why does Thunderbird add ‘\A0’ and other strange-looking strings in e-mails I send?

I use Linux and have used the Thunderbird e-mail client since 2008. I used to use DavMail to enable Thunderbird to access various company Microsoft Exchange WebMail accounts but, several years ago, DavMail would no longer work with a particular Microsoft Exchange account so I switched to the Thunderbird add-on ExQuilla, for which I pay an annual licence fee. I do not know if the more recent versions of DavMail would work with this particular account but ExQuilla got me out of a hole so I stuck with it. Recently this particular corporation decided to stop using an in-house Microsoft Exchange server and switched to Microsoft 365.

Recently people receiving my e-mails sent using this particular account told me there were strange strings of characters in e-mails of mine that quote other e-mails. The most frequent occurrence is the three-character string ‘\A0’, although other strings are sometimes present too. The following e-mail extract illustrates the effect:

Hi Claudia,

I have had a look at your draft and agree with your assessment. Let’s sit down together and prepare a list of possible remedial measures.

Regards,
Fitzcarraldo

On 07/10/2020 13:02, Claudia wrote:
> Hi,
> \A0
> Could you please have a look at the draft I have attached.
> \A0
> There are several main issues requiring attention. The operation was basically run by one person \2013 (John) during the tests, which led to several issues.
> \A0
> He does not have the time to do everything by himself.\A0 The other staff who had assisted him during earlier tests were not present.

Notice various occurrences of ‘\A0’ and an occurrence of ‘\2013’.

I searched the Web to see if other Thunderbird users had come across this problem, and found several reports of similar problems, although not identical. The most promising page I found was in the Mozilla support forums for Thunderbird: Why do my sent messages magically add “�” at the end of my sentences?. However, none of the various fixes suggested in that thread worked in my case. My Thunderbird installation was configured to use ‘Unicode (UTF-8)’ text encoding for Outgoing Mail and Western (ISO-8859-1) for Incoming Mail ( ‘Edit’ > ‘Preferences’ > ‘General’ > ‘Language & Appearance’ > ‘Advanced…’ > ‘Text Encoding’). I changed the text encoding for incoming mail to ‘Unicode (UTF-8)’ but that made no difference. I ticked ‘When possible, use the default text encoding in replies’ but that also made no difference. Anyway, I left the settings like that and hoped an update to Thunderbird would fix the issue.

I was not sure if the problem started with an upgrade to Thunderbird, or whether the switch to Microsoft 365 was the cause. I suspect Microsoft 365 is the culprit because the problem does not occur when I use other e-mail accounts. Anyway, it is annoying and I have still not found a fix for it. One of the replies in the above-mentioned Thunderbird support thread is not identical to what I’m seeing, but it looks to be essentially the same problem:

Jorg K
2/4/18, 6:17 AM

There is NO bug in Thunderbird. Sadly some US ISP’s like AT&T and Bellsouth have started *corrupting* their customers’ e-mail.

If the customer sends in windows-1252 and includes for example special punctuation characters or a non-break space xA0, the ISP doesn’t correctly interpret the the message as windows-1252 but as UTF-8. In UTF-8, xA0 is not valid and gets replaced by the so-called replacement character, � (0xEF 0xBF 0xBD).

Since the e-mail is still windows-1252 encoded, the recipient’s client displays �.

See:
https://bugzilla.mozilla.org/show_bug.cgi?id=1427636
https://bugzilla.mozilla.org/show_bug.cgi?id=1435536

Affected users should complain heavily to their mail providers. As a workaround, they need to send all messages as UTF-8.

This seems to be a possible explanation of what I am experiencing, but it is impractical for me to check what text encoding all my contacts are using, or get them to switch to UTF-8 if they are not already using it in their e-mails. I noticed that there is actually a space in what look like blank lines in the e-mails I quote, and, if I delete that space, the ‘\A0’ no longer appears on those lines when I view the contents of e-mails in the Sent Items mailbox. I think that the space is, in fact, a non-breaking space (xA0), which is apparently invalid in UTF-8 and gets displayed as ‘\A0’ by Thunderbird (I’m currently using Version 78.4.2).

Trying to find and delete all the non-breaking spaces and other non-UTF-8 characters in a quoted e-mail is impractical. However, I found a somewhat cumbersome work-around to the problem of non-breaking spaces (and, I think, other non UTF-8 characters). When I click on the ‘Reply’ button in Thunderbird and a window pops up for me to compose my reply which includes a quoted e-mail or e-mails, I use Ctrl-C to copy all the contents of the window, then Ctrl-V to paste it back into the window. This seems to get rid of the character strings representing non-UTF-8 characters. It does add some extra blank lines in the quoted e-mail(s) in the window in which I am composing my e-mail, but those extra blank lines are normally not present when viewing the e-mail after it has been sent.

This work-around is not ideal as it relies on me remembering to do it when composing an e-mail in which I am quoting a previous e-mail or e-mails. But at least it gets rid of the multiple additional occurrences of ‘\A0’ (non-breaking space). It’s a pity there is no mechanism in Thunderbird to filter out non-UTF-8 characters such as a non-breaking space when quoting other e-mails. Even if Jorg K in the above-mentioned thread is correct and the cause of the problem does not lie in Thunderbird, I would rather Thunderbird act differently if the user has configured it to send e-mails using UTF-8 text encoding, and filter out non-UTF-8 characters rather than including strings of gobbledygook in the e-mail.

Advertisement

Trouble again with PulseAudio and Thunderbird sound notifications

In an earlier post I described how I fixed a scratchy-sounding sound file which the Thunderbird e-mail client plays when a new message arrives. Well, the problem started again recently, but this time the contents of /etc/pulse/daemon.conf looked OK to me. Furthermore, the sound file sounds fine when played using following commands:

aplay ~/Music/wav/E-mail_notifications/halmsg.wav
paplay ~/Music/wav/E-mail_notifications/halmsg.wav
mplayer ~/Music/wav/E-mail_notifications/halmsg.wav
cvlc ~/Music/wav/E-mail_notifications/halmsg.wav

Now, Thunderbird uses libcanberra to play sounds, so I began to wonder if the problem lay with libcanberra. As it happens, libcanberra is maintained by the same person who invented PulseAudio. However, I notice from the libcanberra Git repository that its source code has not been changed since 2012.

My Gentoo Linux installation had libcanberra installed with support for both ALSA and PulseAudio:

root # eix -I libcanberra
[I] media-libs/libcanberra
     Available versions:  0.30-r5 {alsa gnome gstreamer +gtk +gtk3 oss pulseaudio +sound tdb udev ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  0.30-r5(08:27:41 18/05/18)(alsa gtk gtk3 pulseaudio sound udev -gnome -gstreamer -oss -tdb ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://git.0pointer.net/libcanberra.git/
     Description:         Portable sound event library

So, even though my installation uses PulseAudio, I decided to try and re-install libcanberra without PulseAudio support, only ALSA support:

root # USE="-pulseaudio" emerge -1v libcanberra
root # eix -I libcanberra
[I] media-libs/libcanberra
     Available versions:  0.30-r5 {alsa gnome gstreamer +gtk +gtk3 oss pulseaudio +sound tdb udev ABI_MIPS="n32 n64 o32" ABI_PPC="32 64" ABI_S390="32 64" ABI_X86="32 64 x32"}
     Installed versions:  0.30-r5(15:47:14 26/05/18)(alsa gtk gtk3 sound udev -gnome -gstreamer -oss -pulseaudio -tdb ABI_MIPS="-n32 -n64 -o32" ABI_PPC="-32 -64" ABI_S390="-32 -64" ABI_X86="32 64 -x32")
     Homepage:            http://git.0pointer.net/libcanberra.git/
     Description:         Portable sound event library

Lo and behold, Thunderbird (libcanberra) plays the sound file correctly now. So I have added the following line to my file /etc/portage/package.use/thunderbird in order to make the change permanent:

media-libs/libcanberra -pulseaudio

PulseAudio 🙄

Audio in Linux becomes annoying again (continued)

Well, it turned out that the problem playing some system sounds in Thunderbird, described in my previous post, was due to PulseAudio.

Despite sounding fine when played by SMPlayer, the audio clips that sounded distorted/scratchy and too loud when played by Thunderbird also sounded that way when played by VLC. Then I discovered several other .wav files on various Web sites that sounded distorted when played by the browser’s Windows Media Player plug-in (Gecko Media Player). So the problem clearly was not caused by Thunderbird itself. I began to wonder if PulseAudio was the cause. So I adjusted PulseAudio’s sampling frequency, number of fragments and fragment size, and all the clips that previously sounded distorted and too loud now play fine. Here is what I did to fix the problem…

1. Check what PulseAudio is configured to use:

$ pulseaudio --dump-conf
### Read from configuration file: /etc/pulse/daemon.conf ###
daemonize = no
fail = yes
high-priority = yes
nice-level = -11
realtime-scheduling = yes
realtime-priority = 5
allow-module-loading = yes
allow-exit = yes
use-pid-file = yes
system-instance = no
local-server-type = user
cpu-limit = no
enable-shm = yes
flat-volumes = no
lock-memory = no
exit-idle-time = 20
scache-idle-time = 20
dl-search-path = /usr/lib64/pulse-5.0/modules
default-script-file = /etc/pulse/default.pa
load-default-script-file = yes
log-target =
log-level = notice
resample-method = auto
enable-remixing = yes
enable-lfe-remixing = no
default-sample-format = s16le
default-sample-rate = 44100
alternate-sample-rate = 48000
default-sample-channels = 2
default-channel-map = front-left,front-right
default-fragments = 4
default-fragment-size-msec = 25
enable-deferred-volume = yes
deferred-volume-safety-margin-usec = 8000
deferred-volume-extra-delay-usec = 0
shm-size-bytes = 0
log-meta = no
log-time = no
log-backtrace = 0
rlimit-fsize = -1
rlimit-data = -1
rlimit-stack = -1
rlimit-core = -1
rlimit-rss = -1
rlimit-as = -1
rlimit-nproc = -1
rlimit-nofile = 256
rlimit-memlock = -1
rlimit-locks = -1
rlimit-sigpending = -1
rlimit-msgqueue = -1
rlimit-nice = 31
rlimit-rtprio = 9
rlimit-rttime = 1000000
$

2. Check PulseAudio’s output sample rate:

$ pacmd list-sinks | grep sample
sample spec: s16le 2ch 44100Hz
$

So the sample rate is 16 bits @ 44100 Hz and there are 2 output channels (Stereo). My laptop does indeed have two built-in stereo speakers. ‘s16le‘ means ‘signed 16-bit little-endian’.

3. Check PulseAudio’s input sample rate:

$ pacmd list-sources | grep sample
sample spec: s16le 2ch 44100Hz
sample spec: s16le 2ch 44100Hz
$

So the sample rate is 16 bits @ 44100 Hz and there are 2 input channels (Stereo). My laptop does indeed have two built-in microphones. ‘s16le‘ means ‘signed 16-bit little-endian’.

4. Find out the audio card’s maximum sample rate (Hz):

$ arecord -f dat -r 60000 -D hw:0,0 -d 5 test.wav
Recording WAVE 'test.wav' : Signed 16 bit Little Endian, Rate 60000 Hz, Stereo
Warning: rate is not accurate (requested = 60000Hz, got = 48000Hz)
please, try the plug plugin
$

5. As the console output shows that my audio card supports a sample rate of 48000 Hz, edit /etc/pulse/daemon.conf and change the sample rate accordingly:

$ su
Password:
# grep sample-rate /etc/pulse/daemon.conf
; default-sample-rate = 44100
; alternate-sample-rate = 48000
# nano /etc/pulse/daemon.conf
# grep sample-rate /etc/pulse/daemon.conf
default-sample-rate = 48000
; alternate-sample-rate = 48000
# exit
exit
$

6. Find the buffer size and fragment size for each sound card:

$ echo autospawn = no >> ~/.config/pulse/client.conf
$ pulseaudio --kill
$ LANG=C timeout --foreground -k 10 -s kill 10 pulseaudio -vvvv 2>&1 | grep device.buffering -B 10
I: [pulseaudio] sink.c: alsa.driver_name = "snd_hda_intel"
I: [pulseaudio] sink.c: device.bus_path = "pci-0000:00:1b.0"
I: [pulseaudio] sink.c: sysfs.path = "/devices/pci0000:00/0000:00:1b.0/sound/card0"
I: [pulseaudio] sink.c: device.bus = "pci"
I: [pulseaudio] sink.c: device.vendor.id = "8086"
I: [pulseaudio] sink.c: device.vendor.name = "Intel Corporation"
I: [pulseaudio] sink.c: device.product.id = "3b56"
I: [pulseaudio] sink.c: device.product.name = "5 Series/3400 Series Chipset High Definition Audio"
I: [pulseaudio] sink.c: device.form_factor = "internal"
I: [pulseaudio] sink.c: device.string = "front:0"
I: [pulseaudio] sink.c: device.buffering.buffer_size = "19200"
I: [pulseaudio] sink.c: device.buffering.fragment_size = "4800"
--
I: [pulseaudio] source.c: alsa.driver_name = "snd_hda_intel"
I: [pulseaudio] source.c: device.bus_path = "pci-0000:00:1b.0"
I: [pulseaudio] source.c: sysfs.path = "/devices/pci0000:00/0000:00:1b.0/sound/card0"
I: [pulseaudio] source.c: device.bus = "pci"
I: [pulseaudio] source.c: device.vendor.id = "8086"
I: [pulseaudio] source.c: device.vendor.name = "Intel Corporation"
I: [pulseaudio] source.c: device.product.id = "3b56"
I: [pulseaudio] source.c: device.product.name = "5 Series/3400 Series Chipset High Definition Audio"
I: [pulseaudio] source.c: device.form_factor = "internal"
I: [pulseaudio] source.c: device.string = "front:0"
I: [pulseaudio] source.c: device.buffering.buffer_size = "19200"
I: [pulseaudio] source.c: device.buffering.fragment_size = "4800"
$ sed -i '$d' ~/.config/pulse/client.conf # You can just delete the file instead if it didn't exist in the first place.
$

N.B. Depending on your distribution, the PulseAudio file client.conf (if it exists) may be in a different sub-directory of the user’s home directory.

The console output shows that the buffer size was 19200 bits and the fragment size was 4800 bits.

7. Calculate the number of fragments and the fragment size (msec):

default-fragments = 19200 / 4800 = 4

default-fragments-size-msec

= device.buffering.fragment_size [bits] / (sample rate [Hz] x sample width [bits] x number of channels)

= 4800 / ( 48000 x 16 x 2 )

= 0.003125 seconds = 3.125 msec = 3 msec to the nearest integer

8. Edit /etc/pulse/daemon.conf to set these two parameters to the above-mentioned values:

# grep default-fragment /etc/pulse/daemon.conf
; default-fragments = 4
; default-fragment-size-msec = 25
# nano /etc/pulse/daemon.conf
# grep default-fragment /etc/pulse/daemon.conf
default-fragments = 4
default-fragment-size-msec = 3
# exit
exit
$

9. Restart PulseAudio:

$ pulseaudio --kill
$ pulseaudio --start # Only needed if you have 'autospawn = no' in ~/.config/pulse/client.conf
$

10. Check the PulseAudio configuration:

$ pulseaudio --dump-conf
### Read from configuration file: /etc/pulse/daemon.conf ###
daemonize = no
fail = yes
high-priority = yes
nice-level = -11
realtime-scheduling = yes
realtime-priority = 5
allow-module-loading = yes
allow-exit = yes
use-pid-file = yes
system-instance = no
local-server-type = user
cpu-limit = no
enable-shm = yes
flat-volumes = no
lock-memory = no
exit-idle-time = 20
scache-idle-time = 20
dl-search-path = /usr/lib64/pulse-5.0/modules
default-script-file = /etc/pulse/default.pa
load-default-script-file = yes
log-target =
log-level = notice
resample-method = auto
enable-remixing = yes
enable-lfe-remixing = no
default-sample-format = s16le
default-sample-rate = 48000
alternate-sample-rate = 48000
default-sample-channels = 2
default-channel-map = front-left,front-right
default-fragments = 4
default-fragment-size-msec = 3
enable-deferred-volume = yes
deferred-volume-safety-margin-usec = 8000
deferred-volume-extra-delay-usec = 0
shm-size-bytes = 0
log-meta = no
log-time = no
log-backtrace = 0
rlimit-fsize = -1
rlimit-data = -1
rlimit-stack = -1
rlimit-core = -1
rlimit-rss = -1
rlimit-as = -1
rlimit-nproc = -1
rlimit-nofile = 256
rlimit-memlock = -1
rlimit-locks = -1
rlimit-sigpending = -1
rlimit-msgqueue = -1
rlimit-nice = 31
rlimit-rtprio = 9
rlimit-rttime = 1000000
$

Notice that PulseAudio is now configured to use new values for default-sample-rate, default-fragments and default-fragment-size-msec.

$ echo autospawn = no >> ~/.config/pulse/client.conf
$ pulseaudio --kill
$ LANG=C timeout --foreground -k 10 -s kill 10 pulseaudio -vvvv 2>&1 | grep device.buffering -B 10
I: [pulseaudio] sink.c: alsa.driver_name = "snd_hda_intel"
I: [pulseaudio] sink.c: device.bus_path = "pci-0000:00:1b.0"
I: [pulseaudio] sink.c: sysfs.path = "/devices/pci0000:00/0000:00:1b.0/sound/card0"
I: [pulseaudio] sink.c: device.bus = "pci"
I: [pulseaudio] sink.c: device.vendor.id = "8086"
I: [pulseaudio] sink.c: device.vendor.name = "Intel Corporation"
I: [pulseaudio] sink.c: device.product.id = "3b56"
I: [pulseaudio] sink.c: device.product.name = "5 Series/3400 Series Chipset High Definition Audio"
I: [pulseaudio] sink.c: device.form_factor = "internal"
I: [pulseaudio] sink.c: device.string = "front:0"
I: [pulseaudio] sink.c: device.buffering.buffer_size = "2304"
I: [pulseaudio] sink.c: device.buffering.fragment_size = "576"
--
I: [pulseaudio] source.c: alsa.driver_name = "snd_hda_intel"
I: [pulseaudio] source.c: device.bus_path = "pci-0000:00:1b.0"
I: [pulseaudio] source.c: sysfs.path = "/devices/pci0000:00/0000:00:1b.0/sound/card0"
I: [pulseaudio] source.c: device.bus = "pci"
I: [pulseaudio] source.c: device.vendor.id = "8086"
I: [pulseaudio] source.c: device.vendor.name = "Intel Corporation"
I: [pulseaudio] source.c: device.product.id = "3b56"
I: [pulseaudio] source.c: device.product.name = "5 Series/3400 Series Chipset High Definition Audio"
I: [pulseaudio] source.c: device.form_factor = "internal"
I: [pulseaudio] source.c: device.string = "front:0"
I: [pulseaudio] source.c: device.buffering.buffer_size = "2304"
I: [pulseaudio] source.c: device.buffering.fragment_size = "576"
$ rm ~/.config/pulse/client.conf # I didn't have a client.conf to begin with, so I just deleted it.
$ pulseaudio --start
$

Notice that PulseAudio is now using new values for device.buffering.buffer_size and device.buffering.fragment_size.

11. Check PulseAudio’s output sample rate:

$ pacmd list-sinks | grep sample
sample spec: s16le 2ch 48000Hz
$

Notice that PulseAudio is now using a new output sample rate.

12. Check PulseAudio’s input sample rate:

$ pacmd list-sources | grep sample
sample spec: s16le 2ch 48000Hz
sample spec: s16le 2ch 48000Hz
$

Notice that PulseAudio is now using a new input sample rate.

Audio in Linux becomes annoying again

At the moment I seem to be having more audio problems than usual. Last month I blogged about having to fix the ALSA Speaker volume level resetting to zero at boot, and recently two other audio problems have cropped up.

Thunderbird

I have been having trouble with Thunderbird’s ‘system sound’ that announces the arrival of a new e-mail. Lately, Thunderbird has started playing too loud and with significant distortion the audio clip it had been playing perfectly for the last four years. This is especially strange because I created that audio clip with Audacity from another audio clip that sounded too loud when Thunderbird played it. Ironically, the work-around for this latest problem was to switch to the original, much louder sound clip alert.wav instead of the quieter alert_quiet.wav. Not only does Thunderbird now play alert.wav at a lower volume than alert_quiet.wav, but the sound of alert.wav is not distorted when Thunderbird plays it. Yet if I play alert.wav and alert_quiet.wav using SMPlayer, the former is much louder than the latter and neither is distorted. Figure that one out.

The event notification sound that Thunderbird uses to remind me about an impending meeting scheduled in the calendar has now started sounding very distorted too. I still have not found a work-around for that. Event sounds played by the desktop environment I use (KDE) are not distorted, so what is Thunderbird doing? Perhaps the problem is not Thunderbird itself but the audio library it uses, so I need to investigate further.

Skype

Yet another audio problem cropped up this morning when I connected my laptop to an external monitor and keyboard (and thus I left the laptop’s lid almost closed) in an open-plan office and booted the laptop. I entered my username and password on the KDM log-in screen, and the KDE splash screen appeared as usual. After a few seconds the laptop’s speakers suddenly emitted a piercing, continuous howl; the well-known sound of audio feedback from speakers to microphone. It was LOUD. The volume control buttons on the keyboard made no difference, and the sound was so loud that everyone in the office noticed and several people came over to tell me to reset the BIOS (apparently that had fixed the problem for their laptops running Windows).

I kept my finger on the laptop’s power switch and, after several seconds, the laptop powered off. My laptop dual boots Windows 7 and Gentoo Linux, and the audio feedback did not occur when I booted Windows 7. After booting Linux again a couple of times and annoying everyone in the office even more, I discovered I could open the laptop’s lid far enough back to reduce the feedback to a low whine, so I could let KDE finish launching and display the Desktop. I then launched ALSAMixer and discovered that ‘Internal Mic Boost’ was set to 100%. So I immediately lowered it to zero. Then the penny dropped: I had used Skype the previous night without bothering to connect my headphones and external microphone, and Skype had automatically raised ‘Internal Mic Boost’ all the way up to 100%. So I immediately launched Skype, selected ‘Options’ > ‘Sound Devices’ and unticked ‘Allow Skype to automatically adjust my mixer levels’. The next thing I did was add the following lines to the script /etc/local.d/20set_alsa_volume.start mentioned in my previous blog post Fix for ALSA Speaker volume level resetting to zero at boot:

su -c "amixer -c 0 -- sset 'Internal Mic Boost' 0%" -s /bin/sh fitzcarraldo
su -c "amixer -c 0 -- sset 'Internal Mic' 0%" -s /bin/sh fitzcarraldo
su -c "amixer -c 0 -- sset 'Mic Boost' 0%" -s /bin/sh fitzcarraldo
su -c "amixer -c 0 -- sset 'Mic' 0%" -s /bin/sh fitzcarraldo

From now on, only I am allowed to adjust microphone settings! To avoid any possibility of feedback loops in future, the above-mentioned script sets all the microphone channels to zero and I will have to adjust them myself before use. I already have ALSAMixerGUI in the KDE Launcher menu, so it won’t be a big deal to do that.

This fiasco with Skype got me thinking: if Skype is set to automatically adjust mixer levels when you are in a conversation, when you exit Skype why doesn’t it automatically set mixer levels back to the way they were when Skype was launched? It could be done easily and would be more user-friendly than the current way Skype works.

Interrelationship between PulseAudio and ALSA

The final thing I did (yet again) was to adjust all the various ALSA channels and PulseAudio channels to try and get the resulting audio input and output sounding reasonable. This is easier said than done. I often have to mess around with ALSAMixer and PulseAudio Volume Control in order to get audio input and output working satisfactorily in all applications that use audio. It is actually more difficult than it sounds (ouch!) to get ALSA and PulseAudio ‘balanced’ (for want of a better word). In the days before PulseAudio existed, one only had to adjust ALSA. Now, with two agents controlling audio, the task turns out to be surprisingly awkward sometimes.

To sum up, boo to Thunderbird (or whatever it uses to play sounds), boo to Skype and boo to PulseAudio. I’m fed up with audio issues in Linux at the moment. 😡

Update (January 19, 2015): It turns out that the problem in Thunderbird was due to PulseAudio. See my next post for details of how I fixed it.