Why can’t I access a specific Web site?

Is there one specific Web site that you can never access? That always causes your Web browser to stall when you click on a link to that site or enter the site’s URL in the browser’s address bar? Perhaps it’s an Internet banking site? You don’t have trouble with other sites, just this one particular site or perhaps just a couple of sites?

The chances are that this problem is caused by what is called a ‘black hole’ router somewhere between you and that site. For the technical explanation of a black hole router, see e.g. the article Black Hole (networking).

I experienced this precise problem a couple of days ago with the popular Web site IMDb. The old Gateway Solo 9300 laptop I was using has no built-in networking hardware so I use a couple of CardBus cards for wireless and wired network access. The Linksys WPC54G (EU) v7.1 wireless notebook adapter works fine but, irrespective of the Web browser, I could not access the IMDb Web site; the browser always stalled when trying to load the site. I discovered I also had the same problem with my bank’s Web site.

I could ping other sites by IP address and by domain name, but pinging the IMDb site never received a reply (and it still doesn’t, even after the fix explained below which allows browsing of the site).

I also experienced this problem several years ago under Windows XP when trying to access the Amazon UK Web site, using an MTU that I had incorrectly set to a value smaller than possible. The problem this time was due to a correctly-set MTU. In both cases, the two MTUs are smaller than a router somewhere en route to those sites wanted to handle.

BACKGROUND

Bear with me while I explain how the problem arose, as it’s relevant (and useful) information.

I had just installed Sabayon Linux on the laptop and got the wireless network connection up, and the first thing I then did was to launch Firefox. With an MTU of 1500 for wlan0, Firefox would only load Google (and that, only intermittently). All other sites would stall. So I used the following command iteratively in order to find the MTU for this hardware, which is 1464 (the maximum packet size in bytes that does not fragment + 28 bytes):

# ping -M do -s packet_size_in_bytes www.cisco.com

e.g.

# ping -M do -s 1472 www.cisco.com

I just varied the packet size in the ping command until “Frag needed and DF set” was not displayed.

I can reduce the MTU to 1464 as follows:

# ifconfig wlan0 mtu 1464

I could make this permanent by adding an entry (mtu_wlan0=”1464″) to the file /etc/conf.d/net, but I chose instead to make it permanent by adding the above ifconfig command to the file /etc/conf.d/local. This (correct) MTU works fine for all sites I have visited so far except for two, which just stall the browser: IMDb and my bank’s Web site.

I also have a Belkin F5D5010 CardBus Network Card (wired Ethernet) for the laptop. It works fine with an MTU of 1500, and there is no problem accessing the IMDb Web site in Firefox. So, apparently, the MTU is the problem when using the wireless network connection. Now let’s look at the solution…

THE SOLUTION

Well, in the case of Windows it is possible to edit the Registry to solve the problem of network ‘black holes’. But what do you do in the case of Linux? It turns out that the kernel /proc file system provides an easy way to enable and disable TCP MTU Probing by changing a value in the ‘file’ /proc/sys/net/ipv4/tcp_mtu_probing. A value of 0 = disabled; 1 = enabled when a black hole router is detected; 2 = always enabled.

This is what my laptop had in that ‘file’:

# cat /proc/sys/net/ipv4/tcp_mtu_probing
0

So all I needed to do was:

# echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

and, bingo, a browser can access IMDb and my bank’s Web site without a problem. To make it permanent, I just added the above command to the file /etc/conf.d/local so that it is executed every time I boot the laptop. Here is what my file /etc/conf.d/local looks like now:

# Here is where you can put anything you need to start
# that there is not an init script for.

local_start() {
# This is a good place to load any misc programs
# on startup (use &>/dev/null to hide output)

# START OF MY ADDITIONS
# The Linksys CardBus card does not work with a MTU greater than 1464:
ifconfig wlan0 mtu 1464
# Enable TCP MTU Probing in order to deal with black hole routers:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
# END OF MY ADDITIONS

# We should always return 0
return 0
}

local_stop() {
# This is a good place to unload any misc.
# programs you started above.

# We should always return 0
return 0
}

UPDATE (May 13, 2011): OpenRC 0.8.0 and later in Gentoo and Sabayon Linux no longer use the file /etc/conf.d/local but instead require the commands to be in files in the directory /etc/local.d/. So I created a file /etc/local.d/01network.start containing the following lines:

# The Linksys CardBus card does not work with MTU greater than 1464:
ifconfig wlan0 mtu 1464
# Enable TCP MTU probing to deal with black hole routers:
echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing

and made it executable:

# chmod +x /etc/local.d/01network.start

UPDATE (April 11, 2012): If “echo 2” does not solve the problem, try “echo 1” instead. The possible values are:

0   Do not perform PLPMTUD (Packetization Layer Path MTU Discovery)
1   Perform PLPMTUD only after detecting a ‘blackhole’.
2   Always perform PLPMTUD.

About Fitzcarraldo
A Linux user with an interest in all things technical.

19 Responses to Why can’t I access a specific Web site?

  1. Pingback: Cannot browse some Web sites: yet another cause « Fitzcarraldo's Blog

  2. harishsankar says:

    wonderfully explained,i use a ubuntu 14 system.the cat /etc/../mtu_probing returned o, but wen i did echo 2 … bash: /proc/sys/net/ipv4/tcp_mtu_probing: Permission denied.. please advice.

    • Fitzcarraldo says:

      Did you do it as root user? You need to issue the command as root user. I don’t use Ubuntu, but in that distribution you are forced to use the sudo command, so try the following:

      $ sudo su -
      # echo 2 > /proc/sys/net/ipv4/tcp_mtu_probing
      # exit
      $

      • harishsankar says:

        yes i did run the command as superuser.. but still i had the permission denied result, tats wat made me wonder..I use a tplink wireless n adsl2 modem router,the problem started when i changed my modem. both firefox and konqureor gives the server not found message. when i pinged through my router it was a success..

        here is the latest ping result ping -M do -s 1500 shappad.in
        ping: unknown host shappad.in

  3. harishsankar says:

    I use a tplink wireless n adsl2 modem router,the problem started when i changed my modem. both firefox and konqureor gives the server not found message. when i pinged through my router it was a success..

  4. harishsankar says:

    here is the latest ping result ping -M do -s 1500 shappad.in
    ping: unknown host shappad.in

    • Fitzcarraldo says:

      The host does not exist, so try another one:

      # ping -M do -s 1500 www.shappad.in
      ping: unknown host www.shappad.in

      # ping -M do -s 1500 www.google.in
      PING www.google.in (173.194.124.31) 1500(1528) bytes of data.
      ping: local error: Message too long, mtu=1500
      ping: local error: Message too long, mtu=1500
      ping: local error: Message too long, mtu=1500
      ping: local error: Message too long, mtu=1500
      ^C
      --- www.google.in ping statistics ---
      4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2999ms

      # ping -M do -s 1436 www.google.in
      PING www.google.in (173.194.124.24) 1436(1464) bytes of data.
      1444 bytes from 173.194.124.24: icmp_seq=1 ttl=55 time=34.2 ms
      1444 bytes from 173.194.124.24: icmp_seq=2 ttl=55 time=31.2 ms
      1444 bytes from 173.194.124.24: icmp_seq=3 ttl=55 time=30.0 ms
      1444 bytes from 173.194.124.24: icmp_seq=4 ttl=55 time=30.1 ms
      ^C
      --- www.google.in ping statistics ---
      4 packets transmitted, 4 received, 0% packet loss, time 2999ms
      rtt min/avg/max/mdev = 30.015/31.435/34.287/1.728 ms

      • harishsankar says:

        the sudo worked it echoed out 2..so shappad.in doesnt ping for u too?? could u please open it in a browser? after the echoing wat do i do??the files u mentioned in the tutorial /etc/conf.d.. doesnt exist for me.though the echo worked the site still doesnt load. any suggestions

      • harishsankar says:

        I would like to add a thing more, while i access the site from a mobile(my friends) it loads just fine.

        • Fitzcarraldo says:

          I cannot access the domain you gave from any machine, including my mobile. The message is “Server not found” on both laptop (Gentoo Linux) and mobile (Android). What is the IP address of the site? Also, what sort of site is it?

        • harishsankar says:

          188.40.169.249, I would like to add a thing more, while i access the site from a mobile(my friends) through my router wifi , it loads just fine..further the site got hacked 2 weeks back, if u cld share ur email i wld elaborate further

  5. Fitzcarraldo says:

    $ host facebook.com
    facebook.com has address 69.171.230.5
    facebook.com has IPv6 address 2a03:2880:f003:c07:face:b00c:0:2
    facebook.com mail is handled by 10 msgin.vvv.facebook.com.

    $ host cisco.com
    cisco.com has address 72.163.4.161
    cisco.com has IPv6 address 2001:420:1101:1::a
    cisco.com mail is handled by 30 aer-mx-01.cisco.com.
    cisco.com mail is handled by 20 rcdn-mx-01.cisco.com.
    cisco.com mail is handled by 10 alln-mx-01.cisco.com.

    $ host google.in
    google.in has address 173.194.124.51
    google.in has address 173.194.124.49
    google.in has address 173.194.124.50
    google.in has address 173.194.124.52
    google.in has address 173.194.124.48
    google.in has IPv6 address 2404:6800:4009:805::2004

    $ host shappad.in
    shappad.in has address 188.40.169.249
    Host shappad.in not found: 2(SERVFAIL)
    shappad.in mail is handled by 0 shappad.in.

    I recommend you contact your ISP and your Web site hosting provider for advice. As you say your site has been hacked, in any case you should seek the advice of the Web site hosting service provider.

  6. Fitzcarraldo says:

    Have you tried forcing the DHCP client on your PC to acquire a new IP address from the DHCP server? I did that and can now access your site (although that might be a coincidence, as the site page was different to before and displayed a notice that it was under construction):

    First I found out the name of the network interface I am using:

    $ ifconfig | grep mtu | awk '{print $1}'
    enp4s0f1:
    lo:
    wlp3s0:

    I’m using Wi-Fi at the moment, and the wireless interface name in my installation is ‘wlp3s0‘.

    Then I released and renewed the lease from the DHCP server:

    $ sudo dhclient -v -r wlp3s0
    Password:
    Internet Systems Consortium DHCP Client 4.3.1 Gentoo-r2
    Copyright 2004-2014 Internet Systems Consortium.
    All rights reserved.
    For info, please visit https://www.isc.org/software/dhcp/

    Listening on LPF/wlp3s0/a8:bc:27:a2:7e:9b
    Sending on LPF/wlp3s0/a8:bc:27:a2:7e:9b
    Sending on Socket/fallback

    $ sudo dhclient -v wlp3s0
    Internet Systems Consortium DHCP Client 4.3.1 Gentoo-r2
    Copyright 2004-2014 Internet Systems Consortium.
    All rights reserved.
    For info, please visit https://www.isc.org/software/dhcp/

    Listening on LPF/wlp3s0/a8:bc:27:a2:7e:9b
    Sending on LPF/wlp3s0/a8:bc:27:a2:7e:9b
    Sending on Socket/fallback
    DHCPDISCOVER on wlp3s0 to 255.255.255.255 port 67 interval 8
    DHCPREQUEST on wlp3s0 to 255.255.255.255 port 67
    DHCPOFFER from 10.155.240.1
    DHCPACK from 10.155.240.1
    bound to 10.154.245.215 -- renewal in 3417 seconds.

    Then I checked to see if I could ping the site’s IP address:

    $ sudo ping 188.40.169.249
    Password:
    PING 188.40.169.249 (188.40.169.249) 56(84) bytes of data.
    64 bytes from 188.40.169.249: icmp_seq=1 ttl=51 time=183 ms
    64 bytes from 188.40.169.249: icmp_seq=2 ttl=51 time=206 ms
    64 bytes from 188.40.169.249: icmp_seq=3 ttl=51 time=229 ms
    64 bytes from 188.40.169.249: icmp_seq=4 ttl=51 time=149 ms
    ^C
    --- 188.40.169.249 ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3003ms
    rtt min/avg/max/mdev = 149.698/191.963/229.115/29.350 ms

    Then I checked to see if I could ping the site’s domain name:

    $ sudo ping www.shappad.in
    PING shappad.in (188.40.169.249) 56(84) bytes of data.
    64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_seq=1 ttl=51 time=156 ms
    64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_seq=2 ttl=51 time=203 ms
    64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_seq=3 ttl=51 time=225 ms
    64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_seq=4 ttl=51 time=146 ms
    ^C
    --- shappad.in ping statistics ---
    4 packets transmitted, 4 received, 0% packet loss, time 3004ms
    rtt min/avg/max/mdev = 146.766/183.222/225.989/32.635 ms

    Then I checked the site using the host command:

    $ host www.shappad.in
    www.shappad.in is an alias for shappad.in.
    shappad.in has address 188.40.169.249
    shappad.in mail is handled by 0 shappad.in.

    Then I checked if I could access the site shappad.in in a Web browser, and I could (it still displayed a page saying the site was under construction).

    • Fitzcarraldo says:

      I tried accessing the Web site in a browser again a little later, and it now looks like it did originally when you first posted its IP address, so I assume you have reinstalled the Web site.

    • harishsankar says:

      ifconfig | grep mtu merely ran the command but returned nothing then i tried ls /sys/class/net
      which returned etho l0 (letter l and num 0) then i tried the dhcp command which returned

      Internet Systems Consortium DHCP Client 4.1-ESV-R4
      Copyright 2004-2011 Internet Systems Consortium.
      All rights reserved.
      For info, please visit https://www.isc.org/software/dhcp/

      Unsupported device type 772 for “lo”

      the site would show the message underconstruction..now here is a weird fact, my developer uses the same ubuntu 14.01 as me at office, he has the same problem,but the other systems in the office could access the site.. if u cld please share ur email it would give me privacy in exposing the details of the site.

    • harishsankar says:

      ifconfig | grep mtu merely ran the command but returned nothing then i tried ls /sys/class/net
      which returned etho l0 (letter l and num 0) then i tried the dhcp command which returned

      Internet Systems Consortium DHCP Client 4.1-ESV-R4
      Copyright 2004-2011 Internet Systems Consortium.
      All rights reserved.
      For info, please visit https://www.isc.org/software/dhcp/

      Listening on LPF/eth0/44:87:fc:6f:e6:6a
      Sending on LPF/eth0/44:87:fc:6f:e6:6a
      Sending on Socket/fallback
      DHCPREQUEST of 192.168.1.102 on eth0 to 255.255.255.255 port 67
      DHCPACK of 192.168.1.102 from 192.168.1.1
      RTNETLINK answers: File exists
      bound to 192.168.1.102 — renewal in 97678 seconds.
      pinging resulted in ,

      PING shappad.in (188.40.169.249) 56(84) bytes of data.
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=1 ttl=47 time=269 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=2 ttl=47 time=263 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=3 ttl=47 time=262 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=4 ttl=47 time=256 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=5 ttl=47 time=258 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=6 ttl=47 time=258 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=7 ttl=47 time=257 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=8 ttl=47 time=261 ms
      64 bytes from static.249.169.40.188.clients.your-server.de (188.40.169.249): icmp_req=9 ttl=47 time=266 ms
      ^C
      — shappad.in ping statistics —
      10 packets transmitted, 9 received, 10% packet loss, time 9009ms
      rtt min/avg/max/mdev = 256.586/261.598/269.613/4.184 ms

      • Fitzcarraldo says:

        If you are using network interface eth0 then you should have used the following commands to release and renew the lease:

        $ sudo dhclient -v -r eth0
        $ sudo dhclient -v eth0

        As I am not using Ubuntu I suggest you post your request on the Ubuntu Forums, as they will be able to duplicate your commands in a more-similar installation to yours. Also, forums are a better place to deal with problems. This blog is not a support forum. I also recommend that you talk to your ISP and your Web site host provider (not just your site developer) about the problem.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

Join 67 other followers

%d bloggers like this: