Cable Modem Troubleshooting Tips

Web Proxy issues: on this page:

• Transparent web proxy caches: what they do  Updated 2004 Jan 8 
• Which is my web proxy cache? - identifying the address of the web proxy
• Stale pages in a proxy web cache: how to refresh
• Sidestepping the transparent web proxy caches: how to set explicit proxies
• Setting an explicit web proxy in Internet Explorer for Windows
• Setting an explicit web proxy in Internet Explorer for Macintosh
• Setting an explicit web proxy in Netscape or Mozilla
• NTL web proxy caches: list by centre  Updated 2004 Feb 8 
• Blueyonder web proxy caches: list
• Web authoring tips for being cache-aware

The transparent web caches

Your ISP's network might intercept all attempts you make to connect to the web (by means of the HTTP protocol on TCP port 80), and route your request through a transparent web proxy cache, technically known as an interception proxy. NTLworld used Inktomi Traffic Servers, but is now introducing NetApp NetCache appliances. Your ISP does this in order to reduce the traffic on their backbone, and to reduce the traffic they exchange with other operators' networks. If a proxy cache works well, it can also improve performance for the customer, by supplying locally cached copies of web pages and graphics much more quickly than by fetching them from the remote original web server.

However, when the proxy web cache is overloaded, it can make your web browsing appear slower than the nominal speed of your cable modem. For this reason, you should not rely on HTTP download speeds as evidence of the practical speed of your cable modem.

According to reports, Blueyonder no longer enforce transparent web proxies (in some regions?), but permit their users to configure Blueyonder's web proxies explicitly if they wish.

Which is my web proxy cache?

You can find out whether your web connection is being proxy cached, and, if so, the address of the web proxy cache, by visiting either of these sites:

• http://www.all-nettools.com/pr.htm (see Remote address or Remote host)
• http://www.broadband-help.com/cm_diagnose.asp?init=1 (see Proxy Name(R-DNS))

Because the ISP might be load-balancing across more than one web proxy cache, your web connections might use different proxy caches for different web destinations. So you might get different answers from the two sites above. And the actual cache used for your other web accesses might be different yet again. When load-balancing is in operation, the above sites will identify an individual cache in a cluster of caches, and the DNS names of the caches will usually have an obvious structure enabling you to deduce the names of other members of the cluster.

Refreshing stale pages in the web cache

While a copy of a page is stored in a proxy cache, the original page could be updated, and users would never see it, because the proxy will return the stale copy to users. Even if a browser user clicks on Refresh or Reload, the proxy will still return the stale version.

With MSIE for Windows, the fix is to use Ctrl-Refresh (hold down the Ctrl key while clicking on Refresh) or Ctrl-F5, which causes the transparent proxy to refresh its copy of the page from the original web server. However, Ctrl-Refresh does not work with:

• MSIE v5.5 unless it has been updated with Service Pack 1 or higher. MSIE v5.5 (without the service pack) was the version of MSIE supplied with Windows 2000 and Windows ME.
• MSIE 5.01 and MSIE 5.01 SP1.

See http://support.microsoft.com/default.aspx?scid=KB;EN-US;Q266121 for more details. The faulty versions of MSIE send cache-control commands on Refresh or Ctrl-Refresh only if they know it is using a user-configured proxy. A fixed version of MSIE, such as MSIE 5.5+SP1 sends a cache-control command also for Ctrl-Refresh even when functioning without a user-configured proxy, and so works better with transparent proxies. If you have faulty versions of MSIE 5.01 or 5.5, use Windows Update to update MSIE with the latest service pack.

MSIE 5.0 for Mac and earlier versions have similar problems. It sends cache-control requests only if it knows it is using a user-configured proxy, and does not send cache-control requests when working without a configured proxy. Therefore there is no way to flush stale pages from a transparent proxy cache. This was fixed in version MSIE 5.1 for Mac (download http://www.microsoft.com/mac/download/ie/ie51.asp), so that Option-Refresh will always send cache-control commands. An ordinary Refresh will not work.

Another method which works is to use Netscape (on any platform) as your browser. In Netscape, clicking Reload always causes the proxy cache to conditionally refresh the page from the original server. Clicking Shift-Reload (Option-Reload for Mac Netscape earlier than version 6) forces the proxy cache to unconditionally refresh the page from the original server.

As of March 2001, transparent web proxy caches in some regions of the NTL network have been reconfigured so that Ctrl-Refresh/Shift-Reload/Option-Reload no longer force a reload from the original server, unless the page had been updated. This is technically incorrect, though it yields the right visible results. It will however, yield incorrect timings if you had been trying to time downloads from the original server: you will only get a download from the proxy.

In cases where the transparent proxy cache is a Network Appliance NetCache, to ensure that you get a full refresh from the original server, you should perform a full browser refresh (Ctrl-Refresh in IE, or Shift-Reload in Netscape) twice, with the second one following the first after a gap of at least 1 and not more than 10 seconds.

Another technique to force a fresh copy of a page through a proxy cache to your browser is to edit the URL by appending a question mark followed by a string chosen to be different each time you try this trick. For instance, if the URL was originally:

http://www.erehwon.com/somepage.html

then the first time you want a fresh copy, you would edit it to be:

http://www.erehwon.com/somepage.html?1

and the second time you want a fresh copy, you would edit it to be:

http://www.erehwon.com/somepage.html?2

and so on. This works because the cache treats these URLs as being references to different documents. However, there are drawbacks to this technique:

• it works only for the HTML page, and not any embedded images or objects;
• it cannot be used if the URL already has a question mark in this position, because the chances are that you would send a malformed request to a CGI in the server;
• it doesn't actually refresh the cached copy of the original URL: it just adds another cached version, so other users will continue to receive the old cached copy.

How to sidestep the transparent web caches

When your default transparent proxy cache breaks down, you lose all contact with the web. Because it is a transparent cache, there is no setting you can adjust on your computer to cause it to go direct to the remote site rather than via the proxy.

You can however defeat the transparent web cache by configuring your web browser explicitly to use another proxy cache that accepts requests on a port other than 80 (usually 8080 or 3128). Because the request is made on a port other than 80, the transparent cache does not intercept the request.

In choosing an alternative HTTP (web) proxy server name, try the following in order of preference for performance:

• Your default transparent proxy cache (see Which is my cache?): often it works when selected explicitly even when not working transparently;
• Another HTTP proxy cache at the same network centre of your ISP;
• An HTTP proxy cache elsewhere on your ISP's network:
• see below for a list of NTL and BlueYonder web proxy caches;
• An HTTP proxy cache on another network:
• for example, UK users might try webcache.bt.net on port 8080.

The higher in the list above, the better the web browsing performance is likely to be. You should revert to normal working (with no explicit proxy setting) when your default transparent proxy has recovered.

Each application that used the HTTP protocol is likely to have its own HTTP proxy configuration, every one of which might need setting. Just setting the HTTP proxy for Internet Explorer will not affect the HTTP settings for (e.g.) Netscape, RealPlayer, QuickTime Player, etc.

Setting an explicit web proxy in Internet Explorer for Windows

The following procedure works on all Windows platforms running Internet Explorer 5.x/6.x, and the procedure for Internet Explorer 4.x is very similar:

• Open Control Panel Internet Options (or, in IE, pull down menu Tools and select Internet Options...).
• Click on the tab Connections.
• Click on button LAN Settings.
• UNcheck Automatically detect settings.
• UNcheck Use automatic configuration script.
• Check Use a proxy server but ensure that the boxes are left blank.
• Click on button Advanced:
• In the row HTTP fill in the name and port number of your chosen web proxy server (see above). It is essential that you do not select port 80, even if your chosen server offers a proxy service on that port.
• Ensure that the other rows are left blank.
• UNcheck Use the same proxy server for all protocols.
• In the Exceptions box enter: 192.168.*
• Click OK.
• Click OK to exit LAN settings.
• Click OK to apply the changes and exit Internet Options.

The last few stages in Advanced avoid using the proxy for Secure HTTPS and FTP traffic, and ensure that you retain local access to any cable modem diagnostics pages and to any NAT router setup pages, which typically use addresses in the range 192.168.xxx.xxx.

To return Internet Explorer to normal operation with no explicit web proxy:

• Open Control Panel Internet Options (or in IE pull down menu Tools and select Internet Options...).
• Click on the tab Connections.
• Click on button LAN Settings:
• UNcheck Automatically detect settings.
• UNcheck Use automatic configuration script.
• UNcheck Use a proxy server.
• Click OK to exit LAN settings.
• Click OK to exit Internet Options.

Setting an explicit web proxy in Internet Explorer for Macintosh

To set an explicit web proxy in MSIE 5.x:

• Pull down the Edit menu and select Preferences.
• Click the triangle to expand Network.
• Select Proxies.
• Check Web Proxy.
• Uncheck Use Web Proxy for all.
• Check Bypass Web Proxy for FTP.
• Click the Settings button for Web Proxy to open the settings panel.
• Set the Address box to the name of the web proxy server (see above).
• Set the Port box to the port number, usually 8080.
• Set Method to Normal.
• Empty the boxes User Name and Password.
• Click OK to return to the main panel.
• Uncheck Secure Proxy, Mail Proxy, and Gopher Proxy.
• In the box List the sites you want to connect to directly enter: 192.168.*
• Click OK.

Setting an explicit web proxy in Netscape or Mozilla

• Pull down the Edit menu and select Preferences.
• In the left-hand panel, expand Advanced and select Proxies.
• In the right-hand panel, check Manual proxy configuration.
• If there is a button Configure or View on this line, click it.
• In the row HTTP Proxy enter the name of your chosen HTTP proxy server, and in the box Port enter the port number, usually 8080 or 3128. Do not use 80.
• Ensure all other proxy and SOCKS rows are cleared to blank.
• In the box No proxy for enter: 192.168.100.1
• There is no wild card feature: if you have more than one local web server, separate the IP addresses with a comma, such as: 192.168.100.1,192.168.1.1
• If you had to click a button to reach this panel, click OK to exit Manual Proxy Configuration.
• Click OK to exit Preferences.

To restore to normal working:

• Pull down the Edit menu and select Preferences.
• In the left-hand panel, expand Advanced and select Proxies.
• In the right-hand panel, check Direct connection to the Internet.
• Click OK to exit Preferences.

Web authoring tips for dealing with caches

Web authors should (some say must) include hints in their pages as to how web caches should treat them. If your pages are static, and rarely subject to change, then you need take no action: the caches will cache your page and end-users will see correct results. If your pages are dynamic (created on the fly) or subject to frequent updates, then you must include a hint to web caches that they should check back for updates, otherwise they will cache your page and serve stale copies to clients. The mechanism for doing the hinting is the cache-control line in the HTTP headers, fully described in RFC 2616.

There is no mechanism for authors to generate custom HTTP headers on homepage.ntlworld.com.

Some authors advise that web authors can achieve the same effect with a meta tag in the HTML headers of their page, for instance:

<html>
    <head>
        <meta http-equiv="cache-control" content="no-cache">
        <title>My page</title>
    </head>

In fact, those authors are wrong. Web caches take no account of meta HTML headers; they respond only to HTTP headers. The meta tag above would work only if the hosting web server is of the type that promotes meta http-equiv HTML tags to true HTTP headers on the fly as the page is served out.

The server homepage.ntlworld.com does not promote meta http-equiv tags to HTTP headers.

The server xxx.pwp.blueyonder.co.uk does not promote meta http-equiv tags to HTTP headers.

For more detailed information on authoring in a cache-aware manner, see http://www.mnot.net/cache_docs/.

Return to Index.