Help > Trawling for Broken / Invalid Hyperlinks

What does this trawl type find?

As one of its most useful features, DeepTrawl allows you to find broken or invalid hyperlinks. These might be:

• links to pages which no longer exist (404 errors);
• Other HTTP error codes;
• Malformed URLs;
• Non-contactable servers;
• Missing bookmarks (anchors)

Bad links can take the form of:

• Text links
• Images which activate a link when clicked
• Frames
• Client side image maps

Invalid hyperlinks are a major problem for any website. At best they make the site look unprofessional. At worst, they can shake users confidence so much they leave the site.

It is reccommended you run DeepTrawl with this trawl type activated regularly over the entire lifetime of your website, so that you always present the most professional appearence to your users and allow them to find what they're looking for.

How to use this trawl type

You should make sure this trawl type is switched on. There should be a green tick next to this trawl type in the Trawl for... tab. If there is a red cross instead, click the cross to toggle the trawl type on.

This trawl type also has several options contained in its settings dialog. This can be accessed by pressing the Settings link to the right of the trawl type in the Trawl for... tab. The dialog is shown below, all the features are explained beneath...

Above: The link validator settings dialog

The Trawl for malformed URLs check box allows you to enable / prevent checking for badly structured URLs (for example full URLs which have a protocol which is not widely recognised).

The Trawl for problems contacting remote servers check box allows you to stop DeepTrawl from showing a problem when a server cannot be reached over the Internet. This should usually be left on.

The Trawl for missing anchors (bookmarks) check box allows to switch off checking for the existence of anchors in linked pages. Anchors allow a link to point a web browser at a specific section of a web page, but they require a special tag to be included in the target page.

The Trawl for HTTP codes... section of the dialog allows you to switch on / off searching for specific HTTP error codes. For instance you may wish to never see errors caused by 401 (unauthorized) HTTP codes.

How to solve the problems found

Solving 404 errors

Most of the time a 404 error is caused because the owner of the web site the link points to has removed or moved the target page. It is recommended you either remove this link from your site or point it to an alternative page.

Solving Malformed URL errors

Most of the time, malformed URL errors are caused by a typo in your HTML. It is suggested that you review the link URL to make sure it is valid.

Solving Non-contactable server errors

A server may only be temporarily non-contactable. If you are worried the server is unavailable too often, it is recommended you remove this link from your site.

Solving Missing bookmarks (anchor) errors

These errors are caused because the links' tarket page does not contain an appropriate bookmark/anchor. The bookmark/anchor can be added using the following HTML:

<a name="myAnchor"></a>

The following URL would now point to the specific part of the web page where the above tag is located:

http://mySite.com/myPage.com#myAnchor