Help > Trawling password protected pages

Some web pages are password protected and so when a user tries to view them, they will be challenged with dialogs like the one shown below:

Above: Firefox password dialog

Just like all web browsers, DeepTrawl must give a login and password to access these pages

If DeepTrawl has not been provided with the correct password for a page, the problem will show up as a dead link, indicating a 401 error as the reason.

In order to add the password for one or more pages to DeepTrawl, please start by clicking the drop down menu item shown below:

Above: The Password protection manager menu item

Once you have clicked the menu item, you will see the dialog shown below:

Above: The Password protection manager

The password protection manager allows you to enter many full or part URLs for which logins and passwords must be given. To add a new login/password, press the Add button.

You will now see the dialog shown below (each part of the dialog is explained beneath):

Above: The add username and password dialog

Enter a full or part URL for which you want to provide a login/password. A full url might be something like: MyDomain.com/myPage.html If you want to use this login password for this entire web site you could enter: MyDomain.com

Matching of URLs from field 1 is usually not case-sensitive. If you select this check box, pages will only be matched as having the URL entered in field 1 if the case of the URL is correct.

Enter the username to be given for this page / group of pages.

Enter the password to be given for this page / group of pages.

Re-enter the password to be given for this page / group of pages.

After you click OK in this dialog and then OK in the password protection manager, this username / password will be used with this page / group of pages in every trawl from now on.

When in the password protection manager, you can also delete one or more username / password settings by selecting the check box in the left-most column and clicking Delete selected.