Help > Scope

This scope setting provides the widest scope for a trawl. All pages from the same domain as the start page will be trawled, even if they are in a different sub-domain.

For instance if the start page:

http://MyDomain.com/index.htm

was given, all the pages below would be included in the scope of the trawl:

http://MyDomain.com/contact.htm

http://MyDomain.com/products/p1.htm

http://MySubDomain.MyDomain.com/index.htm

This scope setting was designed for sites which include multiple directories and may have multiple sub-domains. It is the default scope level as it is most commonly used and is recommended if you're not sure what scope setting to use.

If DeepTrawl finds pages which you do not wish to be listed because you consider them to be part of a different website you might consider using one of the other, more restrictive, scope settings.

Using this scope setting, all pages from the same domain as the start page will be trawled, except those in different sub-domains.

For instance if the start page:

http://MyDomain.com/index.htm

was given, all the pages below would be included in the scope of the trawl:

http://MyDomain.com/contact.htm

http://MyDomain.com/products/p1.htm

The page below would be excluded:

http://MySubDomain.MyDomain.com/index.htm

This scope setting was designed for sites which include multiple directories within the same domain but not sub-domains.

If DeepTrawl finds pages which you do not wish to be listed because you consider them to be part of a different website you might consider using one of the other, more restrictive, scope settings.

Using this scope setting, all pages from the same directory / any subdirectory of the start page will be trawled.

For instance if the start page:

http://MyDomain.com/MyArea/index.htm

was given, the pages below would be included in the scope of the trawl:

http://MyDomain.com/MyArea/contact.htm

http://MyDomain.com/MyArea/SubDir/contact.htm

The page below would be excluded from the trawl:

http://MyDomain.com/AnotherArea/index.htm

This scope setting was designed for sites which only occupy a few directories on the server. This is common for sites where the domain name is shared, for example this is often the case for web space given by an ISP or blogging site.

Note: If you enter a URL as the starting page which ends in only a directory name, e.g.

http://www.MyDomain/myDir/

You must make sure to add / (forward slash) after the directory name.

Using this scope setting, all pages from the same directory of the start page will be trawled, but not those from sub-directories.

For instance if the start page:

http://MyDomain.com/MyArea/index.htm

was given, the page below would be included in the scope of the trawl:

http://MyDomain.com/MyArea/contact.htm

The pages below would be excluded from the trawl:

http://MyDomain.com/MyArea/MyDir/index.htm

http://MyDomain.com/AnotherArea/index.htm

This scope setting was designed for sites which only occupy a single directory on the server.

Note: If you enter a URL as the starting page which ends in only a directory name, e.g.

http://www.MyDomain/myDir/

You must make sure to add / (forward slash) after the directory name.

Using this scope setting, only the start page will be trawled.

For instance if the start page:

http://MyDomain.com/index.htm

was given, only this page would be trawled.

Using this scope setting, only pages whose URL includes a user defined piece of text will be trawled.

This scope setting is especially useful when trawling sites whose HTML is dynamically generated on the server side. In this case the structure of the site is often not defined by directories but instead by keywords, set in the design of the site. One of those keywords might be entered to define which pages should be trawled.

This scope setting is also useful for web sites which span several domains. The common part of the domain name can be entered to trawl all the required pages. For example, the text Microsoft could be entered to include all the pages within the microsoft.com and microsoft.co.uk domains in the scope.