[Pandora Archive logo] National Library of Australia and Partners
PANDORA Archive
Preserving and Accessing Networked Documentary Resources of Australia

PANDAS Manual


Default Gather Settings

The gather module in PANDAS uses a standard interface developed specifically for PANDAS. It is designed to allow gathering software, such as off-line browsers and web spiders, to be used and upgraded or changed without requiring the user to become familiar with different proprietary softwares and versions. Currently the primary gathering software used is HTTrack (see the HTTrack website copier home page for more information).

NB: The gather filters and settings on PANDAS are based on those used by the website copier HTTrack. Documentation on the HTTrack homepage provides more detailed descriptions of the filters and settings and other useful FAQs. See the HTTrack documentation page.

Basic Settings

The default settings included in the gather module are designed to facilitate gathering procedures. These include the following defaults found on the Settings tab on the Gather Details screen:

  • Directory Travel - default setting is: Can go down. This means that the gatherer will gather all files in the starting directory and any sub-directories of that directory.

    Other Directory Travel options you can select are:
    Stay on the same directory - use this if you do not wish to gather sub directories of the starting directory
    Can go up - use this if your you don't wish to gather sub-directories of the starting directory but you do want to capture files in directories above the starting directory
    Can go both up and down - you could use this option if you want to capture a whole web site but your starting directory is not at the top level. This option will allow you to start your gather at any level in the web site and the gatherer will capture both sub-directories of the starting directory and any directories above the starting directory

  • Host Travel - default setting is: Stay on the same address. This means that the gatherer will remain on the same IP address (host name) of the starting directory. For example if you start on http://www.olympics.com.au/index.html the gather will only gather from the http://www.olympics.com.au/ host, and not, for example, from http://shop.olympics.com.au/

    Other Host Travel options you can select are:
    Stay on the same domain - this option allows the gatherer to gather from various hosts on the same domain. In the URL http://www.olympics.com.au/ the domain is "olympics". You could use this option if, for example, you did wish to gather files on both the http://www.olympics.com.au/ host and the http://shop.olympics.com.au/ host
    Stay on the same location - this options restricts the gatherer to a major portion of the Internet, i.e. the domain type such as .com .gov .edu. It is unlikely you will have a use for this option when gathering specific titles.
    Go everywhere - as the label suggest this means that the gatherer will not be restricted to any address, domain or location. Without using other options such as limiting the maximum depth of the archiving this option will allow you gather the whole Internet, or at least every place that there is a link to. It is strongly recommended that you do not use this option!

  • Max. Depth - i.e. Maximum Depth: default setting is: 50. This means that the gather can go to 50 sub-directories below the start directory. You will probably never need to increase this, although you can, as generally most web sites don't go beyond 20 levels. A default level is set so as to avoid symbolic links causing and infinite recursion of directory levels which is not good for the gatherer and not good for the site being gathered.

    Other options - you may find it useful to change the Max. Depth when gathering sites where you want to gather some documents linked directly (i.e. one level) from your starting page but to do so would mean you will also gather the whole, much larger site, and you are unable to limit the gather effectively using URL filters. By limiting the gather depth to 2 or 3 levels (you may have to experiment to get it right for the particular site) you will prevent the gatherer from gathering the whole site although you may still end up with some pages you don't want.

  • Get Near Files - the default setting is: i.e. ticked to on. Having this option switched on means that non-HTML files like images, sound files, PDF and zip files that are linked to from the gathered pages will also be gathered irrespective of their location. This means that you do not have to create URL filters to gather these types of files which are often located in directories that are not necessarily sub-directories of your starting directory. If you do not wish to gather PDF or zip files, for example, you will need to switch this option to off or create URL files to deny those types of files.

Advanced Settings

The Advance Settings tab is not visible unless you click the Advanced settings tick box on the Basic tab. You should not have any need to change these options. However they are made available should you need to view them. One default to note is the Follow Robots rule which is set to Never. This means that the gather is set so as to ignore robot exclusion rules. Many sites have robot rule text files to discourage web spiders, offline browsers and other robots from doing such things as downloading their sites. However, because the PANDORA Archive only includes sites for which we have gained permission to archive from the publisher our gatherer is set to ignore these robot exclusion rules

 

| EU Manual | PANDAS Contents | Back | Next |


 
Last updated 1 August 2002

[National Library of Australia]