Default Gather Settings
The gather module in PANDAS uses a standard interface
developed specifically for PANDAS. It is designed to allow gathering
software, such as off-line browsers and web spiders, to be used and upgraded or
changed without requiring the user to become familiar with different
proprietary softwares and versions. Currently the primary gathering software
used is HTTrack (see the HTTrack website
copier home page for more information).
NB: The gather filters and settings on PANDAS are based on
those used by the website copier HTTrack. Documentation on the HTTrack homepage
provides more detailed descriptions of the filters and settings and other
useful FAQs. See the HTTrack
documentation page.
Basic Settings
The default settings included in the gather module are designed to
facilitate gathering procedures. These include the following defaults found on
the Settings tab on the Gather Details screen:
- Directory Travel - default setting is: Can go
down. This means that the gatherer will gather all files in the starting
directory and any sub-directories of that directory.
Other Directory
Travel options you can select are: Stay on the same directory -
use this if you do not wish to gather sub directories of the starting
directory Can go up - use this if your you don't wish to gather
sub-directories of the starting directory but you do want to capture files in
directories above the starting directory Can go both up and down -
you could use this option if you want to capture a whole web site but your
starting directory is not at the top level. This option will allow you to start
your gather at any level in the web site and the gatherer will capture both
sub-directories of the starting directory and any directories above the
starting directory
- Host Travel - default setting is: Stay on the same
address. This means that the gatherer will remain on the same IP address
(host name) of the starting directory. For example if you start on
http://www.olympics.com.au/index.html the gather will only gather from the
http://www.olympics.com.au/ host, and not, for example, from
http://shop.olympics.com.au/
Other Host Travel options you can
select are: Stay on the same domain - this option allows the gatherer
to gather from various hosts on the same domain. In the URL
http://www.olympics.com.au/ the domain is "olympics". You could use this option
if, for example, you did wish to gather files on both the
http://www.olympics.com.au/ host and the http://shop.olympics.com.au/
host Stay on the same location - this options restricts the gatherer
to a major portion of the Internet, i.e. the domain type such as .com .gov
.edu. It is unlikely you will have a use for this option when gathering
specific titles. Go everywhere - as the label suggest this means
that the gatherer will not be restricted to any address, domain or location.
Without using other options such as limiting the maximum depth of the archiving
this option will allow you gather the whole Internet, or at least every place
that there is a link to. It is strongly recommended that you do not use this
option!
- Max. Depth - i.e. Maximum Depth: default setting is:
50. This means that the gather can go to 50 sub-directories below the
start directory. You will probably never need to increase this, although you
can, as generally most web sites don't go beyond 20 levels. A default level is
set so as to avoid symbolic links causing and infinite recursion of directory
levels which is not good for the gatherer and not good for the site being
gathered.
Other options - you may find it useful to change the Max.
Depth when gathering sites where you want to gather some documents linked
directly (i.e. one level) from your starting page but to do so would mean you
will also gather the whole, much larger site, and you are unable to limit the
gather effectively using URL filters. By limiting the gather depth to 2 or 3
levels (you may have to experiment to get it right for the particular site) you
will prevent the gatherer from gathering the whole site although you may still
end up with some pages you don't want.
- Get Near Files - the default setting is:
i.e. ticked to
on. Having this option switched on means that non-HTML files like images, sound
files, PDF and zip files that are linked to from the gathered pages will also
be gathered irrespective of their location. This means that you do not have to
create URL filters to gather these types of files which are often located in
directories that are not necessarily sub-directories of your starting
directory. If you do not wish to gather PDF or zip files, for example, you will
need to switch this option to off or create URL files to deny those types of
files.
Advanced Settings
The Advance Settings tab is not visible unless you click
the Advanced settings tick box on the Basic tab. You should not
have any need to change these options. However they are made available should
you need to view them. One default to note is the Follow Robots rule
which is set to Never. This means that the gather is set so as to ignore
robot exclusion rules. Many sites have robot rule text files to discourage web
spiders, offline browsers and other robots from doing such things as
downloading their sites. However, because the PANDORA Archive only includes
sites for which we have gained permission to archive from the publisher our
gatherer is set to ignore these robot exclusion rules
|