Gathering
To begin gathering, access the Gather Details screen from the Title Edit screen where you can specify your gather options and settings.
There are three ways you can access the Gather Details screen:
From the Permission Granted Titles Worktray: Click on the Schedule action button at the end of the title name.
Search and view the title. On the View Title screen, there is a Gather Details section. Click on the Edit button at the end of the Gather Details section.
Search and view the title. Click on the Edit button at the bottom or top right hand corner of the screen. On the Title Edit screen, click on Gather Details tab from the top menu bar.
This screen has three sections.
The top section includes options for the gather method and for scheduling gathers.
The middle section is for the Filter Rules. Click on the right arrow button next to the Filter Rules and a window will be opened up for you to add any URL gather filters.
The bottom section is for the Basic Gather Options. Click on the right arrow button next to the Basic Gather Options and a window will be opened up for you to add any extra seed URLs, authentication and password details.
Gather Methods:
- HTTrack: The default gather method is "HTTrack" which means the gatherer used by PANDAS will download a copy of the resource indicated in the Seed URL from the live web. This URL defaults from the Seed URL field as entered on the title record when it is registered. It can be changed here if desired.
- Upload: You can use this option to upload a single file (e.g. a PDF document supplied as an email attachment) or a collection of files and directories (e.g. a web site) saved in your local drive. To do this:
- Go to the Title Edit - Gather Details screen
- From the Method drop-down box select Upload
- Click Gather Today and the today’s date is set and a one-off gather is scheduled
- Click the Save button (this will create an "instance" immediately) and the title will appear in the Preserve - Instances for Upload Worktray.
- Go the Upload Worktray, click on the Upload link on the far right (you cannot access the instance from anywhere else)
- You will now be taken to the Upload and Post-Process Title page
- From that page select and copy the WebDAV url for this instance by copying the url from the Web Dav URL link
- Use your Windows My Network Places function (or Web Folder depending on the version of Windows you are using) to create a new network place and open access to the PANDAS instance you have created
- Using Windows Explorer, copy and paste the file/s from the local drive to the PANDAS instance
- Once you have added your files to the instance, back in PANDAS, select a start url and enter it in the Archive Url box provided. To do this overwrite any text that is in the box with your chosen file name string, remembering that you need only put in the gathered url string (i.e. if your start page is http://pandas.nla.gov.au/view/3000/20081002-0000/www.nla.gov.au/grants/haroldwhite/index.html you will only enter www.nla.gov.au/grants/haroldwhite/index.html).
Click on Save and the url will be set as the Archive URL (and a message displayed). Return to the Archive URL link above which should have been updated and will now direct to your newly specified starting page. Remember to always test that the changed url works before clicking on the Archive button.
- Once you have added and checked your start url, select the Post Processing button - you will not be able to see the Archive option until post-processing has occurred
- The Upload instance will now re-appear in the gather queue, and once processing is complete will move into the Instances in QA Worktray
- You can select the Process option from the Worktray, and from there see that you may now Archive or Delete the instance.
Note that if you gather files from HTTrack on your desktop for later Uploading you will need to make adjustments to your HTTrack options. The option No External Pages will need to be ticked so that the post processing script will function and will deliver the PANDORA referring page for external web links.
Setting a gather schedule
Schedule a one-off gather
Gather Today
For a small web publication, you can submit a one-off gather request right away by clicking on the Gather Today button. The date in the date box is defaulted to today’s date. On clicking the Gather Today button, the date will be shown immediately underneath the date box. To delete the date, click on the - sign next to it. On saving the record with the date set, the title will appear in the Gather - Scheduled Titles Worktray and will be gathered as soon as there is a vacancy in the gather queue.
With PANDAS 3, more than one instance can be gathered on the same day, so beware of requesting unwanted multiple gathers. If you do not see your title immediately queued after you have selected to gather, do not go back and request another gather, as another new gather will be instituted and the system will not generate an error as in (PANDAS 2)
Add Date - to gather for another day
For a big site, it is best to schedule a one-off gather for the next day (i.e. to be gathered overnight) or over a weekend when the gather queue is not so busy.
To schedule a one-off gather for any other day, edit the date box and press the Add Date button. The one-off gather date will then be set underneath the date box. To delete the date, click on the - sign next to it. On saving the record with the one-off gather date set, the title will appear in the Gather - Scheduled Titles Worktray if the schedule date is within next 30 days.
Schedule a period frequency
To create a period gather schedule:
Select a gather period (e.g. weekly, monthly, annual) from the Schedule gather drop-down box on the Title Edit - Gather Details screen.
You can set a period frequency gather schedule before or after a one-off gather is set or completed.
To set a period gather schedule before a one-off gather
If a period gather schedule is set before a one-off gather instance (via Add Date) is completed, the system will generate a one-off instance to be gathered ‘today’ automatically. On saving the record, the title will appear in the Gather - Scheduled Titles Worktray, the first and last gathered date will be set for the day and the Next Recurring Date will also be defaulted to today’s date.
To set a period gather schedule after a one-off gather
If a gather schedule is set after a one-off gather instance is gathered, the Next Recurring Date will be calculated basing on the last gathered date. On saving the record, the period schedule and the Next Recurring Date will be set.
Note: If you do not need to set a schedule, tick the No Scheduled Gathers tick box. This will prevent the title re-appearing in your Worktrays as requiring a schedule. When selecting this option you will be prompted for a reason. We are planning to update the offered reasons, but in the meantime you are only offered 'No change to contents' or ' Other'. If unsure select 'Other ' until we implement the new options.
When setting filter rules you are offered the standard Rule and Keyword options (as appeared in PANDAS 2 and in HTTrack)
In addition there are now three sets of preset filter rules offered. These can be added by selecting the preferred filter rule from the drop down list and clicking on Add Preset.
The first two presets offer first to disallow all (-*) , and then give rules to allow items required. You can use these when you want to gather a page and its images and style etc. but do not want to gather further html pages. Note that disallowing all, will not disallow the initial web page delivered from your seed url.
The third filter rule, Disallow external PDFs, can be required as the standard gather options will allow non html files to be gathered from anywhere (this is so we get all the images etc. that compose the look of a website). Having this functionality however can be problematic as it will gather unwanted (and without permission) files. We have noticed this particularly with PDF files. Therefore use this to disallow all PDFs and then allow PDFs from your target website only.
-*.pdf
+*target_site_url_here/*.pdf (here you need to overtype target_site_url_here with your target website e.g. +*nla.gov.au/*.pdf )
You can use this filter setting to also restrict other types of files, changing the .pdf to .swf for instance
-*.doc
+*target_site_url_here/*.doc
Note: Once you have selected a preset and clicked Add Preset, from the displayed set of filter rules you can now also add to, delete or alter any of the rules to suit your gather requirements.
Within this section of Gather Details you can add additional seed urls (for use when your target webpages are on more than one address).
You can also gather from password or IP authenticated restricted websites by selecting the offered tickboxes.
When adding a username and password remember to also tick the Password Required tickbox to make them operational.
Title Edit - Gather Options
It is only recommended that experienced users attempt to alter or modify gather options.
The Gather Option tab is located next to the Gather Details tab. By selecting the Edit Gather Options link on the Gather Options page you can change the default settings including gather depth and direction settings. For information on the names and usage of the different gather options please see the Httrack manual at: http://www.httrack.com/html/step9.html
You can also change the gather profile. There are two profile options:
HTTrack defaults - this is the standard set of basic defaults that come with the HTTrack gatherer
PANDAS defaults - this is the standard set of defaults used by PANDAS.
You would only need to use these defaults if you had previously edited the gather Options and wanted to revert to a clean set of default gather options.
Note : Changing the gather options or resetting profiles defaults will also clear the gather settings in the Gather Details tab (filters, additional URLS, passwords). If you do use the Gather Options you will need to reset your Gather Details options. To do this, you will need to Save your changes in the Gather Options, then edit the Gather Details page (with your filters etc.) Save those changes, and then set your schedule and Save again.
Pausing, Checking and Restarting gathers
From your Gather - Gathering Titles Worktray you can view your titles being archived. While the title is gathering you only have the option to stop or pause your gather.
However, if you choose to pause the gather (using the pause button 
) more options will appear.
With these options you can Review the files being gathered, for example to see if the gatherer is picking up excess files, and then choose to modify the Gather Options and Details as necessary - before restarting the gather by clicking on the (
) restart button.
Note before selecting to Pause your gather click on Refresh, to make sure that your title is still gathering. If the gathered instance is post-processing, which it does at the end of a gather, and you select to Pause, the gather may fail.
Note that pausing your gather will keep your title in the Gather Queue but will release the gatherer to pick up another title waiting to gather. When you have finished modifying or checking your gather and click on Restart, the title will still be in the Gather Queue but will not start actually gathering again until there is a free gatherer string. You will know when the gatherer starts re-gathering your instance as the Size and Files indicators will re-appear and begin to update.
Stopping gathers
If you wish to stop a gather, click on the red Stop button. The gather will stop gathering new files, but will continue for a short time completing the gathering of files that were already partially downloaded and then begin post-processing. After this is complete the instance will move into the Preserve worktray and can then be kept and processed or deleted.
Note as the gatherer will continue closing and post-processing the gather for a few minutes after you have clicked on the Stop button, do not try to immediately process the instance.
Deleting gathers
To delete an instance that is gathering, you will first have to Stop it. Once the instance has stopped processing it will move into your Preserved worktray. You may then open the instance in process mode and delete it. You cannot delete a gather while it is in the Gather area.
