Processing Gathered Instances for Quality Assurance
When a resource has been downloaded to PANDAS, the gathered instance will appear in the Preserve - Gathered Instances Worktray. Click on the Process action button provided in the worktray, it will take you directly to the Process Instances screen where you can view the gather details of the specific gathered instance, for example, the size of the gather, gather time, files gathered (including the number of folders) and download rate. You can also view the gathered instance by clicking on the Archive URL link.
Alternatively, you can click on the Process Instances link from the View Title screen. It will take you to the Unarchived Instances screen first where you can view all the gathered instances of the title waiting to be processed. You can choose to process a specific instance by clicking on the Process button provided at the end of each instance.
Process Instances - Process
There are three main URL options available in the Process Instances - Process tab screen.
1. Seed URL:
This is the URL which links to the publisher’s live site. It is also the URL the gatherer used to begin the downloading process.
2. Archive URL:
Click on the hypertext link in the words Archive URL (note that in Firefox it is not an obvious link). The link is to the full gathered URL. This is the link that will be used on the title entry page for the specific instance. You can view the gathered instance in a new browser window and check that the site is complete and functional.
To view the file directories:
This link is to the created instance. It will take you to the file directory screen where you can view (but not edit) all the files gathered without having to use Web Dav.
To specify a starting page:
Occasionally, there may not be the correct start page generated by PANDAS or you may have used the Upload option. You will therefore need to identify the start page you want and enter the location (URL) in the box. To do this overwrite any text that is in the box with your chosen file name string, remembering that you need only put in the gathered url string (i.e. if your start page is http://pandas.nla.gov.au/view/3000/20081002-0000/www.nla.gov.au/grants/haroldwhite/index.html you will only enter www.nla.gov.au/grants/haroldwhite/index.html).
Then press the Save button. Return to the Archive URL link above which should have been updated and will now direct to your newly specified starting page. Remember to always test that the changed url works before clicking on the Archive button.
3. Web Dav URL: This is the URL to be used to access the WebDAV protocol that allows you to add, delete or edit any files on the remote server where the downloaded instances is placed for QA checking (see the FTP access protocol section below to find out how to access files through FTP when WebDAV does not work). Copy the url and go to My Network Places in your Windows Explorer, create a new network place and paste the URL when prompted and you will have access to the PANDAS instance you have created.
Using Windows Explorer, you can perform tasks such as:
- Adding a file/files to an Uploaded instance
- Copying and pasting missing files from a local drive (i.e. download missing files to a local drive first) to the appropriate directory in the gathered copy (create the directory if necessary).
- Deleting files from the gathered copy.
- Copying files from the gathered copy to your desktop, editing the files and copying back to the gathered copy.
Post Processing: This button gives you the opportunity to send an instance for post processing. This processing is what already happens to all titles that are gathered normally, part of its actions are to rewrite the links to external sites, forms and emails. If you have had to add extra pages to an instance, or have used the Upload facility this processing will not have taken place and so this button will need to be used.
Post Processing is only mandatory for Upload instances - when processing an Upload instance the Archive button will be hidden until post processing has been completed.
You do not need to Post Process a title if it has been gathered using the standard PANDAS HTTrack gather method.
Note that if you gather files from HTTrack on your desktop for later Uploading you will need to make adjustments to your HTTrack options. The option No External Pages will need to be ticked so that the post processing script will function once the title is processed. See the Gathering page for further advice.
When you click on the Post Processing button, your page will return to your Home Worktrays page and the title will re-enter the Gather queue. Once the processing is complete, the title will return to your Preserve Worktray and you can continue your quality assurance work.
Process Instances - Problems
There are times when you come across broken images and links in the gathered instance during the QA process. These problems might require assistance from the IT support staff at the National Library. Problems which are reported via PANDAS are normally of a high technical nature, which include Javascript problems and global changes. For simple problems, please refer to the FAQ's document at: http://pandora.nla.gov.au/manual/pandas/faq.html
In PANDAS 3, reported problems are stored and tracked in an issues tracking system called JIRA. This system keeps track of each single problem of a title instance reported by PANDAS users. You are encouraged to add one issue of the problem at a time.
To add a technical problem on PANDAS:
- Click on the Problems tab from the Process Instances screen of the gathered instance.
- Click on the Add New Problem button.
- Describe the problem by selecting the scope of the problem, indicating whether it is a recurring problem, giving the problem a meaningful name (this a worthwhile step as this is a searchable field) and providing a detailed description of the problem with links to examples of the problem.
- Click on the Update button to complete. The details of the problem will be shown in a table.
- Click on Save and the problem will appear in the JIRA issues tracking system which is available to the IT Support and PANDAS administrators.
Once the first QA problem is reported and saved, the instance will move into the Instance Referred to IT Worktray. You should not be able to access the Web Dav instance and the Find & Replace tab. This will provide the IT Support and PANDAS administrator exclusive access to the instance in fixing the problem without interference from the title owner.
You can Edit a problem report at any time, by clicking on the Edit button which appears in the Process Instance - QA Problems page. You can access this page by searching in the main Search - Problems tab or from the Problems tab of the Process Instances screen of the gathered instance
You can Withdraw a problem report at any time, by clicking on the Withdraw button which appears in the Process Instance - QA Problems page. You can access this page by searching in the main Search - Problems tab or from the Problems tab of the Process Instances screen of the gathered instance
If you have another problem to report, repeat steps 1 to 5.
Process Instances: Find & Replace
You can use the Find & Replace tab to fix problems without having to use Web Dav or enlist the help of IT Support. You can find and replace text in a specific file, directory or across all directories.
To Find and Replace text in a specific directory:
- In the Find field box, enter the text which needs to be edited or replaced.
- The filename field box is marked with *.htm* by default. This enables a global search/change. The wildcard * is to find and replace text in all files with .htm or .html extensions in the whole site
- The recursive box is checked by default. This means search and changes will be applied to all sub-directories.
- Click on Generate Report and a report will be sent to your email box.
- Check the report detailing the number of occurrences of the text specified in the Find field box.
- If the report is acceptable to you, you can enter the replacement text in the Replace with field box.
- Click on Replace All to undertake a global change.
To Find and Replace text in a specific directory:
- To change files in a particular directory you use the Directory field box,
- You will need to enter the directory name in the following formats. For example if you want to change the files in the directory /chg/ enter:
/www.nla.gov.au/chg/
however if you do not have www at the start of your string as in era.nla.gov.au/about/, you will need to enter the string in this form adding a . at the start of the string
./era.nla.gov.au/about/
- Make sure that the Find & Replace bullet is marked by default.
- In the Find field box, enter the text which needs to be edited or replaced
- Click on Generate Report and a report will be sent to your email box.
- Check the report detailing the number of occurrences of the text specified in the Find field box.
- If the report is acceptable to you, you can enter the replacement text in the Replace with field box.
- Click on Replace All to undertake the change.
- You will be sent another report detailing the changes made.
To find and replace text in a specific file:
- To change a singe file enter the file string In the Filename field box. This is entered in the following format:
era.nla.gov.au/about/AnswerstoFAQs.html or www.nla.gov.au/about/AnswerstoFAQs.html
You do not need to add a . or / to the start or end of the file.
- Make sure that the Find & Replace bullet is marked by default.
- In the Find field box, enter the text which needs to be edited or replaced
- Click on Generate Report and a report will be sent to your email box.
- Check the report detailing the number of occurrences of the text specified in the Find field box.
- If the report is acceptable to you, you can enter the replacement text in the Replace with field box.
- Click on Replace All to undertake the change.
- You will be sent another report detailing the changes made.
Note that you can change links within pages, for instance converting all .pdf links to .html links. However, this will only change the extensions for links within documents, the Find and Replace function cannot change actual file type extensions. If you want to change file type extensions, you would need to export the files using Webdav and do it manually or through available proprietary applications.
Regexp stands for Regular Expression which is a programming term. Don't use this field unless you are familiar with programming principles.
Using the FTP protocol to access instances for QA checking
The PANDAS interface under the Process Instance view provides the URL for opening up networked access to the downloaded instance files using the WebDAV protocol. When the downloaded instance is large or has many directories - or sometimes because of other issues such as your local firewall settings - you may not be able to successfully open the WebDAV access. If this is the case you can use an alternative protocol, FTP, to open up access to the remote files. To do this follow the instructions below.
- Right click on the Start button at the bottom left of your screen.
- Click on "Explore" (or, just open Windows Explorer).
- Copy the the FTP URL into the location bar . This FTP URL is in the form below. You need to change USERNAME to your pandas user name and replace the /PI/TIMESTAMP bit with the PI and date-time directories of the instance you wish to access.
ftp://USERNAME@pandas-ftp.nla.gov.au/working/PI/TIMESTAMP - You should now be prompted for a password. Use your normal Pandas password.
- Now you should have access to the files, to add, copy to desktop and edit etc in much the same way as you would have used WebDAV.