Annual report to pandora partners 2005-06
Contents
1. Participants working together
2. Growth of the Archive
3. Development of the Archive
4. Focus on users
5. Preservation
6. Legal Deposit
7. International relations
8. Promoting the Archive
9. Concluding summary
Appendix 1 PANDORA Consultative Committee - a list of representatives
PANDORA, Australia's Web Archive, is a selective archive of Australian online publications and web sites, which is built collaboratively by the National Library, all of the mainland State libraries, the Northern Territory Library, the National Film and Sound Archive, the Australian War Memorial, and the Australian Institute of Aboriginal and Torres Strait Islander Studies (AIATSIS). This is a report to contributing partners on activities and developments in the 2005-06 financial year.
1. PANDORA participants working together
1.1 Consultation mechanisms
The National Library continued to inform and consult with other PANDORA participants about the operation of PANDORA through the two email discussion lists – pandoraconsult-l and pandora-l respectively. A teleconference of PANDORA operational staff was held on 19 July.
1.2 Reports
Each month, a report on the growth of the Archive, usage statistics, and a summary of responses to the online PANDORA user survey forms are sent to both email discussion lists. This report includes information about the ten most popular sites for the month and which agency has archived them.
On a bi-monthly basis, the Library compiles two list of instances archived by each partner agency - one list contains all instances archived during the period and the other details government publications only. These lists are put up on the PANDORA web site at http://pandora.nla.gov.au/newtitles/new_titles_reports.html and partners are advised of their availability via a message to the two email discussion lists.
An annual report of progress and activities to the Chief Executive Officers of partner agencies is also provided.
1.3 Training
Taking on board comments about training that had been made by participants at the teleconference on 19 July, National Library PANDORA staff devised two sets of simplified PANDORA training notes and exercises on key aspects of web archiving, and made them available to other participants. Once again usage of the training package was disappointingly very low. The National Library is at a loss to know how to assist other participants with training and skill development when the response to our offerings is so poor.
1.4 Memorandum of Understanding
A major focus of activity this year was formalising the arrangements that participating agencies have with the National Library through a Memorandum of Understanding. The text of the MoU was drafted by the NLA and sent out to other participants for comment in July 2005. Following receipt of comments from some participants a second draft was sent out for comment in December. Agreement on the text was reached with all participants by May 2006 and the MoUs were sent to CEOs for signature. All but one have now been signed ready for the commencement on 1 July 2006.
2. Growth of the Archive
30 June 2005 | 30 June 2006 | Growth 2005-06 | |
---|---|---|---|
Titles |
8,892 |
12,297 |
3,407 (38.3%) |
Instances |
17,947 |
24,536 |
6,589 (36.7%) |
Gigabytes (2) |
884 |
1180 |
296 (33.5%) |
Usage (page views) |
5,390,459 |
8,794,547 |
3,404,088 (63%) |
The Archive showed strong growth during 2005-06, both in terms of the number of titles and instances archived, and in its usage. In November the extraordinary milestone of 10,000 titles archived, 20,000 instances archived and one terabyte of storage for the display Archive was achieved, all in one month.
Government publications comprise approximately 50 per cent of the Archive.
3. Development of the Archive
To keep pace with a rapidly changing web archiving environment the National Library is committed to ongoing development of the policy, procedures and technical infrastructure which support the Archive.
3.1 Development of PANDAS
PANDAS (the PANDORA Digital Archiving System) is web-based software developed by the Library to enable PANDORA staff in participating agencies to carry out all of the tasks involved in contributing selected online publications and web sites to PANDORA. (This does not include cataloguing, which is carried out in separate systems.)
During 2005-06 the Library continued to work intensively on the development of PANDAS 3. The original aim was to tidy up some code and improve stability and maintainability. However, we determined that it would be more effective to do a full re-engineering of the software. This has provided the opportunity to include some enhanced functionality and to redesign the user interface to make it better able to support the web archiving workflows. At the time of writing the last annual report it was expected that deployment of the first release of PANDAS 3 into production would take place in late 2005. However, it has taken much longer than expected and deployment is now planned for the second half of 2006.
3.2 Whole domain harvest
A major innovation in 2005-06 was the whole domain harvest of the Australian web domain. Despite the advantages of the selective approach to archiving, its disadvantages have long been recognised by the Library. Resources are taken out of context, and their links to other web documents are broken. In addition, important resources are missed. Government publications comprise just one category of material that we know none of the PANDORA participants can address adequately via the selective approach. There are just too many titles to be captured.
The National Library contracted the Internet Archive to undertake the whole domain harvest on our behalf. The Internet Archive has extensive experience in this form of web archiving. It provided a ‘seed list’, or list of URLs to be used as a starting point, derived from one of its own recent harvests of the World Wide Web, which had picked up approximately 300,000 Australian documents. The National Library also provided a seed list, which included the URLs of priority web sites, particularly in the government and educational sectors. The intention was to bias the harvest towards these sectors to capture as many of these resources as possible.
The harvest was run for four to five weeks during June and July 2005 and 185 million unique documents were captured, amounting to 6.69 terabytes of data. It is not known what proportion of the Australian web domain this comprises. The Internet Archive indexed the contents and shipped it to the National Library in late 2005, where it was installed for local access. PANDORA participants were provided with access to the whole domain harvest in May 2006.
A report on the first whole domain harvest is available on the PANDORA web site at http://pandora.nla.gov.au/documents/domain_harvest_report_public.pdf
In the absence of legal deposit provisions for online publications and web sites at the Commonwealth level, the access that the Library can provide to the whole domain harvest remains limited. It is not currently available to the general public. Unlike the selective Archive, we have not negotiated permission from publishers to archive and to provide access to the contents of the whole domain harvest. However, there are some categories of publications, such as the government publications of a few jurisdictions, for which permission to archive is held through premiers’ circulars or memoranda. In consultation with the relevant PANDORA participants, the Library will experiment with the possibility of providing access to these resources. The Library also has permission for an increasing number of Commonwealth government domains.
The Library is preparing for a second whole domain harvest commencing in August 2006. The intention this time is to augment the seed list with the URLs of documents that were missed last time and to harvest up to 500 million files – more than twice the size of the first harvest – to attempt to get a better understanding of the size of the Australian web domain and obtain more archival content.
3.3 Search engine
In October 2005 a customised user interface to the Lucene full text search software was installed to run on the PANDORA Archive. For the first time this gave users of PANDORA an effective search service, including advanced search options, to explore and discover resources in the Archive.
4. Focus on users
Once again this year an analysis of usage of the Archive during the previous financial year, 2004 – 2005, was undertaken. It was distributed to PANDORA participants in May 2006.
The analysis showed a continued growth in usage, with an increase of 22.5 per cent over 2003 – 2004.
Usage in 2004 - 2005
Total pageviews |
Average per month |
Month of highest use |
Month of lowest use |
5,390,459 |
449,204 |
May - 649,786 |
December – 276,983 |
Comparison with 2003 - 2004
Total pageviews |
Average per month |
Month of highest use |
Month of lowest use |
4,398,148 |
366,512 |
May – 560,168 |
December – 174,070 |
While only 4.48 per cent of the titles archived in PANDORA have completely disappeared from the live Web, 40 per cent (four) of the ten most heavily-used titles are no longer available from the publishers’ web sites. This suggests that a reasonable proportion of users are coming to the Archive for its primary purpose, which is to provide access to online publications and web sites that are no longer available elsewhere.
Responses to the PANDORA User Survey questionnaire 1 March to 30 June 2005 indicated that usage of PANDORA was spread fairly evenly across all age groups, with those in the 36 – 49 age group being the most frequent users. The primary purpose for visiting the Archive was academic or professional research, but family history was also popular (mainly thanks to First Families 2001, which is consistently the most heavily-used title). Another sizable group of users was just browsing around and came out of curiosity.
Sadly, 42.8 per cent of users did not find what they were looking for. 25 per cent did find what they were looking for, and 28.5 per cent had been ‘just browsing’ and not looking for anything in particular. It is to be hoped that for 2005 – 2006, with the installation of an effective search engine, the analysis of usage will reveal that more of our users will have success in finding what they are looking for. It is clear from comments included in the survey questionnaires that some users are ending up in PANDORA looking for information which is out of its scope.
5. Preservation
The National Library continued to monitor the range of file formats entering the PANDORA archive, maintaining an ongoing profile of its technical makeup.
Preservation work during the year focused on the metadata needed to support preservation of Web materials and a review of criteria for establishing trustworthiness in digital archives.
The National Library, as a partner in the Australian Partnership for Sustainable Repositories (APSR), led a project to develop functional requirements for implementation of the PREMIS preservation metadata recommendations within APSR partner repositories. The outcomes from the project and how they may be implemented within National Library systems, including PANDORA, will be explored in the second half of 2006.
The Library also provided feedback on a draft preservation metadata set for Web archiving, under development by the International Internet Preservation Consortium (IIPC). The proposed metadata set aims to fulfil the requirements for preservation metadata, but in the context of Web archiving practice.
Similarly, comments were provided on a draft set of criteria for certification of trusted digital repositories, which was released for public comment by the Research Libraries Group (RLG) and the US National Archives and Records Administration (NARA) in August 2005. In particular, the Library highlighted several interpretation and compliance issues for Web archives, and will continue to explore how these might be resolved.
6. Legal deposit
The Library was encouraged by contact from by the Department of Communications Information Technology and the Arts in October indicating a renewed interest in amending the legal deposit provisions in the Copyright Act 1968, to include electronic publications. DCITA intended to produce a paper on the subject for public and industry consultation, but, as of writing, this has not yet appeared.
The lack of legal deposit provisions remains one of the main hindrances to efficient selective archiving, and hinders the provision of access to the whole domain harvests. The Northern Territory Library has the advantage of legal deposit provisions for Territory publications, and the State Libraries of New South Wales, Victoria and Western Australia have negotiated agreements with their premiers’ departments for the archiving of government publications. The National Library continues to negotiate blanket permission for Commonwealth government domains through the Commonwealth Copyright Administration (CCA) and now has permission for 269 domains (205 negotiated by the CCA and 64 directly by the NLA). Clearly administration of the Archive would be more efficient if there were uniform legal deposit provisions throughout the country.
7. International relations
During 2005 – 2006 the Library continued its active participation in the International Internet Preservation Consortium, sending a representative to each of the quarterly meetings in Europe and North America. The main contribution this year was to the development of the ‘curator tool’, based on our experience in developing the PANDORA Digital Archiving System (PANDAS), which is a very similar kind of tool.
The first three-year phase of the IIPC has now come to an end and a review of its organisation and objectives has been undertaken to formulate its program for the next three years. The Library will continue to participate in the IIPC but has decided not to continue as a member of the Steering Committee because of the high cost of attending meetings.
8. Promoting the Archive
8.1 PANDORA Fact Sheet
The Library has continued to update the PANDORA Fact Sheet on a monthly basis and to distribute it to participants for publicity purposes. It summarises key information about the Archive and supplements the printed PANDORA Brochure.
8.3 Papers and articles
A number of papers and articles have been written or contributed to for the dual purpose of promoting our work and sharing what we have learned. Some key examples follow:
- Koerbin, P. Current issues in web archiving in Australia: A paper presented at the Open Publish 2005 Conference on 29 July in Sydney. Available at http://www.nla.gov.au/nla/staffpaper/2005/koerbin1.html
- Koerbin, P. Report on the crawl and harvest of the whole Australian web domain undertaken during June and July 2005. Available online at http://pandora.nla.gov.au/documents/domain_harvest_report_public.pdf
- Phillips, M. and P. Koerbin. ‘PANDORA, Australia’s Web Archive: How much metadata is enough?’ To be published in a forthcoming issue of the Journal of Internet Cataloguing.
- Phillips, M. ‘Missing online government publications in Australia: A report to the ALIA Expert Group on Government Publications’. Unpublished report.
- Watson, J. ‘Rock on! Australia’s online musical scene through the PANDORA Archive and MusicAustralia.’ Continuo, Journal of the International Association of Music Libraries, Archives and Document Centres. (IAML), Australia Branch, vol. 34, pp. 5 - 12.
9. Concluding summary
Some of the highlights of 2005 - 06 include:
- All PANDORA participants agreed on the text of a Memorandum of Understanding and by the end of the year most had been signed, ready for the commencement on 1 July 2006 (section 1.4);
- Content of the Archive grew by 36.7 per cent (section 2);
- Substantial progress was made on PANDAS version 3, which will provide greater stability, enhanced functionality and a re-designed user interface. Deployment did not take place at the end of 2005, as expected, and is now likely in August 2006 (section 3.1);
- A whole domain harvest of the .au domain was undertaken June – July 2005 and another is being planned for August – September 2006 (section 3.2);
- An effective search engine was deployed on the Archive in October 2005 (section 3.3.)
- An analysis of usage of the Archive during 2004 – 05 was undertaken and a report distributed to partners. Usage grew by 22.5 per cent over the 2003 - 04 financial year. (section 4);
- Preservation work during the year focused on the metadata needed to support preservation of Web materials (notably through involvement with APSR) and a review of criteria for establishing trustworthiness in digital archives. (section 5);
- Active participation in the International Internet Preservation Consortium continued (section 7).
Appendix 1
PANDORA Consultative Committee – list of representatives
Australian Institute of Aboriginal and Torres Strait Islander Studies - Pat Brady, Collection Manager, AIATSIS Library
Australian War Memorial - Mal Booth, Head, Research Centre
National Film and Sound Archive - Matthew Davies, Manager, Collection Development
National Library of Australia - Margaret Phillips, Director, Digital Archiving (Chair of Committee)
Northern Territory Library and Information Service - Ann Ritchie, Assistant Director
State Library of New South Wales - Jim Tindall, Senior Librarian, Collection Services
State Library of Queensland - Helen Thiselton, Manager, Publications
State Library of South Australia - Tony Leschen, Manager, Collection Development
State Library of Victoria - Liz Jesty, Manager, Collections Management
State Library of Western Australia - Brian Stewart, Manager, Battye Published Collections