policy and practice statement
PANDORA, Australia’s Web Archive, is a collection of Australian online publications and web sites which is being built by the National Library of Australia and ten other participants (1) . This initiative was commenced in 1996 by the National Library in recognition of the fact that an increasing volume of Australia’s documentary heritage was being published in online formats only. Given the mandate under the National Library Act, 1960 to build a comprehensive collection of Australian published materials, collecting online resources was seen as a necessary extension of the Library’s collecting responsibilities.
Cooperating with other Australian collecting and data creating agencies
The Library is committed to cooperation with other Australian agencies that collect or create online information.
Because of the large volume of online resources to be archived and preserved and the high costs associated with this activity, the Library very early established a policy of cooperation with the State libraries and other cultural collecting agencies. It set up technical infrastructure, policy and procedures for archiving online publications and web sites and invited other collecting agencies to join it in building the PANDORA Archive, which is stored centrally on a server at the National Library. It has entered into Memoranda of Understanding with participants.
The Library is working with other Commonwealth agencies which are responsible for the creation and management on online government information, GeoScience, the Australian Bureau of Statistics and the National Archives of Australia, to define responsibilities for preservation and long-term access to this data.
In the absence of suitable, affordable software available for purchase, the Library developed PANDAS in-house to support the tasks and workflows involved in the selective approach to web archiving adopted by the Library and other PANDORA participants. Once the suite of tools developed by the IIPC is available, they will be adopted and customised to meet PANDORA’s needs.
The Library has set up this infrastructure and continues to develop and maintain it without cost to other participating agencies. It intends to continue this supporting role at no cost for as long as its own resources permit.
National libraries around the world have taken a variety of approaches to selecting and archiving the web publications of their respective countries. These can be categorised as the selective (2) approach; periodic harvesting (3) of the country’s whole web domain; thematic collecting (4) ; and the establishment of deposit collections (5) .
The National Library of Australia has placed strong emphasis on the selective approach because, at this stage of technological development of web archiving, it has the following advantages:
- Each item in the Archive is quality assessed and functional to the fullest extent permitted by current technical capabilities.
- A gathering schedule can be individually tailored for each selected title, taking into account its publication schedule or the frequency with which the Web site changes, thus enabling the content gathered to be as complete as possible.
- Each item in the Archive can be fully catalogued and therefore can become part of the national bibliography, and the bibliographic data can be shared. In the Library’s own catalogue web resources are integrated with all other resources and users need look in just one place.
- Each item in the Archive can be made accessible via the Web to readers immediately, because permission to do so has been negotiated with publishers.
- The properties of individual resources and classes of resources within the Archive are known to collection managers. This enhances our ability to develop methods and tools to collect them in the first place, to store them, provide access to them, and also enhances our knowledge of what preservation strategies will be required to keep them accessible for the long term.
- Sites that are inaccessible to harvesting robots can be identified and gathered using other methods, by arrangement with the publisher. This includes commercial titles, which may require the password to be supplied by the publisher, and databases.
However, mindful that the selective approach inevitably misses important resources and takes them out of context of the web pages to which they are linked, the National Library has undertaken annual large scale harvests of the .au domain commencing in June and July 2005. The Library is investigating the legal and technical obstacles to providing access to the content of whole domain harvests.
Each agency contributing to the PANDORA Archive is required to maintain up-to-date selection guidelines indicating the portion of Australian resources published online for which it will take collecting responsibility. These guidelines are made available on the PANDORA web site .
A publication is information, regardless of its format, that is made available to the general public, or to an identified public, either free of charge or for a fee. In theory, this includes everything on every publicly available web site within the Australian web domain. In practice, agencies contributing to PANDORA archive only certain types of online information, as detailed in selection guidelines.
One category of online information specifically excluded from PANDORA is organisational records. The Archive is a collection of published resources, and materials are not included for archival record keeping purposes. For this reason it is generally not considered appropriate to archive entire government agency web sites, but to concentrate on the publications that are published through these web sites.
Titles are selected for archiving giving consideration to the appropriate point of access for the publication. If a title is selected at the whole of web site level, that will be the point of access for the catalogue record and for the title and subject listings on the PANDORA home page. In this case, subsidiary parts of the web site, such as reports or newsletters, will be hidden from access in the title and subject browse lists, although they will be discoverable through the search engine. Our selection policy in this area is a pragmatic one of selecting at a level that will maximise access to important content, while taking into account the workload involved.
The value of the content of the resource under consideration for selection is the dominant factor influencing a selection decision, even if the format or other technical considerations appear to be challenging. The Archive includes dynamic formats, and the National Library has developed a method for including information published in database (6) formats.
Access to the PANDORA Archive is freely available to anyone with an internet connection. A small proportion of titles (about two per cent) are restricted for commercial reasons. Most of these are available on a single PC in the Library’s main reading room, also free of charge.
The Library provides a number of ways to obtain access to the contents of the Archive. The home page of the PANDORA web site provides subject and titles lists for browsing. A search engine, which enables full text searching of the entire PANDORA Archive is also available through the Trove discovery service.
Commercial search engines such as Google and Yahoo! also search PANDORA to the title level. (Note that these search engines do not search the full text content.)
At this stage public access is not available to the whole domain harvests.
Catalogue records are created in the Library’s online catalogue for titles archived in PANDORA and all records are contributed to the Australian National Bibliographic Database, available through the Libraries Australia (7) service. This means that they are recorded in the national bibliography.
Cataloguing of online resources is carried out in accordance with the PANDORA Archive Electronic Cataloguing Manual .
It is a requirement of participation in PANDORA that contributing agencies create catalogue records for the titles they archive and contribute these records to Libraries Australia. These bibliographic records in MARC format are made available as a purchased product from Libraries Australia, the PANDORA e-collection.
The Library has implemented a system of persistent identification and resolution for all titles archived in PANDORA and all components of those titles. The Library undertakes to ensure that these persistent identifiers will not break, and authors, indexing and abstracting agencies, and publishers can use them with confidence knowing that they will be maintained.
Archiving is carried out according to the manuals published on the PANDORA web site. PANDORA participants use PANDAS to undertake the tasks involved. All PANDORA staff are responsible for ensuring that only resources for which permission has been received are added to the Archive, and that gathering filters and quality checking processes are used effectively to this end.
Permission to archive
Archiving in PANDORA is carried out in accordance with applicable legislation, government directives and arrangements with publishers.
Most jurisdictions covered by the PANDORA Archive do not yet have legal deposit provisions which include online publications and web sites. Only the Northern Territory has legislation (8) that unambiguously includes web resources. Three participants, the State Library of New South Wales (9) , the State Library of Victoria (10) and the State Library of Western Australia (11) have arrangements with their governments that permit them to copy titles into the Archive and to provide access to them.
A Whole-of-Government ICT Policy (AGIMO Circula 2010/001) allows the National Library to collect, preserve and make accessible Commonwealth Government online (digital) publications and websites.
Outside of these provisions and arrangements, contributors to PANDORA are obliged to seek permission from publishers to copy a title into the Archive and to provide access to it in perpetuity. This permission is sought by all participants using text supplied by the National Library and this letter and the response from the publisher forms the Copyright Licence . Correspondence with publishers is kept indefinitely in the contributing agency’s records management system.
The National Library and other PANDORA participants recognise the value to all stakeholders of a prosperous online publishing industry. They respect the rights of publishers to a return on commercial publications and are happy to agree to a period of restriction while the publication is commercially viable. The PANDAS software has been designed to enable the imposing of access restrictions. When negotiating with publishers, PANDORA participants aim to achieve the shortest period of restriction that is consistent with the publisher’s commercial interests.
The National Library is committed to working towards the amendment of Copyright Act 1968 to extend legal deposit provisions to online resources.
Quality of archived titles
As mentioned above, one of the advantages of a selective archive, such as PANDORA, is that each title can be checked for completeness and functionality before being added to the Archive. The National Library and other PANDORA participants place high priority on ensuring that titles added to the Archive are complete in terms of their content and represent, as far as technically possible, the ‘look and feel’ (appearance and functionality) of the resource as it was on the publisher’s site. In the absence of suitable software that will do the checking automatically, this involves checking the archived version file by file against the publisher’s version.
The purpose of the Archive is to provide long-term access to Australia’s heritage published on the Web. The contents of the Archive will be preserved according to the Library’s Digital Preservation Policy. As hardware and software changes and the need arises for preservation action to be taken on files in PANDORA, the Library will strive to maintain the ‘look and feel’, as well as the content, of titles. For technical and resource reasons this may not always be possible but it will aim for the ideal on a ‘best efforts’ basis.
Staff training and development
Staff who are engaged in web archiving require specific skills, an aptitude for the work, and a commitment to continual self-learning. Agencies participating in PANDORA need to support their staff engaged in archiving by providing time, funds and opportunities for required training. This is essential in a publication environment that is constantly changing and evolving.
The National Library provides a suite of manuals detailing archiving procedures and the use of the PANDAS software on the PANDORA web page. From time to time, training exercises, which address specific aspects of archiving, are also offered to the staff of participating agencies. The PANDAS Training System is made available to participants for training and experimentation purposes.
The National Library can also provide advice on selection criteria to use when recruiting staff for web archiving.
(1) The other participants are the Northern Territory Library and Information Service, the State Library of Queensland, the State Library of New South Wales, the State Library of Victoria, the State Library of South Australia, the State Library of Western Australia, the National Film and Sound Archive, the Australian War Memorial, the Australian Institute of Aboriginal and Torres Strait Islander Studies, and the National Gallery.
(6) The method involves the use of a tool called Xinq http://www.nla.gov.au/xinq/, which enables a generic interface to the content of archived databases to be set up. It was developed by the Library as part of its contribution to the Deep Web Archiving Group of the International Internet Preservation Consortium. The software is available through SourceForge http://sourceforge.net/projects/xinq.
(7) National Library of Australia. Libraries Australia. Available online at http://librariesaustralia.nla.gov.au/apps/kss
(9) New South Wales Premier’s Department. (2000) Memorandum no. 2000-15: Access to published information –laws, policy and guidelines. Available at http://www.dpc.nsw.gov.au/publications/memos_and_circulars/ministerial_memoranda/2000/m2000-15
(10) Victoria. Premiers Circular No. 2005/2
(11) Western Australia. Premier's Circular 2003/17: Requirements for Western Australian government publications and library collections