NLA Digital collections: Statement of Preservation Intent

Collection Area: Australian Government Web Collection Archive

The Library's Australian Government Web Archive consists of large collections containing a scoped snapshot of web materials mostly from the gov.au second level domain (with some material from other 2nd level .au domains) belonging to Commonwealth Government agencies. The Australian Government Web Archive collections (AGWA) were commenced in 2011 and have been undertaken once annually (within a financial year) since. As of February 2013 two collections have been completed.

The Australian Government Web Archive collections are scoped to collect the content of a specific list of seed URLs covering all FMA Act agencies and those CAC Act agencies with have agreed to comprehensive harvesting; and they represent a collection of a comprehensive range of content and formats found more generally on the .au web domain.

Web Archiving intends that:

  • All Australian Government Web Archive digital preservation masters, consisting of WARC files including all associated metadata (including indexes and CDX files) should be retained in perpetuity. All technical properties should be maintained to the full extent possible.
  • Content, connections and context are of primary importance. How it is ultimately presented to a user is a secondary consideration.

The Library understands that web archives are problematic in that:

  • The Library has no control over the creation of the original content and consequently its format, standards or quality and can only harvest what is delivered in a single published form through a browser/server request (e.g. the original data in the publishers databases are not collected);
  • Current methods for collecting and rendering are also not ideal in ensuring the complete capture of all files or retaining full functionality in the version delivered from the archive;
  • We are only taking artefactual snapshots of web content.

Therefore, the NLA accepts that what is to be preserved is not a mirror representation of the web, or even a website, but a snapshot of content that was once arranged and published as a website, with only limited functionality of the original. The archival artefact is formed out of the web collection process which is inevitably a lossy process in itself. Our concern is to define and control this loss. In addition, the way in which the content is collected and displayed may mean that there is significant limitation on the presentation of the archived artefact as an authentic record of the publishers original data or even the version of that data as originally published on the web.

Other Issues:

  • The intention is long term access for all users. However, over time access to certain content may only be available on site due to technical considerations.
  • The harvested web content being complex objects are contained in a container file (i.e. WARC file) holding random collections of files plus metadata about the container files which are managed and located by indexes (DAT and WAT files) and summaries of the contained web pages (CDX files).
  • The Australian Government Web Archive domain harvest collections can be broadly categorised as consisting of about 86% text (including 4% PDF), 10% images (JPEG, GIF, PNG) and 4% multimedia and style elements (Java script, CSS files, Flash etc.) including linkages. Style elements are problematic from the outset since they are sometimes difficult to harvest and remain often impossible to render. Because content is harvested through a web browser type request on a server, in many cases only a subset of possible style element files are delivered (those required for the browser request). Moreover, harvesters are not able to thoroughly parse complex JavaScript which may also result in the collecting process not identifying and missing many style elements (JS, XML etc).
  • Because of the size of the harvests, comprehensive visual quality assurance is not undertaken at the time of the harvest. Thus, the look of the original may only be surmised from the content collected, the context of embedded links, tags and file types and the context of technologies known to exist at the time of harvesting.
  • The size of the harvests and their coverage of the gov.au web domain collected through scoping rather than selective  individual site curation means that the nature of the contained intellectual content is not fully known or understood. Individual sites are not subject to precise harvesting rules so, while the general categorisation of formats may be similar to the PANDORA collections, the extent of lower value content may be slightly larger though it is likely to be significantly less than for the broader domain harvest collections.
  • The Librarys objective is not to misrepresent the material in any way that would compromise its legal warrant to collect, preserve and make accessible the archival content. Thus particular care in retaining the integrity of the intellectual content including embedded links and domain related image material is a priority.
  • Content has been harvested for the NLA by the Internet Archive (IA). Therefore, the IA has a copy of this content which ultimately forms a part of the Wayback Collection.
  • For each Australian Government Web Archive Domain harvest there are currently no derivatives.
  • This collection is currently in a state of transition in how it is stored, described and understood via technical metadata.

The Australian Government Web Archive Domain collections by definition (being content for the most part on the second level domain gov.au) are a subset of the larger .au Australian Web Domain Harvests and so are contextually related to content in the domain harvests. Conceptually the Australian Government Web Archive includes the content of specific scoped harvests and content obtained as part of the broader Australian domain harvest.

Version 1.0 1 March 2013

The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.

Cultural Notification

Australian Aboriginal and Torres Strait Islander people are advised that this website contains a range of material which may be considered culturally sensitive including the records of people who have passed away.

ENTER SITE