NLA Digital collections: Statement of Preservation Intent

Collection Area: Australian Web Domain Harvests

The Library's Australian web domain archiving program, consists of large collections containing a broad and deep snapshot of web materials from the .au top level domain together with linked content identified through a DNS lookup on other domains (principally the .com domain) that is hosted on Australian located servers. The Australian web domain collections (AusCrawl) were commenced in 2005 and have been undertaken once annually (within a financial year) since. As of July 2011 six domain harvests have been completed.

The domain harvests are scoped to crawl broadly through the .au domain with few exclusions and thus represent a collection of the full range of content and formats found on the .au web domain.

Web Archiving intends that:

  • All Australian web domain digital preservation masters, consisting of ARC and WARC files including all associated metadata (including indexes and CDX files) should be retained in perpetuity. All technical properties should be maintained to the full extent possible.
  • Content, connections and context are of primary importance. How it is ultimately presented to a user is a secondary consideration.

The Library understands that web archives are problematic in that:

  • The Library has no control over the creation of the original content and consequently its format, standards or quality and can only harvest what is delivered in a single published form through a browser/server request (e.g. the original data in the publishers databases are not collected);
  • Current methods for collecting and rendering are also not ideal in ensuring the complete capture of all files or retaining full functionality in the version delivered from the archive;
  • We are only taking artefactual snapshots of web content.

Therefore, the NLA accepts that what is to be preserved is not a mirror representation of the web, or even a website, but a snapshot of content that was once arranged and published as a website, with only limited functionality of the original. The archival artefact is formed out of the web collection process which is inevitably a lossy process in itself. Our concern is to define and control this loss. In addition, the way in which the content is collected and displayed may mean that there is significant limitation on the presentation of the archived artefact as an authentic record of the publishers original data or even the version of that data as originally published on the web.

Other Issues:

  • The intention is long term access for all users. However, over time access to certain content may only be available on site due to technical considerations.
  • The harvested web content being complex objects are contained in a container file (i.e. ARC or WARC file) holding random collections of files plus metadata about the container files which are managed and located by indexes (DAT files) and summaries of the contained web pages (CDX files).
  • The domain harvest collections can be broadly categorised as consisting of about 75% text, 20% images (JPEG, GIF, PNG) and 5% multimedia and style elements (Java script, CSS files, Flash etc.) including linkages. Style elements are problematic from the outset since they are sometimes difficult to harvest and remain often impossible to render. Because content is harvested through a browser type request on a server in many cases only a subset of possible style element files are delivered (those required for the browser request). Moreover, harvesters are not able to thoroughly parse complex JavaScript which may also result in the collecting process not identifying and missing many style elements (JS, XML etc).
  • Because of the size of the domain harvests, visual quality assurance is not undertaken at the time of the harvest. Thus, the look of the original may only be surmised from the content collected, the context of embedded links, tags and file types and the context of technologies known to exist at the time of harvesting.
  • The size of the domain harvests and their coverage of the .au web domain collected through scoping rather than curation means that the nature of the contained intellectual content is not fully known or understood. Individual sites are not subject to precise harvesting rules so, while the general categorisation of formats may be similar to the PANDORA collections, the extent of lower value content may be larger. For example there may be more content like software mirror sites, executable files and the results of crawler traps even though some action during harvesting is taken to reduce such content.
  • The Librarys objective is not to misrepresent the material in any way that would compromise its legal warrant to collect, preserve and make accessible the archival content. Thus particular care in retaining the integrity of the intellectual content including embedded links and domain related image material is a priority.
  • Content has been harvested for the NLA by the Internet Archive (IA). Therefore, the IA has a copy of this content which is a part of the Wayback Collection.
  • For each Domain harvest there are currently no derivatives.
  • This collection is currently in a state of transition in how it is stored, described and understood via technical metadata.

Version 1.0 1 March 2013

The National Library of Australia acknowledges Australia’s First Nations Peoples – the First Australians – as the Traditional Owners and Custodians of this land and gives respect to the Elders – past and present – and through them to all Australian Aboriginal and Torres Strait Islander people.

Cultural Notification

Australian Aboriginal and Torres Strait Islander people are advised that this website contains a range of material which may be considered culturally sensitive including the records of people who have passed away.

ENTER SITE