Technical guide for content partners

Setup Process

The process for beginning contribution to Trove has three stages:

  1. Test harvesting – The Trove team will initiate a test harvest of the contributing organisation’s repository. If this is successful the records will be loaded into the Trove test system.
  2. Test Harvest review – Both the contributing organisation and the Trove team have the opportunity to make any necessary changes. This can involve identifying missing data, changing the mapping of fields, finding broken URLs etc. Re-harvests of the data are often undertaken during this stage.
  3. Production Load – Once the contributing organisation and the Trove team are happy the records are displaying correctly they are ready to be loaded into the live Trove system. This will be done once the Memorandom of Understanding (MoU) has been received.

Transfer Protocols

Trove uses a number of transfer protocols to harvest records from contributors’ systems. These include the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH), Harvest Control Lists, RSS feeds, Sitemaps and custom APIs.

Both interactive and static OAI-PMH as well as the Harvest Control List methods are discussed below. For records available via RSS feed or Sitemap, the same metadata elements will need to be provided, however the structure of the XML document may vary. Please contact us for further information.

This graphic illustrates how records of various formats come into Trove

 

 

Trove harvesting diagram

Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)

OAI-PMH is a web-based protocol that defines a standard way to move records between systems. It allows the Trove harvester to request records from a system in batches. An OAI http request is sent to a repository and 100 records are returned in XML format. A URL to access the next batch of 100 records should be available after the last record – this is called a resumption token.

Once the data is harvested, it is loaded into Trove. The following week an update harvest is conducted; only records added, updated or deleted in the intervening time are returned. Trove refreshes accordingly.

A number of popular library systems and other record management systems have in-built OAI-PMH functionality. This means you may already have the technical capability to contributing records to Trove. 

The standard for OAI-PMH can be found at http://www.openarchives.org/pmh/

To interact with a working OAI-PMH repository see the Library of Congress Sample OAI-PMH requests

A good overview of the OAI-PMH process is available at the CAIRSS website.

Harvest Control List

A Harvest Control List is a single HTML page containing a list of URL hyperlinks. The Trove Harvester goes down the list, visiting every URL on the page. Each URL should lead to an HTML or XML file with information describing a single record. If the files are HTML they need to contain meta tags describing the relevant fields.

The Harvest Control List must be published on the internet and stored at a URL accessible to the Trove Harvester. It should be updated regularly, preferably automatically, to ensure your records in Trove remain current. OAI-PMH is preferable to a Harvest Control List.

Here is a Harvest Control List with links to three separate records named 24597933.xml, 9186335.xml and 3995299.xml

Harvest Control List
<h1>Local History Photograph collection</h1>
<ol>
<li><a href="24597933.xml">24597933</a></li>
<li><a href="9186335.xml">9186335</a></li>
<li><a href="3995299.xml">3995299</a></li>
</ol>

Custom API

Trove now works with a number of Custom APIs. We’re best placed to interoperate with APIs that

  • Offer XML output
  • Output Dublin Core or similar standard metadata
  • Can limit the number of records returned per request to less than 100, and can paginate through results
  • Can provide incremental harvests (“show me all records updated  since 2014-06-18")
  • Provide all metadata fields in the search results, without a second call for the full individual record

None of these are hard and fast rules, they do make setup and maintenance quicker at the Trove end and minimise the number of calls to your API.

To integrate with any custom API requires specialised work by the Trove team. The process of interoperating with a custom API can therefore take longer to setup than other standardised harvesting methods. Please contact us to discuss how Trove can work with your API.

Sitemap

Overview

A sitemap is a list of all the web pages within a website. Search services like Google, Yahoo and Bing use a Sitemap as a guide around a website, to help them find and visit every page. They use this data later to point users towards the page where their search term exists.

Trove uses Sitemaps in a similar way, as a list that points to the different web pages where records are available. Every URL in the list will be visited, and Trove will attempt to ingest each page as a record. Pages not mentioned in the Sitemap are ignored.

We encourage contributing organisations to use Sitemaps as they help get your records into Trove and beyond, making your website more discoverable to the big search engines.

Technical

A Sitemap is a standard XML protocol documented at http://www.sitemaps.org/. It allows a web crawler to navigate through three layers of files
robots.txt – A plain text file that instructs webcrawlers in how they should move around your website. It can contain an optional line 'Sitemap: ' followed by the URL of a Sitemap file, for example
Sitemap: http://www.sitemaphost.com/sitemap-host1.xml
This is the line that Trove reads
Sitemap or Sitemap Index file – a list of URLs. These either resolve to more Sitemap files, or web pages that can be treated by Trove as single record pages.
Single record pages – a single HTML file that will represent a single record in Trove. This might be a single episode of a podcast, a single artwork, a single journal article. We recommend that this be the same page any human visitor would see, with some extra hidden data underneath for services like Trove, as well as the big search engines. This extra data is hidden in <meta> tags, see the HTML record section of this technical overview for more information.

Trove can read both Sitemap and Sitemap index files. The following elements are currently ignored
lastmod
changefreq
priority

Sitemap websites can be crawled on a daily, weekly, monthly, bi-monthly, quarterly, half-yearly or yearly schedule. Nominate the frequency that suits you, we recommend considering how often your content is added or changed.

Record formats

Trove specifies standard data elements it expects from all contributors’ collections. This is called a schema. While the transfer protocols provide a way to move the batches of records into Trove, the schema specifies what kinds of information Trove should expect for each record.

Trove currently works with a variety of metadata schemas, including Unqualified Dublin Core , Encoded Archival Context - Corporate Bodies, Persons, and Families, and Registry Interchange Format - Collections and Services.

Unqualified Dublin Core specifies fifteen fields of data:

  • contributor
  • coverage
  • creator
  • date
  • description
  • format
  • identifier (mandatory)
  • language
  • publisher
  • relation
  • rights
  • source
  • subject
  • title (mandatory)
  • type (mandatory)

Trove accepts and displays all of these fields.

Items do not have to be fully available online for free (eg. a digitised picture or a full thesis). It may be a link to a finding aid describing the box an item is stored in, or an abstract for an article.

Records should be presented in XML, or HTML with meta tags to store the separate fields.

XML record

An XML record can contain both Dublin Core and Picture Australia elements for images, or the extended form of Dublin Core that Trove implements. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<?xml version="1.0" encoding="UTF-8" ?>
<oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://purl.org/dc/elements/1.1/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
<dc:title>Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /</dc:title>
<dc:creator>Scriven, Edward, 1775-1841.</dc:creator>
<dc:contributor>Dance, Nathaniel, 1735-1811.</dc:contributor>
<dc:contributor>Society for the Diffusion of Useful Knowledge (Great Britain)</dc:contributor>
<dc:coverage>1840 - 1849</dc:coverage>
<dc:description>"Under the superintendance of the Society for the Diffusion of Useful Knowledge."; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582.</dc:description>
<dc:format>1 print : engraving ; plate mark 28 x 20 cm.</dc:format>
<dc:identifier>nla.pic-an9186335</dc:identifier>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the <a href="http://www.nla.gov.au/digicoll/copyright.html">National Library of Australia</a> to request permission.</dc:rights>
<dc:publisher>London : Published by Charles Knight, [184-?]</dc:publisher>
<dc:subject>Cook, James, 1728-1779 -- Portraits.</dc:subject>
<dc:subject>Explorers -- Great Britain -- Portraits.</dc:subject>
<dc:type>Image</dc:type>
<dc:identifier linktype="thumbnail">http://nla.gov.au/nla.pic-an9186335-t</dc:identifier>
<dc:identifier linktype="fulltext">http://nla.gov.au/nla.pic-an9186335</dc:identifier>
</oai_dc:dc>

Records for items without thumbnails do not require the extra Picture Australia elements, however it is helpful to include the linktype attribute. This indicates to Trove if the item is fully online. For example this record for a digitised text contains only Dublin Core elements.

<?xml version="1.0" encoding="UTF-8" ?>
<record xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
<dc:title>Some thoughts on examinations [electronic resource]/</dc:title>
<dc:creator>Examiner.</dc:creator>
<dc:publisher>[Oxford] : Privately printed, 1906.</dc:publisher>
<dc:format>11 p. ; 22cm.</dc:format>
<dc:coverage>1906</dc:coverage>
<dc:date>1906</dc:date>
<dc:description>Electronic reproduction. Canberra, A.C.T. National Library of Australia,2010.</dc:description>
<dc:contributor>Examiner.</dc:contributor>
<dc:type>Text</dc:type>
<dc:identifier linktype="fulltext">http://nla.gov.au/nla.gen-vn4807078</dc:identifier>
<dc:identifier>nla.gen-vn4807078</dc:identifier>
<dc:source>Item held by National Library of Australia</dc:source>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission.</dc:rights>
</record>

HTML record

A record in HTML needs to include the meta element within the head section of the file. Other information may be contained in the body, allowing meta tags to be embedded in existing pages. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<html>
<head profile="http://dublincore.org/documents/dc-html/">
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" />
<link rel="schema.PA" href="http://www.pictureaustralia.org/schemas/pa/" />
<meta name="DC.title" content="Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /" />
<meta name="DC.creator" content="Scriven, Edward, 1775-1841." />
<meta name="DC.contributor" content="Dance, Nathaniel, 1735-1811." />
<meta name="DC.contributor" content="Society for the Diffusion of Useful Knowledge (Great Britain)" />
<meta name="DC.coverage" content="1840 - 1849" />
<meta name="DC.description" content="&quot;Under the superintendance of the Society for the Diffusion of Useful Knowledge.&quot;; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582." />
<meta name="DC.format" content="1 print : engraving ; plate mark 28 x 20 cm." />
<meta name="DC.identifier" content="nla.pic-an9186335" />
<meta name="DC.rights" content="You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission." />
<meta name="DC.publisher" content="London : Published by Charles Knight, [184-?]" />
<meta name="DC.subject" content="Cook, James, 1728-1779 -- Portraits."
<meta name="DC.subject" content="Explorers -- Great Britain -- Portraits."
<meta name="DC.type" content="Image" />
<meta name="PA.thumbnail" content="http://nla.gov.au/nla.pic-an9186335-t" />
<meta name="PA.viewcopy" content="http://nla.gov.au/nla.pic-an9186335-v" />
<meta name="PA.location" content="http://nla.gov.au/nla.pic-an9186335" />
</head>
<body>
</body>
</html

NUC symbol

Records in Trove are linked to a National Union Catalogue (NUC) Symbol. This is a brief collection identifier made up of 2-10 letters. NUC Symbols are issued by the National Library of Australia and maintained in the InterLibrary Resource Sharing Directory. Every NUC Symbol is unique and allows Trove to identify the organisation holding the item. A NUC Symbol is added to each incoming record by the Trove harvester.

Organisations are welcome to nominate their pre-existing NUC Symbol or request a new one for a particular collection. They can be obtained by filling in the New Service Notification Form at the ILRS Directory.