For larger collections

If your collection is self-managed, or you have a complex system hosted by a vendor, you will need to contact us to ensure your records display how you want them to in Trove.  To contact us about contributing your records, please follow these steps:

  1. Print out the Memorandum of Understanding, sign it, scan it, and email it to us.
  2. Contact us, using the link on the top right of each page in Trove. Attach the Memorandum of Understanding form.
  3. We'll work with you to make the best of your records in Trove, and when we're all happy, they will be added to Trove.

Once your collections are in Trove, if you change your records they should automatically update in Trove. How regularly the information is updated depends on how your records are organised. It’s best to think about how often your records are updated, and organise the harvesting schedule to match. We recommend at least once a week. If you contact us we can also send you an automated email each time we update your records in Trove. 

Technical guidelines

The follow sections, along with the Data Dictionary, provide technical information for building and maintaining your records collection in Trove. Please contact us with any other questions. 

Setup Process

The process for beginning contribution to Trove has four stages:

  1. Memorandum of UnderstandingThe contributing organisations email the MoU to the Trove team, and provide a link to their organisation’s repository. 
  2. Test harvesting. The Trove team will initiate a test harvest of the contributing organisation’s repository. If this is successful the records will be loaded into the Trove test system.
  3. Test Harvest review. Both the contributing organisation and the Trove team have the opportunity to make any necessary changes. This can involve identifying missing data, changing the mapping of fields, finding broken URLs etc. Re-harvests of the data are often undertaken during this stage.
  4. Production Load. Once the contributing organisation and the Trove team are happy the records are displaying correctly they are ready to be loaded into the live Trove system. This will be done once the Memorandom of Understanding (MoU) has been received.

Transfer Protocols

Trove can work with a variety of transfer protocols to harvest records from contributors’ systems. These include Sitemaps, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and custom APIs.

While we can work with other setups, if you are building a new system with the intent of working with Trove, we recommend building a Sitemap. It has many benefits outside of Trove, making your website more discoverable to search engines.

In the past we have also worked with Harvest Control Lists and RSS feeds. We no longer recommend these for new collections. Instead, consider using a Sitemap or OAI-PMH.

 

Sitemap

A Sitemap is a standard XML protocol. It lists all the web pages within a website. Search services like Google, Yahoo and Bing use a Sitemap as a guide around a website, to help them find and visit every page. They use this data later to point users towards the page where their search term exists.

Trove uses Sitemaps in a similar way, as a list that points to the different web pages where records are available. Every URL in the list is visited, and Trove will attempt to ingest each page as a record. Pages not mentioned in the Sitemap are ignored.

Many popular web hosting platforms such as WordPress and Drupal include Sitemaps functionality by default, or as an optional plugin.

To work with Trove’s web crawler a website needs to have three layers of files:

  • robots.txt - A plain text file that instructs webcrawlers in how they should move around your website. It can contain an optional line 'Sitemap: ' followed by the URL of a Sitemap file, for example Sitemap: http://www.sitemaphost.com/sitemap-host1.xml To contribute to Trove you will need to include this line.
  • Sitemap or Sitemap Index file – a list of URLs. These resolve either to more Sitemap files, or to web pages that can be treated by Trove as single record pages.
  • Single record pages – a single HTML or XML file that will represent a single record in Trove. This might be a single episode of a podcast, a single artwork, a single journal article. We recommend that this be the same page any human visitor would see, with some extra hidden data underneath for services like Trove, as well as the big search engines. This extra data is hidden in <meta> tags, see the HTML record section for more information.

Trove can read both Sitemap and Sitemap index files. The ‘lastmod’, ‘changefreq’, and ‘priority’ elements are currently ignored.

Sitemap websites can be crawled on a daily, weekly, monthly, bi-monthly, quarterly, half-yearly or yearly schedule. Nominate the frequency that suits you, considering how often your content is added or changed.

Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)

OAI-PMH is a web-based protocol that defines a standard way to move batches of records between systems. An OAI http request is sent from the service provider, Trove, to a data provider, a repository. A batch of 100 records is sent back in XML. A URL to access the next batch of records (called a resumption token) should be available after the last record.

As each batch is received the records are immediately loaded into Trove. The following week an update harvest is conducted, and only records added, updated or deleted in the intervening time are returned. Trove refreshes accordingly.

A number of popular library systems, institutional repositories, archival management platforms and other record management systems have in-built OAI-PMH functionality. For others, an optional plugin may exist. This means you may already have the technical capability to contributing records to Trove via OAI-PMH. 

The standard for OAI-PMH can be found at http://www.openarchives.org/pmh/

To interact with a working OAI-PMH repository see the Library of Congress Sample OAI-PMH requests

A good overview of the OAI-PMH process is available at the CAIRSS website.

Custom API

Trove can be configured to work with your organisation’s custom API.  We’re best placed to work with APIs that

  • Are RESTful.
  • Offer XML or JSON output
  • Output Dublin Core or similar standard metadata
  • Include notifications of records that have been deleted
  • Can limit the number of records returned per request to less than 100, and can paginate through results
  • Can provide incremental harvests (“show me all records updated  since 2014-06-18")
  • Provide all metadata fields in the search results, without a second call for the full individual record

None of these are hard and fast rules, they do make setup and maintenance quicker at the Trove end and minimise the number of calls to your API.

To integrate with any custom API requires specialised work by the Trove team. The process of interoperating with a custom API can therefore take longer to setup than other standardised harvesting methods. Please contact us to discuss how Trove can work with your API.

Harvest Control List

A Harvest Control List is no longer recommended as a method for contributing new collections to Trove. It was a single HTML page containing a list of URL hyperlinks. It was used by Trove’s predecessor services to do basic web harvesting.

The Trove Harvester would go down the list, accessing every URL. Each URL led to an HTML or XML file representing a single record. If the file was HTML it would contain meta tags describing the relevant fields.

Harvest Control Lists were publically available on the internet and stored at a URL accessible to the Trove Harvester. It was requested that they be updated regularly, preferably automatically, to ensure records in Trove remained current. Many collections grew to long lists of links, in the tens of thousands, which Trove visited each and every time. This resulted in long and sometimes stalled harvests.

Sitemaps are a more modern version of a Harvest Control List and are recommended for both small and large collections.

Here is a Harvest Control List with links to three separate records named 24597933.xml, 9186335.xml and 3995299.xml.

Harvest Control List
<h1>Local History Photograph collection</h1>
<ol>
<li><a href="24597933.xml">24597933</a></li>
<li><a href="9186335.xml">9186335</a></li>
<li><a href="3995299.xml">3995299</a></li>
</ol>

Record formats

Trove specifies the standard data elements it expects from all contributors’ collections. This is called a schema. While the transfer protocols provide a way to move the batches of records into Trove, the schema specifies what kinds of information Trove should expect for each record.

Items do not have to be fully available online for free (e.g. a digitised picture or a full thesis). It may be a link to a finding aid describing the box an item is stored in, or an abstract for an article.

Published and unpublished material

Trove currently works with a variety of metadata schemas. The majority of published and unpublished material is described using Dublin Core.  Some DC Terms and NISO Access and Licence Indicators (NISO RP-22-2015) fields are also accepted.

Each record contributed to Trove must include these three mandatory fields:

  • Identifier (including a linktype)
  • title
  • type

It is highly recommended that these six fields are included:

  • contributor
  • creator
  • date
  • description
  • subject
  • rights
  • free_to_read
  • license_ref

If known, it is best practice to also include the following:

  • coverage
  • format
  • language
  • publisher
  • relation
  • source

Records should be presented in XML, or HTML with meta tags to store the separate fields. Your records will be aggregated into Trove much faster if your ‘identifier’ tag contains specific labelled information. For example:

<dc:identifier>http://nla.gov.au/nla.pic-an9186335-t</dc:identifier> does not let us know what the link actually contains, and time is spent to work out if it is the actual content, a thumbnail, or something else.

<dc:identifier linktype="thumbnail">http://nla.gov.au/nla.pic-an9186335-t</dc:identifier> lets us know. Read more in the data dictionary.

MARC standard records are accepted via the service Libraries Australia.

People and organisations

Records describing people and organisations are stored within Trove in Encoded Archival Context - Corporate Bodies, Persons, and Families (EAC-CPF) and Registry Interchange Format - Collections and Services (RIF-CS).

Each record contributed to Trove must include these five mandatory fields:

  • agencyCode (ISIL or NUC Symbol)
  • fromDate
  • nameEntry (forename)
  • toDate
  • recordId

It is highly recommended that these four fields are included:

  • agencyName
  • entityId
  • localDescription (nationality)
  • nameEntry (surname)

If known, it is best practice to also include the following:

  • biogHist
  • cpfRelation
  • identifier (thumbnail)
  • localDescription (gender)
  • occupation
  • otherRecordId

These records were previously accepted as part of the People Australia Project.

XML record

An XML record can contain both Dublin Core and Picture Australia elements for images, or the extended form of Dublin Core that Trove implements. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<?xml version="1.0" encoding="UTF-8" ?>
<record><oai_dc:dc xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://purl.org/dc/elements/1.1/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
<dc:title>Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /</dc:title>
<dc:creator>Scriven, Edward, 1775-1841.</dc:creator>
<dc:contributor>Dance, Nathaniel, 1735-1811.</dc:contributor>
<dc:contributor>Society for the Diffusion of Useful Knowledge (Great Britain)</dc:contributor>
<dc:coverage>1840 - 1849</dc:coverage>
<dc:description>"Under the superintendance of the Society for the Diffusion of Useful Knowledge."; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582.</dc:description>
<dc:format>1 print : engraving ; plate mark 28 x 20 cm.</dc:format>
<dc:identifier>nla.pic-an9186335</dc:identifier>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the <a href="http://www.nla.gov.au/digicoll/copyright.html">National Library of Australia</a> to request permission.</dc:rights>
<dc:publisher>London : Published by Charles Knight, [184-?]</dc:publisher>
<dc:subject>Cook, James, 1728-1779 -- Portraits.</dc:subject>
<dc:subject>Explorers -- Great Britain -- Portraits.</dc:subject>
<dc:type>Image</dc:type>
<dc:identifier linktype="thumbnail">http://nla.gov.au/nla.pic-an9186335-t</dc:identifier>
<dc:identifier linktype="fulltext">http://nla.gov.au/nla.pic-an9186335</dc:identifier>
</oai_dc:dc>

Records for items without thumbnails do not require the extra Picture Australia elements, however it is helpful to include the linktype attribute. This indicates to Trove if the item is fully online. For example this record for a digitised text contains only Dublin Core elements.

<?xml version="1.0" encoding="UTF-8" ?>
<record xmlns:dc="http://purl.org/dc/elements/1.1/"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.openarchives.org/OAI/2.0/oai_dc/ http://www.openarchives.org/OAI/2.0/oai_dc.xsd">
<dc:title>Some thoughts on examinations [electronic resource]/</dc:title>
<dc:creator>Examiner.</dc:creator>
<dc:publisher>[Oxford] : Privately printed, 1906.</dc:publisher>
<dc:format>11 p. ; 22cm.</dc:format>
<dc:coverage>1906</dc:coverage>
<dc:date>1906</dc:date>
<dc:description>Electronic reproduction. Canberra, A.C.T. National Library of Australia,2010.</dc:description>
<dc:contributor>Examiner.</dc:contributor>
<dc:type>Text</dc:type>
<dc:identifier linktype="fulltext">http://nla.gov.au/nla.gen-vn4807078</dc:identifier>
<dc:identifier>nla.gen-vn4807078</dc:identifier>
<dc:source>Item held by National Library of Australia</dc:source>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission.</dc:rights>
</record>

HTML record

A record in HTML needs to include the meta element within the head section of the file. Other information may be contained in the body, allowing meta tags to be embedded in existing pages. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.

<html>
<head profile="http://dublincore.org/documents/dc-html/">
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" />
<link rel="schema.PA" href="http://www.pictureaustralia.org/schemas/pa/" />
<meta name="DC.title" content="Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /" />
<meta name="DC.creator" content="Scriven, Edward, 1775-1841." />
<meta name="DC.contributor" content="Dance, Nathaniel, 1735-1811." />
<meta name="DC.contributor" content="Society for the Diffusion of Useful Knowledge (Great Britain)" />
<meta name="DC.coverage" content="1840 - 1849" />
<meta name="DC.description" content="&quot;Under the superintendance of the Society for the Diffusion of Useful Knowledge.&quot;; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582." />
<meta name="DC.format" content="1 print : engraving ; plate mark 28 x 20 cm." />
<meta name="DC.identifier" content="nla.pic-an9186335" />
<meta name="DC.rights" content="You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission." />
<meta name="DC.publisher" content="London : Published by Charles Knight, [184-?]" />
<meta name="DC.subject" content="Cook, James, 1728-1779 -- Portraits."
<meta name="DC.subject" content="Explorers -- Great Britain -- Portraits."
<meta name="DC.type" content="Image" />
<meta name="PA.thumbnail" content="http://nla.gov.au/nla.pic-an9186335-t" />
<meta name="PA.viewcopy" content="http://nla.gov.au/nla.pic-an9186335-v" />
<meta name="PA.location" content="http://nla.gov.au/nla.pic-an9186335" />
</head>
<body>
</body>
</html

NUC symbol

Records in Trove are linked to a National Union Catalogue (NUC) Symbol. This is a brief collection identifier made up of 2-10 letters. NUC Symbols are issued by the National Library of Australia and maintained in the InterLibrary Resource Sharing Directory. Every NUC Symbol is unique and allows Trove to identify the organisation holding the item. It can also identify a sub-collection. A NUC Symbol is added to each incoming record by the Trove harvester.

Organisations are welcome to nominate their pre-existing NUC Symbol or request a new one for a particular collection. They can be obtained by filling in the New Service Notification Form at the ILRS Directory, or the Trove team will set one up on your behalf when you contribute to a new collection.