The process for beginning contribution to Trove has four stages:
- Memorandum of Understanding. The contributing organisations email the MoU to the Trove team, and provide a link to their organisation’s repository.
- Test harvesting. The Trove team will initiate a test harvest of the contributing organisation’s repository. If this is successful the records will be loaded into the Trove test system.
- Test Harvest review. Both the contributing organisation and the Trove team have the opportunity to make any necessary changes. This can involve identifying missing data, changing the mapping of fields, finding broken URLs etc. Re-harvests of the data are often undertaken during this stage.
- Production Load. Once the contributing organisation and the Trove team are happy the records are displaying correctly they are ready to be loaded into the live Trove system. This will be done once the Memorandom of Understanding (MoU) has been received.
Trove can work with a variety of transfer protocols to harvest records from contributors’ systems. These include Sitemaps, the Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH) and custom APIs.
While we can work with other setups, if you are building a new system with the intent of working with Trove, we recommend building a Sitemap. It has many benefits outside of Trove, making your website more discoverable to search engines.
In the past we have also worked with Harvest Control Lists and RSS feeds. We no longer recommend these for new collections. Instead, consider using a Sitemap or OAI-PMH.
Open Archives Initiative Protocol for Metadata Harvesting (OAI-PMH)
OAI-PMH is a web-based protocol that defines a standard way to move batches of records between systems. An OAI http request is sent from the service provider, Trove, to a data provider, a repository. A batch of 100 records is sent back in XML. A URL to access the next batch of records (called a resumption token) should be available after the last record.
As each batch is received the records are immediately loaded into Trove. The following week an update harvest is conducted, and only records added, updated or deleted in the intervening time are returned. Trove refreshes accordingly.
A number of popular library systems, institutional repositories, archival management platforms and other record management systems have in-built OAI-PMH functionality. For others, an optional plugin may exist. This means you may already have the technical capability to contributing records to Trove via OAI-PMH.
The standard for OAI-PMH can be found at http://www.openarchives.org/pmh/
To interact with a working OAI-PMH repository see the Library of Congress Sample OAI-PMH requests
A good overview of the OAI-PMH process is available at the CAIRSS website.
Harvest Control List
A Harvest Control List is no longer recommended as a method for contributing new collections to Trove. It was a single HTML page containing a list of URL hyperlinks. It was used by Trove’s predecessor services to do basic web harvesting.
The Trove Harvester would go down the list, accessing every URL. Each URL led to an HTML or XML file representing a single record. If the file was HTML it would contain meta tags describing the relevant fields.
Harvest Control Lists were publically available on the internet and stored at a URL accessible to the Trove Harvester. It was requested that they be updated regularly, preferably automatically, to ensure records in Trove remained current. Many collections grew to long lists of links, in the tens of thousands, which Trove visited each and every time. This resulted in long and sometimes stalled harvests.
Sitemaps are a more modern version of a Harvest Control List and are recommended for both small and large collections.
Here is a Harvest Control List with links to three separate records named 24597933.xml, 9186335.xml and 3995299.xml.
Harvest Control List
<h1>Local History Photograph collection</h1>
Trove can be configured to work with your organisation’s custom API. We’re best placed to work with APIs that
- Are RESTful.
- Offer XML or JSON output
- Output Dublin Core or similar standard metadata
- Include notifications of records that have been deleted
- Can limit the number of records returned per request to less than 100, and can paginate through results
- Can provide incremental harvests (“show me all records updated since 2014-06-18")
- Provide all metadata fields in the search results, without a second call for the full individual record
None of these are hard and fast rules, they do make setup and maintenance quicker at the Trove end and minimise the number of calls to your API.
To integrate with any custom API requires specialised work by the Trove team. The process of interoperating with a custom API can therefore take longer to setup than other standardised harvesting methods. Please contact us to discuss how Trove can work with your API.
A Sitemap is a standard XML protocol. It lists all the web pages within a website. Search services like Google, Yahoo and Bing use a Sitemap as a guide around a website, to help them find and visit every page. They use this data later to point users towards the page where their search term exists.
Trove uses Sitemaps in a similar way, as a list that points to the different web pages where records are available. Every URL in the list is visited, and Trove will attempt to ingest each page as a record. Pages not mentioned in the Sitemap are ignored.
Many popular web hosting platforms such as WordPress and Drupal include Sitemaps functionality by default, or as an optional plugin.
To work with Trove’s web crawler a website needs to have three layers of files:
- robots.txt - A plain text file that instructs webcrawlers in how they should move around your website. It can contain an optional line 'Sitemap: ' followed by the URL of a Sitemap file, for example Sitemap: http://www.sitemaphost.com/sitemap-host1.xml To contribute to Trove you will need to include this line.
- Sitemap or Sitemap Index file – a list of URLs. These resolve either to more Sitemap files, or to web pages that can be treated by Trove as single record pages.
- Single record pages – a single HTML or XML file that will represent a single record in Trove. This might be a single episode of a podcast, a single artwork, a single journal article. We recommend that this be the same page any human visitor would see, with some extra hidden data underneath for services like Trove, as well as the big search engines. This extra data is hidden in <meta> tags, see the HTML record section for more information.
Trove can read both Sitemap and Sitemap index files. The ‘lastmod’, ‘changefreq’, and ‘priority’ elements are currently ignored.
Sitemap websites can be crawled on a daily, weekly, monthly, bi-monthly, quarterly, half-yearly or yearly schedule. Nominate the frequency that suits you, considering how often your content is added or changed.
Trove specifies standard data elements it expects from all contributors’ collections. This is called a schema. While the transfer protocols provide a way to move the batches of records into Trove, the schema specifies what kinds of information Trove should expect for each record.
Trove currently works with a variety of metadata schemas, including Unqualified Dublin Core , Encoded Archival Context - Corporate Bodies, Persons, and Families, and Registry Interchange Format - Collections and Services.
Unqualified Dublin Core specifies fifteen fields of data:
- identifier (mandatory)
- title (mandatory)
- type (mandatory)
Trove accepts and displays all of these fields.
Items do not have to be fully available online for free (eg. a digitised picture or a full thesis). It may be a link to a finding aid describing the box an item is stored in, or an abstract for an article.
An XML record can contain both Dublin Core and Picture Australia elements for images, or the extended form of Dublin Core that Trove implements. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.
<?xml version="1.0" encoding="UTF-8" ?>
<dc:title>Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /</dc:title>
<dc:creator>Scriven, Edward, 1775-1841.</dc:creator>
<dc:contributor>Dance, Nathaniel, 1735-1811.</dc:contributor>
<dc:contributor>Society for the Diffusion of Useful Knowledge (Great Britain)</dc:contributor>
<dc:coverage>1840 - 1849</dc:coverage>
<dc:description>"Under the superintendance of the Society for the Diffusion of Useful Knowledge."; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582.</dc:description>
<dc:format>1 print : engraving ; plate mark 28 x 20 cm.</dc:format>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the <a href="http://www.nla.gov.au/digicoll/copyright.html">National Library of Australia</a> to request permission.</dc:rights>
<dc:publisher>London : Published by Charles Knight, [184-?]</dc:publisher>
<dc:subject>Cook, James, 1728-1779 -- Portraits.</dc:subject>
<dc:subject>Explorers -- Great Britain -- Portraits.</dc:subject>
Records for items without thumbnails do not require the extra Picture Australia elements, however it is helpful to include the linktype attribute. This indicates to Trove if the item is fully online. For example this record for a digitised text contains only Dublin Core elements.
<?xml version="1.0" encoding="UTF-8" ?>
<dc:title>Some thoughts on examinations [electronic resource]/</dc:title>
<dc:publisher>[Oxford] : Privately printed, 1906.</dc:publisher>
<dc:format>11 p. ; 22cm.</dc:format>
<dc:description>Electronic reproduction. Canberra, A.C.T. National Library of Australia,2010.</dc:description>
<dc:source>Item held by National Library of Australia</dc:source>
<dc:rights>You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission.</dc:rights>
A record in HTML needs to include the meta element within the head section of the file. Other information may be contained in the body, allowing meta tags to be embedded in existing pages. This record describes a picture entitled Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital.
<link rel="schema.DC" href="http://purl.org/dc/elements/1.1/" />
<link rel="schema.PA" href="http://www.pictureaustralia.org/schemas/pa/" />
<meta name="DC.title" content="Captain Cook, from an original picture by Dance in the gallery of Greenwich Hospital [picture] /" />
<meta name="DC.creator" content="Scriven, Edward, 1775-1841." />
<meta name="DC.contributor" content="Dance, Nathaniel, 1735-1811." />
<meta name="DC.contributor" content="Society for the Diffusion of Useful Knowledge (Great Britain)" />
<meta name="DC.coverage" content="1840 - 1849" />
<meta name="DC.description" content=""Under the superintendance of the Society for the Diffusion of Useful Knowledge."; Beddie, 3367.; Also available in an electronic version via the Internet at: http://nla.gov.au/nla.pic-an9186335; S3582." />
<meta name="DC.format" content="1 print : engraving ; plate mark 28 x 20 cm." />
<meta name="DC.identifier" content="nla.pic-an9186335" />
<meta name="DC.rights" content="You may save or print this image for research and study. If you wish to use it for any other purposes, you must contact the National Library of Australia to request permission." />
<meta name="DC.publisher" content="London : Published by Charles Knight, [184-?]" />
<meta name="DC.subject" content="Cook, James, 1728-1779 -- Portraits."
<meta name="DC.subject" content="Explorers -- Great Britain -- Portraits."
<meta name="DC.type" content="Image" />
<meta name="PA.thumbnail" content="http://nla.gov.au/nla.pic-an9186335-t" />
<meta name="PA.viewcopy" content="http://nla.gov.au/nla.pic-an9186335-v" />
<meta name="PA.location" content="http://nla.gov.au/nla.pic-an9186335" />
Records in Trove are linked to a National Union Catalogue (NUC) Symbol. This is a brief collection identifier made up of 2-10 letters. NUC Symbols are issued by the National Library of Australia and maintained in the InterLibrary Resource Sharing Directory. Every NUC Symbol is unique and allows Trove to identify the organisation holding the item. It can also identify a sub-collection. A NUC Symbol is added to each incoming record by the Trove harvester.
Organisations are welcome to nominate their pre-existing NUC Symbol or request a new one for a particular collection. They can be obtained by filling in the New Service Notification Form at the ILRS Directory, or the Trove team will set one up on your behalf when you contribute to a new collection.