Retrieving Metadata from an API

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I get metadata for a DOI automatically?
  • How do I work with JSON data in Python?

Objectives

  • Explain what a web API is.
  • Retrieve metadata for a DOI from the DataCite REST API.
  • Navigate nested JSON data made of dictionaries and lists.
  • Use a loop to retrieve metadata for several DOIs.

DataCite

DataCite is a registration agency for DOIs, mainly for research data, software and other research outputs. Repositories such as Zenodo and figshare register their DOIs with DataCite.

You can search the DataCite metadata manually in DataCite Commons:

  1. Search for git in Works.
  2. Filter by Dataset.
  3. Open a result, and under Download open the metadata as DataCite JSON.

This is the same data that we will now retrieve with Python. Instead of using a web interface, programs can request data from a web service through an Application Programming Interface (API). The DataCite REST API returns the metadata for a DOI if we send a request to a URL that contains the DOI.

Request the metadata for a DOI

PYTHON 

import urllib.request
import json

base_url = "https://api.datacite.org/application/vnd.datacite.datacite+json/"
doi = "10.5281/zenodo.5563162"
full_url = base_url + doi

PYTHON 

doi_json_dataset = urllib.request.urlopen(full_url).read()

The API returns the metadata in JSON format, as text. The json library converts this text into Python data structures:

PYTHON 

doi_dataset = json.loads(doi_json_dataset)
doi_dataset

OUTPUT 

{'id': 'https://doi.org/10.5281/zenodo.5563162',
 'doi': '10.5281/ZENODO.5563162',
 'url': 'https://zenodo.org/record/5563162',
 'types': {'ris': 'DATA',
  'bibtex': 'misc',
  'citeproc': 'dataset',
  'schemaOrg': 'Dataset',
  'resourceTypeGeneral': 'Dataset'},
 'creators': [{'name': 'Banda, Juan M.',
   'givenName': 'Juan M.',
   'familyName': 'Banda',
   ...

PYTHON 

type(doi_dataset)

OUTPUT 

dict

The result is a dictionary, just like the ones we created ourselves. Its keys are the metadata fields:

PYTHON 

doi_dataset.keys()

OUTPUT 

dict_keys(['id', 'doi', 'url', 'types', 'creators', 'titles', 'publisher', 'container', 'subjects', 'contributors', 'dates', 'publicationYear', 'language', 'identifiers', 'sizes', 'formats', 'version', 'rightsList', 'descriptions', 'geoLocations', 'fundingReferences', 'relatedIdentifiers', 'relatedItems', 'schemaVersion', 'providerId', 'clientId', 'agency', 'state'])

Access the year of the publication

PYTHON 

print(doi_dataset["publicationYear"])

OUTPUT 

2021

Access the title of the publication

Some values are not simple strings or numbers, but lists or dictionaries themselves. A dataset can have several titles, so titles is a list of dictionaries:

PYTHON 

# Getting title within a list with a dictionary inside
print(doi_dataset["titles"])

OUTPUT 

[{'title': 'A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration'}]

PYTHON 

# Getting the first element out of the list, which is the dictionary
print(doi_dataset["titles"][0])

OUTPUT 

{'title': 'A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration'}

PYTHON 

# Access the key of the dictionary which is the title of the publication
print(doi_dataset["titles"][0]["title"])

OUTPUT 

A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration

Access the creators of the publication

creators is also a list of dictionaries, one per creator:

PYTHON 

# Access keys from creators list
doi_dataset["creators"][0].keys()

OUTPUT 

dict_keys(['name', 'givenName', 'familyName', 'affiliation', 'nameIdentifiers'])

PYTHON 

# Access creator of the publication
print(doi_dataset["creators"][0]["name"])

OUTPUT 

Banda, Juan M.

PYTHON 

# Next creator
print(doi_dataset["creators"][1]["name"])

OUTPUT 

Tekumalla, Ramya

PYTHON 

# Access only the given name
print(doi_dataset["creators"][0]["givenName"])

OUTPUT 

Juan M.
Challenge

The publisher

Look at doi_dataset["publisher"]. What type of value is it? How do you get just the name of the publisher as a string?

PYTHON 

print(doi_dataset["publisher"])
print(type(doi_dataset["publisher"]))
print(doi_dataset["publisher"]["name"])

OUTPUT 

{'name': 'Zenodo'}
<class 'dict'>
Zenodo

The publisher is a dictionary, so we need a second pair of square brackets with the key "name".

Metadata for several DOIs

Challenge

Loop over DOIs

Use the same approach as above for a larger set of DOIs. For each DOI, print the title, the DOI and the publisher.

Bonus: also print the publication year.

PYTHON 

dois = ["10.6084/m9.figshare.155613",
        "10.6084/m9.figshare.153821.v1",
        "10.5281/zenodo.2599866",
        "10.5281/zenodo.17422255"]

PYTHON 

for doi in dois:
    doi_json_dataset = urllib.request.urlopen(base_url + doi).read()
    doi_dataset = json.loads(doi_json_dataset)
    print(doi_dataset["titles"][0]["title"])
    print("- " + doi)
    print("- " + doi_dataset["publisher"]["name"])
    print("- " + str(doi_dataset["publicationYear"]))
    print("-----------------")

OUTPUT 

git repository for paper on git and reproducible science
- 10.6084/m9.figshare.155613
- figshare
- 2013
-----------------
git can facilitate greater reproducibility and increased transparency in science
- 10.6084/m9.figshare.153821.v1
- figshare
- 2013
-----------------
Nachnutzbare Awarenessmaterialien für Forschungsdatenmanagement (FDM)
- 10.5281/zenodo.2599866
- Zenodo
- 2019
-----------------
Python Basics
- 10.5281/zenodo.17422255
- Zenodo
- 2025
-----------------

Note that publicationYear is a number, so it has to be converted with str() before it can be combined with a string.

Callout

Not every DOI is a DataCite DOI

DOIs for journal articles are usually registered with Crossref, not DataCite. If you request such a DOI from the DataCite API, you get an HTTPError: HTTP Error 404: Not Found. Crossref has its own API at https://api.crossref.org/works/<DOI>.

Optional: all creators of a publication

To get all creators, we can loop over the creators list. After the loop above, doi_dataset contains the metadata of the last DOI:

PYTHON 

for creator in doi_dataset["creators"]:
    print(creator["name"])

OUTPUT 

Förstner, Konrad U.
Müller, Rabea
Sauerwein, Till
Vandendorpe, Justine

A shorter way to create a list from an existing list is a list comprehension:

PYTHON 

creator_names = [creator["name"] for creator in doi_dataset["creators"]]
creator_names

OUTPUT 

['Förstner, Konrad U.',
 'Müller, Rabea',
 'Sauerwein, Till',
 'Vandendorpe, Justine']
Key Points
  • A web API lets programs request data from a web service via a URL.
  • urllib.request.urlopen(url).read() downloads the response of a URL.
  • json.loads() converts JSON text into Python dictionaries and lists.
  • Navigate nested data step by step: data["titles"][0]["title"].
  • DataCite registers DOIs for data and software, Crossref for most articles.