All in One View

Content from Getting Started

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I identify and use key features of JupyterLab to create and manage a Python notebook?
  • How do I run Python code in JupyterLab, and how can I see and interpret the results?

Objectives

  • Identify applications of Python in library and information science environments by the end of this lesson.
  • Launch JupyterLab and create a new Jupyter Notebook.
  • Navigate the JupyterLab interface, including file browsing, cell creation, and cell execution, with confidence.
  • Write and execute Python code in a Jupyter Notebook cell, observing the output and modifying code as needed.
  • Save a Jupyter Notebook as an .ipynb file and verify the file’s location in the directory within the session.

Why Python?

Python is a popular programming language for tasks such as data collection, cleaning, and analysis. Python can help you to create reproducible workflows to accomplish repetitive tasks more efficiently.

This lesson starts with the basics of the Python language and then shows how to analyse tabular data with the pandas library, using data on article processing charges (APCs) from OpenAPC, and how to retrieve publication metadata from the DataCite API.

Challenge

Python in Libraries

There are a lot of ways that library and information science folks use Python in their work. Go around the room and have helpers and co-teachers share how they have used Python.

Learners: Can you think of other ways to use Python in libraries? Do you have hopes for how you’d like to use Python in the future?

Here a few areas where you might apply Python in your work.

Metadata work. Many cataloging teams use Python to migrate, transform and enrich metadata that they receive from different sources. For example, the pymarc library is a popular Python package for working with MARC21 records.

Collection and citation analysis. Python can automate workflows to analyze library collections. In cases where spreadsheets and OpenRefine are unable to support specific forms of analysis, Python is a more flexible and powerful tool.

Assessment. Library workers often need to collect metrics or statistics on some aspect of their work. Python can be a valuable tool to collect, clean, analyze, and visualize that data in a consistent way over time.

Accessing data. Researchers often use Python to collect data (including textual data) from websites and social media platforms. Academic librarians are often well-positioned to help teach these researchers how to use Python for web scraping or querying Application Programming Interfaces (APIs) to access the data they need.

Analyzing data. Python is widely used by scholars who are embarking on different forms of computational research (e.g., network analysis, natural language processing, machine learning). Library workers can leverage Python for their own research in these areas, but also take part in larger networks of academic support related to data science, computational social sciences, and/or digital humanities.

Use JupyterLab to edit and run Python code.

If you haven’t already done so, see the setup instructions for details on how to install JupyterLab and Python using Miniforge. Please also create a folder called lc-python on your Desktop, in which we will save our notebooks. All data used in this lesson is downloaded directly from the web during the lesson.

Getting started with JupyterLab

To run Python, we are going to use Jupyter Notebooks via JupyterLab. Jupyter notebooks are common tools for data science and visualization, and serve as a convenient environment for running Python code interactively where we can view and share the results of our Python code.

Callout

Alternatives to Juypter

There are other ways of editing, managing, and running Python code. Software developers often use an integrated development environment (IDE) like PyCharm, Spyder or Visual Studio Code (VS Code), to create and edit Python scripts. Others use text editors like Vim or Emacs to hand-code Python. After editing and saving Python scripts you can execute those programs within an IDE or directly on the command line.

Jupyter notebooks let us execute and view the results of our Python code immediately within the notebook. JupyterLab has several other handy features:

  • You can easily type, edit, and copy and paste blocks of code.
  • It allows you to annotate your code with links, different sized text, bullets, etc. to make it more accessible to you and your collaborators.
  • It allows you to display figures next to the code to better explore your data and visualize the results of your analysis.
  • Each notebook contains one or more cells that contain code, text, or images.

Start JupyterLab

Once you have created the lc-python directory on your Desktop, you can start JupyterLab by opening a shell command line interface.

Mac users - Command Line

  1. Press the cmd + spacebar keys and search for Terminal. Click the result or press return. (You can also find Terminal in your Applications folder, under Utilities.)
  2. After you have launched Terminal, change directories to the lc-python folder you created earlier.
  3. Next run conda activate carpentries to launch a Python environment which includes Jupyter Notebooks and other tools we will need to proceed through the lesson.
  4. Finally, type jupyter lab which should open a browser window where you can run Jupyter Notebooks.

Note that the $ sign is used to indicate a command to be typed on the command prompt, but we never type the $ sign itself, just what follows after it.

BASH 

$ cd ../Desktop/lc-python
$ conda activate carpentries
$ jupyter lab

Windows users - Command Line

To start the JupyterLab server you will need to access the Anaconda Prompt.

  1. Press the Windows Logo Key and search for Miniforge Prompt, click the result or press enter.
  2. Once you have launched the Miniforge Prompt, change directories to the lc-python folder you created earlier. (The example below assumes a starting path of C:\Users\username)
  3. Next run conda activate carpentries to launch a Python environment which includes Jupyter Notebooks and other tools we will need to proceed through the lesson.
  4. Finally, type jupyter lab which should open a browser window where you can run Jupyter Notebooks.

Note that the $ sign is used to indicate a command to be typed on the command prompt, but we never type the $ sign itself, just what follows after it.

BASH 

$ cd \Desktop\lc-python
$ conda activate carpentries
$ jupyter lab

The JupyterLab Interface

Launching JupyterLab opens a new tab or window in your preferred web browser. While JupyterLab enables you to run code from your browser, it does not require you to be online. If you take a look at the URL in your browser address bar, you should see that the environment is located at your localhost, meaning it is running from your computer: http://localhost:8888/lab.

When you first open JupyterLab you will see two main panels. In the left sidebar is your file browser, which shows the contents of the lc-python folder.

Creating a Juypter Notebook

To the right you will see a Launcher tab. Here we have options to launch a Python 3 notebook, a Terminal (where we can use shell commands), text files, and other items. For now, we want to launch a new Python 3 notebook, so click once on the Python 3 (ipykernel) button underneath the Notebook header. You can also create a new notebook by selecting New -> Notebook from the File menu in the Menu Bar.

screenshot of the JupyterLab for launching notebook
Launching a new Python 3 Notebook

When you start a new Notebook you should see a new tab labeled Untitled.ipynb. You will also see this file listed in the file browser to the left. Right-click on the Untitled.ipynb file in the file browser and choose Rename from the dropdown options. Let’s call the notebook file, workshop.ipynb.

Callout

JupyterLab? What about Jupyter notebooks? Python notebooks? IPython?

JupyterLab is the next stage in the evolution of the Jupyter Notebook. If you have prior experience working with Jupyter notebooks, then you will have a good idea of how to work with JupyterLab. Jupyter was created as a spinoff of IPython in 2014, and includes interactive computing support for languages other than just Python, including R and Julia. While you’ll still see some references to Python and IPython notebooks, IPython notebooks are officially deprecated in favor of Jupyter notebooks.

We will share more features of the JupyterLab environment as we advance through the lesson, but for now let’s turn to how to run Python code.

Running Python code

Jupyter allows you to add code and formatted text in different types of blocks called cells. By default, each new cell in a Jupyter Notebook will be a “code cell” that allows you to input and run Python code. Let’s start by having Python do some arithmetic for us.

In the first cell type 7 * 3, and then press the Shift+Return keys together to execute the contents of the cell. (You can also run a cell by making sure your cursor is in the cell and choosing Run > Run Selected Cells or selecting the “Play” icon (the sideways triangle) at the top of the noteboook.)

PYTHON 

7 * 3

You should see the output appear immediately below the cell, and Jupyter will also create a new code cell for you.

PYTHON 

21

If you move your cursor back to the first cell, just after the 7 * 3 code, and hit the Return key (without shift), you should see a new line in the cell where you can add more Python code. Let’s add another calculation to the same cell:

PYTHON 

7 * 3
2 + 1

While Python runs both calculations Juypter will only display the output from the last line of code in a specific cell, unless you tell it to do otherwise.

PYTHON 

3

Editing the notebook

You can use the icons at the top of your notebook to edit the cells in your Notebook:

  • The + icon adds a new cell below the selected cell.
  • The scissors icon will delete the current cell.

You can move cells around in your notebook by hovering over the left-hand margin of a cell until your cursor changes into a four-pointed arrow, and then dragging and dropping the cell where you want it.

Callout

Markdown

You can add text to a Juypter notebook by selecting a cell, and changing the dropdown above the notebook from Code to Markdown. Markdown is a lightweight language for formatting text. This feature allows you to annotate your code, add headers, and write documentation to help explain the code. While we won’t cover Markdown in this lesson, there are many helpful online guides out there: - Markdown for Jupyter Cheatsheet (IBM) - Markdown Guide (Matt Cone)

screenshot of the Jupyter notebook dropdown to change a cell to Markdown
Changing a cell from Code to Markdown

You can also use “hotkeys”” to change Jupyter cells from Code to Markdown and back:

  • Click on the code cell that you want to convert to a Markdown cell.
  • Press the Esc key to enter command mode.
  • Press the M key to convert the cell to Markdown.
  • Press the y key to convert the cell back to Code.
Key Points
  • You can launch JupyterLab from the command line or from Anaconda Navigator.
  • You can use a JupyterLab notebook to edit and run Python.
  • Notebooks can include both code and markdown (text) cells.

Content from Variables and Data Types

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How do I run my first Python command?
  • How can I store values in Python?
  • What kinds of data can Python work with?

Objectives

  • Use print() to display values in a Jupyter notebook.
  • Write comments to explain code.
  • Assign values to variables.
  • Name and identify the basic data types str, int, float, and bool.

First command

You can use the built-in print() function to print Python objects, such as a string, to the Jupyter notebook output. We will explore functions in more detail later in this lesson. The objects that we want to print are written inside the parentheses, and these are known as the arguments of the print() function.

PYTHON 

print("Hello world!")

OUTPUT 

Hello world!

Comments

The code tells you “how”, comments tell you “why”. Everything after a # on a line is ignored by Python.

PYTHON 

# This is a comment.
print("Hello world!")

OUTPUT 

Hello world!
Callout

Markdown cells

Besides code cells, Jupyter notebooks have Markdown cells where you can provide context for your code. You can write text in italics or bold, or even both, add headings, lists, links and much more.

Variables

A variable is a name given to a value. In Python, the = symbol is used to assign a value to a variable. In this example, Python assigns the value "Lovelace" to the variable name.

PYTHON 

# Variables and assignment
# Lovelace: Ada Lovelace, mathematician, wrote the first computer program
name = "Lovelace"
print(name)

OUTPUT 

Lovelace
Callout

Naming variables

Variable names:

  • cannot start with a digit
  • cannot contain spaces, quotation marks, or other punctuation
  • may contain an underscore (typically used to separate words in long variable names, e.g. first_name)
  • are case sensitive: name and Name are different variables.

Data types

Every value in Python has a type. The most basic types are strings, integers, floating point numbers and booleans.

PYTHON 

# String (str) - Sequence of characters
first_name = "Ada"
print(first_name)

OUTPUT 

Ada

PYTHON 

# Integer (int) - Whole number
age = 37
print(age)

OUTPUT 

37

PYTHON 

# Floating point number (float) - Fractional number
size = 1.7123
print(size)

OUTPUT 

1.7123

PYTHON 

# Boolean (bool) - One of two values
knows_math = True
print(knows_math)
likes_gardening = False
print(likes_gardening)

OUTPUT 

True
False

You can use the built-in function type() to find out the type of a value.

PYTHON 

# Print data type
type(name)

OUTPUT 

str

Displaying values in Jupyter

In Jupyter notebooks, you can omit the print() function for objects, such as variables, that appear on the final line of a cell. If the final line of a Jupyter cell includes the name of a variable, its value will be displayed in the notebook when the cell is run. Only the last line is shown, though:

PYTHON 

# The last value will be output by Jupyter Notebook
name
age

OUTPUT 

37

PYTHON 

# If you want to output other values, you have to use "print"
print(name)
print(age)

OUTPUT 

Lovelace
37
Challenge

What type is it?

What is the type of each of the following values? Check your guesses with type().

  1. "2025"
  2. 2025
  3. 20.25
  4. False
  5. "False"
  1. str, because the value is in quotation marks
  2. int
  3. float
  4. bool
  5. str, again because of the quotation marks
Key Points
  • Use print() to display values.
  • Use # to add comments that explain why the code does what it does.
  • Use = to assign a value to a variable.
  • Basic data types are strings (str), integers (int), floating point numbers (float), and booleans (bool).
  • Use type() to find out the type of a value.
  • Jupyter displays the value of the last line of a cell.

Content from Operators, Functions and Methods

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • What can I do with values of different types?
  • What happens when I combine values of different types?
  • What are functions and methods?

Objectives

  • Use arithmetic operators on numbers and strings.
  • Read and interpret error messages.
  • Convert values from one type to another.
  • Call built-in functions and string methods.

We continue with the variables from the previous episode:

PYTHON 

name = "Lovelace"
age = 37
size = 1.7123

Operators

An object’s type determines what the program can do with it.

PYTHON 

# Addition of integers
5 + 5

OUTPUT 

10

PYTHON 

# Multiplication
age * 100

OUTPUT 

3700

The same operators also work on strings, but they do something different:

PYTHON 

# Concatenation of strings
name + name

OUTPUT 

'LovelaceLovelace'

PYTHON 

# Concatenation of strings x times
name * 10

OUTPUT 

'LovelaceLovelaceLovelaceLovelaceLovelaceLovelaceLovelaceLovelaceLovelaceLovelace'

We get an error if we try to add a number to a string.

PYTHON 

# Does not work: the "+" operator cannot be applied
name + 10

ERROR 

TypeError: can only concatenate str (not "int") to str

PYTHON 

# Works: here, "10" is a string
name + "10"

OUTPUT 

'Lovelace10'
Callout

Reading error messages

Error messages can look intimidating, but they are there to help you. Read the last line first: it tells you the type of the error (here a TypeError) and a short description of what went wrong. Above it, Python shows you the line of code where the error happened.

Dividing two numbers always results in a float:

PYTHON 

# Division
size / age

OUTPUT 

0.04627837837837838

PYTHON 

type(size)

OUTPUT 

float

Type conversion

Data that we read from files or from the web often arrives as text, even if it looks like a number.

PYTHON 

number_of_articles = "50"

PYTHON 

# Does not work: unsupported operand type(s) for /: 'str' and 'int'
number_of_articles / 10

ERROR 

TypeError: unsupported operand type(s) for /: 'str' and 'int'

PYTHON 

# First convert the string to an integer using int()
int(number_of_articles) / 10

OUTPUT 

5.0

Similarly, str() converts a value to a string and float() converts it to a floating point number.

Functions and methods

The len() function tells us the length of an item. In the case of a string, for example, it tells us how many characters it contains.

PYTHON 

len(name)

OUTPUT 

8

Methods are special types of function that are always bound to an object, the thing on the left of the dot. For now, the difference between a function and a method is not that relevant.

PYTHON 

name.upper()

OUTPUT 

'LOVELACE'

PYTHON 

# The string itself is not changed
name

OUTPUT 

'Lovelace'

PYTHON 

name.count("e")

OUTPUT 

2

PYTHON 

# Methods with two arguments
name.replace("lace", " is everywhere")

OUTPUT 

'Love is everywhere'
Callout

Getting help

Use the built-in function help() to see the documentation of a function or method, e.g. help(len) or help(name.replace). In Jupyter, you can also put the cursor inside the parentheses and press Shift+Tab.

Challenge

Fix the error

A colleague wants to print how many articles a library has licensed, but the code below fails. What is wrong, and how can you fix it?

PYTHON 

number_of_articles = 50
print("We licensed " + number_of_articles + " articles.")

number_of_articles is an int, and Python cannot add an int to a str. Convert the number to a string first:

PYTHON 

print("We licensed " + str(number_of_articles) + " articles.")

OUTPUT 

We licensed 50 articles.
Challenge

String methods

Given title = "introduction to python", use string methods to

  1. print the title in capital letters,
  2. count how many times the letter "o" occurs,
  3. replace "python" with "pandas".

PYTHON 

title = "introduction to python"
print(title.upper())
print(title.count("o"))
print(title.replace("python", "pandas"))

OUTPUT 

INTRODUCTION TO PYTHON
4
introduction to pandas
Key Points
  • The type of a value determines what an operator does with it.
  • Read error messages from the bottom up: the last line names the error.
  • Use int(), float() and str() to convert between types.
  • Functions are called with function(arguments), methods with object.method(arguments).
  • String methods return a new value and do not change the original string.

Content from Lists and Dictionaries

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I store many values together?
  • How can I store and retrieve values by name?

Objectives

  • Create lists and access their elements by index.
  • Select parts of a list with slicing.
  • Create dictionaries and look up values by key.

Lists

The most popular type of data collection in Python is the list. Lists have two important characteristics:

  1. They are mutable, meaning they can be changed after creation.
  2. They are heterogeneous, meaning they can store values of many different types.

To create a new list, put some values in square brackets, separated by commas. Let’s create a short list of names.

PYTHON 

# Lovelace: Ada Lovelace, mathematician, wrote the first computer program
# Hopper: Grace Hopper, computing pioneer, invented the first compiler
# Noether: Emmy Noether, mathematician, pioneer of abstract algebra
# Turing: Alan Turing, foundations of computer science
names = ["Lovelace", "Hopper", "Noether", "Turing"]

Indexing

We can use index numbers to reference elements in a list. Python uses square brackets ([ and ]) to access single elements of objects that can be decomposed into parts. Counting starts at 0.

PYTHON 

names[0]

OUTPUT 

'Lovelace'

PYTHON 

names[1]

OUTPUT 

'Hopper'

PYTHON 

names[2]

OUTPUT 

'Noether'

You can also use negative indices: names[-1] is the last element of a list, names[-2] the second last, and so on.

PYTHON 

names[-1]

OUTPUT 

'Turing'

Slicing

We can also use index numbers to select a slice of a list. The slice names[0:3] starts at index 0 and stops before index 3.

PYTHON 

names[0:3]

OUTPUT 

['Lovelace', 'Hopper', 'Noether']

If you leave out the start or the end, the slice starts at the beginning or goes until the end of the list.

PYTHON 

names[:2]

OUTPUT 

['Lovelace', 'Hopper']

PYTHON 

names[2:]

OUTPUT 

['Noether', 'Turing']
Callout

Strings can be indexed, too

Indexing and slicing also work on strings: "Lovelace"[0] is 'L' and "Lovelace"[:4] is 'Love'. More on slicing: https://stackoverflow.com/questions/509211/how-slicing-in-python-works

Challenge

Slicing practice

Using the list names, write code that returns

  1. ['Hopper', 'Noether']
  2. the last two names
  3. the number of names in the list

PYTHON 

print(names[1:3])
print(names[-2:])
print(len(names))

OUTPUT 

['Hopper', 'Noether']
['Noether', 'Turing']
4

Dictionaries

Whereas a list is simply a collection of elements, a dictionary stores key-value pairs. Put into plain English, it lets you store and retrieve data values by name. Dictionaries are written with curly brackets, and each key is separated from its value by a colon.

PYTHON 

authors_and_birth_years = {"Lovelace": 1815,
                           "Hopper": 1906,
                           "Noether": 1882,
                           "Turing": 1912}

PYTHON 

# Retrieve a value from the dictionary
authors_and_birth_years["Hopper"]

OUTPUT 

1906

Keys are case sensitive. If a key does not exist, Python raises a KeyError:

PYTHON 

# Does not work: key is not defined
authors_and_birth_years["HOPPER"]

ERROR 

KeyError: 'HOPPER'
Challenge

Extend the dictionary

Add the computer scientist Margaret Hamilton (born 1936) to authors_and_birth_years and then look up her birth year.

Hint: you can assign a new value with dictionary[key] = value.

PYTHON 

authors_and_birth_years["Hamilton"] = 1936
authors_and_birth_years["Hamilton"]

OUTPUT 

1936
Key Points
  • A list stores many values in order: [value1, value2, ...].
  • Use list[index] to access an element; indexing starts at 0.
  • Use list[start:stop] to select a slice; stop is not included.
  • A dictionary stores key-value pairs: {key: value, ...}.
  • Use dictionary[key] to look up a value.

Content from Loops and Conditionals

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I do the same thing for many values?
  • How can my program make decisions?

Objectives

  • Use a for loop to repeat an operation for each element of a list.
  • Use if, elif and else to run code only under certain conditions.

For loops

A for loop repeats an operation once for each element in a collection. The general structure of a loop is:

PYTHON 

for variable in collection:
    # do things using variable, such as print
  • We can name the loop variable anything we like.
  • There must be a colon at the end of the line that starts the loop.
  • We must indent anything that we want to run inside the loop. Unlike in many other programming languages, there is no command to signify the end of the loop body. Anything indented after the for statement belongs to the loop.

PYTHON 

names = ["Lovelace", "Hopper", "Noether", "Turing"]

for person in names:
    print(person + " did awesome stuff.")

OUTPUT 

Lovelace did awesome stuff.
Hopper did awesome stuff.
Noether did awesome stuff.
Turing did awesome stuff.

Loops are a robust way of dealing with collections such as lists. Even if the values in the list change, the loop will still work.

As with all variable names, it is helpful to give loop variables meaningful names that you will understand as your code grows. For example, for person in names is easier to understand than for kitten in names.

Challenge

Loop over a dictionary

Looping over a dictionary gives you its keys. Use this to print one line per person in authors_and_birth_years, e.g. Lovelace was born in 1815.

PYTHON 

authors_and_birth_years = {"Lovelace": 1815,
                           "Hopper": 1906,
                           "Noether": 1882,
                           "Turing": 1912}

PYTHON 

for person in authors_and_birth_years:
    print(person + " was born in " + str(authors_and_birth_years[person]))

OUTPUT 

Lovelace was born in 1815
Hopper was born in 1906
Noether was born in 1882
Turing was born in 1912

Conditions and if statements

An if statement is a conditional statement that determines whether a block of code is executed. The syntax of an if statement is similar to that of a for statement:

  • The first line begins with if and ends with a colon.
  • The body is indented (usually by four spaces).

PYTHON 

temp = 20
if temp > 15:
    print("It's warm")

OUTPUT 

It's warm

The condition itself is just a value of type bool:

PYTHON 

20 > 15

OUTPUT 

True

We can use an else statement following an if statement to specify an alternative code block to execute when the if branch is not taken.

PYTHON 

temp = 10
if temp > 15:
    print("It's warm")
else:
    print("It's cold")

OUTPUT 

It's cold

You can use elif (short for “else if”) to provide several alternative choices, each with its own test. An elif statement must always follow an if statement and come before the else statement (the catch-all).

PYTHON 

temp = 15
if temp >= 25:
    print("It's hot")
elif temp >= 15:
    print("It's warm")
else:
    print("It's cold")

OUTPUT 

It's warm

Conditions are tested in order and only once. Python steps through the branches of the conditional statement in order, testing each one in turn, so the order of your statements matters.

Callout

Comparison operators

  • == equal to (note the two equals signs!)
  • != not equal to
  • > and < greater than, less than
  • >= and <= greater than or equal to, less than or equal to
Challenge

Combine loops and conditionals

Print only the names of the people in authors_and_birth_years who were born before 1900.

PYTHON 

for person in authors_and_birth_years:
    if authors_and_birth_years[person] < 1900:
        print(person)

OUTPUT 

Lovelace
Noether
Key Points
  • A for loop runs its indented body once for each element of a collection.
  • Use if, elif and else to choose which code to run.
  • Conditions are tested in order; the first one that is True wins.
  • Indentation defines which lines belong to a loop or an if statement.

Content from Introduction to pandas

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I work with tabular data in Python?
  • How much did a university pay on average for article processing charges?

Objectives

  • Import a library and use its functions.
  • Download a CSV file and read it into a pandas DataFrame.
  • Inspect, select and filter data in a DataFrame.
  • Calculate simple statistics and plot a column.

Libraries

Once we have mastered the core language of Python, there are many libraries (i.e. collections of reusable code) that offer additional functionality. Some come with Python itself (the standard library), others are written and shared by the community. One very popular example is pandas, which was written to work with tabular data.

Every good library should have good documentation. The pandas getting started guide answers questions such as:

  • What kind of data does pandas handle? pandas works with so-called DataFrames. The concept comes from the R programming language, and a DataFrame is simply a table consisting of columns and rows. The rows are labelled by an index.
  • How do I read and write tabular data? pandas can read many file formats such as CSV (comma-separated values), TSV (tab-separated values) and XLSX (Excel files), as long as the data is tabular.

We can then filter the DataFrame, calculate new values from its columns, plot the data, and finally save the result in the format we need (e.g. CSV, Excel, JSON or HTML).

The data: article processing charges

APC stands for Article Processing Charges, the fees that publishers charge for publishing an article open access. OpenAPC openly collects data on how much institutions pay for APCs. The data is published in the GitHub repository https://github.com/OpenAPC/openapc-de.

In the data folder of that repository there is a folder for each institution. We will look at the data from FU Berlin for the year 2015: APC_FU_Berlin_2015.csv.

GitHub renders the CSV file as a nicely formatted table with columns such as institution, period, price in euros, DOI, publisher and journal title. If we click on Raw, we can see the file as it really is: plain text, with commas separating the values and the column names in the first row.

Today we want to find out how much FU Berlin paid on average for APCs, and how the prices are distributed.

Download and read the data

We import two libraries: urllib.request from the standard library to download files, and pandas, which is by convention imported under the short name pd.

PYTHON 

import urllib.request  # Module that helps with opening URLs
import pandas as pd

PYTHON 

# Set the source URL
url = "https://raw.githubusercontent.com/OpenAPC/openapc-de/master/data/fuberlin/APC_FU_Berlin_2015.csv"
apc_file = "openapc.csv"

PYTHON 

# Download the data and store them locally
urllib.request.urlretrieve(url, apc_file)

The file openapc.csv should now appear in the JupyterLab file browser. Next, we read it into a DataFrame with pd.read_csv():

PYTHON 

apcs = pd.read_csv(apc_file)

Inspect the DataFrame

If the last line of a cell is a DataFrame, Jupyter displays it as a nicely formatted table. For long tables, only the first and last rows are shown. Use .head() to look at the first five rows:

PYTHON 

apcs.head()

OUTPUT 

  institution  period     euro  ...  ut  url  doaj
0   FU Berlin    2015  2000.00  ... NaN  NaN  True
1   FU Berlin    2015  1503.12  ... NaN  NaN  True
2   FU Berlin    2015  2000.00  ... NaN  NaN  True
3   FU Berlin    2015  2000.00  ... NaN  NaN  True
4   FU Berlin    2015  1805.00  ... NaN  NaN  True

[5 rows x 17 columns]

PYTHON 

# Change the maximum number of rows displayed to 100
pd.set_option("display.max_rows", 100)
apcs

PYTHON 

# Get the number of rows and columns
apcs.shape

OUTPUT 

(61, 17)

PYTHON 

# Show the column names
apcs.columns

OUTPUT 

Index(['institution', 'period', 'euro', 'doi', 'is_hybrid', 'publisher',
       'journal_full_title', 'issn', 'issn_print', 'issn_electronic',
       'license_ref', 'indexed_in_crossref', 'pmid', 'pmcid', 'ut', 'url',
       'doaj'],
      dtype='object')

Select columns and calculate statistics

A single column is selected with square brackets and the column name. The result is a pandas Series.

PYTHON 

apcs["euro"]

OUTPUT 

0     2000.00
1     1503.12
2     2000.00
3     2000.00
4     1805.00
       ...
56    1731.45
57    1600.00
58     219.84
59    1386.35
60    2000.00
Name: euro, Length: 61, dtype: float64

A Series has methods to calculate statistics:

PYTHON 

# Maximum value of a single column
apcs["euro"].max()

OUTPUT 

np.float64(2000.0)

PYTHON 

# Minimum value of a single column
apcs["euro"].min()

OUTPUT 

np.float64(219.84)

PYTHON 

# Average value of a single column
apcs["euro"].mean()

OUTPUT 

np.float64(1419.6080327868851)

PYTHON 

# Median value of a single column
apcs["euro"].median()

OUTPUT 

np.float64(1410.07)
Callout

apcs.euro vs. apcs["euro"]

You can also access a column as an attribute, e.g. apcs.euro. This is less safe, as it fails if the column name contains spaces or has the same name as a DataFrame method. In general, spaces in column names should be avoided.

Filter rows

Comparing a column with a value returns a Series of True and False values, one for each row:

PYTHON 

# Identify cells greater or equal to a number
apcs["euro"] >= 2000

OUTPUT 

0      True
1     False
2      True
3      True
4     False
      ...
Name: euro, dtype: bool

We can use this Series inside square brackets to keep only the rows where the value is True:

PYTHON 

# Show only articles with an APC of 2,000 euros or more
apcs[apcs["euro"] >= 2000]

To display several columns, pass a list of column names:

PYTHON 

apcs[["publisher", "journal_full_title", "euro"]]

OUTPUT 

                           publisher                  journal_full_title     euro
0                    Springer Nature             npj Quantum Information  2000.00
1    American Physical Society (APS)                   Physical Review X  1503.12
2                    Springer Nature               Nature Communications  2000.00
3   Public Library of Science (PLoS)                      PLOS Pathogens  2000.00
4                    Springer Nature                       Gut Pathogens  1805.00
...
Challenge

Filtering data

  1. Filter all rows in which the "euro" value is above the average.
  2. Show only the "journal_full_title", "euro" and "publisher" columns for these entries.

PYTHON 

average_apc = apcs["euro"].mean()                 # Calculate average
over_average_apcs = apcs[apcs["euro"] > average_apc]  # Apply filter
over_average_apcs[["journal_full_title", "euro", "publisher"]]

OUTPUT 

        journal_full_title     euro                         publisher
0  npj Quantum Information  2000.00                   Springer Nature
1        Physical Review X  1503.12   American Physical Society (APS)
2    Nature Communications  2000.00                   Springer Nature
3           PLOS Pathogens  2000.00  Public Library of Science (PLoS)
4            Gut Pathogens  1805.00                   Springer Nature
...

30 of the 61 articles cost more than the average.

Plot the data

pandas can plot data directly (using the matplotlib library in the background). A histogram shows how often APCs in a certain price range occur:

PYTHON 

apcs["euro"].hist()
Histogram of article processing charges in euros, most values between 1,000 and 2,000 euros
Histogram of APCs paid by FU Berlin in 2015

PYTHON 

# Change the number of bins and the colour
apcs["euro"].hist(bins=20, color="black")  # bins = number of equal parts
Histogram of article processing charges in euros with 20 black bins
Histogram with 20 bins

value_counts() counts how often each value occurs in a column:

PYTHON 

# Count the number of articles per publisher
apcs["publisher"].value_counts()

OUTPUT 

publisher
Frontiers Media SA                                  16
Public Library of Science (PLoS)                    13
Springer Nature                                     13
Copernicus GmbH                                      3
Schweizerbart                                        3
Hindawi Publishing Corporation                       3
Scientific Research Publishing, Inc,                 2
IOP Publishing                                       1
PAGEPress Publications                               1
American Physical Society (APS)                      1
Elsevier BV                                          1
Inter-Research Science Center                        1
MDPI AG                                              1
Cogitatio                                            1
The Korean Society of Veterinary Science (KAMJE)     1
Name: count, dtype: int64

PYTHON 

# Plot this count
apcs["publisher"].value_counts().plot.bar()
Vertical bar chart of the number of articles per publisher, Frontiers Media SA has the most
Number of articles per publisher

PYTHON 

# Plot horizontally
apcs["publisher"].value_counts().plot.barh()
Horizontal bar chart of the number of articles per publisher
Number of articles per publisher, horizontal
Challenge

Save the result

Use the method .to_csv() to save over_average_apcs from the exercise above to a file called expensive_apcs.csv. Check in the JupyterLab file browser that the file was created.

PYTHON 

over_average_apcs.to_csv("expensive_apcs.csv", index=False)

index=False prevents pandas from writing the row index as an extra column.

Further reading

Key Points
  • Use import library as alias to load a library.
  • pd.read_csv() reads a CSV file into a DataFrame.
  • Use .head(), .shape and .columns to get an overview of a DataFrame.
  • df["column"] selects a column, df[["col1", "col2"]] several columns.
  • df[condition] keeps only the rows where the condition is True.
  • Series have methods for statistics (.mean(), .median(), …) and plotting (.hist(), .plot.bar(), …).

Content from Retrieving Metadata from an API

Last updated on 2026-10-19 | Edit this page

Overview

Questions

  • How can I get metadata for a DOI automatically?
  • How do I work with JSON data in Python?

Objectives

  • Explain what a web API is.
  • Retrieve metadata for a DOI from the DataCite REST API.
  • Navigate nested JSON data made of dictionaries and lists.
  • Use a loop to retrieve metadata for several DOIs.

DataCite

DataCite is a registration agency for DOIs, mainly for research data, software and other research outputs. Repositories such as Zenodo and figshare register their DOIs with DataCite.

You can search the DataCite metadata manually in DataCite Commons:

  1. Search for git in Works.
  2. Filter by Dataset.
  3. Open a result, and under Download open the metadata as DataCite JSON.

This is the same data that we will now retrieve with Python. Instead of using a web interface, programs can request data from a web service through an Application Programming Interface (API). The DataCite REST API returns the metadata for a DOI if we send a request to a URL that contains the DOI.

Request the metadata for a DOI

PYTHON 

import urllib.request
import json

base_url = "https://api.datacite.org/application/vnd.datacite.datacite+json/"
doi = "10.5281/zenodo.5563162"
full_url = base_url + doi

PYTHON 

doi_json_dataset = urllib.request.urlopen(full_url).read()

The API returns the metadata in JSON format, as text. The json library converts this text into Python data structures:

PYTHON 

doi_dataset = json.loads(doi_json_dataset)
doi_dataset

OUTPUT 

{'id': 'https://doi.org/10.5281/zenodo.5563162',
 'doi': '10.5281/ZENODO.5563162',
 'url': 'https://zenodo.org/record/5563162',
 'types': {'ris': 'DATA',
  'bibtex': 'misc',
  'citeproc': 'dataset',
  'schemaOrg': 'Dataset',
  'resourceTypeGeneral': 'Dataset'},
 'creators': [{'name': 'Banda, Juan M.',
   'givenName': 'Juan M.',
   'familyName': 'Banda',
   ...

PYTHON 

type(doi_dataset)

OUTPUT 

dict

The result is a dictionary, just like the ones we created ourselves. Its keys are the metadata fields:

PYTHON 

doi_dataset.keys()

OUTPUT 

dict_keys(['id', 'doi', 'url', 'types', 'creators', 'titles', 'publisher', 'container', 'subjects', 'contributors', 'dates', 'publicationYear', 'language', 'identifiers', 'sizes', 'formats', 'version', 'rightsList', 'descriptions', 'geoLocations', 'fundingReferences', 'relatedIdentifiers', 'relatedItems', 'schemaVersion', 'providerId', 'clientId', 'agency', 'state'])

Access the year of the publication

PYTHON 

print(doi_dataset["publicationYear"])

OUTPUT 

2021

Access the title of the publication

Some values are not simple strings or numbers, but lists or dictionaries themselves. A dataset can have several titles, so titles is a list of dictionaries:

PYTHON 

# Getting title within a list with a dictionary inside
print(doi_dataset["titles"])

OUTPUT 

[{'title': 'A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration'}]

PYTHON 

# Getting the first element out of the list, which is the dictionary
print(doi_dataset["titles"][0])

OUTPUT 

{'title': 'A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration'}

PYTHON 

# Access the key of the dictionary which is the title of the publication
print(doi_dataset["titles"][0]["title"])

OUTPUT 

A large-scale COVID-19 Twitter chatter dataset for open scientific research - an international collaboration

Access the creators of the publication

creators is also a list of dictionaries, one per creator:

PYTHON 

# Access keys from creators list
doi_dataset["creators"][0].keys()

OUTPUT 

dict_keys(['name', 'givenName', 'familyName', 'affiliation', 'nameIdentifiers'])

PYTHON 

# Access creator of the publication
print(doi_dataset["creators"][0]["name"])

OUTPUT 

Banda, Juan M.

PYTHON 

# Next creator
print(doi_dataset["creators"][1]["name"])

OUTPUT 

Tekumalla, Ramya

PYTHON 

# Access only the given name
print(doi_dataset["creators"][0]["givenName"])

OUTPUT 

Juan M.
Challenge

The publisher

Look at doi_dataset["publisher"]. What type of value is it? How do you get just the name of the publisher as a string?

PYTHON 

print(doi_dataset["publisher"])
print(type(doi_dataset["publisher"]))
print(doi_dataset["publisher"]["name"])

OUTPUT 

{'name': 'Zenodo'}
<class 'dict'>
Zenodo

The publisher is a dictionary, so we need a second pair of square brackets with the key "name".

Metadata for several DOIs

Challenge

Loop over DOIs

Use the same approach as above for a larger set of DOIs. For each DOI, print the title, the DOI and the publisher.

Bonus: also print the publication year.

PYTHON 

dois = ["10.6084/m9.figshare.155613",
        "10.6084/m9.figshare.153821.v1",
        "10.5281/zenodo.2599866",
        "10.5281/zenodo.17422255"]

PYTHON 

for doi in dois:
    doi_json_dataset = urllib.request.urlopen(base_url + doi).read()
    doi_dataset = json.loads(doi_json_dataset)
    print(doi_dataset["titles"][0]["title"])
    print("- " + doi)
    print("- " + doi_dataset["publisher"]["name"])
    print("- " + str(doi_dataset["publicationYear"]))
    print("-----------------")

OUTPUT 

git repository for paper on git and reproducible science
- 10.6084/m9.figshare.155613
- figshare
- 2013
-----------------
git can facilitate greater reproducibility and increased transparency in science
- 10.6084/m9.figshare.153821.v1
- figshare
- 2013
-----------------
Nachnutzbare Awarenessmaterialien für Forschungsdatenmanagement (FDM)
- 10.5281/zenodo.2599866
- Zenodo
- 2019
-----------------
Python Basics
- 10.5281/zenodo.17422255
- Zenodo
- 2025
-----------------

Note that publicationYear is a number, so it has to be converted with str() before it can be combined with a string.

Callout

Not every DOI is a DataCite DOI

DOIs for journal articles are usually registered with Crossref, not DataCite. If you request such a DOI from the DataCite API, you get an HTTPError: HTTP Error 404: Not Found. Crossref has its own API at https://api.crossref.org/works/<DOI>.

Optional: all creators of a publication

To get all creators, we can loop over the creators list. After the loop above, doi_dataset contains the metadata of the last DOI:

PYTHON 

for creator in doi_dataset["creators"]:
    print(creator["name"])

OUTPUT 

Förstner, Konrad U.
Müller, Rabea
Sauerwein, Till
Vandendorpe, Justine

A shorter way to create a list from an existing list is a list comprehension:

PYTHON 

creator_names = [creator["name"] for creator in doi_dataset["creators"]]
creator_names

OUTPUT 

['Förstner, Konrad U.',
 'Müller, Rabea',
 'Sauerwein, Till',
 'Vandendorpe, Justine']
Key Points
  • A web API lets programs request data from a web service via a URL.
  • urllib.request.urlopen(url).read() downloads the response of a URL.
  • json.loads() converts JSON text into Python dictionaries and lists.
  • Navigate nested data step by step: data["titles"][0]["title"].
  • DataCite registers DOIs for data and software, Crossref for most articles.