4  Regression Discontinuity Design

4.1 Causal Inference References

I have mentioned some references on Causal Inference in class that I find extremely useful and that have been an inspiration to my scripts.

The big ones out there that have helped me a lot are:

  • The Effect by Huntington-Klein (n.d.)
  • The Book of Why by Pearl and Mackenzie (2018)
  • The Causal Mixtape Cunningham (2021)

4.2 Regression Discontinuity Design

This session is all about Regression Discontinuity Design (RDD). RDDs are useful when we have a treatment that is assigned discontinuously creating a cutoff or a jump. It creates a divide within a sample at that said cutoff and introduces a treatment. On one side of the cutoff, we have people (observations) which have been treated, and on the other side, people that have not been treated. Most treatments are binary, i.e. can either be 0 or 1.

The variable that creates this discontinuity is called the running variable . It is sometimes also refered to as the forcing variable or forcing function. All these terms refer to the same thing however. The cutoff is the moment where the running variable introduces the treatment; for example:

  • If age is your running variable, turning 21 introduces the treatment of being allowed to drink
  • Barely managing to win slightly more than 50% makes a candidate get into office

A third term that we will have to keep in mind is the bandwidth. In plain English, it simply means how many observations to the left and to the right of the cutoff we are going to compare. We assume that most individuals/observations that are very close to the cutoff are almost the same or at least very similar and thus comparable. The further away we get from the cutoff point the more noise is introduced and the more other and different variables might explain our outcome variable. As we will see later, the choice of bandwidth is highly important for RDDs and must be theoretically and empirically thoroughly discussed. But when is that not the case for any statistical decision that we make?

So to recap: In RDDs, the treatment is assigned based on whether a continuous running variable crosses a cutoff. The difference between the people just left and right of the cutoff is therefore that one side has been treated whereas the other has not. It is important to note that this running variable is somewhat inevitable. It is hard to not turn 21 in your life assuming you make it until then. We all have turned 21 at some point.

Before we get coding, one more thing: what actually identifies the causal effect here? The key identifying assumption of an RDD is the continuity assumption: in the absence of the treatment, the outcome would have evolved smoothly through the cutoff. Everything else about the units – their characteristics, other influences on the outcome – is assumed to change continuously around the threshold; the only thing that jumps at the cutoff is the treatment. If that holds, any jump we observe in the outcome at the cutoff can be attributed to the treatment. If something else also changes exactly at the cutoff (say, another policy kicks in at the same threshold), the design breaks down.

4.2.1 Once again, visualizing your data

I have been saying over and over again that it is very important that you first inspect your data and always visualize your data. RDD is actually a great use case in which visualizing the data at hand helps you identify potential discontinuities in your data.

Below I simulate both the running_variable as well as our outcome to graphically highlight the idea of RDD. Further, the tibble also includes a binary variable that highlights whether the cutoff has taken place or not.

── Attaching core tidyverse packages ──────────────────────── tidyverse 2.0.0 ──
✔ dplyr     1.2.1     ✔ readr     2.2.0
✔ forcats   1.0.1     ✔ stringr   1.6.0
✔ ggplot2   4.0.3     ✔ tibble    3.3.1
✔ lubridate 1.9.5     ✔ tidyr     1.3.2
✔ purrr     1.2.2     
── Conflicts ────────────────────────────────────────── tidyverse_conflicts() ──
✖ dplyr::filter() masks stats::filter()
✖ dplyr::lag()    masks stats::lag()
ℹ Use the conflicted package (<http://conflicted.r-lib.org/>) to force all conflicts to become errors
simulated_rdd_data_before_treatment <- 
  tibble(
    outcome = rnorm(200, mean = 1, sd = 1),
    running_variable = c(1:200),
    treatment = "before treatment"
  ) 

simulated_rdd_data_after_treatment <- 
  tibble(outcome = rnorm(200, mean = 3, sd = 1),
        running_variable = c(201:400),
        treatment = "after treatment"
  )

rbind(simulated_rdd_data_before_treatment, 
      simulated_rdd_data_after_treatment) |> 
  ggplot(aes(x = running_variable, y = outcome, color = treatment)) +
  geom_point(alpha = 0.5) +
  geom_smooth(formula = y ~ x, method = "loess") +
  theme_minimal() +
  theme(legend.position = "bottom") +
  scale_color_brewer(palette = "Set1")

Above you can easily see that there is a cutoff where running variable takes on value 201. This is only simulated data and we do not measure any specific variables yet. But visually speaking, this is something you should observe in your data. I do not want to recommend data mining in any way; it is bad practice and we should work on theoretically driven models. But if you observe such a pattern in your data, maybe there is something happening at the cutoff point that deserves our causal inference and thus RDD attention…?

4.2.2 Sharp vs. fuzzy RDD

We distinguish between two types of RDDs. Below you will find examples for sharp and fuzzy designs. A sharp RD design occurs when crossing the cutoff determines the treatment: everyone above the cutoff is treated, everyone below is not. Age-based rules are the classic case – you can hardly escape turning 21 if you are 20 years and 364 days old, and with it comes legal access to alcohol. A fuzzy RD design occurs when crossing the cutoff only changes the probability of receiving the treatment, without guaranteeing it. Think of a government program that municipalities become eligible for once the governing party holds a local majority: not every eligible municipality actually receives the program, and a few non-eligible ones might obtain it through other channels. Crossing the cutoff then makes treatment much more likely, but the jump in treatment probability is smaller than from 0 to 1.

Put in formal terms: in a sharp RDD the probability of treatment jumps from 0% to 100% at the cutoff; in a fuzzy RDD it jumps from some lower value to some higher value in between (say, from 20% to 80%).

4.2.2.0.1 Sharp RDD

Below, I am using an example by Rohan Alexander from his book Telling Stories with Data again. He reproduces a study by Carpenter and Dobkin (2015). The authors are interested in estimating the causal effect of access to alcohol on crime. They exploit the minimum legal drinking age (MLDA) as a cutoff in a regression discontinuity design: legal access to alcohol jumps discontinuously at the 21st birthday, while everything else about a person changes smoothly around it. You cannot escape turning 21 if you are 20 years and 364 days old. Their data is available here.

library(haven)

carpenter_dobkin <-
  read_dta(
    "data/carpenter_dobkin_replication.dta"
  )

carpenter_dobkin
# A tibble: 2,922 × 143
   days_to_21   all felony misdemeanor fbi_offense_miss violent murder
        <dbl> <dbl>  <dbl>       <dbl>            <dbl>   <dbl>  <dbl>
 1      -1461  5289   1556        3064              206     686      7
 2      -1460  4800   1460        2756              196     581      8
 3      -1459  4464   1356        2589              214     561      8
 4      -1458  4425   1382        2511              180     587     15
 5      -1457  4451   1423        2535              169     596      5
 6      -1456  4501   1375        2593              176     587      9
 7      -1455  4331   1341        2459              159     575     14
 8      -1454  4432   1372        2568              179     599      6
 9      -1453  4576   1401        2650              201     604      3
10      -1452  4608   1486        2577              172     609     14
# ℹ 2,912 more rows
# ℹ 136 more variables: manslaughter <dbl>, rape <dbl>, robbery <dbl>,
#   assault <dbl>, aggravated_assault <dbl>, ot_assault <dbl>, property <dbl>,
#   burglary <dbl>, larceny <dbl>, mv_theft <dbl>,
#   stolen_prop_buy_rec_poss <dbl>, vandalism <dbl>, ill_drugs <dbl>,
#   cocaine_opio_sale_manuf <dbl>, mj_sale_manuf <dbl>,
#   dang_non_narc_sale_manuf <dbl>, cocaine_opio_posses <dbl>, …
carpenter_dobkin_prepared <-
  carpenter_dobkin |>
  mutate(age = 21 + days_to_21 / 365) |>
  select(age, assault, aggravated_assault, dui, traffic_violations) |>
  pivot_longer(
    cols = c(assault, aggravated_assault, dui, traffic_violations),
    names_to = "arrested_for",
    values_to = "number"
  )

carpenter_dobkin_prepared |>
  mutate(
    arrested_for =
      case_when(
        arrested_for == "assault" ~ "Assault",
        arrested_for == "aggravated_assault" ~ "Aggravated assault",
        arrested_for == "dui" ~ "DUI",
        arrested_for == "traffic_violations" ~ "Traffic violations"
      )
  ) |>
  ggplot(aes(x = age, y = number)) +
  geom_point(alpha = 0.05) +
  facet_wrap(facets = vars(arrested_for), scales = "free_y") +
  theme_minimal()

carpenter_dobkin_dui_only <-
  carpenter_dobkin_prepared |>
  filter(
    arrested_for == "dui",
    abs(age - 21) < 2
  ) |>
  mutate(is_21_or_more = if_else(age < 21, 0, 1))

Below I am using the rdrobust() function from the rdrobust package to estimate the causal effect of the minimum legal drinking age on DUI arrests. The rdrobust() function requires the dependent variable y, the running variable x, the cutoff c, and the bandwidth h. Besides the conventional estimate, it also reports bias-corrected and robust versions of the estimate. Note that the choice of bandwidth is extremely important for the result and has to be empirically and theoretically grounded.

library(rdrobust)
rdd_dui <- rdrobust(
  # specifies the DV
  y = carpenter_dobkin_dui_only$number,
  # specifies the running variable of the RDD
  x = carpenter_dobkin_dui_only$age,
  # specifies the cutoff
  c = 21,
  # specifies the bandwidth, here the unit of analysis is years
  h = 2
)

rdd_dui |> summary()
Call: rdrobust

Sharp RD estimates using local polynomial regression.

Number of Obs.                 1459
BW type                      Manual
Kernel                   Triangular
VCE method                       NN

                               Left        Right
Number of Obs.                  729          730
Eff. Number of Obs.             729          730
Order est. (p)                    1            1
Order bias (q)                    2            2
BW est. (h)                   2.000        2.000
BW bias (b)                   2.000        2.000
rho (h/b)                     1.000        1.000
Unique Obs.                     729          730

=====================================================================
                   Point    Robust Inference
                Estimate         z     P>|z|      [ 95% C.I. ]       
---------------------------------------------------------------------
     RD Effect   192.790    17.999     0.000   [184.837 , 230.010]   
=====================================================================

The conventional point estimate suggests that immediately upon crossing the age threshold of 21, there is an estimated increase of approximately 192.79 DUI arrests. This is a substantial effect, indicating a significant jump in DUI arrests that coincides with reaching the legal drinking age. Note that the inference reported in the output (the z-statistic, p-value, and confidence interval) is the robust, bias-corrected inference: rdrobust also computes a bias-corrected version of the estimate (about 207.42, stored in rdd_dui$coef), which adjusts for the bias introduced by approximating the regression function near the cutoff, and pairs it with a larger, robust standard error that accounts for the additional uncertainty of this correction. This is why the confidence interval in the output is not centered on the conventional point estimate. Either way, the conclusion is the same: the effect is large, positive, and clearly statistically significant. A good robustness check that you should always perform in an RDD is to re-estimate the model with different bandwidths (h) and verify that the substantive conclusion does not depend on that choice.

The rdrobust package also comes with rdplot(), which produces the classic RD figure: the raw data is grouped into bins, the binned means are plotted as points, and a polynomial fit is drawn on each side of the cutoff. The discontinuity at age 21 is impossible to miss:

rdplot(
  y = carpenter_dobkin_dui_only$number,
  x = carpenter_dobkin_dui_only$age,
  c = 21,
  x.label = "Age",
  y.label = "Number of DUI arrests",
  title = ""
)

4.2.2.1 Fuzzy RDD example

Below I simulate a fuzzy regression discontinuity design. I am simulating election results in 5000 municipalities. The vote share of the governing party is drawn from a uniform distribution between 45% and 55%, so that all municipalities are reasonably close to the 50% majority threshold. The story: municipalities where the governing party holds a local majority become eligible for a government spending program worth on average 15 (in whatever currency unit you like). Crucially, eligibility does not guarantee the program: among municipalities above the threshold, only 80% actually receive it, while 20% of those below the threshold obtain it anyway (through lobbying, coalitions, or administrative quirks). This is exactly what makes the design fuzzy: crossing the cutoff raises the probability of treatment from 20% to 80%, but not from 0 to 1.

set.seed(2025) # so that the simulated numbers are reproducible

n_municipalities <- 5000

election_data <- tibble(
  municipality = c(1:n_municipalities),
  # vote share around the 50% majority threshold
  vote_share = runif(n_municipalities, min = 45, max = 55),
  # base government spending
  spending = rnorm(n_municipalities, mean = 100, sd = 10)
) |>
  mutate(
    # crossing the threshold raises the probability of receiving the
    # program from 20% to 80% -- it does NOT guarantee it (fuzzy!)
    prob_treatment = if_else(vote_share >= 50, 0.8, 0.2),
    treated = rbinom(n_municipalities, size = 1, prob = prob_treatment),
    # the program is worth 15 on average -- but only for those actually treated
    extra_spending = rnorm(n_municipalities, mean = 15, sd = 5),
    spending = if_else(treated == 1, spending + extra_spending, spending),
    # eligibility indicator: above/below the majority threshold
    majority = if_else(vote_share >= 50, 1, 0)
  )

# the fuzzy first stage in numbers: the share of treated municipalities
# jumps at the cutoff, but not from 0 to 1
election_data |>
  group_by(majority) |>
  summarise(share_treated = mean(treated))
# A tibble: 2 × 2
  majority share_treated
     <dbl>         <dbl>
1        0         0.199
2        1         0.797
# Visualizing the discontinuity in the outcome
election_data |>
  ggplot(aes(x = vote_share, y = spending)) +
  geom_point(alpha = 0.2) +
  geom_smooth(
    data = filter(election_data, vote_share < 50),
    method = "lm",
    color = "blue",
    formula = y ~ poly(x, 2) # Polynomial regression for smoother line
  ) +
  geom_smooth(
    data = filter(election_data, vote_share >= 50),
    method = "lm",
    color = "red",
    formula = y ~ poly(x, 2) # Polynomial regression for smoother line
  ) +
  theme_minimal() +
  labs(x = "Vote Share (%)", y = "Government Spending")

Notice that the jump in spending at the cutoff looks smaller than 15. That is exactly what we expect in a fuzzy design: only some municipalities change their treatment status at the threshold, so the discontinuity in the outcome mixes treated and untreated units on both sides.

Here is our RD analysis using the rdrobust package again. We specify the cutoff point c = 50 and a bandwidth of h = 1. The key novelty is the fuzzy = argument, to which we pass the actual treatment status (treated). Under the hood, rdrobust then rescales the estimate: the estimated treatment effect is the jump in the outcome at the cutoff divided by the jump in the treatment probability at the cutoff (the latter is called the first stage). Intuitively, since only part of the municipalities change their treatment status at the threshold, the raw jump in spending understates the effect of the program itself – dividing by the first stage corrects for this. The result is a local average treatment effect (LATE) for the municipalities whose treatment status is actually moved by crossing the threshold (the so-called compliers).

library(rdrobust)

rd_result <-
  rdrobust(y = election_data$spending,
           x = election_data$vote_share,
           c = 50,
           fuzzy = election_data$treated,
           h = 1)

rd_result |> summary()
Call: rdrobust

Fuzzy RD estimates using local polynomial regression.

Number of Obs.                 5000
BW type                      Manual
Kernel                   Triangular
VCE method                       NN

                               Left        Right
Number of Obs.                 2527         2473
Eff. Number of Obs.             523          520
Order est. (p)                    1            1
Order bias (q)                    2            2
BW est. (h)                   1.000        1.000
BW bias (b)                   1.000        1.000
rho (h/b)                     1.000        1.000
Unique Obs.                    2527         2473

First-stage estimates.

=====================================================================
                   Point    Robust Inference
                Estimate         z     P>|z|      [ 95% C.I. ]       
=====================================================================
     Rd Effect     0.638     6.967     0.000     [0.395 , 0.705]     
=====================================================================

Treatment effect estimates.

=====================================================================
                   Point    Robust Inference
                Estimate         z     P>|z|      [ 95% C.I. ]       
---------------------------------------------------------------------
     RD Effect    10.960     2.676     0.007     [2.316 , 15.001]    
=====================================================================

The output now has two parts. The first-stage estimate (about 0.64) is the jump in the probability of treatment at the cutoff – reassuringly close to the 0.6 (= 0.8 - 0.2) that we built into the simulation. The treatment effect estimate is about 11, with a robust p-value below 0.05 and a robust confidence interval that includes the true effect of 15 which we simulated (right at its upper end – estimates from a single simulated sample scatter around the truth, and here we drew a sample whose estimate came out somewhat below it). Quick reminder that P>|z| is the column with the p-values. We reject the null hypothesis that there is no discontinuity in spending at the majority threshold: crossing the 50% vote share threshold significantly increases spending for the complier municipalities.

4.2.2.2 Replicating the sharp RD estimate by hand

You might be wondering what the rdrobust package actually does. It makes our life easier because estimating an RD by hand using a linear model is not very straightforward. Below I show you a simplified version. To keep it simple, we replicate the sharp version of the estimate on our simulated data: the effect of crossing the threshold (being eligible) on spending, regardless of actual treatment status. This quantity is also known as the intent-to-treat (ITT) effect, and it is what rdrobust estimates when we leave out the fuzzy = argument:

rd_sharp <-
  rdrobust(y = election_data$spending,
           x = election_data$vote_share,
           c = 50,
           h = 1)

rd_sharp |> summary()
Call: rdrobust

Sharp RD estimates using local polynomial regression.

Number of Obs.                 5000
BW type                      Manual
Kernel                   Triangular
VCE method                       NN

                               Left        Right
Number of Obs.                 2527         2473
Eff. Number of Obs.             523          520
Order est. (p)                    1            1
Order bias (q)                    2            2
BW est. (h)                   1.000        1.000
BW bias (b)                   1.000        1.000
rho (h/b)                     1.000        1.000
Unique Obs.                    2527         2473

=====================================================================
                   Point    Robust Inference
                Estimate         z     P>|z|      [ 95% C.I. ]       
---------------------------------------------------------------------
     RD Effect     6.991     1.932     0.053    [-0.067 , 9.190]     
=====================================================================

You will see that our lm() version below does not generate exactly the same results, and I will discuss the reasons for that afterwards. As a rule of thumb, I recommend doing all these things using the rdrobust package while being aware of what it does under the hood and what things to look out for. Hence, the explanation by hand.

To replicate the sharp rdrobust estimate using lm(), we need to manually estimate a local linear regression. The key components to match are:

  • the bandwidth (I take the same bandwidth as above): we limit the data to a ±1 window around the threshold (vote_share = 50%)
  • separate regression lines on both sides of the cutoff: rdrobust fits a separate local linear regression on each side. In a single lm() call, we achieve this with an interaction between the treatment dummy and the centered running variable – the interaction allows the slope to differ on the two sides of the cutoff
  • the treatment effect: the coefficient of interest is the coefficient of majority, i.e. the discrete jump in spending at vote_share = 50%

Below, I filter vote_share to keep only observations within the ±1 bandwidth, and centered_vote is created to measure the distance from the threshold. majority is the eligibility dummy that takes on 1 if vote_share is equal to or bigger than 50, otherwise it is 0.

# Define the bandwidth
h <- 1

# Subset data within the bandwidth around the threshold
rd_data <- election_data |>
  filter(abs(vote_share - 50) <= h) |>
  # create the running variable centered at the threshold
  mutate(centered_vote = vote_share - 50)

# Estimate the local linear regression WITH the interaction, so that
# the slope may differ on both sides of the cutoff
rd_model <- lm(spending ~ majority * centered_vote, data = rd_data)

# Output results
summary(rd_model)

Call:
lm(formula = spending ~ majority * centered_vote, data = rd_data)

Residuals:
    Min      1Q  Median      3Q     Max 
-36.242  -8.292  -0.103   7.824  41.360 

Coefficients:
                       Estimate Std. Error t value Pr(>|t|)    
(Intercept)            102.7229     1.0438  98.414  < 2e-16 ***
majority                 7.8160     1.4669   5.328 1.21e-07 ***
centered_vote           -0.5436     1.8647  -0.291    0.771    
majority:centered_vote   2.2676     2.5961   0.873    0.383    
---
Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Residual standard error: 12.02 on 1039 degrees of freedom
Multiple R-squared:  0.1099,    Adjusted R-squared:  0.1073 
F-statistic: 42.75 on 3 and 1039 DF,  p-value: < 2.2e-16

The majority coefficient (about 7.8) is close to, but not identical with, the conventional sharp estimate from rdrobust() (about 7.0). Why don’t the two match exactly? Because rdrobust does two additional things that our plain lm() does not. First, it uses triangular kernel weighting: observations closer to the cutoff receive more weight, observations further away receive less – our lm() implicitly weights all observations within the bandwidth equally (a uniform kernel). Second, the inference differs: rdrobust reports bias-corrected estimates with robust standard errors, which account for the bias of approximating the regression function near the cutoff, while lm() reports conventional standard errors that are not robust to heteroskedasticity. Also note that both of these numbers estimate the intent-to-treat effect of eligibility – they are smaller than the fuzzy LATE estimate above, because they average over municipalities that did and did not actually receive the program.

4.2.2.2.1 Other robustness checks to consider

To assess if the estimated effect of crossing a specific threshold (e.g., the legal drinking age on DUI arrests) is robust and not sensitive to the choice of bandwidth in a Regression Discontinuity (RD) design, we can employ several robustness checks. These checks help ensure that the findings are not artifacts of particular specifications but reflect a genuine causal relationship. Here are some key strategies:

  • Varying Polynomial Orders: The choice of polynomial order for fitting the regression on either side of the cutoff can influence the results. Testing the model with different polynomial orders and checking for consistency in the estimated effect size can help ensure that the results are not sensitive to this choice.

  • Varying Bandwidths: Conduct the analysis using a range of bandwidths around the cutoff point. This involves re-estimating the model with different bandwidth sizes to observe if the estimated effect significantly changes. rdrobust() chooses the optimal bandwidth automatically. However, this does not mean that we should blindly trust it! If the estimated effect remains stable across a variety of bandwidths, this suggests that the findings are robust to the choice of bandwidth. It is crucial to demonstrate that the effect is stable across different choices of bandwidths. A common method is to visualize how the effect changes as the bandwidth varies, ranging from half to twice the optimal bandwidth size. The best case scenario would be that the Local Average Treatment Effect that you are estimating does not change much.

  • Placebo Tests or Fake Cutoffs: Applying the same RD design to points away from the actual cutoff can serve as a placebo test. If significant effects are detected at these placebo cutoffs, it might suggest that the observed effect at the real cutoff could be due to other factors rather than the policy or intervention being studied. Another way to approach placebo tests would be to show that there is no effect on a number of “fake” outcomes such as a spatial or temporal variables.

You also want to check for sorting at the cutoff. Regression discontinuity designs live and die with the fact that observations left and right of the cutoff are nearly identical and that the only thing that jumps at the cutoff is the probability of the treatment. If however our observations are somehow able to trick themselves either left or right of the cutoff, this would violate this basic assumption of RDDs. You could run a so called McCrary test. Or simply use Cattaneo et al. (2018) estimator to check and account for these things. The package in R would be the rddensity package.

4.2.3 Named Entity Recognition (OPTIONAL!!!)

This is the first time we are really entering the realm of text-as-data methods. The web scraping script in the appendix is actually the first preliminary stage, but this is the first time we will really work with textual statistics. I must have mentioned a few times in class that this part of quantitative methods fascinates me the most and I use these methods on a daily basis. Unfortunately, I won’t have the time to give you a great introduction to it here. You are more than welcome to join the course Malo and I organize on Computational Social Sciences between the 3rd and 4th semester in January!

In this tutorial I will introduce you to Named Entity Recognition (NER). NER is a task from the field of Natural Language Processing (NLP). It is about detecting entities in texts and classifying them according to certain categories. These entities can be places, organizations, persons/names, data or miscellaneous. Strictly speaking, there are two NLP tasks that NER can perform for us: firstly, detecting entities in texts and then, in a second step, classifying them according to predefined categories.

As is so often the case with NLP tasks, this method also works with pre-trained models. There are many NER models, but this script will deal with spaCy; simply because there is an R package for it. NER models are pretrained in the sense that they have seen a human annotated corpus of text before, where the entities have all been detected and classified by hand. In short, the model learns what to look for and what to classify and how once it has been found. These models can be refined later, but for this tutorial we will work with off-the-shelf models as provided by the developers.

4.2.3.1 Installation of spaCy

First, we need to install the spacyr package. This package is an R wrapper for the spaCy NLP library which is usually running in Python. And as a general disclaimer, this might be tricky step – after this, everything will be easy. To be able to do NER in R, we will have to do three things:

  1. Install Python on your computer
  2. Install the reticulate package in R
  3. Install the spacyr package in R

As we need to have a Python environment running, we will have to install Python on your computer first. You can download Python here. Download the latest version of Python and install it. Next, you will have to install the reticulate library in R; here is an in-depth tutorial on how to do this. Here is the next thing, you might want to take a look at which is the spacyr library. You can find the documentation here. For a tutorial on how to use it (that does not necessarily go beyond the documentation), you can check out this blog post.

If you managed to install all three things, these lines of code should run smoothly:

4.2.3.2 Installation of spaCy models

As I mentioned above, Named Entity Recognition relies on pre-trained models that have been trained on a human annotated corpus. These models are not part of the spacyr package, but you can download them from the spaCy website. The spacyr package provides a function to download these models. You can download the English model with the following code:

spacyr::spacy_download_langmodel("en_core_web_lg")

4.2.3.3 What is a corpus?

Before we start, a few words about the corpus we will be working with. We will use the State of the Union corpus, which comes with the quanteda package. quanteda is one of the leading and most important packages in R when it comes to textual statistics. I won’t spend much time explaining the idea of corpora, but here are a few key points:

To use any TaDa methods, we must construct a corpus. You could compare it to the sampling we do from a population of interest. Generally speaking your corpus can be anything as long as it makes sense (very broad statement, I know) but it should at least be useful depending on your dependent variable or guiding research question. A corpus is never value free. This is a key-point. The written text of a society is a certain reflection of society itself. This means that marginalized groups of any given society are also very likely to be marginalized in text. Textual data captures societal structure (structuralism and so on…). We must care about ethics! A corpus must be validated (and validated, and validated again). Ideally we should have the entirety of a corpus. In our example, we are looking at the State of the Union addresses. This is a very specific corpus. We could also look at all speeches by a certain president, all speeches by a certain party, all speeches of a certain type (inaugural addresses, farewell addresses, etc.). But you should definitely justify why we are looking at this specific corpus, this specific set of speeches. Exhaustiveness might be a good rule of thumb. Why stop at only looking at the SOTU from the 1970s on?

4.2.3.4 How to construct a corpus?

We can always do things by hand…

… or we automate it ;) For this I recommend that you go check out the appendix chapter on webscraping but let me tell you that the beautiful Internet is full of corpora which are waiting to be retrieved (scraped), cleaned, and used for text as data! Some might already be available. Party manifestos can be retroengineered from the Comparative Manifesto Project. The State of the Union corpus is available within the quanteda package. Plenty of parliamentary speeches can be found here.

But you can get as creative as you want! You can scrape entire newspaper archives or other digital trace data from the web. Or let’s say you are interested in lyrics of songs. Why not scrape https://genius.com/? Any text that is available on the web can be scraped and used as a corpus!

4.2.3.5 Initializing a spaCy session

Load both the reticulate and the spacyr library to your environment.

Next, we are going to initialize the spaCy session. R will go look for a python environment installed on your computer that it can use to do our NER. You need to specify which spaCy model you would like to use in your session.

spacy_initialize(model = "en_core_web_lg")

It reads as follows: en_ indicates that we are working with a model trained on English text, and core_ means that it could technically detect more than just entities, web_ means that it was trained on web text, and lg means that it is a large model. Other available models in English are en_core_web_sm (small model) and en_core_web_md (medium model). The bigger the model, the better the accuracy of the NER. 1 If you want to use a different model, you can check out this list here of the official spaCy website and see whether your language of interest is available. 2

You always have to finalize your spacy session at the end!

4.3 References

Abou-Chadi, Tarik. 2016. “Niche Party Success and Mainstream Party Policy Shifts – How Green and Radical Right Parties Differ in Their Impact.” British Journal of Political Science 46 (2): 417–36. https://doi.org/10.1017/S0007123414000155.
Adams, James, and Zeynep Somer-Topcu. 2009. “Policy Adjustment by Parties in Response to Rival Parties’ Policy Shifts: Spatial Theory and the Dynamics of Party Competition in Twenty-Five Post-War Democracies.” British Journal of Political Science 39 (4): 825–46. https://doi.org/10.1017/S0007123409000635.
Angelucci, Charles, and Julia Cagé. 2019. “Newspapers in Times of Low Advertising Revenues.” American Economic Journal: Microeconomics 11 (3): 319–64. https://doi.org/10.1257/mic.20170306.
Beck, Nathaniel, and Jonathan N. Katz. 1995. “What to Do (and Not to Do) with Time-Series Cross-Section Data.” The American Political Science Review 89 (3): 634–47. https://doi.org/10.2307/2082979.
Budge, Ian, and Dennis Farlie. 1983. Explaining and Predicting Elections: Issue Effects and Party Strategies in Twenty-Three Democracies. Allen & Unwin.
Carpenter, Christopher, and Carlos Dobkin. 2015. “The Minimum Legal Drinking Age and Crime.” The Review of Economics and Statistics 97 (2): 521–24. https://doi.org/10.1162/REST_a_00489.
Cattaneo, Matias D, Michael Jansson, and Xinwei Ma. 2018. “Manipulation Testing Based on Density Discontinuity.” The Stata Journal 18 (1): 234–61.
Cunningham, Scott. 2021. Causal Inference: The Mixtape. Yale University Press.
Dinas, Elias, and Kostas Gemenis. 2010. “Measuring Parties’ Ideological Positions with Manifesto Data: A Critical Evaluation of the Competing Methods.” Party Politics 16 (4): 427–50. https://doi.org/10.1177/1354068809343107.
Gemenis, Kostas. 2013. “What to Do (and Not to Do) with the Comparative Manifestos Project Data.” Political Studies 61 (April): 23–43. https://doi.org/10.1111/1467-9248.12015.
Green, Jane, and Sara Hobolt. 2008. “Owning the Issue Agenda: Party Strategies and Vote Choices in British Elections.” Electoral Studies 27 (3): 460–76. https://doi.org/10.1016/j.electstud.2008.02.003.
Huntington-Klein, Nick. n.d. The Effect: An Introduction to Research Design and Causality the Effect. Accessed April 10, 2024. https://theeffectbook.net/.
King, Gary, Michael Tomz, and Jason Wittenberg. 2000. “Making the Most of Statistical Analyses: Improving Interpretation and Presentation.” American Journal of Political Science 44 (2): 347–61. https://doi.org/10.2307/2669316.
Nickell, Stephen. 1981. “Biases in Dynamic Models with Fixed Effects.” Econometrica 49 (6): 1417–26. https://doi.org/10.2307/1911408.
Pearl, Judea, and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause and Effect. 1st ed. Basic Books.
Plique, Guillaume, Pauline Breteau, Jules Farjas, et al. 2024. Minet, a Webmining CLI Tool & Library for Python. V. 1.3.2. Zenodo, released January 17. https://doi.org/10.5281/ZENODO.4564399.
Ruedin, Didier, and Laura Morales. 2019. “Estimating Party Positions on Immigration: Assessing the Reliability and Validity of Different Methods.” Party Politics 25 (3): 303–14. https://doi.org/10.1177/1354068817713122.

  1. There is technically also a en_core_web_trf model which is a so-called transformer model. This model is not available in the spacyr package and we would have to use it in Python. In my experience so far, the large model suffices for most purposes!↩︎

  2. Otherwise, you can always try to find a model on huggingface.co. In that case, you will have to code in Python though.↩︎