5  Diff-in-Diff

5.1 Introduction

This week we will start talking about Causal Inference in R. You will have already seen the theoretical and mathematical parts with Brenda in the lecture. This script is supposed to initiate you to the practical part of Causal Inference.

This session will show you how to :

  1. Prepare data for Causal Inference
  2. Perform a simple Diff-in-Diff analysis in R
  3. Interpret the results of a Diff-in-Diff analysis in R

5.2 Diff-in-Diff

For the first example on Diff-in-Diff I am going to use the code written by the brilliant Rohan Alexander who has written one of the (if not the) best introduction to Data Analysis in R which you can find here. Furthermore, the example he uses is a paper written by Charles Angelucci and Julia Cagé who you probably know already.

set.seed(853)

simulated_diff_in_diff <-
  tibble(
    person = rep(c(1:1000), times = 2),
    time = c(rep(0, times = 1000), rep(1, times = 1000)),
    treat_group = rep(sample(x = 0:1, size = 1000, replace = TRUE ), times = 2)
    ) |>
  mutate(
    treat_group = as.factor(treat_group),
    time = as.factor(time)
  )

simulated_diff_in_diff <-
  simulated_diff_in_diff |>
  rowwise() |>
  mutate(
    serve_speed = case_when(
      time == 0 & treat_group == 0 ~ rnorm(n = 1, mean = 5, sd = 1),
      time == 1 & treat_group == 0 ~ rnorm(n = 1, mean = 6, sd = 1),
      time == 0 & treat_group == 1 ~ rnorm(n = 1, mean = 8, sd = 1),
      time == 1 & treat_group == 1 ~ rnorm(n = 1, mean = 14, sd = 1)
    )
  )

simulated_diff_in_diff
# A tibble: 2,000 × 4
# Rowwise: 
   person time  treat_group serve_speed
    <int> <fct> <fct>             <dbl>
 1      1 0     0                  4.43
 2      2 0     1                  6.96
 3      3 0     1                  7.77
 4      4 0     0                  5.31
 5      5 0     0                  4.09
 6      6 0     0                  4.85
 7      7 0     0                  6.43
 8      8 0     0                  5.77
 9      9 0     1                  6.13
10     10 0     1                  7.32
# ℹ 1,990 more rows
simulated_diff_in_diff |>
  ggplot(aes(x = time, y = serve_speed, color = treat_group)) +
  geom_point(alpha = 0.2) +
  geom_line(aes(group = person), alpha = 0.1) +
  theme_minimal() +
  labs(x = "Time period", y = "Serve speed", color = "Person got a new racket") +
  scale_color_brewer(palette = "Set1") +
  theme(legend.position = "bottom")

The paper is called “Newspapers in Times of Low Advertising Revenues” (Angelucci and Cagé 2019) and follows a difference-in-difference analysis and it also comes with replication material which means that we can try to emulate their results.

In 1967, the French government introduced advertisements on French TV programs. This led to a decrease in advertising revenues for newspapers all over France. The idea is therefore to understand if and how the introduction of ads on TV affected the ad revenues for newspapers, both local and national ones, in France. They use a difference-in-difference approach to estimate the effect of the introduction of advertising on the advertising revenues of newspapers. Thus, the treatment is…? Exactly, the introduction of ads on TV. Angelucci and Cagé argue that national newspapers were more affected by this change than local newspapers. They have many more hypotheses which they test in this paper, but for now we will focus on this one. As mentioned above, their data is available and we can use it to replicate their results. You need to sign up, to get it.

newspapers <- read_dta("data/Angelucci_Cage_AEJMicro_dataset.dta")

Next we will have to do some minor data management. Fortunately enough, this replication material is already pretty clean and we do not have to mutate() or filter() our way around too much. The only thing we actually have to do, is to convert some variables into factors and to create a new variable which is the ratio of advertising revenue over circulation. Here I specify the argument across() again of the mutate() function and within it, I give a vector c() containing all the variables that should be transformed to factors.

Here is a brief explanation of the variables in the dataset:

  • year: The year of the observation
  • id_news: A unique identifier for each newspaper
  • local: A binary indicator (0 or 1) representing whether a newspaper is local or national (1 = local, 0 = national)
  • national: A binary indicator (0 or 1) representing whether a newspaper is national or local (1 = national, 0 = local)
  • ra_cst: The advertising revenue of the newspaper (in constant (2014) euros)
  • ps_cst: The price of subscriptions for the newspaper. This variable measures how much the newspaper charges for its subscriptions, reflecting its pricing strategy and revenue from readers
  • qtotal: The circulation of the newspaper. This variable measures the number of copies sold, reflecting the newspaper’s readership and revenue from readers
  • after_national: A binary indicator (0 or 1) for the post-treatment period: it takes on the value 1 for all years from 1967 onwards (after the introduction of television advertising) and 0 for the years before. We construct it ourselves below from the year variable
  • ra_cst_div_qtotal: A derived variable representing the advertising revenue per unit of circulation (ra_cst / qtotal). This variable is calculated to assess the efficiency or effectiveness of advertising revenue in relation to the newspaper’s circulation size
newspapers <-
  newspapers |>
  select(
    year, id_news, after_national, local, national, ra_cst, ps_cst, qtotal
    ) |> 
  mutate(ra_cst_div_qtotal = ra_cst / qtotal,
         after_national =  if_else(year >= 1967, 1, 0),
         across(c(id_news, local, national, after_national), as.factor))

newspapers
# A tibble: 1,196 × 9
    year id_news after_national local national    ra_cst ps_cst  qtotal
   <dbl> <fct>   <fct>          <fct> <fct>        <dbl>  <dbl>   <dbl>
 1  1960 1       0              1     0         52890272   2.29  94478.
 2  1961 1       0              1     0         56601060   2.20  96289.
 3  1962 1       0              1     0         64840752   2.13  97313.
 4  1963 1       0              1     0         70582944   2.43 101068.
 5  1964 1       0              1     0         74977888   2.35 102103.
 6  1965 1       0              1     0         74438248   2.29 105169.
 7  1966 1       0              1     0         81383000   2.31 126235.
 8  1967 1       1              1     0         80263152   2.88 128667.
 9  1968 1       1              1     0         87165704   3.45 131824.
10  1969 1       1              1     0        102596384   3.28 132417.
# ℹ 1,186 more rows
# ℹ 1 more variable: ra_cst_div_qtotal <dbl>

5.2.1 Inspecting your data

One of the first things you can do, is to plot your data points and see if something stands out. This might serve as a first indicator of anything that concerns the parallel trends assumption for example. The code below plots the development of advertising revenue for French newspapers in a given year. The panels are divided into local newspaper or national newspaper. Remember that our control group are local newspapers and the treatment group are the national ones. Essentially, we would expect that before the intervention, both groups should have parallel trajectories in their outcomes. This is the parallel trends assumption. If the parallel trends assumption holds, the difference between the treatment and control group should be constant over time. If the parallel trends assumption is violated, the DiD estimator will be biased.

newspapers |>
  mutate(type = if_else(local == 1, "Local", "National")) |>
  ggplot(aes(x = year, y = ra_cst)) +
  geom_point(alpha = 0.5) +
  scale_y_continuous(
    labels = dollar_format(
      prefix = "$",
      suffix = "M",
      scale = 0.000001)) +
  labs(x = "Year", y = "Advertising revenue") +
  facet_wrap(vars(type), nrow = 2) +
  theme_minimal() +
  geom_vline(xintercept = 1966.5, linetype = "dashed")

In the top panel/the local newspapers, the revenue data points are quite dense and cluster at regular intervals, suggesting that local newspapers had relatively stable advertising revenues year-to-year. There’s no obvious trend or shift in revenue around the dashed vertical line, which likely represents the year 1967, the year when television advertising was introduced in France.

The distribution of data points in the lower panel showing the observations of national newspapers is less dense compared to local newspapers, which could suggest greater variability in the advertising revenues of national newspapers. There appears to be a change around the dashed vertical line at 1967. After this year, there seems to be a wider spread of data points, including some years with significantly lower advertising revenues compared to previous years. This could indicate that national newspapers were more impacted by the introduction of television advertising. Overall, the graph suggests that the introduction of television advertising in 1967 may have had a differential impact on local and national newspapers, with national newspapers possibly experiencing greater negative effects on their advertising revenue.

However, just because some face validity seems to indicate that there is a difference in the treatment and control group, does not mean that it actually is in the data. We need to test this with a model that we will construct in the next section.

5.2.2 Building the DiD model

This here is the regression formula for the DiD analysis, including the interaction term in which we specify the treatment. The treatment is the introduction of ads on TV and the interaction term is the product of the treatment and the national status of the newspaper

\[ \ln(\mathrm{ra\_cst}) = \beta_0 + \beta_1 \mathrm{national} + \beta_2 \mathrm{after\_national} + \beta_3(\mathrm{national} \times \mathrm{after\_national}) + \beta_4 \mathrm{year} + \alpha_i + \epsilon \]

  • ln(ra_cst): The natural logarithm of advertising revenue for a newspaper. Log transformation is often used in economic data to help normalize the distribution of skewed variables and to interpret the coefficients in terms of percentage changes

  • national: This is a dummy variable indicating whether a newspaper is national (1) or local (0). This variable distinguishes the treatment group from the control group.

  • afternational: A dummy variable indicating the time period after the introduction of television advertising in 1967 (1 for years 1967 and later, 0 for earlier years). It captures the before-and-after comparison.

  • national*after_national: The interaction term between national and after_national. This term is crucial for DiD analysis as it estimates the differential effect of the introduction of television advertising on national newspapers compared to local newspapers over time

  • year: A continuous variable representing the year of observation. Including this allows controlling for linear time trends that affect all newspapers

  • alpha: Newspaper fixed effects that control for all unobserved, time-invariant differences between newspapers

  • epsilon: The error term

This is the code to specify exactly this. Note that in R national * after_national includes three variables due to the * operator. It specifies the main effect of national, the main effect of after_national and the interaction effect national:after_national, representing the differential impact of the post-television advertising period specifically on national newspapers relative to local ones. If we had only put in national:after_national, we would have only included the interaction term and would have had to add the main effects manually. This means that the formula log(ra_cst) ~ national*after_national + ... is shorthand for log(ra_cst) ~ national + after_national + national:after_national + ....

newspaper_did_model <-
  lm(log(ra_cst) ~ national*after_national + year + id_news,
     data = newspapers)

You can see that this is as straightforward as what we have already done in Session 1. It is a simple lm() and an interaction effect; nothing more…

5.2.3 Interpreting the DiD model results

Here, I am showing you a different way of displaying models in R/Quarto. I am making use of the fact that my quartobook is rendered to html. I am using the modelsummary package to display the results of the model in a nice table. Similar to the stargazer package, you can change almost every aspect of the table. I am using the coef_map argument to rename the coefficients in the table. A convenient side effect of coef_map is that any coefficient which is not listed in it is dropped from the table – which takes care of the many fixed effects of the individual newspapers. We are not interested in them and they would have cluttered the table with one coefficient per newspaper. This package is maybe a bit more advanced but also more versatile.

model_coefs <- c(
  `national1` = "National Newspapers",
  `after_national1` = "Period After TV Ads",
  `national1:after_national1` = "Interaction Effect (National * After TV Ads)",
  `year` = "Year"
)

modelsummary(newspaper_did_model,
             title = "Difference-in-Differences Model Summary",
             # coef_map renames the listed coefficients and drops all others
             # (here: the newspaper fixed effects)
             coef_map = model_coefs,
             stars = TRUE
             )
Model matrix is rank deficient. Parameters `id_news34689` were not
  estimable.
Difference-in-Differences Model Summary
(1)
+ p < 0.1, * p < 0.05, ** p < 0.01, *** p < 0.001
National Newspapers -1.039***
(0.078)
Period After TV Ads -0.001
(0.022)
Interaction Effect (National * After TV Ads) -0.228***
(0.032)
Year 0.046***
(0.003)
Num.Obs. 1052
R2 0.985
R2 Adj. 0.984
AIC 36169.6
BIC 36581.2
Log.Lik. 345.343
RMSE 0.17

This table reports the results from a difference-in-differences (DiD) regression model, which examines the impact of television advertising on the advertising revenues of national newspapers compared to local newspapers over time. The intercept we can disregard in this case. As a reminder, it is the expected value when all IVs are set to 0 – which does not really make sense in our case, especially due to the year variable (there are no newspapers in the year 0). The coefficients of all the fixed effects for the individual newspapers (contained in the factor variable id_news) are dropped from the table by coef_map because we are not substantively interested in them.

  • National Newspapers: The coefficient for national newspapers is -1.039 and is highly statistically significant (p < 0.001). This suggests that, holding other factors constant, the log of advertising revenue for national newspapers is, on average, 1.039 units lower than for local newspapers – roughly \(e^{-1.039} - 1 \approx -65\%\). National newspapers thus already operated at much lower advertising revenues than local ones, independently of the treatment.

  • Period After TV Ads: The coefficient for the period after the introduction of television advertisements is very small and not statistically significant (-0.001, p > 0.1). This indicates that the post-1967 period, for the control group of local newspapers, is not associated with a level shift in advertising revenues once we account for the general time trend.

  • Year: The coefficient for the year variable is positive and significant (0.046, p < 0.001), indicating that there is a general positive trend in advertising revenue over time across all newspapers in the sample – about 4.7% per year.

  • Interaction Effect (National * After TV Ads): This is the center piece of our model since we try to estimate the causal effect through this interaction term. It is significant and negative (-0.228, p < 0.001). In DiD analyses, this interaction term captures the differential impact of the treatment (here, the introduction of television ads) on the treated group (national newspapers). The negative sign suggests that after television advertising started, national newspapers experienced a significant decrease in advertising revenue relative to local newspapers – about \(e^{-0.228} - 1 \approx -20\%\). This term captures the essence of the DiD strategy: the relative effect post-treatment for the group of interest!

5.3 References