Figure 1 illustrates the main results by summarizing raw data means by treatment status for selected individual outcomes in a series of bar charts (all tests are two-sided, similar graphs for all individual outcomes are presented in Supplementary Information section 2.1). The top row presents data for participants who were initially on the chronological feed. Among these participants, those assigned to switch to the algorithmic feed were 5.2 percentage points less likely to reduce their X usage than those who remained on the chronological feed (95% confidence interval (CI): 0.7, 9.7; P = 0.024). They were 4.7 percentage points more likely to prioritize policy issues considered important by Republicans, such as inflation, immigration and crime (95% CI: 0.7, 8.7; P = 0.023). They were also 5.5 percentage points more likely to believe that the investigations into Trump are unacceptable, describing them as contrary to the rule of law, undermining democracy, an attempt to stop the campaign and an attack on people like themselves (95% CI: 0.8, 10.2; P = 0.022). They were 7.4 percentage points less likely to hold a positive view of Ukrainian President Volodymyr Zelensky (95% CI: 1.8, 13.0; P = 0.009). Finally, they were 3.7 and 2.3 percentage points more likely to follow any conservative account (95% CI: 0.5, 7.0; P = 0.025) and any political activist account (95% CI: −0.1, 4.8; P = 0.061) on X, respectively. The bottom row of Fig. 1 presents the data summaries of the same outcomes for participants who were initially on the algorithmic feed. In sharp contrast to the top row, the outcome means in this group were not significantly affected by treatment assignment.
Fig. 1: Raw data summary for the main outcomes by initial feed setting and treatment. Means for selected outcomes by assigned treatment feed in the two samples: top, summarized outcomes for the sample of users initially on the chronological feed; bottom, summarized outcomes for the sample of participants initially on the algorithmic feed. The difference in the height of the two bars in each chart represents the unconditional ITT effect. Error bars, 95% CI for the ITT effect relative to the control mean (Control mean + ITT ± 1.96 × s.e.). The title of each chart presents the treatment-effect magnitude in percentage points (pp) and its P value (all tests are two-sided). The unit of observation is respondent. Sample sizes for each subgroup are reported below the bars. All outcomes are indicator variables except ‘Shares Republican priorities’, for which the minimum, first quartile, median, third quartile and maximum are: 0, 0, 0.33, 0.67, 1 for initial-Chronological to Chronological subsample; 0, 0, 0.33, 0.67, 1 for initial-Chronological to Algorithmic; 0, 0, 0.33, 0.67, 1 for initial-Algorithmic to Algorithmic; and 0, 0.33, 0.33, 0.67, 1 for initial-Algorithmic to Chronological. Supplementary Information section 2.1 presents similar bar charts for all individual outcome variables. Supplementary Information section 2.4 presents regression results for all outcome variables. Source data Full size image
Regression results using aggregate outcomes are presented in Fig. 2, which displays the results of regression analysis for the composite survey outcomes. Average intention-to-treat (ITT) effects are reported (Methods). On the left, we show the standardized effects of switching the algorithm on for participants initially using the chronological feed; on the right are shown the effects of switching it off for those initially using the algorithmic feed. For each outcome, we report two sets of regression estimates: unconditional, controlling only for the initial feed setting and the pre-treatment value of the outcome variable when available; and conditional, controlling for all collected pre-treatment covariates, with the specification chosen using generalized random forests (GRFs)39.
Fig. 2: ITT estimates of feed-setting changes on engagement and political attitudes. ITT effect estimates of switching the algorithm on and off (in s.d.). Left, effect of moving from the chronological to the algorithmic feed for users initially on the chronological feed. Right, effect of moving in the opposite direction for users initially on the algorithmic feed. For each outcome, the results of two specifications are reported. Blue, unconditional estimates with robust s.e., controlling only for the initial feed setting and, where applicable, pre-treatment outcome levels. Orange: conditional estimates, controlling for pre-treatment covariates using GRFs; 90% and 95% CIs are reported. Numerical effect sizes and P values correspond to the conditional estimates (all tests are two-sided). The unit of observation is respondent. From top to bottom, sample sizes are n = 4,965, n = 3,337, n = 4,965, n = 4,965, n = 4,596, n = 4,596 and n = 4,850. Tests are described in Methods. Supplementary Information Table 2.16 reports the exact numerical point estimates, s.e., CIs and sample sizes for every specification. All outcomes are standardized. Additional results are presented in Supplementary Information section 2. PCA, first principal component from principal component analysis. Source data Full size image
First, we confirmed that the algorithmic feed is more engaging. Among participants initially on the chronological feed, the first principal component of the user engagement measures increased by 0.14 s.d. for those assigned to the algorithmic feed, compared with those who remained on the chronological feed (95% CI: 0.03, 0.25; P = 0.014). For participants initially on the algorithmic feed, engagement declined by 0.06 s.d. for those assigned to the chronological feed, but this effect is not significant at the 5% level (95% CI: −0.12, 0.01; P = 0.081). Second, we find precisely estimated null effects of the algorithm on partisanship and affective polarization, whether the algorithm was switched on or off.
Outcomes related to attitudes towards policies and current political events were affected significantly by turning the algorithm on. Participants who switched from the chronological to the algorithmic feed prioritized a conservative policy agenda over a liberal one by 0.11 s.d., compared with those who remained on the chronological feed (95% CI: 0.02, 0.20; P = 0.016). The experiment also influenced views on the criminal investigations into Trump: those who switched to the algorithmic feed were 0.08 s.d. more likely to consider them completely unacceptable (95% CI: 0.01, 0.16; P = 0.026). This effect is larger and more precisely estimated for aspects of this belief held by a smaller share of participants in the control group (as shown in Supplementary Information Fig. 2.4), indicating that the effect is concentrated among more loyal Trump supporters. Regarding Russia’s invasion of Ukraine, for participants initially on the chronological feed, exposure to the algorithmic feed increased pro-Kremlin attitudes by 0.12 s.d. (95% CI: 0.03, 0.21; P = 0.007). Finally, we address multiple hypothesis testing concerns by aggregating all questions concerning policies and current political news into a single index. We find that this index is also 0.12 s.d. higher for those who were randomized to switch from the chronological to the algorithmic feed, compared with those who stayed on the chronological feed (95% CI: 0.04, 0.21; P = 0.004).
Remarkably, none of the political attitudes were affected by the reverse switch from the algorithmic to the chronological feed, that is, by switching the algorithm off. This is consistent with the earlier findings of the Meta study1. We also found no significant impact of the experimental treatments on subjective wellbeing. As demonstrated by the similarity between the two specifications, unconditional versus conditional on pre-treatment covariates, the significant impact of switching the algorithm on, and the absence of any effect from switching it off, are robust to the choice of covariates and model specification. Supplementary Information Fig. 2.9 presents results for all individual outcomes in the same format as in Fig. 2. We also verify that the ITT estimates are very similar to the local average treatment effects (LATE) on self-reported compliers, presented in Extended Data Fig. 3 and discussed in detail in Methods.
The sample size is insufficient to reliably test for heterogeneities in these effects. However, the split of the sample by self-reported pre-treatment partisanship into Democrats versus Republicans and Independents (or other political parties) indicates strongly that the results are driven by Republicans and Independents, whereas Democrats’ views are largely unaffected by the experiment (Extended Data Fig. 4).
The initial feed setting is correlated with participants’ socio-demographic characteristics, as it may reflect individual preferences and because selecting the chronological feed requires an active choice: those on the chronological feed are more likely (based on two-sided t-tests; Extended Data Table 1) to be female (50% versus 48%; P = 0.15), older (51.6 versus 50.1 years; P < 0.001) and white (82% versus 77%, P < 0.001); to have a college education (63% versus 56%; P < 0.001); to use X more frequently (5.04 versus 4.85 on an ordinal frequency scale; P < 0.001) and to use it more for hard news consumption (73% versus 68%, P < 0.001). We verify that the difference in effects between switching the algorithm on and off is not driven by these socio-demographic differences. First, we show that the results are robust to re-weighting observations so that the socio-demographic characteristics of participants initially on the chronological feed match those of participants initially on the algorithmic feed, and vice versa (Extended Data Fig. 5). Second, we predict the probability of initially using the chronological feed based on pre-treatment covariates and show that there is no significant treatment heterogeneity with respect to the predicted initial feed setting, in contrast to the actual feed setting (details of these exercises are provided in Supplementary Information section 2.5). We also verified that the results are not driven by selective attrition by presenting Lee bounds on our estimates (Extended Data Fig. 6 and Supplementary Information section 1.8.2).
Although it is reassuring that the asymmetry between the political effects of turning the algorithm on and off is not explained by selection into the initial feed setting predicted by socio-demographic characteristics, selection on unobserved factors may contribute to this asymmetry. For example, conditional on socio-demographics, users with different traits—such as autonomy or desire for control—could both choose different initial feed settings and react differently to similar content changes.
... continue reading