No News Is Good News – Monitoring Average Sentiment By News Network With Diffbot’s Knowledge Graph

Ever have the feeling that news used to be more objective? That news organizations — now media empires — have moved into the realm of entertainment? Or that a cluster of news “across the aisle” from your beliefs is completely outrageous?

Many have these feelings, and coverage is rampant on bias and even straight up “fake” facts in news reporting.

With this in mind, we wanted to see if these hunches are valid. Has news gotten more negative over time? Is it a portion of the political spectrum driving this change? Or is it simply that bad things happen in the world and later get reported on?

To jump into this inquiry we utilized Diffbot’s Knowledge Graph. Diffbot is one of the few North American organizations to crawl the entire web. We apply AI-enabled web scrapers to pages that are publicly available to extract entities — think people, places, or things — and facts — think job titles, topics, and funding rounds.

We started our inquiry with some external coverage on bias in journalism provided by AllSides Media Bias Ratings.

(Source)

AllSides’ Media Bias Ratings are one of the most well-regarded presentations on media bias, and we wanted a good starting point for our news monitoring journey, as well as ideological categories we can organize results by.

A quick exploratory query within the Knowledge Graph helped us determine whether each news organization was associated with article entities (they all did) and that plenty of article sentiment scores were recorded.

Within the Knowledge Graph sentiment is a field attached to many article entities. Sentiment scores range from -1 (very negative) to 1 (very positive). Additionally, we provide article-level as well as entity-level sentiment. In this analysis we examine article-level sentiment, which provides a score for an entire document. This takes into account the sentiment and salience (how central an entity is to understanding a given text) entities mentioned in each article are.

A Knowledge Graph query for articles from buzzfeednews.com that have sentiment scores

Seeing as we wanted to look at close to 50 organizations, we utilized API access to the Knowledge Graph. With little wrangling required, this let us quickly look at article sentiment across organizations at scale.

While our initial hunch that most news does tend to be negative held up, other assumptions were challenged. Some publications on the left and right tend to be fairly neutral in their presentation. Though the left and right do tend to present more negatively in general when compared to centrist news organizations.

The most neutral sentiment news organizations of those surveyed in 2021 included Associated Press (-.061 sentiment), as well as Market Watch (-.022 sentiment). The average for all publications in each political category included the following:

  • Left: -.24
  • Leans Left: -.25
  • Center: -.22
  • Leans Right: -.35
  • Right: -.28

Though excluding the single positive average sentiment left publication (New Yorker magazine) brings the left’s average roughly in line with the right at -.27. Also of note is that publications in the left and leans left categories tended to have a wider spread than those in the right or leans right categories.

Diversity of sentiment by publication political leaning

  • Left: .81 spread
  • Leans Left: .53 spread
  • Center: .49 spread
  • Leans Right: .22 spread
  • Right: .35 spread

As we explored the data more, however, we started to ask additional questions. Was news always so negative? How has news coverage on the right and left shifted over time? Are these media empires responding to specific events?

With this in mind we knew we needed some historical data, which for online articles is readily available in the Knowledge Graph.

Partisan News Sentiment 2010-2020

To retrieve historical data at scale from the Knowledge Graph, we relied on simple Python script (feel free to adapt for your needs). In short, we looked at articles indexed from a given publisher URL within a given timeframe and that where articles contained a sentiment score. We averaged sentiment scores on up to the first 500 returned articles.

There’s a lot going on in the above format. But a few trends surface, news from these publishers was by-in-large more positive in 2010 compared to 2020. Sentiment became much more polarized beginning in 2019. Additionally, throughout the 2010’s many publishers followed roughly the same “stairstep” pattern throughout the calendar year, where sentiment dips early in the calendar year, slowly rises, and drops lower the next year.

Using AllSide’s media categories we can mine in a bit more to see how average sentiment was shaped over time.

Far-Left Publications Average Sentiment By Month 2010-2020

Among far-left publications, a wider diversity of sentiments has surfaced over the past decade. While the “stairstep” pattern persists, steadily lowering sentiment for many publications through the 2010’s, extreme volatility in sentiment around the start of the Covid-19 pandemic jostles the layout. Interestingly, sentiment throughout 2020 rose for most publications. Topical tags attached to these articles included Covid-19 and the 2020 presidential election.

Left-Leaning Publications Average Sentiment By Month 2010-2020

Left-leaning publications followed a similar trajectory to far-left publications from 2010 to 2020. A “stairstep” pattern persisted from 2010-2019. A cluster of finance-focused news sites tend towards more neutral presentation. A greater number of left-leaning publications hover around -.1 sentiment than in the far-left publication category.

Centrist Publications Average Sentiment By Month 2010-2020

Centrist publications display a much more stable sentiment across the last decade. Comparing the initial to final lineup of sentiment ranks, more than half of these organizations stay within one rank of where they began. Additionally, the initial drop around the start of reporting on Covid-19 is much less severe. This cluster includes several “papers of record” including the Associated Press, Reuters, BBC, among others.

Right-Leaning Publications Average Sentiment By Month 2010-2020

Save the outlier of Reason.com, right-leaning publications begin and end within the same sentiment range as left-leaning and far-left publications. For the first half of the 2010’s, right-leaning publication sentiment trends downwards at a slower rate than other categories. Around 2015, a more extreme “stairstep” pattern emerges among around half of these publications. The start of 2020 surfaces a smaller spread, though volatility is present. The end of the timeframe shows the most negative cluster examined thus far except for far-left publications.

Far Right Publications Average Sentiment By Month 2010-2020

Far right publication sentiment exhibits a similar pattern to far left publications, though likely more extreme. Within the timeframe, average sentiments begin fairly clustered between 0 and -.3. Around 2015 — alongside 2016 presidential election primaries — far right publications begin to distinguish with growing sentiment spreads. 2019 shows the widest spread of sentiments among any cluster. By the end of 2020, far right publications are trending towards more aligned sentiment, though still one of the widest spreads seen in all years and publication clusters surveyed.