Al Gore’s New Movie Exposes The Big Flaw In Online Movie Ratings


“An Inconvenient Sequel: Truth to Power,” Al Gore’s new documentary, is one of the most interesting films of the year.

We don’t mean its content. The sequel to the Academy Award–winning climate change documentary “An Inconvenient Truth” is a solid movie. But it’s really the film’s online reception that’s intriguing.

“An Inconvenient Sequel” is among the most controversial and polarizing titles of the year. Because of the politics surrounding Gore and climate change, the film divides men and women, critics and fans, and even people who saw the movie and people who are just rating it. But the movie’s aggregate rating hides many of those divisions, giving us a perfect case study for understanding a big weakness of online rating systems: separating the controversial from the mediocre. That weakness could discourage ambitious-but-controversial work.

To start, look at how weird this chart of the film’s ratings distribution is:

In the pantheon of sequels to well-regarded films, “An Inconvenient Sequel” may not be “The Godfather: Part II,” but it’s not exactly “The Godfather: Part III,” either. I think it’s solid, but that seems beside the point. Gore is a capable documentarian, but he’s also a guy for whom 51 million Americans voted and 50.4 million other Americans voted against25 that one time. Climate change is equally divisive. And those politics are coloring the film’s internet reception.

Of the 2,645 IMDb users who rated the film as of August,26 over 38 percent gave the film a 1 out of 10. Of those same 2,645 IMDb users, just under 34 percent gave the film a 10 out of 10. In short: 72 percent of people who rated the movie gave it an extreme score, a 1 or a 10.

This is a massive abnormality for an IMDb rating distribution. As of mid-August, 127 documentaries and feature-length films had been released in 2017 that also earned a Metacritic score and at least 1,000 ratings on IMDb. Here’s a sortable table of all of them.

Strong feelings about climate change, emojis and genocide

Most IMDb users loved or hated “An Inconvenient Sequel,” as of Aug. 16

RATING DIST.
MOVIE
RATING
1 OUT OF 10
2-9 OUT OF 10
10 OUT OF 10
TOTAL POLARIZED VOTES
POLARIZATION SCORE
An Inconvenient Sequel: Truth to Power 5.2 41% 23% 36% 77% 1.373
The Ottoman Lieutenant 7.4 18 20 62 80 1.122
The Emoji Movie 1.8 74 15 10 85 0.920
The Case for Christ 5.6 21 49 29 51 0.676
A Dog’s Purpose 6.8 17 58 25 42 0.549
Snatched 4.0 35 58 6 42 0.537
Diary of a Wimpy Kid: The Long Haul 4.0 32 58 10 42 0.536
Bad Kids of Crestview Academy 6.8 4 63 33 37 0.500
Bitter Harvest 6.1 9 61 30 39 0.484
Girls Trip 6.9 8 61 31 39 0.448
Detroit 7.5 11 73 16 27 0.417
Fifty Shades Darker 4.6 16 69 15 31 0.407
VooDoo 5.6 13 76 11 24 0.394
Megan Leavey 7.1 8 70 22 30 0.392
The Shack 6.3 7 69 24 31 0.390
All Eyez on Me 6.0 9 72 19 28 0.379
The Little Hours 6.6 9 74 17 26 0.368
The Book of Henry 5.7 10 73 17 27 0.367
Nobody Speak: Trials of the Free Press 6.5 10 85 5 15 0.349
Gong fu yu jia 5.3 8 79 14 21 0.339
Transformers: The Last Knight 5.3 11 77 13 23 0.338
Chasing Coral 8.3 1 66 33 34 0.338
Arsenal 3.9 14 78 7 22 0.335
Beatriz at Dinner 6.7 8 79 13 21 0.335
The Dinner 4.6 13 77 9 23 0.334
A Ghost Story 7.8 4 77 19 23 0.332
It Comes at Night 7.0 8 86 6 14 0.329
The Hunter’s Prayer 5.4 3 84 12 16 0.329
Journey to the West: The Demons Strike Back 5.5 9 80 11 20 0.327
Everything 6.3 4 80 16 20 0.323
Rough Night 5.5 9 82 8 18 0.320
How to Be a Latin Lover 5.7 7 82 11 18 0.319
Wish Upon 4.7 10 81 10 19 0.318
Wind River 7.8 2 77 21 23 0.318
Captain Underpants: The First Epic Movie 6.6 5 82 13 18 0.318
Dunkirk 8.4 1 71 28 29 0.316
Annabelle: Creation 7.2 2 81 17 19 0.315
Baywatch 5.8 7 82 11 18 0.315
Wonder Woman 7.9 2 76 22 24 0.314
Kill Switch 5.4 6 91 4 9 0.312
Beauty and the Beast 7.4 2 79 19 21 0.312
First Kill 4.8 9 84 7 16 0.312
The Dark Tower 6.0 7 83 11 17 0.311
I Am Heath Ledger 7.4 1 81 18 19 0.311
The Big Sick 8.1 3 80 17 20 0.310
The Zookeeper’s Wife 7.0 5 86 9 14 0.309
Smurfs: The Lost Village 5.9 3 86 11 14 0.308
Kidnap 6.0 6 84 10 16 0.307
On Body and Soul 8.2 1 74 24 26 0.307
The Wall 6.2 5 88 6 12 0.306
Icarus 8.1 2 75 24 25 0.306
The Incredible Jessica James 6.5 4 86 9 14 0.306
Phoenix Forgotten 5.2 9 84 7 16 0.306
The Bye Bye Man 4.3 12 82 5 18 0.305
Naked 5.3 6 85 9 15 0.305
Drone 5.3 7 85 8 15 0.305
The Fate of the Furious 6.9 3 83 14 17 0.304
Logan 8.2 1 74 25 26 0.304
Cars 3 7.1 3 83 14 17 0.304
xXx: Return of Xander Cage 5.2 8 84 8 16 0.303
Sandy Wexler 5.1 9 84 7 16 0.301
Bokeh 4.9 12 85 4 15 0.301
Viceroy’s House 6.6 4 88 8 12 0.301
Pirates of the Caribbean: Dead Men Tell No Tales 7.0 2 84 15 16 0.301
Burning Sands 6.2 4 86 9 14 0.300
Power Rangers 6.1 3 85 11 15 0.298
The House 5.6 7 87 6 13 0.297
John Wick: Chapter 2 7.6 1 81 18 19 0.297
Spider-Man: Homecoming 7.9 2 81 17 19 0.297
Pilgrimage 5.8 5 86 8 14 0.297
Unforgettable 4.9 8 86 6 14 0.296
Atomic Blonde 7.1 3 87 10 13 0.296
XX 4.6 12 86 3 14 0.296
The Beguiled 7.0 4 90 6 10 0.295
War for the Planet of the Apes 7.9 2 83 16 17 0.295
Song to Song 5.9 7 85 8 15 0.295
Valerian and the City of a Thousand Planets 6.7 3 86 11 14 0.295
The Last Word 6.6 2 87 11 13 0.293
47 Meters Down 5.8 5 88 7 12 0.293
The Boss Baby 6.4 3 88 9 12 0.292
Eloise 4.5 8 86 6 14 0.291
Baby Driver 8.2 1 80 19 20 0.291
Get Out 7.8 2 85 13 15 0.291
The Other Side of Hope 7.4 2 92 6 8 0.290
Casting JonBenet 6.2 5 90 5 10 0.290
CHIPS 6.0 3 90 6 10 0.290
Guardians of the Galaxy Vol. 2 8.0 1 81 18 19 0.290
Rings 4.5 9 87 4 13 0.289
The Most Hated Woman in America 6.1 4 91 5 9 0.289
Once Upon a Time in Venice 5.2 6 90 4 10 0.288
Fist Fight 5.7 4 90 6 10 0.288
Before I Fall 6.4 2 89 9 11 0.288
Okja 7.5 1 86 13 14 0.288
The Sense of an Ending 6.4 3 92 5 8 0.287
The Space Between Us 6.4 2 89 9 11 0.287
Get Me Roger Stone 7.5 2 90 8 10 0.287
Voice from the Stone 5.2 4 90 6 10 0.287
Alien: Covenant 6.6 3 91 6 9 0.287
Unlocked 6.1 3 92 5 8 0.287
Despicable Me 3 6.4 2 91 7 9 0.287
Small Crimes 5.9 3 92 5 8 0.286
The Mummy 5.6 5 90 6 10 0.286
Black Butterfly 6.0 2 91 6 9 0.286
My Cousin Rachel 6.3 3 93 4 7 0.285
Sleepless 5.6 4 92 5 8 0.284
Oklahoma City 7.5 1 92 6 8 0.284
Aftermath 5.7 3 92 4 8 0.283
The Circle 5.3 6 91 3 9 0.283
All Nighter 5.6 3 94 4 6 0.283
Ghost in the Shell 6.5 2 91 7 9 0.283
Sand Castle 6.3 2 94 4 6 0.283
iBoy 6.0 2 92 6 8 0.283
Life 6.7 2 93 5 7 0.283
Berlin Syndrome 6.3 2 93 4 7 0.283
Gifted 7.7 0 86 14 14 0.282
To the Bone 7.0 1 90 9 10 0.282
Wilson 5.8 3 92 4 8 0.282
Kong: Skull Island 6.8 1 93 6 7 0.281
Win It All 6.3 2 95 3 5 0.281
Take Me 6.1 2 93 5 7 0.281
War Machine 6.1 3 93 4 7 0.281
Going in Style 6.6 1 93 6 7 0.280
The LEGO Batman Movie 7.4 1 90 10 10 0.280
T2 Trainspotting 7.4 1 90 9 10 0.279
Table 19 5.8 2 93 5 7 0.279
The Discovery 6.3 2 94 5 6 0.277
I Don’t Feel at Home in This World Anymore. 7.0 1 95 5 5 0.274

Polarized votes are the combined shares of 1 and 10 ratings. Any differences due to rounding. The polarization score is calculated with a formula from Esteban and Ray; the higher the number, the more bimodal the score distribution is, and the highest possible score is 2.25, meaning 50 percent 1s and 50 percent 10s.

Source: IMDB

Simply looking at which films have the most 1s and 10s combined is an easy way to see how divided IMDb users were on the documentary: Only the universally reviled “The Emoji Movie,” and “The Ottoman Lieutenant,” which stirred up a controversy regarding its handling of the Armenian genocide, have more combined extreme scores. But looking at a specialized metric of polarization27 that looks at how evenly split those extreme votes were makes “An Inconvenient Sequel” by far the most heavily and evenly split film of 2017.

The reception to “An Inconvenient Sequel” lays on three major divides: (1) critics vs. audiences, (2) people who saw the movie vs. those who did not, and (3) men vs. women. It’s frankly impressive for a single film to stand astride so many fault lines. But if you’re a casual moviegoer quickly checking IMDb to see if “An Inconvenient Sequel” is worth checking out, you wouldn’t know any of that. You’d think it was just a dumpy movie — the “Breakin’ 2: Electric Boogaloo” of climate documentaries.


Fault Line No. 1:
Critics vs. audiences

The all-too-common critic-versus-fan dance is typically a load of crap. Step No. 1: Some preexisting work of fiction is adapted to the big screen. Step No. 2: Critics don’t like it. Step No. 3: Fans of the original work get mad at the critics. Usually, the lead in the film will say something like “We made this for the fans” to downplay the terrible reviews, and the film will make $20 million and place third at the box office behind the new Pirates of the Caribbean movie and a Chris Pratt flick. Critics gotta critique, fans gotta buy, stars gotta sell.

Roughly three out of four movies have an IMDb audience rating higher than their Metacritic score, so you’re going to see such tension. But we can’t say for certain how much of that is due to differences in how the two groups judge movies, as opposed to the quirks in the rating systems.

Still, let’s take it for granted that, overall, critics are less forgiving than the average moviegoer. “An Inconvenient Sequel” lies way on the other side of that spectrum.

When the critics like a movie more than the fans do

Fans tend to rate movies more favorably than critics, but golly, there are exceptions

FILM
IMDB RATING
METACRITIC SCORE
RESCALED POINT DIFF.
XX 4.6 64 -18
An Inconvenient Sequel: Truth to Power 5.2 68 -16
Win It All 6.3 78 -15
The Other Side of Hope 7.4 88 -14
Casting JonBenet 6.2 74 -12
The Dinner 4.6 58 -12
Dunkirk 8.4 94 -10
It Comes at Night 7.0 78 -8
The Beguiled 7.0 77 -7
The Incredible Jessica James 6.5 72 -7
Berlin Syndrome 6.3 70 -7
Bokeh 4.9 56 -7
Get Out 7.8 84 -6
Pilgrimage 5.8 64 -6
The Big Sick 8.1 86 -5
A Ghost Story 7.8 83 -5
I Don’t Feel at Home in This World Anymore. 7.0 75 -5
Snatched 4.0 45 -5
Baby Driver 8.2 86 -4
Journey to the West: The Demons Strike Back 5.5 59 -4
Chasing Coral 8.3 86 -3
War for the Planet of the Apes 7.9 82 -3
Detroit 7.5 78 -3
Captain Underpants: The First Epic Movie 6.6 69 -3
The Little Hours 6.6 69 -3
Nobody Speak: Trials of the Free Press 6.5 68 -3
Girls Trip 6.9 71 -2
Okja 7.5 76 -1
The LEGO Batman Movie 7.4 75 -1
Beatriz at Dinner 6.7 68 -1
Burning Sands 6.2 63 -1
Small Crimes 5.9 60 -1
Get Me Roger Stone 7.5 75 0
My Cousin Rachel 6.3 63 0
John Wick: Chapter 2 7.6 75 1
Oklahoma City 7.5 74 1
Alien: Covenant 6.6 65 1
Diary of a Wimpy Kid: The Long Haul 4.0 39 1
Wonder Woman 7.9 76 3
The Sense of an Ending 6.4 61 3
How to Be a Latin Lover 5.7 54 3
Gong fu yu jia 5.3 50 3
Rough Night 5.5 51 4
Drone 5.3 49 4
Unforgettable 4.9 45 4
Logan 8.2 77 5
Wind River 7.8 73 5
Megan Leavey 7.1 66 5
The Wall 6.2 57 5
War Machine 6.1 56 5
Take Me 6.1 56 5
Song to Song 5.9 54 5
The Case for Christ 5.6 51 5
Spider-Man: Homecoming 7.9 73 6
I Am Heath Ledger 7.4 68 6
To the Bone 7.0 64 6
Kong: Skull Island 6.8 62 6
Before I Fall 6.4 58 6
47 Meters Down 5.8 52 6
The Bye Bye Man 4.3 37 6
The Emoji Movie 1.8 12 6
T2 Trainspotting 7.4 67 7
Atomic Blonde 7.1 63 8
The Discovery 6.3 55 8
Beauty and the Beast 7.4 65 9
Wilson 5.8 49 9
First Kill 4.8 39 9
Annabelle: Creation 7.2 62 10
iBoy 6.0 50 10
The Circle 5.3 43 10
xXx: Return of Xander Cage 5.2 42 10
Voice from the Stone 5.2 42 10
On Body and Soul 8.2 71 11
Everything 6.3 52 11
Sandy Wexler 5.1 40 11
Cars 3 7.1 59 12
Viceroy’s House 6.6 54 12
Icarus 8.1 68 13
Guardians of the Galaxy Vol. 2 8.0 67 13
The Zookeeper’s Wife 7.0 57 13
The Fate of the Furious 6.9 56 13
Life 6.7 54 13
Ghost in the Shell 6.5 52 13
Aftermath 5.7 44 13
Fifty Shades Darker 4.6 33 13
The Boss Baby 6.4 50 14
VooDoo 5.6 42 14
Arsenal 3.9 25 14
Despicable Me 3 6.4 49 15
Wish Upon 4.7 32 15
Valerian and the City of a Thousand Planets 6.7 51 16
Going in Style 6.6 50 16
Kidnap 6.0 44 16
Gifted 7.7 60 17
Power Rangers 6.1 44 17
Black Butterfly 6.0 43 17
Naked 5.3 36 17
Sand Castle 6.3 45 18
Table 19 5.8 40 18
Unlocked 6.1 42 19
Smurfs: The Lost Village 5.9 40 19
All Nighter 5.6 37 19
The Hunter’s Prayer 5.4 35 19
Phoenix Forgotten 5.2 33 19
The Most Hated Woman in America 6.1 41 20
Fist Fight 5.7 37 20
Rings 4.5 25 20
Baywatch 5.8 37 21
All Eyez on Me 6.0 38 22
The Mummy 5.6 34 22
Sleepless 5.6 34 22
Kill Switch 5.4 31 23
Once Upon a Time in Venice 5.2 28 24
A Dog’s Purpose 6.8 43 25
Transformers: The Last Knight 5.3 28 25
The Last Word 6.6 40 26
The Dark Tower 6.0 34 26
The Book of Henry 5.7 31 26
The House 5.6 30 26
Bitter Harvest 6.1 34 27
Eloise 4.5 15 30
Pirates of the Caribbean: Dead Men Tell No Tales 7.0 39 31
The Space Between Us 6.4 33 31
The Shack 6.3 32 31
CHIPS 6.0 28 32
Bad Kids of Crestview Academy 6.8 22 46
The Ottoman Lieutenant 7.4 26 48

IMDb ratings, which use a 10-point scale, have been multiplied by 10 to match Metacritic’s 100-point scale. All data as of Aug. 16.

Source: IMDB

Its Metacritic score is 68 out of 100. Its IMDb score was 5.2 out of 10 (or 52 on Metacritic’s scale). That’s the second-highest markdown28 by IMDb raters in the entire 2017 data set. And Metacritic is one of the harsher critical aggregations; “An Inconvenient Sequel” has a great 78 percent score on Rotten Tomatoes.29

So while I’m normally skeptical of the critics-versus-fans narrative, it really looks like it exists for “An Inconvenient Sequel” — at least on IMDb — just reversed. The critics thought it was OK-to-fine. The “fans” rated it subpar.


Fault Line No. 2:
People who saw it vs. people who didn’t

But here’s why I put fans in scare quotes.

We initially began monitoring the IMDb ratings of “An Inconvenient Sequel” on July 17, when 678 users had rated the film, including about 45 percent 1s, 38 percent 10s and 17 percent 2-9s. That hundreds of people were able to draw such strong conclusions about the film 11 days before it was released on July 28 is … odd.30 By the time the movie was in more than four cinemas on Aug. 4, more than half of the people who would eventually come to rate it had already done so.

Ticket sales went from 0 per week to 17,000 per week to 150,000 per week while the IMDb score barely budged. Put another way: Despite ticket sales increasing by three orders of magnitude per week, the number of reviewers stayed flat — only about 500 people voted on the film each period.

It’s hard to believe that each and every one of those pre-release reviews is bona fide, especially on a politicized film like this.

Personally, we would not be surprised to find out that a large percentage of these votes aren’t real. IMDb applies some filters to prevent ballot stuffing but refuses as policy to discuss those measures. Nonetheless, how did so many people see this film before its wide release? We made efforts to determine whether automated voting bots were used, but the way ratings update on IMDb makes it hard to look for obvious telltale patterns of bot voting. All told, we think this film’s rating should be viewed with utmost skepticism.

Most likely, the film was doomed before it even premiered by people (or bots) who had not seen it.


Fault Line No. 3:
Men vs. women

First, it’s mostly men who have rated “An Inconvenient Sequel”:

Second, men hate the film way more than women do:

So the movie’s IMDb rating is mostly the work of men who feel substantially different about the film than women do. Which raises the question: Why don’t more review sites do what pollsters do, and adjust their sample to reflect the reality of the world? There are arguments for making the raw data available, yes, but leaving this unadjusted and letting men get 86 percent of the say when it comes to what’s well regarded on IMDb (or any other online rating site) seems specious given that women make up a majority of moviegoers in the United States.

In this case, it seems like the gender gap in opinions on climate change — men are far more skeptical that it’s a problem than women — is affecting views of “An Inconvenient Sequel.”


A Failing Grade

The result of boiling down a piece of culture to one single number is that the rating system inevitably simplifies our understanding of the work. Sometimes that’s useful. But let that happen long enough and you’ll set up an incentive for filmmakers to hedge away from the compelling and the controversial.

Six 2017 films in total had an IMDb rating of 5.2 on Aug. 16, the day we happened to pull all of this data. In addition to “An Inconvenient Sequel,” there was “Voice from the Stone,” “Once Upon a Time in Venice,” “xXx: Return of Xander Cage,” “Phoenix Forgotten” and “Vengeance: A Love Story.”

See if you can spot which one is a special case:

Say what you will, but in addition to being controversial, “An Inconvenient Sequel” was ambitious: Few films involve Arctic expeditions, inside access to the Paris Climate Conference, interviews with the sitting secretary of state and a globe-trotting look at catastrophic weather conditions. If ambitious-yet-controversial films are boiled down to a single number that makes them look identical to mediocre films, what incentive does Hollywood have to continue investing in movies that challenge the audience?

The democratization of film reviews has been one of the most substantial structural changes in the movie business in some time, but there are dangerous side effects. The people who make movies are terrified. IMDb scores represent a few thousand mostly male reviewers who might have seen the film but maybe didn’t, and they’re influencing the scoring system of one of the most popular entertainment sites on the planet.

The thing about a democracy is that it really does fall to the user, the voter, and the vote counter to ensure that the will of the masses is correctly manifested.