Can a smartphone app conduct political polls?

If you, like me, spend too much time perusing 538's polls page, you may have noticed this summer has seen a lot of polls from a new-ish pollster: ActiVote. Since Aug. 1, ActiVote has completed 25 horse-race polls — more than all but two other pollsters in our database.

It's always wise not to put blind trust into pollsters without much of a track record. But ActiVote has proved especially controversial thanks to its unusual methodology: It's a free smartphone app that anyone can download, and it's primarily a nonpartisan voter-education tool, not a tool for measuring public opinion. Some polling analysts say that makes it untrustworthy. The folks at ActiVote argue they're just innovating new methods in an industry facing a lot of challenges. So which is it?

How ActiVote conducts its polls

ActiVote was founded in 2019, and according to CEO and co-founder Victor Allis, the company's primary mission has always been to inform voters about who is on their ballot in upcoming elections. After users input information about where they live, they can view a list of candidates and questions that will appear on their ballot based on their state, district, etc. The app also provides basic information about each candidate, such as their ideology, website and social media accounts.

And, crucially, ActiVote users can also select which candidates they're supporting. The ActiVote staff quickly realized that, in collecting this information, they were essentially conducting a poll — so they decided to publish what they found. Although Allis, a Ph.D. in computer science, had no polling background, he joined the American Association for Public Opinion Research in 2022 and threw himself into the science of polling. As a result, ActiVote is no amateur operation, and its survey results don't just reflect raw data from app users — which, of course, can be pretty unrepresentative of the population they're trying to measure. Instead, ActiVote uses L2, a prominent voter database, to weight its data to match the age, gender, party affiliation, ethnicity, income and education breakdowns of the electorate.

But there are still concerns about the way ActiVote recruits its respondents. First, there's the fact that anyone who wants to can download the app and participate in one of its polls. Traditionally, polls have used what's known as "probability sampling," or obtaining a random sample of the population through techniques like dialing telephone numbers at random. But while ActiVote's app-based approach is unique, its use of nonrandom, or "nonprobability," sampling is actually not all that unusual in modern polling. Many pollsters, including well-reputed ones like YouGov, use nonprobability sampling to recruit people to join an online panel, from which their poll respondents are drawn. However, many pollsters remain skeptical of polls that participants can simply opt in to, as they sometimes produce flawed results.

Recently, social-media users have also flagged the possibility that people could download ActiVote and fill out one of its polls with false information (e.g., a supporter of former President Donald Trump saying they are an Arizona voter in order to make the polls in that crucial swing state look better for their candidate). However, Allis assured 538 that ActiVote confirms that all its respondents are valid registered voters by comparing their name and address to the L2 voter file before including them in its polls.

Others have questioned whether ActiVote is less reliable even than other nonprobability pollsters because it recruits its sample exclusively from its app rather than robustly recruiting participants from across the internet. While Allis told 538 that ActiVote does advertise on Google Play and partners with voter-engagement nonprofits to get more users, he confirmed that the firm does not actively recruit people to participate in its polls other than through the app.

How 538 uses ActiVote's polling

So where does that leave us? At 538, pollsters must clear only a low bar (specifically, a set of basic methodological and ethical standards that any legitimate pollster can easily achieve) in order for us to aggregate their polls. ActiVote meets those standards, so its polls are included on our polls page and in our election models.

How much credence to put in their results, though, is another matter. At 538, including a poll on our polls page and in our models isn't necessarily an endorsement of its quality. We've deliberately adopted an inclusive policy in order to prevent bias from creeping into decisions about what polls to use and not to use, and because we don't think it's our place to decide what the "right" way to conduct a poll is (the proof of that is in the post-election pudding).

Instead, we use our pollster ratings (which give pollsters a grade ranging from 0.5 to 3.0 stars based on their transparency and accuracy in past elections) to assess pollster quality. And unfortunately, ActiVote doesn't have a pollster rating yet because its polls in the 2020 and 2022 cycles did not meet our evaluation criteria,* so we can't be sure how accurate the firm's polls are. It has, however, addressed all 10 of the questions we use to calculate a pollster's Transparency Score, the other component of our pollster ratings. ActiVote's raw Transparency Score is 8 out of 10 — better than 256 of the 320 pollsters for which we have collected transparency data.

So we can't give you a definitive answer about how much trust to put in ActiVote's polling, other than to fall back on the old chestnut that we'd rather have data from a known pollster with a good track record than an unknown pollster with an uncertain one. To that point, there's no need to worry that ActiVote is having too much influence on our polling averages and forecasts, despite the high number of polls it's putting out; 538's polling averages and forecasts put less weight on polls from unrated firms that release a lot of polls from several states in quick succession (a safeguard we implemented to protect against unknown pollsters releasing a glut of polls to bias the aggregates — a dynamic some observers have termed "flooding the zone").

538's models this year also test whether each pollster's polls are consistently more Democratic- or Republican-leaning than average (we call this a "house effect") across states, not just within them. This means that if unrated pollsters — like ActiVote — publish numbers that are unusually good for one party in multiple states, we will pick up on that faster than if we were only looking for house effects at the individual-state level. (So far in the 2024 cycle, ActiVote's polls have been 0.34 percentage points better for Trump than other polls — neither an especially small nor an especially large house effect.)

As with every other pollster, we'll find out how accurate ActiVote is after the election. And Allis is just fine with that. "We're unproven," he acknowledged. But he views ActiVote's novel methodology as just the latest step in the evolution of polling. In the early days, polling was done by going door to door, and there was skepticism when pollsters first tried conducting surveys over the phone — but it eventually became the gold standard. Then, when telephone response rates started to decline, there was skepticism about conducting polls online — but that has now become a widely accepted practice as well. Pollsters constantly "need to find new things," Allis said. "We think we're one of these new things."

Mary Radcliffe and G. Elliott Morris contributed research.

Footnotes

*Specifically, ActiVote's polls in 2020 and 2022 used a very long field period, so the median field dates of its surveys were too far before the elections for us to reliably evaluate them.