What goes into the election ratings that help guide New York’s political parties and consultants?

What tilts an election to “lean Democratic,” “likely Republican” or that closely watched title of “toss-up”? According to the analysts behind some of the country’s most trusted election raters, an immense amount of information that aims to give the general public, would-be donors and the media a strong sense of which party will be celebrating on election night.

“It’s not formulaic,” said Jacob Rubashkin, a deputy editor who has worked with Inside Elections since 2020. “We don’t have an algorithm or a master formula where we plug in a bunch of inputs and it spits out a rating.” Inside Elections, along with competitors The Cook Political Report and Sabato’s Crystal Ball, are regarded as the top nonpartisan organizations – sometimes called “election handicappers” – doing these kinds of election predictions.

Each of those three organizations takes a slightly different approach, but they all cast wide nets capturing a district’s most recent election results, where political parties are spending money, public and private polling, as well as insights derived from interviews with candidates and consultants. Scandals plaguing a candidate or local voting trends don’t escape analysts’ radar either.

Prior to last month’s state Court of Appeals decision that will send New York’s congressional mapmakers back to the drawing board, Cook Political Report, Inside Elections and Sabato’s Crystal Ball had each rated seven districts as somewhere on the scale of “lean” to “toss-up.” All three included the vacant 3rd Congressional District, along with the 4th, 17th and 22nd districts as “toss-ups.” Those seats are currently held by Republican Reps. Anthony D’Esposito, Mike Lawler and Brandon Williams, respectively. Cook and Sabato also included the 19th District, which is held by Republican Rep. Marc Molinaro, as a “toss-up,” while Inside Elections marked it as “tilt Republican.”

Analysts at all three organizations said it was too soon to predict how those ratings will change after redistricting, though some said they’re looking at the 11th Congressional District – held by Rep. Nicole Malliotakis, and rated by all three organizations as a safe or solid Republican seat under its current district lines – as one that could inch closer toward the “toss-up” column, depending on how it’s redrawn.

These nonpartisan ratings have become a trusted source for anyone to follow which races out of hundreds nationwide might be particularly competitive, and campaigns and parties can use them to vie for attention and resources from donors, labor unions and other interest groups. Forecasting models, like one created by FiveThirtyEight, have also used ratings published by each organization as a layer of data, while national and local news outlets often cite them in their own reporting.

“In a world where everyone is overloaded with spin, organizations like ours seek to provide a neutral assessment of where things stand,” said Dave Wasserman, senior editor and elections analyst at The Cook Political Report. “We may be right or we may be wrong, but we care a lot about trying to get it right.” According to Cook’s own internal review, its “lean,” “likely” and “solid” ratings have been more than 95% accurate between 1984 and 2022.

In New York, the ratings are a tool that both parties monitor, even if they’re how insiders learn which races will be toss-ups. “We make those determinations independently,” said state Democratic Party Chair Jay Jacobs. “That’s not to say that we don’t look at the prognosticators. And if something were completely off, I mean, we’d have to take a look at it. But typically, we agree.”

Where the ratings can come in handy is in communicating the state of play for those not plugged into private polling or internal race dynamics. It’s not unheard of for campaigns to prefer a “toss-up” rating that they can use to rally donors instead of one that predicts their candidate performing better, like “leans Democratic,” said Kyle Kondik, managing editor of Sabato’s Crystal Ball.

The Cook Political Report, founded by Charlie Cook and now led by Amy Walter, points to its seven-point scale – including solid Republican, likely Republican, lean Republican, toss-up, lean Democratic, likely Democratic and solid Democratic – as pioneering. The most important contributor to its ratings, Wasserman said, was the underlying partisanship of each district. But ratings are ultimately formed from an array of information gathering, including interviews with campaigns to size up candidates.

Sabato’s Crystal Ball, run out of the University of Virginia Center for Politics and founded by professor Larry Sabato, uses a similar seven-point scale. Sabato’s Crystal Ball also uses off-the-record conversations with campaigns, party brass and consultants to feel out the political environment, which is similar to how reporters stay up to date on the dynamics of a given race. Like Cook, Sabato’s Crystal Ball updates its ratings as needed, shifting districts from one column to the next based on persuasive new information. The announcement that Michigan Democratic Rep. Dan Kildee would be retiring knocked that district out of “leans Democratic” and into “toss-up,” for example.

Inside Elections, born out of Stuart Rothenberg’s newsletter The Rothenberg Report and now led by Nathan Gonzales, uses a nine-point scale that includes a “tilt” category for each party. Inside Elections weights inputs differently based on the point in time within an election cycle, allowing for recalibrations along the way. Early on, ratings primarily assess the district itself, its recent voting history and its demographics. A second phase layers on an assessment of the candidates, taking into consideration data from financial reports and more subjective analysis. Even information like who a candidate is working with can be worth considering – particularly if they’ve got a notorious track record – though Rubashkin said “those kinds of intangibles don’t always rise to the top” in an era with more nationalized, expensive elections. Finally, closer to Election Day, polling becomes more important. Though they’re keeping an eye on both public and private polls throughout the cycle, Rubashkin said, it’s looked at more closely in the weeks and months leading up to the election.

Each of these three organizations have just a handful of analysts keeping track of hundreds of elections between congressional, U.S. Senate and gubernatorial races as well as the Electoral College in presidential election years. And like pollsters, pundits and reporters, the prognosticators can sometimes miss signs about voter behavior or trends. “I would say this about myself, and I don’t think others would disagree, that House projections have been off for the last two cycles,” Kondik said.

While watching how and where political parties spend their money can be helpful, it can also misdirect attention if the parties themselves aren’t spending wisely, Kondik said, offering the example of a flurry of Republican activity in the 25th Congressional District in 2022 – which Democrat Joe Morelle ultimately won by a comfortable margin.

“For what they do focus on, I do think it’s a respected methodology and results that help drive the broader narratives of these races,” said Basil Smikle, a consultant and the former executive director of the state Democratic Party, said of the three nonpartisan groups. “That is very important, whether you’re a pundit or whether you’re a donor, to be able to look at and say, ‘OK, I’ve got to get interested in this race.’”

Ultimately, these ratings are one just one tool among others that aim to make sense of elections – even, as, Smikle said, it’s becoming more difficult to determine what is sending voters to the polls and motivating voters’ decisions.

Noting this “turbulent era in American politics,” the Sabato’s Crystal Ball website offers a warning: “He who lives by the Crystal Ball ends up eating ground glass!”