Dale Emmons, a longtime political operative and Grimes backer who helped her win in 2011, is part of the crowd who believes a Judd candidacy is too risky and means Democrats could lose power in the state House flipping it to the Republicans.
"Everybody wants to get him," Emmon said of McConnell. "If Grimes get this race, I think we get McConnell."
Emmon added that he is "not anti-Ashley Judd."
"We are all proud of Ashley Judd, a native daughter, and her success in the entertainment industry as well as her mother and sister," Emmons said. "But, let's get real about the stakes here. If something makes us lose additional seats [in the statehouse], we become a right-to-work state. We don't want that to happen."
McConnell is seen as the most vulnerable he has ever been. State polls show his popularity low and he has already run ads reaching out to women in the state.
Rep. John Yarmuth (D-KY) is Judd's most vocal proponent in the state and in an interview said Grimes would also "be a strong opponent against Mitch McConnell," but "nobody brings to the table the unique assets that Ashley Judd does."
"There are a lot of Democrats right now who are expressing some concern about her candidacy and what I would say to them is, We've tried five conventional campaigns in a row against Mitch McConnell and been unsuccessful," Yarmuth said, adding that he called Grimes, but she had not called him back yet.
"I don't know of anyone she has ever told that she is interested in this race," Yarmuth said of Grimes. "If she is interested in it, she ought to tell somebody… but to the best of my knowledge she hasn't said anything to anyone."
A Kentucky Democratic source said Yarmuth is not the only elected Democratic official who has reached out to Grimes but hasn't gotten a call back.
In an example of the schism forming between the two camps, Emmons said it is Judd who's not reaching out to Kentucky Democrats, not the other way around.
"I'm mystified why she's not coming to Kentucky," Emmons said. "We haven't heard from her. Whoever is advising her is not from Kentucky. They are running this campaign with an aloof strategy."
Multiple sources have said neither Judd nor Grimes have reached out to each other.
Grimes backers say she is seriously considering entering the race and is waiting until the legislative session wraps up next week to make more concrete moves. In fact, Grimes said on Wednesday she was focused on the end of the session when she was asked about the potential race by reporters, according to the Associated Press. She did say that she has been getting encouragement to run against McConnell.
Grimes wouldn't answer questions about whether Bill Clinton had encouraged her run or other specific questions about the race simply answering: "My focus right now has been on the legislative session."
Grimes' office declined an interview with ABC News, citing the end of the legislative session. Judd's publicist said she is not doing interviews right now.