"If you did this to your own children, you know, you deserve to be locked away," he said. But "I didn't do it and I wasn't going to admit it. You can either roll up in a ball and let life take you or you can just take what comes down the road and maintain integrity."
For twenty years, he fought to prove his innocence, uncovering shocking secrets and a police investigation that may have been tainted from the start, that would reveal the truth, and ultimately exonerate Ray.
In the early 1980s, Ray Spencer was a cop in Vancouver, Wash., where he was married with two young kids. Ray had a reputation around town for being arrogant and for having a weakness for women. It was a weakness Ray admittedly indulged and his wife painfully endured.
"It's not so much the rage and anger I had, but I didn't believe anything he ever said was the truth," his first wife DeAnne Spencer told "20/20."
DeAnne decided she had enough. The couple divorced in 1981 and she took their kids, Katie 2, and Matt, 5, and moved to Sacramento, Calif., to start a new life.
Alone, and separated from his children for months at a time, Ray turned to Shirley Hansen for solace. Two years after his divorce, Ray married Hansen, who had a three year old son of her own, whom they called "Little Matt."
Familiar with the signals children send out, Shirley was alarmed when Ray's daughter Katie, who was visiting during the summer of 1984, suggested she was being inappropriately fondled.
"Katie put her hands between her legs and told Shirley, 'My brother does this, my mom does this, my dad does this,'" Ray recalled. "Now my assumption was that 'Daddy' was a boyfriend that [DeAnne] was dating."
Ray immediately called Child Protection Services and the Sheriffs' departments in both Clark County, Wash., where he worked, and Sacramento, where his children lived with their mother. After a brief investigation, California authorities decided not to pursue the case. But Washington authorities decided that they would look into it.
Ray Spencer was pleased the case was being explored, but he had no idea what lay ahead.
Tasked with the investigation of possible child abuse, Detective Sharon Krause of Clark County made several trips to Sacramento to question Ray's two children. She took Katie and Matt separately to a hotel to be questioned, but did not make audio or video recordings of the sessions, relying instead on handwritten notes.
Katie and Matt -- then 5 and 8 years old -- initially denied any abuse. But then Katie began to give lurid accounts of rape and shockingly, pointed the finger at her own father, according to Krause.
"I said, 'You must be crazy.' I'm the one that reported this," Ray recalled.
Ray was soon charged with raping his own five-year-old daughter. He was released on his own recognizance, but his life continued to spiral downward: he lost his job, separated from his second wife, Shirley Hansen, and began living in a motel.
Shortly after his separation, in February 1985, Shirley asked Ray if her son, Little Matt, could spend a night with him at his motel, which Ray welcomed. Four days later, Ray was arrested and incarcerated -- this time for the rape of his stepson.
"I was thinking, this just can't be happening," he said.
Then Ray was struck with the final blow. After eight months of questioning, his son Matt told investigators that he too had been raped by his father.
"I was starting at that point to question my own sanity," Ray said. "I wanted to commit suicide. I spent two weeks in Oregon Health Science University Hospital in their psyche ward, diagnosed with clinical depression. I would have rather been charged with murder than this."
Ray was charged with several counts of statutory rape and complicity to rape.
Though he maintained his innocence, his mental state had deteriorated and he was worried he wouldn't win over a jury in trial. Ray decided to enter an Alford plea, which allows a defendant to plead guilty without admitting guilt. The move, while it maintained his innocence, cost him with the judge, who sentenced Ray to multiple life sentences plus 14 years.
"It suddenly occurred to me what the judge just said, and my knees gave out and the guards had to hold me up," Ray recalled.
Ray was sent to a maximum security prison in 1985. He would spend the next twenty years behind bars.
Ironically, if Ray had admitted guilt, he might have been able to shorten his prison sentence to possibly two years. But his insistence that he was innocent seemed to hurt him, as he was repeatedly denied consideration for parole.
During this time, Ray was forbidden to communicate with his children.
"It's very difficult because you wonder about those firsts -- baseball games, first date, graduation -- those are the things that haunt you in a prison cell at three in the morning," he told "20/20."
During his time behind bars, Ray rekindled a relationship with his first love, Norma Kohlscheen, who was certain of his innocence and made it her personal crusade to prove it. In the early 1990s, she hired a new defense attorney, Peter Camiel, and private investigator, Paul Henderson, who began to pick apart the original case against Ray.
They first re-examined the accusations of repeated rape by Ray's own children which had come after continued interrogation by Detective Sharon Krause. Major flags were raised when they learned that Krause's sessions were in a private hotel room, were not recorded, and that she bought them gifts, candy and ice cream.
"It was Krause's word and only her word as to what the child said to her," Henderson said.
"20/20" interviewed Dr. Stephen Ceci, an expert on child testimony who found fault with Krause's report.
"She says the child said something, but did the child say that initially or was there denial, denial, denial -- and then finally the child makes a disclosure because she wants to terminate what for her has become a harangue?"
Ceci said that by taking kids to a private hotel room, or by buying them cookies, candy or other sweets, as Krause did, the seeds are sown for highly unreliable statements.
To find out other ways the case was tainted, watch "20/20" TONIGHT at 10 p.m. ET
Sharon Krause declined to speak with "20/20."
Another critical point of concern for the defense was Ray's son Matt's denial of abuse for over eight months until he was told he would have to take a polygraph, which he said scared him.
"When he does come up with a story of being abused, he indicates it wasn't just his father, but that there were other police officers that abused him -- just this fantastic, bizarre story," Camiel said.
And as for allegations that Ray abused his step-son in his motel room, Camiel and Henderson raised some questions about the boy's mother -- Ray's second wife -- Shirley Hansen. Why, after they were separated, would she leave her son alone with Ray, who by then had been charged with raping his own kids?
"One of the explanations is that it was a set-up," Camiel said. "Can I prove that? No. But it sure looks like it."
After a little digging by Camiel and Henderson, the defense believed they found a reason why Shirley would want to set-up Ray; evidence pointed to the fact that Shirley was having an affair with the supervising detective on the case, Sergeant Michael Davidson.
"The motive for going after Ray is clear to me -- it was another police officer's desire for his wife [Shirley]," said Henderson.
Shirley Spencer and Sergeant Davidson refused to speak to "20/20."
Current Clark County Senior Deputy Prosecutor, John Fairgrieve, spoke about the case with ABC News and addressed allegations of an affair.
"Whether it had any impact on the actual investigation itself, there's a question," he said, "But regardless, it was a poor decision by Sergeant Davidson to do that."
But the bombshells kept coming -- medical exams of Katie Spencer and Ray's stepson, Little Matt, were uncovered a decade later, after sitting in a file with Clark County investigators. The results rocked the case: no physical evidence of sexual abuse.
"These were the most graphic, violent, severe allegations. It's exactly the kind of abuse that you would expect would leave physical injury," Camiel noted.
Despite the discoveries, Ray's defense team struck out on appeals. In the fall of 2003, with all other legal avenues exhausted, the defense made a long-shot appeal to Washington Governor Gary Locke to commute Ray's sentence.
More than a year later, in December 2004, Ray's sentence was commuted by Locke. At the time, he had served the longest sentence for a sex crime in Washington State -- almost 20 years.
Locke's decision was so surprising that it caught the attention of local newspaper reporters, Stephanie Rice and Ken Olsen of The Columbian.
"Governor Locke is a very cautious man. For him to commute this sort of sentence, meant there was something really rotten at the heart of this case," Olsen told "20/20."
Olsen made several calls to Matt Spencer following Ray's release. Initially Matt Spencer hung up, but he eventually spoke to Olsen on the phone, and told him that he lived in agony over the allegations for years.
After Olsen's call, over several months, Matt decided he could no longer live with a burden that had haunted him for decades.
"We had been told that we had suppressed it, we were blacking everything out. And I finally realized, I am not blocking anything out. I remember everything else," Matt explained. "I realized, I need to make this right."
In the fall of 2006, Matt boarded a plane in Sacramento headed for Seattle, where his father now lives. During that trip, Matt confessed to his father that he knew nothing happened, and that he had lived with the guilt for years. Matt also met with Ray's lawyer to officially recant the testimony he made twenty years earlier.
Katie, who had started a family of her own in Sacramento, was at first apprehensive. "You don't know how to admit to him that you don't think this ever happened. How do you say that to someone you sent to prison for 20 years?"
Ultimately, she followed Matt's lead and reunited with Ray in 2007.
Matt and Katie told "20/20" that healing with their father has caused turmoil with their mother, who still believes that Ray is guilty.
"I believe what my children told me. I believe what their behavior told me. And something happened," DeAnne Spencer told "20/20."
In July of 2009, Katie and Matt were cross-examined in a hearing where they successfully recanted the testimony that put their dad in prison. They firmly and repeatedly stated that they were never abused by their father.
This year, the Supreme Court of Washington overruled the lower courts during appeals and set a date for Ray to clear his name. The Court was harsh towards the investigators, calling some of the missing evidence "difficult to fathom."
But prosecutors still had the option to re-try Ray, a decision he anxiously awaited.
In September this year, when Ray went into court to withdraw his original plea in court, prosecutors revealed to Ray's attorney they had decided not to re-try him. After the hearing, the Clark County Prosecutor's Office issued a press release that included an explanation of the decision to dismiss the case:
"First, to try the case at this time a jury would need to rely on the memory of witnesses to events from some 25 years ago. For example, the defendant's step-son, Matt Hansen [Little Matt], who is one of the victims, is now 30 years old. He was five at the time of the alleged crimes. Although he still insists he was sexually assaulted by the defendant, one cannot alter the fact that he would now be testifying to events of his very distant past, when he was a very young child. Second, even if a jury nonetheless were to convict the defendant in subsequent trial, the defendant, having already served a substantial prison sentence, would probably only have a duty to register as a sex offender with no additional post-release supervision or conditions. This is because his original sentence was conditionally commuted by Governor Gary Locke in 2004. Given this situation, the value to further prosecution would be far outweighed by the huge cost to our taxpayers of additional litigation."
With the withdrawal of his original plea and the prosecutors' dismissal of the case, Ray was finally a completely free man.
"I can't change what happened. I can't get those years back. So I have to take satisfaction in the fact that I walked out of that courtroom 25 years later a free man," he said.
But would he ever forgive his children for stripping him of twenty years of his life?