"If you did this to your own children, you know, you deserve to be locked away," he said. But "I didn't do it and I wasn't going to admit it. You can either roll up in a ball and let life take you or you can just take what comes down the road and maintain integrity."
For twenty years, he fought to prove his innocence, uncovering shocking secrets and a police investigation that may have been tainted from the start, that would reveal the truth, and ultimately exonerate Ray.
In the early 1980s, Ray Spencer was a cop in Vancouver, Wash., where he was married with two young kids. Ray had a reputation around town for being arrogant and for having a weakness for women. It was a weakness Ray admittedly indulged and his wife painfully endured.
"It's not so much the rage and anger I had, but I didn't believe anything he ever said was the truth," his first wife DeAnne Spencer told "20/20."
DeAnne decided she had enough. The couple divorced in 1981 and she took their kids, Katie 2, and Matt, 5, and moved to Sacramento, Calif., to start a new life.
Alone, and separated from his children for months at a time, Ray turned to Shirley Hansen for solace. Two years after his divorce, Ray married Hansen, who had a three year old son of her own, whom they called "Little Matt."
Familiar with the signals children send out, Shirley was alarmed when Ray's daughter Katie, who was visiting during the summer of 1984, suggested she was being inappropriately fondled.
"Katie put her hands between her legs and told Shirley, 'My brother does this, my mom does this, my dad does this,'" Ray recalled. "Now my assumption was that 'Daddy' was a boyfriend that [DeAnne] was dating."
Ray immediately called Child Protection Services and the Sheriffs' departments in both Clark County, Wash., where he worked, and Sacramento, where his children lived with their mother. After a brief investigation, California authorities decided not to pursue the case. But Washington authorities decided that they would look into it.
Ray Spencer was pleased the case was being explored, but he had no idea what lay ahead.
Tasked with the investigation of possible child abuse, Detective Sharon Krause of Clark County made several trips to Sacramento to question Ray's two children. She took Katie and Matt separately to a hotel to be questioned, but did not make audio or video recordings of the sessions, relying instead on handwritten notes.
Katie and Matt -- then 5 and 8 years old -- initially denied any abuse. But then Katie began to give lurid accounts of rape and shockingly, pointed the finger at her own father, according to Krause.
"I said, 'You must be crazy.' I'm the one that reported this," Ray recalled.
Ray was soon charged with raping his own five-year-old daughter. He was released on his own recognizance, but his life continued to spiral downward: he lost his job, separated from his second wife, Shirley Hansen, and began living in a motel.
Shortly after his separation, in February 1985, Shirley asked Ray if her son, Little Matt, could spend a night with him at his motel, which Ray welcomed. Four days later, Ray was arrested and incarcerated -- this time for the rape of his stepson.