"Actually, I have only met him once," the Kardashian patriarch and 1976 Olympic gold medalist told Extra Friday.
"He's never … he's not around," he laughed, adding, "He was in Paris the whole time writing, and he just hasn't been around."
Even more surprising, Jenner's son Brody Jenner, who joins the family reality-TV series, "Keeping Up With the Kardashians" this season, said, "I've never met him."
But Brody was quick to add, "Kim says great things about him. And so does the entire family."
According to various reports, West, 35, did make an appearance at the end of Kim Kardashian's baby shower Sunday.
Sister Khloe Kardashian had explained Friday to Ryan Seacrest that West originally felt "like it's weird for him to be the only guy."
Khloe's husband, Lamar Odom, and sister Kourtney Kardashian's beau, Scott Disick, had been expected to bring him.
But West was not at the doctor's appointment where Kardashian learned that she is having a girl.
"The baby was always in a position where you couldn't tell the sex, and this time that you're going to see on the show, [Kanye] was in London or Paris," Khloe said. "He had a concert."
On Sunday's season 8 premiere of "Keeping Up With the Kardashians," the future-mama-to-be gushed, "I'm so excited we're having a girl. Who doesn't want a girl? They are the best and I know that's really what Kanye has always wanted. He wanted a little girl."