"A meeting of an executive board that includes many women would take time," Mori was quoted as saying by the newspaper. "Women are competitive. When someone raises his or her hand and speaks, they probably think they should speak too. That is why they all end up making comments."
He also referred to the Tokyo Olympics organizing committee, saying it "includes about seven women, but they all know how to behave," the newspaper reported.
Mori's remarks provoked laughter from some of the committee members, according to the newspaper. But on social media, as reports of his comments surfaced, people quickly began calling for Mori to resign.
Speaking at a hastily-prepared press conference on Thursday, Mori confirmed he made the comments and offered an apology.
"It was an inappropriate remark that went against the spirit of the Olympics and Paralympics," he said. "I deeply regret it and would like to sincerely apologize to anyone whom I have offended."
When asked about the calls for his resignation, Mori told reporters: "I'm not considering resigning."
Japan's chief cabinet secretary, Katsunobu Kato, declined to comment specifically on Mori's remarks during a press conference Thursday but said the government is committed to gender equality and seeing more women in leadership roles in sports organizations.
The controversy came just as Olympics organizers released the first of a series of "playbooks" for how they plan to hold a safe and successful Games in Tokyo this summer amid the coronavirus pandemic.
The 2020 Summer Olympics were supposed to kick off in the Japanese capital last year on July 24. But in late March, amid mounting calls to delay or cancel the upcoming Games, the International Olympic Committee and Japanese organizers announced that the event would be held a year later due to the pandemic. They have been outwardly staunch in their determination to go forward with the Games ever since, despite the fact that Japan -- and much of the world -- has faced a resurgence of COVID-19 infections in recent weeks.
With less than six months left until opening day, Japan's mass vaccination program has yet to begin and Tokyo remains under a state of emergency due to a climbing number of COVID-19 cases and a growing death toll. Moreover, a recent poll by Japanese news agency Kyodo found that around 80% of people in Japan believe the Tokyo Olympics should not be held this summer.