Fianna Fail and Sinn Fein each hold 37 seats in the 160-seat Dail, parliament’s lower house. Fine Gael has 35.
Thursday’s session opened with the election of a speaker, followed by parties nominating their candidates for Taoiseach, or prime minister.
Sinn Fein leader Mary Lou McDonald, Fianna Fail leader Micheal Martin and Fine Gael leader Leo Varadkar were all nominated to the top post, but none secured enough votes to be elected.
Varadkar, the current Taoiseach, formally resigned as prime minister after the votes, but will stay on as caretaker leader until a successor is chosen. He's due to travel to Washington next month for the traditional St. Patrick's Day visit to the White House by Ireland's leader.
The election result was a major blow for Fianna Fail and Fine Gael, which have dominated Irish politics since the country won independence from Britain a century ago.
The two parties have long shunned Sinn Fein because of its historic links to the Irish Republican Army and decades of violence in Northern Ireland.
But Sinn Fein’s radical proposals for tackling Ireland's housing crisis and creaking healthcare system proved a powerful draw for young voters in a country that is still dealing with aftershocks of the 2008 global financial crisis, which hammered its debt-driven “Celtic Tiger” economy.
McDonald said she would continue to seek coalition partners in the coming days and weeks. So far, Fianna Fail has ruled out an alliance with Sinn Fein. Varadkar has said he expects Fine Gael to end up in opposition.
McDonald said Sinn Fein would " talk to everybody because that's what you do in a democratic system, that's what grown-ups do."
"I do know this much, I know that nobody voted for a return of Fianna Fail and Fine Gael,” she said.
"I also know from speaking to people out and about, people of all politics and none, that any suggestion of return of the old Fianna Fail and Fine Gael government will cause huge disappointment right across the country and, I imagine, huge anger.”