Mohamed Abdiwahab/AFP/Getty Images

Thousands of residents are fleeing the last Somalia city controlled by Al Qaeda-affiliated militant group Al Shabab, and there are reports that the militants themselves also are leaving Kismayo ahead of an expected military assault.

Defeating the hard-line Islamist militants with a reputation for brutally violent enforcement of Sharia law is a necessary step toward returning peace to the troubled nation that has been wracked by inter-clan conflict and without a functioning central government for decades.

U.N.-backed African Union troops forced Al Shabab out of the capital city of Mogadishu last year and have been slowly advancing on Kismayo for the past few months. Kenya's army says its troops are now within 20 miles of the port city which is Al Shabab's main source for weapons and funding through the export of charcoal.

"It looks like a wave a panic," said Bruno Geddo, the U.N. refugee agency's representative for Somalia.

"According to our monitors in the field they are moving in a preemptive mode because they fear they might be caught in the crossfire," said Geddo. "There is the fear Al Shabab will use humans as human shields and will fight within the city."

Geddo said of the more than 5,000 civilians who have fled to neighboring towns in the past couple of weeks, more than 3,000 of them left on Monday and Tuesday. The exodus could be a sign Al Shabab is already losing control of the port city where residents previously said Al Shabab barred them from leaving with threats of violence.

Some residents in Kismayo told reporters they have seen Al Shabab fighters leaving the city in trucks with heavy equipment, and an official with the African Union force in Somalia said in a statement today many are surrendering.

"Already a number of them have contacted us indicating their wish to cease fighting and we have assured them of their safety if they give themselves up to our forces," said Deputy Force Commander Maj. Gen. Simon Karanja.

An Al Shabab spokesman denied the claims the militants were pulling out of Kismayo with postings on Twitter calling them "blatant lies." One resident told the Associated Press he still sees some Al Shabab fighters on the streets, but government offices are empty.

"It feels like the town is under no one's control now," said Yusuf Hashi.

Though the military defeat of Al Shabab in Kismayo may be imminent, analysts expect the insurgent group will continue to cause insecurity throughout the country, as they have been able to continue to do in the capital city with frequent grenade attacks and suicide bombings.

"They will have lost their final substantial source of income. They will have been relegated from authorities to outlaws in all important urban centers. But they will not disappear. The group's leaders are still at large and will likely maintain a following," said James Smith, Horn of Africa Project Manager at the Rift Valley Institute.

Smith said a far greater challenge than ousting Al Shabab from key cities will be to replace them with good governance, something most Somalis have never experienced.

"Al Shabab emerged from a vacuum of effective government in Somalia. It or its successor will be quick to re-emerge it the current opportunity to offer a decent alternative is not seized," said Smith.