Lebanese officials say the decision to evacuate is to prevent revenge attacks and further tension after residents of the town threatened to storm the camp.
"This is to ensure their safety and to stop the bloodshed," said Jean Fakhry, a Lebanese municipality official in Deir al-Ahmar told The Associated Press.
The evacuation of more than 90 tents reflects the growing tension in Lebanon over hosting more than 1 million Syrian refugees since 2011.
A country of nearly 5 million, Lebanon's infrastructure and economy has been overwhelmed by the arrival of those fleeing the war next door.
Mostly impoverished and dispossessed, many settled in the country's east, living in squalid camps and struggling to work and survive in the area known for its harsh winters and sizzling summers.
As Lebanon deals with an economic crunch, many Lebanese politicians and groups turned to Syrians, calling for them to go home saying the violence there is winding down.
In the scorching sun of the early afternoon, Syrian men, women and children pulled down cloth and carpets covering the wooden structures that served as their temporary homes. They piled mattresses, pots and pans into trucks headed to a new location miles away. Local officials say 600 lived in the camps but U.N. estimates said it was closer to 400 people.
It is at least the second time the refugees, most of them from Idlib province, had to pack their lives into boxes to find new, safer areas.
Samar Awad, a 27-year old Syrian, said the camp's residents are being sent to a new site with no electricity or water. She said it may be equally dangerous as their reputation as troublemakers will precede them.
"We have been living here for seven years," said Awad. "It has become like home, and more."
Awad hails from Saraqeb, a town in Idlib, the last area controlled by the opposition to Syria's President Bashar Assad.
Seasonal farmers from Syria, like Awad, used come to Lebanon even before the war. But after the violence, many of them stayed, overwhelming the local communities.
Last week, a fire broke out on the hill overlooking the camp sparking panic. When firefighters arrived, residents pelted their vehicle with stones over what they said was the rescuers' late arrival. A brawl ensued and a firefighter was injured and tents were damaged.
The army arrived on the scene and arrested more than 30 Syrians. The camp residents fled and later, unknown assailants set three of the tents ablaze. The tensions led to the evacuation order and the lockdown on refugees in the area.
"There was local anger and revolt," Fakhry, the Lebanese official said. "We decided they should not come back to avoid another problem and bloodshed."
The U.N. refugee agency was on the scene Sunday as the Syrians prepared to leave. In a statement, the agency said all perpetrators should be prosecuted as an investigation is underway.
"A collective punishment of Syrian refugees in Lebanon will further escalate the situation," the UNHCR said. "All refugees in Lebanon should not be punished over one incident and outside the justice system."