Zeke Brown, who lives with his family in the Atlanta area, told ABC News on Thursday that he'd placed the ad looking for a repairman on Facebook's local marketplace.
When the repairman arrived on June 29, Brown said he didn't notice the flag at first until his son mentioned it as he was pulling out of the driveway.
"My son immediately said that he was gonna leave because he knew nothing good was gonna come of it," he said.
Brown said that he spoke to the young man, whom he described as white and in his early 20s, and had what he called a "very polite conversation," before he went inside to speak to his wife.
"I just couldn't believe that he had the flag on the truck and was coming to my house that way," he said. "I was in disbelief."
The Browns' doorbell camera captured the interaction between the couple and the man identified as Mike as they conversed outside the couple's home.
"Hi. You know what? I do apologize ... we're gonna use someone else," Allison Brown can be heard on the video saying to the man before he gets into his truck and drives away.
"I'm beyond upset with the flag," she says. "You continue to believe what you need to believe, sir. But, no, I cannot pay you for your services. Thank you. Have a good day."
Allison Brown told ABC News that she'd handled the situation the way she did because she "wanted to set a good example" for her sons.
"I didn't feel it was necessary for me to raise my voice or to cuss or to fuss," she said Thursday. "There's enough negative image of black women being angry ... I didn't want to step outside of my character."
Nevertheless, she said she wanted to be clear to the man that the flag was unacceptable on her family's property and that he was not welcome.
"Nor did I need his business if that's how he wanted to conduct himself," she said. "Had he not had the flag on his truck and I had not known, I would've allowed him to do the work as he needed -- and probably would have hired him for other jobs."
The Browns said the man had offered to remove the flag from his truck but they still declined his services.
"I felt like at that point, you've already made yourself very clear. You already have shown your true colors," Allison Brown said. "Because he came to our home in that disrespectful, violating manner, I sent him on his way."
They couple told ABC News that they had held onto the video for their own safety, in case they needed to turn it over to police.
Zeke Brown said the man had written him on Facebook soon after leaving, saying that he did not mean to offend the couple.
Brown said that he'd responded by saying the Confederate flag is disrespectful and encouraging him to research what the flag means to people of color.
"The times when that flag was brought up, multiple times during history, it was always to our (African Americans’) detriment ... negative to our plight," he said.