One of the men, Joel Williams, who worked part-time at the stadium, told KHOU-TV that he had his picture taken with Brady just as he was ready to clock out from his shift.
After taking the picture with Brady, Williams said he was run down by a supervisor, who told him he was fired.
Another security guard, Christopher Moore, also had his picture taken with Brady and was fired as well.
Williams said he understood the policy that security guards were not permitted to take pictures with people while on duty but said he was ready to clock out when he ran into Brady.
"I said, 'Hey, Tom Brady, good job, good game, man.' He looked at me and smiled," Williams told KHOU-TV. "He was very polite, very nice guy, humble. Sometimes I see players and say hello and they just keep walking."
The company the men worked for, Contemporary Services Corporation, said in a statement to KHOU-TV that it stood by its decision to fire the two.
"It is strictly against CSC policy for its employees to request photos or autographs from players," the statement said. "CSC stands by its decision to terminate the two employees who violated this policy."
Moore, however, claims that the policy hasn't been strictly enforced.
"What they say and what they practice are two different things," Moore told KHOU-TV. "What we see on a daily basis, whether it be Reliant or CSC employees, is them taking pictures and asking for autographs from players."