It's unclear whether the state will pursue a retrial of the cop, William Porter.
The jury deliberated over three days, signaling at one point that they might not reach a unanimous decision. Jurors told the court Tuesday they were deadlocked but the judge sent them back to deliberate.
Gray died in April from a severe spinal injury while in custody after being arrested when he fled from the police. Porter allegedly failed to get medical help for Gray as the transport vehicle carrying the suspect made several stops in Baltimore after picking him up on the way to the police station.
Baltimore mayor Stephanie Rawlings-Blake urged calm in the wake of the decision.
"This is our American system of justice. Twelve Baltimore residents listened to the evidence presented and were unable to render a unanimous decision," she said in the statement released on her Twitter account.
"In the coming days, if some choose to demonstrate peacefully to express their opinion, that is there constitutional right. I urge everyone to remember that collectively, our reaction needs to be one of respect for our neighborhoods, and for the residents and businesses of our city," she said.
Congressman Elijah Cummings, whose district includes a portion of Baltimore, released a statement about the mistrial and urging people to "ensure that the process of healing our community continues."
"I know that many of my neighbors have been following this trial closely, and many may be disappointed by today’s outcome. Each of us will continue to struggle with the very raw, very real emotions the death of Mr. Freddie Gray invokes," he said.
The police department officials canceled all time off this week so they can be fully staffed if a verdict is reached. The city is on edge as Gray's death in April prompted protests and riots across the city and a temporary state of emergency was put into effect.