"Children would tease me. I would have gestures...flapping of the hands, walking in circles," he said. "The other children would repeat that back to me, call me names."
It was Tammet's obsession with numbers that led to an incredible feat on March 14, 2004, known as Pi Day, when Tammet broke the European record for reciting the number Pi from memory.
Pi, the ratio a circle's circumference to its diameter is considered an "irrational" number in mathematics because it does not end. You may be able to remember the first few digits -- 3.14159 -- but not more.
Tammet says he only read through the digits once and was able to remember 22,514 of them. After a couple weeks to practice reciting the numbers back, in order, it took Tammet just 5 hours and 9 minutes to reel off the numbers while mathematicians listened and simultaneously checked every digit.
To memorize a long number like Pi, Tammet said he just forms a beautiful landscape out of the shapes he pictures in his mind: "I'm taking the numbers, I'm making them into colors and shapes. I'm able to put those into combinations which form hills…or ground or sky…It's another world that I'm able to go into, experience, live within."
Despite Tammet's limitations as an autistic savant, he is able to leave his vibrant autistic world and communicate in a way that's impossible for most savants and remarkable for any person.
"As with numbers, language is something that I have an exceptional ability, and it's something that I can perform very well in beyond most people's ability," he said.
Tammet was even challenged to learn Icelandic, perhaps the most impenetrable language on earth, in just one week. While some said it was impossible, Tammet says, "When people say to me, 'This is something you can't do,' I want to do it. I want to prove them wrong."
Simon Baron-Cohen, director of the Autism Research Center at Cambridge University, said Tammet has mastered languages by understanding them as code. "He's treating vocabulary and grammar as systems and he wants to crack the system, crack the code," he said.
After just seven days of learning Icelandic, Tammet appeared live on an Icelandic television talk show and proved his gift to the world.
These days Tammet shares his intellect in writing. He uses his synesthesia to visualize sentence structure and many have appreciated the beautiful landscape of his prose. His autobiography, "Born on a Blue Day," is a bestseller that has been translated into 20 languages.
His second book, "Embracing the Wide Sky," details how our brains really work and how Tammet overcame autism: He trained himself to make friends and tell jokes, stuff that for most of us is second nature.
"These are skills that can be learned. I'm living proof of that and there are many who are like me who have acquired those skills through effort, through the love and support of their families as well."
Tammet is currently at work on a novel, due out early next year. He spends a lot of time traveling, lecturing and sharing his gifts with scientists and adoring audiences alike.
He's learned to enjoy the disorders that caused him so much pain earlier in life.