Arsenal scored two late goals to ruin Norwich's hopes of a shock
"After the final whistle most of the Arsenal fans around me emptied out but I wanted to clap off the Norwich players because they'd given everything," he said.
"I went to the front of the stand and was clapping and singing in time with the Norwich fans below me. Then my friend said it looked as if an older guy with the Norwich supporters was trying to get my attention as if he knew me.
"I bent down to listen and quickly realised he didn't know me. He probably thought I was an Arsenal fan and he was shouting and swearing at me, saying pretty nasty stuff. I was flabbergasted.
"The player who scored our goal Josh Murphy is mixed-race and some of our great players have been black. This man must have cheered them on in his time.
"He was maybe in his seventies. I'm pretty thick-skinned as a professional athlete but it was really disappointing and a shame that it tarnished what should have been a nice memory for me because we played well even 'though we lost."
Ogogo does not intend to make a formal complaint about the incident and says he tweeted about it to raise awareness.
He said: "I'm not naive. I know things like this still happen but I've never experienced racism at Carrow Road my whole life and it won't stop me watching football anywhere.
"On the train home I was getting quite worked up by it. I don't want negativity or criticism for the club I love but I'm talking about it because this cannot be allowed to go on in 2017."
Ogogo, who he appeared as a contestant on Strictly Come Dancing,suffered a fractured eye socket in his last fight - a defeat by Craig Cunningham - and is preparing for a comeback.