He has revealed that he is "permanently scarred" by abuse and hatred from fans.
"Some fans develop an intense dislike and hatred of you.
"Some are so passionate that if you do something against their team, it's like you did something against their family.
"And they don't forgive you. They remember something you did ten years ago. At one point, I was getting hate letters and death threats at Harrow from a small minority of Manchester United fans. I told security, they told police and then officers came to install Home Office alarms in my house.
"Fans expect me to be right everytime I blow my whistle and they don't remember my right decisions.
"One wonders how I would have coped if I had nothing in my life apart from refereeing."
Elleray described Keane as "aggressive and difficult", but among the best players in the world.
He added: "He doesn't say much, but when he speaks, players take note of what he says.
"He always wants to get on with the game. He's not one to chat like David Beckham and Dwight Yorke.
"You didn't tend to pass the time of day with Roy. He tended to know what was what."
When Elleray retired last year, his refereeing colleagues asked Keane for a signed shirt. Instead, they got a cheeky letter saying he would never forget Elleray for "all my suspensions and fines.