Captain Antolin Alcaraz was dismissed for a second bookable offence after half-time - striker Hugo Rodallega also saw red two minutes later - while Charles N'Zogbia was denied what the Latics felt was a clear penalty with the score still goalless.
"I wasn't even worried he might go off for the [second] challenge, to be honest," said Caldwell of Alcaraz's dismissal after a tackle on Darren Fletcher.
"There were tackles flying in from their side - and I know I'm biased - that weren't getting punished, and could possibly have been bookings.
"But I thought it was an honest challenge from Antolin to get the ball, and although it was a certain foul on Fletch, which we accept, I just felt the referee could have shown a bit of leniency.
"Maybe the pressure from the crowd and whatever meant he sent him off."
Caldwell said: "I also thought it was a stonewall penalty and I'm certain that if it had been a Manchester United player it would have been a penalty.
"I said that to the referee at half-time, and he said he felt that Charlie went down far too easily.
"My argument was why would he go down when he was in such a strong position.
"It was such a big decision that had a bearing on the game, because they got their first goal within a couple of minutes and all of a sudden we're 1-0 down at half-time."
Source: Manchester Evening News