United made it eight goals in two Premier League games by following up their opening victory over West Ham with another four-star show at the Liberty Stadium.
Eric Bailly put them ahead with his first United goal on the stroke of half-time before Romelu Lukaku, Pogba and substitute Anthony Martial scored in the final 10 minutes
Pogba had earlier escaped a second yellow card for a clumsy challenge on the half-hour mark, just four minutes after being booked for a similar foul on Tom Carroll.
But asked if he thought Pogba would see red, Mourinho said: "No, because I didn't think the first yellow card was deserved.
"The first yellow card is a little bit out of context because I saw Jonathan (Moss) had a criteria where certain action wouldn't bring yellow cards.
"There were actions, especially one on (Nemanja) Matic, that is a yellow card and he decided to speak with the players.
"I thought with Paul he would do the same, so when the yellow card came I was a bit surprised
I didn't fear the second yellow card."
United have not started a league campaign as well in their opening two matches since beating Aston Villa 4-1 and Liverpool 4-0 at the start of the 1907-08 season.
But Mourinho refused to get carried away, especially as United began last season with wins against Bournemouth and Southampton.
"The word that describes the team now is confident," Mourinho said.
"I don't want to see it but, if it happens, I want to see the team losing and see the way we emotionally react to it.
"It will be another stage, losing and trying to change the result, because at the moment everything is going in our favour.
"It's not always motorway - you find difficult roads and roadworks.
"I prefer to say we started last season with two matches, six points, and we finished sixth.
"Two matches are not the end of the world for people who lose, and it's not paradise for people who win."
On United's victory, he added: " The last 10 minutes (looked impressive) because the opponent was losing and trying to get the goal
"The reality of the game was 75 minutes 1-0
For 75 minutes the result was open, even if I felt we were in control of the match."
Swansea's shortcomings were plainly evident following the club-record sale of Gylfi Sigurdsson to Everton and the loss of last season's top scorer Fernando Llorente to injury
There was no creativity and - with the generous allowance of a Martin Olsson overhead kick as an on-target attempt by official statisticians - Swansea managed their first goalbound effort in two games.
Head coach Paul Clement now has less than a fortnight to shape the destiny of Swansea's season by investing the Sigurdsson funds in strengthening his squad.
"We absolutely need more quality players to come in, so we have a chance of being competitive this year," Clement said.
"The sooner we can do it the better
It could go later in the window but the necessity is there
"We were competitive for long periods of the game
Up until the second goal the difference was a set-play and generally the shape was good.
"I made some offensive changes because I don't want to be a coach happy to lose 1-0 at home, but we ended up playing to their strengths and they punished us."
Source : PA