Welbeck sent Arsenal through to the semi-finals of the FA Cup with a 61st-minute winner after Wayne Rooney had equalised Nacho Monreal's opener.
To pile yet more misery on the United boss, he watched Angel di Maria get sent off towards the end of the 2-1 defeat for pulling referee Michael Oliver's shirt as he reacted angrily to being booked for diving.
Replays showed Di Maria was pulled back by Aaron Ramsey before he tumbled to the ground, but the way in which he reacted will not please United boss Van Gaal.
Di Maria can have little reason to complain
The Â£59.7million signing snatched at Oliver's shirt as he turned away after issuing him with a booking.
The dismissal is the latest episode in a long list of disappointments for the British record signing, who Van Gaal hoped would bring silverware to Old Trafford.
With United 10 points behind Chelsea in the league, that dream is almost certainly over
After a mildly encouraging performance in the first half, United wilted and looked blunt after the restart
Welbeck, the Longsight lad who left his boyhood club for north London last summer, provided the killer touch after a clanger from Antonio Valencia.
Van Gaal claimed he controversially ditched Welbeck last summer because the striker did not score enough goals.
And while the forward has hardly been prolific at Arsenal, he will have enjoyed showing Van Gaal what he is missing here.
For Van Gaal, the dream of silverware may be over, but Arsene Wenger's side are now very much favourites to regain the FA Cup after registering their first win at Old Trafford in nine years.
A look back at recent history suggested the capacity crowd inside Old Trafford was in for a treat.
Meetings between the two teams have thrown up some classics over the years
Spectacular goals have been scored, intense battles have been fought, red cards have been issued and even pizza has been thrown in the tunnel.
But, with neither team the major force they were 10 years ago, this match had a low-key feel to it at the start.
Both started cautiously
Marouane Fellaini managed to treat United fans to an equal measure of joy and despair in the 13th minute when he turned brilliantly in the box and then clattered into Per Mertesacker and gave away a free-kick.
Welbeck looked cautious on the attack, but it was not the same down the other flank where Alexis Sanchez drilled a low shot which tested David de Gea.
Di Maria cut a frustrated figure on the right
He cut in from the flank and fired in curling crosses, but United's front two of Rooney and Fellaini failed to take advantage
The one time Fellaini did take the ball down well, he laid the ball off to Ashley Young inside the box, but he fired over.
Arsenal raced up the other end on the counter
Alex Oxlade-Chamberlain sprinted past his marker before trying and failing to beat De Gea on the lob.
The impressive England midfielder did much better moments later when he helped the Gunners take the lead
He picked the ball up off Mesut Ozil, tip-toed his way past three United players and squared the ball to Monreal, who stroked the ball past De Gea.
The 9,000 Arsenal fans inside Old Trafford fell silent four minutes later when Rooney equalised.
Di Maria sent over another dangerous dipping cross
This time Rooney had peeled away from his marker Laurent Koscielny and he speared a diving header past Wojciech Szczesny from close range.
Arsenal appealed for a penalty after Marcos Rojo pulled Welbeck back, but referee Oliver waved play on.
Rooney was now brimming with confidence
He plucked a long ball from the sky and scooped a delightful 20-yard pass to Di Maria, but he could only stab the ball at Szczesny.
Di Maria found another way through the Arsenal defence before the break after a cutting pass by Valencia, but again Szczesny denied the Argentinian.
Van Gaal brought Phil Jones and Michael Carrick on for Luke Shaw and Ander Herrera at the break and Arsenal were also forced into a change when Ramsey replaced Oxlade-Chamberlain, who had suffered a hamstring injury.
United started to look more threatening
Fellaini created problems as he tried to plough his way through the middle
Di Maria used his pace to get in behind the Arsenal defence, but he could only blast wide.
Then, with just under half an hour to go, Valencia handed Arsenal a gift-wrapped ticket to Wembley.
The right-back sold De Gea short with a backpass after Jones had chested a long ball down to him and Welbeck sprinted through and tapped into an empty net after beating De Gea to the ball.
The former United striker thought about not celebrating for a second, but then pumped his fists with joy.
United fans responded by demanding their team attack, but they looked inept
De Gea stuck out his left hand to deny Cazorla and Arsenal looked comfortable.
Di Maria rounded off a miserable for the hosts when he was dismissed for tugging Oliver's shirt after he had cautioned him for diving.
Adnan Januzaj appealed for a penalty soon after, but Oliver also booked him for simulation and Arsenal were able to coast to victory.
Source : PA