Australia's suspect batting will need to produce at least the second-highest successful fourth-innings chase at the Nottingham venue after England batted their way to a commanding 261-run lead at 6-326 at stumps.
The visitors were left fuming at an absolute howler from umpire Aleem Dar with England leading by 230.
Broad (47no) refused to walk and Australia couldn't review the decision because they'd already wasted their reviews.
To add to frustrations, Haddin dropped Ian Bell (95no) off Peter Siddle (1-63) in the next over.
Australia battled hard, but couldn't strike the killer blows.
England chased 284 in the fourth innings to beat New Zealand at Trent Bridge in 2004 and, in all likelihood, Australia will have to better that record effort.
Australia will require their second highest successful fourth innings run-chase ever in England, trailing only a 404 chase at Leeds by the 1948 Invincibles, if they're to take a 1-0 lead in the Ashes series.
When Bell and Broad came together, Australia were in control with England six down and leading by just 153.
But their superb unbeaten 108-run, seventh-wicket partnership took the match away from Australia.
Bell was given out lbw to Shane Watson when on 34 but DRS overturned the umpire's decision.
The dream debut of 19-year-old Agar (2-82) continued when he knocked over Alastair Cook (50) and Jonny Bairstow for his first Test wickets.
But he should have had three.
Had Broad gone, Australia would have had a shot at England's tail and been a chance of wrapping the innings up by stumps.
Players converged on the pitch and stood before Dar but he remained unmoved.
Given it was the end of the over, Broad walked quickly up to join Bell and avoided the commotion.
It followed a harsh DRS call against England on day two, when Jonathan Trott was given out by the third umpire.
Australia did have themselves to blame in some respect due to a challenge earlier in the day which proved way off the mark.
They went upstairs for a James Pattinson (1-89) lbw shout, but hawk-eye showed the ball missing leg by at least another stump.
Earlier Kevin Pietersen (64) and a quick-fire 31 from Matt Prior helped England ultimately put Australia to the sword.
Watson bowled 11 maidens in his 15 overs for figures of 0-11.