Tries by Chris Ashton after two minutes and Dylan Hartley had England on a high as the world champions were harried into mistakes in difficult, rainy conditions.
But they pulled back to within five points just before halftime and edged clear soon after and, helped by a key, late TMO decision that cancelled out what England thought was a match-winning try, made it 15 wins in the teams' last 16 meetings.
Despite failing to add to that solitary 2012 success, England, who face Japan next week, will take real heart from their performance while the All Blacks will look to cut out the errors when they face Ireland.
From the moment the Twickenham crowd drowned out the haka with a rousing Swing Low as the rain swept across the stadium, it felt like it might be England’s day and after a breathless opening half an hour everyone watching was starting to believe it as the home team delivered on their promise of trying to knock New Zealand out of their comfort zone.
Relentless forward pressure from the start opened a huge hole which scrumhalf Ben Youngs spotted, looping a long pass for Ashton, making his first start for four years, to slide in the corner.
England continued to dominate, kicking to make the All Blacks turn at every opportunity and seizing upon a rash of errors by the visitors before setting up Farrell to slot a drop goal.
England’s forwards then took centre stage with an unstoppable rolling maul straight out of their 1990s playbook that carried Hartley and half the New Zealand team over the line.
Farrell converted to make it 15-0 with the All Blacks never threatening an attack.
Six years ago, in England’s only win in the last 15 editions of the fixture, they also led 15-0 only for New Zealand to charge back to within a point before England forged clear.
But the Kiwis finally held the ball long enough to force a penalty in front of the posts and skipper Kieran Read opted for a scrum. It looked an ambitious decision but paid off when Barrett’s pop pass sent McKenzie over under the posts.
The All Black first five-eighth then dropped his first goal in 71 tests to close to within two points early in the second half.
England twice opted for lineouts when given very kickable penalties and lived to regret it when the All Blacks were offered the same opportunity on the hour, Barrett taking the shot to put them in the lead for the first time.
Both sides continued with their kicking game, but England failed to take advantage as their lineout fell apart, with man-of-the-match lock Brodie Retallick a constant menace.
England thought they had won it five minutes from time when Courtney Lawes charged down TJ Pereira's clearance kick and flanker Sam Underwood scooped up the ball and threw a dummy to befuddle Barrett and reach the line.
A week ago England got the right side of a late TMO call when they beat South Africa 12-11 but this time Lawes was adjudged to have moved a fraction too quickly and the All Blacks managed to play out the remainder safely.
"It comes right down to the wire, that’s what you want in Test matches and why I always love coming to Twickenham," said Read.
England coach Eddie Jones called it "a fantastic game of rugby".
"We're obviously devastated but you take the good with the bad and we'll take a lot from that," Jones told reporters.
"We had opportunities to win the game. We didn’t take them, they did so they deserved to win but it was a good test-match tussle that we are only going to improve from.
"It was a really good step forward because you benchmark yourself against New Zealand and we will get a lot of reward for the work we've done."
All Blacks 16 (D. McKenzie tries; B. Barrett 2 pens, cons, DG)
England 15 (C. Ashton, D. Hartley tries; O. Farrell cons, DG)