The French paid the price for some sloppy rugby and poor decision-making after halftime as Japan's "Brave Blossoms" took the game to them and at one stage closed to within four points of the nine-times European grand slam winners.
In the end, though, tries from Julien Pierre, Francois Trinh-Duc, Vincent Clerc, Lionel Nallet, Pascal Pape and Morgan Parra secured them a bonus point win in Pool A, which also includes hosts New Zealand, Tonga and Canada.
"I got a victory, which was the most important thing for me, and then I wanted a consistent game but in this regard I wasn't too happy," said France coach Marc Lievremont.
Asian and Pacific champions Japan, overwhelmed and heavily penalised at the scrum, played some superb rugby with ball in hand and raised the biggest cheers of the evening for the two tries from flyhalf James Arlidge, who scored all of his team's points.
"I think it was a positive evening out and we just need to keep working hard," said Japan's coach John Kirwan. "It was important to play well and show the world we are a very competent football team."
The Japanese fightback had looked unlikely at the start of the match when it looked like they would be blown away by France's heavyweight firepower.
Centre Fabrice Estebanez set the tone as early as the third minute when he burst through the Japanese midfield defence only to be denied a try by a last-ditch tackle.
The Tricolore-waving French fans did not have to wait long for the first score, however, with lock Pierre storming over the line two minutes later after a bulldozing charge from number eight Raphael Lakafia.
When flyhalf Trinh-Duc raced 50 metres to score France's second after an interception with just 12 minutes on the clock, it looked like the Japanese, who had only won one previous World Cup match, would be routed before halftime.
New Zealand-born Arlidge scored his team's first points with a penalty, however, and the Japanese restricted France to only two Dimitri Yachvili penalties to leave the score at 20-3 with 25 minutes gone.
Just after the half hour mark, a penalty kicked to touch in the corner was rewarded when Arlidge cut inside the defence to score his first try and, although winger Clerc grabbed one back for France four minutes later, Japan got another penalty to go into the break 25-11 down.
"I wasn't happy even at halftime," Lievremont added. "I wasn't happy at the beginning of the game, we weren't ambitious enough. We didn't seem to be playing with a strong ambition. We had strong scrums ... but we weren't playing together."
The French, playing in white, opened the second half with renewed purpose and power and were denied two tries by television reviews.
But, having weathered that storm, Arlidge opened 20 minutes of Japanese dominance when he skipped over the line to score his second try.
He added the extra points himself and, with France in disarray, kicked another penalty to cut the deficit to 25-21.
Another Yachvili penalty calmed the French nerves a little but Japan were throwing everything at their opponents now and it was only when lock Nallet stormed over with 10 minutes to go that order was properly restored.
"We will be disappointed tomorrow as there were times when we could have done a lot more," New Zealander Kirwan added. "We had them under the pump and they were making uncharacteristic errors. We just didn't quite take it away."
Replacements Pape and Parra put a gloss on the scoreline with two more tries before the end but the French performance will not have main group rivals New Zealand quaking in their boots.