Yannick Agnel, swimming the anchor leg for the French, reeled in Ryan Lochte on the last lap to win gold for France in three minutes, 09.93 seconds.
The Americans, who gambled and lost when they called up Lochte ahead of their sprint specialists, held on to take the silver medal after an exhilarating finish.
Russia got the bronze medal while Australia, who dubbed themselves a "weapon of mass destruction" after winning last year's world championship title, could only manage fourth.
Michael Phelps swam the second leg for the Americans and picked up the first silver medal of his Olympic career to lift his overall medal tally to 17, just one shy of the all-time record.
Four years ago, Phelps won an unprecedented eight golds in Beijing, including the men's relay when the U.S. famously came from behind to beat France on the last lap.
