The 79-year-old, who has held office since 1998, announced the decision at a news conference in Zurich, six days after Swiss police, acting on a request from United States authorities, raided a hotel in the Swiss city and arrested several FIFA officials.
Blatter has not been charged.
"While I have a mandate from the membership of FIFA, I do not feel I have a mandate from the entire world of football - the fans, the players, the clubs, the people who live, breathe and love football as much as we all do at FIFA," Blatter told a news conference called at short notice and held after a delay.
"This is why I will call an extraordinary congress to be held as soon as possible, for a new president to be elected to follow me."
Domenico Scala, head of FIFA's independent audit and compliance committee, said there would need to be four months' notice for any new presidential election.
"The decision for the timing of the election of the next president will be up to the executive committee and will take place any time from December until March."
Blatter had been re-elected after his only rival, Jordan's Prince Ali bin Al Hussein, withdrew after gaining 73 votes to Blatter's 133 in the first round of voting.
FIFA called the news conference at their headquarters in a hilly Zurich suburb at only one hour and 20 minutes notice, an almost unprecedented move.
There was a tense atmosphere among FIFA employees and head of media Walter De Gregorio was close to tears as he sat through Blatter's speech.
Blatter, who began his career as a development officer in Ethiopia and then became secretary general from 1981 to 1988, said criticism from outside FIFA had weighed on his decision.
He was also critical of the structure of the executive committee, over which he said he has no control.
"The executive committee includes representatives of confederations over whom we have no control, but for whose actions FIFA is held responsible," he said. "We need deep-rooted structural change."
• Switzerland's office of attorney general (OAG) said it was not investigating Blatter.
"Joseph S. Blatter is not under investigation by the OAG. His announced resignation will have no influence on the ongoing criminal proceedings," the attorney general said in a brief statement.
The attorney general, which has opened criminal proceedings against persons unknown on suspicion of criminal mismanagement and of money laundering alongside a far wider US corruption investigation into FIFA, said it would release no further information.
Reactions to resignation
English Football Association chairman Greg Dyke: "We welcome today's announcement and believe it is good news for world football and FIFA. Change at the very top of FIFA is the necessary first step in delivering real reform of the organisation. We will now commit ourselves to play whatever role we can to support a positive transformation of FIFA for the benefit of all."
UEFA President Michel Platini: "It was a difficult decision, a brave decision, and the right decision."
Gary Lineker, sports commentator and former England player: "Blatter has resigned. Can't quite believe it. FIFA always appeared to be such a fine upstanding organisation."
Former FIFA presidential candidate Jerome Champagne: "Mr Blatter clearly underlines the fact that it is FIFA that counts, as the world government of football - and in a way he is sacrificing himself for it, for a structure that he developed and that he loves."
British Secretary of State for Culture, Media and Sport John Whittingdale: "I hope football can now come together. The chasm was created by Blatter wanting to hang on. Europe was supporting an alternative candidate and as long as he was there it was going to be difficult to move forward. I now hope everyone can come together to make the changes required."
European Commission spokeswoman for sport Nathalie Vandystadt: "This is an important step but a lot of work remains. We now expect a long process of change that is needed to restore trust and set up a solid system of good governance at FIFA."
