Mattis announced plans to resign after a face-to-face meeting with Trump in which they aired their differences, a senior White House official said.
"Because you have a right to a Secretary of Defense whose views are better aligned with yours on these and other subjects, I believe it is right for me to step down from my position," Mattis said in his resignation letter, released by the Pentagon.
Trump announced on Wednesday (local time) that US troops in Syria would be withdrawn, a decision that upended American policy in the region. On Thursday, officials said the president was considering a substantial US pullout from the 17-year-long conflict in Afghanistan.
Mattis had opposed the decision on Syria, an official said.
His letter indicated that he disagreed with Trump's isolationist policies, writing that it was his belief the United States needed to maintain strong alliances and show allies respect. Trump has withdrawn the United States from several international agreements since taking office in January 2017.
Trump, announcing Mattis' departure on Twitter, said he would nominate a successor soon.
"General Jim Mattis will be retiring, with distinction, at the end of February, after having served my Administration as Secretary of Defense for the past two years," he said.
The two men met in the Oval Office on Thursday afternoon. During the session, Mattis told Trump of his plans to resign, the senior White House official said.
"He and the president had differences on some issues. I don't know if it was specifically Syria," the official said.
Mattis joins a long list of former Trump administration senior figures who have either quit or been removed, some unceremoniously like Secretary of State Rex Tillerson who Trump fired via Twitter in March.
Trump's White House has had the highest turnover of senior-level staff of the past five presidents, according to the Brookings Institution think tank.
Speculation that Mattis might not last long in his post grew in October when Republican Trump said in a CBS interview that the general was "sort of a Democrat" and might be leaving.
Mattis, along with other national security aides, was said to have opposed Trump's decision to remove US troops from Syria. Many US lawmakers have expressed concern about the decision and asked Trump to reconsider.
Mattis has argued for maintaining a strong US military presence in Afghanistan to bolster diplomatic peace efforts.
The Pentagon declined comment on Afghanistan.
Garrett Marquis, a spokesman for the National Security Council, said that the White House was not going to comment "on future strategic developments."
US officials who spoke on condition of anonymity said thousands of the 14,000 troops in Afghanistan could be sent home as a result of the deliberations, the disclosure of which could undermine peace efforts with the Taliban.
