Pyongyang said the test was an act of self-defence against "US hostility" and threatened stronger steps if necessary.
The UN Security Council held an emergency meeting at which its members, including China, "strongly condemned" the test and vowed to start work on appropriate measures in response, the president of the council said.
North Korean leader Kim Jong-un, the third of his line to rule the country, has presided over two long-range rocket launches and a nuclear test during his first year in power, pursuing policies that have propelled his impoverished and malnourished country closer to becoming a nuclear weapons power.
North Korea said the test had "greater explosive force" than those it conducted in 2006 and 2009. Its KCNA news agency said it had used a "miniaturised" and lighter nuclear device, indicating it had again used plutonium, which is suitable for use as a missile warhead.
China, which has shown signs of increasing exasperation with the recent bellicose tone of its reclusive neighbour, summoned the North Korean ambassador in Beijing and protested sternly, the Foreign Ministry said.
Foreign Minister Yang Jiechi said China was "strongly dissatisfied and resolutely opposed" to the test and urged North Korea to "stop any rhetoric or acts that could worsen situations and return to the right course of dialogue and consultation as soon as possible".
Analysts said the test was a major embarrassment to China, which is a permanent member of the Security Council and North Korea's sole major economic and diplomatic ally, because it cast doubt on the extent of Beijing's influence over its ally.
US President Barack Obama called the test a "highly provocative act" that hurt regional stability and pressed for new sanctions.
"The danger posed by North Korea's threatening activities warrants further swift and credible action by the international community. The United States will also continue to take steps necessary to defend ourselves and our allies," Obama said.
US Ambassador to the United Nations Susan Rice said Washington and its allies intended to "augment the sanctions regime" already in place due to Pyongyang's previous atomic tests. North Korea is already one of the most heavily sanctioned states in the world and has few external economic links that can be targeted.
Japanese Prime Minister Shinzo Abe said the test was a "grave threat" that could not be tolerated. UN Secretary-General Ban Ki-moon said the test was a "clear and grave violation" of U.N. Security Council resolutions.
Russian Foreign Minister Sergei Lavrov urged North Korea to abandon its nuclear arms programme and return to talks. NATO condemned the test as an "irresponsible act" that posed a grave threat to world peace.
South Korea, still technically at war with North Korea after a 1950-53 civil war ended in a mere truce, also denounced the test.
North Korea's Foreign Ministry said the test was "only the first response we took with maximum restraint".
"If the United States continues to come out with hostility and complicates the situation, we will be forced to take stronger, second and third responses in consecutive steps," it said in a statement carried by the official KCNA news agency.
North Korea often threatens the United States and its "puppet", South Korea, with destruction in colourful terms.
North Korea told the U.N. disarmament forum in Geneva that it would never bow to resolutions on its nuclear programme and that prospects were "gloomy" for the denuclearisation of the divided Korean peninsula because of a "hostile" U.S. policy.
Suzanne DiMaggio, an analyst at the Asia Society in New York, said North Korea had embarrassed China with the test. "China's inability to dissuade North Korea from carrying through with this third nuclear test reveals Beijing's limited influence over Pyongyang's actions in unusually stark terms," she said.
Mark Fitzpatrick of the International Institute for Strategic Studies think tank, said: "The test is hugely insulting to China, which now can be expected to follow through with threats to impose sanctions."
The magnitude of the explosion was roughly twice that of the 2009 test, according to Lassina Zerbo, director of the international data centre division of the Vienna-based Comprehensive Nuclear-Test Ban Treaty Organization. The U.S. Geological Survey said that a seismic event measuring 5.1 magnitude had occurred.
North Korea trumpeted the announcement on its state television channel to patriotic music against a backdrop of its national flag.
"It was confirmed that the nuclear test that was carried out at a high level in a safe and perfect manner using a miniaturised and lighter nuclear device with greater explosive force than previously did not pose any negative impact on the surrounding ecological environment," KCNA said.
North Korea linked the test to its technical prowess in launching a long-range rocket in December, a move that triggered the U.N. sanctions, backed by China, that Pyongyang said prompted it to take Tuesday's action.
The North's ultimate aim, Washington believes, is to design an intercontinental ballistic missile capable of carrying a nuclear warhead that could hit the United States. North Korea says the programme is aimed merely at putting satellites in space.
Despite its three nuclear tests and long-range rocket tests, North Korea is not believed to be close to manufacturing a nuclear missile capable of hitting the United States.
It used plutonium in previous nuclear tests and before Tuesday there had been speculation that it would use highly enriched uranium so as to conserve its plutonium stocks, as testing eats into its limited supply of materials to construct a nuclear bomb.