(CNN) -- North Korea has started reassembling its main nuclear complex in retaliation for U.S. refusal to remove the Stalinist state from a list of states that sponsor terrorism, it was reported Wednesday.
The report from Japan's Kyodo news agency said work at the Yongbyon complex began Tuesday.
Asked to confirm the report, U.S. National Security Council spokesman Gordon Johndroe would only say that "the process is not moving in the direction we want it to move in."
"We think North Korea is taking these steps because it has not been removed from the terrorism list," he said Wednesday.
North Korea announced last week that it had stopped disabling its nuclear plants on August 14 and would consider rebuilding its reactor because the United States has not removed it from a list of states that sponsor terrorism.
U.S. State Department deputy spokesman Robert Wood said at the time that such a move would be "a step backward" and a violation of North Korea's commitments under a six-party deal involving the U.S., Russia, the two Koreas, China and Japan. Watch Christiane Amanpour explain latest developments »
North Korea agreed to a complete dismantling of its Yongbyon nuclear complex by October. In return, U.S. President George Bush said he would lift some U.S. sanctions against the communist state and remove it from a State Department list of state sponsors of terrorism.
The sticking point between the two countries involves verification. Washington has said it will not remove North Korea from the terrorism list until Pyongyang agrees to set up an internationally recognizable mechanism to verify its declaration.
"As soon as have it, we will remove North Korea from the terror list," Johndroe said, referring to a "credible verification protocol."
"North Korea knows what it needs to do at this point," he added. "The six-party talks operate on a principle of action-for-action."
The United States has demanded that inspectors be given the right to visit all suspected nuclear facilities without notice, said the South Korean news agency, Yonhap.
North Korea rejects that provision.
"The U.S. is gravely mistaken if it thinks it can make a house search in (North Korea) as it pleases just as it did in Iraq," the North Korean Foreign Ministry said in a statement carried by the state-run Korean Central News Agency (KCNA).
In June, North Korean officials turned over to China a 60-page declaration, written in English, that detailed several rounds of plutonium production at the Yongbyon plant, dating to 1986.
In it, North Korea acknowledges producing roughly 40 kilograms of enriched plutonium -- enough for about seven nuclear bombs, according to the U.S. State Department.
Soon after, North Korea publicly destroyed a water cooling tower at the Yongbyon facility.
North Korea agreed to abandon its atomic weapons program on a promise that it would receive energy aid equivalent to one million tons of heavy fuel oil from the five other nations involved in the disarmament talks: the United States, South Korea, China, Japan and Russia.
The parties to the talks also agreed on a system for verifying North Korea's compliance, including "visits to facilities, review of documents, interviews with technical personnel and other measures unanimously agreed upon among the six parties," according to a joint statement they released.
Last week, the North Korean Foreign Ministry said that the United States was in "outright violation" of the agreement.
"The U.S., however, raised all of a sudden an issue of applying an 'international standard' to the verification of the nuclear declaration, abusing this agreed point," the statement said. "It (pressured North Korea) to accept such inspection as scouring any place ... as it pleases to collect samples and measure them."
South Korea's Foreign Ministry said it was disappointed by its neighbor's decision.
"The measure is regrettable as it came at a time when the six parties have to make concerted efforts to complete phase two of the denuclearization process," ministry spokesman Moon Tae-young told reporters.
He said Seoul "will work closely with relevant countries to have North Korea resume the disablement work as early as possible."