The result, which saw Tokyo Governor Yuriko Koike's new Tomin First Party team up with a handful of independents to capture a total of 79 seats, reflected a more general dissatisfaction with Abe's national leadership, said analysts.
Key issues in the election, which is often seen as a bellwether for national elections, included the relocation of the city's famous Tsukiji fish market
, a huge tourist draw, and preparations for the Tokyo 2020 Olympic Games.
The assembly consists of 127 seats, an absolute majority of which are now held by the upstart Tomin First Party and its six allies, according to
the Japanese state broadcaster.
Koike, the Japanese capital's first female governor
, said she was grateful for the results, which exceeded expectations, but is also "feeling the weight of responsibility," according to NHK.
"The result is much better than we had expected," said the lawmaker, who was a former Environment Minister and Defense Minister in the Liberal Democratic Party (LDP) before breaking away to form Tomin First -- which translates to "Tokyoites First" -- last month.
"Our candidates are all fresh, and this new lineup will make the assembly something completely different," Koike said in an interview on Fuji TV.
The landslide victory for Koike meant that Prime Minister Shinzo Abe's LDP, which had been the best represented party in the capital's local government, saw its 57 seats drop to 23, NHK said.
Abe said voters had handed down a "very severe judgment" on his party.
"We must take it seriously as it is severe reprimand against the LDP and reflect deeply on it," he said.
"From now, the party must become united, establish a system and achieve results to regain the public trust," he added.
Some LDP lawmakers are now calling for a cabinet reshuffle.
In the wake of the losses, long-time Abe ally Hakubun Shimomura announced he would step aside as chairman of the party's Tokyo chapter.
"This is a very harsh result. What happened in national politics has evolved into a huge backlash against us. I feel very responsible," he was quoted by Kyodo News as saying.