Angela Merkel's center-right Christian Democratic Union (CDU) was knocked out of a ruling coalition with the center-left Social Democrats, after winning just 17.6% of the vote.
After the poll result, Merkel acknowledged mistakes and shortcomings with the controversial refugee policy which allowed hundreds of thousands into the country.
"If I could I would play back time so I and the German federal government and leaders could have been better prepared," she said.
"We have not done everything right in the last few years. We are not world champions in integration," she added, highlighting that the asylum process was taking too long, that there were too few language courses and not enough labor market integration.
While the Social Democrats won the largest share of the vote, 21.6%, they will have to enter a coalition with two other left wing parties, Die Linke (15.6%) and the Greens (15.2%) to govern.
The Berlin result marks the second recent defeat for Merkel
since Germany adopted its open-door refugee policy.
AfD was set up by a group disgruntled by Merkel's eurozone crisis management -- most notably the Greek bailouts -- but it grew into a more nationalist party that strongly opposed rising immigration levels, particularly of people from Muslim countries.
Merkel has stood firm on Germany's position of accepting nearly all asylum seekers found to be legitimate refugees. Germany took in more than one million refugees in 2015, making it the most open country in Europe to asylum seekers.
In response to a poll indicating that 82% of Germans are unhappy with the refugee policy, Merkel said Monday: "If those 82% means that people do not accept foreigners, particularly people with Islamic background, this is against our constitutional rights, against our Christian democratic... and personal convictions. I and the CDU cannot represent this course."
CNN's correspondent in Berlin, Atika Shubert, reports that refugees were not the top issue there, with rising rents and increasing gentrification being ranked as more important.
But she said: "The nearly 15% win for the right wing AfD even in this very liberal city shows that, like many other parts of the country, it has surged to become a real political force that Merkel cannot ignore."
Although this election is for regional parliament and has no impact on the government, Shubert said Merkel will have to acknowledge that voters are unhappy with her party, especially the refugee policy.
Rise of the right
"She remains the strongest most stable politician not only in Germany, but arguably, Europe. And the European Union needs a strong Germany, not just to deal with the refugee crisis, but also Brexit and the debt crisis," Shubert said.
The rise of AfD in Germany reflects a surge in far-right populist parties across Europe.
"Because of its history, post-war Germany has lacked a viable right-wing political party. But the refugee issue has galvanized some voters to get behind the AfD as a palatable right-wing alternative," said Shubert.
"German voters are not convinced the government is fully prepared for the task of integrating so many thousands of people."
In an interview with CNN this month, AfD party leader Frauke Petry claimed the CDU was "falling apart."
"We see that in many regions of Germany the CDU bases, the party bases, don't agree with Merkel's policy any more," she said.
One of Merkel's current allies, Bavaria governor Horst Seehofer, told German newspaper Der Spiegel
that a rift in policy between them could worsen if the Chancellor fails to impose a ceiling on immigration.
"We don't want unlimited immigration like we saw last year and that's why we need binding measures as a guarantee," he is quoted as saying.
"Our constitution does not require us to take everybody who appears at our borders and demands asylum. And when someone comes from a safe country of origin, we can immediately repatriate them. The ceiling will work and it is consistent with the constitution."
In the run up to the election, current mayor Michael Muller warned
that a large showing by AfD "would be seen around the world as a sign of the return ... of the Nazis."
"Berlin has transformed itself from the capital of Hitler's Germany to a lighthouse of freedom, tolerance, diversity and social cohesion," Muller wrote on Facebook.