Protesters stay in Wisconsin Capitol as defiant governor digs in

Check out CNN iReport to see photos from the protests and read personal stories from Wisconsin residents. If you're participating in the rallies, be sure to share your story!

(CNN) -- Union workers and activists who refused to leave the Wisconsin Capitol building Sunday, against orders, will be allowed to spend the night, police said.

Authorities had set a deadline earlier in the day of 4 p.m. CST for demonstrators to leave the building, saying it needed to be cleaned after roughly two weeks of protests.

While many people left, hundreds of others defied the order and remained inside. Some say they are willing to risk arrest in a conflict that has become a flash point in the nation's debate around labor unions

"We have the right to be here. This is the people's house. This is a house of labor. This is a house that Wisconsin built," Mahlon Mitchell, president of the Professional Fire Fighters of Wisconsin, said from inside the Capitol.

Protesters are upset with Wisconsin Governor Scott Walker's call to curb collective bargaining rights.

The embattled first-term Republican has shown no sign of backing down and defended his proposal Sunday on NBC's "Meet the Press."

Wisconsin is "broke," and unions use their power to block necessary cost-saving measures, Walker argued.

"It's about time somebody stood up and told the truth in this state, and said, 'Here's our problem, here's the solution,' and acted on it," he said. "Because if we don't, we fail to make a commitment to the future."

Minutes later on the program, AFL-CIO President Richard Trumka assailed Walker.

"This isn't about the budget crisis," Trumka said, adding that Walker's arguments have "migrated" as his justifications have been refuted.

"Governors that are willing to sit down and work with their employees can work out problems," Trumka said. "We can solve them. But that's not what Governor Walker is doing. He says, 'I won't talk to you.' "

Walker will talk to big contributors, but he "won't talk to employees," Trumka added.

Earlier on CNN, another union head also accused the governor of using financial arguments as a deceit.

"Walker is saying it's a budget crisis when it isn't a budget crisis," said Randi Weingarten, president of the American Federation of Teachers. "The workers have already said publicly, because he refuses to meet with them, that they would take the cuts to take-home pay that he has asked for here. So this is a ruse to shift power to his friends, because at the same time what he said was that he wanted to give tax breaks to the friends who put him into power."

The Wisconsin Assembly has passed a Republican bill that would strip most state workers of the bulk of their collective-bargaining rights. Among other things, the measure would require workers -- with the exception of police and firefighters -- to cover more of their health care premiums and pension contributions.

Collective bargaining would be limited to wages, though any pay increases beyond the inflation rate would be subject to voter approval.

The bill must now clear the Wisconsin Senate. However, 14 Democratic senators have fled to neighboring Illinois to prevent a quorum from voting on the issue.

Walker called on the lawmakers to "come back and do their job."

"If we fail to pass this bill by Tuesday, we lose $165 million worth of savings. If we continue down that path we start seeing layoffs," he warned.

Walker repeatedly cited his experience as a local official in explaining his call to curtail bargaining rights for government. As a county executive, he tried "modest" changes in pensions and health care, and tried to implement a 35-hour work week "to avoid massive layoffs and furloughs," Walker said. "And the unions said, 'Forget it.' Emboldened by the fact that they had collective bargaining agreements, they said, 'Go ahead, literally lay off 400 or 500 people.' "

Asked why the exemption for firefighters and police, Walker responded the decision "boils down to public safety." If any firefighters or police were to walk off the job in protest, "I can't afford to have a fire or crime committed where there's a gap in service."

Walker also defended himself against attacks stemming from a prank call in which he thought he was speaking privately to a wealthy conservative activist. The caller was in fact liberal blogger Ian Murphy, who posted the call online.

At one point in the call, Walker said, "This is our time to change the course of history."

Critics say his comment is a sign that he is an ideologue looking to reshape the system and end collective bargaining power rather than a pragmatic governor looking to balance a budget.

"I do believe that. This is our moment in Wisconsin's history," Walker told NBC. He added that "legislatures before have kicked the can" by not providing long-term fixes. "I make no apology for the fact this is an important moment in time," he said.

In the prank call, Murphy suggested "planting some troublemakers" among people protesting the measure in Madison. "We thought about that," Walker replied in the call.

Asked Sunday whether he had really thought about such an idea, he responded, "We rejected that." Government officials looked at "all sorts" of suggestions, and rejected that idea "because we've had a civil discourse," he said.

Weingarten, speaking to CNN, said workers in Wisconsin have sent a message to Walker that "'we understand and we will do our fair share.'" But Walker is sending a different message, Weingarten said: "'I'm just taking away your voice at work.'"

In the long term, that message "is ridiculous because collective bargaining is the way to increase quality," she said.

When President Barack Obama was campaigning in 2007, he made a promise: "If American workers are being denied their right to organize and collectively bargain when I'm in the White House, I'll put on a comfortable pair of shoes myself. I'll walk with you on the picket line as president of the United States of America because workers deserve to know that somebody is standing in their corner," he said during a speech in Spartanburg, South Carolina, on November 3, 2007.

While the president has publicly expressed support for those fighting to keep collective bargaining in Wisconsin, he has not joined them at protests.

But Trumka of the AFL-CIO insisted that Obama, who faces federal budget challenges, is "doing it the right way."

"He's not taking on workers like Scott Walker is and trying to take away their ability to come together and negotiate a middle-class way of life," Trumka said.