(CNN) -- While officials say the White House didn't ask for Shirley Sherrod's resignation, the former black Agriculture Department employee insisted Tuesday that the push for her to step down came from the Obama administration.
But Secretary of Agriculture Tom Vilsack said he asked her to resign without interacting with the White House.
"When I saw the statements and the context of the statements, I determined that it would make it difficult for her to do her job as a rural development director, and compromise our capacity to close the chapter on civil rights cases, I didn't want anything to jeopardize her job in terms of getting the job done and getting people to work in Georgia," Vilsack said.
"I made this decision, it's my decision. Nobody from the White House contacted me about this at all."
Sherrod stepped down Monday as the department's director of rural development for Georgia after conservative media outlets aired a video in which she said 24 years ago she shirked from helping a white farmer avoid foreclosure because of his race.
An official from the White House confirmed Vilsack's statement that it didn't influence USDA or Sherrod in the controversy.
"It was the secretary's decision, as he has said," the official told CNN.
Sherrod said on CNN's "Rick's List" that she was told differently.
"That's not what they told me," she said in a phone interview. "That's not what [USDA Deputy Undersecretary for World Development] Cheryl Cook conveyed to me. Each time she said it, she said 'the White House.'"
Roger Spooner told CNN he's the farmer she speaks of in the video and that "there wasn't any white or black or whatnot involved whatsoever" and "if it hadn't been for her, we would've never known who to see or what to do." He and his wife, Eloise, said they were speaking out to help Sherrod as she helped them.
CNN's Suzanne Malveaux contributed to this report
