Issa Hayatou's seven-term reign as head of the Confederation of African Football (CAF) ended when the challenger won by 34-20 as delegates voted in the Ethiopian capital of Addis Ababa.
"When you try to do something, you mean that you can do it," Ahmad said after becoming CAF's seventh leader, and the first since Hayatou came to power in 1988.
"If I can't do it, I never stand," added the 57-year-old, who had campaigned on the basis of bringing about meaningful change.
Hayatou had long been a top-ranking FIFA official, and was acting president for the world game's ruling body before the appointment of Gianni Infantino
Ahmad will replace him on FIFA's Executive Committee -- which has been embroiled in the corruption scandals that have tarnished the game in recent years, and culminated in Sepp Blatter standing down as president.
The brother of Cameroon's former prime minister Sadou Hayatou, the 70-year-old was not implicated in the investigations launched in 2015 by US federal prosecutors.
However, he was sanctioned by the International Olympic Committee in 2011 over a cash payment he received from a collapsed sports marketing agency.
A new dawn for African football?
Ahmad assumes the role after an undistinguished playing career and 14 years at the helm of the Madagascar football federation.
His victory was greeted with cheers in the Plenary Hall and optimism around the continent.
"There is a generational gap between various African football federations, and Ahmad Ahmad can bridge that," Nigerian Football Federation president Amaju Pinnick told CNN ahead of the election.
"We are very keen on this change because football is not played in the banks. Football is played on the pitch.
"It's not Ahmad Ahmad that wins; it's African football that wins."
Hayatou's incumbency might have lasted almost three decades but he had only been challenged twice before, emerging victorious both times with landslide victories.
Ismail Bhamjee of Botswana took him on most recently -- losing by 46-6 votes in 2004 -- while 17 years ago Hayatou beat Angola's Armando Machado 47-4.
"His excellency Issa Hayatou has done a lot for African football but it is time for him to step back," Ghana Football Association vice president George Afriyie said.
Liberia FA president Musa Bility shared the sentiment, telling reporters: "Africa has made an emphatic decision that we are ready for change."
Ahmad will now serve a four-year term, but not everyone was so pleased with the verdict.
As one of just two men to publicly support Hayatou, Guinea-Bissau FA president Manuel Nascimento Irenio had reportedly said before the vote he would resign if Ahmad won.
Zanzibar secures CAF membership
In another historic vote, the archipelago of Zanzibar off the East coast of Africa was unanimously admitted as a full CAF member.
Zanzibar's national team will now be able to enter the Africa Cup of Nations for the first time.
While not yet a member of FIFA -- and therefore ineligible to enter the World Cup -- Zanzibar's governing body will have a vote in CAF debates.
Having admitted a first new member since South Sudan in 2012, the African confederation is now the joint-biggest in the world alongside Europe with 55 associations.