State television showed crowds pouring onto the streets around the campus where the Supreme Leader of Iran, Ayatollah Ali Khamenei, led prayers for Rafsanjani despite the differences between the two men.
President Hassan Rouhani, parliament speaker Ali Larijani and his brother, judiciary chief Ayatollah Sadegh Larijani, joined the prayers.
Rafsanjani died Sunday at the age of 82 after suffering a heart attack.
Three days of official mourning began on Monday.
Various political roles
Rafsanjani, who served two terms as Iran's president from 1989 to 1997, had been an influential figure in Iranian politics.
At the time of his death, he was the chairman of the Expediency Council, which aims to settle disputes between the country's parliament and the Guardian Council.
"I admired him as a shrewd politician who had a calming effect on the nation and the country," Saeed, a 47-year-old local journalist who was among the mourners, told CNN.
"It is because of Rafsanjani that Iran has been safe in our volatile region. Look at Syria, Iraq, Afghanistan. Even outside the region, Libya is in chaos. Iran avoided those problems mostly because of Rafsanjani's influence in several governments that he served."
Throughout his political career, the former President was seen as a conduit for better relations with the West.
He was also seen as a key figure in Iran's moderate movement, losing a presidential bid against the former hardline Iranian President Mahmoud Ahmadinejad in 2005.
During Ahmadinejad's 2009 re-election campaign, Rafsanjani became a vocal critic, chiding the election results that gave Ahmadinejad another overwhelming victory.
His decision put him at odds with Khamenei, his former comrade-in-arms, and set in motion a series of events that sidelined the old Islamic Revolutionary Guard Corps -- involved in cracking down on protests -- from some of the very institutions Rafsanjani had led before.
His support for centrist and moderate forces was believed to be key in the 2013 presidential elections that brought to power Rouhani, a close Rafsanjani ally. Not surprisingly, Rouhani was one of the first to react to the death of his mentor, tweeting: "The spirit of the giant of the #Revolution and politics, the symbol of #patience and #resilience has soared to the heavens. #HashemiRafsanjani."
A nation remembers
"I respected and loved Rafsanjani for what he did on behalf of the Islamic Revolution (in 1979). His family may have caused huge problems for him later in his political life, but I do not blame him for the actions of his children," 63-year-old Rasoul, who was at the funeral, told CNN, referring to the the corruption cases Rafsanjani's children were embroiled in over the past few years.
"Rafsanjani was one of the original founders of the Islamic Revolution and he remained true to the fundamental principles of the revolution and Islam."
The former President will be buried in the crypt of Khomeini, the leader of Iran's Islamic revolution.
Free transportation was provided to the funeral venue.
Messages from around the world
Since Rafsanjani's death, condolences have poured in from around the world, including from the White House -- an unprecedented move by Washington since the 1979 revolution, when ties were cut between the two countries.
"Former President Rafsanjani was a prominent figure throughout the history of the Islamic Republic of Iran and the United States sends our condolences to his family and loved ones," spokesman Josh Earnest said.