The leader of the conservative Popular Party (PP) failed to obtain a simple majority, losing the bid with 180 votes against, 170 votes in favor and zero abstentions.
He was six votes short of the 176 votes needed.
Spain is yet to piece together its first administration since the emergence of the left-wing Podemos and centrist Ciudadanos, as well as other regional parties, ended its traditional two-party system at last December's election.
Neither PP nor the Spanish Socialist Party (PSOE), Spain's two major political forces, has been able to gather enough support from a parliament now representing 12 parties.
Before Friday's vote, Rajoy attempted to sway some opposing MPs to abstain, saying his coalition was the "best option for Spain" and accusing the leader of the PSOE, Pedro Sanchez, of "blocking Spain."
"Your honors, if he doesn't want any of this and wants to sit in the margin of everything and persists in his policy of saying 'no and no,' at least allow Spain to form a government," Rajoy told MPs.
But while Rajoy was thinking of Friday's vote, his opponents were already considering future negotiations to form a government before new elections are called.
Sanchez told the chamber they were responsible for finding an alternative to the "political impasse candidate Rajoy has created."
"I am convinced we will find a solution and, have no doubt, that the Socialist Party will be part of that solution," Sanchez said.
However, the clock is ticking for the Socialist leader to form a coalition capable of convincing the chamber. New national elections will be automatically called if no agreement is reached by October 31.
Sanchez might have to side with anti-austerity party Podemos, whose offer to form a left-wing coalition the Socialist leader rejected earlier in the year.
Podemos leader Pablo Iglesias used Friday's session to urge Sanchez to reconsider.
"Either you reach an agreement with us or you continue to repeat the shame and humiliation that made hundreds of thousands of young people take to the streets and put us, here, in this parliament," Iglesias told MPs.
Sanchez might have to change his mind after all. Following Friday's vote, according to Spanish media outlets, the Socialist leader was quick to suggest that he will try to form an alternative government bid with both Podemos and Ciudadanos.
If that does not work and new elections are called, the next ballot could fall on Christmas Day.
According to Article 51 of the Spanish electoral law, a campaign could start only 38 days after elections are called, would have to last 15 days and finish at 00:00 of the day immediately before voting day.
The whole process would last a total of 54 days, with the vote taking place on December 25th, a major inconvenience in a traditionally Christian country where, according to the Spanish Centre of Sociological Investigations, nearly 70% of people identify as Catholic.