Then an act of kindness revived their spirits -- the leaders of the local Jewish congregation gave them the keys to their synagogue so they could continue to worship.
The leader of the mosque said he wasn't surprised by the gesture.
"I never doubted the support that we were going to get" after the fire, Dr. Shahid Hashmi, a surgeon and president of Victoria Islamic Center, told CNN. "We've always had a good relationship with the community here."
Hashmi said Dr. Gary Branfman -- a member of Temple B'nai Israel in Victoria, as well as a fellow surgeon and friend -- just came by his house and gave him the keys.
Support from around the world
And that wasn't the only offer of a temporary worship space that was extended. Hashmi said three local churches said his congregation could use their buildings. Also offered up was an empty office building, which the congregation used for three days before moving into a mobile home on the mosque property, Hashmi said.
"It's a beautiful community," he said.
Though Hashmi always knew his own east Texas community would support the mosque, he was stunned by the outpouring of support from people outside Victoria.
So far, a GoFundMe page set up to help raise money for the mosque's reconstruction has taken in more than $1 million.
That's on top of other donations the congregation has received from people in 90 different countries.
"We were surprised at the donations from around the world," Hashmi told CNN.
'Humanity at its best'
The Muslim community in Victoria dates back decades, Hashmi said, but the mosque was just 16 years old. Thanks to all of the financial contributions, he expects they'll be able to rebuild it in less than a year.
Hashmi said the congregants at the mosque are still in shock over the fire. Officials with the Texas State Fire Marshal's Office say no cause has been determined yet, and it may be weeks or months before can they figure out what happened, including whether the fire was deliberately set.
But the love that has grown from tragedy is making all the difference.
"It's given us a sense of hope," Omar Rachid, a member of the Muslim congregation, told CNN. "It's humanity at its best."