Jay-Z was spotted wearing the T-shirt showing Lamu's Riyadha Mosque on March 30, as he came out of a restaurant in Santa Monica, California.
Images were uploaded to Instagram by both the T-shirt designer Zeddie Loky, who is Kenyan, and the clothing brand, blkburd genes.
On April 3, the Riyadha Mosque posted a letter to Loky on its Facebook page, saying the management committee and worshipers were "disturbed and actually feel insulted" by the photos of Jay-Z wearing the T-shirt, which features a green image of the mosque on a white background, with the word "Lamu" printed above it.
"Rest assured, we neither consider it an honor nor a privilege for the Historical Mosque and its Founder Habib Swaleh," the letter reads.
"When wearers of these T-shirts end up in bars, clubs and at all sorts of sacrilegious joints, it is certainly an affront to the spiritual respect and dignity towards all those who revere the Mosque, its Founder and the General Muslim community within and outside Lamu."
The letter called on Loky to "show respect for the Mosque by 'removing our Mosque's Portrait.'"
Loky then apologized to the mosque in a letter seen by CNN.
The designer, who is also CEO of blkburd genes, specified that only 20 of the T-shirts have been produced and those who own one have been asked to respect the mosque "by not wearing it in bars nor clubs if they happen to decide to wear it."
Riyadha Mosque confirmed to CNN that it had received Loky's apology.
Clothing brand blkburd genes is "a scientific design company on a mission to educate people around the world about history, nature and current affairs," according its website. It is based in Los Angeles.
CNN has reached out to Jay-Z for comment.
Lamu is situated on the coast in eastern Kenya. Its old town is a designated UNESCO world heritage site.
The United Nations' cultural agency describes it as "the oldest and best-preserved Swahili settlement in East Africa," noting it has "hosted major Muslim religious festivals since the 19th century, and has become a significant centre for the study of Islamic and Swahili cultures."