The activists reported on social media Saturday that ISIS had taken control of neighborhoods in northern and northwestern parts of the city known for some of the world's most treasured ruins.
Intense fighting on the city's outskirts Saturday afternoon forced the withdrawal of hundreds of Syrian troops and Shiite militias, according to the Palmyra Coordination Committee.
CNN has not been able to confirm the developments that appeared on the activists' social media postings.
The development could be a huge setback for Syrian forces and militias loyal to the government who recaptured Palmyra from ISIS in late March, state-run media reported.
The terror group is targeting the Palmyra citadel and its surroundings with mortars and rockets, the activists said via social media. ISIS also appeared to be taking control of a military warehouse and the hills surrounding it.
The Syrian Observatory for Human Rights, a UK-based monitoring group, reported an explosion from what was believed to be an ISIS car bomb east of Palmyra. The assault reportedly cleared the way for the terror group to move near a hospital in the northwest outskirts of the city.
ISIS also appeared to take control of the Amiriyah suburb as well as al-Tar mountain near Palmyra's citadel, according to the monitoring group. The ancient site is taking heavy machine-gun fire.
The Syrian army and militias loyal to the regime chased out ISIS fighters on March 27, according to state-run media. ISIS had seized the city in May 2015.
Palmyra, in the Homs countryside northeast of Damascus, is recognized by the United Nations as a World Heritage Site
ISIS demolished some of the ruins, including the 1,800-year-old Arch of Triumph and the nearly 2,000-year-old Temple of Baalshamin. The group beheaded the antiquities expert who looked after the ruins.
Syria said ISIS also destroyed two Muslim holy sites: a 500-year-old shrine and a tomb where a descendant of the Prophet Mohammed's cousin was reportedly buried.
Palmyra was a caravan oasis when the Romans overtook it in the middle of the first century. In the centuries that followed, the area "stood at the crossroads of several civilizations," with its art and architecture mixing Greek, Roman and Persian influences, according to UNESCO, the UN agency that documents the world's most important cultural and natural sites.