Drew Davies, Fox Fisher and Ugla Stefanía Kristjönudóttir Jónsdóttir (aka Owl) said in a statement that they had invited The Blair Partnership to speak out, but that "they were unable to commit to any action that we thought was appropriate and meaningful."
The three authors, who are understood to have resigned along with a fourth who wanted to remain anonymous, said they did not take the decision to leave lightly and were "saddened and disappointed it has come to this."
In the statement, published online, they wrote: "Freedom of speech can only be upheld if the structural inequalities that hinder equal opportunities for underrepresented groups are challenged and changed."
"Affirmations to support LGBTQIA [Lesbian, Gay, Bisexual, Transgender, Queer, Intersex and Asexual] people as a whole need to be followed up by meaningful and impactful action, both internally and publicly. As LGBTQIA writers ourselves we feel strongly about having an agency that supports our rights at all avenues, and does not endorse views that go against our values and principles."
Rowling has faced renewed criticism in recent weeks after she wrote an essay attempting to explain her views on the transgender community, which have previously triggered a backlash on social media and seen the "Harry Potter" author accused of transphobia.
A spokeswoman for The Blair Partnership, which was established in 2011 with Rowling as its main client, said in a statement that the agency supported "freedom of speech" for all its clients and would not comment on individual views.
"We support the rights of all of our clients to express their thoughts and beliefs, and we believe in freedom of speech," said the statement, emailed to CNN.
"We are disappointed by the decision that four clients have taken to part ways with the agency. To reiterate, we believe in freedom of speech for all; these clients have decided to leave because we did not meet their demands to be re-educated to their point of view."
The agency said it championed "equality and inclusivity," and was committed to making the agency a "welcoming environment."
"The diversity of our clients' voices is our strength and we take enormous pride from each and every one," it added.