Drew Davies, Fox Fisher and Ugla Stefanía Kristjönudóttir Jónsdóttir (aka Owl) said in a statement that they had invited The Blair Partnership to speak out, but that "they were unable to commit to any action that we thought was appropriate and meaningful."
The three authors, who are understood to have resigned along with a fourth who wanted to remain anonymous, said they did not take the decision to leave lightly and were "saddened and disappointed it has come to this."
In the statement, published online, they wrote: "Freedom of speech can only be upheld if the structural inequalities that hinder equal opportunities for underrepresented groups are challenged and changed."
"Affirmations to support LGBTQIA [Lesbian, Gay, Bisexual, Transgender, Queer, Intersex and Asexual] people as a whole need to be followed up by meaningful and impactful action, both internally and publicly. As LGBTQIA writers ourselves we feel strongly about having an agency that supports our rights at all avenues, and does not endorse views that go against our values and principles."
Rowling has faced renewed criticism in recent weeks after she wrote an essay attempting to explain her views on the transgender community, which have previously triggered a backlash on social media and seen the "Harry Potter" author accused of transphobia.
A spokeswoman for The Blair Partnership, which was established in 2011 with Rowling as its main client, said in a statement that the agency supported "freedom of speech" for all its clients and would not comment on individual views.
"We support the rights of all of our clients to express their thoughts and beliefs, and we believe in freedom of speech," said the statement, emailed to CNN.
"We are disappointed by the decision that four clients have taken to part ways with the agency. To reiterate, we believe in freedom of speech for all; these clients have decided to leave because we did not meet their demands to be re-educated to their point of view."
The agency said it championed "equality and inclusivity," and was committed to making the agency a "welcoming environment."
"The diversity of our clients' voices is our strength and we take enormous pride from each and every one," it added.
Rowling's essay -- in which she revealed that she had been a victim of domestic abuse, and had "concerns around single-sex spaces" -- was labeled "devastating" by trans activists. Prominent figures, including "Harry Potter" stars Daniel Radcliffe and Emma Watson, publicly asserted their support for transgender rights.
Authors Davies, Fisher and Jónsdóttir said they stood in solidarity with everyone in publishing who was "working incredibly hard to champion diverse voices and experiences to challenge the homogeneity of the industry."
They said agencies and publishers "need to create platforms for underrepresented groups from the ground up and make meaningful change within their culture. Representation must extend into real and authentic representation of diverse voices."
And they said they wanted to extend "solidarity to the trans community at this time, many who might feel vulnerable, alienated and unsupported right now," adding: "Trans women are women, trans men are men and non-binary identities are valid."
The authors said they had donated to Shakti Women's Aid -- a charity that supports Black, Asian and Minority Ethnic (BAME) women dealing with domestic abuse -- and invited others to do the same or donate to an organization working directly with transgender people.