(CNN) -- Australian Prime Minister Tony Abbott and Indonesian President Susilo Bambang Yudhoyono sought to present a united front on the controversial issue of asylum seekers Monday, as the two leaders held talks in Jakarta.
The talks -- Abbott's first on foreign soil after assuming the premiership last month -- took place in the wake of the deaths of at least 36 people on Saturday off the coast of West Java, Indonesia.
"We are determined to end this scourge, which is not just an affront to our two countries, but which has so often become a humanitarian disaster in the seas between our two countries," said Abbott during a joint press conference with his host.
"They give us a burden socially and economically," agreed Yudhoyono. "When they want to make their way to Australia, they also burden Indonesia. So in this spirit, to enable Indonesia and Australia to solve their problem, we have to work together."
On Monday, 28 survivors were being cared for by immigration officials in the West Java city of Cianjur, according to local police spokesman Martinus Sitompol, who added that rescue and recovery efforts would continue for the next three days.
Most of the boat's occupants were from Lebanon and Iraq, officials said, while others came from Nigeria, Jordan and Iran. It's believed they were heading for the Australian territory of Christmas Island.
Last week, Abbott moved to defuse tensions with Indonesia over how Australia handles asylum seekers entering its waters, dismissing the situation as "a passing irritant."
"The last thing I would ever want to do is anything that doesn't show the fullest possible respect for Indonesia's sovereignty," Abbott told Fairfax Radio. "This is a broad and deep relationship which is going to get broader and deeper over time.
"The last thing anyone should want is to have Australia's relationship with Indonesia defined by this boats issue, which I am sure will be but a passing irritant."
Abbott has ordered a military-led border protection plan in response to an increased number of illegal arrivals by boat -- many of whom use Indonesia as a staging post in onward voyages into Australian waters -- and has told the Royal Australian Navy to turn back vessels when it is safe to do so.
Australia's new Liberal-Coalition government argues that the tow-back policy would show criminal syndicates in Indonesia that it was serious about smashing people-smuggling operations.
Indonesia, however, has been cool towards the tow-back scheme which it sees as a violation of its sovereignty.
At a meeting in New York with Australian Foreign Minister Julie Bishop last week, Indonesia's Foreign Minister Marty Natalegawa warned that Indonesia would not accept violations of its borders, cautioning the Abbott government against taking any "unilateral steps" that would put the "cooperation and trust" between the two countries.
Ignoring a plea from Bishop to deal with the problem "behind the scenes," Indonesia took the step of publishing details of the private conversation, a move seen by commentators as a blunt warning from Indonesia.
''Asian diplomacy in particular is marked by understatement ... this is what makes it such an extraordinary development,'' Acting opposition leader Chris Bowen told ABC TV. ''It's an unprecedented step by an Indonesian foreign minister, I can't recall an Indonesian foreign minister taking a step like this in relation to Australia ever before.''
The Indonesian Foreign Ministry and the Ambassador to Australia both issued statements last week saying the meeting transcript was not intended to be issued.
Former Australian Foreign Minister Alexander Downer weighed into the debate on Friday, saying Indonesia needed to accept responsibility for the problem.
''They do have to understand that it's their boats with their crews, their flagged boats, which are breaking our sovereignty and are breaking our law by transgressing our national borders, our maritime borders,'' Downer told Fairfax Radio. ''There's no point in allowing ourselves to be bullied by the Indonesians in this way. I mean, we have to stand up for ourselves and stand up for our national interest and be prepared to call it as it is."
Indonesian researcher with Human Rights Watch Andreas Harsono told CNN that refugees were caught between harsher policies emanating from Australia and harsh conditions in Indonesia, which is not a signatory to international conventions on refugees.
"The problem in Indonesia is that there is a lack of a protection mechanism within the Indonesia legal system to protect these asylum seekers," Harsono said, adding that refugees -- including unaccompanied minors -- were often placed in detention for a year and had no right to work or go to school on their release.
He described Australia's tow-back policy, meanwhile, as "dangerous."
"Most of these boats are not seaworthy, many of them do not have GPS or other navigational equipment -- to tow them back into international waters is dangerous," he said. "What Tony Abbott can do is to urge Indonesia to ratify the refugee conventions."
According to the latest data from the Department of Immigration and Citizenship there were 17,698 asylum seekers as of March, 2013. The figures show a marked increase in the number of people seeking asylum in Australia, with the first three-quarters of 2012-13 already ahead of the total for the entire 2011-2012 period.
The figures also show that for the first time "irregular maritime arrivals" (IMA), or boat people, for the first time outnumber arrivals by air (non-IMA).
Despite the recent increases in the number of asylum seekers, Australia takes just 3% of the world's refugees, according to figures from the United Nations High Commission of Refugees.
The majority of asylum seekers arriving in Australia by boat were from Afghanistan in 2011-12, with Iran, Sri Lanka and Pakistan the next largest groups.