Anthony Albanese is among 22 world leaders are urgently calling for global guardrails to ensure AI remains "under human control".
The leaders say efforts to ring-fence AI have been failing and rapid development of AI models now pose safety and security risks.
The statement comes on the eve of the United Nations General Assembly where fears of AI is expected to be a hot topic.
Anthony Albanese and more than a dozen world leaders have issued an urgent plea for global guardrails to ensure AI remains "under human control", in a demonstration of middle powers flexing their collective muscle, as the US and China fight for global dominance.
In a statement — issued on the eve of the United Nations General Assembly (UNGA) in New York — the 22 signatories warn "the pace of development could outpace our ability to manage emerging risks" and efforts to ring-fence it are failing.
First flagged by Mr Albanese on Saturday, the declaration warns capable AI systems have already managed to circumvent testing safeguards and gain unauthorised access to real-world systems.
"The rapid development of frontier AI models poses serious risks to safety and security if not appropriately managed," the statement reads.
"To realise AI's potential, industry, governments and society must act now," the statement reads.
The coalition of middle powers — including Australia, Finland, Norway, Canada, Singapore and the European Commission — calls on AI companies to develop transparent safety protocols and governments to develop and coordinate common standards.
They also suggest the UN consider creating an international institution "convene states when capability thresholds are crossed".
Fears of AI, wars in Ukraine and the Middle East and the threat of climate change are set to take centre stage as world leaders gather in New York.
In a speech at the UN headquarters on Monday local time, Mr Albanese will say middle powers need to step up and play a stronger role in maintaining international rules "especially right now, when we are facing one of the toughest periods in modern history".
"The war in the Middle East, an ongoing land war in Europe, the global inflation crisis, the fuel crisis and the rise of AI are incredibly serious challenges," he will tell the Partners For Multilateralism, International Law, Peace And Prosperity forum.
The speech will come a day after Mr Albanese formally launched Australia's campaign for a two-year term on the UN Security Council from 2029, citing the risk of AI as a reason for the push.
Debate over the pace of AI development is reaching an inflection point, with Anthropic chief Dario Amodei, backed by OpenAI's Sam Altman and Space X's Elon Musk, calling for a co-ordinated slowdown.
However, US President Donald Trump has made it clear he views fears the AI poses an existential threat as a hoax.
Ahead of his summit with Chinese President Xi Jinping at the White House on Thursday, Mr Trump repeated, declaring he would establish a new "AI Force" to protect against the "decimation or destruction" of AI.
"We are leading China, and the rest of the World, and I intend to keep it that way! " the president wrote on social media.
Mr Albanese wants the two leaders to cooperate and strike a deal similar to the nuclear non-proliferation treaty, signed in 1970 to stop the spread of nuclear weapons.
"The truth is they're the big two giants here and so we want to see cooperation," he told Insiders on Sunday.
"President Trump, of course, will act in the interests of the United States. But there's a need for all of us to act in the interests of humanity."