US President Donald Trump labelled concerns about AI "a hoax" during a speech to the UNGA.
Professor Yoshua Bengio, one of the 'Godfathers' of AI has pushed back against those comments.
Professor Bengio will address the UN Security Council.
Professor Yoshua Bengio, the co-chair of the UN's Scientific Panel on AI, says big tech companies are taking large risks with their development of the technology and there needs to be an urgent change of direction.
"Its shouldn't be the decisions of just a few CEOs or a few governments to figure out what's good for humanity," Professor Bengio told 7.30 from New York, where he is set to address the UN Security Council overnight.
The winner of the Turing Award for his work on AI said international agreement on safeguards is essential to secure humanity's future.
His comments came after US President Donald Trump labelled concerns about AI "a hoax" in a speech to the UNGA and rejected a joint statement from 22 countries, including Australia, calling for humans to remain in control of AI.
The US "totally rejects any attempt to construct a globalist scheme to control the artificial intelligence being spoken of so much now," Mr Trump said.
However, Professor Bengio, one of the 'Godfathers' of AI, said the plea from the 22 countries was important.
"What happened with this group of countries is extraordinary and it is important," he said.
"And we have to continue in that direction. We need more countries. We need the world to wake up so that we can have influence on our future.
"The reason I am able to sleep at night and the reason why a country like Australia can make a difference is that we have agency right now.
"When we form coalitions, when we put our resources together, our minds, our energy, our talent, we can be in the game and we can bring the world into a place that's better aligned with our values, with democracy, and the idea that we should keep our sovereignty in a multilateral world."
The comments come on the eve of a meeting in New York between Mr Trump and Chinese President Xi Jinping where AI is expected to feature prominently.
Professor Bengio said American AI companies are about six months ahead of their Chinese rivals, but that competition should not get in the way of cooperation between Washington and Beijing.
"It is reason enough to sit at a table with the Chinese because, you know, they're humans too. They want to have a life. They want their children to have a life," he said.
Professor Bengio says the greatest dangers posed by AI come from "recursive self-improvement" being developed by the world's leading tech companies — where artificial intelligence trains the next generation of AI.
"Imagine we have an AI that is smarter than us, so it is able to do that research and engineering all by itself," he said.
"It could design an AI that will be even more powerful than itself and even less controllable and even more malicious.
Professor Bengio will tell world leaders at the UN Security Council that they must act.
"The forces of competition between companies and countries is not healthy," he said.
Watch 7.30, Mondays to Thursdays 7:30pm on ABC iview and ABC TV