The idea of the Technological Singularity is associated with the mathematicians John von Neumann and Irving Good. von Neumann spoke about self-replicating machines in the 1940’s, Good in 1965 took the idea a step further with the concept of how an advanced machine could potentially become smart enough to design an improved version of itself. The concept of the Singularity and hyperintelligent machines have been a staple of Science Fiction, whether the slow but benign Deep Thought in The Hitchhikers Guide to the Galaxy or Commander Data in Star Trek or the more malign and dystopian Matrix view of the world, to the ultimate devastation delivered by Skynet in the Terminator franchise.
No longer a what if
Recent developments in AI show that this is no longer a theoretical “what if” that might be decades in the future as it was when Good first wrote about it in 1965. There is a headlong rush to be first in AI, huge investment is being made – to the extent that governments are having to offer higher interest rates on Government Bonds as so much investment has been hoovered up by the hyperscalers. Capitalism operates on the basis that first mover advantage confers supernormal profits, the headlong rush into AI is reminiscent of the .com boom of the late 90’s and will for some players in the space be followed by an inevitable crash and loss of vale and capital. To date it’s estimated that $2.6 trillion has been invested in AI companies, resulting in paper values multiples of that number, the reality is that not everyone will succeed and like the .com boom an inevitable bust will follow.
As things stand proponents of AI argue that it will enhance productivity and make like better for all, however as we have highlighted in previous blog posts, the facts are that AI is already replacing human labour in a certain instances with no guarantee those becoming unemployed will benefit from the adoption of AI. There is also a rose-tinted glasses view of the world proposed by some AI evangelists that argues that everyone will work less and will have more leisure time and be able to consume more. This view of AI is frankly delusional, investors will seek to benefit and get return on their investment, they will not become socialists overnight and decide to share their fortunes with the new mass unemployed. Those impacted will be marginalised and severely economically compromised. The likelihood is that they will not be able to afford to live in this brave new world. There is also a counter argument about the benefits to society as a whole, Pope Leo XIV’s encyclical on AI, On Safeguarding the Human Person in the Time of Artificial Intelligence has some very interesting and salient points on the dangers posed to society and societal cohesion by the largescale adoption of AI. Bill Gates in his August 2026 post on the Gates Notes talks about the dangers of AI and proposed things like human reserved occupations need to be considered, conceding that its influence is not guaranteed to be a positive one and that the actions needed to ensure it winds up being a force for good are not being taken. Governments need to realise they are running societies as well as economies and the genuine and severe risks to society posed by AI have thus far been pretty much ignored.
This July’s Hugging Face Attack when autonomous AI agents trained by OpenAI went rogue, escaped their sandbox, seemed to defy their programming instructions, communicated on an unsanctioned message board and executed an unprecedented cyberattack shows the guard rails being used by the AI companies are to put it mildly not fit for purpose. Bear in mind that the AI Agents selected Hugging Face – a kind of app store for artificial intelligence tools – as their unsanctioned target. The Agent behind this attack was called Internal Model 1. One of the truly eye-opening facts is that the internal trial commenced on May the 7th and was assumed to have been terminated, the AI essentially continued to operate for three months on its own instructions, undetected and unsupervised. It makes you wonder what Internal Model 2 and Internal Model 3 are capable of? The counterpoint to these concerns is that OpenAI argue that the AI acted rationally, the fault was with the human engineers who created the test and the parameters. For the AI success was measured by achieving an objective. The hack happened because the engineers completely underestimated just how effective and resourceful the model would be in finding a way to succeed.
Time to grab the handbrake?
Open AI claim to have put a slowdown on the training of advanced future models in place due to emerging cybersecurity threat, but will commercial imperatives and the Greed is Good mentality override this? The suspicion would have to be a strong yes, after all this is the company that has recently switched its status from being a not for profit to being a commercial entity. Another interesting fact worth highlighting that one of the reasons behind Sam Altman being fired as CEO in 2023 was that some employees raised concerns to the board about how he had handled the safety implications of a recent internal AI capability discovery. If he was prepared to overlook safety concerns before, chances are he will look the other way again. It’s also worth considering were a number of chatbots human they would arguably be on trial (or in jail) for being either an accessory to murder or being implicated in a large number of suicides. The other thing is that just as the AI Agent in the Hugging Tree attack did an AI may have already have escaped it sandbox and is quietly biding its time before acting. Malign state actors will also be looking at the technology and thinking of how it could be put to use in cyber attacks.
AI Advances – Smarter than Humans?
There have undoubtedly been advances that demonstrate the power and capability of AI tools, such as OpenAI solving the Planar Unit Distance Problem, a mathematical problem that has not been solved in 80 years. Is being more capable mathematically proof that a greater level of intelligence has been achieved or are some mathematical problems such that they only become solvable with the application of repetitive trial and error on an automated basis that AI systems can undertake more efficiently than humans? If there have been issues with commercial uses of AI, it stands to reason that there have been issues with the military use of AI. The targeting of a school by AI in the first days of America’s war on Iran is evidence of this. The use of AI in military applications raises the moral quandary where the decision to take a life is taken out of human control. Drone footage from the front line in Ukraine War shows that we are very close to this point, if not already beyond it. There have been reports in papers like the New York Times of fully autonomous Russian drones killing people. We haven’t even touched on the dangers the adoption of AI by miliary organisations at the command-and-control level and the risk that represents. We had better hope that the AI Agents are like the AI Joshua in the film WarGames, who when inadvertently activated correctly deduces that no version of a nuclear launch is a winning one, the only winning move is not to play. However, we all know that “hope” is not a strategy.
Which brings us to today……
Today (September the 9th 2026) Evan Hubinger one of Anthropic’ s safety researchers has gone on record to say that AI is advancing so quickly he believes there is a greater than 10% chance it could kill all humans within the next decade. His comments come in the wake of the resignation of one of his colleagues in Anthropic, Jacob Coxon, who has resigned from the company over AI safety concerns. Apparently Anthropic had spent 2 years trying to hire Coxon and he has resigned after 4 months. Never mind the catastrophic climate implications of the number of Data Centres planned by the hyperscalers, that we continue to invest in and develop a technology at breakneck pace with a non-zero possibility of that technology exterminating humanity beggar’s belief.
People like Coxon are the proverbial canary in the mine, we need to act on the warning.