The AI Overuse Problem Is Not AI

By Sebastian Scandura

It is the disappearance of the people who know what good looks like.

Following our recent conversations, I was asked a blunt question.

How are we managing the increasing use, and overuse, of AI in Australia?

The honest answer is: not well enough.

Not because AI is the enemy. It is not. AI is already useful. It is already embedded. It is already producing outputs that would have taken a junior analyst, consultant, policy officer, lawyer, project officer, executive assistant or middle manager hours to produce. In some cases, days.

The problem is not adoption.

The problem is practice.

And the practice is now moving faster than the assurance.

·  ·  ·

The pattern is becoming visible.

In government, the Australian Public Service has already completed a whole-of-government Microsoft 365 Copilot trial. The Digital Transformation Agency’s evaluation describes a rapid public-sector experiment with a tool selected partly because it could be deployed quickly inside existing Microsoft arrangements and familiar applications. DTA’s Copilot evaluation

That matters because the findings were not only about productivity. The DTA’s own adoption analysis warned that poor data security and information management processes could lead to Copilot inappropriately accessing sensitive information. It also said tailored training was needed, and that managers require specific training to verify Copilot outputs. DTA whole-of-government adoption analysis.

That should have been the headline.

Not 'AI saves time'. Not 'AI improves productivity'. Not 'AI will transform government'.

The headline should have been: the person approving the output

still needs to know whether it is right.

That is not a small implementation detail.

That is the centre of the risk.

·  ·  ·

The private sector has already given us the warning shot.

Deloitte agreed to provide a partial refund to the federal government after a $440,000 report was found to contain errors, including incorrect references and material later linked to the use of generative AI. The Department of Employment and Workplace Relations said the substance of the report remained unchanged. Deloitte said the updates did not affect the findings. The Guardian report on the Deloitte refund.

That may be true.

It may also be beside the point.

The issue was not merely that AI hallucinated. AI does that. We know that.

The issue was that a professional services firm, operating in a high-trust environment, produced material for government that apparently passed through the human assurance chain without the defects being caught before publication.

That is not an artificial intelligence problem.

That is a human assurance problem.

It is also, more uncomfortably, a procurement problem. A capability problem. A professional standards problem. A 'who is actually doing the work' problem.

In a previous Sincero Field Note, You Are Not a Persona, I asked who in the organisation has the explicit job of standing up and saying, 'wait, what if this AI is wrong for the actual people we serve?' You Are Not a Persona.

In Acceptable Collateral, I argued that the 'human in the loop' claim should be auditable, not asserted in a vendor brochure, and that organisations deploying AI need someone with the authority to call stop when the deployment puts people at unacceptable risk. Acceptable Collateral.

This Field Note sits in the same place.

The missing person is still missing.

Only now, that missing person is also the person who knows whether the AI output is any good.

·  ·  ·

We need to stop pretending the phrase 'human review' solves anything by itself.

A human review by whom?

A graduate with six months in the organisation? A middle manager who has been told to 'use Copilot more' but has not been taught how to interrogate the answer? A senior executive who receives the product in a brief, assumes it has been checked, and signs it forward? A consultant who knows the format of the deliverable but not the substance of the environment?

The old principle still holds.

Crap goes in. Crap comes out.

AI did not repeal that principle. It industrialised it.

An experienced employee with deep organisational knowledge can use AI well. They know what to ask. They know what context matters. They know what the output is meant to feel like before it arrives. They know when a sentence sounds plausible but hollow. They know when a citation is doing too much work. They know when the machine has produced an answer, but not a judgment.

Used that way, AI is not dissimilar to a junior employee.

You give it clear instructions. You give it context. You define the task. You explain what good looks like. You ask it to do the menial first pass so you can operate at a higher level. You review the work. You correct it. You shape it. You own it.

That is useful.

That is not the same thing as outsourcing judgment.

The danger starts when the person at the controls is not suitably qualified and experienced for the task in front of them. They may be qualified on paper. They may be senior enough in structure. They may have the account. They may have the delegation. But if they do not carry the organisational memory, the domain context, or the professional pattern recognition required to assess the output, then AI has not accelerated quality.

It has accelerated uncertainty.

In Sincero’s Self-Healing Risk methodology, the risk environment is described as something that must sense, heal and learn in near-real time. That does not remove the need for human judgment. It raises the standard for it. A living control environment still needs people who understand what the organism is sensing, why it is reacting, and when the reflex is wrong. Sincero Self-Healing Risk Methodology

·  ·  ·

The uncomfortable truth is that many organisations are not using AI to augment their human capability.

They are using it to compensate for the capability they already removed.

That is the part nobody wants to put in the board paper.

Over the last two years, 'efficiency' has become the polite word for workforce thinning. The work did not disappear. The people did. AI then arrived as the convenient bridge between the workload that remained and the headcount that no longer existed.

At first, that looks like productivity.

A brief appears in minutes. A policy summary appears in seconds. A client proposal appears before lunch. A risk register cleans itself up. A board paper becomes readable. A project status update sounds more polished than the project deserves.

But polish is not assurance.

And speed is not quality.

KPMG’s June 2026 Global AI Pulse found that AI adoption is accelerating, but that many organisations are still struggling with cost visibility, accountability and measurable return. The survey covered more than 2,000 senior leaders across 20 markets, including Australia. KPMG Global AI Pulse, June 2026.

That should worry boards.

Because the AI cost problem is not just the licence fee. It is the rework. The checking. The token burn. The duplicated prompting. The hidden labour of making a bad output usable. The opportunity cost of believing the tool saved time when the organisation simply moved the time cost into a less visible part of the workflow.

This is where the business case starts to rot.

An experienced person can get to a strong first draft quickly because their prompt is full of judgment. They know what to include, what to exclude, what assumptions to constrain, what tone to avoid, which legislation matters, which stakeholder will push back, and which sentence will cause trouble in the steering committee.

An inexperienced person has to use more tokens to find the same destination, if they find it at all.

The machine is not expensive because it writes.

It is expensive because people who do not know

what good looks like keep asking it to try again.

·  ·  ·

There is a second-order problem here, and it is more serious.

We are cutting the human supply chain of knowledge.

For decades, organisations trained future experts by giving them the work nobody senior wanted to do. The junior wrote the first draft. The graduate built the spreadsheet. The analyst checked the source. The consultant prepared the minutes. The policy officer chased the evidence. The project officer maintained the register.

That work was sometimes tedious.

It was also how people learned.

They learned what a good brief looked like by writing bad ones and having them marked up. They learned how decisions were made by sitting quietly in rooms where decisions were made. They learned the difference between a real risk and a decorative risk by watching which risks senior people took seriously. They learned judgment through exposure.

AI is now taking a large part of that work.

Again, that is not inherently bad.

But if organisations remove the developmental work without replacing the developmental pathway, they should not be surprised when, five years from now, they have plenty of people who can operate an AI tool and very few people who can assure the output.

The human supply chain is not a metaphor.

It is the way national capability is actually built.

Training, mentoring, task exposure, delegation, review, correction, and repeat performance are not soft cultural extras. They are the mechanism by which junior staff become competent middle managers, competent middle managers become trusted senior practitioners, and trusted senior practitioners become the people who can protect an institution when the machine is wrong.

If we break that chain, those currently tasking AI will not simply become more efficient. Many of them will stall. They will never advance into the level of judgment the organisation assumes they are developing. They will never progress from asking AI for answers to knowing whether the answer is safe, lawful, ethical, accurate, proportionate and useful.

That creates a stale workforce.

Not lazy. Not unintelligent. Stale.

A workforce that can operate tools but cannot mature into judgment. A workforce that can generate outputs but cannot reliably build institutional memory. A workforce that can sound polished but cannot always tell whether the work is true.

That affects more than corporate productivity.

It affects national capability.

It affects the quality of public administration.

It affects regulated services, health decisions, welfare systems, cyber operations, procurement, risk reporting and the advice that reaches ministers, boards, executives and the public.

And it affects everyday Australians, because everyday Australians are the people on the receiving end of those decisions.

That is the core of the problem.

·  ·  ·

The result will not appear immediately. It will not be visible in next quarter’s reporting. It will not show up cleanly in the first AI benefits realisation paper.

It will arrive slowly, then all at once.

A hollowed-out middle.

An aged expert layer.

A junior cohort that knows how to prompt, but not how to decide.

A leadership group receiving machine-polished work from people who cannot reliably tell whether the work is true.

That is not transformation.

That is institutional memory loss

with a productivity dashboard attached.

·  ·  ·

The data problem sits underneath all of this.

AI systems depend on data. That sounds obvious until you follow the implication. If the data supply chain is contaminated, poorly governed, badly labelled, maliciously altered, or recursively polluted by synthetic outputs, the model’s apparent fluency becomes dangerous.

The Australian Signals Directorate’s Australian Cyber Security Centre, with international partners, has warned that AI data security is central to the accuracy and integrity of AI outcomes. The guidance names data supply chain risk, maliciously modified or poisoned data, and data drift as significant risks. ASD/ACSC AI data security guidance

Related ASD/ACSC supply-chain guidance tells organisations to prefer trusted and reputable sources, quarantine and test externally sourced data, review and preprocess data before use, and use provenance or lineage tracking where possible. ASD/ACSC AI and ML supply-chain guidance

Nature published research in 2024 showing that models trained recursively on AI-generated data can suffer 'model collapse', a degenerative process in which models progressively forget the true underlying data distribution. Nature, AI models collapse when trained on recursively generated data

That should end the casual conversation about AI outputs being 'probably fine'.

At scale, 'probably fine' becomes a national vulnerability.

There is also a more adversarial question we need to be mature enough to ask.

I am not saying there is verified evidence that hostile state actors are deliberately feeding Australian-facing AI systems with dirty data at scale to degrade our decision-making.

I am saying that if I were sitting on the other side, looking for a denial-of-capability operation that was cheap, persistent, deniable and difficult to attribute, I would absolutely explore whether the data supply chain of my opponent’s AI ecosystem could be polluted.

Not hacked in the cinematic sense.

Fed.

Skewed.

Saturated.

Flooded with plausible nonsense.

Made just unreliable enough that the human institutions depending on it begin to lose trust in their own outputs.

That is not science fiction. It is an extension of information operations into the knowledge infrastructure itself.

It is why sovereign AI capability matters. It is why provenance matters. It is why government-controlled tools matter. The Department of Finance has described GovAI Chat as a secure, government-controlled generative AI tool for the APS, designed to operate within Australian Government infrastructure so data remains within Australia and under government control. Department of Finance, Introducing the APS AI Plan GovAI’s own material says prompts and files are not used to train models, and that activity is logged for security and accountability. GovAI Chat

That is the right direction.

But a sovereign AI tool without sovereign human judgment

is still only half a control.

·  ·  ·

So what do we do?

The easy answer is policy.

The harder answer is practice.

Australia already has policy movement. The updated Policy for the responsible use of AI in government, effective from 15 December 2025, requires non-corporate Commonwealth entities to meet mandatory requirements for accountable officials, transparency statements, strategic AI adoption, responsible-use operations, use-case accountability, internal registers, staff training and AI use-case impact assessment. Policy for the responsible use of AI in government, version 2.0

That is necessary.

It is not sufficient.

The Department of Finance’s guidance on implementing Australia’s AI Ethics Principles says governments remain responsible for AI outputs, should ensure incorrect outputs are flagged and addressed, should enable human oversight, and should avoid overreliance because overreliance can lead to acceptance of incorrect or biased outputs. Implementing Australia’s AI Ethics Principles in government

Again, necessary.

Again, not sufficient.

Because the failure mode we are now facing is not a lack of words.

It is a lack of teeth.

We cannot afford another five-year maturity plan that reads well, launches politely, and changes little. We cannot keep treating organisations as an adult learning environment, hoping maturity will arrive eventually if we give them enough guidance notes, principles and soft encouragement. We cannot keep throwing coins in the fountain and calling it uplift.

We do not have the benefit of sleeping on this anymore.

The sun is already peering over the horizon.

The expectation must be mature from the beginning. Boards, owners, directors-general and accountable officials must feel the weight of the decision before the system is deployed, not years later after harm, waste, rework or public distrust has already become normalised. Without a stick at the beginning, many organisations will not mature. They will wait. They will posture. They will publish artefacts. And the nation will absorb the risk of inaction.

·  ·  ·

The reset must be sharper.

One. Any AI-generated output used in government decision-making, regulated services, procurement, legal analysis, assurance, risk reporting, policy advice, public-facing communication or executive decision support must have a named human owner. Not a business unit. Not a generic clearance chain. A person. Someone suitably qualified and experienced for the content. Someone who signs for substance, not formatting.

Two. The 'human in the loop' claim must be auditable. If an organisation says a human reviewed the output, it must be able to show who reviewed it, what they checked, what sources they verified, what changes they made and what accountability they accepted. If the review cannot be evidenced, it must not count.

Three. AI adoption funding must not be approved without a human capability plan. Every business case that removes labour through AI must explain how the organisation will maintain the knowledge pipeline that labour used to create. Where will the next senior analyst come from? Where will the next assurance lead learn? Where will the next policy officer develop judgment? If the answer is 'the tool will help them', the business case is incomplete.

Four. Training must mean more than tool familiarisation. It must include mentoring, supervised practice, source checking, critical reasoning, assurance habits, decision ownership and exposure to real organisational context. Juniors must still be given the experience, exposure and correction that turns tasking into judgment. Teaching someone how to prompt is not the same as teaching them how to think.

Five. Token economics must be reported as an operating risk. The board must know not only what the licence costs, but what the organisation is spending in usage, rework, verification, duplication and failed experimentation. AI cost visibility is no longer a finance detail. It is a governance control.

Six. Data provenance must move from technical concern to executive concern. Leaders must know what data their AI tools can access, where that data sits, whether it includes sensitive or classified material, whether it is being used to train anything, how synthetic content is controlled, and what would happen if the data supply chain were polluted.

Seven. Regulators must stop accepting policy artefacts as proof of control. A transparency statement is not transparency. A register is not governance. A training module is not capability. A human-in-the-loop statement is not assurance. The test must move from 'does the artefact exist?' to 'does the control work?'

·  ·  ·

This is the core point.

Australia does not need to become anti-AI.

That would be lazy, unrealistic and strategically harmful.

Australia needs to become serious about AI.

Serious enough to use it where it helps. Serious enough to reject it where it weakens judgment. Serious enough to retain the people who know what good looks like. Serious enough to train the people who will need to know tomorrow. Serious enough to regulate the claims, not just admire the tools.

The future risk is not that AI becomes too powerful.

The nearer risk is that we become too dependent on outputs

nobody can properly assure.

That is how trust collapses.

Not in one scandal. Not in one fake citation. Not in one flawed report. In hundreds of small moments where a machine produces something plausible, a human does not know enough to challenge it, an organisation accepts it, and the public eventually learns that the assurance layer was thinner than advertised.

Then we will not have trust in AI.

We will have negative trust.

People will stop believing the machine.

Then they will stop believing the human who signed the machine’s work.

Then they will stop believing the institution.

That is the near horizon issue.

It is no longer approaching.

The sun is already peering over the horizon.