Is AI-written software safe? The questions a buyer should ask

How do I know if AI wrote the software?

"Did AI write this code?" has no auditable answer. Almost every team uses an assistant today, and a supplier claiming otherwise cannot prove it; writing "AI will not be used" into a contract is writing a clause that cannot be enforced.

The auditable question is this: what did this code pass through before production? Are there automated tests, were the dependencies scanned, did a person review it line by line, is that review recorded? With those four in place the origin of the code is secondary information; without them, a human author does not protect you.

Why does AI write insecure code?

When writing code a language model produces the most likely continuation rather than the correct one. The more often a pattern appeared in its sources, the more readily it suggests it — and most of the examples that have circulated online for a decade are patterns that today's security practice has moved past. The model is not malicious; it repeats what is popular, and what is popular is not always right.

Model-origin flaws therefore cluster predictably rather than randomly: outdated cryptography calls, hand-concatenated database queries, loose access control, and secrets embedded in code. All of them are things automated tooling catches — provided that tooling is in the pipeline.

What are the risks of AI-written code?

Alongside the classic flaws, generative AI carries three risks of its own, and these are exactly the questions to ask as a buyer.

Can AI suggest a library that does not exist?

While suggesting code, a language model can invent a package name that does not exist. On its own that is merely an error; installation fails and the developer fixes it. The problem is that attackers noticed the behaviour: they identify the names models invent most often and publish real packages under those names. That package now installs, and whatever they put inside it enters your system.

The defence is simple and entirely automatic: pin the dependencies, verify signatures and integrity, and scan for known vulnerabilities on every release. Those are already part of a good deployment pipeline — so this risk does not demand a new discipline, it demands that the existing discipline actually be applied.

How do you write an AI clause into a contract?

• All delivered code passes automated tests and human review; the review record is shared on request.
• Dependencies are pinned and scanned against known vulnerabilities on every release; a critical finding stops the release.
• Client code and data are processed in an environment not used for model training; the provider and region are named in writing.
• Every flow where the model can perform a real transaction carries human approval, and the approval is recorded.
• Source code, dependency list and deployment configuration transfer together on delivery — with any one missing, no audit is possible.

What these clauses have in common is that none of them says "do not use AI". Each describes a verifiable behaviour and asks the supplier to evidence it.

Who is liable when AI makes a mistake?

When a security incident happens, what gets examined is not who wrote the code but which measures you took as the data controller. "The model wrote that code" is not a defence; if personal data leaked, the obligations to notify, to keep records and to have technical measures in place sit directly with you. The only thing that helps in an audit is therefore records produced before the incident: test results, scan reports, review records.

Compliance obligations sit on top of those records, and which system falls into which category is a separate subject; our guide on building software with AI covers the risk classification and the obligations it creates in detail.

Should I work with a firm that uses AI?

Using AI is not a disqualifier on its own — because the alternative is not risk-free either. A team not using AI does the same work more slowly and usually more expensively, and that code is just as likely to be unverified. The right comparison is not "model or human" but "verified or unverified". Measured that way, a team using AI on a well-built pipeline ships noticeably safer work than a team with no pipeline at all.

Is AI-written code cheaper to maintain?

The opposite, often: maintenance cost can rise. Rather than following one consistent architecture, a model produces solutions that are individually reasonable but differ from each other, leaving a codebase that does the same job five ways. What prevents that is not the model but human review and a written architectural decision. Skip the review and the time saved is repaid with interest in year two.

My supplier will not say whether they use AI — how do I tell?

Do not try to tell; measure instead. There is no reliable external way to establish the origin of code, and it would not help if there were. Measure the verification: ask for a test coverage report, the pipeline record of the last ten releases, and the review record of a change picked at random. With those three the origin does not matter; without them, a human author does not protect you.

Are open source components a risk?

Not the component itself but how it was chosen and how it is updated. AI assistants recommend abandoned libraries too, because they were once popular. Three things to ask: when was this dependency last updated, how many people maintain it, and does its licence let you keep your product closed. The third is the most skipped and the most consequential.

Does a team using AI deliver faster?

Usually yes, but the gap is smaller than claimed and where it sits matters. What speeds up is the known solution to a known problem; what does not is framing the scope, testing integrations and debugging. That second set decides the delivery date, so the shortening in the overall schedule usually lands between ten and twenty per cent.

“The origin of code cannot be audited; its verification can. The second is what belongs in the contract.”