AI makes software projects faster — and shifts the risk to a place traditional development processes rarely control. A Veracode study across more than 100 language models found that 45% of generated code introduces at least one of the OWASP Top 10 vulnerabilities, and that rate has stagnated for two years despite noticeably better syntax (Veracode GenAI Code Security Report).
The goal is not to avoid AI. In Germany, per Bitkom, 36% of companies now use AI — almost twice as many as a year earlier. The goal is to control AI like any other production dependency: with clear rules, ownership and technical guardrails.
Where the real risks arise — not in the tool
AI risk rarely comes from a single tool — it comes from missing boundaries, missing tests and missing ownership. The sober numbers help sharpen the gut feeling. Veracode had more than 100 models solve 80 coding tasks and checked the result against the OWASP Top 10: 45% of solutions were insecure, Java's failure rate reached 72%, and cross-site scripting and log injection exceeded 85%.
A second risk is more subtle. A study on package hallucinations presented at USENIX Security 2025 generated 2.23 million code samples — 19.7% referenced a package that simply does not exist. Attackers pre-register those invented names on npm or PyPI ("slopsquatting"). Adopt AI suggestions unchecked, and in the worst case you pull a stranger's code into your supply chain.
The lesson is not "AI writes bad code" but: AI output is a suggestion, not a finished result. That is exactly what reviews, tests and clear rules are for — just as we do in a code audit anyway.
Five risk categories traditional processes overlook
Most AI incidents map to five categories — and each has a concrete countermeasure. Cover these five fields systematically and diffuse "AI anxiety" turns into a manageable risk profile.
| Risk category | What happens | Countermeasure |
|---|---|---|
| Privacy | logs, customer data, schemas land in external tools | data classes, anonymization, EU hosting, clear approvals |
| Hallucinations | invented APIs, packages, wrong versions | reference sources, require tests, compare with docs |
| Insecure code | broad permissions, weak error handling, weak typing | architecture-aware review, secret protection, edge-case tests |
| Prompt injection | hidden instructions in content or requests | hard tool boundaries, approvals, input and output filters |
| Traceability | nobody knows what the system saw and why | audit logs, prompt versioning, monitoring, approval history |
Order matters: privacy and traceability are architecture decisions that belong at the start — not something you bolt on shortly before launch.
Prompt injection: the new attack model for AI agents
As soon as an AI system reads external content or uses tools, it gains an attack surface that classical software does not have. A document, website or user request can carry hidden instructions that push the model toward unintended behavior. In the OWASP Top 10 for LLM Applications 2025, prompt injection ranks first (LLM01) — not by accident, since the stochastic nature of the models means there is no foolproof remedy.
In practice: an AI agent must not automatically do everything just because a document says so. What works is defense in depth — input validation, output filtering, restricted permissions and human approval for sensitive operations. The more tool access an agent has, the tighter the guardrails must be. How that looks in production AI agents in the enterprise depends directly on which actions are reversible and which are not.
EU AI Act: risk-based — and the timeline moved in 2026
The EU AI Act follows a risk-based approach: not every AI application carries the same obligations. An internal code-summary tool is assessed differently from a system that screens applicants, prepares credit decisions or supports medical recommendations. And the timeline moved noticeably in 2026: with the Digital Omnibus — a provisional agreement between Council and Parliament of 7 May 2026 — the high-risk obligations are deferred.
Concretely: the obligations for stand-alone high-risk systems under Annex III (recruiting, credit scoring, education and more) move from August 2026 to December 2027, and for AI embedded in regulated products under Annex I to August 2028 (timeline overview). The GPAI rules have applied since August 2025 and the transparency obligations largely stay on the original schedule. For companies the deferral changes little about the principle: what is the system's purpose, who is affected, are decisions automated, is there human approval? That classification belongs at the start, not shortly before launch.
Governance that is not bureaucratic
AI governance does not have to be heavy — it has to be practical. A good start rests on five building blocks: an AI policy (allowed tools, data classes, approvals), a use-case assessment before implementation, technical guardrails (roles, permissions, logging, tool boundaries), quality assurance (tests, evaluation sets, human reviews) and defined operations (monitoring, error analysis, ownership).
On the technical level, AI follows the same rules as any production system — with a few extras:
- no secrets in prompts, server-side tool execution with permission checks
- separate development and production environments
- input and output validation, rate and cost limits
- logging without unnecessary personal data
- human approval for irreversible actions
- tests against prompt injection and unauthorized tool use
Local or private models are not required for every project, but make sense when data must not leave the company or industry-specific compliance imposes strict requirements — for internal knowledge search, contract analysis or sensitive product data. How we think about that alongside privacy and operations is shown in our AI integration.
Next steps
Three questions quickly show how large your AI risk really is today:
- Data: Do you know which data flows into which tools — and which data must not leave the company?
- Code & agents: Is AI output reviewed, tested and bounded by hard tool limits before it reaches production?
- Ownership: Does every AI system have a clear classification, audit logs and a responsible person?
Wherever you hesitate, it's worth a closer look. We treat AI projects as software projects with special requirements — pragmatically, with an eye on roadmap and budget. Take a look at our AI integration or book an intro call directly.
