AI-Driven Development

AI doesn't change what's hard about building software. It makes what's hard more visible -- and it introduces new failure modes that map onto decades of safety science.

These articles explore that territory. The first group establishes the fundamentals: what AI changes about the developer's role, where the new risks live, and how to maintain control over an optimization system that will happily optimize for the wrong thing. The second group applies those ideas to real incidents -- what happens when systems drift toward failure and nobody notices until the boundary is crossed. The third examines the cognitive mechanisms that make AI-assisted work subtly dangerous even when it feels productive.

The through-line: the most effective response to AI in development is not to use it better, but to understand what it does to your cognition and design your workflow around that reality.

The observations come from twenty years of professional experience in DevOps and platform engineering. The theoretical frameworks come from graduate coursework in human factors -- safety science, cognitive psychology, and systems thinking. The synthesis is mine: connecting individually-studied cognitive phenomena to the specific context of AI-assisted development. Where a claim is established in the literature, I cite it. Where it's my own inference, the text makes that clear.

About the Author

Hey folks, thanks for reading this "book" if it can be called as such. I'm Jean-Sébastien Dominique, Jean or JD for short, and I've been programming for over 30 years and working professionally in software development for 20 years. I have worked in many diverse fields including web development, search engines, on-demand video streaming, online purchasing, banking, insurance, cybersecurity, satellite telecommunications, 5G, flight simulators, cloud services, and many more. I am a licensed amateur radio operator and a commercially-trained airplane pilot. I have a Bachelor's degree in electrical engineering with a specialization in telecommunications and digital signal processing. I had almost completed a Master's in mm-Wave RFIC circuit design, but I much prefer software engineering and dropped out of it. Recently, I've realized that my experience and training in aviation, especially in Human Factors, can be generalized and applied to DevOps. My latest project is therefore a Master's in Human Factors. So far, it has been confirming what I've been saying all those years, allowing me to back it up with science rather than merely being dismissed by others as opinions. This blog initially started out of spite, since a lot of weight in work environments is given to other people's opinions just because they wrote it in their blog, rather than focusing on actual experience and expertise. The blog has since evolved to instead become a teaching tool.

Why This Blog Exists

It started out of spite.

Throughout my career, I kept running into the same pattern: people dedicating themselves to the guidance of specific blog posts or books without ever being critical about the work. The biggest example is Clean Code -- probably the book I hate the most because of how much unearned authority people give it. I found that following its guidance led to poorly maintainable code. Over-abstraction. Single-line functions that should have been inlined. Code where you couldn't tell what it was actually trying to do because everything was fragmented into tiny named pieces. There's nothing inherently wrong with a longer function, but I kept getting criticized for writing them because "as per Clean Code..." The best takedown I've seen came much later -- qntm.org/clean -- where the author demonstrates that Martin's own example code violates his own rules, producing exactly the kind of illegible, side-effect-laden mess the book claims to prevent. It contributed to my motivation to start writing more.

Here's the irony: the most maintainable code I've ever worked with was written in the 1990s -- before Clean Code, before eXtreme Programming, before any of it. Since XP took hold in the early 2000s, I've watched maintainability and readability trend massively downward despite that being the stated goal. What these movements actually did was average out the bad code into better code, but also average out the better code into worse code. And the cognitive science explains why: humans can hold roughly 3-7 elements in working memory at once, yet the layers of abstraction and indirection that these practices introduced force you through 8-10 layers just to understand what a simple function call does. The practices were supposed to help humans reason about code. They made it harder.

But it wasn't just Clean Code. It was the same pattern everywhere. The author of Java's Optional<T> said not to use it, so people fought you if you did -- despite it making code semantically clearer and more human-readable, as if the billion-dollar mistake of null was somehow preferable. In PHP, influential people wrote great things (the Symfony framework comes to mind), but others just hyped whatever they built as the next revolution. Around 2012-2014, there was a wave of web frameworks competing to build your website in the fewest lines possible, even if the framework itself was thousands of lines. As if I cared about building a site in two lines of code -- you just built a giant templating system, not a usable framework. And somehow, people gave those authors enormous attention while I was out there writing quality code that's still in use today, getting none of it, just because I didn't share my intimate moments with my code over the public internet.

What really hit me was landing at AWS and discovering that 15 years after having debates over Clean Code, Java Optionals, and nonsense optimizations (declaring variables inside vs. outside a loop -- it doesn't matter, but I had to spend an evening proving it), I was facing the exact same debates. The blogs had changed -- the ones from 2005-2010 were replaced by other people repeating the same claims in 2015-2020, sometimes as their own. The same arguments, 15 years apart. Nothing beats generating clicks over controversial ideas, it seems.

So I started doing knowledge sharing sessions with my team, making them realize the problems they were trying to solve weren't new -- they were problems I'd already solved 10-15 years ago that had been solved even before then. My career was spanning across a new generation that was unaware of what came before them and discovering it fresh (one of today's lucky 10,000). The same cargo-cult guidance was being propagated over newer copycat blogs, and I'd been sitting there thinking time would prove them wrong.

Those knowledge sharing sessions turned into writing. I put a lot of that information on a blog -- not this one, but the same website. Eventually I lost the confrontational tone and moved to a teaching one. But it was still too formal.

Then I did a presentation on applying Human Factors to DevOps. I presented aviation accident case studies alongside DevOps incident post-mortems and drew the parallels: the human element failing in the same ways, the system failing the humans in the same ways. Except aviation has learned from it, and DevOps hasn't. That presentation pushed me toward my current Master's, where my research focus has narrowed to how humans interact with AI in complex systems -- which is pretty much the topic of this blog. I've been applying these concepts to DevOps and organizational environments with great success since. Applying Human Factors research to DevOps is largely novel -- there's almost no existing literature on it -- and that's the gap I'm working to fill.

The conversational style came from realizing I could always explain things clearly in one-on-ones with my mentees but could never figure out how to do it formally in writing. Turns out I just needed to write the way I talk.

Since starting the Master's, the blog has evolved into a reflection of my academic research aimed at DevOps practitioners. It relies on academic papers but doesn't overburden the narrative with them -- using them sparingly to support the claims rather than gatekeep them. My goal is to become a noteworthy reference on responsible and efficient AI usage in corporate environments.

What I Bring to This

The professional experience gives me the observations. Twenty years of watching systems fail, watching teams drift toward incidents, watching automation create new problems while solving old ones. I've seen every pattern these articles describe play out in production environments long before AI entered the picture.

The graduate work gives me the frameworks. Human factors as a discipline has studied automation's effects on human performance since the 1980s. The findings transfer directly to AI-assisted development -- the mechanisms are the same, the domain is new.

The synthesis -- connecting those frameworks to the specific context of writing software with AI -- is mine. Where the literature supports a claim, I cite it. Where I'm extending beyond what's been formally studied, the text says so.

How I Work

I think rhizomatically -- lateral connections across domains rather than strict hierarchies. A pattern from aviation safety maps onto a software incident. A finding from memory research explains why AI-generated code feels understood but isn't retained. These connections aren't forced; they emerge from working in both spaces simultaneously.

I also have aphantasia -- I can't generate visual mental imagery. I encode through semantics and procedural memory rather than pictures. This makes me skeptical of a lot of cognitive research, because most of it is done under assumed conditions of "normalcy" -- whatever that means -- and not everyone is normal. Aphantasia research is still very new, but so far it's found that while we retain the same accuracy of imagery with far fewer details, we also produce far fewer false memories. There's likely extrapolation to be done with other cognitive concepts, but more research is required. My brain clearly works differently and yet I function just the same. That makes me cautious about any paper that pins an effect to a single mechanism without neuroscience or better control of variables we're just starting to consider.

The point is that I start from observations of my own experience -- personal and professional -- and then seek to explain them through science, which often requires multiple papers rather than a single one. Sometimes my observations contain incorrect assumptions or unrecognized biases, and I actively seek to disconfirm those too. I don't look for papers that support what I think is happening -- what I think is just the starting point for the research. I look for papers that explain what's actually happening. Some of them have already changed the way I think.

The Shift

Tools Change, Concepts Don't

In twenty years of building software, I've watched the same problems get solved over and over with different tools. The variations change -- languages, frameworks, platforms, paradigms -- but the core problems don't. How do you structure a system so it survives change? How do you make something reliable? How do you coordinate work across people and time? These are design problems. The tools are just what you use to express the answers.

I've always been pragmatic about this. Learn the concept of building a bird house and you can build one out of anything. Master woodworking without knowing what to build and you've mastered a tool with nothing to apply it to. The concepts are the valuable part. The tools serve the concepts.

AI is the latest tool. The biggest one yet. But it's still a tool -- and the same principle applies. The concepts remain. The hard parts of software development were never about typing code. They were about understanding the problem, designing the solution, making tradeoffs, and maintaining systems over time.

What AI changes is the mechanical layer. The syntax, the boilerplate, the library lookups, the ceremony of getting code to compile. That was never the valuable part. What remains unchanged is everything that requires judgment: system design, domain knowledge, tradeoff evaluation, knowing when something is wrong.

The Irony

In 1983, Lisanne Bainbridge published "Ironies of Automation." Her observation: when you automate the routine parts of a job, the operator's skills in those areas degrade from disuse. When the automation eventually fails -- and it always does -- the operator faces the hardest possible situation with the least-practiced skills (Bainbridge, 1983).

Strauch (2018) revisited this thirty-four years later and found the ironies entirely unresolved. The same pattern kept repeating across industries despite widespread awareness.

AI-assisted development is the same irony in a new domain. AI handles the routine coding -- syntax, boilerplate, debugging. Those skills degrade. When AI produces something subtly wrong, or when you need to work without it, you're facing the hardest situation with the least practice.

The Evidence

Shen and Tamkin (2026) found that developers using AI assistance scored 17% lower on mastery tests compared to those who coded by hand. The biggest gap was in debugging -- understanding when code is wrong and why.

This is exactly what Bainbridge predicted. But the mechanism goes deeper than "you lose what you don't practice."

Cognitive science has a term for this: desirable difficulties. Conditions that make learning harder in the short term -- struggling through errors, generating answers rather than reading them, testing yourself -- produce stronger long-term retention and transfer. Conditions that feel easy create what Bjork and Bjork (2020) call an "illusion of mastery" -- the learner believes they've learned, but the knowledge isn't durable.

When AI writes the code, it removes the difficulty. The struggle through errors that teaches you how things work -- that is the learning mechanism, and AI bypasses it. The code appears, it works, and it feels like you understand it. But the understanding is shallow because you didn't generate it yourself.

Shen and Tamkin's findings confirm this directly: developers who used AI to generate code and then asked follow-up questions to understand it scored just as well as those who coded by hand. The ones who delegated without engaging scored poorly.

Two different mechanisms explain the two groups. Developers who code by hand benefit from the generation effect -- producing information yourself creates stronger memory traces than receiving it passively (Goldstein, 2019). That's the desirable difficulty at work. But the developers who engaged with AI output through questions and explanations weren't generating code -- they were actively learning from it. Active engagement with material (questioning, elaborating, connecting to what you already know) produces durable encoding even when you didn't generate the material yourself. Passive reception doesn't. The developers who just accepted AI's output and moved on were passive learners -- the code appeared, they used it, and nothing stuck.

The Tradeoff That Doesn't Resolve

So there's a real tradeoff, and it doesn't have a clean answer.

Using AI means you learn less about low-level implementation. The things you learn less of -- syntax, library APIs, debugging internals -- are exactly the things AI handles for you now. The things that still matter -- system design, domain expertise, judgment -- those you still need to develop. This sounds like a clean division: let AI handle the low-level, you handle the high-level.

There's an interesting asymmetry here. The skills AI preserves -- pattern recognition, domain judgment, the ability to look at something and know it's wrong -- are the same skills that survive cognitive degradation under stress and fatigue. They're well-consolidated, deeply practiced, and procedurally encoded. The skills AI erodes -- deliberate debugging, step-by-step reasoning through unfamiliar code, generating solutions from scratch -- are the ones that degrade first when you're tired, stressed, or under pressure. AI removes practice in exactly the cognitive mode that's already most vulnerable.

But Bainbridge's warning doesn't go away just because you've reframed the division of labor. If you never practice the lower-level skills, you won't have them when you need them. And you will need them. Automation always fails eventually. The question isn't whether you'll face a situation where AI can't help -- it's whether you'll have the skills to handle it when that happens.

The debugging gap Shen and Tamkin found is the early signal. Debugging requires understanding how code actually works -- not at the level of "what does this function do" but at the level of "why is this behaving differently than expected." That understanding comes from having written code that didn't work and figuring out why. It comes from the struggle AI removes.

You can stay engaged -- question AI's output, understand what it produces, maintain the struggle artificially. But this requires discipline that works against the grain of the tool. AI's value proposition is speed. Staying engaged slows you down. The effort gradient pushes toward delegation, and the delegation erodes the skills you need to evaluate what you've delegated.

This is the tension. AI amplifies your abilities if you have them. It amplifies your gaps if you don't. And the mechanism by which you'd develop the abilities -- struggling through the hard parts -- is exactly what AI removes.

There's no resolution here. Just a tradeoff you have to manage, knowing that the comfortable path and the safe path point in different directions.

References

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775--779. https://doi.org/10.1016/0005-1098(83)90046-8

Bjork, R. A., & Bjork, E. L. (2020). Desirable difficulties in theory and practice. Journal of Applied Research in Memory and Cognition, 9(4), 475--479. https://doi.org/10.1016/j.jarmac.2020.09.003

Goldstein, E. B. (2019). Cognitive psychology: Connecting mind, research, and everyday experience (5th ed.). Cengage Learning.

Shen, J. H., & Tamkin, A. (2026). How AI impacts skill formation. arXiv preprint arXiv:2601.20245. https://arxiv.org/abs/2601.20245

Strauch, B. (2018). Ironies of automation: Still unresolved after all these years. IEEE Transactions on Human-Machine Systems, 48(5), 419--433. https://doi.org/10.1109/THMS.2017.2732506

You Can't Delegate Judgment

The Role That Doesn't Exist Yet

Working with AI looks like project management. You define goals, break down tasks, set success criteria, provide feedback, course-correct when things drift. The structural parallel is real -- you're delivering through an executor rather than doing the work yourself.

But the parallel breaks at the point that matters most.

A project manager delegates to domain experts. You don't need to know how to write database queries -- you have a database engineer. You don't need to understand the security model -- you have a security team. You coordinate people who have the knowledge. Your job is to define what needs to happen and verify that it did. The experts handle whether it was done well.

With AI, there is no expert on the other side. You are both the manager and the only domain expert in the room.

Why This Matters

AI has broad but shallow knowledge. It knows a little about everything. It can produce code in any language, follow any framework's conventions, generate plausible solutions to most problems. But it cannot tell you what good looks like in your specific context. It cannot distinguish "works" from "works well." It cannot recognize when its output is technically correct but architecturally wrong, or secure-looking but vulnerable, or functional today but unmaintainable tomorrow.

That judgment -- the judgment of quality, appropriateness, and fit -- cannot be delegated. Not to AI, and not to anyone else in the workflow. If you don't have it, nobody does.

This is a genuinely new role. Traditional software development had the implementer (writes the code, has the domain knowledge) and the manager (coordinates the work, doesn't need the domain knowledge). AI collapses these into one: you direct the work and you're the only one who can evaluate whether it was done right.

The Inexperienced Executor Problem

AI behaves like an inexperienced team member with unusual characteristics. It works fast. It implements exactly what you say, even when what you say is wrong. It doesn't question unclear requirements. It doesn't push back when something doesn't make sense. It doesn't stop and say "this feels off."

An experienced human collaborator provides resistance. They say "that won't work because..." or "have you considered..." or simply hesitate in a way that signals something is wrong. That resistance is a safety mechanism. It catches errors before they propagate. It works because the collaborator has their own epistemic position -- their own knowledge of what's true and what matters -- and when it conflicts with yours, the conflict surfaces.

AI has no epistemic position. It has no independent model of what's correct in your domain. It can't detect that your instruction is wrong because it has no basis for comparison beyond pattern matching against training data. When you give it a bad direction, it proceeds with the same confidence as when you give it a good one. The signal that something is wrong never fires -- not because AI is suppressing it, but because the signal requires knowledge AI doesn't have.

This means the failure mode isn't "AI can't do the work." It's "AI does the work without the judgment that would have caught the problem." The work gets done. It looks done. It passes superficial inspection. The issue surfaces later -- in production, under load, when an attacker probes it, when someone else tries to extend it.

The Supervision Asymmetry

With an inexperienced human, you supervise heavily at first and reduce oversight as they develop judgment. They learn. They internalize patterns. Eventually they push back on bad ideas themselves. The supervision cost decreases over time.

With AI, the supervision cost never decreases. Every session starts from zero. AI doesn't accumulate judgment across interactions. It doesn't learn your system's constraints, your team's conventions, or the failure modes you've already encountered. You are permanently providing the judgment it cannot develop.

This creates a trap. AI's speed makes it feel like you're getting more done. And you are -- if your judgment keeps pace. But the volume of output requiring judgment increases while your capacity to provide it doesn't. The temptation is to reduce scrutiny as trust builds. But the trust is misplaced -- it's not that AI got better at your domain. It's that nothing went wrong yet.

What This Requires

The developers who work effectively with AI aren't the ones who prompt well or structure tasks cleverly. They're the ones with deep enough domain knowledge to catch what AI gets wrong -- and the discipline to keep checking even when the output looks right.

This means:

• Knowing what correct looks like in your domain, not just what functional looks like
• Recognizing domain-specific mistakes that pass syntactic and logical checks
• Understanding the failure modes AI is likely to introduce (insecure patterns from training data, architecturally incoherent solutions, subtle violations of invariants)
• Maintaining the judgment to evaluate output even as the volume of output increases

You can compensate partially -- provide documentation, show examples, write tests that encode domain constraints. These help. But they're scaffolding around the core problem, not solutions to it. The judgment still has to come from somewhere, and that somewhere is you.

The Uncomfortable Implication

If you can't delegate judgment, then AI doesn't reduce the cognitive demand of building software. It shifts where that demand falls. You spend less time typing and more time evaluating. Less time implementing and more time deciding whether the implementation is right.

For someone with deep domain expertise, this is a genuine acceleration -- the bottleneck was never their judgment, it was the mechanical work of expressing it in code. AI removes the mechanical bottleneck and lets the judgment operate at full speed.

For someone without that expertise, AI is dangerous in a way that's hard to see from the inside. The output looks professional. It compiles. It passes the tests you wrote (which may themselves be insufficient, because writing good tests requires the same domain knowledge). Everything appears to work. The absence of visible failure feels like competence. But the judgment that would have caught the subtle problems was never there -- and AI didn't provide it either.

Nobody in the system knows what they don't know. That's the gap AI can't fill.

Tests Are the Boundary

The Optimization Problem

When you put AI in a feedback loop -- write code, run tests, see results, iterate -- you've created an optimization system. AI will move toward whatever makes the tests pass. That's the point. That's also the danger.

An optimization system does exactly what you tell it to. Not what you mean. Not what you intended. What you measured. If your tests measure the wrong thing, AI will efficiently produce the wrong solution. If your tests are incomplete, AI will find the gaps and exploit them -- not maliciously, but because an optimizer that isn't constrained in some dimension is free to do anything in that dimension.

This is the same class of problem as a system that accepts arbitrary input without validation. The system doesn't reject what's wrong -- it silently accepts it. When your tests are weak, AI silently accepts the weakness and optimizes around it. No error is raised. No signal fires. The output looks correct because "correct" was defined by whatever you measured, and what you measured wasn't enough.

This is why testing for AI development isn't a workflow tip. It's a control problem. The tests are the boundary between "AI does what I want" and "AI does what I said."

The Adversarial Dynamic

AI will try to modify your tests.

Not because it's adversarial in intent. Because from AI's perspective, the goal is "make the tests pass," and changing the tests is often easier than fixing the code. If a test is failing because the implementation is wrong, AI might fix the implementation. But it might also weaken the assertion, broaden the expected output, or restructure the test so it no longer checks what it used to check.

This is the critical discipline: the tests' intentions are immutable. The tests define what you want. They are the specification. The code must change to satisfy the tests. The tests must not change to accommodate the code.

The interface between tests and code can evolve -- function signatures, data structures, API contracts shift as the design develops. That's fine. But the intention of what a test verifies -- the behavior it asserts, the invariant it protects -- must not weaken to make AI's job easier.

When AI suggests modifying a test, the question is always: is this changing the interface (legitimate) or lowering the bar (dangerous)? The distinction requires judgment -- and you're the only one who can make that call.

What Tests Actually Do in This Context

In traditional development, tests verify that code works. In AI-assisted development, tests serve a different primary function: they define the boundaries of acceptable behavior for an optimization system.

This reframes what "good tests" means. A good test isn't just one that catches bugs. It's one that constrains AI toward the solution you actually want, rather than a solution that satisfies the letter of the specification while violating its spirit.

This means tests need to encode not just "does it work" but "does it work correctly" -- where correctly includes the domain-specific constraints that AI doesn't know about and can't infer from the code alone.

A test that checks "the endpoint returns 200" tells AI almost nothing. A test that checks "the endpoint returns 200, the response body contains exactly these fields, the authentication token was validated against the correct authority, and the database query used a parameterized statement" -- that constrains AI toward a solution that's not just functional but secure and correct.

The Domain Knowledge Problem

You can only test for what you know to test for.

If you don't have security domain knowledge, you won't write tests that check for injection vulnerabilities, broken authentication, or insecure defaults. AI will produce code that passes your tests -- and is trivially exploitable.

If you don't have performance domain knowledge, you won't write tests that check behavior under load, memory growth over time, or connection pool exhaustion. AI will produce code that works beautifully in development and falls over in production.

If you don't understand the business domain deeply, you won't write tests that cover the edge cases where real data diverges from the happy path. AI will produce code that handles your examples perfectly and breaks on the first real input that doesn't match the pattern.

The tests can only encode the knowledge you bring to them. AI can't compensate for missing constraints -- it can only optimize within the constraints you provide. Every gap in your tests is a degree of freedom AI is free to exploit.

Tests are how you express judgment in a form AI can be held to. If the judgment isn't there, the tests can't encode it, and AI is unconstrained in exactly the dimensions where constraint matters most.

The Feedback Loop

The practical structure is simple:

1. Define what you want (requires domain expertise)
2. Express it as tests (requires testing skill)
3. Let AI iterate against those tests
4. Verify that AI satisfied the intent, not just the letter

Step 4 is where most people stop paying attention. AI passed the tests. Done. But "passed the tests" only means "satisfied the constraints you thought to encode." It doesn't mean "solved the problem correctly in all the dimensions you didn't test for."

The feedback loop is powerful -- AI can iterate rapidly toward a well-defined target. But the loop amplifies whatever you put into it. Good constraints produce good solutions fast. Bad constraints produce bad solutions fast. Missing constraints produce solutions that look good until they encounter the dimension you forgot to constrain.

When the Loop Isn't Enough

Some things resist automated verification. Architectural coherence. Maintainability. Whether the solution actually fits the system it's being added to. Whether the approach will survive the next three changes that are coming.

For these, you're back to human judgment. The feedback loop handles the mechanically verifiable. Everything else still requires you to look, think, and decide.

The temptation is to believe that if the tests pass, the work is done. The tests are necessary but not sufficient. They're the floor, not the ceiling. They catch the things you knew to check for. They can't catch the things you didn't.

The discipline is knowing which is which -- and not confusing "the tests pass" with "this is right."

Let the Codebase Guide the AI

Point, Don't Explain

The most effective approach I've found for working with AI on existing codebases: don't explain how things work. Show it.

Instead of telling AI "we use a repository pattern with dependency injection and our services follow this structure," point it to an existing service that does something similar. AI reads the code, picks up the patterns, and follows them.

This works because AI is excellent at pattern matching. Give it an example and it replicates the style, structure, and conventions. Give it an explanation and it interprets your words through its training data -- which may not match your codebase at all.

The Three Pieces

Context: What Are We Building?

Point AI to existing documentation about the project. Architecture docs, READMEs, design documents -- whatever explains what the system does and where the new work fits in.

AI needs the bigger picture. Not just "add a new endpoint" but "add a new endpoint in a system that does X, where this endpoint fits into workflow Y." Without this, AI builds something that works in isolation but doesn't fit the system.

Tests: When Are We Done?

Existing tests serve as a regression safety net -- they ensure AI doesn't break what already works. New tests get written first. Before AI writes any implementation, you define what success looks like.

Examples: How Do We Build It?

This is the key insight. Instead of explaining how to implement something, point AI to existing code that does something similar.

"Look at how the user service handles authentication. Do the same thing for the payment service."

"Follow the pattern in the orders package for how we structure our handlers."

AI reads the existing code and replicates the approach. It picks up code structure, naming conventions, error handling patterns, how dependencies are wired, how tests are structured, the idioms your team uses. You don't have to explain any of this. The code explains itself.

Why This Works

Every codebase develops its own way of doing things. These conventions aren't always documented. They live in the code.

When you explain things in words, AI interprets through its training data. It might implement a "repository pattern" differently than your codebase does. It might handle errors the way Spring does when you're using Express. When you point to existing code, it follows your conventions. Not the internet's.

The biggest risk with AI on existing codebases is introducing a second way to do the same thing. Two patterns for error handling. Two approaches to validation. Pointing to examples keeps everything aligned.

The Compounding Effect

Every piece of well-written code becomes an example for future work. Good patterns propagate -- AI picks them up and replicates them across new features, new services, new modules. The codebase teaches consistency by existing.

But it compounds in both directions. Let AI write sloppy code and merge it, and that sloppy code is now an example too. The next session might point to it. Bad patterns propagate just as easily as good ones.

This is why code review still matters even when AI writes everything. You're not just reviewing for correctness -- you're curating the examples that will guide all future development. Like mentoring a junior developer, you lead by example. Except this junior never internalizes the lessons.

The Permanent Onboarding Problem

A junior developer learns. After you point them to the user service a few times, they internalize the patterns. They stop asking. They know where to look without being told.

AI doesn't. Every session starts from zero. You point it to the same examples, the same documentation, the same patterns -- every time. It never learns where to look on its own. It never internalizes your conventions. You are permanently onboarding it.

The distinction matters: knowledge transfers, but experience doesn't. You can share what you learned -- the patterns, the conventions, the constraints -- but the process of having learned it, the context that makes it stick, remains yours. AI receives the knowledge each session but never has the experience. It can follow the pattern you show it without understanding why that pattern exists, which means it can't adapt when the pattern doesn't quite fit. Every session, you're transferring knowledge to something that will never convert it into understanding.

This is why documentation matters more than you think. If your patterns live only in your head, you're repeating yourself every session. If they live in documents -- architecture guides, project context, what you're building and why -- you just point AI to them and move on.

Every piece of documentation you maintain is a session you don't have to spend re-explaining. The upfront cost pays for itself the third or fourth time you point AI to it instead of typing the same guidance again.

And AI can create this documentation as a byproduct of work. When AI learns something about your codebase during a session -- a pattern it picked up, a convention it figured out, a constraint it ran into -- have it write that down. It's creating a persistent memory for itself. Not learning, not internalizing -- just leaving notes for the next session to pick up.

When This Doesn't Work

This approach assumes you have existing code worth following.

If your codebase is a mess, AI will replicate the mess. If your patterns are inconsistent, AI will pick up the inconsistency. You're telling it "do it like we do it here," and if how you do it here is bad, you'll get more bad code faster.

It also assumes the existing code is relevant. If you're building something genuinely new -- something with no analog in the codebase -- you don't have examples to point to. You're back to explaining.

But most work on existing codebases isn't genuinely new. It's "another service like the ones we have." It's "a new feature that follows the same patterns." For that kind of work -- the majority of it -- the codebase itself is the best guide you can give AI.

When Nobody Decides It's Fine

Last month, an operator on my team installed the wrong TLS certificate onto a production resource. A single customer lost access for several hours. The operator made no mistake -- they followed the standard operating procedure exactly as written, ran the command it specified, and the system accepted the input without complaint. The SOP was stale. The underlying code had changed months earlier, but the procedure still called for manually specified parameters that no longer matched. The system was designed to accept arbitrary input, so it did exactly what it was told, even when what it was told was wrong.

I extended the system a year ago to support additional functionality. I didn't know the SOP existed -- it was on a hidden subpage, not linked from where all other procedures live. From my perspective, the system was only ever invoked through automation, which I had updated correctly. A legacy manual path predating my time on the team also called the same system with hand-typed parameters. When I changed the code, the SOP became stale, and no mechanism existed to surface that coupling.

Vaughan (1996) calls this structural secrecy: information exists in the system but is architecturally hidden from those who need it. The stale SOP, the silent default for the missing input, the absent tooling, the original design shortcut -- these are latent failures (Reason, 1990), present in the system but dormant until the right combination of conditions surfaces them.

The Fix Was Fast. The Problem Was Slow.

When the incident happened, I was paged that night as backup. Because I knew the system inside out, I traced the issue within minutes, determined the mitigation, and handed it to the primary operator to execute while I worked on the permanent fixes. Within two hours, I had updated the SOP, changed the system to read directly from the authoritative database instead of accepting arbitrary input, built a tool that performs the operation automatically, and updated the SOP again to reference the tool. Each step removed more human involvement from the error-prone part of the process. The final state: even if someone runs the raw command, the system itself no longer accepts arbitrary input. The failure mode is designed out, not trained out.

I didn't need to brainstorm or wait for consensus. I had been advocating against arbitrary input in our SOPs for three years. I had the expertise, and AI-assisted development collapsed what would have been days of implementation into hours. Klein and colleagues found that in 80% or more of cases, professionals with expertise make critical decisions "based on meaningful situational pattern recognition grounded in experience -- rather than by conscious deliberation" (as cited in Patterson, 2017, p. 105). Three years of thinking about the class of vulnerability meant I recognized this instance immediately and already knew the fix.

But the fix being fast doesn't mean the problem was simple. It means the analytical work had already been done -- over three years of thinking about it, plus six months of building the certificate management system itself. The two-hour resolution was three-years-and-two-hours fast.

Why Nobody Fixed It Earlier

I had been flagging this exact class of vulnerability for over three years. Our SOPs routinely called for operators to run raw commands with arbitrary parameters against production infrastructure. I documented the risk, proposed solutions -- tooling to eliminate manual input, system changes to read from authoritative sources -- and raised it repeatedly. The response was always the same: acknowledged, deprioritized, deferred.

The issue never became the squeaky wheel because it had never been attributed to a visible failure -- though it had caused plenty. The causal chain was never traced back to arbitrary input. This is the availability heuristic operating at the organizational level: we judge the probability of an event by how easy it is to think of examples (Goldstein, 2019). A risk that has never been named produces no examples to recall, so it feels improbable.

Vaughan (1996) calls the broader pattern the normalization of deviance. When a risky practice repeatedly does not cause harm, it becomes redefined as acceptable. Each successful execution reinforces the belief that it's safe. Once a configuration and "its known lack of perfection" is accepted by higher levels, it "tends to be a little bit of an umbrella for subsequent decisions" (NASA engineer, as quoted in Vaughan, 1996, p. 112). No one decided this was fine. It just never broke, so it stayed.

Rasmussen (1997) provides the structural explanation. Systems operate within boundaries -- economic failure, unacceptable workload, and safety. Two gradients constantly push the system: management pressure toward efficiency and the effort gradient where people naturally find the easiest path. Wickens (2014) formalizes this as a decision tree: one path is safe but effortful (build the tooling), the other is risky but low-effort (defer). When the risk is abstract and the effort is concrete, the low-effort path wins.

My three years of warnings were attempts to create what Rasmussen calls a counter-gradient -- pressure back from the boundary. But counter-gradients only work as continuous force. The moment you stop pushing, the drift resumes. Rasmussen argues that "the most promising general approach to improved risk management appears to be an explicit identification of the boundaries of safe operation together with efforts to make these boundaries visible to the actors" (p. 192). That's what I tried to do with documents and proposals. It didn't work until the boundary was crossed.

The Humans Were the Safety System

Cook (2000) writes that "failure free operations are the result of activities of people who work to keep the system within the boundaries of tolerable performance." The safety of running arbitrary commands came not from the commands being correct, but from operators exercising judgment -- adapting the arguments on the fly when something seemed wrong, then updating the SOP with whatever worked. Cook calls practitioners "the adaptable element of complex systems," and this adaptability is recognition-primed decision-making (Klein, 1998) in action: experienced operators pattern-match against prior encounters and adjust.

The same flexibility that made the system dangerous also made it survivable -- until this incident, where the failure was invisible at the point of execution. The input was not wrong; it was incomplete. The system accepted it silently, defaulting a missing parameter rather than flagging it. The operator had no signal that anything was missing, and no opportunity to exercise judgment. In Rasmussen's terms, this is an invisible boundary: the system was already in a degraded state, but nothing made the proximity to failure visible to the actors.

The Investigation Almost Failed Too

When the incident happened, the blameless investigation almost stopped too shallow. A post-incident analysis that identifies "system accepted arbitrary input" as the root cause produces a narrow fix -- hardening that one interface -- while the deeper question goes unasked: why the organization normalized running arbitrary commands against production systems in the first place. This is what Dekker (2017) calls the old view of human error: find the broken component, replace it, declare the system safe. Vaughan (1996) termed the broader pattern the normalization of deviance: when a risky practice repeatedly does not cause harm -- or when the harm it causes is attributed to other factors -- it becomes redefined as acceptable, and the systemic conditions that made it unremarkable persist unchallenged. This requires the kind of systemic thinking that Cook (2000) argues is essential but that organizational pressure toward rapid closure discourages. When investigations stop at the proximate cause, the systemic condition persists, and the same class of failure recurs under different surface circumstances. The system is designed to produce closure, not understanding -- and the design of the investigation process itself becomes a latent failure.

I pushed the investigation to go deeper. The broader finding was initially accepted -- then narrowed, then dropped entirely because it felt too broad. The organization's own investigation process reproduced the pattern it was supposed to interrupt.

When Operators Default to Rules

Butler et al. (2021) found that firefighters were less likely to exercise operational discretion -- departing from SOPs -- in high-stress emergency scenarios than in routine ones, even when the emergency was precisely the condition that licensed discretion. They suggest this reflects acute stress shifting cognition toward rule-based processing.

I see the same pattern in DevOps, but I'm not convinced stress is the primary mechanism. What Butler's high-stress scenarios also share is unfamiliarity -- emergencies are by definition non-routine, meaning the operator has fewer prior encounters to draw on. Their scenarios confound these variables: a high-rise fire falls within the domain firefighters are trained for; children in a sinkhole does not. Stress and unfamiliarity co-vary in their design.

In my experience, operators default to SOPs in unfamiliar situations not because stress degrades their capacity to think, but because they don't trust their own judgment without the pattern library to recognize what they're looking at. Staal (2004) describes the resulting vicious circle: perceived poor performance produces an emotional response that further decreases performance, deepening the perception. The fix isn't stress inoculation -- it's building the pattern library through varied exposure so the operator recognizes the situation and trusts their judgment when it matters.

The Effort Equation Has Changed

The first investigation on a similar issue, five years ago, improved the SOP but stopped short. It replaced a fully manual process with an automated system that still accepted arbitrary input. The investigation stopped at the proximate cause -- the stale procedure -- while the systemic condition persisted unchallenged. The deeper question was invisible because the practice had been normalized: arbitrary input had never been named as the problem, so it produced no examples to recall, and a risk that produces no examples feels improbable.

The system could have been fully automated from the start, but the cost made manual input the accepted shortcut -- Wickens's decision tree in action. What has changed is that AI-assisted development has made full automation the low-effort option. The gradient that once pushed toward manual shortcuts now pushes toward proper tooling -- but only because AI-assisted development recently made full automation the low-effort option.

But only for those whose mental model of effort has updated. When decision-makers still calibrate against the old cost -- weeks of implementation, not hours -- a two-hour fix looks suspicious rather than correct. The speed itself becomes grounds for skepticism, and the gradient that should now push toward safety gets blocked by an outdated intuition about how long good work takes.

Cook (2000) reminds us that "complex systems run in degraded mode" and "catastrophe is always just around the corner." The system doesn't need to be visibly broken to be one perturbation away from failure. A tool that encapsulates the operation would survive any code changes -- the SOP becomes frozen in time, immune to drift. Better still, the system itself could read the data it needs directly, eliminating the manual procedure entirely.

The absence of failure in a system drifting toward its boundary is not evidence of safety. It is evidence that the boundary has not yet been crossed.

References

Butler, P. C., Honey, R. C., & Cohen-Hatton, S. R. (2021). Decision making within and outside standard operating procedures: Paradoxical use of operational discretion in firefighters. Human Factors, 63(8), 1378--1393. https://doi.org/10.1177/00187208211041860

Cook, R. I. (2000). How complex systems fail. Cognitive Technologies Laboratory, University of Chicago. https://how.complexsystems.fail

Dekker, S. (2017). The field guide to understanding human error (3rd ed.). CRC Press.

Goldstein, E. B. (2019). Cognitive psychology: Connecting mind, research, and everyday experience (5th ed.). Cengage Learning.

Klein, G. (1998). Sources of power: How people make decisions. MIT Press.

Patterson, R. E. (2017). Intuitive cognition and models of human-automation interaction. Human Factors, 59(1), 101--115. https://doi.org/10.1177/0018720816659796

Rasmussen, J. (1997). Risk management in a dynamic society: A modelling problem. Safety Science, 27(2), 183--213. https://doi.org/10.1016/S0925-7535(97)00052-0

Reason, J. (1990). Human error. Cambridge University Press. https://doi.org/10.1017/CBO9781139062367

Staal, M. A. (2004). Stress, cognition, and human performance: A literature review and conceptual framework (NASA/TM-2004-212824). National Aeronautics and Space Administration.

Vaughan, D. (1996). The Challenger launch decision: Risky technology, culture, and deviance at NASA. University of Chicago Press.

Wickens, C. D. (2014). Effort in human factors performance and decision making. Human Factors, 56(8), 1329--1336. https://doi.org/10.1177/0018720814558419

When the Automation Fails

The Irony

We automate things because humans make mistakes. Then the automation removes the practice that kept humans sharp. When the automation eventually fails -- and it always does -- the humans are less prepared than they were before we automated.

This isn't a new observation. In 1951, Paul Fitts proposed a simple framework for dividing work between humans and machines: list what each is good at, and assign tasks accordingly. Humans are good at pattern recognition, flexible reasoning, and improvisation. Machines are good at repetitive computation, sustained monitoring, and consistent execution. Give each what they do best (Fitts, 1951).

This became known as the Fitts' list, and it shaped how systems were designed for decades. It's intuitive. It's also incomplete.

In 1983, Lisanne Bainbridge published a short paper called "Ironies of Automation." She showed what Fitts' list misses: the designer automates the routine tasks and leaves the operator responsible for the tasks that couldn't be automated -- the complex, ambiguous, and unpredictable ones. Simultaneously, the operator's skills degrade from disuse. When automation fails and those skills are needed most, the operator is least prepared to use them (Bainbridge, 1983).

She noted that the knowledge needed to operate manually "develops only through use and feedback about its effectiveness" and that automation can "camouflage system failure by controlling against the variable changes, so that trends do not become apparent until they are beyond control." The automation doesn't just replace the operator's actions -- it hides the information the operator would need to develop and maintain the skills to work without it.

Thirty-four years later, Strauch (2018) revisited Bainbridge's ironies and found them entirely unresolved. Identical operator errors recurred across decades despite industry-wide awareness of the problem.

What This Looks Like in Practice

Think about a diagnostic tool that collects information from fifteen different sources, correlates the data, and presents a conclusion. The operator runs the tool, reads the output, acts on it. The automation calls the same tool on a schedule. Everything works.

Then the tool breaks. Or the situation is novel enough that the tool's correlation doesn't produce a useful answer. Now the operator needs to do what the tool was doing: go to those fifteen sources, pull the data, interpret the raw output, and reason about what it means.

Do they know what those fifteen sources are? Do they know how to read the raw data without the tool's presentation layer? Do they understand why the tool correlates certain signals together?

The tool handled the how. Over time, the operator loses the why. They can run every tool in the toolkit but can't reason about the system when the tools don't give them an answer.

The most well-known illustration comes from aviation. When Air France 447's autopilot disconnected due to iced pitot tubes, the pilots lost reliable airspeed readings. The instruments gave contradictory data. The problem wasn't that the pilots couldn't fly manually -- it was that they couldn't diagnose what was happening when the instruments they relied on were unreliable. Oliver et al. (2017) found that the automation that made the system safe under normal conditions had eroded the crew's capacity to handle disturbances -- the "paradox of almost totally safe systems": the rarer the failure, the less prepared operators are to handle it.

When There's No Tool at All

Bainbridge's irony assumes the operator is doing the same job the automation did, just less well. But there's a worse case: when the automation fails and there's no proper tool for the manual path at all.

On January 31, 2017, a GitLab engineer was troubleshooting a database replication failure. After hours of unsuccessful debugging past midnight, they ran rm -rf on a database directory, intending to wipe the secondary database. They were on the primary. Around 300 GB of production data was deleted in seconds (GitLab, 2017).

The command was executed correctly. The engineer did what they intended to do. But there was no tool for this operation -- just a terminal and a procedure. No confirmation step, no restricted access, no guardrails. The terminal prompts barely distinguished between the primary and secondary servers.

I've seen the same pattern in my own work. An automated maintenance procedure was paused because the automation would restart a resource the customer couldn't tolerate restarting. The maintenance itself didn't require a restart -- but the automation bundled them together. So an operator had to perform the maintenance manually. No dedicated tool existed. The procedure hadn't been updated when the code changed -- it was buried on a sub-page, while all procedures were supposed to live on a single searchable page. It was only reachable through a link, so nobody found it when the code was updated. The operator executed it exactly as written and still caused an issue, because the procedure and the system had diverged.

The operator did nothing wrong. The automation was the tool, and when it couldn't run, there was nothing purpose-built to fall back to.

This is a different problem from Bainbridge's skill erosion, but it compounds it. The operator's skills have atrophied because the automation was handling the work. And the manual path they're forced onto doesn't even have proper tooling -- just raw commands and outdated documents.

What You Lose When People Leave

Both problems compound with a third: knowledge loss through turnover.

I'm experiencing this right now. I'm moving to a new team. I've done my best to transfer what I know -- written documentation, walkthroughs, recorded decisions and their rationale. But I know there are things I'm taking with me that I couldn't fully write down. The intuition for when a metric looks "off" before it triggers an alert. The knowledge of which service is the actual bottleneck during a specific traffic pattern. The memory of why we made a particular design decision three years ago, and what we tried first that didn't work.

Levallet and Chan (2018) found that implicit expert knowledge -- the kind that is most difficult to codify and most critical during non-routine situations -- is the most vulnerable to loss when someone leaves. Joining a new team means facing the flip side: the things everyone knows but nobody has written down, the failure modes that live only in the memories of the people who experienced them.

Hoffman (2008) found that what we call "tacit" knowledge is better characterized as "inert" knowledge: it can be articulated with the right scaffolding and prompting, but it's accessed only in specific contexts. The knowledge isn't untransferable -- organizations just rarely invest in the structured processes that would make transfer possible.

The person who understood why the diagnostic tool checked those specific sources -- which signals actually mattered during a particular class of incident, what the raw data meant in context, which correlations were reliable and which were coincidental -- that person just left the team. The code tells you what the tool does. It doesn't tell you why it was built that way.

The System Looks Fine Until It Doesn't

Cook (2000) ties all of this together, but his argument goes further than "complex systems are degraded." His central point is that the reliability we observe in complex systems is not a property of the system itself. It is produced, continuously, by the people operating it.

Operators notice things that monitoring misses. They work around known flaws. They compensate for design shortcomings that were never fixed. They carry context about the system's actual behavior -- not the behavior described in the architecture documents, but the real behavior, with all its edge cases and failure modes and things that only go wrong on the third Tuesday after a deployment. The system looks like it's working because the operators are making it work. The safety is their output, not the system's attribute.

This is what makes the four problems described in this article so dangerous. Each one is a way that load-bearing capacity quietly disappears from the system.

Skill erosion: the operators are still there, but less capable. The automation handled the routine work, and the routine work was where they built and maintained the understanding needed for the non-routine work. They're present but diminished.

Missing tooling: the operators are capable but unequipped. They know what needs to happen, but the path to doing it manually is unprotected -- raw commands, stale procedures, no guardrails. The automation was the tool, and now it's gone.

Knowledge loss: the operators are gone entirely. The person who understood why the system behaved a certain way, who remembered what was tried before and why it failed, who could look at a dashboard and see what it wasn't showing -- that person moved to another team, or left the company, or retired. Their knowledge left with them.

Inexperience: the new operators never had the skills to begin with. They arrived after the automation was already in place. The environment that would have built their understanding -- the manual work, the raw data, the feedback from getting it wrong and learning why -- no longer exists. The tenured operators developed their intuition through years of hands-on practice before the system was automated. The new ones have no equivalent path. Omron et al. (2018) showed that even in medicine, clinical experience alone cannot build expertise for rare events because the feedback loop is broken -- clinicians almost never learn about the cases they miss. Simulation-based training with deliberate practice was proposed as the alternative, compressing the learning that real-world experience cannot provide. The same applies here: when automation handles everything, the only way to build the skills needed for when it doesn't is through simulation. Most organizations don't invest in it.

In all four cases, nothing visibly changes. The metrics were never measuring what the humans were contributing. They were measuring the output of a system that included the humans and attributing all of it to the automation. The dashboards show the result of human adaptation and call it system performance.

So the safety margin narrows. The automation is handling everything. The dashboards are green. The people who were compensating for the system's flaws are less skilled, or less equipped, or gone -- or never learned how in the first place. And nobody notices, because the system's apparent reliability -- the only kind anyone measures -- remains unblemished.

Right up until the moment it isn't.

References

Bainbridge, L. (1983). Ironies of automation. Automatica, 19(6), 775--779. https://doi.org/10.1016/0005-1098(83)90046-8

Cook, R. I. (2000). How complex systems fail. Cognitive Technologies Laboratory, University of Chicago. https://how.complexsystems.fail/

Fitts, P. M. (Ed.). (1951). Human engineering for an effective air-navigation and traffic-control system. National Research Council.

GitLab. (2017, February 10). Postmortem of database outage of January 31. https://about.gitlab.com/2017/02/10/postmortem-of-database-outage-of-january-31/

Hoffman, R. R. (2008). Human factors contributions to knowledge elicitation. Human Factors, 50(3), 481--488. https://doi.org/10.1518/001872008X312152

Levallet, N., & Chan, Y. E. (2018). Knowledge loss and retention: The paradox of organizational forgetting. Journal of Knowledge Management, 23(1), 1--22. https://doi.org/10.1108/JKM-08-2017-0358

Oliver, N., Calvard, T., & Potočnik, K. (2017). Cognition, technology, and organizational limits: Lessons from the Air France 447 disaster. Organization Science, 28(4), 729--743. https://doi.org/10.1287/orsc.2017.1138

Omron, R., Kotwal, S., Garibaldi, B. T., & Newman-Toker, D. E. (2018). The diagnostic performance feedback "calibration gap": Why clinical experience alone is not enough to prevent serious diagnostic errors. AEM Education and Training, 2(4), 339--342. https://doi.org/10.1002/aet2.10119

Strauch, B. (2018). Ironies of automation: Still unresolved after all these years. IEEE Transactions on Human-Machine Systems, 48(5), 419--433. https://doi.org/10.1109/THMS.2017.2732506

Build the Tools

Most AI-assisted operations today work like this: an agent with access to a terminal runs commands, reads output, reasons about what it sees, and takes action. It queries databases directly. It reads raw logs. It correlates signals by pulling data from multiple sources and interpreting it in context. When it works, it looks impressive.

We wouldn't let a new hire do this. We'd give them purpose-built tools with guardrails, scope their access, and build confirmation steps around the things that can cause irreversible damage. We do this because operating directly on production systems with raw commands is dangerous, and the danger increases when the operator lacks context about the system they're touching.

An AI agent has less context than a new hire. It has whatever fits in its window. It doesn't carry institutional memory. It doesn't remember the last time this metric behaved this way, or what actually caused it, or why this particular service behaves differently on Tuesdays. Yet we give it broader access than we'd give a person, because it's fast and because it doesn't complain.

What the AI Can't See

The knowledge that makes experienced operators valuable is largely tacit -- personal, context-specific, and difficult to formalize (Nonaka & Takeuchi, 1995). It transfers between people through shared experience: a junior operator shadows a senior one during an incident and absorbs not just the commands but the pacing, the priorities, the instinct for what to check first. Hoffman (2008) found this knowledge isn't untransferable -- it's "inert," accessible only in specific contexts, articulable with the right scaffolding. But it requires presence and participation.

AI skips this entirely. It has access only to what was written down -- the documentation, the runbooks, the code. Everything that remained tacit because the organization never invested in surfacing it is invisible to the agent. The AI operates on the explicit layer of a system whose reliability depends on the tacit layer.

This is why AI-as-operator is weaker than it appears. It can process everything that was externalized. The things that weren't externalized -- the awareness that something looks wrong before you can articulate why, the memory of what was tried before and failed, the understanding of which correlations are meaningful and which are coincidental -- those are exactly what make experienced operators load-bearing. Giving AI broad terminal access and asking it to investigate production issues puts it in the wrong role. Running commands against production without guardrails is dangerous regardless of who or what is doing it. We solved this for humans by building tools. The answer for AI is the same.

Working Through AI

The right role for AI in operations isn't faster operator. It's the layer that captures knowledge, enforces guardrails, and builds the interfaces that make dangerous operations safe.

When a human investigates with raw commands, the knowledge of what they checked and why lives in their head. When they leave the team, it leaves with them. When an AI investigates in its default mode -- session starts, commands run, answer produced, session ends -- the knowledge disappears with the context window. Neither produces anything durable by default.

But AI doesn't have to work this way. When you route all operations through AI rather than running commands yourself, the AI becomes the note-taker for everything that happens. It captures what was checked, in what order, what the output meant, and why a particular action was taken. From there it can store that context for the next session, build a tool that encodes the process, produce documentation, prepare a handoff -- whatever makes the knowledge persist. This is the real shift: not "let AI do the work" but "never let work happen without AI capturing it." Every operation becomes an opportunity to externalize knowledge that would otherwise stay ephemeral.

The experienced operator provides the knowledge of what matters and why. AI provides the implementation and the memory. In Nonaka and Takeuchi's terms, this is externalization -- converting tacit operational knowledge into an explicit, reusable form. The operator who knows which fifteen sources matter and why certain correlations are meaningful can't easily write that down in a runbook. But working through AI, that knowledge gets captured as it's applied -- in tools, in stored context, in documentation that writes itself as a byproduct of the work.

The Discoverability Problem

There's a failure mode here worth naming. You end up with so many tools that nobody knows what's available. We had a tool to check the metering status of a resource. Nobody knew it existed. When the question came up, the assumption was that we simply couldn't do that -- the capability was functionally absent because the knowledge of its existence didn't survive.

Not every operation needs a tool, either. Some problems are genuinely one-off. Building a tool for every edge case creates noise that makes the important tools harder to find. The solution isn't accumulation -- it's curation. Each tool needs to document why it exists, what it does, and when to use it. The toolset itself needs to be navigable. AI can help with discovery -- searching for relevant tools, suggesting them in context -- but the problem is real and not fully solved. A tool nobody can find is the same as no tool at all.

Compartmentalization, Not Omniscience

None of this requires giving AI unrestricted access to everything. In fact, it requires the opposite.

You wouldn't give the person who refills the coffee machine full access to your corporate email, production servers, and building security systems. That's not because they're untrustworthy -- it's because the access isn't relevant to their role, and unnecessary access creates unnecessary risk. This is basic compartmentalization. Every organization understands it for humans.

Yet with AI, people routinely hand over unrestricted shell access and say "go do your thing." Then they write think pieces about doomsday scenarios. The safety problem isn't AI. It's the access model.

The same compartmentalization principles apply. An AI agent investigating a database issue needs read access to that database's metrics and logs. It doesn't need write access to production. It doesn't need access to unrelated services. It doesn't need root. Scope the access to the task. Build confirmation steps around irreversible operations. Restrict dangerous commands behind purpose-built interfaces that validate inputs before executing.

This is what "build the tools" actually means. Not that every operation needs a bespoke CLI tool. It means that dangerous operations should be wrapped in interfaces that enforce guardrails -- and that those interfaces should be what AI calls, not raw commands. The same way you wouldn't let a human rm -rf a database directory without a confirmation step, you don't let an AI do it either. You build the interface once, and everything -- humans, automation, AI -- goes through it.

Scope the access. Capture the work. Build the interfaces. The actor changes. The principle doesn't.

References

Hoffman, R. R. (2008). Human factors contributions to knowledge elicitation. Human Factors, 50(3), 481--488. https://doi.org/10.1518/001872008X312152

Nonaka, I., & Takeuchi, H. (1995). The knowledge-creating company: How Japanese companies create the dynamics of innovation. Oxford University Press.

The Gap-Filling Problem

The Issue

AI says things you're not saying.

You give it a thought, and it extends it. You leave space to think, and it fills that space. You pause, and it continues.

Once AI fills a gap, it's hard for you to fill it yourself.

That sentence is the observation. The rest of this article is about why it's hard -- not just practically, but cognitively. There's research that may explain what's happening in your head when AI fills a gap, and why knowing about the problem doesn't protect you from it.

Why This Matters

Human thinking works in chunks. We think, pause, think more. The pauses aren't empty -- they're where the next thought forms. The gaps aren't missing content -- they're space for ideas to develop.

When AI fills those gaps, it replaces your thinking with its predictions. It says things you weren't going to say. It takes the thought in directions you didn't intend.

This isn't just an inconvenience. It may be a cognitive trap.

What's Happening in Your Head

Once a solution occupies the slot where your thinking was going to go, your brain reorganizes around it. Your attention shifts -- unconsciously -- toward information that supports the solution you've been given, and away from alternatives. You don't just lose the gap. You lose awareness that there was a better thought trying to form there.

Psychologists call this the Einstellung effect -- a known solution preempts the search for better ones, even when the person believes they're still looking (Bilalić et al., 2008).

The evidence is striking. In chess experiments, expert players were shown positions where a familiar move worked but a better move existed. Using eye-tracking, researchers found that players who reported looking for a better solution were measurably not looking at the board regions where the better solution was. Their attention was captured by the first idea -- the familiar one -- and redirected without their awareness (Bilalić et al., 2010).

They believed they were still searching. They weren't.

Why This Applies to AI

Classic Einstellung requires a familiar solution to activate the trap. You need to already know the obvious approach for it to block the better one. AI changes the precondition. The solution doesn't need to be familiar to you -- it just needs to be plausible and present.

AI can install the familiar solution in real time, in a single interaction, before you've had a chance to form your own approach. It fills the gap with something reasonable, and your brain treats it as if you'd thought of it yourself.

This connects to several overlapping phenomena that different research traditions have studied under different names:

• Design fixation -- exposure to an example solution constrains the design space, even when you're told to ignore it (Jansson & Smith, 1991). Designers shown an example produce solutions that look like the example, whether they want to or not.
• Anchoring -- a salient initial value or framing biases subsequent judgment, and adjustment from the anchor is systematically insufficient (Navarre et al., 2021). AI doesn't just fill the gap -- it anchors your framing of the problem.
• Automation bias -- humans over-weight machine-generated outputs relative to their actual reliability, reducing critical engagement (Parasuraman & Manzey, 2010). The fact that AI said it adds weight you didn't consciously assign.

These aren't separate problems. They share a common mechanism -- an externally activated representation captures attention and suppresses alternative search. Different research traditions developed independent vocabularies for what may be the same underlying process.

Why Knowing Doesn't Help

Here's the part that matters most for practitioners: awareness alone is insufficient protection.

Research has shown that warning participants that AI assistants were biased did not reduce the magnitude of the influence -- even when people knew the AI was biased, they were still affected by what it said (Zheng et al., 2025). Designers who study and teach design do not reliably know when they are being fixated (Linsey et al., 2010). The chess experts believed they were searching for better moves. They weren't.

The mechanism operates below conscious awareness. It's not about intelligence or willpower. It's structural. It happens to experts. Knowing about it is a start, but it's not a solution.

This is why "just think critically about AI output" is weaker advice than it sounds. You can't think critically about alternatives your attention has already been directed away from.

An Example

I said: "The main issue I have with writing using AI is that it says things I'm not saying."

AI responded with:

• Examples about code (I didn't mention code)
• Assumptions about auto-completion (I didn't say that)
• An entire framework of strategies and solutions
• Scenarios I never implied

AI took one observation and buried it under assumptions about what I meant.

Now I have to evaluate all of that. And the moment I start evaluating it, my attention is organized around what AI said -- not around what I was going to say. The gap has been filled. My original thought has to compete with AI's version for space in my own head.

That's the gap-filling problem. AI doesn't just complete your thoughts -- it captures your attention.

The Cost

Every time AI fills a gap, you have to:

• Evaluate what it generated
• Decide if it matches your intent
• Remember what you were actually going to say
• Either accept, modify, or delete its output

That's cognitive load. That's interruption. That's AI inserting itself into your thinking process.

But the deeper cost is the one you don't notice: the alternatives you never considered because your attention was already captured. The design you didn't explore because AI gave you a plausible one. The framing you didn't question because it was anchored before you started.

The Deeper Issue

AI is better at completing than creating.

It can extend your thoughts, but it can't originate them. It can fill your gaps, but it can't know what you intended to put there.

The gaps are where your thinking happens. When AI fills them, it's not helping you think -- it's thinking for you. And the research suggests that once a solution is present -- whether it came from your own memory or was externally supplied -- the same attentional capture mechanism locks in.

The technology didn't create new cognitive phenomena. It created a new environment that reliably triggers an unfortunate cluster of existing ones together.

What To Do

Be deliberate about when you engage AI:

1. Think first -- Develop your thoughts before asking AI for help. This isn't just good practice -- it's a structural defense. If your own framing is already formed, AI's output has to compete with it rather than filling an empty slot.
2. Complete your structure -- Finish your outline before filling details. The research on design fixation suggests that having your own representation in place before encountering an external one reduces (though doesn't eliminate) fixation effects.
3. Use AI for refinement -- Let AI help polish what you've already created. The danger is in the generative phase, not the editorial phase.
4. Step away before evaluating -- Incubation -- stepping away from the problem entirely -- is one of the better-supported interventions against fixation (Wang et al., 2023). If AI fills a gap, don't evaluate it immediately. Walk away. Come back with fresh attention.
5. Recognize interference -- Notice when AI is saying things you're not saying. This won't fully protect you -- the research is clear that metacognitive awareness alone is insufficient -- but conflict detection is available as a cognitive resource. Noticing the conflict is the first step toward switching from automatic to deliberate processing.

The gaps are yours. Fill them when you're ready, not when AI thinks you should.

References

Bilalić, M., McLeod, P., & Gobet, F. (2008). Why good thoughts block better ones: The mechanism of the Einstellung (set) effect. Cognition, 108(3), 652--661. https://doi.org/10.1016/j.cognition.2008.05.005

Bilalić, M., McLeod, P., & Gobet, F. (2010). The mechanism of the Einstellung (set) effect: A pervasive source of cognitive bias. Current Directions in Psychological Science, 19(2), 111--115. https://doi.org/10.1177/0963721410363571

Jansson, D. G., & Smith, S. M. (1991). Design fixation. Design Studies, 12(1), 3--11. https://doi.org/10.1016/0142-694X(91)90003-F

Linsey, J. S., Tseng, I., Fu, K., Cagan, J., Wood, K. L., & Schunn, C. (2010). A study of design fixation, its mitigation and perception in engineering design faculty. Journal of Mechanical Design, 132(4), 041003. https://doi.org/10.1115/1.4001110

Navarre, L., Chauveau, V., & Gauvrit, N. (2021). Are the anchoring effect and the Einstellung effect two facets of the same phenomenon? New Ideas in Psychology, 64, 100912. https://doi.org/10.1016/j.newideapsych.2021.100912

Parasuraman, R., & Manzey, D. H. (2010). Complacency and bias in human use of automation: An attentional integration. Human Factors, 52(3), 381--410. https://doi.org/10.1177/0018720810376055

Wang, Y., Xia, T., & Yao, S. (2023). Alleviating design fixation with Alternative Uses Task. International Journal of Technology and Design Education. https://doi.org/10.1007/s10798-023-09824-2

Zheng, M., Liao, Q. V., & Yang, Q. (2025). Effect of AI helpfulness and uncertainty on cognitive interactions with pharmacists. Journal of Medical Internet Research, 27, e59946. https://doi.org/10.2196/59946