The Five Mistakes Every Forward Deployed AI Engagement Makes
Operational wisdom from running engagements yourself, watching other operators run theirs, and seeing in-house teams attempt them. These five mistakes show up over and over.
I have run Forward Deployed AI engagements for B2B SaaS companies. I have talked extensively with operators running them at Frontier Labs and at independent shops. I have watched in-house engineering teams attempt the same shape of work without the methodology and produce some of the most expensive failure modes I have seen in software. By the end of this article, you’ll know the five mistakes that show up over and over in this work, and how to avoid each one.
These mistakes aren’t theoretical. I have made some of them myself. The list below is what I’d put on a checklist for anyone, operator or in-house team, kicking off an AI workflow project.
Mistake one: confusing “AI feature” with “AI workflow”
The first mistake is linguistic, and it is the most consequential.
“Feature” implies a thing you bolt on: a button, a tab, a screen, a capability the product has. “Workflow” implies a process a user runs: input, action, output, with a specific person at each end.
This distinction sounds semantic. It isn’t. Teams that scope AI work as features end up with prototypes that technically work but don’t get adopted, because they were never tied to a specific user’s specific process. Teams that scope AI work as workflows end up with shipped, used software, because the scoping itself forced the team to name the user and the process before any code was written.
The diagnostic question: Can you fill in the brackets? “For [user], when [trigger] happens, the workflow will [action] and produce [output], so that [business value].” If you can’t fill all five brackets, you don’t have a workflow yet. You have a feature idea.
I cannot count the number of in-house AI projects that died because the team scoped a feature and never converted it to a workflow. The team meant well. The technology worked. The user, however, never existed.
Mistake two: scoping vaguely, hoping a definition emerges from the work
The second mistake is the consequence of the first.
When the work is scoped vaguely (”an AI feature for customer success” or “an AI assistant in the dashboard”), teams sometimes believe that definition will emerge from the engineering. They will start building, they will see what works, and they will iterate toward the right thing.
That never happens. What actually happens is that the team builds something that technically functions but has no specific success criterion, so nobody can say whether it is done. Engineering slips. The PM gets nervous. The CEO asks why this isn’t shipping yet. Six months later, the prototype is in a Notion page, and nobody is working on it.
The fix is to scope on paper before scoping in code. The scope document, call it a brief or a one-pager or whatever, has to name the user, the trigger, the input, the output, the success criterion, and the explicit out-of-scope items. It has to be signed off on by a decision-maker before engineering starts. Not approved verbally; signed off in writing, in an email, with the decision-maker’s name attached.
This is the discipline that separates engagements that ship from engagements that drift. The operators I know who consistently ship are religious about the written brief. The ones who don’t ship are casual about it.
The acceptance criterion specifically has to be concrete. “Make it work” is not a criterion. “Reduce average prep time from 90 minutes to under 15 minutes” is. “Classify 80% or more of sample tickets correctly across a defined sample set” is. The concrete criterion is what makes the end of the engagement unambiguous.
Mistake three: shipping autonomy on day one
The third mistake is reasoning from the end state.
Teams imagine the eventual fully-automated version of the workflow and try to ship it as v1. The customer service AI that auto-replies. The recommendation engine that auto-publishes. The classifier that auto-routes without human review.
These v1s fail in the most expensive way possible. The AI is wrong about 5% of the time; the 5% concentrates at the edges, the edge cases include the high-stakes ones, and the wrong output goes out before anyone catches it. Rollback is messy because customers have already received the wrong outputs.
The right v1 is almost always assistive. The AI drafts; a human reviews, edits, and approves. The user benefits from the AI doing 70% of the work, but doesn’t bear the cost of the AI being wrong, because the human is the checkpoint. Autonomy is added later, gradually, as the eval data accumulates and shows where the AI is reliable enough to remove the checkpoint.
This is the principle that “autonomy is earned, not granted.” Workflows start with maximum human oversight and gain autonomy over time. The teams that skip the assistive phase produce the cautionary-tale deployments. The teams that include the assistive phase ship slowly but durably and end up with autonomous workflows that work because they were tested at every level of autonomy before being promoted.
Mistake four: skipping the eval loop because the prototype “looks fine.”
The fourth mistake is the most seductive.
The prototype runs end-to-end. The output looks reasonable. The PM nods. The CEO approves the demo. Everyone is happy. The team ships without running the eval loop, because the eval loop seems like a formality when the prototype is obviously working.
What the team doesn’t see, what they cannot see without the eval loop, is the distribution of failure modes. The prototype is 95% accurate on the inputs they tested. The 5% they didn’t test concentrates in specific patterns: long inputs, unusual phrasing, missing fields, contradictory information, and low-quality source documents. In production, those edge cases are 20% of real traffic. The prototype that looked 95% accurate is actually 80% accurate, and the 20% failure rate is where the most expensive consequences live.
The eval loop is the discipline that catches this. Take 10 to 20 real inputs (or representative synthetic ones), generate outputs, score them against a rubric dimension by dimension, identify failure patterns, refine, and re-test. Run the loop at least twice: once to validate the prototype, once to refine the production version. Document the cases that the workflow handles badly. Hand off the rubric so the customer’s team can run the loop themselves after the engagement ends.
The teams that skip this step ship workflows that the customer’s team discovers are broken in week three. The teams that don’t skip it ship workflows that the customer’s team can confidently improve over time.
Mistake five: treating handoff as a meeting, not a deliverable
The fifth mistake is the one that determines whether the work survives past the engagement.
The wrong way to do handoff is to schedule a meeting at the end of the engagement, walk through the code, answer questions, and leave. The customer’s team has the code but no documentation, no rubric, no playbook, no roadmap. Three months later, when the AI starts producing weird outputs because the model was updated or because the data shape changed, the customer’s team has no idea how to diagnose or fix it.
The right way to do handoff is to treat it as a deliverable. The handoff package includes the code, brief implementation documentation, a recorded walkthrough video, the eval rubric with worked examples at each score level, a workflow playbook explaining purpose and inputs and outputs, review steps and edge cases, and maintenance, a 30-day post-engagement roadmap with suggested next steps, and a live enablement session with the customer’s engineers.
The rubric and the playbook are the two pieces that separate Forward Deployed work from freelance work. They cost about two hours each to produce well. They make the customer’s team able to maintain and iterate the workflow without you. They also create natural follow-on engagement conversations: “We built the rubric, want us to help maintain it as you ship more workflows?”
The teams I see most disappointed with their AI engagements are the ones where the engagement ended in a meeting rather than a deliverable. They have the code but nothing else. Six months later, the workflow is dead because no one knew how to keep it alive.
The pattern across all five
All five mistakes share a structure. Each one is a place where the team chose speed over discipline, hoping the missing structure wouldn’t matter. Each one produces a specific, predictable failure mode. Each one is preventable, but only if the team is willing to take the structural step that feels slower in the moment.
Confusing feature with workflow is choosing the easy scoping language over the harder definitional one.
Scoping vaguely is choosing to start engineering immediately over writing the brief first.
Shipping autonomy on day one is choosing the end-state product over the assistive v1.
Skipping the eval loop is choosing the demo-looks-fine signal over the systematic evaluation.
Treating handoff as a meeting is choosing the easy end-of-engagement over the deliverable package.
In all five cases, the disciplined path is slightly slower in the engagement and dramatically faster across the customer’s actual relationship with the workflow. The undisciplined path is faster in the engagement and produces failures that consume far more time, money, and reputation across the months that follow.
TLDR
Five mistakes show up in Forward Deployed AI engagements over and over.
One: confusing “AI feature” with “AI workflow.” Features are capabilities; workflows have users. If you can’t fill the brackets (”for [user], when [trigger], the workflow does [action] and produces [output], so that [value]”), you don’t have a workflow.
Two: scoping vaguely and hoping the definition emerges from the engineering. The written brief, signed off by a decision-maker, is the discipline that separates engagements that ship from engagements that drift.
Three: shipping autonomy on day one. The right v1 is almost always assistive. Autonomy is earned over time, not granted at launch.
Four: skipping the eval loop because the prototype looks fine. Failure modes concentrate at the edges. Without the eval loop, the team discovers them in production at maximum cost.
Five: treating handoff as a meeting rather than a deliverable. The handoff package (code, docs, walkthrough, rubric, playbook, roadmap, enablement session) is what determines whether the workflow survives past the engagement.
The pattern across all five: each is a place where the team chose speed over discipline. The disciplined path is slightly slower in the moment and dramatically faster over the customer’s actual relationship with the workflow.
If this topic is useful to you, I go deeper in my book, Forward Deployed AI Engineering: A Working Guide to the Hottest Job in Software.
It’s available now on Amazon: https://www.amazon.com/Forward-Deployed-AI-Engineering-Software/dp/B0H3VKKG38/