How Engineering Uses AI
Rancher Engineeringfor Finance20 min · 10 talk / 10 show
The short version

We took the manual overhead out of engineering.

AI agents now do the slow, repetitive work that used to cap what we could take on — so we ship more, faster, with proof it's correct, including work we used to simply defer.

30+specialized AI tools built
−94%security vulnerabilities cleared
24/7agents working overnight
Numbers shown are a live snapshot captured for this talk. Figures marked verify are still being pulled from our own dashboards.
How we got here — the five steps

Manual → prompt → agent → multi-agent → overnight.

A prompt becomes an agent; one agent becomes a fleet; the fleet runs itself overnight. Each leap moves people further from doing the work and closer to just deciding — here’s what changes.

Manual

You do the whole job by hand.

you: do every step
describe it once
Prompt

You write the instructions once; the AI does the task and hands it back.

you: write the prompt
wrap it in a dashboard
Agent

The prompt becomes a dashboard that runs the whole job and checks itself.

you: press one button
one per job — a fleet
Multi-agent

Many dashboards — a specialized agent for every recurring job (30+).

you: pick the right tool
let them run themselves
Overnight

The agents run on their own, unattended, and report back by morning.

you: just review results
Multi-agent is breadth; overnight is autonomy. Many dashboards — one per job — then they run themselves and report back. Same team, aimed at judgment instead of grunt work.
How we got here · 1 / 5 · the same job: reviewing a code change

By hand.

You open the change and read every file, line by line, until you spot the problem — then you write the comment. Thorough, but it’s hours, on every single change.

hoursyour time
pull request · 6 files changed · reviewing by hand
10export function getUser(id) {
11 const users = store.getUsers()
12 if (!users) return null
13+ const u = users.find(x => x.id == id)
14+ return u.profile.name // u may be undefined
15}
16
17export function listUsers(filter) {
18 return store.getUsers().filter(filter)
19}
R
reviewer commented
Comment
How we got here · 2 / 5

So we wrote it down — a “prompt.”

Instead of reading it all yourself, you write the instructions once. The AI reads the whole change and writes the review back. You read it and decide — minutes.

minutesyou run it
AI assistant · review this change
Assistant
  • bug getUser can dereference undefined when the id isn’t found (line 14).
  • style use strict equality === on line 13.
  • nit listUsers has no guard for a null filter.
How we got here · 3 / 5

Then — one click.

We wrapped that prompt in a dashboard. Paste the change, press one button: it deploys a live preview, takes before/after screenshots, and writes the whole review. Seconds of your time.

secondsone click
code-review console
github.com/rancher/dashboard/pull/18987
Deploy preview
Capture before / after
Write review
Review ready3 findings · preview live
before
after
How we got here · 4 / 5

Then — one for every job.

That one dashboard worked so well we built one for each recurring task. Today it’s a fleet of 30+ specialized tools — each still just seconds of your time, but now covering every job we do again and again.

30+jobs · one click each
the console fleet0 tools
Code review
Vulnerabilities
Flaky tests
Prototype
Extension build
PR re-run
e2e master
Locales
Resolutions
Compatibility
Spend
+ 20 more
How we got here · 5 / 5

Now it runs itself.

No one clicks anything. Overnight, agents pick up every open change, review them all, and the board fills itself in. By morning it’s done — zero effort.

nonehands-off
overnight review board 02:14 · 6 agents running
changeauthorstatus
#18987marceloqueued
#18990codyqueued
#18991richardqueued
#18994neilqueued
#18996edenqueued
reviewed tonight0
The idea in one picture

AI changes what we can say yes to.

Sort any task by how hard it is and how long it takes. Before AI, three of these four boxes got deferred. Now agents do the work and people do the review.

Quick to do
Slow to do
Complexity →
High complexity · quick
Triaging issuesSmall code changesPrototyping
BeforeDone as lower priority, when there was time
With AIAgent drafts it · short human review
High complexity · slow
Major featuresHard bug fixes
BeforeTop priority · long build · long review
With AIAgent builds a first pass · deep human review
Low complexity · quick
Routine edits
BothJust do it — the easy baseline
Low complexity · slow
Code reviewFixing flaky testsDaily task checks
BeforePostponed — too much time for the payoff
With AIFully handed to agents · no human time
← easyTime to executehard →
Now, with our real work

The work we do — and who does it now.

These are jobs we used to do entirely by hand. Sort them by how hard and how slow they are — then hand them over and click any one to see the tool that now runs it.

Today · manual, by hand
Now · run by agents
What it produced

The results, in numbers.

Security · rancher/dashboard
44 security PRs merged
45 raised · 44 merged — open vulnerabilities 18010
open Dependabot alerts · last 12 months

Agents raise the dependency bumps, validate them in a live preview, and ship the fix — clearing the CVE backlog.

▼ 94% of known CVEs cleared
Customer issues · JIRA + mixin
24 issues handled

Customer (JIRA) and small (mixin) issues assigned to me and worked by agents — 17 JIRA + 7 mixin — that would otherwise sit unhandled.

▲ 9 fixed, the rest in progress · click for the breakdown
Automated tests
35 flaky specs fixed
e2e test-failure rate · daily

Hardened across one batch (a second followed) — the failure rate fell to near zero, now runnable unattended overnight.

▲ ~99% attempt-1 pass rate
Scale · the console fleet
30+
Console Manager — one control plane over the fleet of 30 consoles

One control plane manages them all — purpose-built AI consoles, one per recurring job.

▲ and growing
Stale-issue bot · weekly
101 stale issues auto-closed

A weekly agent ages out issues that have gone quiet — 129 flagged, 101 closed — so the backlog reflects real, active work with no manual sweep.

▲ hands-free backlog hygiene · click for detail
Issue-grooming bot · daily
161 issues groomed

Every new issue is read, questioned for missing detail, and flagged ready for triage — humans open a triage-ready queue instead of raw reports.

▲ triage-ready, hands-free · click for detail
Unit-test coverage · Test Improver
83 test PRs merged
line coverage 51.3%58.7% · suite 4,606 → 9,359 tests
line coverage · since Apr

Agents write the missing unit tests every day — steadily lifting coverage on a codebase that had gone years under-tested.

▲ +7.4 pts coverage · click for the graph
Every figure is pulled from our own GitHub + dashboards: 44 security PRs (45 raised), 24 JIRA+mixin issues handled, 35 flaky specs in the first batch, 30+ consoles, 101 stale issues auto-closed, 161 groomed, 83 test-coverage PRs. The 180→10 is the rancher/dashboard vulnerability count.
The obvious question

“But how do you know it’s right?”

We don’t take the AI’s word for it. Every change is checked the way a person would check it — and the AI shows its work.

  1. 01

    It drives a real browser

    The agent clicks through the actual product, like a user would.

  2. 02

    It captures the evidence

    Before/after screenshots and a screen recording of every run.

  3. 03

    It checks against a rulebook

    Automated tests and tagged, labelled checks confirm the expected result.

  4. 04

    A person signs off

    Nothing ships on trust — a human reviews the evidence and approves.

BeforeBefore — captured by the agent
AfterAfter — captured by the agent
▶ Recorded live with Playwright — an agent driving the real Rancher UI
Not just using AI — running it well

Built to be safe and to last.

This isn’t ad-hoc. We treat the AI system like any other piece of infrastructure — with security, standards and a long-term plan.

Contained

Sandboxed by default

Agents run inside isolated containers — no direct access to private or protected company data.

Repeatable

Shared best practices

A common library of prompts and rules so everyone works to the same standard.

Deliberate

Architecture mindset

We weigh long-term trade-offs, not just quick wins — the same way we build products.

Directional

A dev dashboard future

Consolidating the tools into one place: our long-term investment, not a fad.

Further reading — how Engineering uses AI, written down: the AI usage doc ↗
Now let’s show you

10 minutes of it actually working.

The most convincing part isn’t a slide — it’s watching an agent do the job. Here’s the running order (with a recorded version as backup, in case the network isn’t kind).

Demo is pre-recorded and embedded as the reliable backbone; we go live only if the environment is healthy on the day.
The takeaway

Same team. Far more shipped. With the receipts to prove it.

More work delivered per person Work we used to defer, now done Lower security risk Every change evidenced & reviewed
PRESENTED BY   Marcelo Fukumoto  ·  Cody Jackson  ·  with the Rancher UI team
or Space to move