How to Create and Manage Devin Playbooks for Standardized Testing and Repository Guidelines

Engineering teams deploying Cognition AI's Devin at scale often face a common operational challenge: ensuring that an autonomous AI software engineer follows team-specific testing protocols, respects architectural constraints, and writes code that adheres to repository standards without requiring repetitive manual prompting. Devin Playbooks solve this by acting as reusable, shareable, and version-controlled prompt templates that serve as custom system prompts for repeatable engineering tasks.

To standardize testing and repository-specific development guidelines across your organization, engineering leaders need to structure Playbooks around clear procedural steps, deterministic postconditions, and explicit guardrails. Teams must also understand how Playbooks interface with repository-level Skills and organization Knowledge to establish an automated, consistent software delivery pipeline.

Understanding Devin Playbooks: Architecture and Scope

A Devin Playbook is a structured prompt template designed to guide Devin through complex, multi-step engineering workflows. Instead of manually explaining testing frameworks, branching strategies, and linting rules in every chat session, developers can attach a standardized Playbook to execute recurring workflows consistently.

Playbooks vs. Skills vs. Knowledge

To organize automated development workflows effectively, engineering teams must differentiate between three complementary mechanisms within the Devin ecosystem:

Best Practice Rule: If an instruction is tied to how a specific codebase builds, starts, or runs tests locally, commit it to the repository as a Skill. If an instruction defines an end-to-end task workflow (such as triaging a bug report, running a full test-and-fix cycle, or preparing a pull request according to organizational standards), codify it as a Playbook. Playbooks can directly invoke repo-level Skills (for example, @skills:test-before-pr) to execute concrete commands.

Anatomy of a Production-Grade Testing Playbook

A resilient Playbook balances autonomy with deterministic guardrails. In Devin, high-performing Playbooks follow a standardized five-part structure:

1. Procedure

The Procedure section defines the chronological sequence of actions Devin must take.

• Format each instruction as one imperative step per line using active verbs (Run, Inspect, Verify, Generate).
• Design steps to be mutually exclusive and collectively exhaustive (MECE).
• Enforce the complete lifecycle: setup, task execution, validation, and delivery.

2. Specifications (Postconditions)

Specifications describe the verifiable criteria that must be satisfied before Devin may conclude the session. For testing workflows, this section defines what "done" means—such as all unit tests passing, zero lint regressions, and test coverage remaining at or above baseline levels.

3. Advice and Pointers

Advice acts like operational comments on Devin's decision tree. Use this section to calibrate Devin's priors, such as directing it to run headless tests, utilize Docker test fixtures rather than installing local database instances, or inspect specific failure logs first.

4. Forbidden Actions

Forbidden Actions establish strict guardrails to prevent risky or destructive behaviors. Examples include forbidding direct pushes to main, prohibiting changes to test assertions simply to make failing tests pass, and barring the installation of unauthorized external packages.

5. What's Needed from User

This section lists external inputs Devin cannot determine autonomously, such as target branch names, staging environment URLs, or specific issue tracking ticket IDs.

Creating and Managing Playbooks in the Devin Dashboard

Engineering leads and team administrators can create, update, and govern Playbooks directly within the Devin web application:

Step 1: Navigating to the Playbook Manager

1. Open the Devin dashboard and navigate to Settings > Playbooks (https://app.devin.ai/settings/playbooks).
2. Review the three available scope tabs:
   • Organization: Playbooks accessible to all members within your current workspace.
   • Enterprise: Playbooks centrally governed and shared across all organizations in your enterprise account.
   • System: Default, Cognition-maintained reference Playbooks available globally.

Step 2: Creating a Standardized Playbook

1. Under the Organization tab (or Enterprise tab if you have admin privileges), click Create playbook.
2. Enter a descriptive title (e.g., Standard Repository Testing & PR Readiness).
3. Assign a Macro: Define a unique, short identifier starting with an exclamation mark (e.g., !run-repo-tests or !pr-ready). Macros allow developers to attach the Playbook instantly in the chat input box.
4. Input the Markdown body adhering to the five-part structure.
5. Click Save. Devin automatically initiates version tracking for the document.

Step 3: Invoking Playbooks in Development Sessions

Engineers can attach Playbooks using two methods:

• UI Selection: Select the Playbook from the session creation screen. A blue pill and an inline configuration component confirm attachment, allowing engineers to customize parameters before launching.
• Macro Trigger: Type the assigned macro (e.g., !run-repo-tests) directly into the session prompt. The attached Playbook displays with a grey background in the session timeline.

Practical Implementation: Test Standardization & Guideline Enforcement

Below is a complete, production-tested Playbook template designed to standardize automated test execution, code review hygiene, and repository conventions.

Playbook: Repository Testing and PR Readiness Standard

## Overview
Standardize test execution, defect remediation, and pull request hygiene across repository code changes.

## What's Needed From User
- Target branch name to merge into (defaults to `main` if omitted)
- Summary or issue ticket identifier for the planned changes

## Procedure
1. Inspect git status and git diff to identify all modified and untracked files.
2. Check if a repository test skill exists at `.agents/skills/` or `.devin/skills/`; if present, invoke it using `@skills:<skill-name>`.
3. If no repo skill is defined, run the primary test suite: `npm test` (or project equivalent: `pytest`, `cargo test`, `go test ./...`).
4. If any test fails:
   - Analyze stack traces and root causes in test output.
   - Modify implementation code to resolve the defect.
   - Never alter test assertions unless explicitly requested by the user.
   - Re-run failing test files until all tests succeed.
5. Run the repository linter and formatter: `npm run lint` and `npm run format:check`.
6. Fix any linting or formatting violations introduced by the changes.
7. Verify git diff to ensure no stray debug logs, temporary files, or sensitive credentials were added.
8. Commit changes using Conventional Commits format (`fix:`, `feat:`, `test:`, `refactor:`).
9. Prepare a comprehensive PR summary documenting:
   - Root cause analysis
   - Specific files modified
   - Verification commands executed and test output confirmation

## Specifications
1. 100% of existing and new automated tests must pass without skipped suites.
2. Linter must return exit code 0 with zero errors and zero warnings.
3. No secrets, `.env` files, or binary artifacts may be committed.
4. The final summary must include the exact test commands executed and their output.

## Advice and Pointers
1. Prefer running isolated test files first before executing the entire test suite to conserve execution time.
2. If tests depend on local background services, check `docker-compose.yml` and start required services in detached mode.
3. When refactoring code, run tests incrementally after every substantive change.

## Forbidden Actions
1. Do not disable, skip, or delete failing tests to pass CI.
2. Do not use `git push --force` under any circumstance.
3. Do not modify infrastructure or CI/CD workflow configuration files (`.github/workflows/`) without explicit instruction.

Managing and Iterating Playbooks Across Engineering Teams

Writing effective Playbooks is an iterative engineering process akin to software development. To ensure long-term reliability and adoption across team members:

1. In-Session Live Iteration

When running Devin with a newly authored Playbook, monitor the session timeline. If Devin misinterprets an instruction, hesitates on directory structures, or chooses a suboptimal command, chat with Devin to correct course. Immediately update the Playbook with the missing advice or forbidden boundary so future runs succeed autonomously.

2. Version History and Rollbacks

Devin retains a full version history for every Playbook. Each time a team member edits a Playbook in Settings, Devin increments the revision counter. If an updated prompt introduces unexpected behaviors or degraded test accuracy, administrators can inspect the historical diffs and restore a known good version with one click.

3. Promoting Organization Playbooks to Enterprise

In large engineering enterprises, individual squads often test and refine Playbooks for their local repositories. When a squad establishes an exceptionally effective pattern (such as end-to-end integration testing or multi-service database migration), Enterprise Admins can promote the Playbook from the Organization tab to the Enterprise tab, standardizing practices across all business units.

Summary Checklist for Engineering Leads

By codifying testing procedures and development guidelines into Devin Playbooks, engineering organizations transform AI coding assistants from unpredictable experimental tools into disciplined, reliable contributors.

References

• Creating Playbooks - Devin Docs
• Using Playbooks - Devin Docs
• Devin Skills - Devin Docs
• Knowledge - Devin Docs