🔮 Codeball — AI Code Review

Codeball is a code review AI that scores Pull Requests on a grade from 0 (needs careful review) to 1 (you should merge this!)

Use Codeball to add labels to help you focus, to auto approve PRs, and more. The Codeball action is easy to use (sane defaults), and is highly customizeable to fit your workflow when needed.

🏷 Labels PRs when you should review with caution – Stay sharp, don't let the bugs pass through!

✅ Identifies and approves or labels safe PRs – Save time by fast-tracking PRs that are easy to review

🏖 Great defaults, fully customizable and programmable with GitHub Actions

GitHub Action

The Codeball GitHub Action runs Codeball on all new Pull Requests, and approves the Pull Request (example) if the model classifies it as safe.

• Online Demo
• How Codeball Works

Quick Start

1. Create a new file called .github/workflows/codeball.yml with the following content:

name: Codeball
on:
  pull_request: {}
  pull_request_review_comment:
    types: [created, edited]

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          # For all configuration options see https://github.com/sturdy-dev/codeball-action/blob/v2/action.yml
          approvePullRequests: "true"
          labelPullRequestsWhenApproved: "true"
          labelPullRequestsWhenReviewNeeded: "false"
          failJobsWhenReviewNeeded: "false"

2. 🎉 That's it! Codeball will now run on new Pull Requests, and will approve the PR if it's a good one!

Customizations

Codeball Actions are built on multiple smaller building-blocks, that are heavily configurable through GitHub Actions. Here's a few examples:

If you're using Codeball in another way, please let us know in an issue!

Example: "Dry-run" mode, labels all PRs with the Codeball Result

▶️ codeball-dry-run.yml

name: Codeball
on: [pull_request]

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          approvePullRequests: "false"
          labelPullRequestsWhenApproved: "true"
          labelPullRequestsWhenReviewNeeded: "true"
          failJobsWhenReviewNeeded: "false"

Example: Approve only (no labels)

▶️ codeball-approve.yml

name: Codeball
on: [pull_request]

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          approvePullRequests: "true"
          labelPullRequestsWhenApproved: "false"
          labelPullRequestsWhenReviewNeeded: "false"
          failJobsWhenReviewNeeded: "false"

Example: Filter files (run only for PRs modifying a single service)

▶️ codeball-filter-files.yml

name: Codeball
on:
  pull_request:
    # Run Codeball only if files under "/web/" has been modified (and no other files)
    # See: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-including-and-excluding-paths
    paths:
      - '!**'
      - '/web/**'

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          approvePullRequests: "true"
          labelPullRequestsWhenApproved: "true"
          labelPullRequestsWhenReviewNeeded: "false"
          failJobsWhenReviewNeeded: "false"

Example: Fail the Codeball Action (❌) if Codeball does not approve the contribution

▶️ codeball-fail-not-approved.yml

name: Codeball
on: [pull_request]

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          approvePullRequests: "true"
          labelPullRequestsWhenApproved: "true"
          labelPullRequestsWhenReviewNeeded: "false"
          failJobsWhenReviewNeeded: "true"

Example: Skip Draft pull requests

▶️ skip-drafts.yml

name: Codeball
on:
  pull_request:
     types:
     - opened
     - reopened
     - synchronize
     - ready_for_review

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    if: ${{ !github.event.pull_request.draft }}
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2

Example: Running Codeball as a different user (to support CODEOWNERS, etc)

▶️ run-different-user.yml

Instructions

1. If you don't have one already, create a new GitHub bot account (a normal account, named something like your-org-bot)
2. Invite the account to your repository or org, and give it WRITE permissions
3. Generate a Personal Access Token for the bot account
4. Create a new Organization Action Secret (https://github.com/organizations/YOUR-ORG-NAME/settings/secrets/actions), or Repository secret (https://github.com/YOUR-ORG-NAME/YOUR-REPO/settings/secrets/actions) named CODEBALL_BOT_TOKEN with the PAT as the value.
5. Add GITHUB_TOKEN: ${{ secrets.CODEBALL_BOT_TOKEN }} to your codeball.yml (see example below)
6. All Codeball related actions (adding/removing labels, approving PRs, etc) will now run as your newly configured user!
7. (Optional) Add your new user to CODEOWNERS

name: Codeball
on:
  pull_request: {}

jobs:
  codeball_job:
    runs-on: ubuntu-latest
    name: Codeball
    steps:
      - name: Codeball
        uses: sturdy-dev/codeball-action@v2
        with:
          GITHUB_TOKEN: ${{ secrets.CODEBALL_BOT_TOKEN }}

Building Blocks

The Codeball sub-actions are:

• sturdy-dev/codeball-action/baller/@v2 – Triggers new Codeball Jobs
• sturdy-dev/codeball-action/status/@v2 – Waits for the the Codeball result
• sturdy-dev/codeball-action/approver/@v2 - Approves PRs
• sturdy-dev/codeball-action/labeler/@v2 – Adds labels to PRs
• sturdy-dev/codeball-action/suggester/@v2 – Converts comments to code suggestions

How Codeball works

Codeball uses a deep learning model that has been trained on over 1 million Pull Requests. For each contribution it considers in hundreds of inputs, including:

• The code diffs (perceptual hashes)
• The author's recent experience with the affected files
• The change frequency of the affected files
• Past code reversals / fix-ups

Codeball is optimized for precision, which means it only approves contributions that it's really confident in.

Troubleshooting

Permissions

Codeball works out of the box with GitHub Actions.

If you're using non-default permissions, or want to use a custom access token. Make sure that you're running Codeball with the following permissions:

permissions:
  contents: read
  issues: write
  pull-requests: write

To allow PR approvals, make sure that "Allow GitHub Actions to Create and Approve Pull Requests" is enabled in the repository and organization settings on GitHub (under "Settings > Actions > General").

Show recommended GitHub Permissions

If you can not (or do not want to) update the org and repo settings for GitHub Actions, install the "Codeball AI Writer" GitHub App on the repository. When installed, Codeball will use the permissions granted via the app instead of the GitHub Actions token.

Forks and public repositories

GitHub does not offer (and reasonably so) a way for Pull Requests from a fork to a public repository to run with "write" permissions to the parent repository.

If you want to use Codeball on a public repository, install the "Codeball AI Writer" app on the parent repository. This allows the Codeball Action to use the permissions from the app as a fallback if the action is running without write permissions.

Forks and private repositories

By default, Pull Requests from a fork does not have "write" permissions when running in GitHub Actions, and those Pull Requests can not be approved or labeled.

The easiest workaround to this issue is to install the "Codeball AI Writer" app (see instructions for how to use Codeball on a public repository).

Alternatively, you can enable "Send write tokens to workflows from fork pull requests" on the repository (docs) via GitHub.