Spam Filter and Inbox Placement Testing for Transactional Email

Every other test in a QA pipeline asks whether the message renders correctly. This one asks a different question: will it be shown at all. A template can be pixel-perfect in fifteen clients and still be filed in spam because it carries three tracking domains, no plain-text part, and a subject line that trips a rule nobody on the team has read.

Placement testing sits awkwardly between engineering and deliverability, which is why it is so often skipped. It is not deterministic in the way a snapshot test is — the same message can place differently on two consecutive days — and that non-determinism tempts teams into ignoring it entirely. The resolution is to test the parts that are deterministic in CI, and to treat the parts that are not as a monitored trend rather than a pass/fail gate.

What Is Deterministic and What Is Not

Content-based scoring is reproducible: feed the same message to the same rule set and you get the same score. Reputation-driven placement is not: it depends on the sending identity, the recipient's history, and the provider's current model. Splitting the two decides where each check belongs.

Core Implementation: Scoring the Message in CI

SpamAssassin runs locally against a raw message, which makes it the natural CI check. The pattern below renders the real template with fixture data, assembles a complete MIME message, scores it, and fails the build above a threshold set deliberately below the filtering line.

// spam-score.mjs — score the rendered template offline and fail below the bar.
import { execFile } from "node:child_process";
import { promisify } from "node:util";
import MailComposer from "nodemailer/lib/mail-composer/index.js";
import { renderTemplate } from "./render.mjs";

const run = promisify(execFile);

// SpamAssassin's default filtering threshold is 5.0. Gate well below it so a
// regression is caught while there is still headroom.
const SCORE_LIMIT = 3.0;

async function buildMessage(templateId) {
  const { html, text, subject } = await renderTemplate(templateId, {
    customer: { name: "Test Recipient", email: "recipient@example.net" },
    order: { id: "TEST-0001", total: "£42.00" },
  });

  // Score the message you will actually send: both MIME parts, the real
  // headers, the real link domains. Scoring the HTML fragment alone misses
  // the rules that fire on structure.
  return new MailComposer({
    from: "Acme <receipts@mail.example.com>",
    to: "recipient@example.net",
    subject,
    html,
    text,
    headers: {
      "List-Unsubscribe": "<https://example.com/u/abc>, <mailto:u@example.com>",
      "List-Unsubscribe-Post": "List-Unsubscribe=One-Click",
    },
  }).compile().build();
}

function parseReport(stdout) {
  // spamassassin -t writes an X-Spam-Report header listing each rule that fired.
  const rules = [];
  for (const line of stdout.split("\n")) {
    const m = line.match(/^\s*([-\d.]+)\s+(\w+)\s+(.*)$/);
    if (m) rules.push({ points: Number(m[1]), rule: m[2], description: m[3].trim() });
  }
  const total = rules.reduce((sum, r) => sum + r.points, 0);
  return { total, rules: rules.filter((r) => r.points > 0) };
}

export async function checkTemplate(templateId) {
  const raw = await buildMessage(templateId);
  const { stdout } = await run("spamassassin", ["-t", "-D", "none"], {
    input: raw.toString("utf8"),
    maxBuffer: 10 * 1024 * 1024,
  });

  const { total, rules } = parseReport(stdout);
  const ok = total < SCORE_LIMIT;

  if (!ok) {
    // Print the contributing rules, not just the total — "score 4.2" is not
    // actionable, "MIME_HTML_ONLY 1.1" is.
    console.error(`${templateId}: score ${total.toFixed(1)} exceeds ${SCORE_LIMIT}`);
    for (const r of rules.sort((a, b) => b.points - a.points)) {
      console.error(`  +${r.points.toFixed(1)}  ${r.rule}  ${r.description}`);
    }
  }
  return { templateId, total, rules, ok };
}

The detail that makes this useful in practice is printing the contributing rules. A total score tells a developer their change was rejected; the rule list tells them which change to make. Most email spam rules are mechanical and easy to fix once named — a missing text part, an image-to-text ratio, a link whose domain does not match the From domain.

The Content Signals That Actually Move the Score

A handful of structural properties account for most avoidable score in transactional mail.

Two of these deserve extra emphasis for transactional senders. Tracking domains are the most common self-inflicted score: an ESP's default click-tracking domain is shared across its customers, so its reputation is not yours. Configuring a CNAME'd tracking subdomain under your sending domain removes both the mismatch penalty and the shared-reputation exposure. And the plain-text part is not optional — its absence is a rule hit at essentially every filter, and it is trivially generated from the same component tree that produced the HTML.

Second Pattern: Placement Trending, Not Gating

The live half of the problem is answered by a seed panel — a small set of accounts you control across providers, mailed on a schedule with the real production template. Its value is comparative rather than absolute.

Run the panel daily against your highest-volume templates, record the folder each copy landed in, and alert on change rather than on an absolute inbox share. A template that has placed in the inbox for three weeks and lands in spam today is a real signal, whatever the underlying percentage. Seed accounts have no genuine engagement history, so their absolute placement is pessimistic compared with a real recipient's; treating the number as a prediction leads to chasing noise.

Keep the panel stable. Adding and removing accounts changes the denominator and destroys comparability with previous runs — the one property that makes the whole exercise useful. The mechanics of submitting and collecting these runs through a commercial platform are covered under Litmus and Email on Acid workflows; the scheduling and alerting discipline is the same whichever provider renders the panel.

Pipeline Integration Steps

1. Render with fixture data, not with live records, so the scored message is identical between runs.
2. Compose the full MIME message — both parts, real headers, real link domains — before scoring.
3. Score in CI on every pull request that touches a template or the layout, and fail at 3.0 rather than 5.0.
4. Print the contributing rules in the failure output so the fix is obvious without a local reproduction.
5. Verify structure separately: assert the text part exists, List-Unsubscribe is present in both forms, and no link points outside your sending domain.
6. Schedule the seed panel daily against the top templates by volume, and diff against the previous run.
7. Alert on placement change, and route the alert to whoever owns deliverability and reputation rather than to the template author by default — a placement drop is more often a reputation event than a content one.

Debugging: Named Symptoms, Cause, and Fix

Symptom: the score jumps by more than a point after a copy change. Cause: usually a subject-line or body rule such as all-capitals or excessive punctuation, not the change itself. Fix: read the rule list; these rules name exactly what they matched, and the fix is almost always a wording change rather than a structural one.

Symptom: CI score is clean but the message still lands in spam. Cause: reputation, not content. Fix: check the Postmaster Tools signals and authentication pass rates before changing anything in the template — content tuning cannot recover placement for a sender with a damaged reputation.

Symptom: the score differs between a developer's machine and CI. Cause: different SpamAssassin rule versions, or network rules such as DNS blocklist checks resolving differently. Fix: pin the rule set in the CI image and disable network-dependent rules for the deterministic check — those signals belong in the live panel, not in a build gate.

Symptom: MIME_HTML_ONLY fires despite generating a text part. Cause: the text part is empty, or the composer placed the parts in the wrong order. Fix: assert the text part has non-trivial length in the same test, and confirm the message is multipart/alternative with text first.

Symptom: seed placement flips between inbox and spam on alternate days. Cause: normal variance on accounts with no engagement history. Fix: alert on a sustained change across several runs rather than on any single result, and keep the panel membership fixed.

Validation and Deployment Checklist

• Every template is scored in CI from a fully composed MIME message.
• The gate fails at 3.0, leaving headroom below the 5.0 filtering line.
• Failure output lists the contributing rules and their point values.
• The plain-text part is generated from the same source as the HTML and asserted non-empty.
• List-Unsubscribe is present in both HTTPS and mailto: forms, with the one-click header.
• Click tracking uses a subdomain of the sending domain, not the ESP's shared domain.
• No template contains a shortened, redirecting, or IP-address link.
• The SpamAssassin rule set is pinned in the CI image.
• A seed panel runs on a schedule with a fixed membership.
• Placement alerts fire on sustained change, not on a single run.

Reading a Score Report Without Guessing

A SpamAssassin report is a list of rules with point values, and the instinct on first reading it is to attack the largest number. That is usually the wrong move, because the largest contributor is often a rule you cannot act on and the sum of three small, fixable ones outweighs it.

Rules fall into four groups, and knowing which group a rule belongs to tells you whether it is worth touching.

Structural rules fire on the shape of the message: a missing text part, a malformed MIME boundary, an HTML body with no corresponding alternative. These are always worth fixing, always fixable in the build pipeline, and they never come back once the generator is corrected. They are also the cheapest points on the report — a single change to how the message is composed can remove several rule hits at once.

Content rules fire on the words and markup: capitalisation in the subject, a high ratio of image area to text, phrases historically associated with bulk mail. Most are worth fixing, but with judgement, because some are triggered by legitimate transactional language. A receipt genuinely does mention a price and a currency symbol; rewording it to dodge a 0.3-point rule at the cost of clarity is a bad trade.

Network rules query external services at scoring time: DNS blocklists, URI reputation databases, sender authentication lookups. These belong in the live panel rather than in a build gate, because they depend on state that has nothing to do with the change under review and they produce different results on a developer's machine, in CI, and in production. Disabling them in the deterministic check is not cheating — it is putting the signal where it can be interpreted.

Meta rules fire on combinations, scoring a message that matched several other rules together. They are not directly actionable: fix the constituent rules and the meta rule disappears on its own. Chasing a meta rule by name is a common way to waste an afternoon.

The workflow that follows from this is mechanical. Fix every structural hit first and re-score; the total usually drops enough that nothing else is needed. If it does not, review the content hits and fix the ones where a clearer alternative exists anyway. Ignore the network and meta hits in CI entirely. What remains after that pass is your template's genuine floor, and if it is still close to the gate, the message is probably carrying something structural that the rule set does not name well — a tracking wrapper, an unusual encoding, an inherited layout with markup no longer used.

Where This Check Belongs Relative to the Others

Placement testing is the last gate before a template ships, and it depends on every earlier one having passed. Running it first produces confusing results: a template that fails to render also fails to score sensibly, and a broken conditional block can trip structural rules that have nothing to do with the message's actual content.

The sequence that works is render, then structure, then score, then place. A snapshot test confirms the markup is what the generator intended. A structural assertion confirms both MIME parts exist and the headers are present. Only then does scoring tell you something meaningful about the message, because the message is now the one you would actually send. And placement, being slow and non-deterministic, runs on a schedule against the templates that survived all three.

Ordering the pipeline this way also makes failures cheap to attribute. A score regression on a branch where the snapshot also changed is a rendering problem that happens to affect the score; a score regression with an unchanged snapshot is a genuine content change. Without the earlier gates, both look identical.

Frequently Asked Questions

Is a SpamAssassin score still relevant when Gmail does not use it?

Gmail does not run SpamAssassin, and neither does Microsoft. What makes the score useful anyway is that its rules encode structural properties every filter cares about — a missing text part, a mismatched link domain, an image-only message. Treat the score as a lint for message hygiene rather than as a prediction of Gmail's behaviour, and it remains one of the cheapest checks in the pipeline.

Should a failing spam score block a merge?

Yes, at a deterministic threshold, because the check is reproducible and every point it flags is something you chose. What should not block a merge is a live placement result, which varies for reasons unrelated to the change under review. Keeping those two on different sides of the gate is what stops the suite from becoming something people bypass.

How many seed accounts do we need?

Fewer than most teams expect — one per provider you care about, five or six in total. Comparability over time matters far more than breadth, and every additional account adds maintenance without adding signal. What does matter is that they are accounts you control on real consumer providers, not addresses on a testing service's own infrastructure.

Does adding an unsubscribe link to transactional mail hurt anything?

No, and at bulk volume the header form is required. A List-Unsubscribe header does not have to imply a visible unsubscribe link in a password reset — it can point at a preference centre. Its absence is a rule hit and, for senders above the bulk threshold, a compliance failure with Gmail and Yahoo.

Why does our score go up when we add tracking?

Because click tracking rewrites every link to a redirect on a different hostname, which trips both the domain-mismatch rules and, if the ESP's shared tracking domain has been abused, host reputation rules. Configuring a CNAME so tracking runs on a subdomain of your sending domain removes both effects and is usually a ten-minute change in the ESP dashboard.

Can we test placement without sending real mail?

Not meaningfully. Content scoring works offline, and it catches the majority of avoidable problems — but where a message lands depends on the receiving provider's live evaluation of your sending identity, which no offline tool can simulate. The offline check keeps you clean; only a real send tells you where you landed.

---

Related

• Litmus and Email on Acid Workflows — running spam and render panels through a commercial platform
• Automating Litmus Spam Testing in CI — the CI wiring for the hosted equivalent of this check
• Email Deliverability and Reputation — the other half of placement, and where a clean score still fails
• Email Authentication: SPF, DKIM, and DMARC — the authentication signals a placement failure often traces back to

← Back to Email Testing & QA Workflows