Filtering Bot Subscribers to Improve Email Deliverability

Quietly Breaking Email Systems

Email deliverability is an economy of trust. Small shifts in engagement or complaint rates have outsized effects on inbox placement. Once a sender's reputation degrades, recovering it is painful and slow.

At Shopify scale, bot-generated subscribers are a persistent threat to that reputation. Bots can quietly inflate mailing lists by millions of addresses. These addresses never engage, often trigger spam traps, and drag down sender reputation for merchants who did nothing wrong.

We filtered roughly 18 million bot subscribers from Shopify Email sends. The challenge wasn't detecting the bots, upstream systems were already doing that. The challenge was enforcing those signals safely, automatically, and without asking merchants to change their workflows.

Here is how we implemented send-time suppression on existing infrastructure, the testing pitfalls we encountered, and the impact on our sending reputation.

The Problem: Signal Without Enforcement

Shopify's upstream ecosystem was already identifying suspicious signups. A SIGNUP_BOT notification suppression signal was flowing through our data pipeline and persisting on customer records.

However, Shopify Email wasn't consistently enforcing this signal at the moment of delivery. This created a gap: we knew which subscribers were likely bots, but we were still allowing merchants to email them.

We faced a big product decision: filter bots automatically or give merchants control? Though control sounds ideal, automatic filtering was chosen to protect Shopify's email infrastructure, as some merchants would lose over 90% of customers otherwise. This aligns with our long-term goal of shifting filtering upstream.

We needed to close this gap with three strict constraints:

Zero Configuration: Merchants shouldn't need to toggle a "Filter Bots" setting. It had to be the default.

Zero Friction: We could not disrupt existing campaigns or automations.

Safety: We needed a way to ramp this up slowly to monitor for false positives.

The Architecture

We considered filtering bots at the time of list creation, but that wouldn't solve the problem for the millions of bot emails already sitting in merchant databases.

To solve this retroactively and proactively, we moved the enforcement to the send path.

We treated SIGNUP_BOT as a first-class reason to cancel delivery, identical to how we handle bounces or unsubscribes. This logic sits low enough in the stack (SendCustomerDeliveryOperation) to catch both batch marketing campaigns and automated flows.

The Implementation: Exception-Based Flow Control

The implementation utilized a service object pattern. Instead of a simple boolean check, we utilized an exception-based flow to interrupt the delivery process safely.

We introduced a helper method, signup_bot?, on the member model to check thenotification_suppressions array. We then guarded the execution with a Feature Flag (beta flag) to ensure we could control the rollout.

Here is what the core operation logic looked like:

def perform
  # ... validation checks ...
  # The Guard: Check the flag and the customer state
  if shop.has_beta_flag?(:bot_suppression_filtering) && member.signup_bot?
    # We raise a specific error to interrupt flow, rather than just returning false.
    # This allows us to handle the cancellation reason explicitly in the rescue block.
    raise BotCustomerError, "Customer #{member.customer_id} is flagged as a bot"
  end
  # ... render and send logic ...

rescue BotCustomerError => error
  # Explicitly mark the delivery as cancelled due to bot status
  customer_delivery.cancel!(reason: CustomerDelivery::CancelReason::SignupBotCustomer)
  raise error
end

By raising BotCustomerError, we ensured that the job didn't just fail silently; it triggered a specific cancellation workflow that updated the delivery status toSignupBotCustomer. This gave us precise observability into how many sends we were actually stopping.

The Engineering Hurdle: Implicit State in Tests

The implementation seemed straightforward, but it triggered a subtle issue in our test suite.

When we introduced the logic, our tests passed immediately. Too easily.

We discovered that our shared test helpers (specifically in mocking.rb) were implicitly creating customers with suppression flags enabled by default. The mock data assumed a "worst-case scenario" for customers, which meant notification_suppressionsoften contained ['SIGNUP_BOT'].

This meant our new logic was "working" in tests not because the code was robust, but because the test data was biased.

The Fix: We had to refactor the test factories to ensure thatnotification_suppressions were empty by default, forcing developers to be explicit when testing suppression logic.

# BEFORE: Implicit defaults masked the behavior
def member(...)
  notification_suppressions: ['SIGNUP_BOT']
end

# AFTER: Clean slate by default
def member(...)
  notification_suppressions: []
end

We then wrote explicit integration tests to prove the filter worked:

test 'bot customers are filtered out from campaign email sends' do
  # Explicitly flag the customer as a bot
  bot_member = ::Segmentation::Member.new(
    # ...
    notification_suppressions: [NotificationSuppression::SIGNUP_BOT],
  )
  operation = SendSegmentDeliveryBatchCourierUnitEmailOperation.new(batch: batch, member: bot_member)

  result = operation.perform
  # Assert the specific "0" result code we use for filtered members
  assert_equal(0, result.result)
end

This change eliminated false positives and ensured that a passing test actually meant the filter, and the feature flag, were working as intended.

Results: 18 Million Subscribers Filtered

We rolled this out using a phased backfill, targeting high-risk shops first.

Target: 61 Google-flagged merchants and 822 high-risk shops.

Volume: ~18 million bot-generated subscribers identified and suppressed.

Impact: Our goal was to reduce complaint rates for our lowest tier of senders. Post-launch, the complaint rate for this tier dropped from 0.07% to 0.053%.

These numbers proved that the suppression was doing what it was supposed to do: stabilizing sender reputation without harming legitimate sends.

Why this matters

This project started as a deliverability ticket and evolved into a system-level change involving technical design, backfill orchestration, and reputation management.

I shipped this feature during my third week as an intern at Shopify.

I mention this not to highlight my own work, but to highlight the engineering culture that made it possible. We prioritize shipping to learn. By leveraging existing signals (SIGNUP_BOT), respecting established patterns (suppression infrastructure), and focusing on high-leverage changes (send-time filtering), even a newcomer to the codebase can ship changes that impact millions of users.

Strong systems reward focused changes. By enforcing an existing signal at the right point in the stack, we improved email deliverability for thousands of merchants, transparently, effectively, and at scale.