What are the best practices for A/B split testing email send times and subject line variables?

Test send times and subject lines simultaneously using a multivariate split test, but isolate each variable in separate runs to attribute lift accurately.

1. Define hypothesis – e.g., "Sending at 10 AM yields a 3 pp higher open rate than 2 PM" and "Subject with emoji boosts click‑through by 1.5 pp".
2. Segment audience – Pull a random 10 % of the qualified list per variant using a stable hash (e.g., hash(email) % 100 < 10). Keep the remaining 90 % as control for later validation.
3. Configure the test – In SendGrid or Klaviyo, create two campaigns:
Campaign A: send at 10 AM, subject A.
Campaign B: send at 2 PM, subject B.
Enable ABTest flag and set sampleSize=0.10.
4. Run a pilot – Send 1 000 contacts per variant. Collect opens, clicks, and bounce metrics via the provider’s webhook (event_type = "open", "click").
5. Calculate statistical significance – Use a two‑tailed chi‑square test with a 95 % confidence threshold (p < 0.05). Example in Python:

import pandas as pd, scipy.stats as st
# df columns: variant, opens, deliveries
chi2, p, _, _ = st.chi2_contingency([[df.opens.sum(), df.deliveries.sum()-df.opens.sum()],
                                      [df.opens.sum(), df.deliveries.sum()-df.opens.sum()]])
print('Significant' if p < 0.05 else 'Not significant')

6. Scale the winner – If both time and subject pass significance, roll out the winning combination to the remaining 90 %.
7. Document and iterate – Log test_id, start_time, subject, open_rate, ctr, p_value in a shared Airtable for year‑over‑year trend analysis.

Quick reference table

Variable   | Typical Range | Recommended Tool
-----------|---------------|-----------------
Send Time  | 8‑10 AM, 12‑2 PM, 4‑6 PM | SendGrid Scheduler
Subject    | 40‑70 chars, emoji, personalization token | Klaviyo Subject Builder

Gotcha: If your list contains a high proportion of mobile‑only users, an early‑morning send may hit spam filters due to low engagement history; always monitor deliverability spikes before scaling.