Email Creation Optimization Experiment

Overview

This experiment aimed to validate that helping users more easily remove placeholder content from their emails would result in more users successfully sending their first email. Consequently this would positively impact our Activation rate as sending your first email was a critical step in our activation funnel.

Role

Product designer

Research (collaborator)

Workshop facilitator

output

Disposable designs

Final experiment mocks

Problem

Difficulty in Locating & remediating Placeholder Content Lowers Email Send Rates

Within the Mailchimp email creator, if there is any placeholder content present, a user will not be able to send their email. The way we communicate that placeholder content exists in the email creator makes it difficult for users to find the placeholders and remove them resulting in lower send rates for new users.

Goal

Boost Activation by Streamlining the First Email Workflow and Simplifying Placeholder Content Management

Our goal is to increase our Activation rate by reducing the amount of friction found in the workflow of sending the first email. We will achieve this by making it easier to identify and / or remove placeholder content from an email.

Research

Uncovering Activation Funnel Barriers Through Observational Studies

The supporting research for this experiment was informed by observational studies that the entire growth activation team took part in at our Brooklyn, NY offices. Throughout these studies, every user we asked to go through the activation funnel had considerable thrash in the “send your first email” step. This was later validated by data suggesting that users with placeholder content in their emails, that also exhibited the navigation behavior of checking between the checklist and email editor more than once, had an extremely low activation funnel completion rate.





Output

Disposable designs

These are different disposable designs I put together based on our Innovation Lab exercise. Each prompt approaches solving the overall problem in a different way. The prompts are ranked by estimated level of effort from lowest to highest.

Output · Variant one

Combining our lowest-effort approaches into one solution

In this variant we present the user with our Feedback Block component in their email checklist. This feedback block spells out the placeholder content that exists and provides a user with a clear path to get back into the email builder in order to resolve the issue.

Output · Variant Two

Using a modal to inform the user of placeholder content before exiting the email builder

When a user tries to leave the email builder with placeholder content still present, we will fire a modal that let’s them know what the content is. This gives them the opportunity to fix the issue in the same context the warning is in.

Output · Variant Three

Determine whether combining variants has a greater impact on success

This variant combined the tactics of One and Two in order to determine whether the sum of these approaches would have a greater impact on our activation rate.

Solution

Combining the variants won out in this experiment

By integrating both approaches, we addressed all relevant aspects of our problem. We informed users that their email contained placeholder content in the email checklist, along with a straightforward path back to the email builder for its removal. Furthermore, we ensured that users were made aware, before exiting the email builder, of any remaining placeholder content that would prevent them from sending their email.

Results

Optimizing Email Creation for better placeholder identification Yields a 5% Lift in User Activation

In this experiment we optimized the email creation workflow, making it easier for users to identify and remove placeholder content, a significant barrier to sending their first email (a critical step in our Activation funnel). Our winning variant led to a remarkable 5% absolute lift in activation, marking the experiment as the most successful of the year.

Collaborators

Growth Product Manager
Data Analyst

That's a wrap for this project!

See more of my work from

MailChimp