Testing Process & Methodology
BLUF:
This document outlines our A/B testing methodology for optimizing the ArmyMWR.com website. It provides a structured approach to ensure data-driven decisions, efficient project management, and effective communication with all stakeholders.Collect Data
Collect Data:
Our first step is to analyze site analytics using Matomo to identify high-traffic areas and pages with low conversion rates or high drop-off rates. Focusing on these areas allows us to gather data quickly and identify opportunities for improvement.
Write Out a Roadmap:
Creating a detailed roadmap is essential for defining goals, priorities, risks, and timelines. This roadmap will guide our testing process and keep all contributors informed.
Identify Goals:
We define our conversion goals to measure the success of each variation against the original version. Goals may include metrics such as button clicks, link clicks, downloads, and click-throughs.
Generate Hypotheses:
Based on identified goals, we generate hypotheses on why specific variations might outperform the current version. We prioritize these hypotheses by expected impact and implementation difficulty.
Create Variations:
Using Matomo, we make the necessary changes to elements of the website or mobile app. Variations might include changes to button colors, element order, navigation elements, or other custom modifications.
Run an Experiment on a Staging Environment:
We set up the A/B test on a staging site to ensure everything works as expected. The experiment is activated and run internally for quality assurance. Once verified, the A/B testing document is shared with stakeholders to ensure they do not edit the test page/item during the test.
Run Experiment:
The experiment is launched, and visitors are randomly assigned to either the control or variation. Their interactions are measured, counted, and compared to determine performance.
Analyze Results:
After the experiment concludes, we analyze the results using Matomo. The tool presents data showing performance differences between the two versions and indicates whether the differences are statistically significant.
Report on Results:
We prepare a detailed report on the experiment results, including metrics, analysis, and conclusions. This report is shared with stakeholders to inform them of the outcomes and next steps.
