The commerce economics guide asks whether AI-assisted production improves the business result. This brief makes that question testable. It distinguishes a production-cost comparison from an experiment intended to estimate incremental commercial effects.
Choose the claim first
| Claim | Appropriate evidence |
|---|---|
| Production became faster or cheaper | Comparable briefs, recorded human/review time, tool costs and accepted-quality outputs |
| One creative package performed better | A controlled comparison of those packages with defined audience assignment and outcome measures |
| AI-assisted production generally improves performance | Repeated comparisons across representative briefs and production runs; one winning asset is insufficient |
Two ads cannot be identical except for “AI”: the production method changes creative choices. State what treatment is being tested, including human direction, selection and revision. A result applies to that tested production approach and campaign context.
Complete the test record
| Field | Record before launch |
|---|---|
| Decision | Whether to use the approach for a defined set of future briefs |
| Primary outcome | Contribution per assigned eligible user, cost per accepted asset, or another single decision-relevant measure |
| Treatment | AI tools, human work, review, selection and permitted revisions |
| Comparator | Current production process with comparable brief and quality requirements |
| Assignment unit | User, geography or another unit; randomisation method and contamination risks |
| Exposure and denominator | Eligible assigned units, actual delivery and exclusions for each group |
| Commercial conditions | Offer, product, price, media budget, geography and campaign duration |
| Cost | Production, review, tooling, media and relevant variable fulfilment/returns costs |
| Measurement window | Conversion lag, cancellation/return window and final readout date |
| Precision and duration | Minimum decision-relevant effect, expected variation and sample/duration rationale |
| Stopping and changes | Pre-agreed end, harm/operating stop rules and treatment of campaign changes |
| Analysis | How uncertainty, missing data, imbalance and platform optimisation will be reported |
Where assignment is random, compare outcomes using the defined assigned population, including units that did not convert. Do not choose only purchasers as the denominator. If campaign delivery algorithms select different audiences, explain the resulting limit on attributing the difference to creative production.
Plan uncertainty at the level of random assignment. If markets, stores, creators or other clusters are assigned, account for that clustering; thousands of users inside a few clusters do not create thousands of independent assignments. Agree a design appropriate to the available clusters before launch. Without random allocation or a justified alternative design, label the comparison observational and name the confounders.
Avoid repeatedly checking significance and stopping at a favourable moment unless the analysis method explicitly accounts for that process. A small or short test with inconclusive results is not evidence of no effect. Extend or redesign according to the registered rule rather than rewriting the success measure.
Calculate contribution consistently
Calculate each arm using the same cost boundary, then calculate the difference. Start with realised revenue, subtract the specified product, fulfilment, returns and payment costs, then the arm's media, production, review and tool costs. Allocate shared costs once using the declared rule; show non-recurring setup separately. Distinguish cash costs from existing staff time. If capacity is released without reducing payroll, report it separately. Agree the asset-quality acceptance rubric before counting cost per accepted asset, and include rejected variants and rework.
| Readout field | Comparator | Tested approach |
|---|---|---|
| Assigned eligible units and assignment clusters | Record denominator and unit | Record denominator and unit |
| Delivery and outcome completeness | Record exposure and missing/late outcomes | Record exposure and missing/late outcomes |
| Retained orders and order contribution | Record after the agreed return window | Record after the agreed return window |
| Direct media/production/review/tool cost | Record cash and staff capacity separately | Record cash and staff capacity separately |
| Shared allocation and setup | State the allocation rule and period | Apply the same rule |
| Primary outcome and uncertainty | Report the planned measure | Report the planned measure and difference |
| Decision and scope | Record what can be concluded | Expand, revise, obtain more evidence or stop |
For a fictional design rehearsal, a brand proposes testing two production packages for the same product and offer using random user assignment. It can identify the production tasks and costs but has not established eligible audience size, outcome variability or the return window. The correct brief is held for sample planning and measurement design; a media budget alone does not make the experiment ready. This example deliberately contains no campaign result.
Platform-attributed revenue is a measurement input, not profit or causal lift. The Google Conversion Lift documentation illustrates exposed/control designs; its availability varies and an advertising-exposure test does not automatically isolate an AI-production effect.
The final record should state the observed difference, uncertainty, implementation effort and scope of the conclusion. Use it to choose the next production decision. Preserve unsuccessful assets and revisions so the cost of selecting the winner remains visible.
Read our editorial principles for evidence, contribution and correction standards.