Does AI-assisted content improve commerce economics? How to test it

A marketing team can make more content with AI and still leave the business worse off. Production savings may be absorbed by review, weak variants may consume media budget, or reported sales may include customers who would have bought anyway.

The useful question is whether a defined AI-assisted production process improves contribution after the costs of creating, distributing and fulfilling the resulting sales. Answer it with a controlled comparison and a cost record. Content volume and platform-attributed revenue can help explain the outcome, but neither establishes it.

For a regional brand or commerce leader, the first experiment should be narrow enough that the result changes a decision: expand this production approach, revise it or stop paying for it.

Choose the claim you want to test

“AI improves marketing” combines several possible claims. A team might expect faster production, lower production cost, better creative performance or more commercially useful variation. Each requires a different comparison.

Start with one product range, audience, offer and channel. Define the AI-assisted work precisely: generating draft scripts, adapting approved images, localising copy or preparing variants for a human editor. Keep a record of what people still do, including briefing, selection, revision and approval.

The IAB SEA+India's Winning the Scroll playbook page, published in January 2026, places content within a broader chain of discovery, consideration and purchase. It emphasises creators, platform dynamics and consumer trust. That is useful context for choosing an experiment. It does not establish that AI-generated content produces incremental sales.

If the claim is lower production cost at acceptable quality, begin by measuring the full production process against an agreed acceptance standard. If the claim is better sales performance, the experiment must also examine what happens after publication. A saved hour and an additional profitable order are different benefits; record both without merging them into an unsupported uplift percentage.

Decide what the comparison can tell you

A creative comparison usually exposes comparable groups to different creative approaches. An incrementality experiment compares exposure with a suitable holdout to estimate what the advertising itself changed. Both can be useful, but they answer different questions.

Google's Conversion Lift documentation describes test and control groups for measuring conversions caused by advertising exposure. The tool is not available to every account. More importantly, an exposure test alone does not isolate the contribution of AI to creative production.

For an AI-assisted creative test, randomly assign eligible users or suitable groups where feasible and retain the original assignment denominator. Record budgets, offers, actual delivery and platform optimisation. If markets or other groups are assigned, the analysis must account for that grouping; many users in a few markets do not create many independent assignments. Where delivery differs, the result may describe the combined creative-and-delivery approach. Without a justified causal design, describe the comparison as observational and name the likely confounders.

Organic creator content is harder to isolate: audiences overlap, people share posts, and creator trust is not interchangeable. A before-and-after comparison can reveal a useful pattern, but seasonality, promotions and distribution changes can also explain it. Treat such evidence as directional unless the design supports a stronger claim.

Count contribution and the work around production

Define the primary business measure before launch. For a commerce experiment, contribution after campaign costs is often more useful than gross attributed revenue. Agree with finance which product, fulfilment, transaction, return and other variable costs belong in the order contribution, then subtract media and creative costs consistently.

The following calculation is an illustrative scenario, not campaign research. Assume two groups of 10,000 eligible assigned users, a completed return window, and S$25 contribution per retained order after the declared variable order costs. Media spend is the same in each arm. Production costs include creation, approval, allocated staff time, tool usage and each arm's share of subscriptions or purchased assets. Assume no additional setup outlay in this example; show it separately when present. Use the same cost boundary for both arms before calculating the difference.

The hypothetical difference is S$900: S$500 from the additional orders and S$400 from lower production and approval cost. That calculation shows how to separate the mechanisms. It does not show that the order difference is statistically reliable or caused by AI.

If both arms produced 200 retained orders, the saving under these assumptions would be S$400. If the AI-assisted arm produced 180, its contribution after the stated costs would fall to S$700, below the existing process. Cheaper content can therefore be a poor purchase when its performance falls sufficiently.

Time savings also require careful treatment. Reducing the hours allocated to production releases capacity; it becomes a cash saving only if spending falls or an otherwise necessary cost is avoided. Keep the operational benefit visible without reporting the same staff time as both an immediate cash saving and free capacity for another project.

Write the experiment brief before seeing results

Use this worksheet to agree the decision and prevent the metric from changing after the campaign starts.

Have the analyst check whether the available traffic and conversion rate can resolve a commercially meaningful difference. A small campaign may produce an inconclusive result even when the business benefit could matter. “We cannot distinguish the approaches yet” is a valid finding; declaring a winner from noise can lock the team into a worse process.

Do not stop at the first favourable result unless the planned analysis explicitly supports sequential stopping. One conventional asset against one AI-assisted asset tests those packages under those conditions. A broader claim about production needs repeated comparisons across independent briefs, including selection effort, rejected variants and revisions. The experiment brief records these choices and the per-arm costs before launch.

Record tool and model versions, material prompt changes and the amount of human revision. If the production approach changes halfway through, the experiment is no longer comparing the originally specified processes.

Expand only the finding you actually established

A result for one product, language or creator relationship is a reason to test adjacent conditions. It is not a regional forecast. Preserve the elements that appeared to drive the result and change one meaningful condition at a time where practical.

Bring marketing, finance and the person designing the experiment together to complete the brief. The first approval should fund a test that can answer a business question. A larger content budget can follow when the evidence supports the production approach and the team understands the work required to sustain it.