How to A/B Test In-App Campaigns Without Engineering

A woman wearing a yellow embroidered top and a gray hoodie stands outdoors near a roadside, gently touching her hair, with trees and a hazy sky in the background.

Ritul Singh

Published 22 min read
A dark, minimalist scene showing a glowing, arched doorway with a shadowy figure standing inside, partially reflected on a glossy floor, creating a mysterious and atmospheric mood.

TL;DR

  • The largest brake on in-app experimentation is engineering dependency, not a lack of ideas. Every test that requires a sprint adds two to three weeks and costs multiple learning cycles a competitor without that constraint does not pay.
  • Teams that run A/B tests without engineering run more experiments, learn faster, and compound the advantage over successive quarters.
  • This article covers why engineering dependency kills experimentation velocity specifically.
  • It covers what "without engineering" actually means in practice.
  • It covers the three things that typically require engineering and how purpose-built tools eliminate each one.
  • It covers what to test and how to prioritise the backlog by expected impact.
  • It covers the statistical significance requirements for fast-moving in-app tests.
  • It covers when a multi-armed bandit approach fits better than a traditional A/B test.
  • It covers what to evaluate in a testing platform, and how to build a compounding, well-documented experiment backlog.
  • Sourcing note: All statistics are attributed to their sources throughout, and this article is a companion to Digia's existing guide on no-code in-app campaign infrastructure, which it does not repeat.

Mobile app release cycles typically run two weeks per sprint for teams following standard Agile, and adding App Store review time, staged rollout, and the reality that growth-adjacent tickets often sit below feature work in sprint priority, a realistic end-to-end timeline for a single in-app change can hit three to four weeks for teams without dedicated in-app tooling. That timeline describes shipping one change. A/B testing requires shipping at least two, usually more, and then waiting for each variant to accumulate enough traffic to reach a reliable conclusion. A team running experiments through an engineering-dependent pipeline is not just slow. It is running a fundamentally smaller number of experiments per year than the format allows, and every experiment not run is a learning the team does not have when it needs it.

This article is a companion to Digia's existing guide on no-code in-app campaign infrastructure, which covers the architecture, workflow, and evaluation criteria for a genuine no-code tool in depth. This piece goes narrower: specifically how to run a rigorous A/B test once that infrastructure exists, what to prioritise, how much traffic each variant actually needs, and when a bandit approach beats a traditional test.

Why Engineering Dependency Kills Experimentation Velocity

The cost of engineering-dependent testing is not simply "tests take longer." It compounds in three specific ways that a single delayed campaign does not fully capture.

The sprint cycle cost. Each variant in a test that requires custom implementation needs its own ticket, its own QA pass, and its own place in the release queue. A two-variant test does not cost one sprint cycle. It costs the coordination overhead of two separate implementations landing in the same release, which in practice often means the test does not launch until both are ready, adding the slower variant's timeline to the whole experiment.

The opportunity cost of delayed learning. The experiment deficit created by engineering-dependent workflows is structural: a growth team running one in-app experiment per month because of engineering bottlenecks is operating well below its potential contribution, and Gartner's forecast that 70% of new enterprise applications will use no-code or low-code technology by 2026 reflects, in part, a recognised competitive disadvantage accumulating for teams that remain dependent on engineering for every change. A learning delayed by three weeks is not a learning that arrives three weeks late with the same value. The market condition, competitor move, or seasonal window that made the hypothesis worth testing has frequently shifted by the time the result comes back, which means the insight is stale before it is even actionable.

What the experiment backlog looks like under an engineering bottleneck. In practice, this produces a specific and recognisable pattern: a backlog of hypotheses that never gets tested because each one, individually, does not seem important enough to justify a sprint ticket. Small, high-frequency questions, does this CTA copy convert better than that one, does this trigger moment outperform that one, are exactly the questions an engineering-dependent pipeline is worst suited to answer, because the fixed cost of shipping a variant does not scale down for a small question the way it needs to. The result is a backlog dominated by untested assumptions, not because the team lacks curiosity, but because the infrastructure makes testing expensive regardless of the question's actual size.

What "Without Engineering" Actually Means in Practice

The phrase gets used loosely enough that it is worth being precise about what genuinely removes the dependency, as distinct from tools that simply relabel it.

No-code campaign creation means a growth or product team member can build a new campaign variant, copy, visual treatment, CTA, entirely through a dashboard interface, with no code written and no developer review required before it can go live. This is the baseline requirement, but it is not sufficient on its own.

SDK-based rendering with server-controlled variants means the variants themselves are rendered by a pre-installed SDK reading configuration from a server, rather than being compiled into the app binary. A genuine no-code in-app tool ships native components rendered by the SDK without an HTML bridge, and campaigns can trigger in under 100ms because the component is rendered natively rather than downloaded and parsed through a WebView. This is the specific architectural decision that decouples variant creation from the app release cycle. Without it, "no-code" campaign creation still ultimately requires a new app version to actually reach users, which reintroduces the exact bottleneck the tool claims to remove.

Event tracking that doesn't require custom implementation per test means the underlying behavioural events a test needs to measure, an impression, a click, a downstream conversion, are already instrumented once, generically, and available to any campaign or test configured afterward, rather than requiring a developer to add tracking code specific to each new experiment. A platform that requires a custom analytics event to be implemented for every new test variant has only removed the engineering dependency from variant creation, not from measurement, which means the team is still gated by engineering capacity at the exact moment a test needs to go live.

All three conditions have to be true simultaneously for a testing programme to genuinely operate without engineering. A tool that satisfies two out of three still leaves a residual bottleneck at whichever condition is missing.

The Three Things That Typically Require Engineering, and How Purpose-Built Tools Eliminate Each

Custom UI element creation. In a standard engineering-dependent setup, every new variant of an in-app element, a different modal layout, a new banner treatment, a redesigned card, requires a developer to build it as a new UI component, because the app's codebase has no generic, reusable rendering system for arbitrary in-app content. Purpose-built no-code platforms eliminate this by shipping a library of pre-built native components, tooltips, bottom sheets, persistent nudges, gamification cards, inline widgets, and in-app video, that a growth team configures through a visual builder rather than requesting as new code. The component exists once, engineered well, and every subsequent variant is a configuration of that existing component rather than a new build.

Variant routing logic. Deciding which user sees which variant, and ensuring that assignment is stable, random, and correctly proportioned, traditionally requires custom logic written into the app or a backend service maintained by engineering. A purpose-built testing layer handles this natively: the platform's own audience and targeting engine assigns users to variants, tracks the assignment persistently so a user does not flip between variants across sessions, and exposes the resulting split to the growth team as a dashboard configuration rather than a codebase change.

Event instrumentation. Measuring whether a variant actually performed better requires tracking specific behavioural events tied to that variant, which in a custom-built system means a developer adding tracking calls for each new test. A platform with generic, pre-instrumented event tracking, exposure, interaction, and downstream conversion events that apply to any campaign configured on the platform, removes this dependency by making measurement a property of the platform itself rather than something built fresh per experiment.

The pattern across all three: engineering dependency exists wherever a capability has to be built fresh for each new test rather than existing once as reusable infrastructure. A genuinely no-code testing platform has already paid the engineering cost, once, at the infrastructure level, so that every subsequent test is a configuration exercise rather than a development project.

What to Test: The Experiment Hierarchy by Expected Impact

Not every test is worth the same investment of attention, and prioritising a testing backlog requires a rough sense of which variable categories tend to produce the largest effect sizes.

Format (bottom sheet versus tooltip versus persistent nudge versus interstitial, for the same underlying message) typically produces the largest effect sizes of any variable category, because format determines the fundamental interaction cost and visual prominence of the entire campaign, not just one element within it. A message delivered as a full-attention interstitial and the same message delivered as a small persistent banner are not variations on a theme. They are different products competing for a different quality of attention.

Trigger (the specific event or moment that fires the campaign) is the second-highest-leverage variable, because it determines whether the campaign reaches the user in a contextually relevant moment or an arbitrary one. A campaign fired at the correct trigger and a campaign fired at the wrong trigger, with identical creative, routinely produce very different results, because the trigger determines the user's receptiveness before the creative has any chance to work.

CTA (the specific action the campaign asks the user to take, and how it is worded) typically produces a moderate effect size, meaningful but generally smaller than format or trigger, because it operates within whatever attention and context the format and trigger have already established.

Copy (the specific wording of the message body, independent of the CTA) usually produces the smallest reliable effect size of the four categories for a well-targeted, well-triggered campaign, though copy testing remains valuable for optimisation once format and trigger have already been validated as correct.

Creative (visual treatment, imagery, colour, animation) effect sizes vary the most unpredictably of any category, capable of producing large effects in specific cases (an image that clarifies an otherwise confusing offer) and negligible effects in others (a colour change with no functional difference to comprehension).

The practical prioritisation rule that follows from this hierarchy: when a testing backlog contains untested format or trigger hypotheses alongside untested copy hypotheses, the format and trigger tests should generally run first, because they are more likely to reveal a large effect that changes the entire approach, whereas a copy test is more appropriately run as a refinement once the format and trigger are already validated as reasonably correct.

Statistical Significance for In-App Tests

The mechanics of calculating whether a result is real, rather than noise, are the same for in-app tests as for any other conversion experiment, but the practical thresholds and traffic requirements are worth stating explicitly for teams new to running this discipline internally.

How much traffic each variant needs. A reasonable rule of thumb for most mobile app tests is at least 1,000 to 2,000 users per variant to reliably detect an effect of 5% or larger, while detecting a smaller effect in the 2 to 3% range requires 5,000 or more users per variant. Apps with a smaller active user base, under roughly 10,000 monthly active users, should generally focus on tests where the expected effect size is large and obvious rather than attempting to detect subtle differences that would require more traffic than the app produces in a reasonable testing window. This is a direct, practical reason the experiment hierarchy above matters: a format test, which is likely to produce a large effect, is achievable with a smaller sample than a copy test chasing a marginal improvement, which means smaller apps should weight their testing backlog even more heavily toward the high-leverage categories.

Which p-value thresholds are appropriate for fast-moving campaign tests. The conventional 95% confidence threshold (p less than 0.05) remains the reasonable default for any test informing a decision the team intends to keep in place for months, such as a permanent change to an onboarding flow's format. For genuinely fast-moving, low-stakes campaign tests, seasonal promotions, one-off event-tied messaging, where the cost of being wrong is limited to the campaign's own short window and the decision will not persist beyond it, a team can reasonably accept a lower confidence threshold in exchange for a faster read, provided this trade-off is made deliberately and disclosed when the result is reported, not silently substituted for rigour without acknowledgement.

The sample size calculator approach. Before launching any test, calculate the required sample size from three inputs: the current baseline conversion rate for the metric being tested, the minimum effect size that would be meaningful enough to act on, and the desired confidence level. This calculation should happen before the test starts, not be checked retroactively against whatever traffic accumulated by an arbitrary stopping date, because checking significance repeatedly as data accumulates and stopping the moment a result crosses the threshold by chance is one of the most common and most avoidable errors in applied A/B testing, producing false positives at a rate meaningfully higher than the nominal significance threshold implies.

Multi-Armed Bandit vs Traditional A/B Testing

The choice between these two approaches turns on a specific trade-off: speed of learning and minimised opportunity cost during the test, versus statistical confidence in the declared winner and the ability to analyse results by segment afterward.

How a multi-armed bandit works, mechanically. Rather than holding traffic evenly split between variants for the test's full duration, a multi-armed bandit continuously learns from engagement data as it accumulates and redistributes traffic toward the better-performing variant while the test is still running, using an algorithm such as Upper Confidence Bound, which prioritises testing options with more uncertain performance, or Thompson Sampling, which predicts each variant's probability of success based on all results collected so far. The practical effect is that a bandit test minimises the number of users who see the underperforming variant for the test's full duration, because traffic shifts toward the leader as evidence accumulates rather than remaining fixed at an even split.

When traditional A/B testing fits better. A/B testing is the better choice for validating a big, consequential shift, a redesign, a pricing change, a new navigation structure, where a wrong move could be costly, and for any comparison where the team specifically wants confidence that the measured difference is genuinely caused by the variable being tested, not an artifact of a shifting traffic mix during the test. A/B tests also preserve the ability to slice results by any dimension after the fact, mobile versus desktop, new versus returning users, because traffic remains evenly split for the full test duration, which means every segment accumulates enough data to be analysed meaningfully. In a bandit test, weaker variants receive so little traffic by the end that slicing their data becomes unreliable, meaning the team learns which variant won without being able to reliably investigate why.

When a multi-armed bandit fits better. Bandits are the stronger fit when speed and adaptability matter more than post-test analytical depth, which describes most short-duration, time-boxed in-app campaigns directly: a seasonal promotion, a limited-time offer, or any campaign where the underlying opportunity itself will have expired before a traditional even-split A/B test would reach significance. The trade-off is explicit: a bandit test optimises for maximising conversions during the live campaign window, at the cost of the deep post-hoc segment analysis a traditional test would allow, and it typically watches and optimises for a single primary metric, which means a secondary metric quietly deteriorating in the background is not something the algorithm will surface or correct for.

A practical decision rule. If the test result needs to inform a durable, structural decision the team will keep in place for months and wants to interrogate by segment afterward, run a traditional A/B test with an appropriately calculated sample size. If the test is tied to a genuinely time-boxed campaign where the value of the underlying opportunity decays before a traditional test would reach significance, and where maximising in-window performance matters more than deep post-hoc analysis, a multi-armed bandit is the better-suited tool. One caution applies specifically to bandit tests run during unusually volatile periods, a product launch, a major ad campaign, a live event, where the traffic mix itself is shifting: the winner a bandit identifies under those conditions may reflect the specific, temporary traffic composition rather than a durable preference, and should be treated with appropriate caution before being generalised to other periods.

Platforms That Enable No-Code In-App A/B Testing: What to Look For

The evaluation criteria for a testing-specific capability build directly on the general no-code platform criteria, with a few additions specific to experimentation.

Does variant configuration require a new implementation per test, or is it a dashboard exercise? On a platform where every in-app change requires engineering, each test variant needs a separate implementation, a QA pass, and a release, which means most teams end up testing far fewer hypotheses per quarter than they intend to, because the cost of each individual test is high. Confirm specifically that adding a second, third, or fourth variant to an existing test is a configuration action within the dashboard, not a new development request.

Does the platform calculate and surface statistical significance natively, or does the team need to export data to a separate tool? A platform that shows raw engagement numbers per variant without a built-in significance calculation forces the team to maintain a parallel analysis workflow, which reintroduces friction at the exact step where a fast decision matters most.

Does the platform support both traditional even-split testing and bandit-style dynamic allocation? Given that the two approaches suit genuinely different test types, a platform that only supports one forces every test into a single methodology regardless of fit, which produces either unnecessarily slow results for time-boxed campaigns or unnecessarily shallow analysis for structural decisions.

The specific workflow from hypothesis to live test to result should be traceable end to end within the platform: define the hypothesis and the metric it targets, configure the variants, set the traffic split or select the dynamic allocation method, launch without an engineering ticket, and retrieve a significance-checked result, all as a continuous workflow rather than a process that hands off to a different tool or a different team at any step.

Building a Testing Backlog

A single well-run test is a data point. A maintained, prioritised backlog of tests, documented consistently, is what actually compounds into a durable competitive advantage over time.

How to maintain a prioritised experiment list. The backlog should rank hypotheses using the same expected-impact hierarchy covered earlier, format and trigger hypotheses generally prioritised above copy and creative hypotheses, combined with a rough estimate of how quickly each hypothesis can reach statistical significance given current traffic levels. A hypothesis with a large expected effect and a fast time-to-significance should be tested before a hypothesis with a small expected effect and a slow time-to-significance, all else equal, because it returns a usable answer sooner and at lower opportunity cost.

How to document hypotheses and results. Every test entry should record, before the test launches, the specific hypothesis being tested, the primary metric it is measured against, the minimum effect size the team considers meaningful, and the calculated required sample size. After the test concludes, the entry should record the actual result, whether it reached the pre-specified significance threshold, and a brief note on whether the finding is likely to generalise beyond the specific context it was tested in (a specific season, a specific user segment, a specific competing campaign running concurrently) or whether it is context-bound and should not be assumed to hold under different conditions.

How to compound learnings across tests. The value of a documented backlog compounds specifically because later hypotheses can build on earlier confirmed findings rather than re-testing settled questions. A team that has already confirmed, through a properly powered test, that bottom-sheet format outperforms tooltip format for a specific campaign category does not need to re-litigate that finding every time a new campaign in the same category is proposed. It can treat the earlier result as an established input and focus new testing capacity on the next open question, which is what allows testing velocity to genuinely accelerate over successive quarters rather than plateauing at whatever rate a fixed amount of testing capacity produces when every test starts from zero prior knowledge.

Topics Not in the Brief That Teams Should Know

Sequential testing and the peeking problem deserve explicit team-wide agreement, not just individual discipline. Repeatedly checking a test's results as data accumulates, and stopping the moment the result happens to cross a significance threshold, inflates the false positive rate well beyond the nominal p-value the team believes it is operating at. The fix is not simply "don't do this." It is agreeing on and documenting the sample size and stopping rule before the test launches, and treating any interim check as directional monitoring only, not as a basis for an early stopping decision, unless the team has specifically adopted a sequential testing methodology designed to control for this.

Novelty effects distort early results for any test involving a genuinely new format or mechanic. A test comparing a familiar campaign format against a format users have never encountered before will often show an inflated early result for the new format, purely because novelty itself drives engagement independent of the format's actual merit. Running the test long enough to see whether the effect persists past the first exposure, rather than concluding from an early lead, is the practical mitigation.

Interaction effects between concurrent tests are easy to miss on a platform running many simultaneous experiments. Two tests running at the same time, targeting overlapping audiences, can each individually appear to show clean results while actually interacting with each other in ways that distort both. A documented backlog should track which audience segments are currently exposed to which live tests, so a team can identify overlap before drawing conclusions from either test in isolation.

The organisational habit of testing matters as much as the infrastructure that enables it. A platform that removes the engineering dependency does not automatically produce a high-velocity testing culture. Teams that see the largest compounding benefit tend to treat testing as a standing operating rhythm, a fixed cadence of tests always in flight, rather than an occasional activity reached for only when a specific high-stakes decision demands it.

Key Takeaways

Engineering dependency does not just slow individual tests. It compounds into a structural experiment deficit, where small, high-frequency hypotheses go untested because the fixed cost of shipping a variant does not scale down for a small question, leaving a backlog dominated by untested assumptions.

Genuine engineering-free testing requires three conditions together: no-code variant creation, SDK-based rendering with server-controlled variants that reach users without an app release, and event tracking that does not require custom implementation per test. A platform satisfying only some of these still leaves a residual bottleneck.

Custom UI element creation, variant routing logic, and event instrumentation are the three specific capabilities that traditionally require engineering, and purpose-built no-code platforms eliminate each one by paying the engineering cost once, at the infrastructure level, rather than once per test.

Format and trigger hypotheses typically produce the largest effect sizes and should generally be prioritised in a testing backlog over copy and creative hypotheses, which tend to produce smaller, more refinement-oriented effects once the higher-leverage variables are already validated.

Most mobile app tests need at least 1,000 to 2,000 users per variant to reliably detect a 5% effect, and 5,000 or more per variant for a 2 to 3% effect, which means smaller apps should weight their testing backlog toward large-effect-size hypotheses that are actually detectable with the traffic they have.

Traditional A/B testing fits durable, structural decisions where post-hoc segment analysis matters and confidence in the winner is paramount. Multi-armed bandit testing fits time-boxed campaigns where the underlying opportunity would expire before an even-split test reaches significance, at the cost of reduced ability to analyse results by segment afterward.

A maintained, documented testing backlog, prioritised by expected impact and time-to-significance, is what converts individual test results into a compounding advantage, because later hypotheses can build on confirmed prior findings rather than re-testing settled questions from zero every time.

Further Reading

From Digia Engage:

External Sources:

The no-code campaign builder, native SDK rendering, generic event tracking, and audience targeting engine described in this article are built into Digia Engage, letting growth teams configure and launch A/B test variants directly from the dashboard without an engineering ticket per test, after a one-time SDK integration of around 20 minutes. Book a demo to see the testing workflow from hypothesis to significance-checked result, or read the no-code campaign architecture guide for the underlying infrastructure this testing capability depends on.

Frequently Asked Questions

Why does engineering dependency reduce how many in-app A/B tests a team actually runs?
Each variant that requires custom implementation needs its own development ticket, QA pass, and place in the release queue, which means a two-variant test costs the coordination overhead of two separate builds landing in the same release. Beyond the direct time cost, this produces a structural experiment deficit: small, high-frequency hypotheses go untested because the fixed engineering cost of shipping a variant does not scale down for a small question, leaving a testing backlog dominated by untested assumptions rather than a lack of ideas.
What does it actually mean to A/B test in-app campaigns without engineering?
It requires three conditions holding simultaneously: no-code variant creation, where a growth team member builds a new campaign version through a dashboard with no code written, SDK-based rendering with server-controlled variants, where the variant reaches users through a pre-installed SDK reading server configuration rather than requiring a new app release, and generic event tracking, where the behavioural events needed to measure a test are already instrumented once and available to any new experiment without custom implementation. A platform satisfying only some of these conditions still leaves a residual engineering bottleneck at whichever condition is missing.
What should teams test first when building an in-app experimentation backlog?
Format and trigger hypotheses should generally be prioritised over copy and creative hypotheses, because format and trigger typically produce the largest effect sizes. Format determines the fundamental interaction cost and attention level of an entire campaign, while trigger determines whether the campaign reaches users in a contextually relevant moment. CTA and copy testing are valuable for refinement but usually produce smaller, more marginal effects, and are best run once format and trigger have already been validated as reasonably correct for a given campaign.
How much traffic does an in-app A/B test variant need to reach statistical significance?
A reasonable baseline is 1,000 to 2,000 users per variant to reliably detect an effect of 5% or larger, and 5,000 or more users per variant to detect a smaller 2 to 3% effect. Apps with fewer than roughly 10,000 monthly active users should generally prioritise tests where the expected effect size is large, such as format tests, rather than attempting to detect subtle differences that require more traffic than the app produces within a reasonable testing window. The required sample size should be calculated from the baseline conversion rate and minimum meaningful effect size before the test launches, not checked retroactively.
When should a team use a multi-armed bandit instead of a traditional A/B test?
Multi-armed bandit testing fits situations where speed and adaptability matter more than deep post-test analysis, particularly time-boxed campaigns such as a seasonal promotion or limited-time offer, where the underlying opportunity would expire before an even-split A/B test reaches statistical significance. Traditional A/B testing fits durable, structural decisions, such as a permanent onboarding flow change, where the team wants confidence that the measured difference is genuinely caused by the tested variable and wants the ability to slice results by segment after the test concludes, since bandit tests reduce traffic to underperforming variants over time in a way that makes their segment-level data unreliable to analyse afterward.
A woman wearing a yellow embroidered top and a gray hoodie stands outdoors near a roadside, gently touching her hair, with trees and a hazy sky in the background.

About Ritul Singh

I am a tech-focused creative building engaging digital experiences.

LinkedIn →