Magento CRO agencies, ranked on the test evidence they publish

Takes the heaviest criterion on three named Magento tests. The Byggmax A/B testing case study opens by stating the estate it ran on, "over 190 brick-and-mortar stores, 4 Magento (Adobe Commerce) stores, and 55k+ products in their catalog", then publishes each test with the control and the variant described, the metrics split into primary and secondary, the device segmentation and the result. Test one, "the current grey CTA on the hero banner (control) vs. a green color CTA (variant)", measured "Hero banner click-through rate (primary)" against two secondaries and produced "more than a 15% increase in revenue and an 11% increase in purchases". Test two, white cart icon against green, produced "a significant increase in cart icon clicks by more than 18%". Nobody else in this field publishes a named Magento client, a described variant, a separated primary metric and a statistical reading in the same write-up.

The statistical reading is where scandiweb loses the criterion it should win. Byggmax test one reports "The chance to beat control with the variant in terms of this KPI equals 73%" and test two "equal to 92%", while the framework article on the same site sets the bar at "The eCommerce default is 95% (a 5% chance of a false positive)". Test three publishes no probability at all. Two wins called at 73 and 92 percent are below scandiweb's own published threshold, and the page says so rather than reprinting the headline. On this criterion MageCloud scores 16 and scandiweb 13.

The method is published in full, which is rare. The seven-step A/B testing framework carries a hypothesis format ("a falsifiable claim with one isolated change, one expected metric movement, and one reason the change should work"), a one-variable rule ("A/B testing earns its keep only when the two versions are isolated by one change"), a duration rule ("at least two business cycles" and "The test also needs to reach its calculated sample size, whichever is longer is the actual duration"), the sample-size formula written out as "n per variant ~ 16 x p x (1 - p) / d^2", a confidence-interval reading rule, a guardrail period after launch, and the refusal that matters most: "Do not lower the confidence threshold to 80% to make the math work, you are not running a test then, you are coin-flipping with a calculator."

A CORRECTION, printed rather than quietly fixed. The same article states "A defensible eCommerce default for a 5% minimum detectable effect at 95% confidence with 80% power is roughly 300 to 400 conversions per variant", and then prints the formula that contradicts it. Run that formula at the article's own example baseline of 8 percent with a 5 percent relative effect and it returns about 73,600 visitors per variant, which is roughly 5,900 conversions per variant, about fifteen times the published default. The formula is right and the rule of thumb beside it is wrong. It cost scandiweb three points here and it is the reason MageCloud takes the statistical-honesty criterion.

Ties Tom&Co at the top of the Magento mechanics criterion, from the other end of the same problem. Server-side generated A/B tests on a Magento 2 store is the only published account in this field of running an experiment inside Magento's own rendering path rather than over the top of it: "It became obvious quite soon, the server will have to be the one to track and switch page versions", then the implementation, "On the server side, it was enough to check for additional query parameter to decide on which layout to load", and the platform hook by name, "there's an event called 'layout_load_before', where we can append the layout changes we want. Even more conveniently, we use Magento's layout file-handles to load the appropriate layout files". It also records the constraint it avoided, that splitting traffic at the servers "might cause issues with the multi-instance server setup".

Volume and people behind the hypotheses. The conversion rate optimization service page publishes "858+ User tests run across client programs", "21 full market research projects", "150+ eCommerce stores optimized" and a practice that has run "as one continuous program since 2015", with moderated user testing, five-second tests, heatmaps and session recordings named as the inputs. Olga Kimalana is named as the lead, with an openable profile, "Head of Digital Experience", "11+ years of experience", and bylined articles including one on Magento homepage design. The framework article closes on "what we run across 1,000+ production tests".

The programme evidence, and its limits. A 23-test Adobe Target programme for a US wine distributor publishes the stopping rule as three conditions, "A duration of at least two weeks / Over 300 conversions for each primary segment of the variant / Statistically significant results", the analysis method, "using Frequentist and Bayesian statistical methods", a cadence of "8-10 tests per month" and "a 70% winning test ratio". That client is anonymised. The Cervera programme publishes "78% win-rate for A/B testing program" against a named Swedish retailer. Against that, Northerner at "+12% in checkout conversion rate" and Nicokick at "+5.6%" are labelled on the same page as comparisons against a previous period, not tests, which is honest and is scored as such.

The published limits are real and one of them is the wrong number. Checkout optimization prints a qualification test a buyer can run on themselves, "Two numbers decide this: how many sessions reach your checkout, and how many finish", and a not-relevant-if list that turns work away, including "You want a continuous testing program across every page". The framework article tells a merchant what to do when the traffic is not there, "widen the detectable effect (run for a 20% relative lift instead of 10%) or move the test up the funnel where volume is higher". A separate study publishes an engagement where testing could not conclude at all, at "around 400-500" daily visitors. Elsewhere, the cart abandonment guide is the diagnostic end of the same practice. One more note a reader should have: the portfolio still names Google Optimize as the testing tool on one older engagement with no date attached. Google's own help centre article, titled "[Sunset September 2023] Google Optimize", states that Optimize and Optimize 360 have been "no longer available as of September 30, 2023", and the framework article above gets that right and says so, so this is a dating problem on one page rather than an error. Bemeir, ranked tenth here, is the only other agency in this field to publish the retirement at all.

Publishes the single best test report found anywhere in this research, and it is the reason MageCloud beats scandiweb on statistical honesty. Its Thermos delivery progress bar test states the hypothesis under its own heading, the window, "we ran a controlled A/B test from 23 June to 4 July 2026", the allocation, "the original cart (12,792 visitors) and the variation with the progress bar (13,058 visitors) 25,850 visitors in total", the pre-declared metric, "The primary metric was completed checkouts", and the result with the baseline, the variant and the confidence on one line: "1.74% to 2.37% conversion rate, 99.75% confidence".

It then does the thing almost nobody does. Secondary metrics carry their own confidence figures, "+66.40% revenue per visitor (£0.50 to £0.83, 99.36% confidence)" and "+20.03% checkout starts (95.36% confidence)", and the metrics that failed are disclosed rather than dropped: "Later checkout steps (contact and shipping details) showed directional improvements that did not reach significance on their own". It also separates proof from interpretation, "the mechanisms themselves are interpretation rather than something the data directly proves", and states the operating principle, "Ideas earn their place on the site by winning a test, not by being someone's preference".

Where it loses is Magento specificity and depth of practice. The Magento CRO page publishes "12 YEARS OF EXPERIENCE", "MAGENTO CERTIFIED TEAM" and A/B testing as a service line, and the testing offer covers "Magento, Shopify, and WooCommerce", but no platform-specific testing mechanic was found on the pages read, and the Thermos page does not state which platform thermos.co.uk runs on. The case cards on the Magento page itself carry no test numbers. No testing tool is named for the Thermos test, no research volume and no named optimisation lead were found on the pages read, and the only published limit is that "Initial gains can appear within weeks".

The only agency here that publishes both a real Adobe Commerce practice and a fully designed experiment. Its SeaWorld conversion test publishes the allocation, "Each customer had an equal chance (33 percent chance each) of being in the Control, Variation 1, or Variation 2 group", both variants in words, the results by device, "a 4.7% lift in Conversion Rate, a 5.92% lift in RPV" on mobile against "2.74%" and "3.83%" on desktop, and the arm that lost: "Users in Variation 2 who were directed immediately to the Tickets landing page did not perform as well as those in Variation 1 or the Control." The stopping rule is stated, the experiment was complete "once the metrics reached statistical significance and traffic volume was high enough", but no confidence figure is attached to the lift, which is the four points between this entry and the top rung.

The method article behind it is the most complete published by any agency in the pool that also works on Adobe Commerce. Conversion testing: what brands really need to know publishes a hypothesis format with a worked example, the choice between A/B, split-URL and multivariate testing with the rule for each, the sample-size arithmetic as "Sample Size x Number of Variations in Your Experience = Total Number of Visitors You Need", and a numeric threshold as the house standard: "Optimization experiments typically run at a 90 percent statistical significance... Anything below 90 percent will increase the risk of deploying a 'loser' page or funnel." It also publishes what happens when nothing wins, "If there is no clear winner, then put a hold on deployment", and a seasonal refusal, "We do not recommend any significant UX changes during the holiday season".

The Adobe Commerce credentials are current and the testing content sits on separate pages from them. The commerce practice page publishes a solution "Built on Adobe Commerce and Adobe Experience Manager" that "has been accredited by Adobe", and the about page carries "Blue Acorn iCi Recognized as the 2020 Adobe Emerging Solution Partner of the Year". What is missing is the join: no Magento or Adobe Commerce testing mechanic was found on the pages read, and the named optimisation leads, a Head of Strategy and Analytics and a Director of Insights and UX, are quoted in body copy without individual profile pages. Most other case studies are download-gated, so SeaWorld is the only numeric test a reader can open.

Takes the test-design criterion outright, which nobody else in the top three manages. Its Splash About A/B test case study publishes the sequence, "a full CRO workshop, research and data collection, hypothesis generation, prioritisation matrix, design, A/B tests and implementation", a hypothesis in the exact words it uses, "Changing the Buy Now CTA button colour to a more vibrant colour will increase click-through rate because it stands out to the user more", and a prioritisation model with its three inputs named: "business value... implementation complexity... and certainty of improvement". The one-variable rule is published too, on the how-to article: "Ensure that only one element is changed at a time to isolate the impact of the tested variable."

The results are real and the platform is the catch. The same case study reports "a revenue increase of 4%, conversion rate increase of 15.44% and a revenue per customer increase of 22.55%", with the tool named, and states what happened next, "these tests... have been promoted to the main variant". But the client was moved to Shopify Plus before the testing began, so the published tests are not Magento tests. Magento work is published separately, including "Magento Hyva Theme & Hyva Checkout for Charles Bentley", and the CRO service page names "Adobe Silver Solution Partner and Shopify Plus Partner", but no Magento testing mechanic was found on the pages read.

The statistical half is the weak half. Significance appears as a requirement with no number, "set a sufficient test duration to achieve statistically significant results", and the low-traffic problem is named as a challenge rather than a floor: "achieving statistically significant results, especially for stores with lower traffic volumes... businesses can focus on high-impact areas". There is a published observation minimum, "Typically we need at least one month so that we can capture week-over-week trends", and a policy on tests that fail, "even inconclusive tests offer valuable learning opportunities", but no confidence figure, no sample size and no win rate were found on the pages read. Heatmap and behaviour tools are named; no research volume and no named CRO lead were.

Ties scandiweb at the top of the Magento mechanics criterion and is the only agency in the pool to name the failure mode that kills client-side testing on a commerce storefront. Its Adobe Target page puts it first: "Flicker, first. Nothing damages confidence in a testing programme faster than customers watching the page change under them. Getting the implementation right, pre-hiding, async loading tuned properly, server-side delivery where it matters, is the difference between a programme people trust and one the brand team quietly kills." It carries the headless case too, "On a decoupled storefront, client-side delivery gets awkward. Server-side and hybrid delivery are usually the right answer", and the inherited-instance diagnosis, "We usually start by auditing the implementation for flicker and delivery problems".

It also takes the published-limits criterion outright, with a number. "A site with 50,000 monthly sessions cannot run six concurrent tests and detect a 2% lift in anything. We size the roadmap to the traffic you have, and say so when the honest answer is that a test will never reach significance." And the guidance below that floor, which no other agency here supplies: "as a rough guide, tens of thousands of sessions a month per test if you are looking for single-digit conversion lifts. Below that, personalisation and targeting rules deliver more than classical A/B testing. We will tell you which camp you are in before you buy anything." On null results it is equally plain: "A test that shows no difference is a real result and it saves you building the thing."

What is missing is the evidence. No concluded A/B test, no per-test lift, no confidence figure, no sample size and no test duration were found on any page read. The only figure attached to testing is a band, "Typical clients see conversion uplifts in the 15-30% range over the first six months. That figure comes from the programme, not the software", and every named-client number, including "45% increase in mobile conversion rate" for one Adobe Commerce retailer, is a replatform or build outcome. The Adobe credentials are strong, "an official Adobe Solutions Partner and certified Adobe Commerce partner, with 30+ certified specialists", and a dedicated CRO service page returns 404. A buyer should ask for one concluded test read with its confidence figure before the first invoice.

Publishes the clearest hypothesis discipline of any consultancy in the pool. How to generate ideas for CRO tests sets the format, "usually in an 'If, Then, Because' format", and then works a real one end to end, from the insight, "From GA4, we noticed a high frequency of sessions who add an item to the cart, but do not complete the checkout", through the question and the experiment idea to the hypothesis itself. It names its research partner and its ideation sources, "Baymard Institute, Guess the Test, GoodUI and Evidoo", and publishes a corroboration rule, "We often try to corroborate any insight found by one method with another". Two named authors carry openable biographies.

Magento-specific testing is published, and it is old. Improving the conversion rate of Magento stores describes a real Magento checkout experiment, "we found that removing the header and footer and various other unnecessary elements of the checkout pages had a positive impact on abandonment rates, and the one step checkout did perform slightly better than our alternative (three steps, email, delivery, payment, executed via AJAX)", which is a test run against a Magento component and published as such. It also states the precondition nobody else states so plainly, "realistically you need to have a certain amount of traffic and trade to effectively use testing". The article carries a 2015 byline and recommends a Magento Enterprise full-page-cache module, so a reader should treat the tooling as historical and the mechanics as current.

The gap is numbers. No concluded test of theirs is published with a figure: the results that do carry figures are anonymised and attached to other work, "a luxury retailer... managed to increase their conversion rate by 50% through significant performance improvements". The service page publishes two tiers, CRO Establish and CRO Embed, with what each one tests, and an impartiality pledge, "we work with all technology partners and do not favour solutions based on commission". The only win rates on the site are quoted from a third party and clearly attributed as such. No significance figure, no sample size, no stopping rule and no losing-test policy were found on the pages read.

Publishes the most Magento-native experiment in the field: three checkouts tested head to head on one store. How we increased checkout conversion by 67.5% names the client, "Bedre Naetter is a Danish company, one of Europe's largest manufacturers and suppliers of beds, mattresses, sofas, quilts, pillows and bedding", and the arms, "We ran 3 simultaneous tests... React Checkout / Hyva Checkout (based on Magewire) / Luma Checkout". The design is published in four steps including "Random assignment: We then randomly assigned customers to one of three payment options", the measurement source is named, "data from New Relic", and the winner is reasoned rather than asserted.

The headline number is not the test result, and the page says which it is. "Between June and August 2024, Bedre Naetter achieved a 67.5% increase in checkout conversion rate compared to the same period in 2023" is a year-on-year comparison after the winner was rolled out. No in-test delta between the three checkouts is published, and the only statistical statement is "we used statistical tools to accurately compare the performance of the three payment options". That is why a test with genuinely good design scores 16 rather than 26 on the heaviest criterion.

The Adobe Commerce integration route is published as a tutorial, which is useful and undisciplined in equal measure. Starting A/B testing with Optimizely and Adobe Commerce walks a merchant through the snippet, the variation and the click tracking, then reads the result with no waiting period and no significance gate: "The results are visible immediately so you can quickly and easily check what works best for your business." Read against every vendor's own documentation, including Optimizely's own, that is the single most expensive habit in experimentation. Hatimeria is a Hyva Gold Partner with a headless Magento history going back to 2017 and three named authors; no traffic minimum, no win rate and no losing-test policy were found on the pages read.

Publishes three CRO case studies with numbers, one platform sentence nobody else writes, and no statistics at all. The CRO and A/B testing service page is the only page in this entire field to say that the platform bounds the recommendation: "we have deep experience working within the constraints and architectures of Shopify Plus, Adobe Commerce, and Magento, ensuring our CRO recommendations are scoped to what your specific platform allows". It names its experimentation stack, "Optimizely, GrowthBook, VWO", and its retainer scope includes "Experiment Design & Implementation", "Experiment Monitoring & Reporting" and "Experimentation Governance".

The Perfect Water case study names the client and the method, "Key pages were audited to identify friction points, followed by hypothesis-led A/B testing across product and landing pages using VWO", and reports "38% improvement in conversions achieved following optimized layout and messaging" and "18% reduction in form abandonment". A second case names XLfeet and reports "27% increase in Add to Cart following validated PDP fit content restructuring" and "19% lift in checkout completion", with "Controlled A/B tests were conducted" and "Only validated improvements with measurable conversion gains were deployed".

Two things a buyer should press on. The headline figure on the Perfect Water case is "11,179% lift in high-impact conversion over the baseline experience (VWO A/B Testing)", published with no baseline value, no sample size, no duration and no confidence figure, and a lift of that magnitude on a purchase-adjacent metric is unusual enough to need its arithmetic shown. And the XLfeet page opens "XLfeet is a niche U.S. retailer specializing in large-size footwear" while a pull-quote on the same page describes "A footwear brand from Europe", with a returns figure that appears as 32 percent in one place and 41 percent in another, so a reader cannot reconcile the two from what is published. No significance level, no sample size, no test duration, no win rate and no traffic minimum were found on the pages read, and none of the three CRO cases names its platform.

The strongest payment-step evidence in the pool, and it is not a test. The 3DS case study names the client and the problem, "Choice Shops operate a number of Magento sites" whose healthcare customers "found the 3DS authentication process confusing or challenging", then the result with its window: "In the last two months, we noticed a 77% reduction in 3DS rejections after implementing the exemption. This translated to an increase of around 300 orders per week, or a 4% increase in sales." That is a before-and-after over two months, which the page states plainly, so it is scored as a period comparison rather than as a test.

It publishes something rarer than a lift: the risk it took on. "It's important to note that requesting a 3DS exemption comes with the shift of liability for any fraudulent transactions back to the merchant. To mitigate this risk, we set a cap of £100 for exemption requests." Two further checkout results appear with the client withheld, "a 3% lift in conversion by deleting just two fields" and "20% of failed submissions resolved at the first attempt simply by improving error handling", alongside a fully worked uplift model on 100,000 sessions a month and a published engineering threshold, "Keep the checkout route under 150KB of JavaScript".

Nothing behind it is a published testing practice. No hypothesis format, no primary metric, no one-variable rule, no duration or sample-size rule, no prioritisation model and no statistical statement of any kind were found on the pages read, and the sitemap contains no CRO or experimentation service page. Adobe Target is covered at length in an Adobe Target article and the advice on Magento checkouts is to "A/B test different optimisation strategies and listen to what your data tells you", which is the right instruction with no method attached. Worth knowing: three of the insight posts carry a joke byline rather than a real author, so those are not scored as named people.

Publishes the best architectural thinking about testing on Magento and not one test result. Customization flexibility and conversion optimization opens on the mistake it sees most: "Most conversion optimizers think of A/B testing tools as a layer you add on top of your platform. Install Optimizely or VWO, paste a snippet, start testing. That works for simple visual tests, but it breaks down when you need to test structural changes to the buying experience." It then publishes a four-level table, from "Visual overlay... Client-side JS snippet" through "Component-level... Theme-level branching with feature flags" and "Flow-level... Platform API integration with server-side routing" to "Architectural... Backend customization with data layer segmentation", and states its own build practice, "feature flagging at the theme level, server-side experiment routing for checkout flow tests".

The tool review adds the platform-specific caveat that matters and the retirement nobody else names. On tooling for a customised storefront: "For heavily customized Magento or Shopware storefronts with complex JavaScript-driven interactions, the visual editor sometimes produces unreliable variations. In those cases, developer-implemented variations using VWO's code editor are more reliable." And on the tool half this field still recommends, it is the only agency here to state the fact: "Google Optimize was retired in September 2023, leaving a gap in the free A/B testing market." It also publishes a numeric revenue floor with the alternative below it, behaviour analytics under roughly five million dollars and "adding VWO for A/B testing" above it, plus test-volume bands by tool.

The scoring consequence is blunt. No concluded test, no client with a conversion figure, no hypothesis, no primary metric, no duration or sample-size rule, no significance statement and no win rate were found on the pages read, and both a CRO service page and a services page return 404, so the testing content lives entirely in articles. One published passage reads like a record and is not: a contrast between a "flexible team" that "has run fifty tests, found eight winners" and a rigid one is an illustrative scenario in context, and it is not scored as Bemeir's own result. The Optimizely partner page still carries a 2022 copyright line and describes the product under its pre-rebrand name.