Growth Experimentation Velocity

Build a high-output engine to compound small wins into massive growth.

Help the user with growth experimentation velocity using insights from 10 guests and posts across Lenny's Podcast and Newsletter.

How to Help

1. Establish the Baseline - Analyze current conversion funnels and identify the single North Star metric to focus on.
2. Prioritize and Plan - Use frameworks like ICE or RICE to rank experiments by impact and engineering cost.
3. Execute and Iterate - Launch scrappy tests quickly to find signals of life before scaling into robust features.
4. Scale and Socialize - Systematize the sharing of wins and failures across the organization to multiply the impact of every insight.

Core Principles

Search for signs of life

Timothy Davis: "You can always do a very, very small test. You can just put a little money into a platform, see if there's a sign of life. If there is, then you can pull back and say, 'Okay, we have signs of life. Now let's build a campaign around that.'"

Validate new channels or ideas using low-budget tests and narrow match thresholds before committing significant resources.

Embrace the counterfactual

From "How today’s top consumer brands measure marketing’s impact": "Testing/conversion lift studies (CLS): regularly run by marketers to validate what performance would look like if you switched a channel off, or scaled spend up or down."

Use randomized testing and lift studies as the gold standard to observe what would happen without your intervention.

Leverage compounding effects

From "The secret to Duolingo’s exponential growth": "To get the best long-term gains, you should always have a sense of urgency. The quicker you launch winning experiments, the quicker those changes impact your growth. Not only that, but these improvements compound!"

Focus on high experiment velocity because early small wins multiply over time into significant competitive advantages.

Optimize psychological commitment

Jackson Shuttleworth: "We've actually set up really good infrastructure for copy testing. We used to say continue, our standard CTA is continue, and we changed that to commit to my goal, and it was a massive win."

Shift from generic microcopy to intentional language that reinforces the user's specific goals and psychological state.

Lower friction with scrappy tools

From "Fostering a culture of experimentation": "When systems are still in flux, you don't want to overinvest in tooling that will become outdated immediately when your data schema gets updated or some other piece of infrastructure changes. However, it is essential to have a way to rapidly iterate, and that means quick access to experiment results data. So if you need to in the early days, build something simple and scrappy at first, and over time evolve it to support the team's needs."

Prioritize rapid iteration over perfect infrastructure by starting with simple internal tools to prove the value of testing.

Templates & Frameworks

• EVELYN (Experiment Velocity Engine Lifting Your Numbers) - Airtable Template (Introducing DRICE: a modern prioritization framework) - A batteries-included Airtable template for managing growth experiment prioritization using RICE/DRICE
• Noom's Experimentation Velocity Principles (How to win in consumer subscription) - A set of operating principles for running a high-velocity experimentation program in growth
• 4-Step Conversion Optimization Process (Prioritizing conversion opportunities) - A structured end-to-end process for identifying, prioritizing, executing, and learning from conversion optimization work
• Experiment Design Template (Breaking into growth) - A Google Doc template for designing and running growth experiments
• Impact and Learnings Review Meeting (Ben Williams) - A weekly document and meeting structure used by growth teams to discuss and socialize experiment learnings.
• 6 Guidelines for Experiment Urgency (The secret to Duolingo’s exponential growth) - Tactical guidelines for moving quickly on experiments to maximize compound growth, used at Duolingo
• Growth Ideas Brainstorming Framework ('How Might We…?') (Growth ideas) - A facilitation approach for running team brainstorming sessions where you go through a categorized list of growth ideas and apply 'How might we…?' framing to ge
• Conversion Optimization Decision Tree: Experiment vs. Ship (Strategy and tactics for increasing conversion) - Guidance on when to A/B test conversion changes vs. when to just ship them

See references/artifacts.md for the full list with details.

Questions to Help Users

• "What is the single North Star metric you are currently trying to move?"
• "How many experiments are you currently running per week?"
• "What is the estimated engineering cost versus the predicted impact for your top three ideas?"
• "Do you have a standardized process for sharing experiment learnings across the whole team?"
• "Is your team autonomous enough to launch experiments without multi-level approvals?"
• "What percentage of your user base actually encounters the flow you are planning to optimize?"

Common Mistakes to Flag

• Waiting for silver bullets - Teams often stall growth by looking for one massive feature instead of accumulating many small optimizations.
• Paralysis by testing - Applying rigorous A/B testing to every minor change can slow down execution if there is not enough data volume.
• Ignoring the addressable pie - Failing to factor in how many users actually see a change leads to overestimating the real-world impact.
• High-friction approvals - Requiring multiple levels of sign-off for experiments kills the momentum needed for a high-velocity culture.

Deep Dive

For all 16 sourced insights from 10 guests, see references/guest-insights.md

Related Skills

• Growth Model
• Acquisition Channels
• User Onboarding Activation
• Retention Engagement

---

Supporting file: references/artifacts.md

Growth Experimentation Velocity - Frameworks, Templates & Checklists

23 artifacts extracted from Lenny's Podcast and Newsletter

Frameworks

Conclusive Experiment Design (Failure)

A method for designing growth experiments in low-volume (B2B) environments.

How it works: When sample size (N) is low, maximize the treatment effect by combining all possible levers (trigger, text, personalization, design) into one test. If it fails, you can conclusively discard the hypothesis. If it succeeds, you can cost-rationalize later.

Conversion Optimization Decision Tree: Experiment vs. Ship (Strategy and tactics for increasing conversion)

Guidance on when to A/B test conversion changes vs. when to just ship them

How it works: Decision process:

1. Check if you have sufficient volume to run experiments — Use a sample size calculator (e.g. Optimizely's)
2. IF YES: Experiment with as many ideas as you can. Don't bet on just a few amazing ideas — volume of experiments matters.
3. IF NO (not enough scale): Do your due diligence: a) Talk to users b) Evaluate upside potential c) Then feel comfortable trusting yourself to go with your best guesses and ship without an experiment

Effort vs. Reward Evaluation for Experiments (Tim Holley)

Mental model for evaluating experiment wins not just by absolute impact but by the effort required to achieve them

How it works: Consider: 1) What was the GMS/conversion win? 2) What was the effort (e.g., one-line text change vs. major feature build)? Example: Adding 'Etsy offsets carbon emissions from every delivery' to the cart — a single line of copy that drove huge unexpected conversion uplift. Small effort, outsized reward.

Growth Channel Experimentation Guardrails (Yuriy Timen)

Approach for testing growth channels without premature abandonment: set objective KPI benchmarks, try multiple creative angles, and define clear continue/abandon criteria

How it works: Two types of channels: 1) Channels with objective guardrails - set minimum impressions, try 2-3 creative angles, define target CTR ranges. If within ranges, keep going; if not, abandon (but abandonment doesn't mean never revisit). 2) Channels requiring more judgment - evaluate incremental lift of next phase vs. opportunity cost of other high-profile things to try. Periodic reevaluation applying sunk cost concept.

Growth Ideas Brainstorming Framework ('How Might We…?') (Growth ideas)

A facilitation approach for running team brainstorming sessions where you go through a categorized list of growth ideas and apply 'How might we…?' framing to generate specific experiments for your product.

How it works: Steps:

1. Gather your team together
2. Pull up the categorized growth ideas list (6 categories: short-term top-of-funnel, long-term top-of-funnel, monetization, conversion/activation, retention, team velocity)
3. Go through each idea with the mindset of 'How might we…?' — translating each general idea into a specific experiment for your product
4. Prioritize the resulting ideas based on impact and effort

The framework ensures coverage across the full growth funnel rather than fixating on one area.

Lead Bullets vs. Cannonballs Portfolio (Adriel Frederick)

A mental model for balancing growth experiments and investments.

How it works: Allocate resources between 'lead bullets' (small, incremental experiments and optimizations) and 'cannonballs' (massive, fundamental product changes like SMS phone registration). Early stage products should be 100% cannonballs; mature products shift to a mix like 80% cannonballs / 20% lead bullets to avoid incremental laziness.

Marketing vs. Product Cost of Failure (Raaz Herzberg)

A mental model for understanding why marketing should be high-experimentation versus product's careful, additive approach

How it works: Product features: High cost of failure — consumes engineering time (most valuable resource), can never truly be removed, complicates the product permanently, every new feature must work with all existing features. Marketing activities: Low cost of failure — no maintenance, no technical debt, if a LinkedIn video flops, nothing happened, post a different one tomorrow, no permanent consequences. Implication: In marketing, try everything. In product, think very hard before adding anything.

Noom's Experimentation Velocity Principles (How to win in consumer subscription)

A set of operating principles for running a high-velocity experimentation program in growth

How it works: 1. Run a lot of tests — Up to 6 per week per PM at highest velocity. More learnings lead to improved performance faster. 90% of ideas failed, so you need to know quickly.

1. Always have an experiment running — Night/weekend launches are common. Never waste your bottleneck resource (traffic/users).

2. Aim for large minimum detectable effect (MDE) — Only target +20-30% improvements. Don't sweat the small stuff.

3. Optimize for fast learning over quality — Companies underestimate the power of experimentation volume. High-effort experiments don't necessarily increase win rate or size. You lose out on learnings by going slow.

4. Match speed and quality to what you're validating — Appropriate level depends on risk; no one size fits all.

Infrastructure requirements: Sandbox environment for rapid experimentation, modularized website, Optimizely, Mixpanel, fully autonomous growth team (1 PM, 1 eng, 1 designer) with full ownership of tech stack.

Noom's Growth Infrastructure Stack (How to win in consumer subscription)

The testing infrastructure and team structure Noom used to enable rapid experimentation in paid acquisition

How it works: Infrastructure components:

• Sandbox environment designed for rapid experimentation
• Modularized website (for easy A/B testing)
• Optimizely (experimentation platform)
• Mixpanel (analytics)

Team structure:

• Fully resourced and autonomous growth team
• One PM, one engineer, one designer
• Full ownership of tech stack
• Didn't have to bother anyone to get stuff done

Prerequisites:

• Stakeholder buy-in from the start
• Super-high autonomy
• Full ownership of paid marketing (ads → funnel)

Turbo Boost Anatomy Template (60 ideas to boost your growth)

A consistent structure used throughout the newsletter to evaluate and plan turbo boost tactics, applicable to any new growth experiment

How it works: For each turbo boost, document:

1. Company name and tactic name
2. Category (viral video, mini-product/drop, limited-time offer, influencer promotion, co-marketing, offline experience, pick a fight)
3. Description of the tactic
4. Distribution channel(s): How it reached people (PR outlets, social platforms, partnerships, existing customers, etc.)
5. Why it worked: The core psychological or situational driver (e.g., surprise, controversy, humor, FOMO, great timing, trust in endorser, solved a real problem, social flex, free money, David vs Goliath narrative)

Templates

EVELYN (Experiment Velocity Engine Lifting Your Numbers) - Airtable Template (Introducing DRICE: a modern prioritization framework)

A batteries-included Airtable template for managing growth experiment prioritization using RICE/DRICE

How it works: Full Airtable template for tracking and scoring growth experiments. Includes RICE scoring fields and more comprehensive tracking. URL: https://www.airtable.com/universe/expZpCNVlkaoLGNAr/evelyn-experiment-velocity-engine-lifting-your-numbers

Experiment Design Template (Breaking into growth)

A Google Doc template for designing and running growth experiments

How it works: Google Doc template linked at: https://docs.google.com/document/d/1aPAOEvYbT9gsSUUtucANjcKQf1J5CqMwLgbMhuAiQxY/edit — Referenced as a starting point for designing and running growth experiments as practical hands-on practice

Impact and Learnings Review Meeting (Ben Williams)

A weekly document and meeting structure used by growth teams to discuss and socialize experiment learnings.

How it works: Run by the PM. Zero time is spent reviewing what the team actually did. The entire meeting is focused on discussing documented learnings from data exploration, experiments, and user research, their implications, and how they can be leveraged by other teams.

Checklists

4-Step Conversion Optimization Process (Prioritizing conversion opportunities)

A structured end-to-end process for identifying, prioritizing, executing, and learning from conversion optimization work

How it works: Step 1: Audit your funnel to identify potential reasons for drop-off 1a. Research your audience — understand what people want and why they're using the experience. Talk to marketing teams about which messaging is most effective to understand motivations. 1b. Go through your own flow — put yourself in the shoes of major customer groups. Where do you get stuck, confused about what to do next, or distracted? Does messaging speak to core motivations? Use different devices and browsers to try to break the experience. 1c. Study others — watch non tech-savvy friends or older relatives try to get through the experience. Where do they get stuck or confused? 1d. Look at the data — examine funnel step by step, compare across user groups (device, browser, demographic, channel, etc.). Look for sharp, unexpected, or inconsistent drop-offs. Identify conditions that decrease or increase conversion (page load speed, # of results, etc.).

Step 2: Convert your hypotheses into concrete ideas

Step 3: Prioritize & execute based on ROI 3a. Estimate impact, probability of success, and cost (often in terms of engineering days). 3b. Generally do the highest expected impact / cost items first.

Step 4: Create a strong feedback loop 4a. Run lightweight tests to reduce risk for costly items. 4b. Go deep where you find heat — every shipped project is an opportunity to learn what works or doesn't. 4c. Re-evaluate ROI after getting feedback. If you see standout results around a particular hypothesis, double down with additional related feature work. If you see diminishing returns, consider changing focus.

6 Guidelines for Experiment Urgency (The secret to Duolingo’s exponential growth)

Tactical guidelines for moving quickly on experiments to maximize compound growth, used at Duolingo

How it works: 1. Make a decision AS SOON AS you're confident you have the data to do so. Don't let experiments sit on the back burner to 'collect more data.' Regularly examine experiments to know when you have sufficient data. 2. Roll out experiments to as many users as you can, as quickly as you can. Increase rollout percentage as soon as you're confident a change is safe. The sooner you get data, the sooner you launch and start compounding. 3. Don't pull your punches. If an imperfect version is strictly an improvement, launch it to users (starting compound gains) and then iterate with follow-up experiments. Don't shut down, tweak, and re-run. 4. Consider ROI when determining the ORDER of experiments. Do the most efficient, high-ROI projects FIRST because they have a longer period of paying compound interest. Order matters, not just the quarterly list. 5. Launch on your biggest platforms first. If Android has more users, launch there first to get bigger compound gains, then port to iOS. 6. Port the wins quickly. At Duolingo, ~50% of experiments launch overall, but ports are close to 100% launched. Nearly everything that wins on one platform wins on the other.

How to Run a Conversion Lift Study (How today’s top consumer brands measure marketing’s impact)

Step-by-step guide for running incrementality experiments, from in-platform tests to geo-region experiments

How it works: Option 1: In-Platform Lift Tests (easiest starting point)

• Use Meta's native CLS solution or Google's equivalent
• Limitations: Only tells you about one channel; not every channel offers it; smaller channels may only offer if you're a big spender; some channels technically can't (e.g., can't de-rank from Google search for some users)

Option 2: Geo-Region Tests (most common for independence from platforms)

• Divide audience into geographical regions (cities, counties, states)
• Turn off ads in some regions to measure incremental impact
• Use Meta's open source GeoLift library for calculations and synthetic control estimation
• Benefits: Works independently of platform; don't have to trust platforms to grade their own homework

Advanced Techniques:

• Test various channel combinations (turn off some channels but not others)
• Run graduated scale tests: increase spend in steps in some regions but not others to find saturation/diminishing returns point
• This reveals how much headroom you have to expand investment and at what point it becomes too expensive

Cadence Recommendations:

• Major channels (80%+ of budget): Quarterly
• Smaller channels: Annually
• More regular testing as budget grows into millions

Critical Warning: Incrementality tests are just a snapshot in time. A channel found 90% incremental can drop to <20% incremental just 4 months later (e.g., if the ads team drops the setting that excludes existing customers).

Examples

EarnUp: Round-Up Mortgage Payment Nudge (How behavioral science can boost your conversion rates)

Real case study of using round numbers to increase mortgage overpayment adoption by 40%

How it works: Company: EarnUp (fintech app) Problem: Users not paying down debt faster Goal: Get users to pay more than minimum payment

Experiment: A/B email test

• Control: Asked users to increase their mortgage payments
• Treatment: Asked users to 'round up' to an even number

Results:

• Control: 10% of people increased payment by $60, saving $8,000 in interest, reducing mortgage by 2 years
• Treatment (round up): 40% increase from baseline in people choosing to overpay
• Impact: Additional 2 years shaved off mortgage on average

Why it worked: People already mentally round up loan amounts. The round-up feature aligned with existing mental behavior and made it psychologically easy.

Email Opt-in Copy Experiment (25% to 55% opt-in rate) (How GiveDirectly increased donations by over $3 million/year through experimentation)

Changing checkbox copy to be transparent about email frequency and value doubled the email subscription rate during checkout.

How it works: Before copy: Generic/unclear email opt-in checkbox (25% opt-in rate). After copy: Updated to be upfront about how many emails donors should expect to get and why they should subscribe (55% opt-in rate). Impact: Subscribers retain 2-3x as long as non-subscribers. Conservatively estimated at $400,000/year in incremental donations. Key insight: Transparency about email frequency increases trust and opt-in rates.

Google Search: Increasing sticky search awareness (How to accelerate growth by focusing on the features you already have)

Real example of how Google's growth team nearly doubled usage of high-value search types by focusing on existing features rather than building new ones

How it works: Context: In 2015, the average Google Search user in the U.S. did only 12 searches per month. More searches = more ad opportunities = more revenue.

Analysis: Searches whose results changed frequently (sports scores, weather, movie times) were the 'stickiest' — users who did them repeated them often. But only ~15% of users were aware of these search types.

Approach: Instead of building new features, the team focused on raising awareness and engagement with existing sticky search types using ARIA principles (Reduce friction, Introduce in context, Assist).

Result: Nearly doubled the number of users doing these types of searches, translating into millions of dollars of incremental ad revenue.

Lyft Friday Testing Ritual (What They'd Do Differently 🔮 Kickstarting and Scaling a Marketplace Business)

A lightweight weekly testing practice where the Lyft growth team ran experiments over Friday drinks using Optimizely

How it works: Process: Every Friday after work, over drinks, the Lyft growth team used Optimizely to test changes on core landing pages. Cadence: 1-2 tests launched per week. Constraint: Each change had to be under an hour of work. Scope: Tested across emails, website, and app. Results: Found insights like 'this type of messaging resonates a lot more with drivers,' which informed the next set of big changes. Key benefit: Low effort, high influence on product direction.

Monthly Donation Default Experiment ($500K/year lift) (How GiveDirectly increased donations by over $3 million/year through experimentation)

Two iterations of encouraging monthly donations—a modal that failed and a default change that succeeded.

How it works: Context: Average LTV of a recurring donor is 12x that of a one-time donor.

Failed approach: Modal asking people to switch to monthly ($X/12) after starting a one-time donation of $X. Result: 7x more likely to start monthly donation, BUT 10% decrease in one-time donation conversion. Net negative—increase in monthly gifts didn't compensate for decrease in one-time gifts.

Successful approach: Changed default donation frequency from one-time to monthly. Result: One-time donations decreased slightly, but lifetime value of increased monthly donations outweighed the decrease. Net positive: ~$500,000/year incremental.

Key insight: Modals add friction. Changing defaults is frictionless and more effective.

Transaction Fee Coverage Experiment ($200K/year lift) (How GiveDirectly increased donations by over $3 million/year through experimentation)

Two iterations of asking donors to cover payment processing fees—checked by default failed, unchecked by default succeeded.

How it works: Context: GiveDirectly pays ~$400,000/year in payment transaction fees.

Failed approach: Checkbox to cover processing fees, CHECKED by default. Result: 85% of completing donors covered fees, BUT overall donation conversion rate dropped 6%. Hypothesis: People were put off by seeing a different (higher) amount on the donate button than they intended.

Successful approach: Checkbox UNCHECKED by default. Result: 60% of donors opted in to cover fees (down from 85%), BUT the conversion decrease disappeared. Net positive: ~$200,000/year in recovered fees.

Key insight: Pre-checked boxes that change the user's expected total feel manipulative. Unchecked opt-in respects user agency while still capturing majority of the upside.

Tools

Recommended Reading List for Conversion Optimization (Strategy and tactics for increasing conversion)

Five curated articles for going deeper on conversion optimization

How it works: 1. 'A/B Testing' by Julian Shapiro — https://www.julian.com/guide/growth/ab-testing 2. 'Conversion Rate Optimization' by Moz — https://moz.com/learn/seo/conversion-rate-optimization 3. 'The Conversion Optimization Rulebook' by Unbounce — https://unbounce.com/conversion-rate-optimization/rulebook/ 4. 'Psych'd: A new user psychology framework for increasing funnel conversion' by Darius Contractor (via Andrew Chen) — https://andrewchen.co/psychd-funnel-conversion/ 5. '13 Examples of Re-Engagement Emails' by HubSpot — https://blog.hubspot.com/marketing/10-examples-of-effective-re-engagement-emails

---

Supporting file: references/guest-insights.md

Growth Experimentation Velocity - All Guest Insights

10 sources, 26 insights

---

Albert Cheng

Insight: To scale the impact of experimentation, treat every individual win as a blueprint that adjacent teams must audit and apply to their own product areas.

Tactical advice:

• Establish a consistent system for sharing experiment insights across the organization to prevent 'siloed' learning.
• Scale proven psychological wins 10X by replicating successful patterns across different features and user segments.
• Oscillate between an exploratory phase of finding 'the right mountain' and an exploitation phase of scaling those proven insights.

Source: How to find hidden growth opportunities in your product | Albert Cheng (Duolingo, Grammarly, Chess.com) (https://www.youtube.com/watch?v=2BKmNmnEj9w) @ 00:55:36

---

Ben Williams

Insight:

Source: How Snyk built a product-led growth juggernaut | Ben Williams (VP of Product at Snyk) (https://www.youtube.com/watch?v=21sFTZzIfUk) @ 01:06:50

---

Deb Liu

Insight: Product growth is driven by a steady accumulation of small optimizations and experiments rather than by searching for a single silver-bullet feature.

Tactical advice:

• Focus on a high volume of small experiments to find the 'inches' that aggregate into massive growth.
• Avoid stalling growth by waiting for a single step-function change that may never come.
• Continuously optimize the core user journey to ensure sustainable long-term expansion.

Source: How to own your career growth and become a powerful product leader | Deb Liu, Ancestry (ex-Facebook, PayPal) @ 00:46:34

---

Elena Verna 3.0

Insight: Rigorous experimentation can become a 'paralyzing disease' if applied to every minor initiative or used without a sufficient volume of data to drive insights.

Tactical advice:

• Avoid requiring A/B tests for every roadmap item to prevent operational paralysis.
• Shift from shipment-focused to experiment-focused growth only once you have a high volume of users.
• Base experimentation on specific hypotheses derived from deep data analysis rather than random testing.

Source: Elena Verna on how B2B growth is changing, product-led growth, product-led sales, why you should go freemium not trial, what features to make free, and much more @ 01:05:59

---

Failure

Insight:

Source: Failure (https://www.youtube.com/watch?v=9euy9gC48lc) @ 00:33:38

---

Jackson Shuttleworth

"We've actually set up really good infrastructure for copy testing. We used to say continue, our standard CTA is continue, and we changed that to commit to my goal, and it was a massive win."

Insight: Moving from generic microcopy to language that reinforces a user's psychological commitment can lead to substantial increases in conversion and retention.

Tactical advice:

• Audit standard CTAs for opportunities to use more intentional, goal-oriented language.
• Invest in infrastructure that allows product teams to test copy variations independently.
• Align microcopy with the user's stated objectives to increase the perceived value of an action.

Source: Behind the product: Duolingo streaks | Jackson Shuttleworth (Group PM, Retention Team) (https://www.youtube.com/watch?v=_CCwoQZH5hI) @ 00:26:49

---

Laura Schaffer

Insight: Higher experimentation velocity is achieved by lowering statistical bars and launching iterative, simple tests that prioritize speed over perfection.

Tactical advice:

• Lower confidence intervals strategically to launch and conclude more tests.
• Run 'embarrassing' or low-fidelity experiments to fail faster and learn.
• Prioritize iteration speed to increase the total volume of successful outcomes.

Source: Career frameworks, A/B testing mistakes, counterintuitive onboarding tips, selling to developers | Laura Schaffer (VP of Growth at Amplitude) (https://www.youtube.com/watch?v=UMAmj1bciww) @ 00:34:26

---

Lenny Rachitsky

Insight: Modeling the potential impact of an optimized conversion rate helps justify prioritizing specific growth loops over other acquisition channels.

Tactical advice:

• Use data modeling to estimate how a lift in demand-to-supply conversion rates would impact overall organic growth.
• Focus experimentation efforts on the specific months where data shows demand-side users are most likely to convert.
• Run product experiments that aim to increase user awareness of the full supply-side product offering.

Source: Demand driving supply: The little-understood growth loop behind a surprising number of iconic billion-dollar companies (https://www.lennysnewsletter.com/p/demand-driving-supply-the-little-understood-growth-loop-behind-a-surprising-numb)

---

"Testing/conversion lift studies (CLS): regularly run by marketers to validate what performance would look like if you switched a channel off, or scaled spend up or down."

Insight: Randomized testing is the gold standard for marketing measurement because it provides the only definitive way to observe the counterfactual.

Tactical advice:

• Run 'lift' studies regularly to validate the ROI reported by your attribution platform.
• Experiment with switching specific channels off or scaling spend to observe the actual change in sales.
• Use platform-native testing tools or geo-lift tests to establish causal proof of conversion.

Source: How today’s top consumer brands measure marketing’s impact (https://www.lennysnewsletter.com/p/how-todays-top-consumer-brands-measure-marketings-impact)

---

"To get the best long-term gains, you should always have a sense of urgency. The quicker you launch winning experiments, the quicker those changes impact your growth. Not only that, but these improvements compound!"

Insight: High experiment velocity creates a compounding effect where early wins continuously multiply future growth gains over time.

Tactical advice:

• Launch winning experiments immediately once you have statistical confidence in the data.
• Increase experiment rollouts to the maximum number of users as quickly as safety allows.
• Prioritize porting wins to other platforms over fine-tuning iterations to start compound gains sooner.

Source: The secret to Duolingo’s exponential growth (https://www.lennysnewsletter.com/p/the-secret-to-duolingos-exponential-growth)

---

Insight: Executing real-world growth experiments builds the practical skills and internal credibility needed to transition into a formal growth role.

Tactical advice:

• Use a standardized experimentation template to document your hypotheses, methods, and key learnings.
• Execute a handful of small-scale experiments to demonstrate your ability to handle the end-to-end growth process.
• Present the results of your experiments to stakeholders to build a reputation as an effective, data-driven practitioner.

Source: Breaking into growth (https://www.lennysnewsletter.com/p/breaking-into-growth)

---

"When systems are still in flux, you don't want to overinvest in tooling that will become outdated immediately when your data schema gets updated or some other piece of infrastructure changes. However, it is essential to have a way to rapidly iterate, and that means quick access to experiment results data. So if you need to in the early days, build something simple and scrappy at first, and over time evolve it to support the team's needs."

Insight: High-velocity experimentation requires reducing friction in the testing process, starting with scrappy internal tools and evolving into robust infrastructure as the company scales.

Tactical advice:

• Start with simple, scrappy experimentation tools to enable rapid iteration without over-engineering early infrastructure.
• Prioritize quick access to experiment results data to ensure teams can learn and move fast.
• Devote a full-time team to building and supporting data tooling once the experimentation volume justifies it.

Source: Fostering a culture of experimentation (https://www.lennysnewsletter.com/p/fostering-a-culture-of-experimentation)

---

Insight: Effective brainstorming sessions leverage structured prompts and categorized lists to shift the team's perspective toward actionable, specific solutions.

Tactical advice:

• Gather the team to review a curated list of growth strategies across acquisition, monetization, and retention.
• Adopt a “How might we...?” mindset to reframe general growth tactics into specific company initiatives.
• Narrow the team's focus during the session to identify the highest-impact ideas to test immediately.

Source: Growth ideas (https://www.lennysnewsletter.com/p/growth-ideas)

---

"When it comes to product nudges and defaults, it’s worth testing a few iterations to see if you can find a win. Staying focused on our North Star metric (total dollars raised) also allowed us to recognize when to accept decreases in other submetrics."

Insight: A successful experimentation strategy balances low-effort optimizations with high-risk bets while maintaining focus on a single North Star metric to navigate trade-offs.

Tactical advice:

• Prioritize experimentation on a single critical flow where tiny changes can have outsized impact.
• Iterate on failed tests by trying second versions with different default settings or copy.
• Stay focused on the primary goal (e.g., total revenue) rather than optimizing submetrics in isolation.

Source: How GiveDirectly increased donations by over $3 million/year through experimentation (https://www.lennysnewsletter.com/p/how-givedirectly-increased-donations-by-over-3-millionyear-through-experimentati)

---

"We invested heavily in our testing infrastructure. For example, we built a sandbox environment designed to allow us to rapidly experiment (e.g. a modularized website, Optimizely, Mixpanel, etc.). To do this, we had stakeholder buy-in from the start, and super-high autonomy."

Insight: High-velocity experimentation is only possible when a growth team has full ownership of their tech stack and a modular environment that doesn't rely on core engineering for every change.

Tactical advice:

• Form an autonomous growth team consisting of one PM, one engineer, and one designer.
• Build or implement a sandbox environment to test ads and funnels without bothering other departments.
• Prioritize ruthless experimentation on your performance marketing engine to hit one-month payback targets.

Source: How to win in consumer subscription (https://www.lennysnewsletter.com/p/how-to-win-in-consumer-subscription)

---

"Overall, we found that Dropbox teams that adopted DRICE were able to move their key metric by twice as much as teams that stuck to a simpler prioritization process."

Insight: Rigorous prioritization allows teams to ignore 'favourite' low-impact ideas and discover 'nice-to-have' features that are actually massive drivers of growth.

Tactical advice:

• Run the laziest possible tests to validate high-impact ideas with the least amount of engineering work.
• Look for projects that address sensitive-to-change portions of the customer journey, like landing page pricing.
• Increase confidence scores for ideas where competitors are already using the tactic or customers are repeatedly complaining.

Source: Introducing DRICE: a modern prioritization framework (https://www.lennysnewsletter.com/p/introducing-drice-a-modern-prioritization-framework)

---

Insight: Before pivoting your entire company toward a single growth engine, you must validate that the channel shows early data signals of efficacy.

Tactical advice:

• Run small-scale experiments for 3-6 months to look for a 'trickle' of results.
• Analyze early traffic to ensure it converts and retains at an acceptable rate before scaling spend.
• Make a 'company betting decision' to commit resources only after you have enough data to analyze the channel's potential.

Source: Kickstarting and scaling a consumer business—Step 6: SCALE: Build your growth engine (https://www.lennysnewsletter.com/p/kickstarting-and-scaling-a-consumer-business-step-6-scale-build-your-growth-engi)

---

"One thing you’ll notice is that although there are many ways to look at it, in the end it always comes down to your best guess at ROI — the ratio of (1) effort to launch, and (2) expected impact."

Insight: Regardless of the framework used, prioritization of conversion experiments should be driven by the ratio of engineering cost to the predicted improvement in user behavior.

Tactical advice:

• Create an Impact Calculator to structure thinking and quantify potential gains before starting work.
• Estimate cost in terms of engineering days to calculate the highest expected impact-to-cost items.
• Incorporate customer and data inputs into your calculations to avoid moving metrics that don't impact the central goal.

Source: Prioritizing conversion opportunities (https://www.lennysnewsletter.com/p/prioritizing-conversion-opportunities)

---

"Every project you ship is an opportunity to learn about what works or doesn’t. Re-evaluate ROI after you get feedback. If you see standout results around a particular hypothesis, double down with additional related feature work."

Insight: Successful conversion optimization relies on a continuous feedback loop where you re-evaluate ROI after every launch to decide whether to double down or shift focus.

Tactical advice:

• Run lightweight tests to reduce risk for potentially costly experiments.
• Go deep with additional feature work specifically where you find 'heat' or standout results.
• Consider changing focus if you observe diminishing returns on a particular hypothesis.

Source: Prioritizing conversion opportunities (https://www.lennysnewsletter.com/p/prioritizing-conversion-opportunities)

---

"When I left in 2012, we were running more than 100 concurrent experiments any given time, and incredibly fast paced. Anybody in the product team could still have and idea in the morning and have it running live on customers in the afternoon."

Insight: A high-velocity A/B testing culture compounds small conversion gains into massive competitive advantages in paid marketing.

Tactical advice:

• Empower anyone in the product organization to launch experiments without multi-level approvals.
• Maintain a high volume of concurrent experiments to maximize the speed of learning and optimization.
• Focus testing on conversion rates to make unviable marketing campaigns competitive.

Source: Top 5 most interesting things about Booking.com's early growth strategy – Issue 46 (https://www.lennysnewsletter.com/p/top-5-most-interesting-things-about-bookingcoms-early-growth-strategy---issue-46)

---

Insight: Fostering a culture of experimentation requires a holistic approach that integrates data science into hiring, team structure, and core infrastructure from the earliest stages of the company.

Tactical advice:

• Hire a full-time data scientist early to establish data as a strategic asset through direct relationships with leadership.
• Embed data scientists within product teams so they are active voices in roadmap planning rather than just ad-hoc consultants.
• Invest in internal tooling that makes it effortless for teams to run experiments and access trusted results.

Source: Fostering a culture of experimentation (https://www.lennysnewsletter.com/p/fostering-a-culture-of-experimentation)

---

Insight: Factor in the percentage of the user base that will actually encounter a change to determine its true addressable impact and ROI.

Tactical advice:

• Calculate the 'whole pie' of users who will actually see an experiment.
• Prioritize experiments based on total addressable impact rather than just percentage improvement.
• Target experiments at high-traffic areas of the product to maximize absolute gains.

Source: The secret to Duolingo’s exponential growth (https://www.lennysnewsletter.com/p/the-secret-to-duolingos-exponential-growth)

---

Insight: A high-velocity experimentation engine requires disciplined, regular data reviews to ensure winning changes are shipped the moment significance is reached.

Tactical advice:

• Establish a regular cadence for examining experiment data to prevent features from sitting on the back burner.
• Launch on your largest platforms first to maximize the absolute volume of compounding gains.
• Port winning experiments across platforms quickly, as user behavior is often consistent across devices.

Source: The secret to Duolingo’s exponential growth (https://www.lennysnewsletter.com/p/the-secret-to-duolingos-exponential-growth)

---

Sri Batchu

Insight:

Source: Lessons from scaling Ramp | Sri Batchu (Ramp, Instacart, Opendoor) (https://www.youtube.com/watch?v=RcYCU5UAZOk) @ 00:51:43

---

Timothy Davis

"You can always do a very, very small test. You can just put a little money into a platform, see if there's a sign of life. If there is, then you can pull back and say, 'Okay, we have signs of life. Now let's build a campaign around that.'"

Insight: Validate new channels by using existing customer data to run low-budget lookalike tests that search for initial signs of life before scaling.

Tactical advice:

• Build lookalike audiences using 1% match thresholds from your own customer data for the highest correlation.
• Run small, low-budget experiments to find signs of life before committing to a full campaign.
• Wait for a positive signal before investing in platform-specific creative and messaging.

Source: The ultimate guide to performance marketing | Timothy Davis (Shopify) (https://www.youtube.com/watch?v=zNJyb3R_Pnc) @ 00:11:38

---

Insight:

Source: The ultimate guide to performance marketing | Timothy Davis (Shopify) (https://www.youtube.com/watch?v=zNJyb3R_Pnc) @ 01:05:51

---