Experiment Design & Methodology.

AI as Methodological Reviewer

The best time to find flaws in your methodology is before you collect data, not during peer review. AI serves as an always-available methodological reviewer that challenges your design from multiple perspectives. ``` DESIGN REVIEW PROMPT: I am designing an experiment to test: Hypothesis: {hypothesis} Population: {target_population} Design: {between/within/mixed-subjects} IV: {independent_variable} with levels {levels} DV: {dependent_variable} measured by {measurement_method} Controls: {control_conditions} Duration: {timeline} Act as three different reviewers: REVIEWER 1 (Methodologist): What are the internal validity threats? What confounds exist? What alternative explanations could account for the predicted results? REVIEWER 2 (Statistician): Is the design powered to detect the expected effect? Are the measures appropriate? What statistical assumptions should be verified? REVIEWER 3 (Domain Expert in {field}): What does the existing literature suggest about this design? What has been tried before and what were the results? What would make this contribution novel? ``` This multi-reviewer prompt forces the AI to attack your design from three angles simultaneously. The output is not a rubber stamp -- it is a list of problems to solve before you start.

Power Analysis and Sample Size

Underpowered studies waste resources and produce unreliable results. AI makes power analysis accessible, even for complex designs. ```python POWER_ANALYSIS_PROMPT = """ I need a power analysis for this design: - Test: {statistical_test} - Expected effect size: {effect_size} (based on {justification}) - Desired power: 0.80 - Alpha: 0.05 - Groups: {n_groups} - Repeated measures: {n_timepoints} Generate Python code using statsmodels or pingouin to: 1. Calculate required sample size 2. Create a power curve showing sample size vs. power for effect sizes from {small} to {large} 3. Calculate achieved power if I can only recruit {max_n} participants 4. Account for expected attrition of {attrition_rate}% """ # Example output code: from statsmodels.stats.power import TTestIndPower analysis = TTestIndPower() # Required sample size for d=0.5, power=0.80, alpha=0.05 n = analysis.solve_power( effect_size=0.5, power=0.80, alpha=0.05, ratio=1.0, # Equal group sizes alternative='two-sided' ) print(f"Required N per group: {int(np.ceil(n))}") # Adjust for 15% attrition n_adjusted = int(np.ceil(n / (1 - 0.15))) print(f"Recruit per group (with 15% attrition): {n_adjusted}") # Power curve import matplotlib.pyplot as plt effect_sizes = np.arange(0.2, 1.0, 0.05) sample_sizes = [analysis.solve_power(es, power=0.80, alpha=0.05) for es in effect_sizes] plt.plot(effect_sizes, sample_sizes) plt.xlabel('Effect Size (Cohen\'s d)') plt.ylabel('Required N per group') plt.title('Sample Size Requirements by Effect Size') plt.grid(True) plt.savefig('power_curve.png', dpi=300, bbox_inches='tight') ```