Detecting Network Effects: Randomizing Over Randomized Experiments
Detecting Network Effects: Randomizing Over Randomized Experiments
Martin Saveski (MIT)
Jean Pouget-Abadie (Harvard University)
Guillaume Saint-Jacques (MIT)
Weitao Duan (LinkedIn)
Souvik Ghosh (LinkedIn)
Ya Xu (LinkedIn)
Edo Airoldi (Harvard University)
Randomized experiments—A/B tests—are the standard approach for evaluating the effect of new product features. They rely on the “stable treatment value assumption” (SUTVA) which states that treatment only affects treated users and does not spill over to their friends. Violations of SUTVA, common in features that exhibit network effects, result in inaccurate estimates of the treatment effect. In this paper, we leverage a new experimental design for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group. We do so by simultaneously running completely randomized and cluster-based randomized experiments and comparing the difference of resulting estimates, detailing known theoretical bounds on the Type I error rate. We provide practical guidelines for implementing this design on large-scale experimentation platforms. Finally, we deploy this design to LinkedIn’s experimentation platform and apply it to two online experiments, highlighting the presence of network effects and bias in standard A/B testing approaches in a “real-world” setting.
More on http://www.kdd.org/kdd2017/
Видео Detecting Network Effects: Randomizing Over Randomized Experiments канала KDD2017 video
Martin Saveski (MIT)
Jean Pouget-Abadie (Harvard University)
Guillaume Saint-Jacques (MIT)
Weitao Duan (LinkedIn)
Souvik Ghosh (LinkedIn)
Ya Xu (LinkedIn)
Edo Airoldi (Harvard University)
Randomized experiments—A/B tests—are the standard approach for evaluating the effect of new product features. They rely on the “stable treatment value assumption” (SUTVA) which states that treatment only affects treated users and does not spill over to their friends. Violations of SUTVA, common in features that exhibit network effects, result in inaccurate estimates of the treatment effect. In this paper, we leverage a new experimental design for testing whether SUTVA holds, without making any assumptions on how treatment effects may spill over between the treatment and the control group. We do so by simultaneously running completely randomized and cluster-based randomized experiments and comparing the difference of resulting estimates, detailing known theoretical bounds on the Type I error rate. We provide practical guidelines for implementing this design on large-scale experimentation platforms. Finally, we deploy this design to LinkedIn’s experimentation platform and apply it to two online experiments, highlighting the presence of network effects and bias in standard A/B testing approaches in a “real-world” setting.
More on http://www.kdd.org/kdd2017/
Видео Detecting Network Effects: Randomizing Over Randomized Experiments канала KDD2017 video
Показать
Комментарии отсутствуют
Информация о видео
Другие видео канала
Designing AI at Scale to Power Everyday LifeEstimation of Recent Ancestral Origins of Individuals on a Large Scalestruc2vec: Learning Node Representations from Structural IdentityLearning to Generate Rock Descriptions from Multivariate Well Logs with Hierarchical AttentionA Local Algorithm for StructurePreserving Graph CutMulti-Aspect Streaming Tensor CompletionVisualizing Deep Learning Activations for Improved Malaria Cell ClassificationPlanning Bike Lanes based on SharingBikes' TrajectoriesTripoles: A New Class of Relationships in Time Series DataIncorporating Feedback into Tree-based Anomaly DetectionIndustrial Machine LearningWeisfeiler-Lehman Neural Machine for Link PredictionOptimal Reserve Price for Online Ads Trading Based on Inventory IdentificationMulti-Aspect Streaming Tensor CompletionDispatch with Confidence: Integration of machine learningLearning certifiably optimal rule lists for categorical dataInterpretable Predictions of Tree-based Ensembles via Actionable Feature TweakingMultitask Learning using Task ClusteringThe Co-Evolution Model for Social Network Evolving and Opinion MigrationRevisiting power-law distributions in spectra of real world networksKDD Business Lunch