Research and analysis

Road-testing remedies, experimental techniques and evaluation

Published 3 August 2017

The content is divided into 바카라 사이트˜core reading바카라 사이트™ and 바카라 사이트˜further reading바카라 사이트™ lists. Core reading captures fundamental and/or important documents of use primarily to inexperienced users while further reading lists capture more specific material intended for experienced practitioners.

1. Overview of the key approaches

1.1 Core reading

Document	Topics covered	Notes
EC (2013),	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs.	Explains when and how behavioural science should be applied to policy making including an overview of the different research methods.
EC (2015),	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs.	This policy brief is a follow-up to EC (2013) on applying behavioural sciences and aims to provide policy-makers with practical guidance for planning and managing a behavioural study.
Francesco Guala (2005),	Experimental economics, remedy testing.	This book provides a comprehensive analysis and critical discussion of the methodology of experimental economics. It outlines the fundamental principles of experimental inference in order to investigate their power, scope and limitations.
Bardsley, Cubitt, Loomes, Moffatt, Starmer and Sugden (2010),	Economic behaviour, experiments, RCTs	The book reviews a wide range of experiments, all carried out with the intention of contributing in some way or other to the understanding of economic behaviour.

1.2 Further reading 바카라 사이트“ guidance

Document	Topics covered	Notes
OFT1099 (2009),	Remedy testing, qualitative research, surveys, online experiments, lab trials, field trials, RCTs, simulation, quasi-experiments, econometrics.	This report provides rich information on the different research methods to assess consumer remedies prior to their implementation. Covers lab and field trials under the broader category of economic experiments.
Commonwealth of Australia, Department of the Prime Minister and Cabinet (2016),	Remedy testing, RCTs, pre-testing.	Covers 4 project stages to developing behaviour interventions for RCTs focusing on the first two stages: - Discovery - defining outcomes of interest; establishing whether they can be measured; and whether standardised interventions are feasible and can be delivered to a large enough randomised population. - Diagnosis 바카라 사이트“ understanding behaviour and how interventions might influence this

2. Experimental and survey-based techniques

2.1 Core reading

Document	Topics covered	Notes
List J, Sadoff S and Wagner M (2012),	RCTs, lab trials, field trials, experiments.	Develops several simple rules of thumb that researchers can apply to improve the efficiency of their experimental designs. This is a more technical article on experimental design than the guidance documents listed in the overview section above. Also includes empirical examples from the literature.
CC/OFT (2011), 바카라 사이트˜Good practice in the design and presentation of consumer survey evidence in merger inquiries바카라 사이트™	Surveys	While the focus of this guidance is on surveys in merger inquiries, the good practice principles it sets out have wider relevance to survey work for consumer remedies.

2.2 Further reading 바카라 사이트“ selected academic papers

Document	Topics covered	Notes
Al-Ubaydli O and List J (2012),	RCTs, lab trials, field trials, external validity, generalisability.	This study provides an overview of experimental methods in economics, with a special focus on developing an economic theory of generalisability. It concludes that until we conduct more field experiments that build a bridge between the lab and the naturally-occurring settings of interest, we cannot begin to make strong conclusions empirically on the crucial question of generalisability from the lab to the field.
Mitchell G (2012), . Perspectives on Psychological Science, 7(2), 109-117	RCTs, lab trials, field trials, external validity, generalisability.	Using 217 lab-field comparisons from 82 meta-analyses, this paper finds that the external validity of laboratory research differed considerably by psychological subfield, research topic, and effect size. Laboratory results from industrial-organisational psychology most reliably predicted field results, although results from IO labs still varied in their generalisability.

2.3 Further reading 바카라 사이트“ case studies

Document	Topics covered	Notes
FCA (2016),	Remedy testing, RCTs, online experiments, field trials.	This round-up paper presents the results of 5 field trials and 3 online experiments, which have tested the effect of interventions that draw on behavioural theory, such as increasing salience or personalisation. It also shares some practical lessons learnt.
Optimisa (2016),	Remedy testing, qualitative research.	Used as an input to the design of consumer remedies in the CMA Retail Banking Market Investigation (2016), including the recommendation to undertake RCTs on prompting customers to consider their banking arrangements.
CC (2009), , Appendices 3.9, 10.9, 10.10 and 10.11 and associated	Remedy testing, simulation, surveys.	Includes using survey evidence as part of a simulation of the effects of remedies aimed at increasing the proportion of consumers that shop around for PPI.
FCA (2014),	Remedy testing, online experiments.	This paper discusses the pros and cons of using behavioural experiments in creating regulatory policy. The paper illustrates this by describing an innovative experiment, which finds that when general insurance is sold as an add-on product to a primary purchase, this weakens consumers바카라 사이트™ ability to discipline firms by shopping around and comparing products effectively. This in turn has implications for possible remedies.
FCA (2015), . Research by London Economics and YouGov	Remedy testing, online experiments, credit.	Used an online experiment to test the impact of imposing a series of potential standards on price comparison websites of high-cost short-term credit products.
Ofcom (2010),	Remedy testing, lab trials.	To assess the potential benefits and limitations of experiments, Ofcom commissioned this experimental study on devising effective ways of providing information on the price of telephone calls. Includes a discussion on using experiments in economics for regulatory policy.
OFT (2010),	Remedy testing, lab trials.	This study examines the impact on consumer decision-making of different ways in which prices can be framed using a controlled experiment. It fed into the OFT바카라 사이트™s study on the .
OFT (2013),	Remedy testing, lab trials.	Examines the impact of partitioned pricing on consumer decision-making. Forms part of a wider set of of the OFT in 2013. This research was related to the OFT investigations on .

3. Randomised controlled trials in the field

3.1 Core reading

Document	Topics covered	Notes
Behavioural Insights Team (2012), 바카라 사이트˜Test, Learn Adapt바카라 사이트™	Remedy testing, RCTs, field trials.	The paper argues that RCTs, should be used much more extensively in public policy to enable policymakers to test which interventions are most effective. It also sets out 9 separate steps that are required to set up any RCT.
List A (2012),	RCTs, lab trials, field trials, external validity, generalisability.	Offers an overview of experimental methods in economics from laboratory experiments to field experiments with a focus on field experiments.

3.2 Further reading 바카라 사이트“ practical considerations

Document	Topics covered	Notes
J-PAL (2013),	RCTs, evaluation.	Covers the what, why, who, when, and how of randomised evaluations.
IPA (2015),	RCTs	Introductory toolkit aimed at researchers seeking to conduct RCTs with financial service providers. This toolkit also has relevance to other sectors.
Australian 바카라 사이트 Department of Industry and Science (2015),	Remedy testing, RCTs, field trials, evaluation, generalisability.	Develops a framework that can be used to assess the suitability of evaluating policy interventions using RCTs.
Cartwright, Deaton (2016),	RCTs, generalisability.	Raises a number of concerns relating to the ability of RCTs to provide robust and/or generalisable evidence.

3.3 Further reading 바카라 사이트“ case studies

Document	Topics covered	Notes
FCA (2016),	Remedy testing, RCTs, field trials, switching triggers, retail banking, field trials.	This paper describes the findings of a series of field trials that explored how disclosure-type regulatory interventions could encourage consumers to switch to higher-paying savings accounts.
FCA (2015),	Remedy testing, RCTs, field trials, switching triggers, retail insurance.	This paper presents the results of field trials testing the potential for improved renewal notices to encourage consumers to switch or negotiate their home or motor insurance policy at renewal.
Bertrand M and Morse A (2011),	Remedy testing, RCTs, field trials, credit.	Uses a field experiment to assess whether psychology-guided information disclosure induces borrowers to lower their use of high-cost debt.

4. Evaluation

4.1 Core reading

Document	Topics covered	Notes
HM Treasury (2011), 바카라 사이트˜The Magenta Book바카라 사이트™	Evaluation	Guidance on what to consider when designing an evaluation.

4.2 Further reading 바카라 사이트“ guidance

Document	Topics covered	Notes
Ofcom (2013),	Evaluation, information remedies.	Section 6 of this review explores the issues around evaluating the effectiveness of an information remedy.
National Audit Office (2011),	Evaluation	Explains what auditors look for in when assessing the design and evaluation of a behavioural change intervention.
IPA (2016),	Evaluation	Beginning with the need for a theory-driven evaluation, and ending with a set of concrete tools to use in running an evaluation, this site provides references to a range of practical materials useful for organizations that are considering a rigorous impact evaluation.

4.3 Further reading 바카라 사이트“ case studies

Document	Topics covered	Notes
FCA (2015),	Evaluation, switching remedies, banking.	-
Ofgem (2015),	Evaluation, econometrics, information remedies, restricting choice, comparison tools, energy.	As part of implementing reforms to the retail energy market in 2013/14, Ofgem designed an evaluation framework to go beyond straightforward monitoring of market changes. This report presents the results of this evaluation on Ofgem바카라 사이트™s reforms to improve tariff comparability and annual summaries.
CMA (2016), 바카라 사이트˜Evaluation of the OFT바카라 사이트™s 2012 consumer enforcement action in relation to health and fitness contract terms바카라 사이트™	Evaluation, surveys.	Evaluates the outcomes of a consumer enforcement case on improved cancellation rights and reducing the length of minimum contract terms. This is similar to an evaluation of consumer remedies and is a useful example of bringing together evidence from a number of sources including information from competitors and a customer survey.
CC (2013),	Evaluation, information remedies, search remedies.	Evaluates the remedies on the CC바카라 사이트™s home credit market investigation. It finds that there have been few, if any, unintended consequences from the CC바카라 사이트™s remedies and that profit per customer of the largest provider (Provident) fell. However, it also found that the CC바카라 사이트™s remedies have not necessarily removed all the features the CC identified in 2006.
CC (2011),	Evaluation, econometrics, information remedies.	Highlights the potential difficulties in data gathering and in isolating with precision the impact of consumer information remedies against changes to other drivers of consumer behaviour. Appendix B of the report applies the 바카라 사이트˜difference-in-differences바카라 사이트™ approach to evaluate the introduction of APR warnings to sub-set of store cards. Also see this related econometric study: Elliott and Wei (2010), The Impact of Regulatory Intervention in the UK Store Card Industry, International Journal of the Economics of Business, Volume 17, Issue 1, pages 87-88. .
OFT (2011),	Evaluation, surveys, consumer education, information remedies.	Evaluates the OFT바카라 사이트™s Just tick it campaign to raise consumer awareness of scam ticket websites and Skilled to go toolkit for teachers, which aimed to increase consumer knowledge, skills and confidence. Includes a discussion of methodological approaches and the value of early discussions on the evaluation framework before implementing a remedy.
OFT (2008),	Evaluation, surveys, mystery shopping, information remedies.	An independent evaluation of changes to the extended warranties market after the 2005 Order, drawing on interviews, consumer surveys and a mystery shopping exercise (following an OFT market study and CC enquiry). Employed a before and after comparison on the impact of consumer information remedies and a remedy giving consumers the right to cancel an extended warranty.