Ben “Bad Science” Goldacre has just blogged on the importance of randomised controlled trials in the development of policy and has co-authored a Cabinet Office paper which looks well worth a read. He has also written on paucity of RCTs in criminal justice policy to which I would add the situation is even worse in civil justice. I can think of only one RCT (on the impact of debt advice: see Pleasence et al (£)) and another close to it which abandoned a quasi-random element (when the Scottish Public Defenders Offices were initially set up clients were allocated to the SPDOs if born in certain months and to private practitioners if borne in others; exceptions were negotiated for certain types of case and then coach and horse was driven through the attempted ‘randomisation’). This was not assisted by the ability to claim that lawyer allocation was now being determined astrologically. One of the more shocking omissions is in mediation: the government have put inordinate faith in mediation which is not supported by the evidence. An attempt at randomisation by Genn et al, was stymied by a judicial ruling which prevented fully random allocation of cases.
A number of barriers stand in the way of RCTs in legal work: natural conservatism; attempts to run randomisation within adversarial systems especially where one party is the State and the RCT is state sponsored); lack of understanding of the benefits; the cost; ethical concerns (overstated in my view), etc. Legal researchers are not typically equipped with the understanding and interest of such approaches which would make them more likely to be developed, and developed well. I have even seen hostility to the suggestion that we should be doing more quantitative work attacked as undermining the benefits of qualitative work. In truth there is not enough of either. All approaches have strengths and weaknesses.
An interesting question which might be posed for legal service innovators starting to make sometime dramatic claims for their own service models is how well tested are they? There are business benefits in testing different approaches properly to see if they really do deliver better outcomes for clients; cost benefits and efficiency savings to the provider. See Robert Richard’s post here which stimulated that thought. How many do such testing: very few if any is my guess and I would wager it is often, though not always, back of an envelope stuff. This will change, expecially as the innovators and providers get bigger and cannier; but it may be a long time coming and it may pass the legal academy by.