The challenges of using randomised control trials to investigate careers work

I’ve been talking to a number of different people about the idea of doing randomised control trials of various kinds of careers work. In principle I’m really excited by this as it is clearly a gap in the careers work literature (as we argued in Fostering College and Career Readiness). However there are a number of problems with this kind of approach that it is worth briefly rehearsing before going on to look at some work that takes this kind of approach.

Problem 1 – Is it ethical? The first issue that people normally throw at you is the idea that it would be unethical to provide a service for one group of people but not for another group. I haven’t got too much sympathy with this position as the idea that it is ethical for medics (who deal in life and death) to use this approach but not for educators seems fairly ludicrous. However, there are clearly political and reputational implications of (a) admitting that we are not sure whether something works and (b) withholding it from some people.

Problem 2 – What impact are you looking for? Careers work interventions act on multiple factors. They may influence people’s aspirations, confidence, happiness, decisiveness and so on and may have impacts on people’s academic attainment, vocational choices, labour market outcome, income and so on. Randomised control trial approaches really require us to narrow down what we are looking for to a relatively small number of hypotheses (ideally one) rather than to go out casting for impacts wherever we find them. The problem is that this can be reductive and leave a study in danger of missing actual impacts.

Problem 3 – Effects are likely to be small. Careers work is a relatively small part of everyone’s life. If you are looking for factors that explain academic or labour market performance and participation you are going to be far more likely to find these in the usual spread of socio-economic factors (race, class, gender) than in an individuals’ participation in career development activities. Even within the education system careers work remains as a relatively small component. Consequently it is reasonable to expect that effect sizes are small. If careers work tends to results in a 1 mark exam improvement or a 1% increase in the likelihood of getting a job it may be well worth the investment but will not necessarily be easy to spot. The smaller the effect size the larger the sample size needs to be to be sure that you’ve spotted it. See Wikipedia’s article on Effect Sizes if you are interested in this stuff.

Problem 4 – What level of impact is worthwhile?  Even if randomised control trials can identify that careers work has a statistically significant impact on something this doesn’t necessarily mean that it is worthwhile. If, for example, to ensure that school students received an intervention that improved their exam result by 1 mark you had to spend £1 million per student you might conclude that this wasn’t worthwhile. Equally you might search to identify whether the same result could be achieved some other way. In other words a statistically robust impact does not necessarily mean that something is inherently valuable or a good use of money, nor does it mean that the same impact cannot be achieved through some other mechanism.

So now I’ve flagged some of these problems I thought that it might be useful to look at some of the studies that do look at careers work through a randomised control trial methodology and summarise what they tell us.

de Raaf, S., T. Shek-wai Hui, and C. Vincent (2012). Career Motion: How Web-based technologies can improve the career choices of young people. Ottawa: The Social Research and Demonstration Corporation (SRDC).

This study looked at the effectiveness of an online information environment for unemployed adults. Participants were randomly assigned to either a programme group (which got access to a web tool) or a control group (which did not). The study set out two main hypotheses.

  • Can the provision of labour market information through Web-based technologies improve participants’ level of confidence and competency with regards to career and employment decision-making?
  • Can the provision of labour market information through Web-based technologies increase participants’ job search and improve labour market outcomes?

The study used a variety of different instruments to quantify participants’ confidence about their career and job seeking. The study found that the intervention did have an impact on these factors. However the study also measured actual labour market outcomes a year after the intervention and did not find any statistically significant impacts.

Yamagishi, M., Kobayashi, T., & Nakamura, Y. (2008). Effects of web-based career identity training for stress management among japanese nurses: A randomized control trial. Journal of Occupational Health, 50 , 191-193.

This article also looked at an online career support intervention, but this time one targeted at Japanese nurses. Again the participants were divided into two groups. The intervention group received an online career development intervention while the control group did not. Impacts were observed through the use of questionnaire based instruments that measured career identity and work-related stress. The research was able to identify impacts on some components of both career identity (e.g. confidence that you can influence the organisation) and work-related stress (e.g. ability to manage workload.

McKay, H., Bright, J. E. H., & Pryor, R. G. L. (2005). Finding order and direction from chaos: a comparison of chaos career counseling and trait matching counseling. Journal of Employment Counseling, 42 (3), 98-112.

In this article students who were seeking careers advice were randomly assigned to either (a) a chaos career counselling intervention (b) a trait matching intervention or (c) a wait list control group. The researchers then used questionnaire based instruments to measure participant satisfaction and changes in career thinking/decision-making. Participants were tested before and after the intervention and then one month later. The research found that both types of counselling were resulted in greater impacts that no counselling. It also found that the chaos approach had a more lasting impact.

In conclusion

The articles above (and there are others – although not as many as I’d like) demonstrate that it is possible to apply a randomised control trial approach to careers work. It also shows that it is possible to identify impacts through this method. However, these articles also demonstrate both the complexity of developing meaningful measures and the relative levels of subjectivity about what researchers decide to measure and what instruments they use to do it.

If you are interested in finding out more about this you may also be interested in having a look at

  • Goldacre, B. (2007). Bad science. London: HarperPress.
  • Mosteller, F. and R. Boruch (2002). Evidence Matters: Randomized Trials in Education Research. Brookings Inst Pr.

I’d also be interested to hear further thoughts about this from other people and to be directed to further articles.

Advertisements

2 thoughts on “The challenges of using randomised control trials to investigate careers work

  1. Really interesting post. This is something I’m really quite interested in and have read quite a bit about recently. I’ve been tasked with developing a new feedback and evaluation strategy for our service and to consider how as a service we can measure the impact of what we do in order to meet the requirements for the revised Matrix standard. I’ve read quite a bit around the topic of measuring impact of career guidance activities and come across a few, albeit not many, examples of the use of control groups which I’d be happy to e-mail on to you if you’d like? There don’t seem to be many examples in the literature I’ve come across and limited to a few examples drawn from adult guidance comparing a guidance group with a control or ‘information only’ group. Happy to send details.Your comment about the ethical argument is an interesting one and made me re-think things a bit. Much of the literature I’ve read accounting for the lack of work in this area uses the ethical argument as a reason not to do it but I think the point you make is a valid one. The challenge then is to figure out what it is we are looking for and how do we know it’s us that has had the impact, given that career guidance doesn’t operate in a vacuum…

  2. I’d really appreciate your list of control trial studies. I know that I’ve seen others in the past, but I couldn’t find them when I was writing this. I’m not sure that all education ethics committees will agree with my stance on this.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s