Rocket Surgery Made Easy

I’ve been a massive fan of Steve Krug for years. I discovered him during a brief detour in my career when I was working as a web designer. I bought a book called Don’t Make Me Think which was an introduction to both web usability and to the process of usability testing. I realised that most website were badly designed because most web designers never showed their work to anyone before it was released. Krug argued for an iterative and consumer informed approach to design. In essence you have an idea – you show it to some people, you build a prototype – you show it to some people, you get your first page up – you show it to some people and so on.

One of the things that I really remember from that book was the way that Krug talks about how the idea of doing things properly stops them from happening at all. So because you can’t find the full demographic spread of users you don’t show your design to anyone. Or because your site is meant to be viewed in schools you hold off on showing it because it is difficult to get into schools. Krug makes a simple point that chimes with me. The more you show your work to other people the better. The more the people you show it to are like your intended audience the better, but don’t let this stop you showing it to someone. This principle works for website design, but it also works for pretty much everything else (academic articles, your CV etc.) Of course there are important qualifiers to this. Some sense of representativeness should be an aim of evaluation projects, but Krug’s point is essentially that the perfect is the enemy of the good. One brilliantly representative evaluation of a product once the product is designed is probably a lot less effective than 10 imperfect evaluations as you develop the product.

Anyway, this post was prompted by chancing across this video of Krug explaining how to run a simple usability test.

Enjoy

Advertisements

The challenges of using randomised control trials to investigate careers work

I’ve been talking to a number of different people about the idea of doing randomised control trials of various kinds of careers work. In principle I’m really excited by this as it is clearly a gap in the careers work literature (as we argued in Fostering College and Career Readiness). However there are a number of problems with this kind of approach that it is worth briefly rehearsing before going on to look at some work that takes this kind of approach.

Problem 1 – Is it ethical? The first issue that people normally throw at you is the idea that it would be unethical to provide a service for one group of people but not for another group. I haven’t got too much sympathy with this position as the idea that it is ethical for medics (who deal in life and death) to use this approach but not for educators seems fairly ludicrous. However, there are clearly political and reputational implications of (a) admitting that we are not sure whether something works and (b) withholding it from some people.

Problem 2 – What impact are you looking for? Careers work interventions act on multiple factors. They may influence people’s aspirations, confidence, happiness, decisiveness and so on and may have impacts on people’s academic attainment, vocational choices, labour market outcome, income and so on. Randomised control trial approaches really require us to narrow down what we are looking for to a relatively small number of hypotheses (ideally one) rather than to go out casting for impacts wherever we find them. The problem is that this can be reductive and leave a study in danger of missing actual impacts.

Problem 3 – Effects are likely to be small. Careers work is a relatively small part of everyone’s life. If you are looking for factors that explain academic or labour market performance and participation you are going to be far more likely to find these in the usual spread of socio-economic factors (race, class, gender) than in an individuals’ participation in career development activities. Even within the education system careers work remains as a relatively small component. Consequently it is reasonable to expect that effect sizes are small. If careers work tends to results in a 1 mark exam improvement or a 1% increase in the likelihood of getting a job it may be well worth the investment but will not necessarily be easy to spot. The smaller the effect size the larger the sample size needs to be to be sure that you’ve spotted it. See Wikipedia’s article on Effect Sizes if you are interested in this stuff.

Problem 4 – What level of impact is worthwhile?  Even if randomised control trials can identify that careers work has a statistically significant impact on something this doesn’t necessarily mean that it is worthwhile. If, for example, to ensure that school students received an intervention that improved their exam result by 1 mark you had to spend £1 million per student you might conclude that this wasn’t worthwhile. Equally you might search to identify whether the same result could be achieved some other way. In other words a statistically robust impact does not necessarily mean that something is inherently valuable or a good use of money, nor does it mean that the same impact cannot be achieved through some other mechanism.

So now I’ve flagged some of these problems I thought that it might be useful to look at some of the studies that do look at careers work through a randomised control trial methodology and summarise what they tell us.

de Raaf, S., T. Shek-wai Hui, and C. Vincent (2012). Career Motion: How Web-based technologies can improve the career choices of young people. Ottawa: The Social Research and Demonstration Corporation (SRDC).

This study looked at the effectiveness of an online information environment for unemployed adults. Participants were randomly assigned to either a programme group (which got access to a web tool) or a control group (which did not). The study set out two main hypotheses.

  • Can the provision of labour market information through Web-based technologies improve participants’ level of confidence and competency with regards to career and employment decision-making?
  • Can the provision of labour market information through Web-based technologies increase participants’ job search and improve labour market outcomes?

The study used a variety of different instruments to quantify participants’ confidence about their career and job seeking. The study found that the intervention did have an impact on these factors. However the study also measured actual labour market outcomes a year after the intervention and did not find any statistically significant impacts.

Yamagishi, M., Kobayashi, T., & Nakamura, Y. (2008). Effects of web-based career identity training for stress management among japanese nurses: A randomized control trial. Journal of Occupational Health, 50 , 191-193.

This article also looked at an online career support intervention, but this time one targeted at Japanese nurses. Again the participants were divided into two groups. The intervention group received an online career development intervention while the control group did not. Impacts were observed through the use of questionnaire based instruments that measured career identity and work-related stress. The research was able to identify impacts on some components of both career identity (e.g. confidence that you can influence the organisation) and work-related stress (e.g. ability to manage workload.

McKay, H., Bright, J. E. H., & Pryor, R. G. L. (2005). Finding order and direction from chaos: a comparison of chaos career counseling and trait matching counseling. Journal of Employment Counseling, 42 (3), 98-112.

In this article students who were seeking careers advice were randomly assigned to either (a) a chaos career counselling intervention (b) a trait matching intervention or (c) a wait list control group. The researchers then used questionnaire based instruments to measure participant satisfaction and changes in career thinking/decision-making. Participants were tested before and after the intervention and then one month later. The research found that both types of counselling were resulted in greater impacts that no counselling. It also found that the chaos approach had a more lasting impact.

In conclusion

The articles above (and there are others – although not as many as I’d like) demonstrate that it is possible to apply a randomised control trial approach to careers work. It also shows that it is possible to identify impacts through this method. However, these articles also demonstrate both the complexity of developing meaningful measures and the relative levels of subjectivity about what researchers decide to measure and what instruments they use to do it.

If you are interested in finding out more about this you may also be interested in having a look at

  • Goldacre, B. (2007). Bad science. London: HarperPress.
  • Mosteller, F. and R. Boruch (2002). Evidence Matters: Randomized Trials in Education Research. Brookings Inst Pr.

I’d also be interested to hear further thoughts about this from other people and to be directed to further articles.

Employer practice in progressing low-paid staff

Low-pay

My iCeGS colleagues and I have recently been working with the Policy Research Institute and around the theme of progressing into and on in employment and employer practice in this area.
 
We spoke to 28 employers both large and small about how they progress their low skilled, low paid staff. The research found that when employers progress their staff, they reap a number of organisational benefits, including: improved retention of staff and the subsequent reduction of recruitment costs; a general improvement in employee satisfaction; greater organisational flexibility; improved productivity; and a positive impact on organisational reputation.

To access the report, please click here. To read the employer case studies associated with the report, click here.

 

What is the best piece of careers advice you???ve ever been given?

Signpost

Time for a parlour game!

I’d like everyone to play along with this one. Just answer the questions

What is the best piece of careers advice you’ve ever been given?

And of course tell us why!

I can imagine that for some people this might be a sensitive subject and so I’m happy if people want to email me their answer to t.hooley@derby.ac.uk so that I can anonymise and post up here. Everyone else can just add their answer as a comment.

To get us started I’ll share mine.

… After four hard years slog I went into my PhD viva having given very little thought to the outcome. I certainly hadn’t given any thought to the official regulations that surrounded PhD vivas and consequently didn’t really know what the possible outcomes were. Consequently when I was informed that I had another year’s work to do I assumed that I had failed the thing and that my personal and intellectual inadequacies were now a matter of public record.

One long lost weekend later I crawled back into the office and sat flicking through the fragments of my broken dreams (AKA my thesis). A postdoc in the office above me happened by at the right time and I unburdened my tail of woe to him.

“Yes, that happened to me as well.” He said. “It’s awful isn’t it. First I blamed myself, then I blamed my supervisor, then I blamed my examiner and then I condemned the whole system. But, eventually I realised that I just had some more work to do, and I got on with it.”
This was like a bolt of lightning to me as it jolted me out of my self-pity and communicated two key messages.

  1. There is no point dwelling on the past, particularly if you are searching for reasons why the present shouldn’t be as it is. Life isn’t fair – get used to it. It is only by focusing on the future that life and career can move on.
  2. Purposeful decisions and practical hard work move you from a bad situation to a good one.

This piece of advice was the best piece of advice that I’ve ever had because it was wise, because it came from experience, but probably most of all, because it was timely. I needed to hear it at that point in my life, I needed a ladder out of my crisis and the postdocs comments provided me with that. From then on I got focused, got cracking and got realistic about my thesis. What ended up in the public record was my thesis itself which although of extremely limited interest to the world at large is not actively a record of my intellectual inadequacy.

Would I have done this without this piece of advice? Maybe? Probably? In all likelihood someone else would have provided the advice I needed a few days or weeks later, but it is also possible that without some advice and encouragement I would have thrown in the towel and headed off to do something different. Would that have been a bad thing? Who knows….

So that’s my story. Let’s hear yours.

As we say goodbye to the summer…

On_the_beach

Like most other parents I’ve spent the summer juggling kids, work, holidays and various other bits and pieces. In some ways this isn’t really any different from any other time, but the length of the summer holidays, the long days and the essential appeal of sitting out in the sun means that, for me at least, during the summer the see-saw of my work-life balance tips in an interesting and unfamiliar direction.

Most of the time, work fills up most of my life. This isn’t to say I don’t spend time with my family or do anything else. But the 5 X 8 hours discipline of the working week means that I simply spend more time at work than I do doing any other thing. What is more I frequently slop outside of the 8 hours, finishing up things that “need doing”, checking email and so on. What is more I like it, I’m not someone who has a strict distinction between work and play and most of the time I’m happy to be working. I find it difficult to leave work behind and often find that something that the kids have said or I’ve seen on the TV sparks some work-related idea.

However, during the summer, because of holidays and child care responsibilities work has to take a back seat. Consequently I fill my days with other things which I start to find just as interesting as work. Projects with the kids absorb me, the challenge of existing in a tent for a week engages me, I read fiction again, the guitar comes out for the first time for ages, I think about learning to Lindy Hop and so on. I don’t do nothing very well, but I do find it easy to throw my self into something or indeed anything.

Inevitably this leads me towards reflection. Do I really love what I do as much as I think? Would a career change be more interesting? For about five minutes last week I contemplated running off to join the circus after a morning failing to learn how to walk on a circus ball and do a Diablo. More realistically I fantasise about balancing my work and play differently so that I can devote Thursdays to learning the tremolo harp or Friday’s to running a counter-cultural happening.
So where does all of this reflective daydreaming lead me? In all honesty, probably nowhere. In fact, probably straight back to work. What I’ve recognised about myself in the past, and what this reminds me, is that I don’t have a vocation. I can be happy (and sad) doing a pretty wide variety of stuff. I throw myself into things and try and push them as far as I can. Career is a lived experience – it is about making something meaningful of your life experiences. It is hardly surprising if holidays, in which your lived experiences are radically different from the rest of your life can make you rethink and seek alternative narratives.

I seek creativity, progression, social engagement and the vain hope of doing some good in the world around me (oh and money of course). If I can find that I’ll probably keep on keeping on. If I can’t you might just find me working up a juggling act in Ibiza.