I was going through my archives of presentations, papers etc I’ve written over the years and stumbled on to a presentation I prepared on writing qual recruitment screeners.

Boring, basic, who cares? Right? Well yes and no….

It is really my firm belief that the qual recruitment questionnaire is one of the most important parts of a qualitative project – yet it is often dismissed, hurried through, passed on to more junior staff and generally paid very little attention.

….

So why do I think it is important?

Well firstly, there are some differences between qualitative and quantitative projects that you may have noticed.

….

One… Numbers

In qual we generally have a lower margin of error to work with…

We all know that qualitative samples are smaller – however what does this mean for the project management?

  • If one person is wrong – it’s noticeable
    • If 4 people don’t participate – we need to repeat our fieldwork
    • If the invites don’t go out – it’s hard to recover from
    • If we have the wrong quotas / sample etc – we can’t easily remove them from our sample
    • It’s one night critical – if we get the wrong night, the wrong venue, the wrong participants – there’s little room for comeback
….

Two… Visibility

Where can we get our reality TV fix now that Big Brother has been cancelled?

The fieldwork component in qualitative is generally a lot more transparent and visible than in a quantitative survey – having a good night out, watching the group often forms an important part of what the client is paying for. Attendance at qualitative fieldwork can be part of KPIs (customer contact); it is also the opportunity for any negative stakeholders to dispute or undermine the findings of the research.

So we have clients right there watching us when the participant who is supposed to be a user of Brand A, tells us how their husband works for Brand B, and they actually always buy Brand C. There is no buffer, we can’t cleanly and clinically remove their responses from the sample like we would in a questionnaire – this off spec participant WILL impact on the rest of the group no matter what we do next (remove them will unsettle the other participants, leaving them in will skew the results).

….

Three… Time and Costs

The very nature of focus groups and a qualitative project also increase the pressure of getting the screening right on the night. There are a number of additional pressures that are inherent in many qualitative projects that make poor recruitment even harder to come back from:

  • Shorter turnarounds: The report is due next week – we’ve only one night to get it right!
  • Costs: It has cost roughly $2,080 just to get these people here, fed, in a room – if recruitment is off, you still need to pay them an incentive – we can’t really afford to do this twice
….

Impact of poor recruitment

Still many have a near enough is good enough mindset when it comes to writing their screeners. Surely a good moderator can ‘cover’ these issues up? Surely it isn’t really that much of an issue – there are still other participants in the room that are on-spec?

Not so. Poor recruitment is always obvious…

….

So, what do we do to make sure that recruitment is on spec, every time.

Off spec participants, clearly the recruiters fault, right?

Not necessarily. Often when I go back to look at why recruitment is off spec it is the screening questionnaire that is at fault, not the recruitment itself. Failure of the researcher to apply critical thought to the drafting of the recruitment questionnaire is often where the error arises. Yes, the recruiter could/ should have flagged an issue with a question. Yes, the client should have picked that up when they approved the question. Excuses, excuses and I don’t want to hear them. As researchers, the buck stops with us.

….

Key things to remember when writing screeners

Structure, Structure, Structure

Start Broad – Order questions to make sure those which screen out the most people are first.

Don’t Give the Game Away – Try to put more general questions first, specific questions later.

Apply the KISS principle: Imagine your Mum, reading this out from paper at home (sorry Mum!). This is how many recruiters are set up – at home, using paper based questionaries. There are no complex CATI systems or online programming like in your quant questionnaire. For qualitative recruitment, keep it simple. Complex routing is unlikely to understood and will lead to mistakes – simplify all instructions.

….

Questions must not ‘give away’ the screening requirement

Don’t make it easy to cheat – Keep in mind what you are telling them in the question – are you letting them know how to answer to qualify? Amazing but true, there are people out there who are willing to LIE, yes LIE, to get $80+ for a couple of hours work. Don’t make it easy for them.

….

Tight specifications lead to tight groups

Detailed and specific specs – Specs should be clearly set down, everything your participants should be must be specified at the beginning of the questionnaire. (I prefer to have a separate page at the beginning of the screener that details who exactly should be in each group and what the group should look like).

Secondly, each specification must have a question. Sounds obvious, but often it is not the case. If there is no question there, then the specification won’t be delivered.

Remember, this is qualitative research – we need the most representative people in the group, so we shouldn’t be afraid to write tight specifications. More and more in qualitative research we want the “sharp point” of the behaviour or attitude. In order to get that ‘sharp point’ we need to write detailed specs. As researchers we shouldn’t be afraid to:

  • Make it difficult to qualify, make the specs tight, define exactly who it is you want to talk to
  • Be descriptive, specific and detailed in the statement of the specification
  • Link the questions back to the specifications
  • THINK CRITICALLY. Ask yourself, does answering this question this way mean that this person will meet this spec (what else could it mean?)
….

My Six top tips for Qualitative Recruitment Management

There are six key things I tell researchers who are starting writing their own recruitment screeners

1. Anything that can go wrong, will go wrong

Be paranoid. It is important to think about all the possible things that could go wrong with a project. e.g. food testing and not excluding food allergies or diets. Copy testing with no glasses or can’t read English. Aside from the ‘research’ issues, what other issues could arise from left field that would impact on the study

2. Apply the ‘stupid’ factor to anything you ask others to do

I am normally a ‘glass half full’ kinda gal – I like to think the best of people and generally like to believe ‘it’ll be right on the night’. However this will not fly when you’re thinking about your recruitment screener. No one will give your project as much thought as you do – because no one’s ass is on the line like yours’ is if the project goes wrong. It is important to look at things critically, to understand how people who are looking at it in a non-critical way could misinterpret. For example, once we changed groups from Parramatta (about 1hr drive out of the Sydney CBD) to be held in a Sydney CBD venue. The recruiter only moved the first group of the evening to Sydney CBD, and continued to recruit the second group to the Parramatta venue.

3. Check it, double check it and check it again.

See Parramatta example above. Don’t assume that because you’ve asked for something to be done, that it will be. Check changes have been implemented, ask for screener responses and check them.

4. Get everything in writing

If these things go wrong, having it in writing is the best protection we have. Confirm all telephone conversations with venues / recruiters etc in writing to ensure we have adequate protection if they make a mistake. If something does go wrong, this could mean the difference between having to wear the costs yourself or being able to trace the mistake back to the supplier (or the client!) and getting them to foot the cost for the error.

5. Copy, copy, copy

Short turnarounds mean that if you are away the project will still need to move. Important that there is someone else across the project at all times. Copying another researcher involved in the project in on emails is essential to ensure that projects will continue if you are hit by a bus or are out sick for a day. It is not checking up on you (well, at least that’s not the main reason), it is making sure that that the project can progress in your absence.

6. Your recruiter will either be your best friend or your worst enemy – relationships are key

Spend time getting to know and building relationships with your recruiters. A good recruiter will be your partner and advisor. I value recruiters who critically look at your screening questionnaires and say ‘that just won’t work, have you considered this instead?’ – this shows me that they are invested in your study and getting it right. Red flags emerge when recruiters take screeners as written and do not offer any feedback or raise any questions about the study.

8 thoughts on “The Devil in the Details: Qualitative Recruitment Screeners

  1. I read your article re qual recruitment with interest, Victoria. As a data collection agency we occasionally undertake qualititative recruitment for groups and also for depth and/or F2F interviews, but still I have often needed to clarify objectives and to edit loose screening questions on clients’ behalf in order to get things right for their group or depth interview. I agree that qual recruitment screening needs to have more time devoted to it than quant, not less – not only is the quality of the research more easily affected by the smaller numbers, but also there is the ‘double jeopardy’ of having to pay an incentive to an incorrect recruit.

    Another element of qual recruitment that can be overlooked is one that should be obvious but is not always so – i.e. whether the recruit is actually suitable for the target focus group, in terms of the type of input being sought from them. They may fit the screening criteria perfectly, but if the recruiter has to coax information out of a shy / reluctant target on the phone, they should be asking themselves “What will this person be like in a group context?” It’s perhaps the opposite conundrum to that of avoiding serial / professional focus group attendance – yes, it’s always good to bring in fresh voices, but only if those voices will actually be used and the recruit will not remain largely silent during the process! Having ‘sold’ the idea of the research to the potential recruit and screened them cleanly, the recruiter should almost be asking himself/herself “What reasons might there be for this person not being right for the group?” It’s arguably worth putting includeing some ‘words’ to this effect at the end of the screener.

    I guess the bottom line is the adequacy of the budget. Given that the recruitment is so important, trying to achieve it on the cheap is arguably asking for a low-qual approach. Having had a couple of instances of clients being unhappy with incorrect or insufficient recruitment, we now monitor 100% of all recruitment calls (either live or on the day from recordings), and we cost recruitment accordingly. That isn’t cheap, but it does help to ensure that the pitfalls you describe are spotted in advance and avoided where possible – and before it’s too late to act. It also helps us to preserve our reputation as a high quality data provider.

  2. Thanks for your comments Ann-Marie. I think you make a great point too. All too often I will see requests for groups with people who just shouldn’t be approached that way. And as you point out, there are certain people who, whilst on spec, will just not be good participants because they are reluctant or shy. I think this is where the dialogue between researcher and recruiter and client is really key. Researchers must be open to and listen to feedback from their recruiters, but also provide feedback at the end of the group (this person was shy, had trouble contributing, had difficulties reading etc) – so that the recruiter can mark their database, to ensure that this participant is only used when they would be appropriate (for instance, a one on one setting).

    Another point I should have made, but didn’t (there were just too many points to make!), is that whilst tight specs are important – they still need to be achievable and realistic. I think there is a real art to getting the balance between recruitment specifications that are ‘specific’ enough and those which are unrealistic and prohibitive. Again, client and researcher needs to have flexibility when dealing with tight specs – knowing which areas may be relaxed and which are essential.

    It sounds like you have some good measures in place to ensure data quality – it is great to see! Thanks again for your comments

  3. Looking through your ‘achieves’ or ‘archives’ of preso etc V?!? 🙂 though I’d say all of them are achieves (especially around screeners!!) S

    1. damnation – spell check doesn’t pick those things up! Thanks for the heads up!!!

  4. Hi Victoria –

    Very interesting piece! Over the course of the past few years working in online qual, I’ve found there are some other considerations to keep in mind that differ from quant and in-person qual.

    For online qualitative, I’ve found the articulation question works best when it’s a writing sample. Having research participants who are able to communicate effectively in writing is key in an online environment.

    In our practice, most projects take place over the course of several days (with some spanning weeks or months). We look at the start rate but also the “stay rate” – i.e. how many folks went the distance through the full project. A strong stay rate is a good indicator that the project was a success.

    And our services team focuses on three stages of recruitment:

    Acquire – get the participants who qualify, have access to the required technology and can commit to the time frame.

    Engage – In the rush to get things done for the client, it can be easy to forget that participants are working to fit this research in between work, home and other obligations. Be realistic in the amount of work you expect participants to do. Also, keep it interesting for them! Doing the same diary entry every day for two weeks can get boring for the participants.

    Retain – cheerlead throughout the project, offer “carrot” incentives when needed for longer projects or more time consuming assignments like videos or photo tours. And be flexible when participants fall behind – maybe give them an extra day to catch up on assignments. As we know, sometimes life gets in the way!

    Victoria, I’m interested in hearing what you feel has worked successfully for you when recruiting for online qual.

    1. Hi Rachel – I’ve been sitting on your comment for some time as it has given me a bit of food for thought.

      So far I haven’t been doing too much different for online qual recruitment to what we do for face-to-face – but you are right that there are differences in method and we should be reflecting that in recruitment.

      I think your tips above are all very good and valid. I think a written articulation question is a great idea – though I’m trying to think how I would do this as most of my recruitment for online we do via telephone…. but definitely a good idea!

      Thanks for posting!

Comments are closed.

Additional comments powered byBackType