Many of you know that I am currently wrapping up a year’s worth of research with the CCO. My job this year has been to try to paint an accurate picture of the work CCO has have been doing in the realm of discipleship with college students. Coming into this year, I had exactly no experience in conducting formal research. You can probably imagine some of the challenges that came along with that.
On this side of the research, I wanted to chronicle some of the things I learned along the way both as a way to bring closure to the project. As I started writing, however, I realized that the lessons I’ve learned might be helpful to others who interact with research in some way.
So I identified 10 lessons hoping that they would be of some use to you as you navigate your professional (and maybe personal) lives.
Lesson #1: Questions trump Answers in Research
The best research begins with the best questions. Just about anyone can ask a question, but it takes a thoughtful and careful person to ask the right question. I learned that they ways in which you phrase questions, the words you choose, and even the word order of a question can all drastically altar the response you get. Most of those things are based on your assumptions and your bias (more on that in a minute), which tend to leak out through the questions you ask. This was an long exercise in learning to be careful.
Lesson #2: Your team matters more than you think…but not for the reason that you think
Surrounding yourself with people who think like you, talk like you, and are motivated like you are is a recipe for them agreeing with you. That’s problematic if you have a bias in your research (which we all have.) A highly homogenous team will tend to ask narrow — and most likely unhelpful — questions. So we need other to help us identify our blind spots because you literally can’t do that alone.
Lesson #3: “Measure twice, cut once.”
Excellent planning sets up superb research. It was pretty common to need to rewrite a question because someone (or a few someones) thought I meant something else. When you realize that your wording isn’t getting you the data you’re looking for then you need to change no matter how right you think you are. Why? Because its better to find out that you’re asking the wrong question after 3 test surveys than it is to figure out that same thing after 70 open-ended surveys you can’t administer again.
Lesson #4: Keep your data clean
You can have the most compelling and accurate data in the world but if you and others can’t make sense of it because of its organization, then its practically worthless. I’ve found that you can tell how well a researcher has kept their data set through the ways in which they choose to communicate their findings. Simplistic and caricature explanations based on loose figures point to a researcher who was merely looking to confirm their own pre-scripted conclusion, rather than one who has followed the data they’ve discovered. Clean data usually leads to intuitive interpretations and communications. (**For the record, intuitive and simplistic are different things.**)
Lesson #5: People process data in extremes
There are two common errors people make when interpreting data. The first is to rely purely on the numbers and to reject the stories or exceptions behind them. The second error is operate solely in anecdote and exception in spite of data-driven trends. Most people tend towards one or the other, so you need to be able to speak both languages when you talk about your data.
Lesson #6: Sample size matters…a lot
Sample size refers to the number of entries you have in a given data set. This gives your findings authority to speak into the action taken as a result of your research. For example, suppose you gathered information from 10 people who work for a 1,000 person organization and you use that data to make sweeping policy changes within that organization. That would suggest an abuse of the authority of that data because those 10 people likely don’t totally represent all 990 of the other people. Conversely, receiving 800 responses in the same company doesn’t guarantee that your data speaks for every person even if it does speak for most.
Lesson #7: Read the literature landscape before you get started
I used to think that the literature review section of a research report was nothing more than courtesy. In reality, this sections help to set the context of why the research being collected is even necessary! Without the pre-work of understanding the broader landscape of our topic before diving in, we would have made some rookie mistakes in the conceptual understanding of our problem.
Lesson #8: Ask even the obvious questions
I can guarantee that even the most senior researcher has assumptions or preconceptions about a subject they plan to investigate. Sometimes you find that you’re mostly right, but maybe not for the right reasons. Confirming “obvious” data will help round out your interpretations and give credibility to your findings as a whole.
Lesson #9: Investigate with Complexity, Communicate in Simplicity
This was advice given to me by someone who is much better at research than I am. They suggested I use methods of capture and analysis which respect the subject matter you’re dealing with. I.e., if you’re studying something complex, then use match its complexity where necessary. The trick is to communicate simply enough that anyone looking at your research can understand what you’re trying to say.
Lesson #10: Correlation does not imply causation
This is essentially a warning to hold your findings open-handedly because its nearly impossible to account for every possible influence on your subject matter, especially when that subject matter is a human being. Best practice after identifying an insight is to step back and ask, “What all could have affected this?” and, “What all might this effect?”
There are certainly more things that I learned throughout the process of developing a research project from scratch than what you see here. But these lessons seems to cover the breadth of my training-on-the-go, including the hard lessons I needed to learn. What do you think? Are there any surprises on this list?