How-To

In order to be considered solid scientific research, content analysis should be conducted in accordance with thorough procedures and guidelines. Researchers should follow nine essential steps below to successfully utilize the content analysis technique. The outline and description of those steps, with minor adjustments, were taken directly from Kimberly A. Neuendorf’s The Content Analysis Guidebook :

1. Theory and Rationale: What content will be examined, and why? Are there certain theories or perspectives that indicate that this particular message content is important to study? (Is there a difference in the way Putin addresses the domestic and the international populace? Does the way he communicates change over time?) Library work is needed here to conduct a good literature review. Will you be using an integrative mode, linking content analysis with other data to show relationships with source or receiver characteristics? Do you have research questions? Hypotheses?

2. Conceptualization: What variables will be used in the study, and how do you define them conceptually (i.e., with dictionary-type definitions)? Remember, you are the boss! There are many ways to define a given construct, and there is no one right way. You may want to screen some examples of the content you’re going to analyze, to make sure you’ve covered everything you want.

3. Operationalizations (measures): Your measures should match your conceptualizations (this is called internal validity). What unit of data collection will you use? You may have more then one unit (e.g., a by-utterance coding scheme and a by-speaker coding scheme). Are the variables measured well (i.e., at a high level of measurement, with categories that are exhaustive and mutually exclusive)? An a priori coding scheme describing all measures must be created. Both face and content validity may also be assessed at this point.

4. Coding: You will need to select the type of coding you are going to use. Two options are available:
Human Coding
4a. Coding schemes: You need to create the following materials:
a. Codebook (with all variable measures fully explained)
b. Coding form
Computer Coding
4b. Coding schemes: With computer text content analysis, you still need a codebook of sorts-a full explanation of your dictionaries and method of applying them. You may use standard dictionaries (i.e., those in Hart’s program, Diction) or originally created dictionaries. When creating custom dictionaries, be sure to first generate a frequencies list from your text sample and examine for key words and phrases.

5. Sampling: Is a census of the content possible? (If yes, go to #6, if no, go to step 7b) How will you randomly sample a subset of the content? This could be by time period, by issue, by page, by channel, and so forth.

6. Training and pilot reliability: During a training session in which coders work together, find out whether they can agree on the coding of variables. Then, in an independent coding test note the reliability on each variable. At each stage, revise the codebook or coding form as needed.

7. Coding: Based on the above sampling (step 5) select the type of coding you are going to use. Two options are available:
Human Coding
7a. Coding: Use at least two coders to establish inter-coder reliability. Coding should be done independently, with at least 10% overlap for the reliability test.
Computer Coding
7b. Coding: Apply dictionaries to the sample text to generate per-unit (e.g., per-news-story) frequencies for each dictionary. Do some spot check for validation. Skip step 8.

8. Final reliability: Calculate a reliability figure (percent agreement, Scott’s pi, Spearman’s rho, or Pearson’s r, for example) for each variable.

9. Tabulation and reporting: See various examples of content analysis results to see the way in which results can be reported. Figures and statistics may be reported one variable at a time (univariate), or variables may be cross-tabulated in different ways (bivariate and multivariate techniques). Over-time trends are also a common reporting method. In the long run, relationships between content analysis variables and other measures may establish criterion and construct validity.