Exploring Reliability and Validity Concerns in the Values and Motives Inventory
Exploring Reliability and Validity
Researchers have constructed instruments in attempt to empirically measure values and culture belief systems of people and societies. In order for one to measure what some may consider subjective meanings; concrete definitions were given to the types of values that were examined in the “Values and Motives Questionnaire” (Values and Motives Questionnaire, n.d., p.1). The values examined were categorized as interpersonal, extrinsic, and intrinsic values (Values and Motives Questionnaire, n.d.). These categories cover relationships one has with others, themselves, and values that are seen as motivational (Values and Motives Questionnaire, n.d.).
Two pertinent areas to cover when examining the Values and Motives Inventory (VMI) are the issues that encompass the reliability and validity of this instrument. Reliability not only examines the dependability of scores, but also measures and accounts for the standard error of measurement (SEM) within the instrument (Whiston, 2009). The SEM is a hypothesis of what the scores would be if one took the test more than once (Whiston, 2009). Validity is when the test actually “measures what it claims to measure” (Values and Motives Questionnaire, n.d., p.15). So, not only is it pertinent to have consistency with a low amount of error, but it is essential that the instrument accurately measures what it intends to measure (Whiston, 2009).
Whiston (2009) outlines three types of reliability that must be considered when examining an instrument. First “test-retest”, calculates the coefficients between taking the test more than once (Whiston, 2009, p.53). The coefficient is the correlation/relationship between two different sets of data after a test was given (Whiston, 2009). This measurement was not conducted with the VMI, but if so it may have further validated the reliability of the instrument. The coefficient that was used for this instrument was the coefficient alpha, due to the responses having a range from strongly agree to strongly disagree (Whiston, 2009). The coefficients from this data ranged from .52 thru .83 (Values and Motives Questionnaire, n.d.). Although this number varied; all were over half. These numbers are seen as acceptable, but do fall on the weaker end of the continuum when evaluating the reliability of this test.
Next, when examining reliability one can see if “alternative or parallel forms” were conducted when giving the instrument (Whiston, 2009, p.53). This is when there is more than one form of the instrument given (Whiston, 2009). If a correlation occurs between the two sets of data then the instrument can be seen as reliable. Although, the VMI does not have more than one version; various instruments, which are similar in what was being measured was implemented to the sample who also participated in the VMI.
Nonetheless, this procedure falls more on the continuum of proving the validity of the instrument, and not necessarily the reliability. The reason being is that the results proved the VMI was accurately assessing values, which is more a test of validity. Finally, it is essential that the test has an internal consistency of measurement (Whiston, 2009). When examining reliability one must ensure the instrument is stable in measuring what it sets out to measure (Values and Motives Questionnaire, n.d.). This means the instrument produces data that is congruent with what the instrument is intended to measure.
The VMI shows an overall of internal dependability, and a low level of SEM (Values and Motives Questionnaire, n.d.). Overall, the internal consistency surpasses the requirements for a reliable instrument (Values and Motives Questionnaire, n.d.). Nonetheless, the scores of this test are not normally distributed, which impacts the standard deviations of the scores (Values and Motives Questionnaire, n.d.). Regardless, research proved that the deviation of scores was still acceptable, but did not have an extremely high correlation (Values and Motives Questionnaire, n.d.). This test was given in an unsystematic manner and it is unknown if other variables affected the results of the test.
Next, Whiston (2009) outlines what encompasses an “alternate or parallel form” of a test (p.53). This means that two instruments are created from the same instrument and therefore are not only strongly correlated, but are two of the same test. Although, this was not done specifically from the VMI; the results were compared to other tests that examined values.
Weaknesses in the reliability may be influenced greatly by examining the different values, which are particularly important to different cultures, and/or family systems. One should consider conducting this instrument to different demographic backgrounds and compare the varying backgrounds to one another to gain insight into how this can affect the overall findings to make sure all persons are adequately represented in the results.
When one is examining the validity one must ensure that the content is measuring what it is suppose to measure. First, “construct validity” involves ensuring the test is congruent with what the instrument states it is measuring (Whiston, 2009, p.67 In order for a test to be valid; it must be appropriate for its intended use. This was evident with the VMI, because the data showed to be comparable to other instruments, which were measuring values and motivational factors.
Next, “criterion validity” measures if the results are a good predictor for what the instrument is measuring (Whiston, 2009, p.67). The correlations within the instrument appeared to be average and independent of one another (Values and Motives Questionnaire, n.d.). Finally, when assessing the validity of an instrument it is pertinent that the instrument meets the requirements outlined by Whiston (2009) that encompasses “content-related validity” (p.67). This entails the instrument is appropriate for the person who will be given the test. The VMI does encompass “content-related validity”, accurately measuring values and motivation factors among individuals (Whiston, 2009, p.67).
The population that was studied was one-hundred and fifty-nine psychology and MBA students (Values and Motives Questionnaire, n.d.). Although, research needs a starting point; this population of people is not going to produce results that represent the population as a whole. The sample studied may produce skewed results on topics such as achievement considering the total population was continuing their education.
The sample size was not only small, but more than one third consisted of psychology students. Perhaps, people with certain value systems are geared towards having a desire to work in particular fields. In this case, psychology students and business students were only assessed on their value systems. These variables must be taken into consideration when taking the presented results as an actual representation of the population.
Overall, the idea of what the VMI is measuring is interesting, due to the subjective feelings that are being measured. Measuring things such as ones value system is not easily empirically proven. One must take into account how varying cultures, and so forth will reply very differently than a small sample of one-hundred and fifty-nine college students. Further research is needed in order to provide more concrete information on the topics included in the VMI. Although, this instrument proved to be reliable and valid on many levels addressed previously; it is pertinent research is continually conducted to find higher correlations than what was discussed in the Values and Motives Questionnaire.
Values and Motives Questionnaire: The Technical Manual. (nd). Psytech International.
Retrieved from the Liberty COUN 521 website.
Whiston, S.C. (2009). Principles and applications of assessment in counseling (3rd ed.).