Building on my previous post about innovative research methodologies in the field of education and digital media, I wanted to take a minute to consider the issue of assessment - an aspect whose significance is often overlooked or undervalued in the context of digital media education programs. Recently I have become very interested in issues of assessment, after having tested and validated a comprehensive survey tool to measure new media literacies in both youth and adult populations (the full report of the study is forthcoming, but in the meantime you can check out an executive summary here). I decided to pursue this line of research partly due to my disappointment at the lack of an appropriate tool to measure such skills, even though their increasing significance in the educational realm is, as of now, widely (dare I say unanimously?) recognized. Having an appropriate assessment tool to measure students' progress in the field of digital technologies and media literacy would not only paint a valuable picture of learners' skills across a variety of domains, but it would also facilitate the effective crafting of future educational interventions that are in line with the needs and propensities of the program beneficiaries.
One of the main challenges, however, is the methodology issue: what is the best way to assess learning in digital environments and get a sense of learners' digital skills and media literacy levels? Previous attempts to assess media literacy have mostly been centered on self-reported quantitative measures, due to the difficulty of feasibly gauging media literacy levels across different platforms and various media content. However, the reliance on self-reported measures brings with it a wide set of problems concerning the accuracy and reliability of the findings; in the case of new technologies and digital media, this is further complicated by the issue of social desirability, which may affect the empirical/objective accuracy of the self-reported responses. Conversely, the reliance on qualitative data - which is typical of most studies in the field of media literacy - means that such assessment projects are not feasibly replicable with larger groups.
In my experience with the educational assessment of digital skills and media literacy, and considering the inherent shortcomings of self-report methods, there is the need to combine quantitative measures with more qualitative evaluations, in order to achieve a fuller and more precise. For instance, a survey can be combined with practical workshops, where participants are able to actually demonstrate the skills that they report having. There is also a need to better understand the relationship between baseline and endline evaluations. Especially in the case of program evaluations, it is crucial to determine whether surveys such as this one can be used for both baseline and endline assessments, and specifically, what particular changes need to be made in order to best underscore the differences in media literacy levels between these two points of evaluation.
Some researchers and educators are pointing to the need to create national standards of media literacy assessment; the infamous iSkills, developed by the Educational Testing Service (ETS), is one such example. Although iSkills in particular did not gain substantial popularity and has been criticized widely for its rather narrow approach, it is expected that such measurement tools will become more common - and hopefully more refined - in the near future. Nevertheless, in developing such standardized measures, one must consider the issue of scalability and cross-cultural applicability. Can such an assessment tool be used universally, with different populations and in different types of program evaluations? While a national (or perhaps even international) standard of media literacy evaluation would certainly be useful and instrumental in educational endeavors, my experience with the new media literacies survey indicates that an over-standardization of such assessment tools can lead to a decrease in the validity of the findings.
