July 2, 2001

Evaluation Standards for MERLOT/Physics

The following information is provided for users of and contributors to MERLOT/Physics to help them better understand the criteria used to evaluate items in the database and the Peer Reviews. This information is also used to provide new reviewers the background necessary to quickly participate in the MERLOT Peer Review process.

GENERAL CATEGORIES:

The three general categories for evaluation to be used within MERLOT/Physics are:

1. Quality of Content
2. Effectiveness as a Teaching-Learning Tool
3. Ease of Use

These three categories are interdependent to some degree, but separate ratings are given for each. The "Effectiveness" rating is given with respect to a specific use of the material (e.g., Lecture/Demo, Tutorial, Homework Assignment).

The purpose of MERLOT/Physics Peer Reviews is to provide information for users on the best material in the database. All items with Peer Reviews have been identified as quality material, useful for instructors and students. Reviews are on a 1 to 5 scale, with reviews of 3 and higher posted on MERLOT and lower reviews sent to authors for their information. Authors see all reviews before they are posted and have a chance to respond to the review, or not have it posted. The MERLOT/Physics Editorial Board is working to write Peer Reviews for as many quality items as possible. Two reviewers are assigned to each item. They each perform an individual review, then collaborate to create a composite review. This is sent to the author and, if approved, posted on the MERLOT site. If needed, a third reviewer and/or an Editorial Board Co-leader will help with the composite review.

The criteria listed below outline the characteristics of quality material in the areas of Quality of Content, Effectiveness as a Teach/Learning Tool, and Ease of Use. This list is not meant to imply that materials need to meet all these criteria to be highly rated on MERLOT. Rather, these questions are used in the MERLOT review process. More specifically, material that is deemed basically useful is given a rating of 3. Material that positively satisfies two or three of the main questions in each of the categories below is given a rating of 4. Material that positively satisfies most of the main questions is given a rating of 5. As more experience is gained in reviewing multimedia, online materials, this list may change.

QUALITY OF CONTENT:

1. Does the material present valid (correct) concepts, models, and results?
   a. For quantitative material, are the numerical results correct and in agreement with graphics or figures?
   b. For qualitative material, are the trends correct (e.g., doubling an input doubles an output for a linear relation)?
   c. Is the textual material accurate and precise?
   d. Does the material follow clearly defined physics notation and conventions?
   
2. Does the material present important physics concepts or models?
   a. Is the content part of the core curriculum in physics?
   b. Is the content difficult to teach and/or learn?
   c. Is the content a prerequisite for more advanced material?
   d. Is the content unique, not covered in standard collections of learning material?
3. Does the material help develop conceptual understanding of physics?
   a. Does the material use a variety of ways to present physical concepts?
   b. Does the material represent an original approach to the topic covered?
   c. Can students interact with the material and receive timely feedback about the impact of changing physical parameters?
4. Does the material make effective use of graphics and multimedia?
   a. Are the graphics attractive?
   b. Do the graphics/media effectively illustrate physical quantities?
5. Is the material flexible?
   a. Can the material be used to explore a range of different physical situations, or is it applicable to only one or a few physical systems?
   b. Is there control over the initial state of the system and other important physical parameters?
   c. Is the material modular so that it can be adapted to different physics problems?

EFFECTIVENESS AS A TEACHING-LEARNING TOOL: (Use Stated)

1. Does the material improve faculty and students' abilities to teach and learn?
   a. Does the material effectively demonstrate physical concepts?
   b. Is the material engaging and/or entertaining?
   c. Does the material encourage students to generalize concepts or make predictions?
2. Does the material promote and/or use effective learning strategies?

a. Does the material promote active student engagement?
b. Does the material help develop critical thinking skills?
c. Is the material suitable for group projects or peer instruction?
d. Will the material promote student discovery?

3. Can the material be readily integrated into the physics curriculum?
   a. Does the material fit into standard presentations of physics?
   b. Can the material be used with standard textbooks or problems?
   c. Does the material extend the presentations in standard textbooks?
4. Can the material be used in a variety of ways?
   a. Are there uses that the material is particularly well suited for? (e.g., lecture/demo, group projects, homework assignments, tutorials, …)
   b. Are there uses that the material is particularly poorly suited for?
5. Are the teaching-learning goals for the material easy to identify?

EASE OF USE:

1. Does the material operate in an understandable manner?
   a. Does the material load and run on (a) standard computer system(s) in a manner transparent to experienced users?
   b. Can the material be used by students with different levels of experience with computers and/or the web?
   c. What is the expected learning curve for students and faculty? d. Will students become lost or confused while using the material?
2. Is the general layout of the material consistent and intuitive?
   a. Is the design attractive and inviting for students to use?
   b. Do buttons, sliders, and other elements have obvious uses and work well?
   c. Are the input and output elements easy to understand, operate, and read?
3. Does the material provide effective feedback?
   a. Is information displayed in a manner with which students will be familiar?
   b. Is information displayed in a graphical manner?
   c. Is feedback immediate, while users are interacting with the material?
4. Is the material documented and does it have useful instructions?
   a. Does the material include physical explanations, equations, or derivations?
   b. Are documentation and instructions available and easy to understand?
   c. Is there interactive or case-sensitive help?
   d. Is technical support necessary for either instructors or students?
   e. Have the authors described any technical problems with the material?