Using Reference Structure to Quantitatively Engineer Class Integration

Contents

• Introduction
• Referencing as a Quantitative Measure of Class Integration
• Reference Examples
• Class Measurement Details
• Class Reference Matrix
• Class Reference Diagram
• Using Reference Measurement to Educationally Engineering a Science Class
• Reference Structure as a Design-Time Statistic
• Portability Issues
• Definition of Textbook Measurement
• Textbook Reference Matrix
• Textbook Chapter Diagram
• Textbook Engineering Using References
• Reference Statistics
• Block Re-Use Statistics
• Block Spread Statistics
• Block Localization
• Conclusion

Introduction

A potential goal for a class designer is to communicate the connection of information, so that the student feels they are experiencing a uniform body of information. In this document, we describe the use of a measurement of the linguistic references made in class materials to quantitatively characterize the integration of the class.

A reference occurs when one piece of communication explicitly references another piece of communication. For example, if the lecturer says "as you saw in the first homework problem this week", then a reference between lecture and homework is recorded.

Beyond the goal of integration, a reference measurement can also be used to optimize the choice of discretionary content of the course, such as demonstrations and examples. A example which is never referenced is a less efficient use of material than an example which is referenced.

Referencing As a Quantitative Measure

of Class Integration

Reference- Any time a piece of communication refers in some way to another piece of communication.

Explicit Class Blocks- Part of the class that can be identified without resort to educational expertise: Lecture, Lab, Homework Assignment, Test, Demonstration, or Question/Answer segments of Lecture.

References Between Explicit Class Blocks- In the experiments that follow, references connecting Explicit Blocks of the Class are catalogued. For example, if the lecturer refers to homework problem 29.1, then a reference is recorded between the lecture and the homework.

The Reference Pattern as a Basic Model of Class Integration- If a class actually is well integrated where the various parts work together, then this integration should be exposed in the language used by class personnel and materials. If the lecture never mentions the lab, then it is very difficult to claim that the two are well connected.

The Reference Pattern as a Quantitative Measure of the Class- A reference is a physical feature of the language of the class. It either exists or doesn’t exist. The detection of references does not depend on expert opinion. References are therefore a good candidate for a statistic useful in educational engineering.

Examples of References

• By Label – "As in Figure 2.1"

• By Temporal Location – "In the previous lecture"

• By Spatial Location – "At the top of the page"

• By Name with Referencing Language – "Atwood’s Machine Revisited"

• By Proximity – "The accompanying figure."

• By Description – "In the lab, where you measured g."

Details of the Measurement

The referencing structure of two introductory physics classes were measured at the University of Arkansas.

UPI- a traditionally taught introductory calculus-based mechanics course. All tests, in-class quizzes, and laboratory descriptions were measured. The lectures were video taped, and six, chosen at random, were measured. Homework problems are drawn exclusively from Tipler, so they share the reference structure presented later in the talk.

UPII- a modified-traditional introductory calculus-based electricity, magnetism, and optics course which combines a traditional lecture with hands-on activities. All tests, homework problems, and laboratory descriptions were measured. The lectures were video taped, and six, chosen at random, were measured.

UPI(Traditional) Reference Matrix

UPII(Modified) Reference Matrix

Reference Diagram for UPI/UPII

Using a Reference Measurement to Engineer

UPI

The measurement quantitatively shows that UPI is very sparsely integrated. This results from a number of decisions about the structure of the course; decisions which are being revised:

• different individuals managing the lecture and the laboratory, making lecture -> lab integration difficult.
• a pre-printed laboratory manual, preventing laboratory class integration.
• reliance on textbook homework problems.

After these physical changes are made, increased integration requires the lead instructor to design references integrating the lecture, lab, homework, and tests.

Using a Reference Measurement to Engineer

UPII

The measurement quantitatively shows that UPII is well integrated, but additional improvement is possible:

• use of test problems as the core of future lecture examples.
• requiring the use of a homework solution within a lab report.

Referencing as a Design-Time Engineering

Tool

Many tools to improve education require that the instructor make a change and then measure whether the thing that they wanted to accomplish actually happened.

Referencing can be added to a class before it is presented to students, at design-time. It is further totally reliable in that if a reference is added to a class, the class will have that reference, period.

I note that it is not the references themselves which add the educational value, but the changes in class structure and the selection of integrating problems, tests, demos, and labs which improve the educational value. Referencing is an engineering statistic which measures the extent to which the changes in class components accomplish integration.

Portability Issues Arising from Referencing

Educational Reform initiatives have a poor record of portability between institutions.

Referencing is presumably one of many engineering statistics which could be applied to an educational offering. The specification of a class by published materials will not capture the reference structure of the successful experimental class. This lack of complete specification will mean that there are essential features of the class which are not transported.

Our initial goal for educational engineering is to provide as complete as possible a characterization of a class so it can be transported effectively.

Textbook Measurement

The reference measurement technique was also applied to the best selling introductory physics textbook.

Explicit Textbook Blocks-The textbook has a richer structure of explicitly identified blocks than lecture or lab: Chapters, Sections, Strategies, Problems, Examples, Illustrations, Tables, etc. This allows referencing to provide even more information about the integration structure of a textbook.

Reference Matrix for a Leading Textbook

To Block Type

Chapter Integration Diagram for Leading Textbook

Using Referencing to Engineer a Textbook

The reference graphs show the section and example text does a good job of integrating the section material.

However, there are certain integration problems which were uncovered by the reference analysis.

• No chapter integration is achieved by re-using discretionary blocks like examples, problems, and illustrations.
• Photographs and examples have a poor record of re-use.
• Little integration is achieved through problems referencing plots and data tables.

Statistics Derivable from Referencing

We have been using the graphical representations of the raw data derived from a reference measurement to analyze educational offerings. There are also many useful statistics which can be derived from a reference measurement.

Re-Use- The number of times a block is referenced shows how often a block is re-used. A block that is re-used often is more efficient than a block which is used once and then discarded.

Locality- References can be filtered by how local they are. References which are local to the week of the class or to a section in a textbook provide less integration than longer distance references.

Localization- References can be used to determine how localized a block is in the class. A block than is only discussed within one lecture provides no integration and replacement might be considered.

Spread- The spread of the references from a block shows how well the individual block integrates the class. A ladder of block spreads is probably appropriate.

Textbook Block Re-Use

Textbook Block Spread

Textbook Block Localization

Conclusion

• The measurement of references between explicit class blocks is a powerful technique to examine the integration of an educational offering.
• Traditional classes may have poor integration which is reparable if the designer makes integration a priority.
• Textbooks are well integrated section to section, but do a poor job integrating discretionary objects like examples, illustrations and assignments.