Advancing Quality Through Additional Attention to Results
by Judith S. Eaton
"Quality is Job 1"
Remember the advertisement from a major automobile company that says, "Quality is Job 1?" Quality here is a matter mainly of how the product performs: perhaps the car runs without incident for 50,000 miles, is fuel-efficient, and boasts a classic design. This advertisement does not invite us to consider how the company goes about producing cars; the product is offered as the ultimate test of quality. "Quality is Job 1" is based primarily on demonstrable results.
Consider a familiar alternative approach to warranting quality - an approach based on resources expended and processes used - and apply it to an automobile company. Following this approach, we would focus on how much steel and energy the plant uses to manufacture a car. We would review the size and condition of the facility and ask whether state-of-the-art machinery and manufacturing methods were being used. We would look at the number of employees and how they were compensated, and verify how many cars were produced and how many were sold.
This approach to warranting quality describes accreditation as it has traditionally been practiced in higher education. Accreditation has depended on an examination of institutional budgets and endowments, physical facilities, library and equipment resources, entrance requirements, faculty salaries and credentials, administration and staff size, and the number of students enrolled and graduated. This is still the standard approach to accreditation, although recently we have seen some changes, with more attention to assessments of institutional effectiveness.
The accreditation claim of quality that is based primarily on resources and processes has provided an important foundation for trust among higher education insiders. Educational institutions that grant admission and academic credit for student work done at other institutions rely on one another's judgment about, for example, curricula and standards. Accreditation provides a basis for trusting that judgment.
By contrast, "Quality is Job 1" conveys a clear message to the world at large - a message about results; a message that traditional accreditation typically does not deliver. Would you buy a car from a company that based its claim of quality exclusively on the resources it expended and the processes it employed, and provided no information about the performance of its product?
We all know what is wrong with this comparison: Producing cars and educating students are not analogous activities. Educating students depends on what students do as well as what the college or university does; the same cannot be said of car manufacturing. The results of educating students are not always immediately apparent and may not be apparent for years; the results of car production are measurable relatively quickly. Students are not "products."
But while we do not have a perfect analogy here, we do have an important message: The claim of quality in higher education cannot be isolated from the educational results that are produced.
Barriers to Connecting
Quality and Results
What if we tried to adapt accreditation in order to tighten the connection between quality and results and to enhance the clarity of accreditation's message about quality? While some accrediting organizations have begun to address this challenge, their efforts have confronted barriers from resistance to discomfort to confusion.
Resistance emerges when the effort to pay attention to results is perceived as equating students with "consumers" and treating higher education as a "market." Although the higher education enterprise is responsive and competitive (both terms that have market connotations), this has usually been tempered by a conviction that colleges and universities must hold themselves a bit apart from the market in order to avoid becoming captive to fads and fleeting consumer interests. Academic culture has deep roots in a long tradition that views colleges and universities as, first and foremost, communities of intellectual growth and knowledge development. This tradition is not easily reconciled with an emphasis on results perceived as tied to a market mentality.
Resistance also emerges when higher education's critics fail to understand a crucial point about quality assurance through accreditation: that accreditation, as it has evolved over
the years, is very much an affirmation of certain values which are central to our thinking about higher education. These values include collegiality (hence accreditation's extensive reliance on a volunteer corps of professionals who undertake peer review, not compliance review); self-improvement (hence our determination to make accreditation a means of helping colleagues to improve their institutions, rather than a means of dictating improvement); and institutional autonomy (hence our great reluctance to establish inviolate standards in the accreditation process independent of institutional purpose).
Discomfort emerges when the quality discussion turns to a particular kind of results, "outcomes." Reflective education leaders who make the case against outcomes offer several points:
- Outcomes are usually attached only to student learning. Our institutions are about more than student learning; they are also about research and service;
- Outcomes are hard to measure;
- We are already engaged in faculty development and attempting to increase our resources, leading to improvement in student learning.
Confusion about quality and results abounds in part because we frequently speak of the "quality" of our resources and processes, not taking into account that this is different from the "quality" of our results. We confuse, for example, the quality of the teaching environment (resources and processes) with the quality of student learning (a result). Both are important; they are connected; but they are not the same. This confusion is compounded by a tendency among some of us to interpret public call for information about results as a call for better public relations rather than a substantive accounting to society in return for its investment in higher education. Some believe, for example, that if people just understood our issues, everything would be fine.
Is There a Quality and Results Scenario That Works?
Yes. We can diminish the barriers of resistance, discomfort and confusion by sustaining our commitment to scrutinizing resources and processes, while adding an intensified scrutiny of results. Accreditation practice might then be viewed as an examination of three key aspects of institutional quality: resources invested, processes followed, and results achieved. A quality and results scenario, in other words, calls for additional attention to results, but not at the expense of attention to resources and processes.
Such an approach to accreditation might start with a review of resources available to an institution, including reasonable expectations of its particular student population. We would then set institutional goals that reflect the expected results of institutional efforts, given the mix of resources available. This would be followed by compiling the evidence - confirming the progress that is being made toward those goals over time. This evaluation of progress should ideally be informed by a comparison of similar institutions.
To expand this a bit, a quality and results scenario involves:
- Making a commitment to results. This means adding an expanded examination of results to the scrutiny of resources and processes. We should continue to confirm the adequacy of resources and processes for our institutions, while focusing more attention on educational results. For despite one of the points about outcomes above, increased investment (in faculty development, for example) does not automatically yield enhanced results (student learning, for example); the link must be demonstrated.
- Examining resources. This examination should include an institution's fiscal and other resources and involve assessment of an institution's student population (academic preparation, educational goals, and socio-economic status), and is the basis for setting defensible expectations, for example, of student performance.
- Tying results to institutional purpose. Institutional goals describing expected results need to be appropriate to each institution's purposes.
- Expecting results in research and service as well as student learning. Expecting results in terms of the research and service purposes as well as the teaching purposes of our institutions will broaden the "outcomes" discussion and address the first point about outcomes described above.
- Deciding what evidence is needed to determine results. Institutions need to decide what constitutes evidence for results. Evidence can embrace, for example, student achievement while in school and success in future education. It can include gains in research and success in public service initiatives. This evidence, coupled with resource and process indicators, provides a comprehensive institutional profile of evidence for quality.
- Confirming progress toward results. Institutions then need to collect their evidence to confirm progress toward goals. While measuring progress is not easy (as those who argue against "outcomes" suggest), that is no reason not to try it; few of us entered higher edu-cation because we believe teaching and learning, research, and service are easy.
- Comparing results. Over time, institutions should consider benchmarking their expectations and accomplishments and moving toward community standards. Ultimately, this will make the measurement of educational results both easier and more reliable. For example, examining the accomplishments of institutions with similar student populations and resources will assist in setting expectations for results in the future.
Higher education can enhance the credibility of its quality claim if colleges and universities set their own quality expectations and indicators; if these expectations are based on institutional purposes, resources, and student profiles; if results are routinely measured and used to guide change; and if, ultimately, a benchmarking capacity is established.
Students are not cars. Colleges and universities are not automobile factories. But we are concerned about the performance of students and cars, colleges and factories. We do not have to like the comparison to learn from it.
Our society has invested heavily in us. It is an investment, not just of money, but of faith in the value of education for individual benefit and the public good. That investment of faith places an obligation on every one of us in the higher education community. It is an obligation to be attentive to the results of our enterprise -- for the wellbeing of society and of the enterprise itself.