Does performance-based funding work?

View the full results of the Wall Street Journal/Times Higher Education College Rankings 2020

Like most public institutions in the US, the California State University system – the nation’s biggest – has long been locked in a pattern of low graduation rates and declining state funding.

Three years ago, the institution embarked on a strategy to break out, convincing state lawmakers to provide more money in return for a promise to boost completion rates across a system known for serving disadvantaged students.

“We went to the legislature and said: ‘Listen, we have a lot of ambition to serve the students of California a lot more effectively,’” says James T. Minor, a senior strategist in the CSU system. “‘But success is not free: it costs money, and we are woefully underfunded in our public higher education system.’”

There is a lot of nuance and certainly some disagreement about what’s happened since then. But, as data assembled for the Wall Street Journal/Times Higher Education US College Rankings 2020 help demonstrate, at least two important things have changed.

First, state funding for CSU has increased, from just under $6 billion (£4.9 billion) in 2016-17 to more than $6.4 billion in 2018-19. Second, graduation rates have shot up across the system, with many CSU campuses reporting four-year gains in the order of 30 to 40 per cent, much of them coming in the past year or two.

Performance-based funding for higher education has become both commonplace and controversial across the US in recent years. The first state to adopt it was Tennessee, in 1979. A 2017 analysis by the policy study group Research for Action found that it had had mixed results, with gains on some student outcome measures and drops in others. More broadly, the Century Foundation found in 2016 that performance-based funding models reinforce institutional disparities and make little impact on improving completion rates. And a report by the centre-left US thinktank Third Way last October said that “overwhelmingly, the empirical research on performance funding suggests that in most current iterations at the state level, the policy fails to improve degree completions and graduation rates”.

But such doubts about its efficacy haven’t stopped the idea spreading across the country – and, indeed, over the border. Earlier this year, Doug Ford, the premier of Ontario, followed up budget cuts with plans to allocate 60 per cent of the Canadian province’s university funding by 2024 on the basis of institutional performance across 10 metrics – six of which reportedly will concern student outcomes, with the others reflecting external engagement. Ford depicted the move as “restoring accountability to Ontario’s postsecondary education system”.

Top 20: resources

Meanwhile, across the Pacific, the Australian government recently published a report that it commissioned on how performance-based funding might work there. The report, drawn up by a panel headed by Paul Wellings, vice-chancellor of the University of Wollongong, suggests that 7.5 per cent of universities’ teaching grants should be allocated on the basis of performance on four measures: access, retention, student satisfaction and graduate employment.

The scheme has some similarities with the UK’s teaching excellence framework, introduced in 2017, which was originally supposed to determine whether universities would be able to raise their tuition fees. However, critics fear that the Australian move could see universities punished for things they have little control over, or pushed in contradictory directions.

Nevertheless, the WSJ/THE data for CSU and beyond suggest that a meaningful combination of money and motivation can – in the right conditions, at least – breed success. As a general trend, those states that have most revived their spending since the Great Recession a decade ago – often with some incentives attached – are seeing more successful institutions.

At CSU, Minor says, the pressure to meet goals set jointly with the legislature has helped “our campus leaders, our faculty, our students even, to be more aware and conscious in their day-to-day decision-making”.

It is a similar story in Florida, which is six years into a formula pitting 11 institutions against each other on a set of 10 criteria that cover many of the same factors as the WSJ/THE data, including graduation rates, job success and access for low-income students. Florida’s formula gives extra credit for filling jobs in areas that lawmakers have designated as deserving special emphasis, such as healthcare, science and financial services.

Those cheering the approach include Ralph Wilcox, provost at the University of South Florida, which has been ranked in the top three among the 11 for all but one year of the programme. The incentive-based distributions cover only about a quarter of state funding for higher education – which, in itself, accounts for only about a fifth of USF’s overall budget. Yet that still amounts to tens of millions of dollars, and can mean the difference between a net budget increase and a decline.

“We’ve done really quite well,” says Wilcox, describing strategies that include sharing valuable research dollars with undergraduate students to keep them more invested in completing their degrees.

The state overall seems happy too. A report this summer by the State University System of Florida’s Board of Governors touted “significant improvements in metric scores” across the 11 institutions participating in the performance-based funding model.

Examples, it said, include average system-wide four-year graduation rates of 53 per cent, up from 41 per cent for 2008 freshmen, and already exceeding its 2025 goal of 50 per cent.

However, the stress involved in meeting such targets quickly may be growing less welcome.

At Florida A&M University, one of the nation’s largest and most accomplished historically black universities, 63 per cent of students rely on federal Pell Grants. That puts the institution ahead of the pack in terms of helping lower-income families: one of the measures in the formula that Florida has used to distribute nearly $1 billion in performance funding since 2014. Overall, however, that kind of service hurts Florida A&M on many of the state’s other metrics, placing it in the bottom three among the 11 Floridian participants: a position that translates into very little extra revenue.

Florida A&M’s advocates have suggested changes to the system that would make the distribution formula far more sensitive to each university’s individual mission and student body.

From his perch among the top-three performers, however, Wilcox gets “a little nervous when there are whispers of changing the methodology”. South Florida is “generally happy” with the system in its current form, crediting it for the incentive it creates for institutions to develop “a pretty strategic rhythm” in addressing the legislature’s goals. “If, all of a sudden, someone determines that we should take a sharp right or left turn, that doesn’t happen without some challenges,” Wilcox says. “That would be pretty devastating because, remember, we have an obligation to serve students on an ongoing basis – and to see budgets collapse overnight would not be a healthy way of [doing so].”

Trends in university resources

Even if that degree of crisis can be avoided, there are other questions tied to performance-based budgets, such as how to improve a university while stuck on the wrong side of the funding formula, and how to avoid various temptations to exclude challenging students or weaken academic standards.

These are very real questions for Robert Gregerson, who moved this summer from the relatively well-funded state of Florida to a far more challenging budgetary situation in Pennsylvania, which has seen some of the nation’s deepest post-recession cuts to higher education.

As the new president of the University of Pittsburgh at Greensburg, Gregerson has one helpful advantage: the University of Pittsburgh has been around for more than 200 years, mostly as a private institution, and has accumulated a $4 billion endowment. With that resource, the five-campus system is pledging, from this autumn, to match any Pell Grant award any of its students receive. That means an average annual gift of about $4,000 per student, which adds a little more than $1 million a year to the $17 million budget of the Greensburg campus.

Top 20: engagement

The WSJ/THE data show Pitt-Greensburg struggling in measures of both its proportion of Pell Grant recipients and its graduation rates, and Gregerson is grateful for the help the institution’s endowment affords while waiting for state funding to recover.

He is not sure whether the extra money might be better spent as part of a sophisticated data-driven system – pioneered at places such as Georgia State University and the University of Maryland, Baltimore County – to help students finish their degrees. That route appears effective but costly, he says, given the need for both the computer tools and the staff: “When your software says, ‘Here are 100 students of concern’, you need to have the person power to reach out and have an hour-long conversation to plot out their future,” he notes. So, for now, Gregerson is eager to use Pitt-Greensburg’s endowment to encourage more low-income students to enrol at the institution in the first place.

For his part, he is sceptical about performance-based funding systems, which Pennsylvania has used in the past and may soon accelerate. One of his concerns matches that cited by Wilcox in Florida: the harsh and sudden implications for all students at a university if the institution unexpectedly falls behind the statewide pace in a key metric.

“What if you promise 10 per cent and you deliver 5 per cent” on a key measure, Gregerson wonders. “Do [the politicians] claw back half of the money they gave you? What sort of arrangement is there?”

Another major concern, he says, is the likelihood of faculty feeling that they must lower their academic standards if their institution’s survival appears to be at stake. “If you cut a deal that involves millions of dollars, it seems to me that there could be pressure to do that,” Gregerson says.

Top 20: outcomes

However, Wilcox and Minor say they aren’t feeling that in Florida or California. A main strategy of the University of South Florida involves keeping students busy and involved, and therefore less likely to walk away before graduating. That means steps to boost student participation in professional internships, study abroad activities and undergraduate-level research.

“The research shows in a pretty compelling fashion that the more engaged students are in the intellectual life of the university, [the more they] will realise greater measures of success,” Wilcox says. Some of that costs money, he acknowledges. But internships can be funded by companies, and undergraduate research can just mean siphoning off portions of grant awards to bring more students into science labs.

That is an example of a university using one of its strengths: Southern Florida attracts about $550 million a year in research grants, ranking it in the top 25 nationally among public institutions.

The results include a four-year graduation rate of 62 per cent and a six-year graduation rate of 75 per cent, with only about 52 per cent of those diplomas accompanied by any debt, Wilcox says. Of those, the average debt is $21,000: well below state and national averages.

Trends in student perceptions

Back in California, CSU, with 23 campuses spread across 800 miles, has come up with a list of tools for producing the graduation rate gains promised to lawmakers and reflected in the WSJ/THE data. One of the most foundational, Minor says, involves allowing local campus leaders wide latitude to set their own courses, while providing them with details of how well other campuses are doing.

Another involves the data-intensive methods of tracking individual students seen in Georgia and Baltimore, with an attitude of offering more resources before imposing the possible penalty of taking them away. That, Minor says, means looking for courses with unusually high failure rates and pointing out that fact to the campuses and their faculty.

“We’ll say, ‘Listen, you’ve got three semesters to figure it out: let us know what kind of help you need,’” Minor says. “That has had a dramatic effect in terms of students’ time to degree.”

View the Wall Street Journal/Times Higher Education College Rankings 2020 methodology

This approach has helped four or five CSU campuses to record graduation rates better than those of Georgia State at half the price: an in-state tuition rate of $5,742.

But some of CSU’s initiatives have been more controversial, Minor concedes. These include eliminating all stand-alone remedial courses this past year and integrating extra help into its existing course structures instead. Initial numbers show strong gains in the numbers of CSU students passing mathematics courses, although some faculty remain sceptical, fearing longer-term discouragement of students or lowering of standards.

Changes to faculty attitudes are also part of CSU’s new approach. One basic need, Minor says, involves getting instructors more invested in student success. An especially long-standing detrimental practice involves faculty scheduling courses for times that are convenient to them rather than their students. Modern communications technology allows institutions to understand what class times work best for the students, and to put a greater priority on that, Minor says.

More broadly, he says, CSU is hunting for any obstacles that could be pushing students off the track towards graduation. That means pulling together administrators and staff throughout the system to identify possibilities.

One discovery, Minor says, concerned the practice of disenrolling students over unpaid bills, which sometimes amount to just a few dollars. The penalty may have made sense to the chief financial officer, he says, but the real-world result was that students were losing their entire class schedules, throwing their timelines off by months or more.

The difficulty of paying bills is part of the overall financial struggle that many students face, Minor says, and CSU is paying much more attention to the trade-offs. In some cases, such as loans and housing, this can just mean connecting students to services that already exist in their local community.

The problems faced by poorer students could increase the incentives, in a performance-based funding regime, for CSU campuses to increase their graduation rates by reducing their intake of needier students. CSU plans to closely monitor that, Minor says, by watching the demographic mix of graduates, especially in fields such as the sciences.

Top 20: environment

One institutional strategy, in fact, could be very helpful to lower-income students: accepting more transfers. Too many leading US universities refuse to take undergraduates moving from other institutions, Ariana González Stokas, vice-president of diversity, equity and inclusion at Barnard College, told THE’s US Student Success Forum in New York.

Yet an analysis of transfer acceptance rates shows that several top performers on graduation rates in the WSJ/THE US College Rankings 2020 – including the University of South Florida and several CSU campuses – are also among the nation’s leaders in accepting students who change institutions.

As universities struggle to predict which freshmen will perform well, studies have shown that transfers give colleges a real-world opportunity to find students who already have demonstrated they can work at the college level.

In that regard, González Stokas said, the potential for relationships with community colleges is too often overlooked by leading four-year institutions. Community colleges are “where you find the highest number of low-income students of colour,” she said. Helping them can “create incredible social mobility”.

Colleges, however, also face another potential negative incentive: to weaken academic standards to increase graduation rates. However, Minor doesn’t feel much need to worry on this score: “I trust the faculty 100 per cent to maintain quality and rigour, if not increase it, as we go along,” he says.

Whether his confidence is well founded remains to be seen. But with performance-based funding on the rise, it is no doubt an issue that will be monitored very closely by employers and commentators, to ensure that good teaching and pastoral care – as opposed to skilful game-playing – are truly being rewarded.