Thanks to Nate Johnson for a fantastic new chart that clearly illustrates who's to blame for the incredible growth in Pell Grant expenditures: private for-profit higher education. Switch the settings from "all institutions" to "for profits" and watch the red line (for-profits) pull dramatically away from the grey (national average). Now Nate, please do this for federal student loans!
Saturday, April 28, 2012
Monday, April 23, 2012
Elites to 99%: Resistance is Futile
A few highlights:
- The co-founder and former chief executive officer of CarMax told a crowd attending the Association of Governing Boards of Colleges and Universities 2012 National Conference on Trusteeship that public universities should strive for major tuition increases. Reports the Chronicle of Higher Education, "Poor kids borrow money so that the rich kids can get a tuition discount," said Mr. Auston Ligon, now a member of the Board of Visitors at St. John's College in Annapolis, Md. "Quit subsidizing people like my kids."
- Gordon Gee of The Ohio State (and buddy of Biddy Martin) is promoting a forthcoming book from Stanford University Press called "Public No More." This little ditty plays a familiar tune, sung by two business school types. Again we are told, the current business model of higher education is broken (duh) and public higher ed's "longstanding dependence on state subsidies...is unsustainable...recent cuts are permanent...public universities either recognize this...or face decline....attempts to block competitive forces by resistance and delaying actions are self-defeating." Apparently these dudes never heard of the need to present and evaluate without pre-judgement alternative models in policy prescriptions.
- According to Inside Higher Ed, some educators are full-on gung-ho about privatization and not even experiencing "angst" about it (sidenote to IHE--nice framing, making having reservations sound like neuroses). The chancellor of Maricopa Community College, a man in charge of guiding the futures of thousands of black and brown students, apparently has an oracle. Rufus Glasper tells us "We have no choice. The state funds are gone forever." There's no point in anything but his kind of "realism," and his so-called solution is a private for-profit model.
Second, when did smart people all start singing in unison about simplistic, singular solutions to complex problems? Did they all attend a special dinner party together where primers were distributed, and the private monetary incentives for making the education "public no more" were explained? Sure seems like it. Because they are talking to highly educated people in a way that is utterly pedantic-- there is one solution and one solution only -- pass the buck onto the "consumer"? Can you imagine if instead they said, "Hey 5th graders, pay your own way through elementary school?"
Third, how much longer are you people (yes you, our readers) going to take this? For-profit leaders clearly worked this out quite well ages ago, using their massive profits paid for with your federal tax dollars to lobby legislators and university leaders into believing the future lies in private, for-profit education. They're doing it from up high in the skyscrapers around the world, while many higher ed leaders are out there wittingly and unwittingly carrying their water and doing their bidding. We mere "academics" and "students" who won't admit that really we are "obstacles" and "consumers" are simply in the way.
PUBLIC NO MORE. WE HAVE NO CHOICE. RESISTANCE IS FUTILE.
Where have we heard that before?
Friday, April 20, 2012
On social media
Colleagues at the American Educational Research Association's annual meeting asked me to speak informally at a Sunday morning workshop on the topic of social media. I covered a range of topics, including what it's like to write for the Education Optimists. In case you're wondering what it's felt like "behind the scenes" here are the videos.
Thursday, April 19, 2012
Stop Subsidizing the Upper Middle-Class
Today Stephen Burd from Education Sector released a provocative new report that fully supports my contention (and that many others including Sandy Baum, Mike McPherson, Rick Kahlenberg) that we should stop subsidizing the upper middle-class with tax credits for college, and start focusing federal financial aid on those who need it most: Pell recipients.
Every time I've publicly discussed this idea I've been attacked as not caring about the middle-class. This is a red herring-- suggesting that scarce dollars should be targeted to those who most need and will most benefit from them is simply good policy making. It's not about "who cares about whom." As I pointed out following Obama's latest speech in Michigan, tax credits are demonstrably ineffective at their goals. Burd calls a spade a spade when he adds, "Notably, while policymakers continue to tout the tuition tax breaks as a middle-class benefit, the introduction of the AOTC led to significant reductions in the share of the overall benefits going to families making between $25,000 and $75,000."
As a result, of the $55 billion distributed in college tax credits between 2010-2014, most will go to families earning over $100,000. Tax credits don't make or break their children's decisions about attending or college, and are unlikely to even affect where they attend or how long they take to finish. Instead they operate as a sort of "reward" to the family for having a college-bound child, and a little "apology" for the high costs. Of course these are nice things for the government to do for families, but since they don't change student outcomes, they simply aren't necessary. Well, mostly. The one caveat is that they may incur some political support for aid programs generally, a benefit that accrues to all recipients. But that's very hard to demonstrate, and probably isn't worth their high cost.
Let's hope that Congress is listening, and stops attacking the Pell program as inflated and unbearable. What's clearly not needed are these tax credits. Enough already.
Every time I've publicly discussed this idea I've been attacked as not caring about the middle-class. This is a red herring-- suggesting that scarce dollars should be targeted to those who most need and will most benefit from them is simply good policy making. It's not about "who cares about whom." As I pointed out following Obama's latest speech in Michigan, tax credits are demonstrably ineffective at their goals. Burd calls a spade a spade when he adds, "Notably, while policymakers continue to tout the tuition tax breaks as a middle-class benefit, the introduction of the AOTC led to significant reductions in the share of the overall benefits going to families making between $25,000 and $75,000."
As a result, of the $55 billion distributed in college tax credits between 2010-2014, most will go to families earning over $100,000. Tax credits don't make or break their children's decisions about attending or college, and are unlikely to even affect where they attend or how long they take to finish. Instead they operate as a sort of "reward" to the family for having a college-bound child, and a little "apology" for the high costs. Of course these are nice things for the government to do for families, but since they don't change student outcomes, they simply aren't necessary. Well, mostly. The one caveat is that they may incur some political support for aid programs generally, a benefit that accrues to all recipients. But that's very hard to demonstrate, and probably isn't worth their high cost.
Let's hope that Congress is listening, and stops attacking the Pell program as inflated and unbearable. What's clearly not needed are these tax credits. Enough already.
Sunday, April 15, 2012
Derek Bok & the Path to Changing Faculty Teaching Practices
Last night Liam and I attended a talk by Derek Bok, Harvard's president emeritus, hosted by the Spencer Foundation at the meetings of the American Educational Research Association in Vancouver. Due to a lack of Wifi and data service, I couldn't tweet the speech, which was probably good because we both got a little worked up. Here's a bit about why.
Bok is a thoughtful, experienced leader in higher education and I have long appreciated his efforts to get colleges and universities to pay attention to undergraduate education. He's written a book on the topic, and found a set of Bok Centers on many campuses to try and get faculty involved (unfortunately, as he admitted last night, engagement in the centers is often low).
The main thrust of his speech was that professors need to get focused on rigorously improving undergraduate education because policy changes are bringing a reform agenda focused on student outcomes, and we'd best get prepared. We ought to do this, he suggested, by acting as the good researchers we are and attending to and creating new research on what works to improve student learning and graduation rates. We ignore those studies at our peril, he said, instead going about our teaching in un-informed ways -- lecturing, failing to use technology, failing to conduct formative assessments etc-- and it's partly because there's a dearth of good research on quality teaching in undergraduate education. It's time to wake up and embrace our role in the problems we "know" exist-- a lack of learning in higher education, students who don't study, and falling graduation rates.
His contentions were on the one hand laudable -- I'm always a fan of people who push the comfortable elite to wake up-- and on the other hand deeply problematic.
First, Bok spoke about the faculty as if we are a homogeneous bunch. Only once did he mention adjuncts, and it was when he said they were the workforce of for-profits, which are organizations that do pay attention to pedagogy, according to him. So my open question to him, and the first question asked after his talk was "It is increasingly the case that we research types are not 'the faculty' -- the faculty are the enormous number of part-time, contingent, and adjunct workers used by administrations to teach for cheap. What are the implications of your argument for them-- and what are the implications for tenure?" I don't think Bok really understood my question since he respond simply that they 'they' needed to care about good teaching too. (He also made some statements about the potential that the use of adjuncts reduces graduation rates and promotes grade inflation--things that I have commentary on but will take up another day.)
Well, part-time, contingent, and adjunct faculty do care about teaching practices -- and they are arguably more experienced than those of us who teach a few times a year. They also know quite a bit about technology and contemporary teaching practices. But the big difference between "us" and "them" is tenure, status, and pay. They teach very frequently with little job security, no perks like offices to meet with students, and for very little money. They are not segregated to for-profits as Bok suggested, but are employed nationwide in all types of colleges and universities. And they are the workers whom the accountability movement will hit first, hit hardest, and undoubtedly change forever.
When it does, "our" response will have everything to do with tenure. And it will have everything to do with the future of tenure. If those without tenure respond in ways policymakers "like," then you can be sure that tenure will bd deemed the obstacle to student success -- just as it has in k-12 education -- and will be under steady attack. We tenured professors will be pitted against our students in a classic "who cares most about student achievement" false dichotomy, and that is the situation we must prepare for-- and work to avoid. That is what I'd hoped Bok would address.
A few other thoughts. I'm tired of the movement to improve undergraduate outcomes being led by people at institutions where everyone finishes college and money appears to grow on trees. I'm not saying people at those schools don't care about these issues, but most speak in ways that suggest they are out-of-touch with the 99.9% of the rest of us. (There are big exceptions to this rule-- Bridget Terry Long is one.) One could make the case that Harvard got us into this mess -- leading the arms race, raising the costs of attendance like it was going out of style, and setting up an idealized standard in the public imagination that could never be realistically achieved. The more public higher education tries to be like Harvard in any way, the more our doors close rather than open-- leaving the vast majority of students outside in the cold, just waiting to be devoured by the for-profits. Again, I'm so happy people at elite places care about these issues, but I wish that they would (at minimum) partner with people in settings where the real problems actually exist. And I think that wonderful foundations like Spencer should elevate the stature and share the work of people whose research struggles in focused, daily ways with the reality of students dropping out of college and faculty working over-time and under financial constraints to serve them.
I also fervently hope that leaders like Bok will stop repeating shaky empirical research findings that cast undergraduates as fundamentally lazy and underachieving. Throughout his talk, Bok showed a recognition of the importance of rigorous research in establishing cause and effect. Yet he gave great credence to studies of student time use that have enormous problems with measurement error, failed to recognize the role of technology in changing both study and leisure time, and again imposed a homogeneity assumption on undergraduates. Ask yourself, what if undergraduates were mainly a hard-working bunch, with a strong desire to learn -- wouldn't you still want to work harder to teach them well? Why do we feel we must establish a crisis by saying they are unengaged partiers, playing more and doing less?
Finally, I take issue with a point Bok ended with -- the challenge of measuring learning outcomes in higher education. When asked whether he agreed that some goals of higher education are more difficult to measure than others, he responded that that's "mainly because people haven't thought through the issues of measurement enough and aren't clear enough on what those goals entail." While I agree there is too much hand-waving at broad goals, and we often aren't specific enough about what we want students to actually learn, I disagree that everything is quantifiable and readily assessed. College today is a place where life begins to come together for students-- and that happens alongside textbook learning and is a key piece of faculty work. Those successes should be recognized and we deserve credit for them. But they will not be easily measured.
Bok is a thoughtful, experienced leader in higher education and I have long appreciated his efforts to get colleges and universities to pay attention to undergraduate education. He's written a book on the topic, and found a set of Bok Centers on many campuses to try and get faculty involved (unfortunately, as he admitted last night, engagement in the centers is often low).
The main thrust of his speech was that professors need to get focused on rigorously improving undergraduate education because policy changes are bringing a reform agenda focused on student outcomes, and we'd best get prepared. We ought to do this, he suggested, by acting as the good researchers we are and attending to and creating new research on what works to improve student learning and graduation rates. We ignore those studies at our peril, he said, instead going about our teaching in un-informed ways -- lecturing, failing to use technology, failing to conduct formative assessments etc-- and it's partly because there's a dearth of good research on quality teaching in undergraduate education. It's time to wake up and embrace our role in the problems we "know" exist-- a lack of learning in higher education, students who don't study, and falling graduation rates.
His contentions were on the one hand laudable -- I'm always a fan of people who push the comfortable elite to wake up-- and on the other hand deeply problematic.
First, Bok spoke about the faculty as if we are a homogeneous bunch. Only once did he mention adjuncts, and it was when he said they were the workforce of for-profits, which are organizations that do pay attention to pedagogy, according to him. So my open question to him, and the first question asked after his talk was "It is increasingly the case that we research types are not 'the faculty' -- the faculty are the enormous number of part-time, contingent, and adjunct workers used by administrations to teach for cheap. What are the implications of your argument for them-- and what are the implications for tenure?" I don't think Bok really understood my question since he respond simply that they 'they' needed to care about good teaching too. (He also made some statements about the potential that the use of adjuncts reduces graduation rates and promotes grade inflation--things that I have commentary on but will take up another day.)
Well, part-time, contingent, and adjunct faculty do care about teaching practices -- and they are arguably more experienced than those of us who teach a few times a year. They also know quite a bit about technology and contemporary teaching practices. But the big difference between "us" and "them" is tenure, status, and pay. They teach very frequently with little job security, no perks like offices to meet with students, and for very little money. They are not segregated to for-profits as Bok suggested, but are employed nationwide in all types of colleges and universities. And they are the workers whom the accountability movement will hit first, hit hardest, and undoubtedly change forever.
When it does, "our" response will have everything to do with tenure. And it will have everything to do with the future of tenure. If those without tenure respond in ways policymakers "like," then you can be sure that tenure will bd deemed the obstacle to student success -- just as it has in k-12 education -- and will be under steady attack. We tenured professors will be pitted against our students in a classic "who cares most about student achievement" false dichotomy, and that is the situation we must prepare for-- and work to avoid. That is what I'd hoped Bok would address.
A few other thoughts. I'm tired of the movement to improve undergraduate outcomes being led by people at institutions where everyone finishes college and money appears to grow on trees. I'm not saying people at those schools don't care about these issues, but most speak in ways that suggest they are out-of-touch with the 99.9% of the rest of us. (There are big exceptions to this rule-- Bridget Terry Long is one.) One could make the case that Harvard got us into this mess -- leading the arms race, raising the costs of attendance like it was going out of style, and setting up an idealized standard in the public imagination that could never be realistically achieved. The more public higher education tries to be like Harvard in any way, the more our doors close rather than open-- leaving the vast majority of students outside in the cold, just waiting to be devoured by the for-profits. Again, I'm so happy people at elite places care about these issues, but I wish that they would (at minimum) partner with people in settings where the real problems actually exist. And I think that wonderful foundations like Spencer should elevate the stature and share the work of people whose research struggles in focused, daily ways with the reality of students dropping out of college and faculty working over-time and under financial constraints to serve them.
I also fervently hope that leaders like Bok will stop repeating shaky empirical research findings that cast undergraduates as fundamentally lazy and underachieving. Throughout his talk, Bok showed a recognition of the importance of rigorous research in establishing cause and effect. Yet he gave great credence to studies of student time use that have enormous problems with measurement error, failed to recognize the role of technology in changing both study and leisure time, and again imposed a homogeneity assumption on undergraduates. Ask yourself, what if undergraduates were mainly a hard-working bunch, with a strong desire to learn -- wouldn't you still want to work harder to teach them well? Why do we feel we must establish a crisis by saying they are unengaged partiers, playing more and doing less?
Finally, I take issue with a point Bok ended with -- the challenge of measuring learning outcomes in higher education. When asked whether he agreed that some goals of higher education are more difficult to measure than others, he responded that that's "mainly because people haven't thought through the issues of measurement enough and aren't clear enough on what those goals entail." While I agree there is too much hand-waving at broad goals, and we often aren't specific enough about what we want students to actually learn, I disagree that everything is quantifiable and readily assessed. College today is a place where life begins to come together for students-- and that happens alongside textbook learning and is a key piece of faculty work. Those successes should be recognized and we deserve credit for them. But they will not be easily measured.
Thursday, April 12, 2012
Student Responds to UW System Board of Regents Meeting
This week is also another meeting of the UW System Board of Regents. Consistent with yesterday's blog, here I am sharing some thoughts from one of my students who watched the February 9 meeting of that Board. This was the meeting at which members discussed rising costs, cost containment, and the potential for cutting enrollment throughout UW System.
"The debate following the panel of chancellors reflected the three corners of the Iron Triangle: access, funding, and quality. Surprisingly, most of the concerns seemed to revolve around issues of quality, and to some extent access, not around funding concerns...When pressed on the issue, [President Kevin] Reilly was forced to admit that he did not know when or if quality decline would result in reduced enrollment...
Chancellors Sorenson and Wachter noted students' reluctance to leave universities for the workforce. While not using the same intense and accusatory rhetoric of a Jackson Toby, they did claim that students lack efficiency in pursuing their education. At this point [Vice-President] Mark Nook interjected to provide an anecdote about how his daughter ... had managed to graduate in 4 years...This reflected a misguided assumption that his daughter's experience is typical, rather than the reality that 3/4 of today's college students face serious constraints and pressures that could impede their academic progress.
In one particularly poignant moment, Regent Jose Vasquez questioned how the System would provide for students of color and those with disabilities. He was the only member to directly address issues of access for underserved populations. He noted that these students cost more than what he called 'ideal' and 'easy' students, and wondered how they'd be impacted by cost-cutting measures. His remarks highlighted the the non-financial values of public higher education and provided a moment to undercut finance as the "privileged language of reality." Sadly, none of the board members or chancellors responded to his concern."
"The debate following the panel of chancellors reflected the three corners of the Iron Triangle: access, funding, and quality. Surprisingly, most of the concerns seemed to revolve around issues of quality, and to some extent access, not around funding concerns...When pressed on the issue, [President Kevin] Reilly was forced to admit that he did not know when or if quality decline would result in reduced enrollment...
Chancellors Sorenson and Wachter noted students' reluctance to leave universities for the workforce. While not using the same intense and accusatory rhetoric of a Jackson Toby, they did claim that students lack efficiency in pursuing their education. At this point [Vice-President] Mark Nook interjected to provide an anecdote about how his daughter ... had managed to graduate in 4 years...This reflected a misguided assumption that his daughter's experience is typical, rather than the reality that 3/4 of today's college students face serious constraints and pressures that could impede their academic progress.
In one particularly poignant moment, Regent Jose Vasquez questioned how the System would provide for students of color and those with disabilities. He was the only member to directly address issues of access for underserved populations. He noted that these students cost more than what he called 'ideal' and 'easy' students, and wondered how they'd be impacted by cost-cutting measures. His remarks highlighted the the non-financial values of public higher education and provided a moment to undercut finance as the "privileged language of reality." Sadly, none of the board members or chancellors responded to his concern."
Wednesday, April 11, 2012
Students Respond to the UW Taskforce
Today the Wisconsin Legislature's "Special Task Force on UW Restructuring and Operational Flexibilities" meets again in Madison. In honor of that, I want to bring you some student perspectives on one of the prior task force meetings--the one that took place on February 8, and included presentations from the chancellors of UW-Madison and Milwaukee. I'm doing this because student voices are notably absent from these meetings-- students have not been given a chance to present (they will, for the first time, on May 9) and they do not serve on the task force. A few have written letters or spoken publicly on the topic, but most have not.
Recently, students in my Introduction to Debates in Higher Education Policy course (EPS 518) were asked to view a legislative or regents hearing or meeting of their choosing and write a response paper. Below, I provide some representative examples of their responses -- these are deliberately provided without attribution to the student (all are undergraduates) and are posted with their specific permission. My intention is to simply allow the voices of students to emerge, as I think their comments and questions are critical to the discussion. If other students wish to share their considered opinions of hearings, please do send me your memos, and I'm happy to post thoughtful excerpts.
Student 1: "..The nature of the meeting itself...was self-congratulatory and generally insufficient in data. The meeting focused on individual knowledge and individual power, that is they spoke of their personal bailiwicks, which, while it makes sense for a panel of experts, was insufficient...though the panel brought up several reforms, these reforms were often self-serving, under-supported by data, and/or uncertain in their impact."
Student 2: [Flexibilities were a primary topic of discussion at the hearing and yet] "there was an utter lack of understanding about what was being discussed...Despite the apparent knowledge gap about what flexibilities were, they were the main focus of discussion and seemed to be the only thing anyone believed could save the UW System money...What is perplexing about the deregulation rhetoric is that, according to Gary Rhoades, this behavior is...a trickle-down model of funding. In exchange for deregulation and flexibilities, institutions receive less state support. This ends up privileging the elite institutions while creating problems for local institutions. However, it was chancellors of schools like UW-Oshkosh and Platteville who were calling for this deregulation..I cannot help but wonder why the chancellors of these schools would call for deregulation when it would mean less money from the state."
Student 3: "I was surprised at the small number of women on the task force-- just 3. I was disappointed at the lack of minority representation, but not surprised....[many spoke about the word 'product'] and the word 'product' is a difficult one, and its use underscores the different positions and value systems of the task force members. [Most] seemed to think that having a better education and a lower price were mutually exclusive things, and that one must be sacrificed for the other."
Student 4: "As a student, a major concern became evident at this meeting. Members of this task force have been charged with creating innovative solutions to the challenges facing the UW System, challenges that have arisen from a lack of funding. The majority of task force members, however, are not even close to specialists in higher education, let alone public higher education. In fact these people who are supposed to be coming up with solutions are primarily business people who have spent most of their professional careers in the private sector. [Thus] it is clear these are powerful voices denouncing the importance of public funding for various reasons."
Recently, students in my Introduction to Debates in Higher Education Policy course (EPS 518) were asked to view a legislative or regents hearing or meeting of their choosing and write a response paper. Below, I provide some representative examples of their responses -- these are deliberately provided without attribution to the student (all are undergraduates) and are posted with their specific permission. My intention is to simply allow the voices of students to emerge, as I think their comments and questions are critical to the discussion. If other students wish to share their considered opinions of hearings, please do send me your memos, and I'm happy to post thoughtful excerpts.
Student 1: "..The nature of the meeting itself...was self-congratulatory and generally insufficient in data. The meeting focused on individual knowledge and individual power, that is they spoke of their personal bailiwicks, which, while it makes sense for a panel of experts, was insufficient...though the panel brought up several reforms, these reforms were often self-serving, under-supported by data, and/or uncertain in their impact."
Student 2: [Flexibilities were a primary topic of discussion at the hearing and yet] "there was an utter lack of understanding about what was being discussed...Despite the apparent knowledge gap about what flexibilities were, they were the main focus of discussion and seemed to be the only thing anyone believed could save the UW System money...What is perplexing about the deregulation rhetoric is that, according to Gary Rhoades, this behavior is...a trickle-down model of funding. In exchange for deregulation and flexibilities, institutions receive less state support. This ends up privileging the elite institutions while creating problems for local institutions. However, it was chancellors of schools like UW-Oshkosh and Platteville who were calling for this deregulation..I cannot help but wonder why the chancellors of these schools would call for deregulation when it would mean less money from the state."
Student 3: "I was surprised at the small number of women on the task force-- just 3. I was disappointed at the lack of minority representation, but not surprised....[many spoke about the word 'product'] and the word 'product' is a difficult one, and its use underscores the different positions and value systems of the task force members. [Most] seemed to think that having a better education and a lower price were mutually exclusive things, and that one must be sacrificed for the other."
Student 4: "As a student, a major concern became evident at this meeting. Members of this task force have been charged with creating innovative solutions to the challenges facing the UW System, challenges that have arisen from a lack of funding. The majority of task force members, however, are not even close to specialists in higher education, let alone public higher education. In fact these people who are supposed to be coming up with solutions are primarily business people who have spent most of their professional careers in the private sector. [Thus] it is clear these are powerful voices denouncing the importance of public funding for various reasons."
Monday, April 9, 2012
The Value of Looking at Local Results
The report we released today has an interesting history that shows the value of looking beyond the initial results of an experiment. Later this week, we are presenting a paper at AERA entitled "In School Settings, Are All RCTs Exploratory?" The findings we report from our experiment with an iPad application were part of the inspiration for this. If Riverside Unified had not looked at its own data, we would not, in the normal course of data analysis, have broken the results out by individual districts, and our conclusion would have been that there was no discernible impact of the app. We can cite many other cases where looking at subgroups leads us to conclusions different from the conclusion based on the result averaged across the whole sample. Our report on AMSTI is another case we will cite in our AERA paper.
We agree with the Institute of Education Sciences (IES) in taking a disciplined approach in requiring that researchers "call their shots" by naming the small number of outcomes considered most important in any experiment. All other questions are fine to look at but fall into the category of exploratory work. What we want to guard against, however, is the implication that answers to primary questions, which often are concerned with average impacts for the study sample as a whole, must apply to various subgroups within the sample, and therefore can be broadly generalized by practitioners, developers, and policy makers.
If we find an average impact but in exploratory analysis discover plausible, policy-relevant, and statistically strong differential effects for subgroups, then some doubt about completeness may be cast on the value of the confirmatory finding. We may not be certain of a moderator effect--for example--but once it comes to light, the value of the average impact can also be considered incomplete or misleading for practical purposes. If it is necessary to conduct an additional experiment to verify a differential subgroup impact, the same experiment may verify that the average impact is not what practitioners, developers, and policy makers should be concerned with.
In our paper at AERA, we are proposing that any result from a school-based experiment should be treated as provisional by practitioners, developers, and policy-makers. The results of RCTs can be very useful, but the challenges of generalizability of the results from even the most stringently designed experiment mean that the results should be considered the basis for a hypothesis that the intervention may work under similar conditions. For a developer considering how to improve an intervention, the specific conditions under which it appeared to work or not work is the critical information to have. For a school system decision maker, the most useful pieces of information are insight into subpopulations that appear to benefit and conditions that are favorable for implementation. For those concerned with educational policy, it is often the case that conditions and interventions change and develop more rapidly than research studies can be conducted. Using available evidence may mean digging through studies that have confirmatory results in contexts similar or different from their own and examining exploratory analyses that provide useful hints as to the most productive steps to take next. The practitioner in this case is in a similar position to the researcher considering the design of the next experiment. The practitioner also has to come to a hypothesis about how things work as the basis for action.
-- DN & AJ
We agree with the Institute of Education Sciences (IES) in taking a disciplined approach in requiring that researchers "call their shots" by naming the small number of outcomes considered most important in any experiment. All other questions are fine to look at but fall into the category of exploratory work. What we want to guard against, however, is the implication that answers to primary questions, which often are concerned with average impacts for the study sample as a whole, must apply to various subgroups within the sample, and therefore can be broadly generalized by practitioners, developers, and policy makers.
If we find an average impact but in exploratory analysis discover plausible, policy-relevant, and statistically strong differential effects for subgroups, then some doubt about completeness may be cast on the value of the confirmatory finding. We may not be certain of a moderator effect--for example--but once it comes to light, the value of the average impact can also be considered incomplete or misleading for practical purposes. If it is necessary to conduct an additional experiment to verify a differential subgroup impact, the same experiment may verify that the average impact is not what practitioners, developers, and policy makers should be concerned with.
In our paper at AERA, we are proposing that any result from a school-based experiment should be treated as provisional by practitioners, developers, and policy-makers. The results of RCTs can be very useful, but the challenges of generalizability of the results from even the most stringently designed experiment mean that the results should be considered the basis for a hypothesis that the intervention may work under similar conditions. For a developer considering how to improve an intervention, the specific conditions under which it appeared to work or not work is the critical information to have. For a school system decision maker, the most useful pieces of information are insight into subpopulations that appear to benefit and conditions that are favorable for implementation. For those concerned with educational policy, it is often the case that conditions and interventions change and develop more rapidly than research studies can be conducted. Using available evidence may mean digging through studies that have confirmatory results in contexts similar or different from their own and examining exploratory analyses that provide useful hints as to the most productive steps to take next. The practitioner in this case is in a similar position to the researcher considering the design of the next experiment. The practitioner also has to come to a hypothesis about how things work as the basis for action.
-- DN & AJ
Subscribe to:
Posts (Atom)