Panel Discussion: Evaluation Policies and Approaches Across Administrations
NANCY KINGSBURY: Well this is going to be something of a personal journey.
I think of the people here I have the longest government service I’m pretty sure…
47 years. If Joe shows up, he gets a little bit more
than me. [Laughter] I started at what is now NBS, the National
Bureau of Standards in 1969
–the early days of the Nixon administration– and I got into an organization
and NBS was a very technical organization but it had a small group that offered up evaluative
services to other government agencies
on memorandive understandings so I did things like look at the use of video
recordings in the courts when the only court that was doing video recording in the country
was in Alaska So it was a good grounding.
I mean, my academic career through social science research being what it is
and I wandered down to the Department of Commerce after that and ended up at the Civil Service
Commission That route was sort of funny, but
nonetheless that’s where I ended up. in what was then the Intergovernmental Personnel
and Grant Program It was a program that
–and now we’re into the Ford administration– It was a program that provided small grants
to state and local governments to build merit-based personnel systems
and somewhere along in there my boss, about 1974 or so,
my boss came to me and said “You know how to do evaluation, don’t you?”
And I said, “Why?” And he said, “Because OMB wants us to evaluate
the grant program.” So I did that.
And it provided enough evidence that the grant program
stayed alive for another 3 years, which I was proud of.
So at the Civil Service Commission under President Carter
there was a lot of interest, not only at the Civil Service Commission,
but I think in general around the government and the aftermath of the growth of the
social programs that were passed in the ’70s to start looking at whether programs are working
or not and there was also interest in the Carter
administration in doing civil service reforms
so I got involved in the civil service reform activities
but coming out of that there was a lot of support to have
an evaluation office at what then became OPM in order to evaluate how well the civil service
reform was going. So again, it was a fairly good environment
for evaluation and I think if we looked around government
at that time there were a fair number of pockets of evaluation
going on. Then of course, in 1980,
well, in the little sojourn at the end of the Carter administration
I ended up spending two years at the Peace Corps as the Senior Administrative Manager
of the Peace Corps where among other things I learned that the
Peace Corps had had an evaluation office from the outset in 1960.
It was small, but it was very active and I thought that was kind of neat.
I was there when Mr. Carter, whose mother had been a Peace Corps volunteer,
decided he wanted to make the Peace Corps independant.
It had been in the Action Agency which you’re familiar with, before that.
He, like many Presidential candidates, had run on
a platform of reducing the number of federal agencies
so he didn’t think he could make it completely independent,
so instead he issued an executive order– Those were always very popular–
to make the Peace Corps an autonomous agency within Action.
Very interesting bureaucratic challenge, but we made it work.
The Peace Corps became completely independent about 3 or 4 months
after I left it in 1981. Then President Reagan was elected.
And I have to tell you, I’m seeing very similar attitudes today
to what I saw then which was, a President that was elected on
ideological grounds and wanted to make decisions on ideological
grounds, and was not particularly grounded in whether
or not the facts supported those decisions. And my impression at that point
–I ended up staying at what was then OPM until
I found my way to GAO in 1984– it was a laissez-faire kind of thing
but evaluation wasn’t considered nominally very useful
for that period of time in the executive branch. Now, fast forward: I end up at GAO.
What has GAO been doing? Towards the middle of the 1970s,
GAO was given additional authority to do economy and efficiency kinds of evaluations
so they stopped hiring accountants and started hiring economists and social scientists and
people of that sort, established an institute for program, well,
a program analysis division, and then an institute for program evaluation,
and then the program evaluation methodology division which some of you are familiar with.
We’ve got a couple graduates of that program in the audience.
And PEMD was a real force in evaluation both methodologically, because we took the
effort to, and this was before my time, took the effort to develop methodology papers,
many of which are still available on our website. But they also did program evaluations.
And pretty good ones. Now at that time in GAO the average engagement
took 2 years, so there was time to do real evaluation work.
That has somewhat changed in recent years. But anyways, so I ended up at GAO in 1984.
Was I doing evaluation? No. I wasn’t in PEMD.
I was hired in part because the leadership of PEMD knew me from the American Evaluation
Association and the Evaluation Research Society. Because I had been a leader in those organizations
for 15 or 16 years. So I got there, and I remember having an interview
with the Comptroller General, Chuck Bowsher, who said, “Have you given any thought to where
you’d like to work in GAO?” And I said, “Well, I’d like to do international
work. Peace Corps was really interesting.
I didn’t want to do civilian personnel work. I hated the guy who was running GAO, I mean
OPM, and he said, Chuck said, “Well, our international
work in the division with our defense work. How do you feel about defense work?”
And I said, “My husband worked at the Pentagon for seven years.
I can probably manage that.” I spent the next 10 years of my life doing
defense work, which is not evaluation.
The first engagement I got involved with I suggested to the fellow running the job
that we might do a sample, and he looked at me and said,
“If you’re going to do a sample, you’re going to have to write the report.”
I buried my PhD, by the way, at that time. [Laughter.]
It did me no good, and unless I got into an argument with the two other people who had them on their
business cards, and wanted to try to use them to throw their
weight around. So I did defense work until up until 1993.
So at this point, we’re at the end of the H. W. Bush administration.
Clinton gets elected, etc. I get picked up and plopped down in civilian
personnel work for a while. But then I started actually doing some work
that some of the people in this room might begin to call evaluation.
We also had begun to be more responsive to the congress,
and more concerned about meeting their needs. So the amount of time we had to do engagements
started shrinking. And so forth.
A couple years later I was made a deputy in the division,
a couple years I was the division director. So I was responsible for a fairly significant
amount of GAO’s work. But when Dave Walker was made Comptroller
General in 1998, during that first year he was there,
he began to notice that we had a lot of technical people.
And congress was beginning to get interested in doing more sophisticated work.
We were getting, we went from having 40% of our work requested to congress to 70%, 80%
now 95%. So we have to sort of work to, as Dave Walker
used to say, “Give them what they want, but give them what
they need.” And giving them what they need means designing
the work in a rigorous, non-partisan, non-ideological, fair, balanced and analytically sophisticated
way. So Dave asked me to figure out where the technical
people in the group, in the organization, were
and stand up a new group. Which I did in October of 2000.
So now, I am responsible for providing technical support to all of the other teams.
And we do, we look at almost everything the federal government does.
But we’re much more oriented to focusing on what are researching questions are,
focusing on how to collect information to answer those questions,
and having those discussions up front. And we’ve just recently redesigned our whole
engagement system so that the folks from my group who we refer
internally as “stakeholders” are at the table early in most of the engagements.
It’s a very satisfying job. Now, to come back to the presidential theme,
Clinton had the national performance review. In 1993, congress passed the Government Performance
and Results Act which required agencies to do strategic plans
for performance plans and to consider evaluations in those plans.
That tended to encourage evaluation. It was the first place that evaluation was actually
talked about in awe that I can remember, that I can remember.
Agencies were sort of good at it. We get to the Bush administration and it gets
ratched up to another notch, which is that they developed the Performance Assessment
Rating Tool and put every program through this exercise.
Three of the questions on the Performance Assessment Rating Tool asked about evaluations,
but at that time there was a mindset at OMB that evaluations a) had to be independent, so if
an agency did an evaluation it didn’t count. If they contracted it out, it didn’t count.
And Stephanie Shipman that’s sitting over there, and I, and some others were involved in actually
trying to train OMB examiners that there are other methods besides randomized control trials.
[laughter] We were modestly successful.
And not altogether. I mean, it was really unfortunate because
there was this emphasis on evaluation and yet a constraint about what kind of evaluation
was useful or could be trusted. So that part of it was uncomfortable.
It was also probably unworkable to try to do it for every program.
If you had a program that was coherent the National Weather Service, something like
that, then you could think about the Program Assessment
Rating Tool but to try to apply a single tool to every,
all of the tactical aircraft programs of the Department of Defense was nuts. I mean,
it was just nuts. It didn’t work.
Okay. When President Obama was elected,
I frankly got very excited. The very first speech that Peter Orszag gave
as Director of OMB talked about evidence-based policy.
And I was there. It was kind of encouraging.
They put… Here’s Joe! Yay!
[laughter, applause] They put it in their budget.
Exciting.They had a whole section of the analytic perspectives in the budget
talking about program evaluation. And in the early years they had a little extra
money thrown in, where agencies could sort of compete, to get
extra money to do evaluations. It’s been very interesting to watch that happen.
Over time, they moved on from “program evaluation” per se,
to using administrative data, other things of that sort.
But it was a very, a pretty positive effort and I was telling somebody earlier
one of the things we do at GAO is, when we kept being asked in the 90s
to do reports on who was doing evaluation and it was very difficult to do.
So the last time we did that, we kept the addresses,
email addresses of people and we created the Federal Evaluators’ Network,
which Stephanie runs and we manage, and the website associated with that,
and when we started it was, Stephanie, what 400 people or so?
And it’s now 2000, something like that? So we’re sort of aware of who’s interested
in evaluation around the government. I’m not really, yet, ready to talk very much
about the current transition. I’m very encouraged the commission was created
and I’m looking forward to what you have to say about that.
I said earlier I’m reminded of the early Reagan years where decisions were being made on a
vast scale without any interest in evaluation at all
as far as I can tell. And budget reductions are being proposed
for non-defense spending, which if they were carried out,
would be catastrophic, I think for evaluation at least in the short term.
I’m also skeptical they’re actually going to be carried out in that way,
so you know it’s really, really, really early right now to tell
where we go from here and I think on that note I’ll stop. NICK HART:
I just want to say thank you all for coming on a Friday night to listen to us talk about
what I think is a really important conversation. So I am in my third administration, [inaudible],
I started in the federal government at OMB during the Bush administration.
I was there through the entirety of the Obama administration,
and am now privileged to work with the Commission on Evidence-Based Policy Making
in the Trump administration. So I want to say at the outset that my remarks
are my own. So I give you my standard disclaimer [Audience laughter]
to give myself a little bit of latitude. So please don’t prejudge that anything I’m
going to say is an affectation of what the commission will recommend later this year.
So I want to be optimistic in this conversation because I think it’s really important
for our field and every way that we think about what we do.
Back on January 20, I wrote a letter to my friends and family on FaceBook,
and I tried my best to take this really optimistic approach.
And I think it’s worth saying that we have a lot of shared values in our country.
And it’s important to not forget that. At the end of the day, we have a common goal
of improving people’s lives, regardless of political philosophies.
I mean, at the end of the day we are trying keep homeless veterans off the streets.
We’re trying to make sure people have enough food to eat.
We’re trying to reduce recidivism. We’re trying to give people better opportunities
in life. And because our policy-makers have that shared
goal, regardless or not of whether you necessarily
agree with that, I think it’s an important point, is that this
is the starting frame for this entire conversation.
So if we have the common goal, I think we can also say that,
“How do you prove, define good policy” is really a moral imperative.
We’re talking about people’s lives, at the end of the day.
So I also want to say that evaluation is inherently political.
And I hope that’s a statement that most of you will agree with.
That means small “p” politics. I mean, at the end of the day, we are passing
some judgement on how well a program is doing,
whether a process is working, how to make it better,
work for who… and these are all important questions.
I think if you look at principles and practices that many of us follow,
this is going to be something that is inherent in a lot of the different angles of those
principles. Now I wrote down this quote:
Pericles said that “Just because you don’t want to take an interest in politics
does not mean politics will not take an interest in you.”
[laughter] And this is where we find ourselves as evaluators,
whether or not we want to operate in this world,
that’s where we are, and so we’ve got to make the best of it.
So that’s a good segueway to say new administrations will always have new priorities.
I think we find ourselves here today, fifty days into a new administration.
Priorities have shifted from the last eight years.
Eight years ago, priorities shifted from the proceeding eight years
and so on and so forth. And I think it’s come up already a little bit,
even in the social hour before the panel. People ask, “Will there be evidence-based
policy making in the new administration?” [some laughter]
And have to say, it’s gonna be a lot of the same.
Remember, we want to believe that things are going to be fundamentally different across
time. But I want to be much more optimistic than
that. I don’t think that you’re going to see these
drastic pendulum shifts to the extent that we are
at least in some corners, maybe, hearing. I think in practice we know that when
budgets have expanded, evaluation has expanded in some circles.
But we also know that when budgets have contracted, there have also been more evaluations.
At the same time, my doctoral research was on evaluation capacity at the Environmental
Protection Agency. During two separate Democratic administrations,
they effectively killed the central evaluation function.
So what does that say? Well what that says is that there’s a lot
of priorities that are always at play with what we do.
A couple years ago, Kathy Newcomer and I started writing a working paper, comparing
across administrations the different performance management and evaluation
agendas. The Bush administration and the Obama administration.
And I think what you may be surprised to know is that there are a striking number of similarities
in actuality across administrations. So what tells you is that it’s not political
philosophy that dictates that we want to make government
work better so that we can serve Americans and improve
people’s lives. It is a fundamental value that our leaders
share. So Nancy talked a little bit about the Bush
administration initiatives. Part of course included questions about evaluation.
And actually I think this is really interesting. So you may not have agreed with the framing
of those particular questions: Should evaluations be independent?
How do you define independent? Should it be a randomized control trial?
Are there other things that we consider? But we know today that there are agencies
that conducted evaluation because that question
was on the PART. So it served as a really valuable motivator
for conducting evaluation and I think we can criticize exactly how the
questions were, what the rating system of PART looked like,
but it actually sowed a very very valuable function within the field.
And that was in the Bush administration. Fast-forward to the Obama administration.
I think you gave a nice little overview of some of the pieces that have intersected with
CNCS’s work, and you gave a nice overview of a handful
of the other pieces. I would describe the Obama initiatives as
1,000 flowers blooming. There were so many initiatives in the Obama
administration it was really impossible to keep track of
them all: Pay for Success, Social Impact Bond, funding
mechanisms, tiered access… I mean, everywhere you turned there was something
going on at least in conversation.
And at the end of the day, it’s a question of how much of it actually got implemented.
I think there was a lot of conversation and there’s a gap between the conversation and
what actually happened. But the point being, there’s very clear
attention on this topic, still today, so I’ve been working for the
Commission for Evidence-Based Policy Making. It’s a bi-partisan commission,
created under the Obama administration, but the commission always knew that its final
report was going to get submitted to a different
president. We didn’t know what the political affiliation
would be, but we knew it was going to be a different
person. And what that tells you is that, in congress,
there is an interest in this topic. So it’s not going away.
That’s my optimism again. [Laughter]
So when Kathy Newcomer and I looked across all these initiatives in the Obama and the
Bush administrations, we looked at them very exhaustively,
and identified three really fundamental challenges that are going to exist,
and do exist, as we think about these. One: balancing the purposes of our evidence-building
activities. So classically in the evaluation field we
say that there are two predominant purposes: accountability
and learning. And very rarely will you find a case
where something can accomplish both purposes for every audience that they intended to.
In fact, I would say never, because it’s impossible. So even these best initiatives that we can
come up with I mean, think about priority goals in the
Obama administration. At the end of the day,
it’s always going to have to face an issue of mandates.
And there are real challenges when we mandate that agencies do something.
So I know some of us would say, “We should mandate that every federal agency conduct
an evaluation for every program, every year.” And I would say, “Whoah! Ok, hang on, let’s
maybe take that down a notch.” [laughter]
I’ve heard individuals say, “There should be 1% federal funding set aside for every
program across the federal government.” I say, “Wow. The Social Security Administration
spends a trillion dollars a year.” [chuckles]
“Do you really want 1% of a trillion dollars for evaluators,
as you say, or are there better uses for the money?”
So I think the challenge here is that we have to think a little about what the purpose that
we’re actually interested in is. The second challenge was acquiring and sustaining
audiences. And this has already come up a couple times,
and this is not going to be news to most of you.
We’ve seen plenty of cases where there are major proponents or champions of evaluation
initiatives or performance initiatives across the government.
And when that champion leaves, sometimes that initiative falls to the wayside.
And our challenge as evaluators is making sure there is a sustained interest in the
field as we go forward.
And it doesn’t matter whether we’re talking about performance management,
performance measurement (which has a important role in this conversation),
or the production of individual analysis that are evaluations.
The third thing, the third challenge is effective implementation of initiatives.
And this sounds so silly that, it’s… it’s not profound by any means.
But it’s one of these things that we often forget.
We’ve seen a lot of initiatives that even when they are well-designed,
they’re based on theory– they skip important steps
in thinking about how you role it out, how you scale it up,
and that has huge implications for how we conduct ourselves
in doing evaluations. I wasn’t actually planning to talk about administrative
data, so if there were things that you wanted me
to talk about, ask questions.
[laughter] But to the extent that we want to use administrative
data, in evaluations, we have to think about
what is an appropriate scaling for how that happens.
Do we say today, “Is it appropriate for every snap agency across the country
to supply data for a national-scale evaluation?” Maybe, but can that happen tomorrow?
That’s going to be really, really hard. And so phasing of initiatives is really important.
Our most important observation in thinking about
implementation of initiatives was really around cross-agency collaboration,
though. I like to say cross-agency collaboration has
long been an oxymoron in government. And it is something that is sorely needed.
And it’s true for something as simple as establishing a priority goal
and something as complicated as conducting an evaluation of a range of human services
programs. Today we know you can be a WIC beneficiary
and you can receive other human service benefits and we may not even know that.
So in conducting an evaluation, often we’re going to look at one silo.
But we’re interested in all the benefits that you’re receiving.
And so as a government, we have to think about how to break down those silos.
No administration has perfected that. There are a lot of challenges in doing just
that. Alright, so what is to be done as we move
forward? This is my really optimistic part.
One: stay calm. [laughter]
So, transitions are a routine part pf our democracy.
And when new political appointees are coming into office,
they have a lot to learn. Many of them have not been in government before.
They’ve been in different roles in government. So evaluation may not be the first thing they
walk in the door and want to talk about. And they have a lot of learning to do.
But when they leave? They want to talk about it
because they want to say, “I accomplished X, Y, and Z.”
And so, I think that there is a real opportunity for this conversation
around what the role of evaluation will be in any administration going forward.
But it takes time. Number two: We all have to take some effort
to explain the value of evaluation, and that includes to policy makers.
I think one important way to think about this is that
you alluded to the Fed Eval Network. We have a lot of networks within government.
We have a lot of networks outside of government. This event is co-sponsored by Washington Evaluators,
which is one of them. And we have to think about how to use our
networks to make sure that we’re talking both intelligently
and insightfully about the value evaluation can actually have for making policy decisions,
and informing how those decisions are framed and executed.
Number three: We have to offer to help inform decision makers
how evaluation can actually be useful. I would say, as a field, we’re actually really
bad at this. I think in a lot of cases we’re happy to hand
over a report and say, “Alright, I’m done.”
And that’s not really the end game. If our goal is for evaluations to actually
be used, the usefulness has to be evident.
And sometimes that requires a little bit of explanation.
So one of the challenges for us is to figure out how we explain
the opportunities and think about how we can educate
new political appointees, senior managers in government,
even outside of government, CEOs of non-profits,
what the role of evaluation can appropriately be
in accomplishing our shared goals. Number four: we have to find opportunities
to do better analyses and I know as I worked at OMB, previously
at OMB, you’re probably expecting me to say that RCTs
are the only way you can do that.
But that is not my message, my message at all.
Of course my PhD work will validate this, that a range of methods is always necessary
and it doesn’t matter if we’re talking about environmental science,
or human services evaluation. So I want to concede that up front.
What I’m really talking about though is thinking about strategies to actually make our
evaluations more useful. Back in January, the American Evaluation Association
hosted a dialogue on race and class. And there will be several more of these dialogues
as the year goes forward. It’s a priority for AEA’s President, Kathryn
Newcomer. And one of the points I made in that panel
back in January is that thinking about disparities in our
programs is often something that gets overlooked.
And this would be sub-group analysis. That seems really obvious if you’re thinking
about classic analytical techniques
and what you might be interested in but it’s not always the first question somebody’s
going to put on the agenda for what they want to know the answer to.
And so that’s incumbent on all of us to say, “If it’s relevant, we should make
sure that it’s part of the conversation.” The last, well, two points.
I added one while Mary was talking. [Laughter]
Number five: We have to recognize that evaluation has multiple purposes.
And this gets also into the range of methods issue that I raised.
I’m not talking about just informing budget decisions,
though in many cases we hope that evaluations are useful in those conversations
because if they’re not, we’re missing an important opportunity.
And it may be that we’re presenting some evidence that says,
“This program actually needs more money because it’s working really well, and it can
work better, and we can learn more.”
So I think we have to think intelligently about
how that framing occurs. And the point that I added, which was my only
reference to the Commission on the paper, is that…
So the Commission on Evidence-Based Policy is having a meeting on Monday,
and I encourage you all to attend. [Laughter] MARY HYDE: I started working as…
With the Trump administration, I have now worked as an evaluator
in four administrations. I started working as a professional evaluator
under the Clinton administration and your point about the sort of the GPRA
being sort of the focus then I wasn’t working as a federal employee then
I was working as a post-doc, actually, and what was exciting coming right out of
graduate school at that time was the big social policy research that was
going on in addition to the performance focus
or new focus, new iteration of that focus. There was this really exciting wealth of reform
policy work going on. To your point about where there were coherent
policies and programs there was some really interesting work going
on. So that was sort of my first foray into at least federally-supported evaluation. I also, in another professional life, worked
with an agency who was trying to implement PART
under the Bush administration, and again to your point it was very challenging
because most of these programs were not this sort of coherent,
“This is what we do, and therefore there’s an easy way to ask a question,
and identify the change and measure that.” So that was all very challenging.
I came to the Corporation for National and Community Service
in President Obama’s second term and I was also very excited
to work in that evidence movement. But I will say, the only thing I can add
is my personal experience, working in the evaluation office at CNCS.
I can give you some examples of how, I think, every point you’ve made has played out
in this one particular agency, and as someone who has been responsible for
an agency’s’ evaluation agenda the challenges that you’ve laid out
resonate very much, so. I came in to the agency’s evaluation office
towards the end of 2012 and at that time
the way that the agency was talking about evaluation was largely around statistical data. There was a CPS supplement that we implemented around volunteering data, and civic engagement data,
and the agency was saying “These actual statistics prove that we’re
effective.” And so, we were like, “Wait a minute, what?”
[Laughter] How did we make that leap from,
you know, “Here’s some nice national data” to
“See? We work!” So that’s kind of where we started.
We were also, the agency had also invested quite a lot of resources into a longitudinal
study of Americorps members. So they had followed a cohort of members for I think eight or nine years.
It was a very expensive survey. The attrition rate, as you might imagine,
was getting pretty high towards the end of it. And even then, in just 2012, the last cohort had been surveyed in 2009.
So in addition to this statistical data that was loosely, loosely, loosely affiliated
with our programs, we had this member data that was kind of outdated, and at the time OMB was really doubling down on this idea that, “Outcome data over time, that’s not evidence.” I think my main takeaway over working across
different administrations is that the biggest differences
are the way in which people define evaluation and what people mean by evaluation,
and what is determined to be credible evidence. And that has varied.
It’s a variation on a theme, but it has varied in kind of wild swings, in some ways.
So fast-forward a year or two into my tenure at CNCS
and what I had started to realize also was the agency had invested lots of resources
and emotional energy in performance measures in reaction to, in some ways, PART, but also in sort of this of this effort to get on board with “Outcomes are the new results.
This is what we need. We need outcomes, we need outcomes, we need
outcomes. And if we ask all of our grantees to give
outcome data, then we can talk about the impact that we’re
having.” And at about that time, literally we were
getting guidance that said, “No, you need to start looking
at some experimental data. Performance measures are not enough.”
So you have this mandate which is very exciting as an evaluator
to start talking about different types of designs
and different options for getting at different policy and programmatic questions,
and an agency in a field of funded grantees who
had barely just gotten on board with this idea of moving from outputs to outcomes
and now you’re raising the bar and saying, “And now we’d like you to do some experimental
design.” And you know, you could kind of get attacked
in the hallways. [Laughter]
“We’re not happy!” [Laughter]
You’d have these convenings, and people were very hostile and angry,
and very upset, and understandably so in many ways because you had gotten them all, you know, rallied around this idea that,
“Ok, we just need to link some outputs to outcomes and that’s the right path,” and then we’ve added this element of design
which obviously comes with a whole different set
of expertise and understandings, and people just felt like the rug had been pulled out from underneath them. So you know, that was sort of my little journey inside of four years
from working with an agency and their grantees around these very different definitions of
evidence and credible data.
So in terms of, I mean I think we’ve had some successes in terms of shifting the mindset
and the culture a bit but you know, there is still this expectation,
and who knows in the current administration what will happen, but,
there’s this expectation that the ultimate use for
this experimental data, and quite frankly that’s the fear around this more experimental data, is to give you a thumbs up or a thumbs down.
So in the process of actually trying to help people understand that
performance measures can be a part of an evaluation strategy,
part of a more rigorous design, we’ve tried to encourage different ways of
using data, not only with our grantees but within the
agency. So we talk about the importance of organizational
learning, the importance of program improvement,
the importance of using data of all kinds to inform critical decisions.
So I would say one of our biggest successes at the agency
has been shifting funding decisions, at least based in part on the level of evidence
that people are either striving for, or come in as as applicants.
So we’ve really shifted our grant-making practices. So I feel like in many ways, that’s in theory
a more sustainable use and change in norms than getting people all rallied around
a big thumbs up or thumbs down depending on whether a particular finding
is. So that’s just my one very narrow experience at one federal agency. I would say, in terms of pros and cons with
that, I think I’ve identified the cons, that it
makes people really upset. Some trust is compromised because I think we all know that at the end of the day as program evaluators, relationships and trust are essential.
If you’re going to work with your program colleagues
if you’re going to work with a field of grantees, if they don’t trust what you’re saying or
trust that you’re sharing the same goal of delivering good services
and making good change in communities, you’ve kind of lost the game.
I would say one other con is at least for our agency
a vast amount of money was spent on data systems, on reporting systems,
on training, on all sorts of things around this performance measure movement.
And suddenly, we’re like, “Nope, nevermind! That’s not as important. Now we need you to
do this.” So just from a resource perspective,
that can be a big con, when you make these shifts in emphasis over
time across administrations.
The upside, and I feel like there would be at least a
substantial percentage of program staff at the agency as well as in the field
that say, “No matter what the pendulum is doing, whether it’s PART, whether it’s GPRA, whether it’s experimental design, whether it’s an ROI, whatever the fad is at the moment,
at the end of the day it really does get people thinking
a little bit more evaluatively. They think a little bit more around data.
They think about what good data is, whether you’re talking about an output and
outcome or a design.
They start to understand there is a relationship between research design and just data.
So I feel like that’s a pro. No matter where the winds may sail next,
people at least have built some capacity in their organizations to have good data habits,
basically. No matter what kind of data they’re talking
about, and to think about how that can help them
improve their organization as well as their services.
So I think that capacity is built across all of these efforts,
no matter what the focus is of an administration. Like my distinguished panelist, I can’t even
begin to speculate what this new administration is gonna do.
I have no idea. I don’t even want to pretend to think that
there’s a specific evidence or evaluation agenda with this administration.
On my more optimistic days, I feel like, well, that means that it’s not going to be
a big priority, so we can just sort of do what we need to
do and carry on business as usual.
That’s what I’m going to keep telling myself. [laughter]
And if that’s the case, I think that, you know, my hope is that we will continue to use evaluation in all of its different iterations and forms
as a tool for organizational learning, as a tool for improvement,
and as a more and more institutionalized source of information
for any decision making. I think it’s naive to think it’s ever going
to be the only piece of information that people use: ideology, politics, practicality, all sorts of things come to the table
when you’re making big decisions but the more that this is at least at the
table, as one consideration,
I think would be a win. And I think as long as there is some adherence
and expectation of evaluation no matter how it’s defined,
that will continue to happen. I would say also that if things continue to
at least be able to move along
in an evaluation and evidence track, that we at least start to think as a group,
whether you’re a federal evaluator or not, that the more we can stick to our fundamentals, I think the easier it’s going to be for non-evaluators to come along this journey
with us. So sticking to fundamentals like,
What is the question you’re trying to answer? Let’s start with that.
Let’s start thinking about how we raise good questions. And then we can talk about what the best strategies might be for addressing
them. And it’s going to require a holistic approach.
As an evaluation office, we can’t just say, “We’re going to do this, and this.”
It has to be a little bit more nimble and flexible and a little bit richer.
I would hope that we have space as we move forward
to think about rigor and robustness around critical thinking and quality of studies
and not equating it with one particular design. That I think would be some progress,
if we could do that. And then of course I think you’re never going
to get around –this was sort of my earlier point–
the importance of quality data the importance of good measurement
and the importance of good infrastructure to look at data, because I do think that if I were to place a bet on where this administration’s going to go, we’re going to see the pendulum swing back to administrative data
which obviously Nick is going to talk to us about given the focus of the Commission. But I do think that’s probably likely.
If we can start using these large programmatic and administrative databases
that the federal government spends a lot of money on,
quite frankly, on gathering, whether it’s survey data or just programmatic
data, and maximizing those like Raj Chetty has done.
I mean, we’ve done that at a smaller scale looking at longitudinal volunteering and employment
data. There’s some important policy questions that could be asked, using these types of data sets with sophisticated statistical methods. That’s not to say it’s not good.
I also think that there’ll probably be some emphasis on cost-benefit type analysis, return on investment, those types of methodologies,
but again, at the heart of those approaches those assumptions have to be based on some
evidence that something works.
So you’re going to build a model around something, you still need some evidence somewhere that
these outputs are going to happen, in theory. It can’t all be speculative. And then I’d say my third bet, my third wager
I’d put down, is these social investment bonds, the paper
success models. I think this administration is probably likely to latch on to those sort of models.
And again, you’re going to pay for results. You still need effective services to pay for
results for. So I would say that’s probably where it’s
going to go. But that’s all I know.
[Laughter] JOSEPH WHOLEY: So getting ready for tonight,
and again, thanks for coming to this event, I said to myself,
I’ve been watching from different vantage points nine administrations and nine transitions. I only watched some of the time.
And sometimes I was inside, but sometimes out. So I’d like you to imagine with me –Tarek asked me if I’d want to have any slides
for tonight and I said the print would be too tiny!–
[Laughter] so just imagine with me a matrix that goes
about nine rows down
and about, well, we’ll just stick to three columns now.
And so we’re interested in whether people use information,
too, what do they do with it. So, who might be using information?
So one of the columns will be the present administration, political appointees, and senior people in the agencies.
That’s one column. Then there’s the congress,
and then there’s the agency itself, which might be a department or a bureau,
or whatever. So I had this huge, uh, little tiny print,
and so forth, and I’m just going to give you some vignettes
from different cells in that imaginary matrix. So let’s see…
We had the Bay of Pigs, I won’t go into that. [laughter]
There was evaluation, and learning. Also, there was the Peace Corps–
came out of a campaign speech. The Peace Corps, Sargent Shriver was the head
of the Peace Corps, when he wasn’t being head of OEO,
which he also was head of at the same time. But he would send journalists out as his evaluators. NANCY KINGSBURY (off screen): I told them
about Charlie, too. JOSEPH WHOLEY: And I really love qualitative
information. I don’t like “data” and so forth because it
sounds like numbers but I like “evidence” and so forth
but he used journalists as his evaluators. So that’s from the Kennedy years.
I was in the Johnson administration, planning program budgeting systems,
a lot of stuff there. But we were doing policy analysis
and heady stuff for a young guy. His study is in the State of the Union message,
and is a new law, called EPSDT, Early and Periodic Screening, Diagnostic and
Treatment. In that same law, the Internal Child Health
Act, was not a 1/2 a percent for evaluation,
UP TO a 1/2 a percent. And just in grade, UP TO a 1/2 a percent.
And the Public Health Service Act, the next year, up to 1% may be taken for evaluation. So never did we say “1% for evaluation.” We said “Up to.” And that’s important.
So that was in the Internal Child Health Act. Now let’s see…
President Johnson… Hmm… [Joseph] Califano was running, by the way,
the whole domestic side of the government, because the President was worried about
something called the Vietnam War. Califano later was the Secretary of Health
Education and Welfare, became Secretary of Health and Human Services while I was there. So I was in those two administrations on the inside. And Califano knew that he could, if he commissioned evaluations he would be long gone before the information came back.
So he took the planning evaluation staffs from the ten regions
–there were ten in those days– and he had 40 people, his evaluators,
and they would do looks around the country. The first one was Head Start, a politically
important program. Up to 1/3 of the projects are any good, in
the Head Start program. It lives on because of the political support
that it has. So anyhow, Califano sends out people and all
of a sudden he wants outcome information, outcome monitoring. “Joe Wholey, would you go and create a monitoring system?” Well, we brought back classroom observations, and how… No! He wanted to know, will the kids flunk kindergarten? Or how are they doing by the 2nd grade? I mean, that’s what he wanted in outcome monitoring systems for Head Start. So anyhow, that idea of–and also I wanted
to say, he had the first Inspector General in government,
and later on a lot of other Inspectors General who do a lot of evaluation.
Tom Morris was supervising. The evaluation was done by these,
you’ve got people in 10 regions, you can see a lot of things in a short time.
And so I want to emphasize this: the reports the evaluators wrote, because
they were evaluation people, they knew that they were 300 pages long.
These reports could be no more than 20 pages long and the information was conveyed through
briefings. So the Secretary, or in those days Under-Secretary, Dale Champion, took it home. The next day, the people who did the study
are giving the briefings. And right in front of you can see the top
guy (and they were alternates for each other)
making decisions. It was heady stuff. I remember Mike Hendricks who was the close
in supervising did a little briefing on briefings: how do
you brief the Secretary? And he said, “Never sit down, as a briefer. Never sit down. As soon as you sit down, you’re the junior person in the room.
If you stay on your feet, you can be part of the conversation about fixing this program and making it better.” Was really impressive, and I told the FBI
later, when some of their people, top executives
were in my classes, “These reports are fine, ok. Briefings is
the way.” And it helped, too, in the FBI.
I’m just making a few points here and there. So we get to Clinton.
Government Performance and Results Act. Heady stuff. We thought it up at
the National Academy of Public Administration. The American side of public administration thought of something similar, and we got it in, and then we’re teaching
the agencies how to do it. Strategic planning. The first mention of evaluation in a statute. What evaluations were used in the next strategic plan? And so forth. Pretty important stuff.
But as we learned both in the Program Assessment Ratings Tool
and the GPRA reports and so forth: a lot of information produced; very little
used. Very little used. Go ahead to the Obama years,
Government Performance and Results Act… Modernization Act. Let’s have, not measure everything, which you still do, but let’s have goals and progress reports
that are owned by the top people in the agency. And General Shinseki was head of the V.A.
(I don’t remember who was head a part of the time) but he insisted on the goal being, “No homeless veterans.” It unleashed efforts–federal, state, local, public, private, nonprofit–
across the country. He wouldn’t settle. He said he’d go to the
President on this. I’m impressed they didn’t put the targets
in for every two years as zero, but my community doesn’t have any homeless
veterans. Other communities don’t. And there’s been
a dramatic reduction on how many homeless veterans there are in
this country. [sirens from outside]
Now if this new President tries to give up on cutting down the number of homeless veterans, it may not matter anymore because the country is very interested in this
and is going to keep working on it. And I might say that when this president was
elected, I did stay up and listen to his speech, and
so forth. When he said he was going to bring people
together and all that. He’s said other things too, since that night.
But we have two sovereigns in this country if you read your Constitution:
the states are sovereign, and the national government is sovereign.
It’s not like any other country that I know of in the world.
The states are sovereign. I won’t belabor that point.
But it helps, because I’d like to be optimistic as others do.
[Laughter] What else? I don’t remember when the GAO High Risk Series started. That what the first one, wasn’t it? NANCY KINGSBURY: It’s been 9 years? No, 13
years. JOSEPH WHOLEY: So every two years, the General… well, it was the U.S… NANCY KINGSBURY: Government Accountability
Office. [Laughter] JOSEPH WHOLEY: It was called–oh, I was working for Dave Walker at the time– It was called the U.S. General Accounting
Office. Or “GAO”, ok, and he put Nancy and other top people in a room, and made them struggle with “What are our core values?” And then he carved them into the front of
the building. Accountability. Integrity. Reliability. Look at it when you go in the door at GAO. And then put them to the strategic planning
and all that stuff. He said he’s not bound by the Government Performance
Results Act, but he’s going to have the best strategic
plan in the whole federal government. As a model, and so forth.
And then he had GAO assessing how well all the agencies did strategic planning
and they thought they knew, the agencies, but they didn’t have a clue
because the committees would bring them back to do deeper studies
on little pieces of the hard ones. But now, the GAO people had to help the whole
department do a better job. So it was a quite, quite exciting time.
Let’s see, what else? Oh, PART–the Program Assessment Rating Tool
has a wonderful question in it: How has the information been used to improve
programs? You can throw in the whole thing, but that
question was a wonderful question and should be in our minds all the time.
I don’t think of two uses for evaluation, accountability and learning.
When Carol Weiss invented conceptual use, I thought she was cheating.
[Laughter] But I have in my mind program improvement.
That’s my goal in life, program improvement. So I would imagine that learning does get
to program improvement, but Carol Weiss counted it if we were learning more. That’s fine. That’s good. I like Carol Weiss.
I met her daughter… What’s her first name? Janet?
She goes over to the Government Performance Coalition meetings
and she’s spending a sabbatical-type thing with Kathy Newcomer this year at GW.
And I said, “Are you Carol Weiss’ daughter?” She admitted that she was.
[Laughter] She’s a professor at University of Michigan
or someplace like that, spending the year at GW.
A very good person to know and talk to. I don’t think I’ll say any more.
Cross-agency collaboration came up, and my interest now is in evaluating collaborative
efforts to lift up the people who need it,
especially the poor, people of color, etc. That’s my goal in life at this point,
and probably a lot of people in the room, too. Thanks for the chance to talk to you. TAREK AZZAM: Mark? MARK LIPSEY (offscreen): Mark Lipsey from
Vanderbilt. TEI instructor. Something I’ve been pondering that some of you have touched on and I’d be interested in hearing more:
I’ve been around a long time, too, and it seems to me that evaluation,
use of data, and so on has gotten more and more embedded
in thinking, and agencies, and certainly these executive administrations make a difference. I mean, things where you’ve all talked about this. But I’m wondering how much of a difference? How much will continue to go on, because
it’s in many ways, I think, part of the thinking and part of the action of many of the agencies
now in ways that kind of, I think, works underneath what the executive priorities are. And I may be wrong about that, but the current context kind of raises the question of how, how vulnerable is the concept of using evidence and the concept of doing evaluation, and my optimism is that actually a lot of it is embedded in ways where it won’t make that much difference what happens at the top. But I’d be interested in your sense of whether or not I’ve just gone off the rails here. [Chuckles] NICK HART:Totally off the rails. [Laughter] NICK HART: So I think, if I can go first, my take is that that’s about right. The big risk seems to be in areas where we
do a lot of ad hoc analysis. So thinking about Health and Human Services:
they have an authority that allows states to waive certain provisions of human service laws, and do demonstration projects, practically, with evaluations attached to them. And each of those requires a very discrete decision to take some action. So will the number of those projects change?
Potentially, although it depends on how many states request them. But for things like routine monitoring of performance? I think, as others have alluded to, with GPRA, GPRA Modernization,
these things are very much on every agency’s radar, at least in the federal government. But I think more importantly, if you look
at municipalities and states, even at a state level there’s
been a really interesting growth in this conversation about how you can use
open data to look at potholes, and figure out how to deploy your transportation services
to fix those roads better and more efficiently. And a lot of states have really started to
figure out how to make the cost argument, that doing good analysis can actually help
them save taxpayer dollars for very targeted types of projects.
I think it’s been really interesting to see that get embedded
in the last, really, 5 to 10 years? And I don’t know how much of it has to do
with advances in technology or level of interest, but I imagine that there’s
always been a level of interest in some of these questions,
and we’ve just figured out better ways to monitor how we deploy ambulances, and whether firetrucks are going to the right address.
But there are a lot of municipalities that are able to do that now because they have the data systems that exist to do it So fundamentally I think, I mean, part of the Commission’s work is really looking
at how government already has a lot of data.
What are we using well? What can we use better? And at the federal level, we know we collect
a lot of data from our administrative programs that we can use for evaluation and research purposes that we don’t. And I think that there’s clearly a signal that that whole effort is going to continue
moving forward. To what degree? We might have
some productive conversations about that. JOSEPH WHOLEY: I’m interested, Mark,
that the Government Performance and Results Act Modernization Act requires by February of 2018 that the departments develop new strategic plans. It also requires that the departments come forward with what they call “Agency Priority Goals” in the lingo that OMB uses, that might be slightly different words in the statute. It’s worth reading the statute. And the thing is that as I mentioned before,
the people at the top who choose which goals we’re going to be accountable on
and there’s a whole machine, a whole support structure available to them, the people are based over at GSA, but that’s just a device, to kind of like, in a way, White House staff or supplementary OMB staff or something like that, to help the departments learn how to do better in achieving the agency heads’ top priority goals. Ok, and they do have some cross-cutting goals as well that come out of the White House and OMB. But I’m interested in the agency priority
goals, where every it’s either 3 months or 6 months,
they’re putting up in public view–the congress told them to do that–
how are we doing? You know? So typically it’s a mixture of output and
outcome goals but some of them will resonate with the new
administration like, trying to improve the permitting process so
that you’re trying to build something or whatever and it’s a nightmare, especially in California. There may be 10 or 15 different people who can say no.
People wonder if the interstate highway system could ever have been built.
And they’re trying to build a trolley type or light rail or something
in Maryland right now and they’re worried about: first of all, the owls up there–will they get enough food and whatever? Will their babies hatch and fly away before they start cutting down some trees? Plus, there’s some little, tiny little thing
in the water that they have to worry about too.
And so, it’s good that we have the pendulum in this country, and maybe there’s a book you might read sometime called “The End of Liberalism” by Theodore
Lowi. And we get all bound up with these good liberal ideas, and so in a way if some pushback occurs,
the congress won’t let the pushback go too far. But the congress said they want this and that from the departments,
and they want the top people to say “What are the goals we are pursuing?” And then tell the general public it’s either
every 3 months or 6 months how it’s going. So I think there will be machinery there that
encourages use of evidence. NANCY KINGSBURY: I would add a couple of things to that. And before we leave, your very nice story
about our core values… one of the things that really amuses me about GAO is when Dave Walker came in and wanted to do our core values, he brought our managers together and I was one of them at the time and he said he’d been around saying he’d collected 25 words or so, It took GAO’s senior managers 18 minutes to agree on “accountability”, “integrity”, and “reliability”. The walls knew what our values were. They’ve always been our values, and with the words on it.
The next thing he wanted to do, by the way, was our congressional protocols,
and I later asked him 2 or 3 years later if he thought he thought we were going to
do it in one more meeting, and he said yes and it took 8 months and 65 hours. [Laughter]
I like telling that story. The other thing I would say is, that I didn’t
mention earlier, is that one of the other interesting developments is that a law was passed recently, didn’t get headlines,
to give the Inspectors General additional authority to get information from agencies, and relieve them from the burdens of the Paperwork Reduction Act.
Now that is, excuse me, the GAO is not subject to the Paperwork Reduction Act, so we do lot a of surveys as original data collection. And we don’t have to ask OMB squat. Right? And now the Inspectors General don’t either. We’re gonna have a meeting in 3 weeks’ time with the senior Inspector Generals
and two of the topics on that meeting are surveys and data matching.
Because there is now much more likelihood that they will get access to data
that they’ve had trouble getting access to in the past, and I think that’s an area that will expand. Now the difficulty is they don’t currently have the skill set that they may need to do that.
But I think that is also an interesting trend. It’s hard for me to… I know somebody wrote
a column recently about trying to get away from Trump
and they couldn’t do it. They couldn’t go anywhere for information
anywhere that didn’t have that that ego in the middle of it.
And when that begins to calm down, and you get agency heads, when you get the
next layer of people which they don’t have now
and the next layer below that, which they don’t have now,
I think there will be receptivity to information if only because a lot of these people are
coming out of the business world, and that’s what they’re used to doing.
But until that happens, the stuff that’s on the front page
is a little hard to get past. Our experience, on the other hand,
because we do work for the congress, is there is no lack of interest.
There is no lack of interest. The National Defense Authorization Act had
44 mandates for GAO. The 21st Century Cures Acts had 27 mandates
for GAO, something like that. So the interest is there in the congress,
which is also sovereign. JOSEPH WHOLEY: Even more so! [Laughter] NANCY KINGSBURY: And the other thing that
we engage with is with the state and local audit community
that is increasingly getting involved in doing work of this sort.
And so, and of course we’re involved with the international auditing community which is a whole different thing. So I think there’s lots of audiences there
for the kind of work that we do as evaluators. But at the moment it’s hard to focus on what
questions they want to have answered because, again, to go back to my–I started with when Reagan came in… JOSEPH WHOLEY: You’re so young! NANCY KINGSBURY: I know.
I knew I was going to be the oldest one here until you got here! [Laughter]
But I mean in the early days of the Reagan administration nobody wanted evidence done on anything, right? That was your experience too.
And that’s what it feels, right now. I mean that’s sort of the way it feels at
least in the headlines right now. And I’m going to say again to Nick, that we
are very pleased that the congress decided to stand up the
Commission and keep this issue on the front burner at
some point in time, because it’s very important, and we’ve had
a lot of people in your meetings who’ve said so. MARY HYDE: I would say that your assessment
is true at our agency. A lot of what has been going on over the years is really now part of standard operating procedures.
Even in my short tenure I’ve seen pretty significant shifts in appetite among career staff and changes in significant ways in which we
do grant making that, short of the entire agency walking out,
I think it would be hard for it to just die off because everyone, not everyone, a lot of people have bought into the value of using different sorts of evaluative strategies in operations.
I think it’s true that there’s no appetite in recognizing the value and the interest
and desire for the information. PETER BERNSTEIN (off camera): Yeah hi–Peter
Bernstein from WestEd and Washington Evaluators. Can I say I was particularly struck by your comments because, roughly right out of grad school, I was at the Office of the Assistant Secretary for Planning and Evaluation riding in the wake of Joe Wholey and doing task order contracts. I was there when the 1983-84 Reduction of course was put in place. So you all have talked about money a little bit. You’ve talked about the set-aside, you’ve talked about “It’s not 1%, it’s ‘Up
to'”. Let’s talk about money a little bit.
So the Reagan administration shifted a bunch of resources from domestic programs to defense, spiked the deficit to levels that
had not been seen before. I agree with you Nancy, because we have a
similar situation now. Follow the money. We’ve all been very optimistic
about the role of evaluation. I’m very pessimistic about the funding for
evaluation. Not the desire for it, not the intent of it, not the purpose of it, and not the professionalism of it, all of which I’m a firm believer in and have been for many years. But if we don’t have money for programs, there’ll be no money to evaluate those programs. I’d like you to talk about that a little bit, and how that has sort of ebbed and flowed
across administrations. JOSEPH WHOLEY (quietly): Well let’s think now. WHOLEY (Out loud): Well, Nick’s already said that there were some surprises that maybe in the Republican administrations the resources for evaluation sometimes went up. I had this quote from my wife, but I didn’t use it but, it is true that neither Reagan nor Trump needed information because they knew already what they wanted to do. So they don’t need evidence. But it also is true that at least in the first
case, with Reagan, that the congress–when the programs had evidence
to support them– I looked at, had my minions out looking at
four or five different departments, I guess four departments, five programs–
and when there was evidence, either from the performance monitoring schemes or from evaluation studies that the programs were good, then the congress didn’t resist. Like, Job Corps was one.
Senator Hatch said, “We’re going with the President on a lot of things,
and his program for economic recovery. We’re not going with him on this one.”
And it was a benefit-cost study which you maybe never could do a perfect job,
but it was a pretty good job at the time, and it seemed as if the society gained more than it lost from the big bucks spent on each person in the Job Corps. But I think you’re right. I think that we
may have a reduction in resources for, quote, “evaluation”, unquote. But you can get evidence lots of different ways, and it doesn’t need to be called “evaluation” or “evaluation studies” or “evaluation research”.
The bigger worry, though, is that the staffs with the technical expertise are swept out. I won’t say more, but I share that concern, and I know that the other concern is particularly important for those of us who have been evaluation contractors.
[Laughs] NANCY KINGSBURY: I would add something a little longer-term perspective about that. I agree with that. I mentioned in my comments earlier that the budget numbers for the non-defense spending that are being thrown around at the moment are draconian
and do not bear well for a whole lot of things, including whole agencies.
But I am intrigued. My boss, the Comptroller General had occasion to meet with the Army’s Long-term Strategic Science Board a few weeks ago, a few days ago, and they wanted to meet with us because we had just reissued a report on the long-term fiscal sustainability of the federal government at all. Right?
And I have to admit, even Gene Dodaro was impressed that the generals that were worrying about how to plan for the Army
30 years from now are dealing with that issue. Which is, you referred to the growing of the
deficit and all that kind of problem, so in the larger context we have a huge problem
in this country. You know, evaluation won’t fix it.
I mean, we’ve got to use lots of other things. And if we got to doing lots of those other things, we’d have some money for evaluation because then it’d be important. NICK HART: So having worked at OMB, you probably would expect me to have a perfect answer to this question. But I don’t, because at the end of the day, money matters when we’re talking about evaluation.
But it’s not the first thing that matters. You’ll find that an agency head or a political appointee who wants an evaluation in a time of tight budgets, they’re going to find the money to do it. And so I think this is an issue about, again,
back to my earlier comments, how we articulate the value of evaluation
and make sure that political appointees, career senior civil servants, know what role evaluation can play in their decisions. So let’s talk about the money specifically then. So discretionary funding is only one vehicle for funding an evaluation. It’s not the only one.
The Department of Labor uses fees to fund some of their evaluations.
HHS relies largely on mandatory funding for the biggest portion of their evaluations.
State and local governments are still going to be pursuing some level of evidence-building
activities as we’ve already talked about.
So I think the point is that there’s a lot of promise
for a lot of different mechanisms existing to fund evaluation. We shouldn’t think about the discretionary budget in the federal government as the sole source of funding for evaluation
because it it’s not. And it doesn’t mean that if the number goes
down it’s going to be pretty, but I don’t think it’s the death of evaluation.
In in fact, based on some of the examples that I’ve seen, it just makes the value of evaluation even stronger in some cases.
So that’s me being optimistic again. MARY HYDE: I think that I’d agree with Nick. I’d probably add that if you want to start talking about a line item in the agency around evaluation and if it gets cut, how that money gets redistributed, I think that there are way to be creative
with cuts in budgets. I do think that there will be certain parties
that do evaluation that will feel that cut first.
If your budget is cut, in an evaluation office, you are going to scale back contracted out
evaluations. That’s inevitable. Even in the best of fiscal
times, there’s probably not enough money for evaluation. TAREK AZZAM (off screen): I just want to be respectful of people’s times. It is after 8:00 right now. And so you’re welcome to talk with the panelists afterwards if they’re willing to talk to you.
[Laughter] I don’t want to speak on their behalf.
But I just wanted to also thank the Washington Evaluators for co-sponsoring this session with TEI, and please join me in thanking the panelists. [Applause]