There were so many thoughts and things I wanted to say during my talk on the OpenCon Research Evaluation panel but I just didn’t have the time to include them all. Here I’m filling in some of the gaps of the talk I actually gave a bit.
My thanks to Nick for inviting me to be here today. I’m so honored to speak to you all and to be a part of this panel. I have gained so much from the OpenCon community that I’m thrilled to be able to give back.
Let’s end on a note of hope. We talk a lot about what is wrong with the scholarly communication system, but I’d like to take a few moments to envision a better way.
First, I have to acknowledge the privilege of my position. I work in the US, in a high research activity university, on a unique campus that supports innovation, in a well resourced library, with administrative support for openness.
I was invited to speak to you today because I work at a rather unique place. Civic engagement has been a part of our campus mission since it’s inception. This means that we don’t function in isolation – we are a part of the community. We need to work with the community, along side the community members. This emphasis creates an atmosphere that is perhaps more willing to embrace open.
While it is really important to have champions, but those values need to be encoded in policies and institutional practices in order to enable sustainable change.
These are some of the ways that we have tried to encode openness in our campus policies and practices.
Engagement in faculty governance has led to addition of OA as a campus value and we are collaborating with specific schools and departments to revise guidelines at those levels. Our education efforts include workshops & consultations to help faculty make a stronger case for promotion and tenure. We have been able to change practice by modelling the practices we advocate for, openly discussing how those practices are affecting our own professional advancement, and by establishing an Open Access Fund.
Our current practices emphasize the value of research product, rather than valuing all of the work that we do as scholar – including teaching and service.
To successfully advocate for the values we want to embed in scholarly dissemination, the dossier should function as a communication tool – the story you tell should embody the values you hold.
The choices that we make about how we communicate and disseminate our work should reflect our values. We can be more strategic about planning for dissemination and impact. Think more broadly – who are you trying to impact? who are you trying to communicate with? A clearer understanding of these goals makes us better communicators.
As I help faculty develop their dossiers, I tried to break it down into simple categories. Good scientific communication principles emphasize the first three, but I think we also need to keep in mind the type of impact we are trying to achieve. The next time you think about your work, think about how to create or contribute to the dialog.
When you think about the types of impact your work can have, there are a couple of helpful resources. The first is ImpactStory’s “flavors of impact” concept. The second is the Becker Model for Assessment of Research Impact.
While this came out of a medical research environment, it can be translated to non-medical research. For example, at IUPUI I adapt the clinical implementation category to a campus value and initiative called TRIP – translating research into practice.
Right now, promotion and tenure is largely based on articles and citations.
Until a few years ago perhaps, we were dependent on the linkages between these very static and narrow products (articles, books, presentations).
In this model, narrative statements focus on quality, reputation, prestige. This way of valuing scholarship is very self-referential and closed.
We are in a transition period. The types of scholarly products that “count” is expanding.
As are the types of metrics available, but we are still learning what these metrics data mean. Too often, their use is driven by sheer availability, rather than the strategic and thoughtful dissemination.
We have lots of data, but data and metrics are not the solution. This is a cultural problem. Metrics can be a great way to start the conversation.
I believe a better way for scholars to demonstrate the impact of their scholarship is by taking a program evaluation approach. This means identifying the products created, the stakeholders or audience for those products, and the desired short-term, intermediate, and maybe even long-term outcomes. When a strategic plan is created, it enables a scholar to gather, present, and understand how specific evidence (including metrics) can be used to demonstrate the impact of their work.
At my own institution, a great example of community engaged research with direct community benefit comes from our school of public health
Silvia Bigatti: Your Life. Your Story Latino Youth Summit
This is a powerful story that demonstrates the impact of her research, teaching, and professional practice, but our current system does not really provide guidance in how to document this type of impact, particularly the face-to-face, personal contact.
Her work is beginning to get local and statewide news coverage. How can this faculty member tell the story of this project? How can we help her demonstrate impact that isn’t captured by metrics?
First and foremost, the dossier is the story of a scholar. Metrics and evidence are great, but the story needs to come first. As we support faculty in developing dossiers that tell their story and document the impact that their work has had on the community. These are some of the ways I think that we can change how dossiers can communicate diverse impact and advocate for greater openness in research.
Open is a means to an end, not the ultimate goal. Open is a set of practices that promote the values we want our scholarly ecosystem to embody.
TAKE ACTION: Like Ahmed said, change is hard and it may take time to for your impact to be felt or seen.
I hope these questions can stimulate our discussion about how to improve research evaluation.