Consultation and feedback

Consultation and feedback

Consultation methodology

The stakeholder consultation program for this review was divided into three phases.

In phase I, we reached out mainly to the vice-presidents of research at Ontario’s universities, colleges and hospitals. We also reached out to the Ontario Council on University Research (OCUR), Council of Academic Hospitals of Ontario (CAHO), and Colleges Ontario.

Using a questionnaire format, we requested feedback in four key areas:

• Are the program designs doing what they intended and having the desired impact?
• Do the programs still have the right focus?
• Are there programmatic gaps?
• Are there opportunities to streamline?

We expanded our consultation in phase II to include private sector partners, specific Ontario research institutes, such as the Perimeter Institute, and selected federal government organizations such as Innovation, Science and Economic Development Canada, and Canada Foundation for Innovation (CFI). Outreach to these stakeholders was done through a variety of methods, including focus groups, surveys and teleconference meetings.

In addition, Ministry staff helped create focus groups and/or conducted surveys of former ORF and ERA Peer Review Panel Chairs/Panel Members, multiple award recipients (ORF and/or ERA), and unsuccessful award recipients.

As part of Phase III, we returned to some of the leading representative research organizations to test and validate our recommendations. This included meeting with the Ontario Research Fund Advisory Board, whom we engaged from the start of the consultation process; and with Ontario’s Chief Scientist, Dr. Molly Shoichet.

A complete listing of our contacts with stakeholders in the course of this review may be found in Appendix A.

Stakeholder feedback summary

The majority of stakeholder feedback was directed at ORF-RE, although some of it was expressed in broader terms about “ORF” collectively to include ERA, ORF-RE and ORF-RI.

Overall, many stakeholders confirmed that the three competitive programs are an important tool for attracting and retaining top research talent. “The ORF is a primary driver of discovery and innovation; and is a bright light for researchers in this province.”

The importance of early researcher awards, in particular, as one stakeholder pointed out, goes beyond financial assistance. “They help establish greater recognition for the recipient; and increase the likelihood of success in other funding programs.” But the feedback on ERA also pointed to various areas for improvement, with a focus on administrative efficiency, funding allocations, performance metrics. Some institutions also suggested that an increased emphasis on mid-career researchers would be of benefit.

There was also strong stakeholder support for the design of the two ORF programs, with some tweaks.

With respect to ORF-RE, stakeholders appreciated the flexibility to use the funds for operational purposes, including hiring post-doctoral fellows and support staff (technicians, administrators, research assistants, etc.). They were also pleased with the 40% top up of project costs dedicated to overhead, a proportion that is aligned with true costs when compared to research funding programs in other jurisdictions.

As well, they noted that the program has become more inclusive of research disciplines beyond the natural and life sciences, especially in the social sciences, humanities and the arts. However, some suggested that greater emphasis should be placed on multidisciplinary research projects, which may involve collaborators across many disciplines working on a common problem.

Some institutions expressed some concerns about the ORF-RE program and how it affects their ability to do world class research and to attract exceptional talent.

The perception is that all three flagship research programs (ORF-RE in particular) are oriented towards “industry relevant” research targeted towards a commercial application or outcome, even though the application form covers benefits and impacts that are much broader in scope. Nevertheless, many stakeholders indicated that the application process places fundamental or basic research, where potential end-users are further downstream on the innovation pathway, at a disadvantage.

Some institutions, especially colleges, the smaller ones or those without significant medical research capacity, find it very difficult and time-consuming to raise private sector contributions. They say that the high level of required partnered funding associated with the ORF-RE program limits the pool of researchers who are able to apply; and that it disadvantages those who conduct basic research, as well as those in the social sciences and humanities.

Stakeholders expressed a general frustration with the administrative and other application aspects of the three programs. Applications are quite lengthy and complex. Budgets are also especially complex and require the institution to devote significant administrative resources, which colleges and smaller institutions do not necessarily have. They suggest that the Ministry implement an effective electronic submission process for applications, including all aspects of the submission such as letters of reference.

Analysis

In the Introduction we proposed a framework of analysis based on what we refer to as the six attributes of an effective research support programs:

• excellence and innovation
• talent and teams
• connections and networks
• technical and financial sustainability
• outreach and public engagement
• administrative efficiency and accountability

The framework was developed as a systematic means of organizing and analyzing the large amount of information that we encountered in the course of reviewing Ontario’s three competitive research programs. We suggested that it could be applied to individual programs as well as a suite of programs; realizing, however, that depending on the objectives and design of a program, not every attribute applies equally or sometimes not at all.

Below, we have attempted to synthesize our analysis across the three programs, with a focus mostly on ERA and ORF-RE which is where we received the most stakeholder feedback. Overall, we find that the program individually and together make a strong contribution to Ontario’s research and innovation system by providing sustainable and flexible funding support to academic research, leveraged by private funding. It is important to maintain the pivotal role that these funding programs play in creating opportunities for research talent, enhancing the research competitiveness of Ontario’s publicly-funded institutions, and for attracting investment to Ontario’s economy.

Excellence and innovation

We are impressed by the high quality of research that is being funded by the Early Researcher Awards program. We encountered many examples of where research funded through ERA that have contributed to scientific knowledge and enhanced understanding of social problems. It has opened doors to innovative and more complex research projects.

We are also impressed with the high quality of research that is being funded through ORF-RE and ORF-RI. The peer review process is working. Without this support, Ontario would surely lag other jurisdictions in research output; and Ontario’s institutions would be less able to compete for world class talent and R&D investment.

Nevertheless, there is a perception that ORF-RE, in particular, is tied to industrial collaboration which, as expected, tends to favour projects with more commercial and industrial relevance.

ORF-RE application guidelines do not explicitly distinguish fundamental research projects with a theoretical or experimental bent from those with a more practical purpose. And in our view, it should not do this as the two types of research orientations cannot be meaningfully separated at the scale of a research funding program.

Research excellence means that a project has to demonstrate a potential contribution to the world stock of scientific knowledge, which is both fundamental and applied; and only then to the areas of strategic relevance for Ontario’s economy, which are not even clearly defined or current as the Ontario Innovation Agenda has not been updated for nearly a decade.

Talent and teams

We heard from stakeholders, including many researchers, that ERA has played an important role in launching the careers of many researchers, and who have gone on to become principal investigators in other research funding programs. It has also played a key role in the development of research teams.

We heard from both institutions and researchers that ORF-RE is effective in attracting, recruiting and retaining excellent research talent. We have less information, however, on how effective it has been to help individual researchers advance their career path from early stage to mid-stage and beyond. We have some anecdotal evidence that the requirements for preparing and application and lining up partners is a major hurdle for many younger researchers; and, therefore, a potential hindrance to their career advancement.

We also understand that researchers in smaller institutions are at a disadvantage because they have less access to resources for support. Finally, we lack a comprehensive picture of how well ORF-RE promotes inclusivity and diversity of research talent.

Connections and networks

ORF-RE is designed to promote collaboration and linkages across research institutions and with community and business partners. Building such connections, however, appears to be easier for researchers in some disciplines than in others; for example, the health sciences vs. social sciences.

Although this was not a major objective of the ERA program, some institutional stakeholders and researchers who participated in the consultation process noted that it had a modest impact on helping them to establish connections with other researchers in academia and the private sector, and in other parts of the world. There is evidence that the number of ERA-funded research collaborations has been on the rise.

Technical and financial sustainability

One of the key features of the ERA program is that it provides an appropriate amount of funding to cover the indirect costs of a research project. Similarly, with respect to ORF-RE, most stakeholders were very appreciative of the 40% funding support for indirect costs.

We note, however, some concern about the extent to which industry contribution is calculated based on cash or as “in-kind” value equivalent for equipment, databases, and potentially personnel. There appears to be some confusion over which is preferred by the Ministry – or if there is even a preference. The “mix” of industry support, whether it is tilted to cash or to in-kind, matters only in how well it supports research excellence and the infrastructure that is created and becomes a building block for future research projects.

But we also would like to highlight that industry or non-profit partner contribution is more than about money and other resources. Perhaps even more important of industry partnership is the contribution that is made in terms of expertise, data, and market knowledge that can leverage the impact of the research for the good of society.

Outreach and public engagement

Encouraging youth to explore and potentially pursue careers in science and engineering is one of the unique elements of the ERA and ORF-RE program. Youth outreach is not a requirement for other research support programs anywhere in Canada. Other countries have initiatives to engage youth in science, but not directly as part of a funded research project. ERA and ORF-RE allow for the allocation of up to1% of the project budget towards outreach. A detailed youth outreach plan in the application is required. Proposals must outline a plan and include (where funds are allocated) the spending in the proposed budget.

The number of young Ontarians exposed to the work and methods of researchers through the ERA and ORF-RE programs is substantial and unique when compared to research programs in other jurisdictions. The number of researchers willing to do numerous and varied outreach activities over the life of their projects in both programs is encouraging. The Ministry should also publicly report on the progress made through the funded projects, including youth outreach activities.

Administrative efficiency and accountability

The time and resources invested in preparing and submitting an application to an ORF program is significant, and perhaps daunting enough that it discourages applications from early-stage researchers, colleges and smaller institutions.

Much the same can be said of the post-award reporting requirements for meeting project milestones and demonstrating outcomes and impacts or what economists refer to as a social return on investment.

This comes back to the question of what gets counted as a “return”: knowledge, talent, technology or other tangible and sometimes intangible outcomes per dollar spent that are difficult to quantify but are required to demonstrate accountability. It is important to understand that there is not a one-to-one relationship between science and innovation; and that all kinds of scientific endeavor contribute to our well-being, from the curiosity-driven, fundamental science that generates new insights about the physical and human world to the science that is more directly linked to evidence-based decisions about policy and investment.

There is much that the Ministry can do to streamline and ease the administrative burden on researchers, who should be spending more time on their research projects rather than on filling out forms. It would also help if competitive rounds are held on a more predictable schedule. A related point, that perhaps also touches upon the excellence and talent attributes of an effective program, is that more effort should be placed on providing meaningful and timely feedback to the successful applicants – and to the unsuccessful ones so that they are encouraged to apply the next round.