Appendices

Appendix A: Abbreviations

Abbreviation Explanation

CAHO Council of Academic Hospitals of Ontario

CANSIM Canadian Socio-Economic Information Management System

CFI Canada Foundation for Innovation

CIHR Canadian Institutes of Health Research

CIIF College-Industry Innovation Fund

CIMD Centre for Industrial Material Development

D.V.M. Doctor of Veterinary Medicine

ERA Early Researcher Award

HIFU high-intensity focused ultrasound

HQP highly qualified personnel

IEEE Institute of Electrical and Electronics Engineers

ISF Israel Science Foundation

LIF Large Infrastructure Fund

LSARP Large-Scale Applied Research Projects

M.D Medical Doctor

MSI Major Sciences Initiatives

NSERC Natural Sciences and Engineering Research Council of Canada

OCUR Ontario Council on University Research

OIA Ontario Innovation Agenda

ORF Ontario Research Fund

ORF-RE Ontario Research Fund-Research Excellence

ORF-RI Ontario Research Fund-Research Infrastructure

Ph.D Doctor of Philosophy

PREA Premier’s Research Excellence Awards

R&D research and development

SIF Small Infrastructure Fund

SNOLAB Sudbury Neutrino Observatory Laboratory

SSAH social sciences, arts and humanities

SSHRC Social Sciences and Humanities Research Council of Canada

Appendix B: Submissions and meetings with stakeholders

This appendix provides a summary of the consultations undertaken by the Expert Review Panel and secretariat with a number of key stakeholders, both within and outside of Ontario. Submissions and meetings with stakeholders with members of the science and innovation ecosystem were imperative in providing feedback to the Panel. We are pleased with the responses received, as they provided a vital contribution in helping inform and shape the recommendations of this report. Please note, where individual responses were provided, names are not listed or identified by name in this report.

Methodology

Consultations with external stakeholders were conducted in three Phases, with a variety of methods implemented, and discussions taking place via in-person meetings, phone calls, e-mail, teleconference, and formal electronic written submissions. In each case, a series of open-ended questions tailored to each stakeholder was provided.

Phase I: Institutions and Associations

A number of Ontario’s key institutions and associations were contacted via formal e-mails and encouraged to engage in the ORF Review consultations, and submit electronic written responses. Phase I of the consultations process was facilitated through CAHO, OCUR and Colleges Ontario. A number key of institutions and associations from across the province were contacted, with 21 institutions providing submissions.

A comprehensive list of the institutions and associations that provided submissions are listed below:

• Colleges Ontario
• The Council of Academic Hospitals of Ontario (CAHO)
• The Centre for Addiction and Mental Health (CAMH)
• Hamilton Health Sciences
• Health Sciences North Research Institute
• Holland Bloorview Research Institute
• Ontario Council on University Research (OCUR)
• Queen’s University
• Ryerson University
• Sheridan College Institute of Technology and Advanced Learning
• Sick Kids Research Institute
• St. Michael’s Hospital
• Sunnybrook Research Institute
• Toronto Academic Health Science Network
• University Health Network
• University of Guelph
• University of Ontario Institute of Technology (UOIT)
• University of Ottawa
• University of Toronto
• University of Windsor
• Women’s College Hospital

Phase II:

During Phase II of the consultations process, a series of open-ended questions tailored to each stakeholder was shared through e-mail by Ministry staff from the Research Division. Participants were asked to provide input on several aspects of the ORF and ERA programs including strengths of the programs, potential gaps, advice for improvement in policies, guidelines or procedures as well as any challenges that they may have experienced. Please find below a comprehensive listed of participant groups as part of Phase II of the ORF Review consultations:

• Federal Government (via teleconference):
  • Canada Foundation for Innovation (CFI)
  • Genome Canada
  • Innovation, Science and Economic Development Canada (ISED)
• Research Institutes (via email submission)
  • Ontario Brain Institute (OBI)
  • Ontario Institute for Cancer Research (OICR)
  • Ontario Institute for Regenerative Medicine (OIRM)
  • Ontario Genomics (OG)
  • Perimeter Institute for Theoretical Physics
• Private Sector Partners (invited via email):
  • ARISE Technologies*
  • Bell Canada
  • GE Healthcare
  • Hewlett Packard
  • IBM
  • Rimon Therapeutics Inc.*
  • Siemens

*Please note: Organizations that may no longer be active are indicated with an asterisk.

The secretariat exercised its best efforts in contacting individuals who were private sector partners on specific projects with researchers funded by ORF-RE; however only three private sector partners responded to the request to participate.

• ORF-RE and ERA (via email submission):
  • Panel Chairs and Members
  • Multiple Award Recipients
  • ORF and ERA Previous Applicants

Phase III:

The Secretariat shared the key recommendations prior to the report’s release with Ontario’s Chief Scientist, ORFAB, CAHO, OCUR and Colleges Ontario.

Consultation Questions by Phase

Phase I Questions

Please answer the following questions. You may answer them individually or address them as a whole.

1. Are the program designs doing what they intended and having the desired impact?
   1. The Early Researcher Awards provide funding to new researchers working at publicly funded Ontario research institutions to build a research team. How has the program played a role in your institution’s recruitment and retention of new talent?
   2. How valuable has the Ontario Research Fund – Research Excellence program been in supporting researchers conducting leading-edge, transformative, and internationally significant research at your institution?
   3. In what ways does the Ontario Research Fund – Research Infrastructure program provide research institutions with funding to help support infrastructure needs, such as modern facilities and equipment?
   4. What could the programs do to further support the recruitment, training and retention of top research talent in the province?
   5. Are there ways to more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate Ontario’s research funding?
2. Do the programs still have the right focus?
   1. When looking at your researchers and their career transitions, what supports are most needed at your institution?
   2. The Research Excellence program currently funds projects between $1 million and $4 million for the general and clean technology streams and between $200,000 and $1 million for the social sciences, arts and humanities stream. Are the minimum and maximum amounts set for the applications appropriate? How do these limits impact your application?
   3. Are there any concerns regarding the required partner funding associated with ORF awards? If so, what are some suggested improvements?
3. Are there programmatic gaps?
   1. Do researchers feel that the appropriate measures are in place to effectively promote and support all disciplines through Ontario’s competitive research programs?
   2. How can the Ministry best address the perception of a lack of continuity in the funding opportunities for the researchers at different levels in their career? Is there a need for a dedicated program supporting excellence for mid-career researchers?
   3. Are there any perceived barriers regarding the ability to receive support for investigator-led research? Do researchers feel that the Ministry’s programs support their ability to do independent research?
   4. What are some suggestions for areas of improvement?
4. Are there opportunities to streamline?
   1. Are there changes to the existing application, peer review and reporting processes for funding that could be improved?

Phase II Questions

The following questions were asked to all Phase II consultation stakeholders:

1. Are the program designs doing what they intended and having the desired impact?
   1. The Early Researcher Awards provide funding to new researchers working at publicly funded Ontario research institutions to build a research team. In your view, how has the program played a role in the recruitment and retention of new talent?
   2. What could the programs do to further support the recruitment, training and retention of top research talent in the province?
   3. How valuable has the Ontario Research Fund – Research Excellence program been in supporting researchers conducting leading-edge, transformative, and internationally significant research?
   4. In what ways does the Ontario Research Fund – Research Infrastructure program provide researchers with funding to help support infrastructure needs, such as modern facilities and equipment?
   5. What do you believe to be the main strengths of the Ontario Research Fund and Early Researcher Awards programs?
   6. Do you believe there to be any gaps in the Ministry’s current research funding mechanisms? Yes or no? Please explain.
   7. Are there ways to more efficiently or more accurately assess the outputs, impacts and contexts of research in order to allocate Ontario’s research funding?
2. Do the programs still have the right focus?
   1. When looking at your researchers and their career transitions, what supports do you believe are most needed?
   2. The Research Excellence program currently funds projects between $1 million and $4 million for the general and clean technology streams and between $200,000 and $1 million for the social sciences, arts and humanities stream. Are the minimum and maximum amounts set for the applications appropriate? How might these limits impact an application?
   3. Are there any concerns regarding the required partner funding associated with Ontario Research Fund awards? If so, what are some suggested improvements?
3. Are there programmatic gaps?
   1. Do researchers feel that the appropriate measures are in place to effectively promote and support all disciplines through Ontario’s competitive research programs?
   2. How can the Ministry best address the perception of a lack of continuity in the funding opportunities for the researchers at different levels in their career? Is there a need for a dedicated program supporting excellence for mid-career researchers?
   3. Are there any perceived barriers regarding the ability to receive support for investigator-led research? Do researchers feel that the Ministry’s programs support their ability to do independent research?
   4. What are some suggestions for areas of improvement?
   5. Are there opportunities to streamline?
4. Are there any other issues or questions that you would like to raise and address?

In addition, the following questions were asked to the Ministry’s Federal Government counterparts and the Canada Foundation for Innovation (CFI):

1. Are there opportunities to better streamline processes between CFI and the Ministry with respect to the administration of programs?
2. Do you have suggestions for how the federal and provincial government can better coordinate its efforts in supporting projects requiring co-funding?
3. Are there any gaps in the processes, policies, guidelines or procedures? What improvements would you suggests?
4. In general, what are the major issues faced by researchers regarding infrastructure funding?
5. In cases where there projects do not receive a consensus by both funding bodies (CFI and the Ministry), what are some alternative options for researchers in seeking out additional research funding? Should researchers be advised to develop a strategy surrounding this potential issue prior to applying?
6. Does the Ministry have an appropriate level of emphasis on the commercial potential of research? Is there too much or too little emphasis? Please explain.
7. Are there opportunities for the federal and provincial research funding bodies to partner further on projects with socioeconomic impacts?

The following questions were posed to Genome Canada:

1. What would you say are the main strengths of the Ministry’s research funding programs? What works well in the programs?
2. Are there any gaps in the processes, policies, guidelines or procedures? What improvements would you suggests?
3. Are there opportunities to better streamline processes between Genome Canada and the Ministry with regards to the administration of the GAPP, LSARP, and DIG programs?
4. In your view, are there more efficient ways to streamline processes between the federal funding agencies and Ontario?
5. Do you have suggestions for how the federal and provincial government can better coordinate its efforts in supporting projects requiring co-funding?
6. What are the main issues or challenges you experienced?
7. Are there opportunities for the federal and provincial research funding bodies to partner further on projects with socioeconomic impacts?

The following questions were also asked to Innovation, Science and Economic Development Canada (ISED):

1. How is the federal government working to address and implement the recommendations from the recent Naylor Report?
2. What is the federal government doing to support early and mid-career researchers?
3. Do you have suggestions for how the federal and provincial government can better coordinate its efforts in supporting projects requiring co-funding?
4. In your view, are there more efficient ways to streamline processes between the federal funding agencies and Ontario?
5. Is the Ministry’s research funding programs sufficient in meeting the needs of the researchers within Ontario? Why or why not? Please explain.
6. Are there opportunities for the federal and provincial research funding bodies to partner further on projects with socioeconomic impacts?

The following additional questions were asked to Panel Chairs and Members:

As a Chair or Panel Member of the Ontario Research Fund – Research Excellence (ORF – RE) program,

1. What do you believe are the main strengths of the ORF – RE program? What works well in the program?
2. Are there any gaps in the program’s current policies or procedures? If so, what improvements do you suggest?
3. Are the Ministry’s guidelines clear in establishing its criteria for researchers applying to ORF – RE?
4. From your perspective, what were the main challenges you experienced?
5. Overall, what improvements could you suggest to the Ministry’s ORF – RE program?
6. Are there any other questions or issues you would like to address?

The following additional questions were asked to Multiple Award Recipients:

As a recipient of multiple awards from the Ministry,

1. What do you believe are the main strengths of the programs? Are there any programmatic gaps?
2. Are the Ministry’s guidelines clear in establishing its criteria for researchers applying to our programs?
3. How have the Ministry’s programs and research funding helped you in your career as a researcher?
4. In your opinion, what were the critical factors for establishing independence early on in your research career?
5. Do you believe you received the appropriate level of advice and feedback from program peer review panels? Why or why not?
6. Overall, what improvements could you suggest to the Ministry’s programs?
7. Are there any other questions or issues you would like to address?

The following additional questions were asked to Previous Applicants:

As a researcher who has previously applied to the Ministry’s ORF and/or ERA programs,

1. Do you feel the Ministry’s programs are well-structured to support research excellence in terms of:
   1. Areas of focus?
   2. Size of grant available? (i.e. too small or too large)
2. Was the process for proposal preparation clear and reasonable? Do you have any recommendations to clarify or strengthen the process?
3. Were the criteria for proposal evaluation clear and reasonable? Do you have any recommendations that could be enhanced or improved?
4. Did you receive valuable and sufficient feedback on your proposal to understand where you could make adjustments and enable you to successfully reapply to a future competition?
5. From your perspective, what challenges did you experience in securing funding from the Ministry? Please explain.
6. Overall, what improvements could you suggest to the Ministry’s programs?
7. Are there any other issues you would like to address?

The following questions were asked to Private Sector Partners:

1. Why did you choose to partner on this project with the research institution through the ORF program? What were you hoping the partnership would bring to your company? What value did this partnership add or bring as a result?
2. What outcomes did your company achieve as a result of the partnership? (i.e. licenses, HQP, etc.)
3. Did your company experience any issues or challenges associated with the Ministry’s programs? If yes, please explain.
4. Were there any issues or challenges establishing and/or maintaining a relationship with the research institution?
5. Do you have any suggestions or recommendations for improving the Ministry’s programs?
6. Do you believe greater flexibility should be given for research institutions seeking matching funds? (i.e. partnership eligibility, matching criteria, timing of partner contributions, how matching funds are obtained).
7. Do you believe that the Ontario Research Fund should maintain its current level of required matching funds from private sector partners? If no, why not? And what level would you suggest?
8. Does the Ministry have an appropriate level of emphasis on the commercial potential of research? Is there too much or too little emphasis? Please explain.
9. Based on your response, how may this impact the researcher’s success in applying for funding?

Appendix C: Ontario’s research and innovation system – emerging technological fields

This appendix examines the impact Ontario researchers are having in five emerging technological fields – artificial intelligence, fuel cells, quantum computing, quantum science (broadly defined) and regenerative medicine. It looks at Ontario’s research efforts in national and global perspective by examining outputs of scholarly publications, the impact these outputs are having on the larger academic community as measured by citations, citations per publication and the share of publications which are among the most highly cited as well as the extent to which Ontario researchers are engaged in international collaborations in these areas.

Methodology

A series of keywords were defined for each of the five research areas (the specific terms used are outlined at the end of this appendix). These keywords were then inputted into SciVal, an online bibliometric tool that is integrated with the Scopus database.^{footnote 29[29]}

Both Scopus and SciVal are products produced by Elsevier publishing. Detailed information regarding the data coverage of SciVal/Scopus can be found here: elsevier.com/__data/assets/pdf_file/0020/53327/scival-metrics-guidebook-v1_01-february2014.pdf.

A series of metrics regarding scholarly output, citations and international collaborations were utilized for the analysis:

Publications

This metric covers all articles, review papers and conference papers produced by researchers in the five emerging fields. It should be noted that the number of publications can vary by discipline so comparisons across research areas is not recommended.

Citations

This represents the total number of times an article is cited by other researchers and can serve as a broad indicator of the influence that researchers in a specific field from a particular jurisdiction are having on the wider academic research efforts in a given field. To be conservative self-citations were excluded from the results presented in this appendix.

Citations per publication

This measures the total number of citations divided by the total number of publications produced in a given field. It can serve as a broad indicator of research impact that controls for the volume of publications produced.

Top cited publications

This metric reports upon the number of publications from a particular jurisdiction which are among the most cited (top 10%) of publications in a specific area of research. This measure can serve to highlight the most highly regarded papers being produced in a jurisdiction. The results for this metric are field-weighted to take into account differences in publication rates.

International collaborations

This metric reports upon the percentage of publications involving a Canadian or Ontario researcher which also involved an international co-author. This can serve as an indicator of both the level of importance attached to Ontario-based research by international researchers and the ability of Ontario researchers to tap into global academic networks.

The Canadian and Ontario results reported in the Findings section were constructed as follows. After running the search terms against the list of all publications available through SciVal a two stage filtering process was carried out. First the results were filtered to look for any papers featuring a Canadian author. This generated lists of publications which included not only Canadian institutions but also foreign institutions since many papers involved international co-authors. Thus a second institutional filter was introduced which only selected Canadian institutions or Ontario institutions as the case need be to try to eliminate the double counting caused by the presence of the foreign institutions. An element of double-counting will still be present, however, due to potential collaborations between Canadian based researchers.

The results presented below should be treated as a quick, initial cut at a more fulsome bibliometric analysis. A more comprehensive approach would move beyond the use of keywords to the use of some of the pre-defined subject area categories available through SciVal.

Findings

Table Appendix C1 provides an overview of where Ontario stands, both nationally and globally, in terms of the quantity and quality of research being conducted within the province in five emerging fields – artificial intelligence, fuel cells, quantum computing, quantum science and regenerative medicine.

Table Appendix C1: Ontario research performance in select emerging technology fields, 2011 to 2017 ^{footnote 30[30]} , ^{footnote 31[31]}

Alt-text Description: Table Appendix C1 provides a breakout of research generated by Ontario researchers in five emerging fields, artificial intelligence, fuel cells, quantum computing, quantum science and regenerative medicine. Ontario’s results are compared against the Canadian national and global totals in terms of publications, citations, citation per publication, top cited publications and international collaborations.

On a national basis Ontario researchers were involved in producing approximately half of all Canadian publications in the areas of fuel cells, quantum computing and regenerative medicine showing Ontario’s particular strength in these areas. Ontario researchers involvement in the national production of articles in artificial intelligence (39%) and quantum science (41%) is on par with the provinces overall share of the nation’s population and economy.

Metrics on citation rates also suggests the research being conducted in these emerging fields within the province is regarded highly by other scholars. In terms of citations per publication Ontario scores higher than the global average across all fields and leads nationally in four of the five areas examined. Additionally when one examines the share of all Ontario publications which rank among the top 10% of most cited papers one finds that Ontario surpasses the global equivalent measure in all of these fields and outperforms the national scores in four of the five areas.

Ontario researchers also demonstrate a strong willingness to engage in international collaborations in these emerging fields. Researchers in the areas of quantum computing and quantum science particularly stand out with 68% of researchers in quantum computing and 63% engaged in quantum science having produced publications in collaboration with international partners

Keyword Searches

The following keyword text searchers were used to generate the publications total in SciVal. As noted above this is an initial bibliometric analysis and could be followed up with a more comprehensive analysis.

• Artificial Intelligence: Text search “machine learning” OR “artificial intelligen*” OR “neural net*”
• Fuel cells: Text search “Fuel cell*”
• Quantum computing: Text search “quantum comput*” OR “qubit”
• Quantum Science: Text search “quantum comput*” OR “qubit” OR “quantum crypt*” OR “quantum informat*” OR “quantum communic*” OR “quantum key” OR “quantum security” OR “quantum dot*” OR “quantum photo*” OR “quantum entangle*” OR “photon entangle*” OR “quantum superposit*” OR “quantum teleport*” OR “quantum metro*” OR “quantum squeez*” OR “quantum control” OR “quantum device” OR “quantum measure*”
• Regenerative Medicine: Text search “regenerative medicine” OR “tissue engineering” OR “cell therapy”