Introduction

How can young people who may need residential services, their families and helping professionals who work with them find out what kinds of options are available and might best meet their needs? How do service providers, the Ministry and the public know how well young people in residential care are doing? What types of planning activities are completed to ensure that high quality residential services are being delivered in an optimal fashion on the basis of the best available evidence? These are the kinds of information and data access questions that are fundamental to the design of any public service delivery system in order to: (1) inform choice and facilitate access to services; (2) monitor service trends and outcomes; and (3) support innovation and implementation of quality of care enhancements and evidence based practices. Access to such information is particularly important in the context of a highly decentralized service system that includes a mix of over 600 locally operated transfer payment agencies, directly operated residential services, and private for-profit and not-for-profit residential service organizations. To build effectively on the strengths of such a diversified service delivery model, it is essential that service users, operators, funders and oversight bodies have access to accurate, relevant and timely information.

Every review of residential services that has been conducted in Ontario has cited the lack of information and data as core concerns. In a 2003 report, the Ombudsman’s investigation revealed that “the Ministry did not have the necessary data to decide what level of residential service was required in Ontario for children with special needs and had no set timetable for consideration of this issue” (Ombudsman Ontario, 2003, p.17). The 2006 Children and Youth Residential Services Review conducted by the Bay Consulting Group concluded that “there is a lack of consistent, centralized information for planning and managing the overall system and for monitoring, evaluating and improving system performance and outcomes” (Bay Consulting Group, 2006, p. 86). In the 2012 My Real Life Book: Report from the Youth Leaving Care Hearings, it was recommended that the Ministry “commit to collecting and publishing information on how children and youth in care are doing” (Provincial Advocate for Children and Youth, 2012, p. 32). In its 2013 Blueprint for Fundamental Change, the Ministry Youth Leaving Care Working Group stressed the critical importance of tracking “outcome data about children and youth in and from care” (MCYS, 2013, p. 20). Most recently, the 2015 Auditor General review found that despite a clear need to better assess outcomes following child welfare services, the Ministry “does not have sufficient information to monitor the performance of the Child Protection Services Program,” further noting that the data collected for the newly established child welfare performance indicators “is not sufficient to adequately monitor and assess” performance (Office of the Auditor General of Ontario, 2015, p. 147, p. 150).

In addition to considering the data and information issues raised in previous reports and echoed further in many of our consultations, the Panel reviewed a range of government, agency, association and researcher reports describing residential services and service outcomes, and examined various data collection systems in place across different service delivery and monitoring systems. The Panel also examined several information tracking, outcome reporting systems and research and analysis infrastructures in other jurisdictions in Canada and internationally.

While many of the information and data collection gaps identified in previous reviews were confirmed in our own review, we also came across some very promising initiatives and were generally struck by the rich potential of many existing sources of data. We have therefore tried to focus our review of information and data issues beyond simply identifying gaps but also examining data access, sharing, and analysis issues. In a sector where staff and, to a lesser extent, youth already spend a lot of time documenting, assessing, and responding to questionnaires and information requests, our approach has been to understand whether existing data and information are being used in an optimal fashion, while also identifying important information gaps that may require new forms of data collection.

Issues

Access to information

There are over 600 residential service providers across the province but there is currently no province-wide mechanism for potential users and placing agencies to get information about those services. The Ministry directly operates residential services (e.g., certain youth justice custody and detention centres and the Child Parent Resource Institute), while it also transfers payments to community agencies such as children’s aid societies to deliver residential and other services. Some transfer payment agencies (TPAs) responsible for placing children in residential care operate their own internal residential resources, whereas others utilize outside paid resources (OPRs) to provide residential services. Adding to the complexity of the service system, some community agencies use a mix of internal and external residential services.

Children’s aid societies are transfer payment agencies that are responsible for caring for children and youth in need of protection. The Panel heard about the challenges that children’s aid societies face in locating suitable residential services and ensuring that the services are high quality and well-suited to the specific and unique needs of individual children and youth. This was particularly difficult when the children’s aid society relied predominantly on outside paid resource residential services. In some cases, children’s aid societies have sought to develop efficiencies by sharing information about the quality of external residential services. For instance, children’s aid societies in the Greater Toronto Area have developed and implemented a shared assessment tool for OPR residential service providers. This tool allows for information sharing regarding the OPR program philosophy, client groups served, staffing levels, serious occurrences, and outcomes for children and youth following services. Other children’s aid societies share quality assurance managers, which may facilitate sharing best practices for analyzing data about children in care.

In addition to transferring payments to child welfare agencies, MCYS currently provides transfer payments to more than 400 agencies that provide services related to children and youth mental health, some of which provide residential services. Children, youth and their caregivers can access Ministry-funded mental health and complex special needs services by contacting service providers directly or receiving a referral from a school or other health or social service professional. Access to residential services for young people with mental health needs is managed through community tables where children and youth mental health service providers, local child welfare agencies, and other human service organizations are represented, including local hospitals. In the GTA, for example, this mechanism is referred to as Centralized Access to Residential Services (CARS). As with most such community tables managing a centralized access point for mental health residential services, referral to CARS can only be made through professionals and human service organizations, including schools. Outside of that process, there is no list describing mental health service providers that offer residential programs along with detailed information about the type and quality of care offered. Information about mental health service providers is expected to be more readily available through the work of lead agencies and the recently updated Health Care Options website (https://www.ontario.ca/ locations/health/index.php?lang=en), however these resources provide general information and do not specifically assist families in accessing residential services.

For young people with complex special needs, the Ministry funds transfer payment agencies to coordinate services and in some instances provide respite. A complex system to navigate, we heard that parents have tremendous difficulty locating residential services, and some parents have resorted to accessing protection services through children’s aid societies even when no child protection concerns were present but rather the issue was a lack of access to more appropriate residential services. This issue was the focus of a 2005 Ombudsman report and was discussed in our consultations 10 years later (Ombudsman Ontario, 2005). Like in the child welfare and children and youth mental health sectors, there is a lack of information available to children and youth with special needs and their families about residential service options.

The Ministry’s 2007 Child and Youth Residential Services Action Plan committed to improving access to information on residential services by developing a public website designed “to provide agencies, families, and ministry staff with a single, integrated database of reliable information” (MCYS, 2007, p.4). Following the 2007 Action Plan, the Ministry started to develop a province-wide public website that would provide “one-stop access to information about specific residential programs,” but unfortunately this website is not operational (MCYS, 2007, p.4). Families and service providers can access information about respite services provided by agencies funded by the Ministry of Community and Social Services and the Ministry of Children and Youth Services through a central website (respiteservices.com), which is a step in the right direction.

Limits to access to information about residential services are partly a result of the differences in routes for accessing these services. Access to residential services in Ontario is determined largely on the basis of how the need for out-of­home placement is identified (child welfare, youth justice, mental health, complex special needs), the region a young person lives in, and the funding mechanisms in place (direct operated facilities, transfer payment agencies, and outside paid resources). Some resources can be accessed through multiple sources. For instance, parents, child welfare agencies and even out-of-province organizations can access OPR services directly, as long as they have the capacity to cover the costs. Others, such as services run by child welfare agencies or youth justice, are typically only accessed through those systems. As such, each access point maintains its own list of resources. Differences in mandates, funding models and governance structures in many instances explain why access to some residential services is restricted; youth justice custody/detention facilities, for instance, are designed to be accessed through youth justice, which includes a court order. These restrictions, however, do not mean that access to information about the nature, type and quality of residential services in the province cannot be made more readily available to potential users of those services.

Access to services and access to information about services have been identified as important issues needing to be addressed in several previous reports. Throughout our consultations access was also identified as a critical issue, especially with respect to lack of placement resources in some jurisdictions or lack of options for young people with complex special needs. Access to information about resources supports a transparent consumer focused model that gives young people and their families more control over the difficult decisions that often need to be made in placing young people in out-of-home care. While access to information does not necessarily resolve lack of capacity and resources, easier access to information about the full provincial network of service providers can help (1) increase access to resources that service users would otherwise not be aware of, (2) identify service gaps or duplications to support more efficient resource planning, and (3) identify barriers to accessing underutilized services.

The need for information goes beyond simply knowing which residential services exist in the province. Service users would benefit immensely from having access to detailed information about the programming offered within various residential services, the target client group, the certifications of staff and management, and previous client experiences and outcomes. This would allow for service users to understand the quality and specialization of various residential service providers, and could increase the probability of locating a successful match between the child or youth and the residential service. Through our consultations, we learned of the challenges that families and community agencies face in finding residential resources that adequately match the unique needs of children and youth.

Monitoring trends and outcomes

Effective oversight of the over 600 residential service providers across Ontario requires both the capacity (1) to ensure that every individual service provider meets provincial standards for quality care (see Quality of Care) and (2) to track service trends and monitor outcomes to determine at the aggregate level whether residential services are effectively supporting young people. While the youth justice sector and some service providers are reporting information that provides a sense of service trajectories and outcomes, it is not possible at this time to determine the extent to which the current design and delivery system is effectively meeting the needs of children and youth in residential care in Ontario.

The province has no easily accessible and reliable information to answer the most basic questions, for example: What is the age and sex distribution of young people entering residential care? Are there any trends in admissions and discharges broken down by Aboriginal status? What is the average number of placement changes young people experience while in care? Where do young people go when they leave residential care? How well do children and youth in care do on educational and other outcomes? Even where data is collected in these areas, it is not effectively managed and analyzed to provide insight into these questions.

Answering these and other questions requires a comprehensive and integrated approach to collecting data over time from the child welfare, children and youth mental health, youth justice, and complex special needs sectors. It also involves examining existing sources of data to identify how to maximize the utility of these sources by creating user-friendly databases that can be easily analyzed to provide both simple descriptive outputs as well as more complex longitudinal and multilevel analyses. This exercise of transforming existing data sources into datasets rich with information must occur in the context of collaboration across sectors, in order to move toward comparable cross-sector approaches to asking and answering critical questions about children and youth in residential care as well as the quality of care provided by residential services.

Existing reporting on service trends, service provider quality, and child and youth outcomes in Ontario is limited. Available data on children in residential care is typically presented as point-in-time snapshots, with little information on trajectories over time. Trends data, where available, only represent trends over a few years making it very difficult to assess the impact of incremental changes. Data is rarely representative of the population of children living in residential services across sectors and there is no way to track young people between sectors. Random sampling techniques, which can maximize representativeness while limiting the costs and time associated with population-based research, are not commonly employed in the child welfare, youth justice, children and youth mental health, and complex special needs sectors. Finally, issues of the reliability and validity of data are rarely discussed in reports on residential services, and findings are rarely presented with a clear explanation of the research methods and the limitations of those methods.

The province does not systematically collect important information in order to monitor the safety and well-being of young people receiving residential services. MCYS and residential service providers also do not have a method of consistently and publicly reporting on other important issues for children and youth in care, treatment, and custody/ detention, including schooling, friends, skills, significant adults and relationships, and satisfaction with services. Even basic demographic and service data that exist are not being fully accessed and analysed.

Promising sources of data

There are a growing number of promising sources of clinical data and potentially useful administrative data available in Ontario. A collection of these data sources is listed below with a short description and consideration of the strengths and limitations of each.

  • The Ontario Association of Residences Treating Youth (OARTY) reports on clinical data from approximately one quarter of children served by member agencies. The OARTY database includes numerous outcome measures for children and youth that are measured at various time points. However, the specific research methods employed by OARTY are not described in detail, and it is unclear whether outcome measures are collected on a representative sample of children and youth or instead a convenience sample. Further, there is no third-party validation of the outcomes reported by OARTY, which is concerning given that the reports on outcomes are used as marketing tools.
  • The Ontario Looking After Children (OnLAC) project uses an approach aimed at raising the standard of care for children receiving residential services. It focuses on seven key developmental areas for children: health, education, identity, family, social presentation, emotional and behavioural development, and self-care skills. OnLAC involves collecting information from young people in out-of-home care in these key areas using the Assessment and Action Record (AAR) data collection instrument. This review is conducted for all young people who are in child welfare care for a continuous 12 months. The Panel heard that the AAR questionnaire is very lengthy, which led to poor response rates and a feeling of burden among professionals, caregivers, and young people who completed them. The Panel also heard anecdotal evidence indicating that appropriate data collection procedures are rarely followed.
  • The Ontario Child Abuse and Neglect Data System (OCANDS) is a provincial database composed of administrative data from child welfare agencies. The database is child-specific, event-level, and longitudinal. It is designed to track children from initial report through termination of services. It is currently used to generate data for three of Ontario’s five publicly reported performance indicators. At present, 44 of 47 children’s aid societies have signed on to participate. However, the database requires complex programming in order to conduct even basic analysis.
  • The Crown Ward Review (CWR) is an annual legislated review conducted at CASs across Ontario. A comprehensive file review is completed and young people are provided an opportunity to complete a questionnaire and interview. There is generally a low response rate for questionnaires and interviews, and the review is limited to only those children who have been crown wards for 24 consecutive months. The Panel heard in our consultations that the data produced through the Crown Ward Review is cumbersome and difficult to analyze.
  • Serious Occurrence Report (SOR) forms are completed if there is a death, serious injury, physical restraint, missing person, abuse/neglect, complaint, or disaster in a residential service funded by the province. MCYS has developed a new tool that aims to enhance the capacity to examine SORs from across the province. However, the data collection process for SORs is outdated and impedes the ability for province-wide analyses. Residential service operators are required to fax SORs and then MCYS regional offices must manually enter the data to send to MCYS. It is an inefficient and burdensome data collection and management process.
  • MCYS developed the Child Protection Information Network (CPIN) to enable timely sharing of critical child protection information among children’s aid societies. The Auditor General recently reported that “CPIN implementation has suffered significant cost overruns and delays due to poor project planning” (Office of the Auditor General of Ontario, 2015, p. 161). The Auditor General also found that other jurisdictions using the same case management software have experienced positive results. CPIN has been implemented in only five of Ontario’s 47 children’s aid societies and six additional CASs are planning on implementing CPIN in 2016. While CPIN holds great potential for collecting and analyzing data about children and youth in the care of a children’s aid society, it does not appear that the Ministry has built infrastructure to ensure this occurs. In addition, there are issues of data ownership that will need to be resolved in order for the province to meaningfully analyze trends in the characteristics and outcomes of young people in care at the aggregate level. At present, children’s aid societies who have implemented CPIN have agreed that while the information system belongs to MCYS, case-level data belongs to the agency.

Limited analytic capacity

There is an enormous amount of fragmented information being collected about children and youth and their placements, but no one has either the mandate or the analytic infrastructure to synthesize the information. Each data collection initiative appears to be conducted independently without coordination with other initiatives. Further, there has been no effort to develop a systematic approach to research and analysis that reduces duplication, maximizes efficiency, and ensures that the data collected is of the highest quality possible.

Innovation and evidence-based practices

While the capacity to monitor service trends and outcomes is the cornerstone of any effective service delivery system, not all questions are appropriately addressed by rolling up information collected on every service and service user. Evaluation research following selected samples of service users can produce rich and detailed qualitative and quantitative information that can guide the development of innovative approaches and support the implementation of evidence based programs. There is a lack of independent analysis of available service data. Information is usually provided by, analysed and presented by the service providers themselves.

Residential services in Ontario do not consistently utilize evidence-based practices and even when such practices are used, there seems to be limited tracking of long-term impact. There are several examples of research initiatives that use high quality approaches to data collection and longer term follow-up, such as the clinical trials of multisystemic therapy (MST) conducted as a collaborative effort of children’s services in Ottawa, Simcoe County, Mississauga, and London (Leschied & Cunningham, 1999). More collaborative high quality evaluations are needed to assess the impact of residential services on outcomes for children and youth.

Jurisdictional review

A number of models are available from different jurisdictions, ranging from specialized within government units, to arms-length government agencies, to university based research centres. Three examples are described in detail below.

  • Centre for Excellence for Looked After Children in Scotland (CELCIS) is an organization that conducts research, influences policy, and engages in consultations to improve service delivery. It is primarily funded and monitored by the Scottish Government and also supported by the University of Strathclyde. CELCIS is guided by a Strategic Steering Group (including academics, social workers, and administrators from the field), which provides a connection between CELCIS and University of Strathclyde and develops strategic directions. CELCIS research teams provide specialist support to organizations interested in conducting research and is affiliated with various academic research projects. CELCIS also conducts evaluations. CELCIS produces briefings, comments on government actions, summaries of research, summaries of promising practices around the globe, and reports, and these are published online.
  • California Child Welfare Indicators Project (CCWIP) is a collaborative venture between University of California at Berkeley and the California Department of Social Services. The project is housed at the School of Social Welfare and base funding is provided by the state. Data in California are collected at the county level and every county uses the same information system. Aggregate data are made publicly available through CCWIP and refreshed quarterly. Berkeley provides technical assistance and support to counties and the state to understand the performance of child welfare agencies. Examples of performance indicators measured include: maltreatment in foster care; permanency; placement stability; timely caseworker visits and health/dental exams; exit outcomes for youth aging out of care.
  • The Center for State Child Welfare Data is housed at Chapin Hall at the University of Chicago and represents a partnership between state child welfare agencies, Chapin Hall, the American Public Human Services Association, and the University of California at Berkeley. The mission of the Center is to use research and technical assistance to promote evidence-based decision making in child welfare. The Center generates knowledge on key issues in contemporary child welfare practice. The cornerstone of the Centre’s research program is the Multistate Foster Care Data Archive, a longitudinal data warehouse developed and maintained by the Center that contains decades of state data on approximately 3 million children in over two dozen states who have spent time in foster care. Technical assistance is provided to member agencies to analyze data, and agencies from states across the U.S. can be members. The Center also offers formal training opportunities.

Implications for recommendations

An effective province-wide cross-sectoral information and data management system is essential to ensure that Ontario is able to offer young people in residential care the quality of services that they need. Such a system should include (1) a clearinghouse which provides young people, their parents and professionals working with them with information about the full range of placement options that are available to them; (2) a data repository that can be used to monitor services, analyze service trends and track outcomes; (3) a research and evaluation strategy that supports innovation and implementation of evidence-based practices.

Existing Information and Data Collection Systems

Although there currently is very little province-wide information being reported about young people in residential care, the Ministry collects an enormous amount of data about these services. Crown Ward reviews, licensing reports and serious occurrence reports are rich under-utilized sources of data. The Ministry has already developed a province-wide Youth Justice information system (Youth Offender Tracking and Information System). The Ministry is also investing significant resources into the development of CPIN, the province-wide child welfare information management system. CPIN tracks many key indicators that could be used to monitor services, analyze trends and track outcomes. The placement module in CPIN can be expanded to include information about all residential service providers. The deployment of CPIN has been slower than expected and in several of our consultations concerns were raised about its implementation. It is also important to note that CPIN has limited capacity to allow for complex analyses to be conducted with data from pre-existing systems. To analyze long term trends, analysts will need to access legacy systems; OCANDS has the capacity to be used to tap into those legacy systems.

Service providers also collect a range of important information: the Assessment and Action Records used by child welfare agencies through the OnLAC initiative and the Partners in Care project conducted by OARTY include a rich amount of information about young people’s relationships, education and well-being that can be used to help understand service needs and outcomes. As discussed, these data collection efforts have serious limitations alongside their strengths. In reviewing these various tools we considered the merits of selecting a single assessment tool to be used across all service providers. While a common tool would have some advantages, such as simplifying referral procedures between service providers, and facilitating comparisons of services and outcomes between different service providers, we were concerned that imposing a single assessment tool could (1) undermine clinical approaches that integrate existing tools, (2) be poorly completed by staff who may fail to buy into a Ministry imposed tool, and (3) risk moving the Ministry’s role from oversight to administration. These tools nevertheless provide helpful information that include indicators of key dimensions that can be compared across different assessment platforms and analyzed at the aggregate level. MCYS does not intend to mandate the use of any specific assessment tool and the Panel found this to be appropriate.

User satisfaction ratings are one area where new data collection systems may be required. Other jurisdictions, including British Columbia, collect client satisfaction information following the provision of services to children and youth. In Ontario, informal reviews for certain residential services such as shelters are sometimes posted in online forums such as Google Reviews, but there has been no systematic collection of user satisfaction from young people who have experienced residential services. The exception is youth justice, where youth feedback is gathered through a youth experience survey completed with the probation officer following discharge from custody. A parent/guardian experience survey is also used.

Legal and ethical considerations

Opening up of access to information about services must be balanced against a range of privacy and safety considerations. Accessing and sharing information about residential service providers was identified as a major challenge in many of our consultations:

  • Some placement agencies and licensing staff said that they did not believe they could share information about service providers because of uncertainty about the legality of sharing information and concern about law suits. We were troubled to hear of several examples of placement staff who said that there were some resources that they would not use because of concerns about quality of care, but they were not able to share these concerns with other placement agencies.
  • Some service providers raised concerns about the framework and conditions for sharing and protecting what they perceived to be private information about their services.
  • Privacy and safety considerations, given the difficult and at times very conflictual circumstances leading to young people coming into care, are also important factors to take into consideration.
  • There is a limited tradition of sharing information between child welfare, youth justice and mental health sectors and no unique identifier which would support tracking residential services across these sectors. Issues with respect to accessing education information will also need to be examined.

Some of these issues were examined through the Ministry’s recently completed review of the CFSA. Given the Ministry’s role in funding residential services and its licensing function, it will be important to ensure that the Ministry is able to access information needed by the proposed Quality Inspectorate to meet its function. A framework will also need to be developed to determine what types of information can be shared at what level of aggregation and with whom.

Data infrastructure needs

Given the wealth of existing data and the significant investments that are being made to develop common information in the child welfare sector, we do not think that a new residential services information system is the appropriate way to go. Initiatives like OCANDS demonstrate how programming and data analytics can be used to effectively combine information across different platforms. While some information collection systems require some enhancements – Serious Occurrence Reports are currently faxed into regional offices, Crown Ward Review data are provided to children’s aid societies in spreadsheets, CPIN has limited cross-sector information sharing capacity and there is no comprehensive strategy for using the data to inform policy – we have concluded that a focused investment in developing programming and data analysis capacity would be more effective and expedient than creating new data collection systems dedicated to residential care.

In addition to developing the analytic infrastructure for making better use of existing data, a commitment to a standard and comprehensive set of public reports is a powerful mechanism for ensuring that data are used and for creating incentives for improving data quality. The Ministry is currently reporting on several child welfare indicators and on important trends in Youth Justice. These initiatives could be significantly expanded by including a broader array of indicators. In Chapter 10, we review many of the indicators used in other jurisdictions and present a framework for selecting key indicators that should be reported on an annual basis. We also discuss some of the analytic issues that have to be addressed before reporting publicly on meaningful data comparing service providers.

One of the information and data challenges that will need to be addressed will be finding effective methods to track young people who experience placements across different sectors. The sector-specific information systems that are currently in place fail to provide a good mechanism for tracking and understanding the placement trajectories of these “cross-over” youth. Given that these are some of the most vulnerable young people in residential care, it is critical that enhancements made to existing information systems provide the Ministry with the ability to track and analyze the service trajectories and outcomes for these young people.

Along with enhancing the Ministry’s data analytic capacity, service providers and independent researchers also need to be supported to make better use of the data that they produce. Independent researchers should be encouraged to access non-identifying data to conduct more in depth analyses of these administrative datasets. In addition, research studies based on selected samples of young people in residential care are important tools to generate more nuanced and in-depth understanding of the experiences of young people and the outcomes of services.