Handbook for Regulatory Proposals: Performance Measurement and Evaluation Plan

We have archived this page and will not be updating it.

You can use it for research or reference. Consult our Cabinet Directive on Regulations: Policies, guidance and tools web page for the policy instruments and guidance in effect.

Introduction

This handbook outlines the purpose of a Performance Measurement and Evaluation Plan (PMEP) for regulatory activities and provides guidance for its development and for completing the PMEP Template. The handbook supports the implementation of the Cabinet Directive on Streamlining Regulation (CDSR).

The intended users of this handbook are government officials who need to develop and implement a PMEP for their regulatory proposals as well as the analysts in the Regulatory Affairs Sector of the Treasury Board of Canada Secretariat (Secretariat) who perform a challenge function.

Throughout the handbook, regulatory activities are understood to mean the regulation(s), the regulatory program, and the regulatory program's related activities, such as communications, inspection, and enforcement.

What are the performance measurement requirements under the Cabinet Directive on Streamlining Regulation?

The requirements for carrying out performance measurement for regulatory activities are outlined in Section 4.6 of the CDSR, "Measuring, evaluating, and reviewing regulation." They are also outlined in the Regulatory Impact Analysis Statement (RIAS) Template.

What is the purpose of a Performance Measurement and Evaluation Plan?

The purpose of a PMEP is to ensure that regulatory activities continue to meet their initial policy objectives and are accordingly renewed on an ongoing basis. A PMEP provides a concise statement or road map to plan, monitor, evaluate, and report on results throughout the regulatory life cycle. When implemented, it helps a regulator:

• ensure a clear and logical design that ties resources and activities to expected results;
• describe the roles and responsibilities of the main players involved in the regulatory proposal;
• make sound judgments on how to improve performance on an ongoing basis;
• demonstrate accountability and benefits to Canadians;
• ensure reliable and timely information is available to decision makers in the regulatory organizations and central agencies as well as to Canadians; and
• ensure that the information gathered will effectively support an evaluation.

When is a Performance Measurement and Evaluation Plan required?

Before submitting a regulatory proposal, departments and agencies are expected to conduct an assessment, which is based on the  and performed in collaboration with the Secretariat's Regulatory Affairs Sector, to determine the level of impact (Low, Medium, or High) of the proposed regulation.

Completion of a PMEP Template is required when the answer to one or more of questions 1 through 6 in the Triage Statement is "High." For regulatory proposals of Medium impact, completing a PMEP Template is optional and left to the discretion of the regulatory organization.

Where are the regulatory activities situated in the department's Program Activity Architecture and Performance Measurement Framework?

When developing the PMEP, it is important to ask the following question: How does this PMEP fit into the departmental Management, Resources, and Results Structures (MRRS), specifically the Program Activity Architecture (PAA) and the Performance Measurement Framework (PMF)?

Regulatory activities would be represented, when appropriate, in a departmental PAA and supporting PMF as a program's lowest level component, i.e., a program's sub-subactivity level (see graphic under Linkage to the Program Activity Architecture). Where regulatory activities are significant to understanding why and how funds are being spent to achieve a program's stated expected results, they must be mentioned in the program description unless a compelling rationale for omitting them is provided. Program managers will need to consider how performance indicators supporting regulatory activities factor into a program's PMF. For example, regulatory activity indicators may support a program output tracked in the PMF or a program's expected result. Managers are strongly encouraged to consult with key corporate groups (e.g., heads of Evaluation, program performance measurement teams or units) to determine the most appropriate way to align performance indicators within the MRRS.

To ensure the departmental PAA and PMF reflect a new PMEP, they should be updated in accordance with the next scheduled review of the PAA and PMF, but only after the regulation has been published in Canada Gazette, Part II.

How to complete a Performance Measurement and Evaluation Plan Template

This section contains a description of the nine core components of the PMEP Template (see Appendix A). Definitions related to performance measurement can be found in the Results-Based Management Lexicon. Information from the PMEP Template is carried forward into the Performance measurement and evaluation section of the RIAS. If the regulatory proposal involves a Treasury Board submission, the PMEP should be consistent with that submission (see A Guide to Preparing Treasury Board Submissions).

Description and Overview of the Regulatory Proposal

The introductory component of a PMEP should provide a thorough description of the problems and risks that the regulatory proposal aims to address, as identified in the Triage Statement. This section should also specify the proposed regulation's target audience, the intended beneficiaries, and the behavioural changes it seeks to bring about among specific groups. This element of the PMEP Template is linked to the Issue, Objectives, and Description sections of the RIAS.

The questions to be answered are as follows:

• What is the issue the regulatory proposal aims to address?
  • What evidence exists to show that there is an issue?
  • Why is the issue important?
• In concrete terms, what are the objectives of the regulatory proposal?
• How will the regulatory proposal achieve the objectives for solving or mitigating the issue?
• Who are the target audiences (i.e., regulated individuals and organizations) of the proposed regulation?
• Who are the intended beneficiaries of the proposed regulation (e.g., Canadian public, specific groups within the Canadian public such as children under the age of four)?
• What behavioural changes in the target audience need to be addressed (e.g., awareness, understanding, capacity, compliance)?

This section should also describe how the proposed regulation fits into the bigger picture (i.e., how it contributes to the regulatory program, the department's strategic outcomes (PAA), other overarching initiatives). Furthermore, identify if and how the regulatory activities cut across multiple regulatory organizations. Finally, indicate how the information summarized and reported will be used to improve the performance of the regulatory activities.

Logic Model

A logic model, which is composed of a graphic and accompanying text, tells the story of regulatory activities. It connects the inputs (resources) and activities (what one does) to the outputs (products or services generated from the activities), the groups reached (regulated parties or beneficiaries), and the expected outcomes of the initiative (the sequence of changes among groups outside the control of the regulatory organization). In its simplest form, the logic model is composed of the following five logically interrelated components.

• Inputs: The human and financial resources used to undertake the regulatory activities and consequently produce outputs (i.e., services and products). Inputs include personnel, physical facilities, equipment, materials, and funding. In some cases, they include the legislative or regulatory authority necessary to undertake regulatory activities.
• Activities: Actions that the department or agency undertakes to produce its outputs. For instance, inspection and licensing are two regulatory activities that are commonly found in regulatory organizations.
• Outputs: The products or services produced by regulatory activities. Outputs are deliverables wholly under the control of an organization. The results that occur beyond outputs are not within the full control of the regulatory organization.
• Target groups (reach): The individuals, groups, or organizations that the regulatory activity is intended to reach and influence. This includes both regulatees (i.e., those subject to regulations) and other key groups who are important to the success of the initiative (e.g., public, private, and not-for-profit organizations, other institutions, and individual Canadians, among others).
• Outcomes: Results attributable to a regulatory organization. Outcomes are not the direct result of a single regulatory activity; rather, they are affected by what the organization does. Outcomes are further qualified as immediate (also known as direct), intermediate, or ultimate and are linked to the target groups that the regulatory organization is trying to influence.

The problems or risks identified in Section 1 of the PMEP Template help to define the inputs, activities, outputs, and outcomes stated in the logic model. See Appendix B for more on how risks and problems set the vital context for results (this is also covered in a course offered at the Canada School of Public Service).

In developing the logic of the regulatory proposal, organizations can draw on other departmental documents related to MRRS. Where possible, the logic model for the regulatory activity should be linked to related programs and the strategic outcomes of the regulatory organization.

Logic models should be reviewed and validated in conjunction with the appropriate department or agency personnel to confirm the accuracy of the program logic and to facilitate buy-in among those who will be involved in implementing and maintaining the ongoing performance measurement monitoring system. In the spirit of ongoing consultation and engagement with stakeholders, departments and agencies are encouraged to involve, when appropriate, external stakeholders with an interest in the regulatory activity.

Indicators

An indicator is a quantitative or qualitative means of gauging an initiative's performance or the progress made toward its expected desired results. Indicators operationally describe the intended output or outcome one is seeking to achieve over time.

Indicators should be developed from the logic model in the PMEP Template. Indicators need to be prioritized and limited in number when selected for monitoring. It is more effective to measure the critical few rather than the trivial many. A small set of highly meaningful indicators need to be specified to track overall performance with respect to the intended outcomes and policy objectives of the regulatory proposal (e.g., health, safety, security, environmental protection, business and trade, Aboriginal prosperity). Where possible, these indicators should be consistent with, and ideally support, indicators found in the department's PMF.

Indicators should be expressed as numerical forms when possible (e.g., as raw numbers, averages, percentages, rates, ratios, or indexes). Where qualitative indicators are used, they should be objectively verifiable. Certain criteria should be kept in mind when selecting indicators. Indicators should be:

• Relevant and valid;
• Prioritized and limited in number;
• Balanced and comprehensive;
• Meaningful and understandable;
• Timely and actionable; and
• Cost-effective to measure.