Changes

16 bytes removed , 20:01, 3 October 2021

no edit summary

Line 4: Line 4:

<span class="plainlinks">[[Manuel des projets de réglementation : Plan de mesure du rendement et d'évaluation|Français]]</span>

</div>

+

Line 17: Line 20:

Throughout the handbook, regulatory activities are understood to mean the regulation(s), the regulatory program, and the regulatory program's related activities, such as communications, inspection, and enforcement.

−

== What are the performance measurement requirements under the ''Cabinet Directive on Streamlining Regulation''?==

+

==What are the performance measurement requirements under the ''Cabinet Directive on Streamlining Regulation''?==

The requirements for carrying out performance measurement for regulatory activities are outlined in Section 4.6 of the CDSR, "Measuring, evaluating, and reviewing regulation." They are also outlined in the Regulatory Impact Analysis Statement (RIAS) Template.

−

==What is the purpose of a Performance Measurement and Evaluation Plan? ==

+

==What is the purpose of a Performance Measurement and Evaluation Plan?==

The purpose of a PMEP is to ensure that regulatory activities continue to meet their initial policy objectives and are accordingly renewed on an ongoing basis. A PMEP provides a concise statement or road map to plan, monitor, evaluate, and report on results throughout the regulatory life cycle. When implemented, it helps a regulator:

Line 30: Line 33:

*ensure that the information gathered will effectively support an evaluation.

−

== When is a Performance Measurement and Evaluation Plan required?==

+

==When is a Performance Measurement and Evaluation Plan required?==

Before submitting a regulatory proposal, departments and agencies are expected to conduct an assessment, which is based on the and performed in collaboration with the Secretariat's Regulatory Affairs Sector, to determine the level of impact (Low, Medium, or High) of the proposed regulation.

Line 42: Line 45:

To ensure the departmental PAA and PMF reflect a new PMEP, they should be updated in accordance with the next scheduled review of the PAA and PMF, but only after the regulation has been published in ''Canada Gazette'', Part II.

−

==How to complete a Performance Measurement and Evaluation Plan Template==

+

== How to complete a Performance Measurement and Evaluation Plan Template==

This section contains a description of the nine core components of the PMEP Template (see Appendix A). Definitions related to performance measurement can be found in the [https://www.canada.ca/content/canadasite/en/treasury-board-secretariat/services/audit-evaluation/centre-excellence-evaluation/results-based-management-lexicon.html Results-Based Management Lexicon]. Information from the PMEP Template is carried forward into the Performance measurement and evaluation section of the RIAS. If the regulatory proposal involves a Treasury Board submission, the PMEP should be consistent with that submission (see ''[http://www.tbs-sct.gc.ca/pubs_pol/opepubs/TBM_162/gptbs-gppct-eng.asp A Guide to Preparing Treasury Board Submissions]'').

Line 52: Line 55:

*What is the issue the regulatory proposal aims to address?

**What evidence exists to show that there is an issue?

−

**Why is the issue important?

+

** Why is the issue important?

*In concrete terms, what are the objectives of the regulatory proposal?

−

*How will the regulatory proposal achieve the objectives for solving or mitigating the issue?

+

* How will the regulatory proposal achieve the objectives for solving or mitigating the issue?

−

*Who are the target audiences (i.e., regulated individuals and organizations) of the proposed regulation?

+

* Who are the target audiences (i.e., regulated individuals and organizations) of the proposed regulation?

−

*Who are the intended beneficiaries of the proposed regulation (e.g., Canadian public, specific groups within the Canadian public such as children under the age of four)?

+

* Who are the intended beneficiaries of the proposed regulation (e.g., Canadian public, specific groups within the Canadian public such as children under the age of four)?

−

* What behavioural changes in the target audience need to be addressed (e.g., awareness, understanding, capacity, compliance)?

+

*What behavioural changes in the target audience need to be addressed (e.g., awareness, understanding, capacity, compliance)?

This section should also describe how the proposed regulation fits into the bigger picture (i.e., how it contributes to the regulatory program, the department's strategic outcomes (PAA), other overarching initiatives). Furthermore, identify if and how the regulatory activities cut across multiple regulatory organizations. Finally, indicate how the information summarized and reported will be used to improve the performance of the regulatory activities.

−

===Logic Model ===

+

===Logic Model===

A logic model, which is composed of a graphic and accompanying text, tells the story of regulatory activities. It connects the inputs (resources) and activities (what one does) to the outputs (products or services generated from the activities), the groups reached (regulated parties or beneficiaries), and the expected outcomes of the initiative (the sequence of changes among groups outside the control of the regulatory organization). In its simplest form, the logic model is composed of the following five logically interrelated components.

*Inputs: The human and financial resources used to undertake the regulatory activities and consequently produce outputs (i.e., services and products). Inputs include personnel, physical facilities, equipment, materials, and funding. In some cases, they include the legislative or regulatory authority necessary to undertake regulatory activities.

*Activities: Actions that the department or agency undertakes to produce its outputs. For instance, inspection and licensing are two regulatory activities that are commonly found in regulatory organizations.

−

* Outputs: The products or services produced by regulatory activities. Outputs are deliverables wholly under the control of an organization. The results that occur beyond outputs are not within the full control of the regulatory organization.

+

*Outputs: The products or services produced by regulatory activities. Outputs are deliverables wholly under the control of an organization. The results that occur beyond outputs are not within the full control of the regulatory organization.

*Target groups (reach): The individuals, groups, or organizations that the regulatory activity is intended to reach and influence. This includes both regulatees (i.e., those subject to regulations) and other key groups who are important to the success of the initiative (e.g., public, private, and not-for-profit organizations, other institutions, and individual Canadians, among others).

*Outcomes: Results attributable to a regulatory organization. Outcomes are not the direct result of a single regulatory activity; rather, they are affected by what the organization does. Outcomes are further qualified as immediate (also known as direct), intermediate, or ultimate and are linked to the target groups that the regulatory organization is trying to influence.

Line 99: Line 102:

#Suggest the measure of change, such as its number, proportion, percentage, rate, ratio, amount, or level.

#Specify who is changing. This may include the population target group, program participant, client, individual, organization, agency, or community. The more you specifically describe the group who should be changing or from whom you expect a reaction, the more precise the indicator.

−

# Specify what is changing or happening. This may include changes to awareness, attitude, knowledge, skill, aspirations, commitments, behaviours, or practices. It may also include simple reactions such as perception or satisfaction. Note that this may not always be a change per se; sometimes, one is seeking to maintain existing reactions and behaviours.</blockquote>A key principle for developing indicators is that they should always follow the logic of an initiative's expected results. Examples are as follows:

+

#Specify what is changing or happening. This may include changes to awareness, attitude, knowledge, skill, aspirations, commitments, behaviours, or practices. It may also include simple reactions such as perception or satisfaction. Note that this may not always be a change per se; sometimes, one is seeking to maintain existing reactions and behaviours.</blockquote>A key principle for developing indicators is that they should always follow the logic of an initiative's expected results. Examples are as follows:

{| class="wikitable"

!Level

Line 106: Line 109:

|Ultimate Outcomes

The highest level outcome that can be reasonably attributed to a policy, program, or initiative and that is the consequence of one or more intermediate outcomes having been achieved. Such outcomes represent a change of state in a target population.

−

|

+

|

*Level of health-related incidence (e.g., incidents per 100,000 of the population)

−

* Level of environmental degradation (e.g., level of soil erosion, air and water pollution counts)

+

*Level of environmental degradation (e.g., level of soil erosion, air and water pollution counts)

*Level of economic activity or growth (e.g., percentage of change in the GDP, in direct foreign investment, in employment)

|-

Line 123: Line 126:

|

*Percentage of target enterprises that attended information session

−

* Percentage of information (specific report) users stating that they found the information at least somewhat valuable

+

*Percentage of information (specific report) users stating that they found the information at least somewhat valuable

−

* Level (exceeded, met, did not meet) of support for agreed terms of MOU from the point of view of signatory A

+

*Level (exceeded, met, did not meet) of support for agreed terms of MOU from the point of view of signatory A

*Number of downloads of safety information from organization's website

|-

Line 131: Line 134:

|

*Numbers of communications, events, inspections, citations, etc.

−

* Deliverable produced or achieved (yes/no) within expected time frame

+

*Deliverable produced or achieved (yes/no) within expected time frame

|}

Line 159: Line 162:

*Administrative data—Information that is being collected through day-to-day activities, such as permit and licensing applications, audits, inspections, and enforcement. Regulatory organizations should be mindful of the administrative burden that data collection may place on businesses and individuals.

*Primary data—Information that is collected through specialized data collection exercises, such as surveys, focus groups, expert panels, or specialized evaluation studies.

−

* Secondary data—Data that have been collected by other organizations, such as national statistics on health and economic status.

+

*Secondary data—Data that have been collected by other organizations, such as national statistics on health and economic status.

Baseline measurements establish critical reference points from which subsequent changes in indicators can be measured. If reliable historical data on the performance indicator exists, they should be used. Otherwise, it will be necessary to collect baseline data at the first opportunity.

Line 174: Line 177:

Regulatory organizations have a responsibility to report to Canadians on an annual basis through the Report on Plans and Priorities (RPP) and Departmental Performance Report (DPR). This reporting responsibility provides an excellent opportunity to roll up the findings of ongoing performance measurement activities. Other reporting instruments such as websites, annual reports, and newsletters are also effective and timely means for communicating progress on a regulatory proposal to Canadians.

−

===Evaluation Strategy ===

+

===Evaluation Strategy===

Regulatory organizations are required to evaluate their regulatory activities in accordance with the time frames and cycle established in the ''[http://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=15024 Policy on Evaluation]'', which defines evaluation as "the systematic collection and analysis of evidence on the outcomes of programs to make judgments about their relevance, performance and alternative ways to deliver them or to achieve the same results." The evaluation of regulatory activities should be referenced in the annual departmental evaluation plan.

Line 189: Line 192:

[[File:Pmep-pmre1-eng.gif|alt=This is an example that indicates how the regulatory activities are situated in the PAA. The sample shows multiple levels of program activities, sub-activities and sub-sub-activities and how they relate to one another.|thumb|Strategic Outcome]]

−

=== Linkage to the Program Activity Architecture===

+

===Linkage to the Program Activity Architecture===

Briefly describe where the regulatory activities are situated in the PAA. If the regulatory activities do not yet figure in the PAA, indicate when they are expected to be integrated. An example of how regulatory activities are represented in a PAA follows.

Line 195: Line 198:

Before the PMEP is signed off by the regulatory organization's responsible Assistant Deputy Minister, a draft copy of the PMEP must be sent to the Secretariat's Regulatory Affairs Sector portfolio analyst for review to confirm that the PMEP meets CDSR requirements. When seeking confirmation from the Secretariat's Regulatory Affairs Sector, the Head of Evaluation should have provided comments on the PMEP.

−

=== Assistant Deputy Minister Sign-off ===

+

===Assistant Deputy Minister Sign-off===

The Head of Evaluation must review the PMEP and agree that it effectively supports the conduct of an eventual evaluation. The PMEP is then signed by the Assistant Deputy Minister (or equivalent) responsible for the regulatory proposal.

Approval criteria for the Assistant Deputy Minister (or equivalent) are the following:

−

*The scope and detail of the PMEP are commensurate with the impact of the regulatory proposal;

+

* The scope and detail of the PMEP are commensurate with the impact of the regulatory proposal;

−

*The PMEP's content is accurate and reflects the design, implementation, and evaluation of the regulatory activities;

+

* The PMEP's content is accurate and reflects the design, implementation, and evaluation of the regulatory activities;

*There is commitment to monitor, evaluate, and report on the performance of the regulatory proposal, including implementation of the PMEP, through the use of progress information to support decision making and the conduct of periodic evaluation exercises according to the departmental evaluation plan;

*Accountabilities for delivery, collection, and timely reporting of performance information and for evaluation activities are clear;

*Resources are sufficient to implement the PMEP; and

−

* The Head of Evaluation has reviewed the PMEP and agrees that it effectively supports the conduct of an eventual evaluation.

+

*The Head of Evaluation has reviewed the PMEP and agrees that it effectively supports the conduct of an eventual evaluation.

All sign-offs on the PMEP must be obtained before final approval of the RIAS by the Secretariat's Regulatory Affairs Sector. The regulatory organization must send two signed copies of the final PMEP to Regulatory Affairs.

Line 212: Line 215:

Identify the contact person(s) and contact information for enquiries.

−

== What needs to be in the Regulatory Impact Assessment Statement? ==

+

==What needs to be in the Regulatory Impact Assessment Statement?==

A summary (maximum 2 pages) of the PMEP is to be included in the Performance measurement and evaluation section of the RIAS, consisting of the following key elements:

Line 221: Line 224:

*Indication of the PMEP's availability upon request.

−

==Who can provide advice and assistance and where can you get training?==

+

== Who can provide advice and assistance and where can you get training?==

Departmental performance measurement and evaluation specialists can provide guidance on the articulation of key outcomes, results logic, and intended policies, the identification and selection of indicators, and the development of strategies for ongoing monitoring and reporting as well as for evaluation. Specialists involved in the cost-benefit analysis of the regulatory proposal may also be of value to you. Information management and information technology (IM/IT) personnel can also contribute to the PMEP development process by providing expertise on data system design or redesign. They can identify what is already being collected, what would be easy to collect, what mechanisms are already in place to collect data, and what the system implications might be of choosing a certain indicator for regular monitoring.

Line 255: Line 258:

The regulatory organization should send two signed copies of the final PMEP to the Secretariat's Regulatory Affairs Sector. Regulatory Affairs will then sign the two PMEPs and return one copy to the regulatory organization.

−

=== Step 4===

+

===Step 4===

A summary of the PMEP is to be included in the Performance measurement and evaluation section of the RIAS; however, the PMEP is not part of the regulatory package that goes to Treasury Board.

Line 261: Line 264:

To ensure the departmental PAA and PMF reflect new regulatory activities, they should be updated at the earliest opportunity, but only after the regulation has been published in ''Canada Gazette'', Part II.

−

=== Step 6===

+

===Step 6===

Regulatory organizations are required to evaluate their regulatory activities according to the time frames and cycle established in the Policy on Evaluation. Regulatory activities should be referenced in the annual departmental evaluation plan.

A regulatory review may be required if the evaluation's findings demonstrate that the regulatory activities are not achieving the intended outcome(s).

−

=== Step 7===

+

===Step 7===

Reporting on the performance of regulatory activities should be done on an annual basis through the evaluation reports, the RPP, and the DPR. Other reporting instruments such as websites, annual reports, and newsletters are also effective and timely means for communicating progress on regulatory activities to Canadians.

−

== Contact information==

+

==Contact information==

For general public enquiries, please contact the Treasury Board of Canada Secretariat at:

Line 303: Line 306:

</blockquote>

−

==Appendix B: Developing a logic model for regulatory activities ==

+

==Appendix B: Developing a logic model for regulatory activities==

A logic model (also known as results logic and theory of action or intervention) is a graphic (usually accompanied by text) that tells the story of a regulatory initiative. It connects the inputs (resources) and activities (internal processes) to the outputs (products or services generated from the activities), the groups reached (partners, intermediaries, and target groups), and the expected outcomes of that initiative (the sequence of changes in groups outside the control of the regulator).

[[File:Pmep-pmre3-eng.gif|thumb|400x400px|Figure 2: Example of How Problems and Needs Inform the Results Logic of an Environmental Initiative|alt=]]

Line 336: Line 339:

{| class="wikitable"

!Logic model component

−

! Ask yourself about the problem

+

!Ask yourself about the problem

!Examples

−

!Ask yourself about the result you want

+

! Ask yourself about the result you want

!Examples

|-

Line 345: Line 348:

|

*What is the social or cultural problem?

−

* What is the risk to public health, the environment, public safety, or the economy?

+

*What is the risk to public health, the environment, public safety, or the economy?

*What is the severity of the risk?

|Rate of health incidents, hectares of land contaminated, rate of injuries or deaths caused by airplanes

−

|

+

|

*When will the problem or risk no longer be an issue?

*What will the problem or risk look like when it is no longer an issue?

−

*How does the program line up with the department's Strategic Outcome(s)?

+

* How does the program line up with the department's Strategic Outcome(s)?

−

| Reduced rate of health incidents, hectares of land remediated, reduced rate of injuries or deaths caused by airplanes

+

|Reduced rate of health incidents, hectares of land remediated, reduced rate of injuries or deaths caused by airplanes

|-

!Intermediate outcomes

Stakeholder behaviour

|

−

*How do stakeholder behaviours influence the social or cultural situation?

+

* How do stakeholder behaviours influence the social or cultural situation?

*What behaviours are having a negative impact on the social or cultural situation?

|Industry is not adhering to voluntary guidelines

Line 363: Line 366:

|

*What behaviours would you like to see change?

−

*How do the behaviours need to change?

+

* How do the behaviours need to change?

−

| Industry compliance with new regulations

+

|Industry compliance with new regulations

Reduction in percentage of Canadians putting toxic cleaning products down the drain

|-

!Immediate outcomes

Stakeholder knowledge

−

|

+

|

*What gaps exist in your target population's knowledge, abilities, skills, or aspirations?

|Poor industry awareness and acceptance of standards

Poor Canadian knowledge of risks

|

−

*How do we expect our target audience(s) to react immediately to the deliverables? How will the target audience's knowledge, abilities, skills, or aspirations change?

+

* How do we expect our target audience(s) to react immediately to the deliverables? How will the target audience's knowledge, abilities, skills, or aspirations change?

|Industry awareness and understanding of new regulations

Canadian awareness and acceptance of risks and need for new rules

Line 381: Line 384:

Stakeholder participation

|

−

*Are there gaps, which are critical to achieving your objective, in the participation or engagement of groups?

+

* Are there gaps, which are critical to achieving your objective, in the participation or engagement of groups?

−

|Low industry participation Low engagement from Canadians

+

| Low industry participation Low engagement from Canadians

|

−

*Who is this initiative intended to reach or who will be affected?

+

* Who is this initiative intended to reach or who will be affected?

−

* Whose behaviour needs to change and among which group or population?

+

*Whose behaviour needs to change and among which group or population?

−

*How will the participation or engagement of important groups change as a result of your program?

+

* How will the participation or engagement of important groups change as a result of your program?

|Increase in participation and engagement from Canadians, industry, stakeholders, and other governments and jurisdictions

|-

Line 392: Line 395:

Programs or supports in place for addressing the problem

|

−

*Are there gaps in the suite of supports, programs, or services that are currently in place to address the problem or risk?

+

* Are there gaps in the suite of supports, programs, or services that are currently in place to address the problem or risk?

−

| Industry is under no legal obligation to comply with the voluntary standards currently in place

+

|Industry is under no legal obligation to comply with the voluntary standards currently in place

Information bulletins to advise Canadians about certain risks are not resulting in a significant enough change in the number of incidents

|

−

* What product or service will we deliver in order to fill the gaps?

+

*What product or service will we deliver in order to fill the gaps?

|Regulations, inspection reports, information campaigns

|-

−

! Activities

+

!Activities

Internal practices

|

−

* Are there problems with the current delivery practices?

+

*Are there problems with the current delivery practices?

−

* Are there programs or services being offered in other jurisdictions that are demonstrating better results than our programs or services?

+

*Are there programs or services being offered in other jurisdictions that are demonstrating better results than our programs or services?

|Inefficient delivery practices, incomplete assessment of criteria, gaps in communication

|

Line 414: Line 417:

Internal resources

|

−

*Are there gaps in the financial, human, or technical resources available? Is data availability a problem?

+

* Are there gaps in the financial, human, or technical resources available? Is data availability a problem?

−

| Lack of resources, lack of data, or lack of information

+

|Lack of resources, lack of data, or lack of information

|

*What resources do we have for this regulatory initiative?

Nancy.luo

430

edits

Changes

Handbook for Regulatory Proposals: Performance Measurement and Evaluation Plan (view source)

Revision as of 20:01, 3 October 2021