Internal Audit and Program Evaluation Directorate
Evaluation of the Canada Border Services Agency Targeting Program

January 2016

Note

[*] An asterisk appears where sensitive information has been removed in accordance with the Access to Information Act and the Privacy Act.

Acronyms and Abbreviations

ACI
Advance Commercial Information
API/PNR
Advance Passenger Information/Passenger Name Record
CBSA
Canada Border Services Agency
CRA
Canada Revenue Agency
FY
Fiscal Year
FTE
Full Time Equivalent
NHQ
National Headquarters
NTC
National Targeting Centre
OGD
Other Government Department
PM
Performance Measurement
POE
Port of entry
RPO
Regional program officer
WCO
World Customs Organization

Executive Summary

Program Profile

The objective of the National Targeting Program is to develop and maintain the Canada Border Services Agency’s (CBSA) ability to identify suspected high-risk people, goods and conveyances in order to alert the appropriate CBSA personnel of an impending suspected risk or threat to national security and/or public safety priorities, supported by evidence-based data and intelligence. By identifying potentially high-risk travellers and goods for examination using advance information and automated risk assessment systems, the CBSA contributes to the protections of Canada's security and prosperity while minimizing the disruption to legitimate trade. The assessment of this information is referred to as targeting under the CBSA Program Alignment Architecture.

A decision was made to implement a centralized Targeting Business Model in order to make targeting functions more efficient. As a result, on April 1, 2012, the National Targeting Centre (NTC) was created.

Evaluation Purpose and Scope

The Evaluation of the CBSA Targeting Program was approved as part of the Agency’s 2014–2019 Five-Year Program Evaluation Plan. Although the NTC was officially created on April 1, 2012, the targeting business lines vary in maturity and stages of implementation. This evaluation looked at the Air Passenger Targeting and Marine and Air Commercial Targeting, for which the Agency has the mandate and the advance information to conduct targeting activities. The evaluation aimed to determine lessons learned from the current targeting business lines that could be applied to future enhancements of the Targeting Business Model.

List of Findings and Recommendations

Continued Need

Targeting is relevant, addresses an ongoing need, and supports the priorities and strategic outcomes of the Government of Canada and the CBSA.

Alignment with Government Priorities

Targeting supports the Government of Canada’s “Beyond the Border” commitment to harmonizing with the United States methodology.

Targeting contributes to Canada’s international commitment to establish a risk-management system to identify high-risk individuals and shipments.

Alignment with Federal Roles and Responsibilities

The Targeting Program is aligned with the federal roles and responsibilities and supports the Agency’s responsibilities and priorities to secure the border strategically and modernize processes to develop an integrated approach to targeting.

Achievement of Expected Outcomes

The Targeting Program Logic Model and Performance Management Strategy Framework have not been finalized and are missing key measures to assess overall program performance.

Recommendation 1:

That the Vice-President, Programs Branch, in collaboration with the Operations Branch, finalize and approve the Targeting Program Logic Model, Performance Measurement Strategy Framework and program performance indicators [*].

[*]

Recommendation 2:

That the Vice-President, Operations Branch improve communication mechanisms between regional staff and National Headquarters so that regional feedback and intelligence information is considered in the development and issuance of targets and to support continuous improvement.

[*]

[*]

Demonstration of Efficiency and Economy

Operational expenditures related to targeting have slightly decreased when comparing before and after the centralization of activities within the NTC.

Since the centralization of the targeting function, along with system improvements, the CBSA has improved its ability to issue targets when faced with fluctuating volumes in travellers and goods. In addition, the NTC continues to take on additional targeting activities within the same resourcing base. These are indications that the Targeting Program is delivered more cost-efficiently post-centralization.

Implementation of the CBSA Targeting Business Model

The consolidation objectives as identified in the CBSA Targeting Business Model have been met.

Since the centralization of the targeting function, more targets are being issued annually and the Program is being delivered at a lower cost. However, in the absence of more pre- and post-centralization data, further program effectiveness comparisons cannot be made at the national level.

Targeting Program management structure has been confirmed in both Programs and Operations Branches; however, there are no regional representatives on any of the operational committees.

The regional program officers’ (Targeting) roles and responsibilities are not being fulfilled as envisioned.

Recommendation 3:

That the Vice-President, Programs Branch re-assess the need for, the roles and responsibilities and governance structure related to regional program officers (Targeting).

Introduction

The Canada Border Services Agency (CBSA) provides integrated border services that support national security priorities and facilitates the free flow of admissible people and goods, including food, plants, animals and related products, across the border. In Fiscal Year (FY) 2014–2015, the CBSA processed close to 98 million travellers and cleared more than 15 million commercial releases. Footnote 1About 29 million travellers (or 29.9% of all travellers) were processed in the air mode, about 469,000 commercial releases were processed in marine cargo (3.0% of all commercial releases), and approximately 3.9 million commercial releases (or 25.4% of all commercial releases) in air cargo.Footnote 2

Under the Customs ActFootnote 3 and the Immigration and Refugee Protection Act,Footnote 4 information about the conveyance and persons and goods on board the conveyance is to be provided to the CBSA prior its arrival in Canada. The legislation and regulations stipulate what information must be provided, who has responsibilities for providing it, as well as the circumstances, timeframe and manner in which the information is to be provided.

To facilitate the entry of lawful travellers and goods into Canada, the CBSA reviews the pre-arrival information to identify potentially high-risk travellers and goods for examination. The risk assessment of this information is referred to as targeting. The CBSA's targeting activities are multi-faceted, multi-dimensional and embedded in both commercial and traveller processing. They involve the scrutiny of goods, people, food, plants and animals entering the country by different modes of transportation.

Program Description

The CBSA's Targeting Program identifies people and goods bound for Canada that may pose a threat to the security and safety of the country. The targeting process employs a complex risk-based approach. Advance Passenger Information/Passenger Name Record (API/PNR) and Advance Commercial Information (ACI) provide the CBSA with pre-arrival data on people and goods that can be used to perform risk assessments in advance of their arrival in Canada. People and goods that are identified as posing a threat to Canada are referred by targeting officers for verification and examination upon their arrival at a port of entry (POE).Footnote 5 Border services officers intercept and conduct examination of targets, and document and report results.

Prior to the implementation of the current centralized Targeting Business Model, there was no single “pre-arrival Targeting Program” in place. Instead, regional offices and National Headquarters (NHQ) worked together to perform the pre-arrival targeting function. As such, the targeting activities were conducted both at a national and regional level.

Following recommendations from the Report of the Auditor General of Canada – Chapter 5 “Keeping the Border Open and Secure” and the Pre-Arrival Targeting Evaluation StudyFootnote 6, the CBSA created a targeting functional authority on April 1, 2010 to lead the Agency’s Targeting Program as part of the Agency’s Change Agenda. The intent was to make the program more effective and gain efficiencies by developing and implementing a new centralized Targeting Business Model. The new service delivery model was to develop national targeting policies, performance measurement and monitoring tools, make system enhancements, and establish closer working relationships with Other Government Departments (OGDs) and international partners. The targeting functional authority now lies within the Enforcement and Intelligence Programs Directorate, Programs Branch.

The Agency also decided to consolidate all targeting resources into a National Targeting Centre (NTC). The NTC was officially launched on April 1, 2012 and the air traveller, air cargo and marine cargo regional targeting activities were consolidated into the NTC by February 2013. The goal was to build the capacity to target for all modes and streams (Rail, Courier, Low Value Shipments, Postal, Marine Passenger, Highway and Exports).

Program Objectives

The objective of the National Targeting Program is to develop and maintain the CBSA’s ability to identify suspected high-risk people, goods and conveyances prior to their arrival in Canada, in order to alert the appropriate CBSA personnel or OGDs of an impending suspected risk or threat to national security and/or public safety priorities at the earliest opportunity.

The draft expected outcomes of the Targeting Program are:

Evaluation Purpose and Scope

The Evaluation of the CBSA Targeting Program was approved as part of the Agency’s 2014–2019 Five-Year Program Evaluation Plan. In addition to the core issues from the Treasury Board Secretariat evaluation policyFootnote 7, senior management asked that this evaluation provide an analysis on the effectiveness and efficiency of the new centralized model vis-à-vis the previous regionally based delivery model.

In preparation for the evaluation, the evaluation team reviewed previously conducted studies and the terms of reference (where available) of ongoing and future studies to avoid research duplication and to minimize operational disruption. The scope of the evaluation focused on the centralization of targeting activities.

Although the Agency conducts other types of targeting activities, the evaluation only assessed the CBSA Targeting Business Model by studying targeting activities in air cargo, marine cargo and air travellers, which are conducted at the NTC. The evaluation excluded the targeting activities for which the Agency does not receive mandatory electronic pre-arrival information, or which are conducted in the regions or use proprietary systems. The targeting activities that were scoped in or out of the evaluation are outlined in Exhibit 1.

Exhibit 1: Scope of the Evaluation

In Scope Out of Scope
Air cargo targeting (NTC) Vessel and crew targeting (NTC)
Marine cargo targeting (NTC) Cruise ship passenger screening (NTC)
Air traveller targeting (NTC) Rail cargo targeting (Regional)
  Courier/Low Value Shipments (CLVS) targeting (Regional)

Evaluation Methodology

The evaluation research phase was conducted between January and April 2015 and included a variety of quantitative and qualitative research methods (see detailed description in Appendix B). Data was collected from a range of sources to ensure multiple lines of evidence were available when analyzing data and formulating findings. The Evaluation Team conducted interviews, focus groups, analysed program performance data, human resources and financial data, reviewed key program documentation and conducted field research in three regions as well as at the National Targeting Centre. Each finding was triangulated using a mix of quantitative and qualitative data. The recommendations provided are based on these findings.

By using multiple lines of evidence, limitations of any single methodology are mitigated. For this evaluation, the following research limitations should be considered:

Findings and Recommendations

Continued Need

Growing travel flows, changing travel patterns, such as increased direct international flightsFootnote 8 and increased commercial traffic are placing the traditional model of border management under pressure. As an example, the volume of travellers arriving by commercial passenger flights increased by 17% over the past five years from 24.6 million in FY 2010–2011 to 29 million in FY 2014–2015.Footnote 9 This increase puts a greater demand on the Agency to implement pre-arrival targeting strategies to identify suspected high-risk people and goods while facilitating the free flow of admissible people and goods. There is, therefore, a continued need to provide activities that support effective border management and that contribute to the safety and security of Canada. Through the CBSA Targeting Program, the Agency is fulfilling this need. Furthermore, identifying people and goods as early as possible prior to their arrival contributes to the Agency's risk-based approach to border management by 'pushing the border out'.

It is within this context that the CBSA restructured its targeting activities and implemented the new Targeting Business Model to facilitate a more integrated and consistent approach to targeting.

Targeting is relevant, addresses an ongoing need, and supports the priorities and strategic outcomes of the Government of Canada and the CBSA.

Alignment with Government Priorities

On February 4, 2011, the Prime Minister of Canada and the President of the United States issued a Declaration of a Shared Vision for Perimeter Security and Economic Competitiveness. As a result of this Declaration, on December 7, 2011, a joint Beyond the Border: Perimeter Security and Economic Competitiveness Action Plan was announced that aimed to enhance security and economic competitiveness, not just at, but beyond the border. One of the objectives of the Action Plan was to address threats at the earliest point possible in the travel and trade continuum to ensure the safety and security of Canadians. By identifying suspected high-risk people, goods, and conveyances through an integrated risk assessment process, the Targeting Program supports this Government of Canada priority.

As identified within the Action Plan, Canada was to establish a common approach to perimeter screening and to promote security and border efficiency. As a result, the Agency created a scenario-based traveller targeting methodology in March 2014. This methodology is aligned with that of the United States and with existing bilateral information-sharing agreements, such as the sharing of information provided by airlines to screen inbound flights for persons at high-risk of being engaged in terrorism or serious criminal activity.Footnote 10

Targeting supports the Government of Canada’s “Beyond the Border” commitment to harmonizing with the United States methodology.

The Targeting Program activities support Canada’s international commitments to control and administer the movement of goods and people to enhance security while facilitating trade and travel. The World Customs Organization (WCO) Risk Management CompendiumFootnote 11, Revised Kyoto ConventionsFootnote 12 and SAFE FrameworkFootnote 13 outline how countries should implement risk-based procedures for the provision of advance information including automated risk-management systems, and that selectivity and targeting should be used to identify potentially high-risk people and cargo. To do this, the WCO recommends that risk assessment/targeting Centres could serve as a tool to enhance information exchange and inter-agency cooperation.Footnote 14 The CBSA Targeting Program is in line with the WCO’s guidelines and what other Border FiveFootnote 15 countries are doing. [*] Automated tools also make it possible to process the growing volume of travellers and goods coming to Canada.

Targeting contributes to Canada’s international commitment to establish a risk-management system to identify high-risk individuals and shipments.

Alignment with Federal Roles and Responsibilities

The Agency's mandate for providing integrated border services that support national security and public safety priorities while facilitating the free flow of admissible people and goods is clearly established in the Canada Border Services Act.Footnote 16 In addition, both the Customs ActFootnote 17 and the Immigration and Refugee Protection Act (IRPA)Footnote 18 specify the provision and/or access of advance commercial and passenger information to the CBSA prior to the arrival of a conveyance, person or goods. The Reporting of Imported Goods RegulationsFootnote 19 stipulates requirements and specific timeframes for providing ACI to the CBSA. The CBSA utilizes API/PNR and ACI data to risk assess and identify unknown and high-risk goods and people bound for Canada prior to their arrival.

Furthermore, targeting is an integral component of the Agency’s Risk Assessment Program. Thus, the creation of a functional authority and the NTC to ensure an integrated targeting approach supports the Agency’s priorities to secure the border strategically and modernization processes.Footnote 20

The Targeting Program is aligned with the federal roles and responsibilities and supports the Agency’s responsibilities and priorities to secure the border strategically and modernize processes to develop an integrated approach to targeting.

Achievement of Expected Outcomes

Targeting Program Performance Measurement

Following the implementation of the Targeting Business Model, the Agency standardised the data collection model for the assessment of the Targeting Program success. The use of a uniform template allowed for a clearer picture of the overall performance by ensuring that the definition of what constitutes a resultant target is consistent across all regions. [*]

These elements contribute to the overall measurement of program success. All these data points contribute to the development and changes to risk rules and indicators and the program’s ability to adapt to threats. They could also be integrated in the Performance Management (PM) Strategy Framework in the future.

The Targeting Program does not have a finalized and approved Logic Model (LM). There are currently two draft LMs for targeting with very similar activities and outputs and identical outcomes, one for each stream (travellers and commercial). Furthermore, as currently drafted, the outputs identified in these documents would not allow the program area to measure its performance against the established outcomes. For example, outputs such as partnerships agreements, priorities, plans, policies, SOP’s, training sessions and materials would not allow the measurement of its related outcome, “Ability to identify people, goods, commercial and associated entities that may pose a threat to Canada prior to arrival”.

Likewise, the Targeting Program does not have a finalized and approved PM Strategy Framework. As with the LM, there are two draft PM Strategy Frameworks and they are missing key performance measures that could help demonstrate program success. [*]

Together, the LM and PM Strategy Framework would enable the collection of performance data for effective decision making. Footnote 21 [*]

The Targeting Program Logic Model and Performance Management Strategy Framework have not been finalized and are missing key measures to assess overall program performance.

It is recommended:

Recommendation 1:

That the Vice-President, Programs Branch, in collaboration with the Operations Branch, finalize and approve the Targeting Program Logic Model, Performance Measurement Strategy Framework and program performance indicators [*].

Two-way Communication

The focus of the targeting officer’s responsibility is to generate and disseminate target information in a clear, efficient and timely manner to alert appropriate CBSA personnel.Footnote 22 POEs are collectively responsible for entering detailed examination results into appropriate systems once the examination has been completed.Footnote 23 The targeting outcome is dependent upon the details of the target issued, the quality of the examination conducted and the level of detail provided in the examination results.

During the evaluation, some border services officers who participated in focus groups [*]. Though the targeting officer can include a narrative in the system, they are limited in the number of characters that they can use. Targeting officers are also limited, by legislation as well as regulation and policyFootnote 24, in the type of information that they can share with the frontline.

The NTC has created a feedback form to receive suggestions from the regional staff and it also conducts semi-regular conference calls with some of the POEs. In turn, the regional intelligence and examining officers are also responsible for sharing information, local trends and risks with the NTC on an ongoing basis. Strengthening the information sharing and collaborative processes would facilitate the decision-making process. Furthermore, a more effective feedback loop is expected to enhance the Targeting Program’s ability to include regional trends, risks and realities in its analysis and as such, be more responsive to the dynamic targeting environment.

The centralization of the targeting function within the [*]. In some cases under the new model, officers were unaware of communication protocols for contacting the NTC. To mitigate this gap, the NTC has conducted outreach sessions to create awareness and provide information on the Targeting Business Model including publishing targeting results to ensure visibility and has worked closely with the Regions on ad hoc “Special Projects”.Footnote 25

[*]

It is recommended:

Recommendation 2:

That the Vice-President, Operations Branch improve communication mechanisms between regional staff and National Headquarters so that regional feedback and intelligence information is considered in the development and issuance of targets and to support continuous improvement.

Systems

[*]

The CBSA is in the process of modernizing its IT systems and their successful implementation will further improve targeting performance across modes.

[*]

Pre-arrival information

The quality of pre-arrival information received from carriers and importers is critical to the effectiveness of targeting.

Quality issues regarding ACI and API/PNR data identified in the 2008 Pre-Arrival Targeting Evaluation Study remain.Footnote 26 [*] While transporters compliance in terms of ACI accuracy and timeliness is being monitored, [*].

Exhibits 2 and 3 look at the state of API/PNR information in order to assess any possible impact on program delivery.

Exhibit 2: API/PNR Compliance per Flights for FY 2013–2014

Data Type Total Compliant Flights (Non-Compliant) Percentage of Compliant Flights (Non-Compliant)
Crew API 252,715 (2,197) 99.1% (0.9%)
Passenger API 252,167 (2,745) 98.9% (1.1%)
Passenger PNR 250,337 (4,575) 98.2% (1.8%)

Source: Internal CBSA documentation.

The CBSA measures carrier compliance by assessing the number of flights for which it received API and PNR data. [*]

Exhibit 3: [*]

  [*] [*] [*]
[*] [*] [*] [*]
[*] [*] [*] [*]
[*] [*] [*] [*]
[*] [*] [*] [*]
[*] [*] [*] [*]

[*]

[*]

[*]

[*]

Demonstration of Efficiency and Economy

As per the scope of the evaluation, exhibit 4 looks at the Targeting Program operational expenditure over time in order to assess program efficiency and determine if the implementation of the Targeting Business Model has achieved cost savings. Fiscal years 2012–2013 and 2013–2014 are considered transition years when the NTC operations were being set up and the CBSA incurred a number of one-time transitional costs. As such, FY 2011–2012 financial costs represent the last FY before the implementation of the new Targeting Business Model and FY 2014–2015 represents the post centralization operational costs.

Exhibit 4: Table of Agency Targeting ExpendituresFootnote 27 at the Program Sub Sub Activity Level per FY.

  Pre-centralization Post-centralization
Branch FY 2011–2012 FY 2012–2013 FY 2013–2014 FY 2014–2015
NHQ Total $9,157,774 $14,715,338 $17,007,124 $16,922,346
Operations Branch $9,157,774 $14,715,338 $17,007,124 $16,922,346
Regions Total $9,672,321 $5,842,862 $2,727,277 $1,413,717
Greater Toronto Area Region $2,944,815 $1,741,070 $296,078 $334,392
Northern Ontario Region $67,135 $4,896 $15,855 $78,679
Atlantic Region $831,528 $576,469 $235,565 $83,560
Pacific Region $1,786,112 $1,165,891 $725,177 $95,315
Prairie Region $1,242,770 $814,766 $918,142 $402,700
Quebec Region $2,221,892 $1,290,850 $307,688 $198,404
Southern Ontario RegionFootnote 28 $578,069 $248,921 $228,772 $220,667
Total $18,830,095 $20,558,200 $19,734,401 $18,336,063

Source: CBSA systems Data as of May 8, 2015.

This table focuses on the operational costs because centralization savings were identified in regional and NHQ operational expenditures. Although other branches incur direct and indirect costs related to targeting, they have not been included in the table above as the cost comparison is not within the approved scope. For example, $3.38 million Programs Branch expenditure can be directly attributed to the Targeting Program Management Unit. The CBSA has changed how it reports the Information, Science and Technology Branch costs and the Programs Branch has re-organized its structure within the last two years. The inclusion of these costs without re-coding previous years of expenditures does not provide an accurate comparison of pre and post centralization costs. 

In FY 2014–2015 the CBSA spent $18.3 million on operational targeting activities while the operational expenditure in FY 2011–2012 was $18.8 million, which is a decrease of $0.5 million. The corresponding decrease of targeting expenditures in the regions and increase in Operations Branch are in line with the centralization of activities within the NTC.

Exhibit 5 looks at the Targeting Program resources over time in order to assess program efficiency and determine if the implementation of the Targeting Business Model has achieved resource savings.

Exhibit 5: Number of Full Time Equivalents (FTEs) for Targeting per FY.

  FY 2010–2011 FY 2011–2012 FY 2012–2013 FY 2013–2014 FY 2014–2015
Total FTEs 312 n/a 148 195 213

Source: Internal CBSA documentation

From FY 2010–2011 to FY 2014–2015, the number of FTEs has decreased by almost a third (31.7% or 99 FTEs). The current resource level remains below the FY 2010–2011 numbers. Given that the overall number of targets has increased while the number of resources has decreased, this is an indication that the Targeting Program is delivered more cost-efficiently.

Operational expenditures related to targeting have slightly decreased when comparing before and after the centralization of activities within the NTC.

With the same level of resourcing, the NTC has also taken on several additional responsibilities such as, cruise ship passenger screening, Seasonal Agricultural Worker Program Criminality Verification Process, 2015 Pan American and Parapan American Games as well as Ebola screening. Expenditures related to targeting have decreased and the reach of the Agency has increased suggesting that the Targeting Program is being delivered more efficiently.

Since the centralization of the targeting function, along with system improvements, CBSA has improved its ability to issue targets when faced with fluctuating volumes in travellers and goods. In addition, the NTC continues to take on additional targeting activities within the same resourcing base. These are indications that the Targeting Program is delivered more cost-efficiently post-centralization.

Implementation of the CBSA Targeting Business Model

In developing the Targeting Business Model, the Agency had set specific implementation objectives while making an effort to restructure the way targeting operational services was provided. As such, the CBSA:

The consolidation objectives as identified in the CBSA Targeting Business Model have been met.

Prior to centralization, there was not a single functional authority for targeting. Performance data was not collected, partially collected or not collected consistently. This lack of pre-NTC performance data limited the direct comparison of performance of before and after the implementation of the Targeting Business Model.

Since the centralization of the targeting function, more targets are being issued annually and the Program is being delivered at a lower cost. However, in the absence of more pre and post centralization data, further program effectiveness comparison cannot be made at the national level.

Overall, the targeting management structure provides the framework to set accountabilities as well as roles and responsibilities for all program activities. The Targeting Program management structure is comprised of a number of committees. The strategic direction and policy guidance of the national Targeting Program falls within the Enforcement and Intelligence Program Management Table. This director-general level committee sets the strategic direction, approves the priorities and performance objectives and serves as a primary body to support the effective governance and functional management of the Enforcement and Intelligence Programs.

The Agency has also recently created a Targeting Program Management Committee (March 2015) chaired by the Targeting Program Unit and a Traveller Scenario Management Committee (June 2015) chaired by the NTC. According to the terms of references, the formerFootnote 31 is responsible for ensuring the efficiency and effectiveness in the management of the Traveller Targeting Program and its compliance with international agreements and legislative requirements for Scenario Based Targeting. The latterFootnote 32 is responsible for validating and monitoring scenarios before they are implemented or removed from the system.

[*]

Targeting Program management structure has been confirmed in both Programs and Operations Branches; however, there are no regional representatives on any of the operational committees.

Furthermore, in the CBSA Targeting Business Model, the regional program officers (RPO) (Targeting) positions were to serve as a liaison between the region they represent, the NTC and the Targeting Program authority. Though not an exhaustive list, RPOs were toFootnote 33:

Although these roles and responsibilities are clearly outlined in the CBSA National Targeting Policy as well as in the Key Activities for RPOs document, there is no evidence to demonstrate that the RPOs are currently providing regional feedback to NHQ. In addition, conference calls that were to be held on a regular basisFootnote 34 are not being conducted which, limits the opportunities for RPOs to provide feedback to NHQ.

RPOs are, however, supporting the NTC by facilitating the collection of missing examination results for targets but they are not contributing to the Targeting Program as originally planned. Though this is an NHQ funded position, in some regions the RPOs have been partially reassigned to other duties. In one region, the roles and responsibilities of the RPO are being conducted by an Intelligence Officer within their triage unit.

The regional program officers’(Targeting) roles and responsibilities are not being fulfilled as envisioned.

It is recommended:

Recommendation 3:

That the Vice-President, Programs Branch re-assess the need for, the roles and responsibilities and governance structure related to regional program officers (Targeting).


Appendix A – Management Response

Recommendation 1:

That the Vice-President, Programs Branch, in collaboration with the Operations Branch, finalize and approve the Targeting Program Logic Model, Performance Measurement Strategy Framework and program performance indicators [*].

Management Response: Implementation Date:
The Programs Branch agrees with this recommendation and will work in collaboration with the Operations Branch to finalize and approve the Targeting Vision Document, Targeting Program Logic Model and the Performance Measurement Strategy Framework. June 2016

Recommendation 2

That the Vice-President, Operations Branch improve communication mechanisms between regional staff and National Headquarters so that regional feedback and intelligence information is considered in the development and issuance of targets and to support continuous improvement.

Management Response: Implementation Date:
The Operations Branch agrees with this recommendation and will work collaboratively with Programs Branch to address the communications gap between the region and NHQ to ensure that timely, accurate, and complete information is shared for the purpose of effective program delivery and management decision making. June 2016

Recommendation 3

That the Vice-President, Programs Branch re-assess the need for, the roles and responsibilities and governance structure related to regional program officers (Targeting).

Management Response: Implementation Date:
The Programs Branch agrees with this recommendation and will re-assess the need for, the roles and responsibilities and the governance structure related to regional program officers (Targeting). July 2016

Appendix B – Evaluation Methodology

The evaluation used a multi-method research approach to generate multiple lines of evidence. Data and findings were recorded in an evidence matrix, and only data that could be triangulated with multiple lines of evidence were used in the final evaluation report. The research methodologies used in the evaluation of the Targeting Program are described below:

Calibration

The evaluation team reviewed past and planned evaluation and audit studies to mitigate duplication and ensure an effective use of resources. Where possible, past findings and recommendations were incorporated into the evaluation to assess the degree to which issues remain and the degree to which recommendations have been implemented. Although the audit and evaluation have different focuses, the Program Evaluation Division coordinated with the Internal Audit Division, where possible, on requests for information and documentation to minimized and streamlined requests for information from program stakeholders.

Review of Documents

Relevant documents supplement findings from other lines of evidence, providing either support for the findings or conflicting information. The evaluation team reviewed a variety of documentation to gain a solid understanding of the program and its linkages to other areas in the Agency and connections to OGDs and other external bodies. For example, a review of other Border five members’ administrative practices was conducted for targeting related aspects and compared to the Agency’s model to determine international best practices. In addition, a literature review of centralized business models was conducted to gain an understanding of the context, strengths and weaknesses of centralization to see how the new Centralized Business Model compares. The World Customs Organizations Risk Management Framework was also reviewed for targeting related aspects.

While the documents provided a general overview of the Centralized Business Model, there were some limitations. For example, the program Performance Measurement (PM) Strategy Framework and Logic Model (LM) documents have not been finalized and approved. There are currently two draft LMs for targeting with very similar activities and outputs and identical outcomes, one for each stream (passengers and commercial) and the draft PM Strategy Frameworks are missing key performance measure. Thus, this made it difficult for the evaluation to determine if the program is achieving its intended outcomes and it is possible there may be documents that were not reviewed because the evaluation team was not made aware of them.

Review of Operational, Performance and Financial Data

The evaluation team worked closely with the National Targeting Centre to determine the performance data that could be extracted from the air traveller, air cargo and marine cargo Reporting Suites, the Performance Reporting Unit, Programs Branch for data and with the Regions for regional data where available. Not all data was available leading to gaps in the analysis. [*]

The evaluation team also requested financial data from Comptrollership Branch to assess efficiencies gained from the implementation of the Targeting Business Model (i.e. reduced number of resources and operating expenses (O&M) necessary to deliver targeting activities). The lack of formalized mechanisms to collect performance data pre-centralization limited the ability to measure the increased capacity of the Targeting Business Model and to compare with pre-centralization.

Key Informant Discussions

Key informant discussions were used to gather in-depth information of program design and delivery, performance measurement, program impacts, and areas for improvement. Key informant discussions complement evidence gathered as part of the evaluation, as they provided qualitative information that clarified and contextualized data collected through other methodologies. Interviews were held with relevant managers, supervisors, directors, and specialized personnel in the regions and with NHQ Programs Branch, NHQ Operations Branch and Information, Science and Technology Branch. In addition, standardized questionnaire were drafted by profile (e.g., Manager, Director). This allowed the evaluation team to explore key topics and gathered knowledge of the Targeting Program from a range of participants.

Key informant discussions responses were enter into an excel spreadsheet by themes and analyzed by reoccurring themes. Furthermore, the evaluation team reviewed the answers to the interview questions and determined whether all or most interviewees had a similar view on something or if they were all different. Key findings gathered from the field research was summarized and recorded in the evidence matrix. Given that information collected during interviews may contain bias; other lines of evidence were used to validate key findings to establish overall evaluation findings and recommendations.

Focus Groups

Focus groups were held with targeting officers, superintendents/chiefs and border services officers at all three field research locations. Each focus group comprised of 4-6 officers and superintendents where possible, focusing on the timeliness, completeness and accuracy of the targets. For the targeting officers at NHQ, focus groups discussions consisted of a walkthrough of targeting activities for each modes with either one or two officers. The evaluation team also looked at the challenges/obstacles in completing examinations as prescribed and lessons learned that could enhance the efficiency and effectiveness of the targeting process overall. An experienced facilitator lead the sessions while two evaluators took notes that were cross-validated for findings.

Semi-structured focus group guides were designed based on the key evaluation questions in the methodology report and other questions arising from observations during the field research. This semi-structured approach facilitated discussions between the moderator and participants and help identified relevant issues. The evaluation team conducted a comparative analysis of the focus groups responses by recurring themes, summarized and recorded key findings in the evidence matrix. Sticky notes and flip charts were used to incorporate participants inputs to overcome the limitations identified (i.e., biases resulting from group dynamics, small sample size) as well as observation of the natural work. Furthermore, focus groups findings were corroborated with other lines of evidence to established evaluation findings and recommendations.

Exhibit c-1: Number of Key Informant Discussions and Focus Groups Conducted and Participants.

  Number Conducted Number of Participants
Interviews
Directors and directors general, managers (NHQ) 19 30
Officers (NHQ) 2 2
Regional director general (Regions) 6 7
Management (Regions) 9 13
Regional program officers/Intelligence officers 4 8
Focus Groups
Targeting officers 6 9
Border services officers 6 24
Superintendents/chiefs 4 6

Field Research

Field research was conducted at four locations NHQ (NTC), Pacific (Vancouver), GTA, and Quebec (Montréal) in order to gain a better understanding of the program delivery, to observe and compare differences in the operational contexts across regions. Informal discussions were held with regional directors during the field research to gather information on regional targeting activities. Engagement at a regional level help addressed concerns raised in the Evaluation Neutral Assessment regarding the evaluator's knowledge gap of the program and program delivery. In addition, the field visits provided regional management and staff with the opportunity to contribute to the evaluation and to provide their insights on what works well and what could be improved.

The field research also permitted the team to see the impacts of centralizing all targeting activities in the NTC as well as potential delivery gaps. Overall, the field research allowed for a better understanding on the extent in which performance measure and examination results are fed back into the targeting process to improve the risk assessment capacity of the targeting function. Moreover, field research enabled the evaluation team to address the value proposition that the Targeting Business Model allows for targeting activities to be delivered in a consolidated manner, allowing for a single assessment of cargo/travellers/conveyances and thus, improving program consistency and gains in efficiencies.

Summaries of visits findings were sent to regional directors general following the visits for fact-checked and validation. Findings from the field research were analysed in conjunction with other lines of evidence to determine the findings and recommendations of the evaluation.


Date modified: