Business Continuity Review
- City Auditor
- Apr 23
- 7 min read
Updated: Jun 17
Why the Office of the City Auditor (OCA) did this project
Assess business continuity planning across departments to ensure the City of Tulsa is mitigating risks to critical service interruptions, legal and regulatory noncompliance, restoration costs, and reputational damage.
Project Scope
Internal controls and processes for business continuity within the organization. Citywide priority operations include critical systems and applications such as performing critical operations, collecting revenue, and processing payroll. The focus of this report is on the City of Tulsa’s business continuity program and documented business continuity plans completed by June 30, 2024.
How OCA did this project
Following the ISACA “IT Business Continuity Disaster Recovery Audit Program” tool we conducted the following procedures:
Review documentation of business continuity (BC) and disaster recovery (DR) requirements and procedures for governance and monitoring.
Confirm identification and prioritization of business operations.
Ensure identification of requirements for alternative locations for continued operations in the event of an emergency or service disruption.
Assess backup/scenario and workaround planning for emergency preparedness and communications.
Test backup and recovery systems, locations and personnel.
Track and report business continuity testing results.
Key Observations
The City of Tulsa lacks a formalized policy mandating the City to create and maintain business continuity plans.
All but two departments completed business continuity plans in 2024.
Current versions of business continuity plans contain most of the required planning documentation. Next required steps include plans for testing, monitoring, and continuous improvement.
Several areas for improvement of reviewed business continuity plans are noted. See Observations 3-8.
Instructions to recover the prior version of application/system are not always present for each change in the change management system.
Business Continuity
Fiscal Year Ended June 30, 2024
Background
In May 2021, the City of Tulsa experienced a ransomware attack. An organized crime unit attempted to extort the City of Tulsa into paying a ransom by accessing the City network and installing malicious software (malware) that prevented the City from accessing computer files, systems and networks. Instead of paying the ransom, the City of Tulsa contacted the FBI and rebuilt their network.
Impact
Critical city services continued after the attack without interruption, however administrative and support processes were greatly affected. Additional overtime was needed, and recovery costs were incurred. Six months post incident, the City of Tulsa estimated the recovery cost upwards of $2 million dollars.
What Has Been Done
The City of Tulsa has made large investments in our Information Technology department, both in technology and additional staff. Part of this large investment has been to hire a Disaster Recovery Architect to create and implement an organized citywide business continuity program. The first iterations of approved business continuity plans by departments and the IT disaster recovery plan were created in 2024.
The Review
Scope
The focus of this report is on the City of Tulsa’s business continuity program and documented BC plans to continue providing City services during a sudden disruption. All documentation gathered ended by June 30, 2024. This review did not include the City of Tulsa and Tulsa County’s cooperative emergency program, Tulsa Area Emergency Management Agency (TAEMA).
Objective
Assess business continuity planning across departments to ensure the City is mitigating the risks to critical service interruptions, legal and regulatory noncompliance, restoration costs, and reputational damage.
Methodology
This performance audit followed the ISACA audit program, “Audit Program IT Business Continuity Disaster Recovery” tool. We reviewed enterprise-wide areas of business continuity for eight main components: 1. governance and monitoring, 2. business impact analysis, 3. workforce, 4. location, 5. applications-systems, 6. emergency preparedness and communications, 7. business continuity planning scenario plans and disaster recovery testing, and 8. continuous improvement and reports.
The OCA interviewed City staff and reviewed completed City department business continuity plans, the City of Tulsa Disaster Recovery plan and additional supporting documentation. At the time of testing, eleven business continuity plans were completed. We tested seven of the eight components of enterprise-wide areas of business continuity on all eleven completed business continuity plans. We tested one component, (emergency preparedness and communications) on a sample selection of four business continuity plans, due to the level of detail required for reviewing department call trees.
The Results
After this first iteration of development of the City of Tulsa’s business continuity program, most of the required planning documentation is in place and consistent. As the City of Tulsa’s business continuity program continues to mature, testing and continuous improvement will take place.
Governance and Monitoring
Governance and monitoring establishes that business continuity plans are aligned with organizational goals and ensures compliance, consistency, refinement, and effective implementation.
Completion rate: 46%
Although governance documentation of business continuity and disaster recovery requirements and procedures is generally complete, there is no City policy requiring the business continuity plan program and monitoring of governance testing is not being performed yet.
Business Impact Analysis
Business impact analysis identifies and prioritizes operations so that business operations continue after a disruption.
Completion rate: 50%
Identification and prioritization of business operations have been created on a macro department level with all but two departments participating. A Citywide Business Impact Analysis at the service level has not been completed.
Workforce
Necessary workforce requirements ensure continued operations in the event of an emergency or service disruption (e.g. tornados, flood, cyber-attack). These requirements include verifying the alignment of job role functions, mapping these roles to skill set priorities, alignment of vendor tasks to enterprise job functions, and vendor service level agreements (SLAs).
Completion rate: 60%
The City of Tulsa's workforce requirements have not been full identified for continued operations in the event of an emergency or service disruption.
Location
Backup location requirements for the physical requirements and the notification systems are critical for continued operations in the event of an emergency or service disruption.
Completion rate: 67%
Not all location requirements for continued operations in the event of an emergency or service disruption have been identified. In several Business Continuity Plans, departments list that the Asset Management department will be finding the location for them.
Application Systems
Application systems requirements must be identified for continued operations in the event of an emergency or service disruption.
Completion rate: 69%
The City of Tulsa has demonstrated it has generally identified application systems requirements for continued operations in the event of an emergency or service disruption. The City has prioritized application systems and addressed source code executables for outsourced software. The workaround plans and version documentation system change, and release management are mostly in place. Instructions to recover the prior version of application/system are not always present for each change in the change management system. This could cause delays in recovering software and applications.
Emergency Preparedness and Communications
Backup, scenario, and workaround planning provides assurance that the plans are effective in continuing business operations.
Completion rate: 75%
This planning is generally sufficient. However, the personal contact information contained in the ERP system and in the Business Continuity Plans are incomplete and contain inaccuracies.
Business Continuity Planning, Scenario Planning, and Disaster Recovery Testing & Continuous Improvement & Reports - Incident and Problem Management
Completion rate: 0%
As of this review, testing has not been performed on any of the business continuity plans across the departments and therefore no continuous improvements or reporting is taking place.
Observations
The City of Tulsa does not have a written policy to create and maintain a business continuity plan. |
|
|
Two departments (former Mayor and City Council) did not complete business continuity plans in 2024. |
|
|
|
Departments did not prioritize functions or include a dependency criticality in their respective Business Continuity Plans. |
|
|
Several departments' Business Continuity Plans did not identify alternative locations. |
|
|
Business Continuity Plans do not list the functions that vendors perform. Vendors need to be listed for their risk ratings and prioritized. |
|
|
Departments were missing workaround documentation for applications that are in line with their priority ratings. |
|
|
The department business continuity plan call trees are incomplete and contain inaccuracies. (e.g. missing active employee information, including inactive employees, not including staffing agencies or temporary employees.) |
|
|
The call trees completed by Departments were incomplete and personal contact information found in our official employee record system was also found to have missing information. |
|
|
Instructions to recover the prior version of application/system are not always present for each change in the change management system. |
|
|