To: Board of Supervisors
Department or Agency Name(s): Information Systems
Staff Name and Phone Number: John Hartwig 565-6055
Vote Requirement: 4/5th
Supervisorial District(s): Countywide
Title:
Title
Information Technology Resiliency Program
End
Recommended Actions:
Recommended action
Receive an update on Information Technology Recovery and Resiliency efforts to date and establish an Information Technology (IT) Resiliency Program and authorize use of $1.7 million from the FY 2018-19 Disaster & Recovery General Fund Reserves established by the Board of Supervisors for information technology resiliency efforts. (4/5th Vote Required)
end
Executive Summary:
The 2017 October Disaster and proximity to the County Campus highlighted the potential loss of buildings, infrastructure, and equipment which could have meant the loss of the vast majority of information technology systems and data and the inability to provide critical functions and services. In response, during the FY 2018/19 Budget Hearings, the Board approved redirection of $1.7 million in General Fund Reserves for information technology recovery and resiliency efforts.
The IT Resiliency Program will assess, develop and implement solutions to mitigate the risks to the IT environment. The department is proposing to establish ongoing IT Resiliency Program efforts, using the $1.7 million established by the board for information technology resiliency efforts including implementation of a more resilient server architecture, a redesign of the network connectivity architecture, issuance of Request for Proposals for Disaster Recovery/Business Continuity Planning Consultant, and a Request for Proposal for consulting services to review alternatives and develop detailed plans to transition the County from the Microsoft Office tools to new cloud based productivity and collaboration suites, and funding for Information Systems Department resources.
Discussion:
Pursuant to the adopted Sonoma County Recovery & Resiliency Framework Goal C4, Information Systems Department is actively working to address the following risks which would affect emergency response and continuity of operations:
• Loss of network connectivity.
• Loss of core productivity applications including email, electronic documents, and collaboration tools.
• Loss of enterprise and line of business applications including 911 and Computer Aided Dispatch, financial systems, and County websites.
• Loss of electrical power and Internet carrier services to the ISD Data Center that would impact services.
Pursuant to the adopted Sonoma County Recovery & Resiliency Framework Goal C4, the IT Resiliency Program will:
• Assess departmental Continuity of Operations Plans to determine IT Resiliency expectations for the various business applications in use throughout the County.
• Prepare Business Impact Analysis (BIA) for County technology infrastructure and community infrastructure and utilities.
• Prepare Notice of Intent (NOI) and sub-applications under the Hazard Mitigation Grant Program (HMGP) for critical facilities, including power and Internet redundancy and wireless/cellular communications hardening.
Recovery & Resiliency Accomplishments
Over the past year, Information Systems has taken specific actions to support recovery and resiliency within existing appropriations. The actions intend to mitigate identified gaps in securing County data and establishing baseline recovery sites that can help in recovery scenarios consistent with the Recovery Framework. The primary accomplishments include:
• Supported emergency response and ongoing recovery and resiliency efforts for the Emergency Operations Center and Office of Recovery & Resiliency.
• Prioritized IT Disaster Planning using existing staff.
• Implemented new cloud based public website for emergency information designed for high traffic and optimized for mobile devices.
• Entered into a Memorandum of Understanding (MOU) with County of Alameda for data center space and replicated data that will enhance offsite protection of data and potentially allow for recovery of operations.
• Improved data protection by establishing connections to cloud providers and implementing daily replication of critical data.
• Expedited purchase and installation of hardware to support virtualization of remaining physical server roles to better enable applications portability.
• Contracted for services to migrate SharePoint data a web based collaborative tool that integrates with current enterprise software, to a cloud provider to improve resiliency.
• Completed HMGP sub-application for Data Center Generator which has been approved by California Department of Emergency Services (CALOES), Federal Emergency Management Agency (FEMA) review pending.
• Completed initial concept designs for future network and data center infrastructure architecture to improve reliability, availability, and stability of systems and data.
Recovery & Resiliency - The Future and Next Steps
As described above, the Information Systems Department has made strides to mitigate risks and improve the recovery and resiliency footing of the information technology environment by reprioritizing our existing resources. To continue the progress made and accomplish the goals outlined in the adopted Sonoma County Recovery & Resiliency Framework Goal C4 will require additional investment financed with the $1.7 million set aside established by the Board during the FY 2018/19 budget hearings.
The Information Technology Resiliency Program proposes the following:
1. Development and issuance of a Request for Proposal for IT Disaster Recovery/Business Continuity Consultant to a) assess current and future risks to County IT infrastructure and facilities b) evaluate proposed technical designs including network redesign and utilization of cloud based office productivity and collaboration suite (e.g. to replace Microsoft Office on-site systems) for IT service resiliency, as well as to offer additional options, c) provide organizational recommendations for a sustained program in Business Continuity to ensure effective emergency response, continuity of government, and minimal disruption to public services and d) assess staffing and use of limited resources
Development and issuance of a Request for Proposal for consulting services to review alternatives and develop detailed plans to transition the County from the Microsoft Office tools to new cloud based productivity and collaboration suites to a) evaluate existing data volume, infrastructure and licensing b) recommend licensing option and forecast future costs c) prepare migration plan and training schedule and d) migrate data and provide post-migration support.
2. Implement resilient server architecture by virtualizing critical server, storage, and database structures to allow movement of service delivery between data centers and cloud providers. Implementation will allow the shifting of workloads (compute, storage, application hosting) to available resource based on disaster scenario. This will improve dramatically the time-to-recovery in the event of Data Center failure. Resilient server architecture to be funded through Infrastructure Replacement funds.
3. Redesign network connectivity architecture to ensure access to Internet and to applications running offsite by adding circuits to ensure diverse paths and automated rerouting of traffic. Implementing this design would allow off-campus remote sites to continue to access Internet and cloud services even if the ISD Data Center and County campus were unusable. This is also a requirement to reliably shift enterprise and line of business solutions to cloud service providers.
4. Approve funding for Information System Department using existing allocated positions to a) engage with Disaster Recovery/Business Continuity and cloud based office productivity and collaboration suite consultants; b) prepare technical designs to improve recovery and resiliency; c) assess the departments’ and county agencies’ completed Continuity of Operations Plans to determine IT Resiliency expectations for the various business applications in use throughout the County; d) prepare Business Impact Analysis (BIA) for County technology infrastructure and community infrastructure and utilities; and e) prepare Notice of Intent (NOI) and sub-applications under the Hazard Mitigation Grant Program (HMGP) for IT Infrastructure. Approximately 4800 hours of ISD staff time will be required to work with consultants, departments, and implement anticipated resiliency activities
Prior Board Actions:
12/11/18 - Board approved the Recovery and Resiliency Framework
6/15/18 - FY 2018/19 Budget Hearing Actions: Board established $1.7 million in General Fund Reserves for Information Technology Continuity of Operations.
5/22/2018-Memorandum of Understanding with County of Alameda for Reciprocal Data Center Space
Fiscal Summary
|
Expenditures |
FY 18-19 Adopted |
FY19-20 Projected |
FY 20-21 Projected |
|
Budgeted Expenses |
|
$1,700,000 |
|
|
Additional Appropriation Requested |
|
|
|
|
Total Expenditures |
|
$1,700,000 |
|
|
Funding Sources |
|
|
|
|
General Fund/WA GF |
|
|
|
|
State/Federal |
|
|
|
|
Fees/Other |
|
|
|
|
Use of Fund Balance - GF Reserves |
|
$1,700,000 |
|
|
Contingencies |
|
|
|
|
Total Sources |
|
$1,700,000 |
|
Narrative Explanation of Fiscal Impacts:
Funding for information technology resiliency efforts and use of the $1.7 million set aside established by the Board of Supervisors during FY 2018/19 budget hearings will be requested as part of the FY 2019/20 supplemental budget process. Expenditure of funds will be phased and subject to approval from the CAO.
|
Staffing Impacts: |
|
|
|
|
Position Title (Payroll Classification) |
Monthly Salary Range (A - I Step) |
Additions (number) |
Deletions (number) |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Narrative Explanation of Staffing Impacts (If Required):
Allocated positions, filled and vacant, will be utilized to provide the staff time required to complete these work efforts.
Attachments:
Click or tap here to enter text.
Related Items “On File” with the Clerk of the Board:
Click or tap here to enter text.