вход по аккаунту

код для вставкиСкачать
Business Continuity Planning
In a Post-Katrina Environment
May 2006
Brian D. Voss
Chief Information Officer
Enterprise 2006—The EDUCAUSE Enterprise Technology Conference
Copyright Brian D. Voss, 2006. This work is the intellectual property of the author. Permission is granted for this material to
be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced
materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish
requires written permission from the author.
A few caveats
• I was not at the center of Katrina
– John Lawson, my colleague from Tulane; Jim Burgard, my colleague from
University of New Orleans; Dave Troendle, my colleague from LSU
Health Science Center NO were … and they have stories to tell
– Our IT colleagues at Xavier, Loyola, Dillard, Southern (NO), and
Southeastern, in the Louisiana Technical and Community Colleges, and
those in the states of Mississippi, Alabama, and Texas impacted by
Katrina and Rita.
– Clifford Woodruff at Lamar University, who rode out Rita, has some
excellent insights to share; ask him
I was on this disaster’s edge; they were at its center.
• My campus – LSU A&M in Baton Rouge was not subject to any
devastation or flooding. We were, however, impacted greatly by
the aftermath of the storm
Don’t call me Doctor …
• I am not an expert on disaster recovery
– But I am learning more and faster than I could
have imagined
• I don’t have all the answers
– But I sure have thought about a lot of questions
and some possible answers
• LSU does not have a ‘model’ BCP to
– But we’re working on one with urgency
A Little History from 2005:
Katrina’s Impacts on Louisiana Higher Ed
Basically, those in Katrina’s path were put
temporarily out of business
• UNO/LSUHSC in New Orleans
– portions of campus under water
– classes resumed Oct-05 at Jefferson Facility
– Resumed operations in NO Spring Semester 06
• Tulane, Xavier, Dillard & Loyola
– Campuses all impacted to one degree or another
– Established temporary admin HQs elsewhere
– Resumed operations in NO Spring Semester 06
Katrina’s Impacts on Louisiana
Higher Ed
• Louisiana Technical Colleges & Community
– Lost several NO-based campuses; serving students at
other campuses that were less (or not at all) impacted
• Louisiana State University
– Became perhaps the most critical facility in support of
disaster relief/response in the State of Louisiana
– Despite this, continued to function as the State’s
Flagship Research University; added over 3,000
temporary/displaced students; serve over 9,000
students from areas impacted by Katrina & Rita
Katrina Impact Continues:
Smaller Classes in New Orleans
The Chronicle of Higher Education May 26 issue
Story by Jeffrey Selingo
Katrina Impact Continues:
Smaller Classes in New Orleans
The Chronicle of Higher Education May 26 issue
Story by Jeffrey Selingo
A word about New Orleans
Don’t believe all of what you see in the media – good or
bad; you need to see it to understand
• The city is about 40% its pre-Katrina population
• Whole areas – many square miles – are not inhabited
and little or no recovery has occurred
• Blue tarps, FEMA trailers, and abandoned vehicles
abound; storm damage is still evident, even downtown
• The French Quarter is pretty much the same as it was
and tourism has resumed
• Optimism and hope fight a daily battle with despair
• It will take a decade or more for the city to become
whatever it will become – but it will never be like it was
before (which may have a hopeful aspect)
Post-Katrina Disaster Recovery
• Traditional Disaster Recovery
– What if my data center is lost
• Broader Disaster Recovery
– What if my campus is lost
– What if the city where my campus is located is lost
• Survivor Disaster Recovery
– What if I’m fine … but everyone around me is not
Traditional Disaster Recovery
You’re down, everything else is fine
• Do you have a workable DR plan?
• Do you know where on campus you’ll go?
• Did you take necessary back-ups and do you have
them ready to re-produce production files?
• What vendors will you need to tap – and for what?
• How will you quickly re-establish network
connectivity? Phone service? Web presence? Email? Mission critical information systems?
Traditional Disaster Recovery
You’re down, everything else is fine
• Lessons:
– It’s the data, stupid. Hardware can be
replaced; data can not be
– If you lose just your data center, the need
for rapid response will be acute and
immediate; your institution can not operate
without IT and it will notice it (maybe for the
first time)
– The CIO will be definitely on the hot seat;
job at risk
Your Views/Experiences?
• Do you have a “traditional” DR Plan?
– Are you building one?
• Have you had to use it? How’d that go?
• Do you ‘drill’ on this plan?
• What has been your experience with
your administration’s responsiveness to
– Has that changed since 9-11, or Katrina?
Broader Disaster Recovery
You and everyone around you is down
• Are your off-sites conveniently (and perhaps
tragically) close?
• Do you have arrangements to get key services
restored at a distance
– Web, E-mail, Financial/HR, Student Information, CMS
• Hot-sites may be too much $$$$ – but can you find
suitable raised floor/HVAC/power to ‘re-build’
• Can you support your administration “in exile?”
– Internet access, computers, cell phones, e-mail, IM
• Is your ‘life-boat’ plan portable over larger distances?
• Can you grab your key people? Can you care for
Broader Disaster Recovery
You and everyone around you is down
• Lessons:
– People are your most key resource – but
expect them to be burdened with other
– Knowing what you’ll need to do and having it
organized is more important than knowing all
about ‘how’ you’ll do it when you get there
(wherever ‘there’ might be)
– Once again, the CIO will be on the hot seat as
the institution realizes just how dependent it is
upon IT
One Possible Tool In The Arsenal:
Data Center Lifeboat
• Situation: What if we had very short
notice (4-8 hours) notice of the need to
abandon our data center/campus and
set-up elsewhere (>50miles away)
• Goal #1: Re-establish some critical
subset of services
• Goal #2: Support the re-establishment
of some subset of university
• Key things to recover:
• Key things to address
– Payroll/Financial Data
– Web presence
– Off-site storage of critical back-ups
– Ability to ‘grab and go’ key data and
• Splash/priority information
– List of key hardware needed later
• As much content as possible
from vendors
E-mail service for
– Disaster Supplies Crate
• What would we put into an 8x12
truck for rapid evac?
Portal interface
– Equipment for a mobile or relocated
Student Information Systems
university command post
HR, Procurement Systems
• Laptops, radios, phones, etc.
– Identify Key IT personnel
What else?
• Budgets ($25K, $50K, $100K)
• Who does what w/back-up
• “Scoop ‘em up”
– Where might we go?
Your Views/Experiences?
• Does your IT DR plan included a
‘broader impact’ consideration? Or
Does your Institution have such a plan
– Has that changed since 9-11 or Katrina?
• Have you had to use it? How’d that go?
• How do you think your Administration
would respond to this type of broader
Survivor Disaster Recovery
You’re the last ones standing
• Dealing with unimaginable demands
– Start imagining it
• Do you have a stock of equipment to set up a large
support operation in short-order?
– Networking gear, computers, cables, supplies, telephone service
• Value of a flexible and capable staff
– They’ll see things no one should have to see
• Consider how you’ll do all this on top of your normal
jobs, as campus life resumes and student enrollment
• How ready is your campus administration to take on the
role of disaster response center?
– Facilities, public safety/police, communications, academic affairs
– Is the CEO (Chancellor, or President) prepared?
Lessons Learned at LSU
• Have a good stock of networking equipment,
and mobile and desktop computing in the
– Plan to raid campus labs & empty desks if need be
• Have strong relationships with key vendors
• Great to have terrific, dedicated, service-oriented
• Architectures count – how divisible are the
components? What’s removable as a
component, and what’s too-tightly integrated?
• Be prepared to be flexible; adapt, improvise,
– Don’t be thin-skinned
Lessons Learned continued
• Good to “sit at the big table”
– But know when to speak, and when not to – if you’ve
watched ‘Survivor’ you know what I mean
• Keep your friends close – and don’t have
• Everything we’ve been saying about the
strategic value of IT is valid; IT enables
everything in the 21st Century
– But even now – does HE administration ‘get it?’
• Disaster Recovery and Business Continuity
Planning is not a luxury
Your Views/Experiences?
• Have you ever been on the edge of
someone else’s disaster?
• Does your DR/BCP plan include this
aspect? Does your institution have
such a plan component?
– Has that changed since 9-11 or Katrina?
• How do you think your Administration
would respond to this type of broader
Hurricane Season 2006 Starts
… next week!
• We’re working on a basic, nimble IT DR/BCP document
– will be done by 5/31
– Addressing all three forms of disaster
– Requesting funds ~$500K
– Primed the pump with $100K from year-end cash
• Campus is constructing a ‘permanent’ EOC
– Rather than relying upon hastily assembled one
– Spending ~$150K to do it
• Chancellor put out a call for DR/BCP plans across key
campus units
– Met with EOC commander to review plans and coordinate
IT DR/BCP at LSU – Post-Katrina
Before 29-August 2005
• Information Technology Services (ITS) has always
made backups of data on its servers
- Mainframe (daily, stored off-site in Port Allen)
- Lots of non-mainframe servers (housed in BR)
• Fairly strong, redundant network topology
• Computer Center has a UPS, generator, chillers
• Stand-by or parallel servers in separate building on
IT DR/BCP at LSU – Post-Katrina
After 15-September 2005
• Organizational focus/emphasis on DR/BCP
- Creates IT DR/BCP Officer; within IT Policy &
Security Office
• Started to update disaster recovery plan
– The ‘old’ formal one was circa 1984 (useless)
• Double-checked Payroll contingency
- Able to meet payroll deadlines without mainframe
IT DR/BCP at LSU – Post-Katrina
More work
• Long distance offsite storage --- Houston,
Birmingham, Nashville, Dallas?
– Security of tapes with Breach Notification Law
• Establish LSU Rapid Recovery Site
- 100 + miles, not south
- Relocate servers
- Increase stand-by presence
• Formal hot-site contract for mainframe
• E-mail emergency service contract
Some Key Items to Address
Conduct a risk evaluation and business impact
Define and prioritize your mission-critical systems.
What must be recovered immediately (within 24 hours)?
What can wait (and how long)?
Identify your backup/recovery site.
Vendors provide offsite storage of mission-critical backup
tapes, remote data centers, and temporary office locations.
Consider co-sourcing or reciprocal agreements with other
regional higher education institutions or for facility and
equipment use.
Develop a plan with your key hardware vendors to rapidly
replace any damaged hardware/communication systems.
Some Key Items to Address
Develop and document a communications and
contact plan.
The Internet can be a crucial external communications tool.
Designate and equip a central command and
communication center. Who will be primary spokesperson
to respond to questions, as well as how information will be
Be wary of relying on wireless.
Cellular circuits can quickly become overloaded and
unavailable during a regional or national incident.
At your centralized command and communication center,
use a variety of communication links (Web, cellular, fax,
landline, radio, and sticky-note bulletin boards).
Key Items to Address in a DRP
• Don’t forget about the people side of your institution.
– Do you know where your staff, faculty, and students are? Do
you have a tracking or check-in system?
– Who is on your IT Emergency Response Team? How will you
communicate with them?
– Do you need temporary offices, temporary classrooms, or
temporary housing?
• Finally, document and distribute your plan (including
hard copies).
– Test it, evaluate it, fix it, and retest it.
– Do this at least annually, as well as after major system or
infrastructure upgrades.
I met a traveler from an antique land
Who said: Two vast and trunkless legs of stone
Stand in the desert. Near them, on the sand,
Half sunk, a shattered visage lies, whose frown,
And wrinkled lip, and sneer of cold command,
Tell that its sculptor well those passions read,
Which yet survive, stamped on these lifeless things,
The hand that mocked them, and the heart that fed,
And on the pedestal these words appear:
"My name is Ozymandias, King of Kings:
Look upon my works, ye Mighty, and despair!“
Nothing beside remains. Round the decay
Of that colossal wreck, boundless and bare
The lone and level sands stretch far away.
-Percy Bysshe Shelley
Thoughts 8 months after
In a disaster, do the rules – and the plan
– go out the window?
• Isn’t the very nature of disaster its
• Can we ever plan for every possible event
and circumstance?
• During the crisis will we have time to refer
to a detailed disaster plan document?
What then should we do?
• Focus on the process of planning and not on
the plan itself (so sayeth Capt. Joe Castillo USCG)
• Examine how we will position ourselves and our
assets to be flexible in responding to a disaster
• Focus on knowing what will need to be done in
the first stages, what we’ll need to do those
things, and who will do them
• Plan to be flexible. Plan to improvise, adapt,
and overcome
• Drill on these things
Your Views?
• Do you think having a DR/BCP matters,
or is ad-hoc the best way to handle
• How much is too much to spend on
– From your perspective
– From the viewpoint of your Administration
Again, What’s Important?
• Hardware and facilities can be replaced
in the periods following a disaster
• Data is the primary focus of what you
need to be prepared to restore and the
basis of continuity
• People are your most key asset
IT Personnel = First Responders
• Everyone on campus uses and relies
upon IT today (whether they fully realize
it or not)
• If Anyone is on campus, they’ll need IT
(and thus need IT support)
• Ergo, if someone is on campus during a
disaster, they’ll need IT and IT
personnel on campus also.
Your Views?
• Do you think this is true – that IT is now
a ‘first responder?’
• Is your organization (i.e., staff) ready to
accept and fulfill this role?
• Are you ready for them to do this – are
you (as a CIO or IT leader) prepared to
deal with this aspect of our role?
“We had a failure of imagination.”
• Lessons from NASA; Apollo 1 and Columbia Shuttle
• We need to imagine the questions first so that we can
find the answers
• We need to – as a community – seek answers together
– How can we leverage national cyberinfrastructure?
– Individual arrangements versus broader approaches
• How seriously do CIOs, take the strategic nature of IT?
How about your administration?
– IT truly is an enabler of everything we do now
– Are our people ready to be First Responders?
Those who cannot learn from
history are doomed to repeat it.
-George Santayana
• CIOs can no longer say they can’t imagine what
could happen – because it just did.
– Or an earthquake, or a tsunami, or a terrorist attack,
or an accident, or a pandemic
• Next time, you may not be watching it on CNN –
you may be living it
• Now is the time to think, plan, and take action –
later it will be too late
Business Continuity Planning
In a Post-Katrina Environment
May 2006
Brian D. Voss
Chief Information Officer
Пожаловаться на содержимое документа