Maintaining Mission Critical Systems in a 24/7 Environment, 2nd Edition

Foreword xvii

Preface xix

Acknowlegments xxi

1 An Overview of Reliability and Resiliency in Today's Mission Critical Environment 1

1.1 Introduction 2

1.2 Risk Assessment 4

1.2.1 Levels of Risk 6

1.3 Capital Cost Versus Operation Cost 6

1.4 Critical Environment Workflow and Change Management 8

1.4.1 Change Management 9

1.4.2 Escalation Procedures 10

1.5 Testing and Commissioning 10

1.6 Documentation and the Human Factor 14

1.7 Education and Training 18

1.8 Operation and Maintenance 19

1.9 Employee Certification 20

1.10 Standards and Benchmarking 21

1.11 Conclusion 22

1.12 Risk Analysis and Improvement 22

2 Energy Security and Its Effect on Business Resiliency 25

2.1 Introduction 25

2.2 Risks Related to Information Security 29

2.3 How Risks are Addressed 34

2.4 Use of Distributed Generation 37

2.5 Documentation and Its Relation to Information Security 40

2.6 Smart Grid 42

2.7 Conclusion 44

2.8 Risk Analysis and Improvement 45

3 Mission Critical Engineering with an Overview of Green Technologies 47

3.1 Introduction 47

3.2 Companies’ Expectations: Risk Tolerance and Reliability 48

3.3 Identifying the Appropriate Redundancy in a Mission Critical Facility 50

3.3.1 Load Classifications 51

3.4 Improving Reliability, Maintainability, and Proactive Preventative Maintenance 52

3.5 The Mission Critical Facilities Manager and the Importance of the Boardroom 53

3.6 Quantifying Reliability and Availability 54

3.6.1 Re vie w of Reliability Terminology 55

3.7 Design Considerations for the Mission Critical Data Center 56

3.7.1 Data Center Certification 57

3.8 The Evolution of Mission Critical Facility Design 58

3.9 Human Factors and the Commissioning Process 59

3.10 Short-Circuit and Coordination Studies 60

3.10.1 Short-Circuit Study 60

3.10.2 Coordination Study 63

3.11 Introduction to Direct Current in the Data Center 65

3.11.1 Advantages of DC Distribution 65

3.11.2 DC Lighting 67

3.11.3 DC Storage Options 67

3.11.4 Renewable Energy Integration 68

3.11.5 DC and Combined Cooling, Heat, and Power 68

3.11.6 Current State of the Art 68

3.11.7 Safety Issues 70

3.11.8 Maintenance 71

3.11.9 Education and Training 71

3.11.10 Future Vision 71

3.12 Containerized Systems Overview 72

3.13 Conclusion 73

4 Mission Critical Electrical System Maintenance and Safety 81

4.1 Introduction 81

4.2 The History of the Maintenance Supervisor and the Evolution of the Mission Critical Facilities Engineer 83

4.3 Internal Building Deficiencies and Analysis 85

4.4 Evaluating Your System 86

4.5 Choosing a Maintenance Approach 87

4.5.1 Annual Preventive Maintenance 88

4.6 Safe Electrical Maintenance 89

4.6.1 Standards and Regulations 89

4.6.2 Electrical Safety: Arc Flash 90

4.6.3 Personal Protective Equipment (PPE) 92

4,6.4 Lockout/Tagout 98

4.7 Maintenance of Typical Electrical Distribution Equipment 99

4.7.1 Thermal Scanning and Thermal Monitoring 100

4.7.2 15 kV Class Equipment 102

4.7.3 480 Volt Switchgear 102

4.7.4 Motor Control Centers and Panel Boards 103

4.7.5 Automatic Transfer Switches 103

4.7.6 Automatic Static Transfer Switches (ASTS) 104

4.7.7 Power Distribution Units 105

4.7.8 277/480 Volt Transformers 105

4.7.9 Uninterruptible Power Systems 105

4.8 Being Proactive in Evaluating Test Reports 107

4.9 Conclusion 107

5 Standby Generators: Operations and Maintenance 109

5.1 Introduction 109

5.2 The Necessity for Standby Power 110

5.3 Emergency, Legally Required, and Optional Systems 111

5.4 Standby Systems That Are Legally Required 112

5.5 Optional Standby Systems 113

5.6 Understanding Your Power Requirements 113

5.7 Management Commitment and Training 113

5.7.1 Lockout/Tagout 114

5.7.2 Training 115

5.8 Standby Generator Systems Maintenance Procedures 115

5.8.1 Maintenance Record Keeping and Data Trending 116

5.8.2 Engine 116

5.8.3 Coolant System 117

5.8.4 Control System 117

5.8.5 Generator Mechanics 117

5.8.6 Automatic and Manual Switchgear 117

5.8.7 Load Bank Testing 118

5.9 Documentation Plan 118

5.9.1 Proper Documentation and Forms 118

5.9.2 Record Keeping 119

5.10 Emergency Procedures 119

5.11 Cold Start and Load Acceptance 120

5.12 Nonlinear Load Problems 121

5.12.1 Line Notches and Harmonic Current 121

5.12.2 Step Loading 121

5.12.3 Voltage Rise 122

5.12.4 Frequency Fluctuation 122

5.12.5 Synchronizing to Bypass 122

5.12.6 Automatic Transfer Switch 123

5.13 Conclusion 123 

6 Fuel Systems Design and Maintenance 125

6.1 Introduction 125

6.2 Brief Discussion on Diesel Engines 126

6.3 Bulk Storage Tank Selection 126

6.3.1 Aboveground Tanks 127

6.3.2 Modern Underground Tanks and Piping Systems 128

6.4 Codes and Standards 128

6.5 Recommended Practices for All Tanks 129

6.6 Fuel Distribution System Configuration 133

6.7 Day Tank Control System 135

6.8 Diesel Fuel and A Fuel Quality Assurance Program 139

6.8.1 Fuel Needs and Procurement Guidelines 141

6.8.2 New Fuel Shipment Prereceipt Inspection 141

6.8.3 Analysis of New Fuel Prior to Transfer to On-Site Storage 144

6.8.4 Monthly Fuel System Maintenance 145

6.8.5 Quarterly or Semiannual Monitoring of On-Site Bulk Fuel 146

6.8.6 Remediation 146

6.9 Conclusion 148

7 Power Transfer Switch Technology, Applications, and Maintenance 149

7.1 Introduction 149

7.2 Transfer Switch Technology and Applications 151

7.3 Types of Power Transfer Switches 152

7.3.1 Manual Transfer Switches 152

7.3.2 Automatic Transfer Switches 153

7.4 Control Devices 163

7.4.1 Time Delays 163

7.4.2 ln-Phase Monitor 164

7.4.3 Test Switches 165

7.4.4 Exercise Clock 165

7.4.5 Voltage and Frequency Sensing Controls 166

7.5 Design Features 166

7.5.1 Close Against High In-Rush Currents 166

7.5.2 Withstand and Closing Rating (WCR) 167

7.5.3 Carry Full Rated Current Continuously 167

7.5.4 Interrupt Current 167

7.6 Additional Characteristics and Ratings of ATS 167

7.6.1 NEMA Classification 167

7.6.2 System Voltage Ratings 168

7.6.3 ATS Sizing 168

7.6.4 Seismic Requirement 168

7.7 Installation and Commissioning, Maintenance, and Safety 168

7.7.1 Installation and Commissioning 168

7.7.2 Maintenance and Safety 170

7.7.3 Maintenance Tasks 173

7.7.4 Drawing and Manuals 173

7.7.5 Testing and Training 173

7.8 General Recommendations 176

7.9 Conclusion 176

8 Static Transfer Switch 179

8.1 Introduction 179

8.2 Overview 180

8.2.1 Major Components 180

8.3 Typical Static Switch, One-Line Diagram 181

8.3.1 Normal Operation 181

8.3.2 Bypass Operation 182

8.3.3 STS and STS/Transformers Configurations 183

8.4 STS Technology and Application 183

8.4,1 General Parameters 184

8.4.2 STS Location and Type 184

8.4.3 Advantages and Disadvantages of the Primary and Secondary STS/Transformer Systems 184

8.4.4 Monitoring, Data Logging, and Data Management 184

8.4.5 Downstream Device Monitoring 185

8.4.6 STS Remote Communication 185

8.4.7 Security 186

8.4,8 Human Engineering and Eliminating Human Errors 187

8.4.9 Reliability and Availability 187

8.4.10 Repairability and Maintainability 189

8.4.11 Fault Tolerance and Abnormal Operation 189

8.5 Testing 190

8.6 Conclusion 190

9 Fundamentals of Power Quality 193

9.1 Introduction 193

9.2 Electricity Basics 195

9.2.1 Basic Circuit 196

9.2.2 Power Factor 196

9.3 Transmission of Power 197

9.3.1 Life Cycle of Electricity 198

9.3.2 Single-Phase and Three-Phase Power Basics 199

9.3.3 Unreliable Power versus Reliable Power 201

9.4 Understanding Power Problems 202

9.4.1 Power Quality Transients 202

9.4.2 RMS Variations 204

9.4.3 Causes of Power Line Disturbances 207

9.4.4 Power Line Disturbance Levels 212

9.5 Tolerances of Computer Equipment 212

9.5.1 CBEMA Curve 214

9.5.2 ITIC Curve 215

9.5.3 Purpose of Curves 215

9.6 Power Monitoring 215

9.7 The Deregulation Wildcard 217

9.8 Conclusion 221

10 UPS Systems: Applications and Maintenance with an Overview of Green Technologies 223

10.1 Introduction 223

10.1.1 Green Technologies and Reliability Overview 223

10.2 Purpose of UPS Systems 225

10.3 General Description of UPS Systems 228

10.3.1 What is a UPS System? 228

10.3.2 How Does a UPS System Work? 228

10.3.3 Static UPS Systems 229

10.3.4 Online 230

10.3.5 Double Conversion 230

10.3.6 Double Conversion UPS Power Path 231

10.4 Components of a Static UPS System 232

10.4.1 Power Control Devices 232

10.5 Online Line Interactive UPS Systems 238

10.6 Offline (Standby) 239

10.7 The Evolution of Static UPS Technology 240

10.7.1 Emergence of the IGBT 240

10.7.2 Two- and Three-Level Rectifier/Inverter Topology 241

10.8 Rotary UPS Systems 242

10.8.1 UPSs Using Diesel 243

10.8.2 Hybrid UPS Systems 244

10.9 Redundancy, Configurations, and Topology 245

10.9.1 N Configuration 245

10.9.2 N+1 Configuration 246

10.9.3 Isolated Redundant Configuration 246

10.9.4 N+2 Configurati on 246

10.9.5 2N Configuration 248

10.9.6 2N+ 1 Configuration 248

10.9.7 Distributed Redundant/Catcher UPS 249

10.9.8 Eco-Mode for Static UPS 249

10.9.9 Availability Calculations 250

10.10 Energy Storage Devices 251

10.10.1 Battery 251

10.10.2 Flywheel Energy 255

10.11 UPS Maintenance and Testing 256

10.11.1 Physical Preventive Maintenance (PM) 257

10,11,2 Protection Settings, Calibration, and Guidelines 258

10.11.3 Functional Load Testing 258

10.11.4 Steady-State Load Test 259

10.11.5 Steady-State Load Test at 0%, 50%, and 100% of Load 259

10.11.6 Harmonic Analysis and Testing 259

10.11.7 Filter Integrity and Testing 260

10.11.8 Transient Response Load Test 261

10.11.9 Module Fault Test 261

10.11.10 Battery Rundown Test 261

10.12 Static UPS and Maintenance 262

10.13 UPS Management 262

10.14 Conclusion 263

11 Data Center Cooling Systems 265

11.1 Introduction 265

11.2 Background Information 266

11.3 Cooling within Datacom Rooms 266

11.4 Cooling Systems 267

11.4,1 Air Side 267

11.4.2 Cooling-Medium Side 267

11.5 Components Outside the Datacom Room 269

11.5.1 Refrigeration Equipment—Chillers 269

11.5.2 Heat-Rejection Equipment 273

11.5.3 Energy-Recovery Equipment 282

11.5.4 Heat Exchangers 287

11.6 Components Inside the Datacom Rooms 290

11.6,1 CRAC Units 290

11.7 Conclusion 295

12 Data Center Cooling Efficiency: Concepts and Advanced Technologies 297

12.1 Introduction 297

12.1.1 Data Center Efficiency Measurement

12.2 Heat Transfer Inside Data Centers 300

12.2.1 Heat Generation 301

12.2.2 Heat Return 302

12.2.3 Cooling Air 302

12.3 Cooling and Other Airflow Topics 303

12.3.1 Leakage 303

12.3.2 Mixing and Its Relationship to Efficiency 303

12.3.3 Recirculation 303

12.3.4 Venturi Effect 304

12.3.5 Vortex Effect 304

12.3.6 CRAC/CRAH Types 304

12.3.7 Potential CRAC Operation Issues 305

12.3.8 Sensible Versus Latent Cooling 305

12.3.9 Humidity Control 307

12.3.10 CRAC Fighting—Too Many CRACs 308

12.4 Design Approaches for Data Center Cooling 308

12.4.1 Hot Aisle/Cold Aisle 308

12.4.2 Cold-Aisle Containment 309

12.4.3 In-Row Cooling with Hot-Aisle Containment 309

12.4.4 Overhead Supplemental Cooling 309

12.4.5 Chimney or Ducted Returns 310

12.4.6 Advanced Active Airflow Management for Server Cabinets 310

12.5 Additional Considerations 310

12.5.1 Active Air Movement 310

12.5.2 Adaptive Capacity 311

12.5.3 Liquid Cooling 311

12.5.4 Cold Storage 312

12.6 Hardware and Associated Efficiencies 312

12.6.1 Server Efficiency 312

12.6.2 Server Virtualization 313

12.6.3 Multicore Processors 313

12.6.4 Blade Servers 313

12.6.5 Energy-Efficient Servers 313

12.6.6 Power Managed Servers 313

12.6.7 Effects of Dynamic Server Loads on Cooling 313

12.7 Best Practices 314

12.8 Efficiency Problem Solving 314

12.9 Conclusion 316

12.10 Conversions, Formulas, Guidelines 316

13 Raised Access Floors 317

13.1 Introduction 317

13.1.1 What is an Access Floor? 317

13.1.2 What are Typical Applications for Access Floors? 318

13.1.3 Why Use an Access Floor? 319

13.2 Design Considerations 319

13.2.1 Determine the Structural Performance Required 320

13.2.2 Determine the Required Finished Floor Height 322

13.2.3 Determine the Understructure Support Design Type Required 323

13.2.4 Determine the Appropriate Floor Finish 325

13.2.5 Airflow Requirements 326

13.3 Safety Concerns 328

13.3.1 Removal and Reinstallation of Panels 328

13.3.2 Removing Panels 328

13.3.3 Stringer Systems 330

13.3.4 Protection of the Floor from Heavy Loads 331

13.3.5 Grounding the Access Floor 336

13.3.6 Fire Protection 337

13.3.7 Zinc Whiskers 337

13.4 Panel Cutting 328

13.4.1 Safety Requirements for Cutting Panels 328

13.4.2 Guidelines for Cutting Panels 328

13.4.3 Cutout Locations in Panel—Supplemental Support for Cut Panels 338

13.4.4 Saws and Blades for Panel Cutting 339

13.4.5 Interior Cutout Procedure 339

13.4.6 Round Cutout Procedure 339

13.4.7 Installing Protective Trim Around Cut Edges 340

13.4.8 Cutting and Installing the Trim 340

13.5 Access Floor Maintenance 340

13.5.1 Best Practices for Standard High-Pressure Laminate Floor Tile (HPL) and for Vinyl Conductive and Static Dissipative Tile 341

13.5.2 Damp Mopping Procedure for HPL and Conductive and Static Dissipative Vinyl Tile 342

13.5.3 Cleaning the Floor Cavity 342

13.6 Troubleshooting 343

13.6.1 Making Pedestal Height Adjustments 343

13.6.2 Rocking Panel Condition 343

13.6.3 Panel Lipping Condition (Panel Sitting High) 343

13.6.4 Out-of-Square Stringer Grid (Twisted Grid) 344

13.6.5 Tipping at Perimeter Panels 345

13.6.6 Tight Floor or Loose Floor—Floor Systems Laminated with HPL Tile 345

13.7 Additional Design Considerations 346

13.7.1 LEED Certification 346

13.7.2 Energy Efficiency—Hot and Cold Air Containment 346

13.7.3 Airflow Distribution and CFD Analysis 347

13.8 Conclusion 354

14 Fire Protection in Mission Critical Infrastructures 357

14.1 Introduction 357

14.2 Philosophy 358

14.3 Alarm and Notification 359

14.4 Early Detection 361

14.5 Fire Suppression 362

14.6 System Designs 364

14.6.1 Stages of a Fire 364

14.6.2 Fire and Building Codes 365

14.7 Fire Detection 366

14.8 Fire Suppression Systems 374

14.8.1 Water Mist Systems 379

14.8.2 Carbon Dioxide Systems 382

14.8.3 Clean Agent Systems 384

14.8.4 Inert Gas Agents 384

14.8.5 IG-541 385

14.8.6 IG-55 385

14.8.7 Chemical Clean Agents 386

14.8.8 Portable Fire Extinguishers 390

14.8.9 Clean Agents and the Environment 390

14.8.10 Conclusion 391

Appendix A Policies and Regulations 393

A.1 Introduction 393

A.2 Industry Policies and Regulations 395

A.2.1 USA PATRIOT Act 396

A.2.2 Sarbanes-Oxley Act (SOX) 397

A.2.3 Comprehensive Environmental Response, Compensation, and Liability Act of 1980 399

A.2.4 Executive Order 13423—Strengthening Federal Environmental, Energy, and Transportation Management 399

A.2.5 IS027000 Information Security Management Systems (ISMS) 400

A.2.6 The National Strategy for the Physical Protection of Critical Infrastructure and Key Assets 403

A.2.7 2009 National Infrastructure Protection Plan 404

A.2.8 North American Electric Reliability Corporation (NERC) Critical Infrastructure Protection Program 405

A.2.9 U.S. Security and Exchange Commission (SEC) 405

A.2.10 Sound Practices to Strengthen the Resilience of the U.S, Financial System 405

A.2.11 C⁴I—Command, Control, Communications, Computers, and Intelligence 407

A.2.12 Basel II Accord 408

A.2.13 National Institute of Standards and Technology (NIST) 408

A.2.14 Business Continuity Management Agencies and Regulating Organizations410

A.2.15 FFIEC—Federal Financial Institutions Examination Council 412

A.2.16 National Fire Prevention Association 1600 Standard on Disaster/Emergency Management and Business Continuity Programs 412

A.2.17 Private Sector Preparedness Act 414

A.3 Data Protection 414

A.4 Encryption 416

A.4.1 Protecting Critical Data through Security and Vaulting 417

A.5 Business Continuity Plan (BCP) 417

A. 6 Conclusion 419

Appendix B Consolidated List of Key Questions 421

Appendix C Airflow Management: A Systems Approach 441

C.l Introduction 441

C.2 Control is the Key 442

C.2.1 Benefits of Control 444

C.3 Obtaining Control 445

C.3.1 Lower Return Air ΔT Versus Higher ΔT 445

C.4 Air Management Technologies 451

C.4.1 In-Row Cooling 452

C.4.2 Overhead Cooling 452

C.4.3 Containment Strategies 452

C.4.4 Active-Air Management 453

C.4.5 A Benchmark Study for Comparison 454

C.5 Conclusion 456

Glossary 459

Bibliography 473

Index 479