Chapter 61 Β· Examples

Infrastructure Considerations in Practice

MTTR cost analysis, elasticity security blind spots, cybersecurity insurance claims, unpatched embedded device attacks, and power failure recovery walkthroughs.

Example 1 Β· MTTR Cost Analysis β€” Why Resilience Investment Pays Off

A regional bank's core banking application goes offline on a Friday afternoon. The root cause is a failed SSD in the primary database server. The IT team must determine: how quickly can they restore service, and what does the downtime cost?

Scenario A β€” No pre-planned resilience (cold spare, no HA):

2:00 PM β€” Database server fails; banking application goes offline
2:05 PM β€” Alert fires; on-call engineer paged
2:20 PM β€” Engineer identifies failed SSD (root cause: 15 min)
2:20 PM β€” Spare SSD not in stock; orders emergency delivery (2-hour wait)
4:30 PM β€” Replacement SSD arrives
4:45 PM β€” SSD installed; recovery from last backup begins
5:30 PM β€” Database recovery complete
5:45 PM β€” Application tested and validated
6:00 PM β€” Banking application restored

Total downtime: 4 hours
MTTR: 4 hours

Cost calculation:
  Transaction revenue lost: $2,200/hour Γ— 4 hours = $8,800
  Staff overtime (3 engineers Γ— 4 hours Γ— $75/hour) = $900
  Emergency SSD delivery = $150
  Customer compensation / goodwill: $2,000
  Total incident cost: ~$11,850

Scenario B β€” Active/Passive HA + on-site spares:

2:00 PM β€” Primary database server fails
2:00 PM β€” HA heartbeat detects failure; passive standby promotes to primary
2:00 PM β€” Banking application continues on standby β€” no visible interruption
2:02 PM β€” Alert fires; engineer paged to investigate and replace SSD
3:30 PM β€” Failed SSD replaced from on-site spare; rebuilt as new standby

Total customer-visible downtime: 0 seconds (HA failover transparent)
MTTR (to restore HA pair): 1.5 hours

Cost of HA infrastructure:
  Second database server: $8,000 (one-time CapEx, depreciated over 5 years = $1,600/year)
  SSD spare kit: $200
  HA clustering software: $1,200/year
  Annual HA cost: ~$3,000/year

Single incident savings vs. Scenario A: $11,850
HA pays for itself after less than one 4-hour outage per year.

The lesson: MTTR is not just a technical metric β€” it is a financial one. Calculating the cost of downtime per hour and comparing it to the cost of resilience infrastructure produces the business case for HA investment. This is the conversation security architects must be able to have with business leadership.

Example 2 Β· Elasticity Security Blind Spot β€” Scale-Out Creates Unmonitored Instances

A retail company's e-commerce platform uses auto-scaling: during normal hours, 4 web server instances handle traffic. On Black Friday, auto-scaling adds 20 additional instances within 10 minutes to handle the traffic surge. The security team has configured SIEM logging, WAF policies, and vulnerability scanning for the baseline 4 instances. The auto-scaling configuration was set up by the infrastructure team without involving security.

9:00 AM Black Friday β€” Traffic surges; auto-scaling triggers
9:01 AM β€” 20 new web server instances provisioned and serving traffic
9:01 AM β€” These instances are NOT sending logs to SIEM (config not in auto-scale template)
9:01 AM β€” WAF policy covers the load balancer but new instances have no host-level monitoring

10:30 AM β€” Attacker identifies the retail site is under load; launches targeted attack
10:45 AM β€” Attacker exploits a known vulnerability in the web application framework
10:45 AM β€” Compromises web-server-instance-07 (one of the 20 new scaled instances)
10:45–3:00 PM β€” Attacker exfiltrates session tokens from instance-07
  β†’ No alerts: instance-07 not in SIEM
  β†’ No anomaly detection: no baseline to compare against
  β†’ No endpoint monitoring: agent not installed on scaled instances

3:00 PM β€” Traffic drops; auto-scaling terminates instances including instance-07
  β†’ Evidence destroyed when instance terminated
  β†’ Attack may never be discovered

The fix β€” security must be in the auto-scale template:

  1. Include security agent installation in the instance launch configuration: The IaC template for new instances must include SIEM log agent installation and configuration as part of the bootstrap process. Every new instance reports to SIEM from its first second of operation.
  2. WAF and network security must cover all instances, not just the load balancer: Host-level monitoring on each instance catches attacks that bypass the perimeter WAF.
  3. Security team reviews auto-scaling configurations: Infrastructure and security teams must collaborate on auto-scaling setup β€” not just infrastructure performance requirements.
  4. Centralized logging before instances go live: Instances should not be added to the load balancer pool until their log agent is confirmed healthy β€” security readiness as a deployment gate.

Example 3 Β· Cybersecurity Insurance Claim β€” Ransomware Recovery

A mid-sized manufacturing company is hit by a ransomware attack on a Tuesday morning. The attackers encrypt all file servers, the ERP system, and three production databases. The ransom demand is $450,000. The company has a cybersecurity insurance policy with a $500,000 per-incident limit.

Day 1-2 (Tues-Wed): Systems encrypted; manufacturing halted
  β†’ Insurance notified immediately (policy requires notification within 24 hours)
  β†’ Insurer assigns incident response firm

Day 3-5: IR firm performs forensics; scope of compromise determined
  β†’ IR firm cost: $45,000 (covered by policy)

Day 6: Decision: pay ransom or restore from backups
  β†’ Backups exist but are 3 days old; 3 days of production records lost
  β†’ Restoring from backups estimated at 5 additional days of downtime
  β†’ Total downtime if restore: ~10 days
  β†’ Ransom payment: $450,000 (insurer approves payment)
  β†’ Total downtime if ransom paid: ~3 days (decryption key received)

Financial outcome:
  Ransom payment: $450,000 (covered by policy)
  IR firm fees: $45,000 (covered by policy)
  Business interruption (3 days Γ— $80,000/day): $240,000 (covered by policy)
  Legal notification costs: $15,000 (covered by policy)
  Total claim: $750,000 β€” exceeds $500,000 policy limit
  Company pays out-of-pocket: $250,000

Without insurance: company would have paid $750,000 out of pocket.

What the insurance did not cover:

The lesson: Insurance transferred significant financial risk, but the $250,000 out-of-pocket cost was a direct result of insufficient policy limits. Pre-incident analysis should have calculated: what would a worst-case ransomware event cost? The policy limit should have been set to cover that estimate. Insurance is a risk transference tool that requires accurate risk quantification to be effective.

Example 4 Β· Inability to Patch β€” HVAC Controller Exploited for Network Access

A university hospital has 200 HVAC controllers managing temperature and airflow across its facilities. The controllers run firmware from 2014 β€” the vendor discontinued support in 2019 and no further patches will be released. A security researcher published a proof-of-concept exploit for a remote code execution vulnerability in this firmware version in 2022.

Current state (unmitigated):
  HVAC controllers on hospital corporate network (same segment as clinical systems)
  Controllers accessible on TCP port 4840 from any device on the network
  Known RCE CVE with public exploit code; no patch available

Attack path:
  Attacker on hospital guest Wi-Fi (same flat network as HVAC)
  Discovers HVAC controller at 192.168.100.45:4840
  Applies public RCE exploit β†’ code execution on controller
  Controller has no security software; no logging
  Attacker pivots from controller to clinical systems (same network segment)
  Accesses Electronic Health Records: 15,000 patient records exfiltrated

Remediation timeline after breach discovered:
  No patch available β†’ cannot fix the root vulnerability
  200 controllers must be replaced ($2,400 each = $480,000)
  Breach notification cost: $85,000
  Regulatory fine (HIPAA): $250,000
  Total incident cost: ~$815,000

Controls that should have been in place:

  1. Network isolation when vendor support ended (2019): The moment vendor support ended, the HVAC controllers should have been moved to a dedicated VLAN with no access to clinical systems or patient data networks. The public exploit was published 3 years later β€” but isolation should have happened at end-of-support, not after a breach.
  2. Firewall restricting port 4840: Access to the HVAC management port should have been restricted to the facilities management team's subnet only β€” not accessible from any device on the hospital network.
  3. End-of-support lifecycle policy: A policy requiring replacement of systems within a defined window after vendor support ends would have mandated controller replacement before the exploit was published. The $480,000 replacement cost, spread over a planned 3-year lifecycle, would have been ~$160,000/year β€” far less than the $815,000 incident cost.

Exam Scenario Β· Infrastructure Design Review

Scenario: A security architect is reviewing the infrastructure design for a new financial services application. Answer the following based on the infrastructure considerations framework:

A. The application must maintain 99.99% availability. What is the maximum permissible downtime per year, and what does this imply for infrastructure design?

99.99% ("four nines") allows a maximum of 52.6 minutes of downtime per year β€” approximately 4.4 minutes per month. This implies: (1) Active/Passive or Active/Active high availability for all critical components β€” single-component failures must not cause more than seconds of visible disruption; (2) MTTR must be very low β€” recovery must be measured in seconds, not hours; (3) Redundant power (UPS + generator + dual utility feeds), redundant networking, and redundant compute all required β€” any single infrastructure failure must be absorbed by a redundant component; (4) Automated failover β€” manual failover is too slow for a 52-minute annual budget.

B. The application will handle 1,000 users during off-peak hours and 50,000 users during peak. The team proposes sizing the infrastructure for 50,000 users permanently. The architect objects. What is the correct approach and why?

Sizing permanently for peak load (50,000 users) is wasteful β€” the infrastructure would be idle at 98% capacity during off-peak hours, paying for compute, memory, and licensing that is not being used. The correct approach is elastic auto-scaling: baseline infrastructure sized for normal load (1,000 users), with auto-scaling configured to automatically provision additional capacity as demand increases, and automatically decommission it as demand decreases. This matches cost to actual usage. Critical security requirement: the auto-scaling configuration must include security tool deployment (SIEM agents, endpoint monitoring, WAF policy) in the launch template β€” every new instance must be monitored from provisioning.

C. The development team wants to deploy quickly and proposes manual configuration of each of the 12 application components. The architect recommends orchestration. What is the security argument for orchestration?

Manual configuration of 12 components is error-prone. In a rushed deployment, individual steps are skipped or misconfigured: a firewall rule is forgotten, a monitoring agent is not installed, a certificate is left self-signed, a database is left with default credentials. These omissions become security vulnerabilities that persist until discovered. Orchestration (Terraform, CloudFormation, Ansible) encodes the entire deployment β€” including all security configurations β€” as code that executes the same way every time. Security configurations are peer-reviewed before deployment, version-controlled, and repeatable. The security argument: orchestration eliminates the manual-step omissions that create vulnerabilities, and it ensures that every deployment of the application β€” development, staging, production, disaster recovery β€” has identical security configurations.

D. The CISO suggests purchasing cybersecurity insurance. A board member argues that good security controls make insurance unnecessary. Who is correct?

Both are partially right, but the CISO's recommendation is correct for a financial services organization. Security controls reduce the probability of incidents; insurance transfers the financial consequences when incidents occur despite those controls. No security posture eliminates 100% of risk. For a financial services firm, the cost of a major breach β€” business interruption, forensics, legal proceedings, customer notification, regulatory fines β€” can run into millions of dollars. Insurance caps the financial exposure and enables predictable budgeting. The board member's error: conflating risk reduction (what security controls do) with risk elimination (which no control achieves). Insurance is not a substitute for security controls β€” insurers require minimum controls before issuing a policy. It is a complement.