Skip to content

Attack encyclopedia

Oldsmar - remote change of chemical dosing at a Florida water treatment plant

Incident at the Oldsmar water treatment plant in February 2021: the sodium hydroxide dosing setpoint was changed remotely from 100 to 11,100 ppm. The FBI did not confirm an intrusion and the cause remains unexplained. Lessons for OT: setpoint limits and two-person approval of changes.

Józef SulwińskiJózef Sulwiński

Published 6 min read

Oldsmarwater treatmentSCADAremote accesscritical infrastructureICSsodium hydroxide
Oldsmar - remote change of chemical dosing at a Florida water treatment plant
Contents
  1. 1:30 PM - dosing setpoint changed remotely
  2. Sodium hydroxide - what would have happened if no one noticed
  3. What is known about the cause
  4. Timeline
  5. Attack or error - the cause remains unexplained
  6. Lessons for critical infrastructure operators
  7. Oldsmar in the broader context
  8. Sources

Friday, February 5, 2021, 8:00 AM. An employee at the water treatment plant in Oldsmar, Florida, a city of just under 15,000 residents, sees on the monitor that someone is logging into the system remotely. The employee treats it as routine, because remote access is an everyday matter at the plant.

A few hours later, someone changes a setpoint at the same plant that could have endangered drinking water.

1:30 PM - dosing setpoint changed remotely

Around 1:30 PM the employee sees someone remotely change the sodium hydroxide (NaOH) dosing setpoint in the SCADA software. Plants use this chemical to regulate the pH of water. According to the Pinellas County sheriff, the setting moves from 100 to 11,100 ppm (parts per million), a factor of 111.

The employee immediately reverts the change to a safe value. According to CISA, staff noticed the change before the software raised an alarm, so the treatment process was not disrupted.

The Pinellas County sheriff presented the event as the work of a person who wanted to cause harm. In 2023 the FBI stated that it could not confirm a targeted intrusion. A former city manager pointed to an employee error, but neither the FBI nor the employee confirmed this. The cause remains unexplained.

Sodium hydroxide - what would have happened if no one noticed

Sodium hydroxide at low concentrations is a standard reagent in water treatment: it neutralizes acidity and raises pH. At 100 ppm it is safe, and at 11,100 ppm it is highly corrosive.

Contact with such a solution causes chemical burns to the skin, mucous membranes, and eyes. Ingestion can burn the esophagus, stomach, and intestines.

What is known about the cause

CISA advisory AA21-042A describes an incident at a US water treatment plant without naming the town.

What is confirmed

The sources agree on the course of events:

  • On February 5, 2021, someone remotely changed a setpoint in the SCADA system and increased the sodium hydroxide dose
  • Employees noticed the change at once and reverted it before the software raised an alarm
  • The treatment process was not disrupted, and the altered water did not reach customers

What CISA considers likely or possible

CISA words its conclusions about the access path with care. According to the advisory, weak passwords and an outdated operating system likely enabled the access. Use of desktop-sharing software is possible, but CISA states that this cannot be confirmed. These sources contain no document that points to a shared password or to a specific operating system version.

What remains unexplained

The FBI could not confirm that the incident was initiated by a targeted intrusion. Al Braithwaite, the former city manager, said at a 2023 conference that the FBI found no evidence of access from outside and that an employee may have changed the setting by accident. An FBI Tampa spokesperson declined to comment, and neither the employee nor the FBI confirmed the employee-error version. Water-ISAC allowed from the start that an authorized user could have entered a wrong value.

Timeline

Date Event
February 5, 2021, ~8:00 AM An employee sees a remote login to the system and treats it as routine, because remote access is common at the plant
February 5, 2021, ~1:30 PM The NaOH dosing setpoint is changed remotely from 100 to 11,100 ppm, and the employee reverts it
February 8, 2021 The Pinellas County sheriff announces the incident at a press conference
February 11, 2021 CISA, FBI, EPA, and MS-ISAC publish Joint Advisory AA21-042A
2023 Former city manager Al Braithwaite says the FBI found no evidence of access from outside; the FBI states that it could not confirm a targeted intrusion

Attack or error - the cause remains unexplained

The Oldsmar story has two versions, and the available sources do not settle which one is true. The sheriff spoke of a person who wanted to cause harm. The former city manager called the event an employee error and a non-event resolved in two minutes. That is his account, and the FBI has not confirmed it.

The lesson for OT is the same in both versions. If an intruder changed the setpoint, the system allowed the change remotely. If an authorized employee changed it, the system allowed the mistake. Either way the setpoint could rise 111-fold without a second confirmation and without a value limit.

Safeguards must therefore work even when an authorized person makes a mistake. Limits on physically plausible setpoints, two-person approval of critical changes, and alarms on setpoint changes provide that.

Lessons for critical infrastructure operators

The CISA advisory recommends, among other things, multi-factor authentication, strong passwords, review of remote connection logs, and isolation of systems that cannot be updated. The lessons below do not depend on who changed the setpoint in Oldsmar.

1. Remote access requires controls

Every remote connection to ICS systems should pass through a secured tunnel with multi-factor authentication. Sessions should be logged and time-limited, because after an incident the logs show who changed which setting and when.

More on securing remote access to control systems: Remote access to ICS - how to do it safely.

2. Every user has their own account and password

With shared accounts it is impossible to establish who performed an action. In Oldsmar it is unknown whether an authorized employee or someone from outside changed the setpoint, so individual accounts and a change log would have answered that question. The password policy should enforce unique, strong passwords for every user.

3. OT network segmentation

Computers controlling technological processes should not have direct internet access. The OT network should be separated from the corporate network and the internet using DMZ zones and firewalls with traffic inspection, because every external connection lengthens the path to process setpoints.

Details on segmentation architecture: OT network segmentation - designing zones and conduits.

4. Operating system updates

CISA points to an outdated operating system as a likely weak point. A system past the end of vendor support receives no security patches. Where migration is not possible because of SCADA software compatibility, additional layers of protection are needed: an application allowlist, segmentation, and behavioral monitoring.

5. Limits and alarms on process parameter changes

The SCADA system should enforce limits on critical process parameters. An 11,000% change in NaOH concentration should automatically trigger an alarm and require confirmation by a second operator, so that one person’s mistake does not end in a physical consequence.

Oldsmar in the broader context

One remote setpoint change was enough to alter the quality of drinking water, because the system set no limit on it and no second person confirmed it.

Sources

  1. CISA Joint Cybersecurity Advisory AA21-042A - Compromise of U.S. Water Treatment Facility (accessed: )
  2. CyberScoop - Did someone really hack into the Oldsmar, Florida, water treatment plant? (accessed: )
  3. Water Online - Former official claims Oldsmar drinking water hack was really operator error (accessed: )
  4. ScienceDirect - Accidental contamination of a German town's drinking water with sodium hydroxide
  5. The Register - Someone tried to poison a Florida city by hijacking its water treatment plant via TeamViewer, says sheriff (accessed: )

Need help in this area?

Our experts will help you assess the risk and plan next steps.

Talk to an expert

We'll discuss scope, methodology, and timeline.

Book a call