Cybersecurity testing for ICS – pitfalls and wins

This is a (long overdue) followup post of the talk I gave at the SANS ICS Summit in 2021 – a recording of this talk can be found on youtube: https://www.youtube.com/watch?v=Qpl8eI8Tn0s

I suggest you to first look at the recording of the talk before reading further about some questions I have received during the talk.

I think the ‘independant’ pentest / security focussed FAT is great. Specifically because it can be objective.
Otherwise, the vendor typically has a lot of control on establishing the procedures, setting up scenario’s etc… So you are only testing a limited subset of system functionalities with vendor locked-in tests.

Q: Do you have tips when discussing security from your supplier / integrator if security knowledge is very low on the contracting party? Without formal requirements but security is a must…

I would perform security testing and list up all discovered issues as always. With these issues, the associated risks can be discussed openly together with the contracting party and the business owner. It is important the these issues and risks are clearly explained and that the possible solutions are also explained. The security FAT/SAT process is often also an awareness or learning path for the contracting parties. In the end, this is a win-win for both sides. Business owners get a secured system, contracting parties learn to implement security in their solutions.

Q: What I’m missing in your cycle is testing (reviewing) the design during the engineering phase against those security requirements.

This is indeed an important step not to forget. However, most of the times engineering does not include (yet) security where it actually should. I did not mention this in my talk.

Q: During the last phases of testing the (cyber) security processes of the organisation that will operate the system should be tested as well in a Site Integration test, do you agree?

Yes, this is what I (partially) mentioned with testing Human elements/aspects – this includes processes on how people are or will be using the system.

Q: What to be tested? What I see that in ICS input validation is often forgotten and could be a huge vulnerability. Is this something that can be caught during engineering/design, that should be testing during FAT/SAT as well?

Input validation controls are indeed very important to include in security testing. However, it is often not possible (or even allowed) due to time constraints, vendor contract clauses and such. During FAT/SAT it is not (always) the goal to perform full software security testing where this is more the task of the vendor themselves.

Q: If you are with a Solution Provider, wanting to offer Cyber tests during FAT/SAT, but the client disregards it, due to monetary issues, then what?

The first thing you run into is that as a Solution Provider, you are not (completely) independant (enough) to be able to provide an acceptable security test result. So an option might be to cont(r)act another (independant) party that can do such tests to make sure those independancy concerns are out of the way and properly covered.

Solving the monetary issue can be fixed by letting the client run their own security tests – which will give them more value as at the same time the independancy issue as described earlier will be out of the way as well. It also helps if the client has a policy that ALL new or upgraded systems or solutions MUST be security tested in a FAT/SAT testing approach.

Q: Do you have a recommended Vulnerability Scanning tool for ICS

In security FAT/SAT tests, I generally use Nessus but any vulnerability scanner would do just fine.

Q: Any recommendations on how to record testing & results during the test (similar for pentesting) rather than a paper based pass/fail checklist?

Any of the following helps in this: console/terminal logging, traffic capturing while testing, screen recording while testing…

Q; Who defines what tests should be done in SAT and FAT?

That is dictated by the (default) testing protocol you should build and have, together with the information received through answers on the security requirements (if any have been set upfront). It will also be important to include any additional testing should you see the need for this and not just stick to the default testing protocol, but do make sure that your default testing protocol is executed as a minimum.

Q: What do you think of ICS vendors now becoming themselves Cyber Security Vendors with the excuse that they know your ICS system better than anyone else?

This can certainly be of value for the vendor itself and may has its place. However those teams can not be completely independant in my point of view. Most customers want to have an independant point of view, not influenced in any way by any vendor. This is also the reason why some of my customers request second opinions of vendor performed security tests 😉

Q; Does this testing means production testing with real data? How to ensure business continuity and availability during ordering tests in 3rd parties agreement responsibilities aren’t solution for critical infra aren’t they?

In security FAT/SAT testing, the systems are not (yet) operational, so there are no business continuity risks with regards the testing itself. If the system fails, operations are not impacted (or should not be).
In live production environments that is of course a completely different story as those 3rd party agreements do indeed not always cover such business continuity impacts.