Skip to content
Ask A Question
  • There are no suggestions because the search field is empty.

Meltdown and Spectre Vulnerability Guidance for Operations Technology (OT) professionals - (US CERT VU#584653,TA18-004A)

What guidance can you offer related to your products that we use and the Meltdown and Spectre Vulnerabilities announced on January 3, 2018 by US-CERT references VU#584653, TA18-004A and NIST references CVE-2017-5753, CVE-2017-5715,CVE-2017-5754?

Software Toolbox is aware of the side-channel analysis attacks (also known as Meltdown and Spectre) affecting many modern microprocessors. Although these are hardware issues, they affect everyone. We take cybersecurity seriously and are providing the following guidance to clients, and partners concerned about these vulnerabilities. Reference links are provided at the end of this document for those that want to learn more about the vulnerabilities. Of course, our IT security team is working to insure systems we use to develop on, support you, and that you access to interact with us are secured following industry best practices and guidelines.  We are constantly adding things to this document, and you can subscribe to notifications when we update it with the "Notify Me" link at the bottom right of this article. As we update the article, we'll update the revision history at the bottom as well.

What are these vulnerabilities?

The Spectre and Meltdown vulnerabilities are CPU hardware vulnerabilities that are affecting Intel, AMD, ARM processors to name a few. Bottom line, they have the ability to be used to allow unauthorized disclosure of information to an attacker in ways not seen before since this involves CPU hardware. The issues are below the level of even operating system and application developer tools used to build our software products. They are in the architecture of the chips.

For those reasons, you can't "patch the chip" but operating system providers such as Microsoft and others are actively working to provide mitigation patches in their products. The risks of the vulnerabilities are described well in the references we provide so we will focus on guidance on actions you should take.

Software Toolbox products run on top of an operating system, access the hardware through the operating system, and thus depend on the security of the operating system and hardware.  That said, there are indirect ways that the performance of our products could be affected which we are actively researching and evaluating and will address later in this document.

Recommendations

#1 - Collaborate and engage with your company's IT and cybersecurity teams.

They should be acting on these vulnerabilities and taking responsibility for the mitigating steps, including, but not limited to patching, user informing and training, limiting local user access to systems, and other application and user specific pragmatic mitigation steps. If IT has not already reached out to you, reach out to them.  This is not a time for Operations Technology (OT) and IT to work in isolation.

#2 - Become informed, learn.

See the US-CERT and Microsoft Technical Advisories at the bottom of this article to learn more about these issues. If you are operating in a critical environment and haven't looked at the NIST Cybersecurity framework, this might be the tipping point to consider reviewing it using the link at the end of this document.

#3 - When you decide if and when to patch Operating Systems, do it carefully following proper change management and preparation - it's possible your situation warrants waiting!

  • Microsoft in their advisories talks about interactions between some anti-virus software and Microsoft's updates that if not addressed with patches to Anti-virus first, will result in "blue-screen" crashes of systems!  
    • What we've seen is that on January 4, 2018, Microsoft rolled out Windows Updates to machines that had a special registry key present already that had been made by their anti-virus systems. We saw this on systems with Symantec Antivirus and Microsoft's own Windows Defender.  No other systems were even offered the updates
    • On "Patch Tuesday" January 9, 2018, the only patches offered to machines that did not have the special registry key present were non Meltdown and Spectre related updates. From reading Microsoft updates online, it appears they are holding off on general availability and letting those who want to manually put in the registry key do so, but it definitely feels like a "proceed at your own risk and apply your own assessment of whether you need to".  To us this supports our recommendation #1 and the other points here.
  • Intel has pulled their patches while they try to fix the "infinite reboot" issues that some machines were seeing. See Intel News Byte Jan 22, 2018
  • We have seen notices from HMI/SCADA vendors indicating that they want users to wait to apply patches for their product before applying Microsoft updates and to "hold off" for now.  If you have other software running on machines with ours, and those vendors ask you to hold off patching, then do follow their guidance. 
  • When you do patch be sure you have backups of your systems, configurations and data to avoid unexpected operational downtime.  
  • Also see point #6, the cure might be worse than the risk, especially if you have extremely isolated, secure systems, without multi-user or remote access.  
  • See point #1 and work with your IT team to assess the risk/reward and understand from a practical perspective how these new vulnerabilities might be exploited in your systems.

#4 - Revisit OS modernization plans.

It's time to get off of unsupported Microsoft operating systems.  That's anything older than Windows Server 2008 and Windows 7.  There won't be any patches for those operating systems!  

#5 - Review your application versions.

Run current versions of our products. Our current versions typically can take advantage of newer hardware, newer operating system and development environment features that help performance. Current version information is found on the support tab of specific product pages on our website.

#6 - VERY IMPORTANT - Review your CPU loading on systems.

As we will discuss later, the operating system mitigation steps will affect system performance.  If your systems aren't loaded up to 70, 80% CPU load or higher, and you don't have fail-over scenarios between machines that could double loading on a moments, notice, then you may never notice.  However, if you have chosen to push your system's CPU usage hard, you could be impacted and need to reconsider how you distribute workloads before you begin patching operating systems. Contact us if you need application specific guidance involving your specific usage of our products.

In our testing with some of our products, we've seen some performance impacts but they are not as great as we and others feared.  If you would like to discuss our findings and how they might apply to you, please submit a question to our team.

If you are updating your systems to newer operating systems as a result of IT requirements, action, or recommendations #4 or 5, you may need product updates.  If you are concerned about loading of your systems per recommendation #6 and want to talk, please contact us and our team can advise what your options are.  

Users on active support & maintenance agreements as always will receive priority support. If you aren't on support, we can still help as paid services as resources allow, and/or get you back on support.

Is Software Toolbox patching their products, what are they testing?

As these vulnerabilities are in the platform below us, there is not, at this time, anything we can do in the products.  You must work with your IT to insure a pragmatic, best practices approach for your operations is taken to patch, mitigate, and protect the underlying operating system and access to systems.  We will update this FAQ should this situation change.  Our teams monitor US-CERT, NIST, Microsoft, and SANS advisories so that we can act quickly on new information from security experts.

As Microsoft patches are released, we will be doing tests to check and see if there are unexpected direct impacts. We'll update this advisory with links to product specific information if we find anything.   

What indirect effects could there be on my Software Toolbox products?

Performance

It has been stated by security experts that the patching for mitigation that operating system vendors is doing will make non-negligible impacts on operating system performance. (See SANS webcast reference).  What that means will vary but we have anecdotally heard 10, 20, 30% impacts.  See recommendation #6, and review your CPU loadings.  

There may be impacts on your performance that are totally outside of our control depending on your application, how heavily you have loaded your systems, etc.  For example, certain CPU intensive operations, like retrieving the entire tag database from supported PLC CPUs, may see performance impacts.  As we know more, we'll update this advisory.  

Microsoft patches that they ended up having to patch again

As this situation develops/developed, given it's a very challenging issue for OS vendors, there have been issues created by OS patches, that then had to be patched again. An example of this is are the patches released 3 January 2018 that all begin with KB405689# where "#" varies by Windows OS.  (Example: Windows 10 Version 1709 was KB4056892).  Products using OPC Classic, which depends on COM, have been impacted where under some conditions, simple COM calls like an OPC client and server connecting locally would fail if impersonation level is set to none, or authentication level is set to none. 

So Microsoft then released "Patches to the patches".  The one referenced above was then patched with Microsoft KB4073291

This "proliferation" of patches emphasizes why if you have IT resources to help you, you should use them - see advisory point #1 and point #3 above.

We are finding that the KB articles on Windows will state the known issues, and then as resolutions are made available, links to those new KBs are provided, but you must pay attention to any known issues on those KBs to see if they impact you. If you look at the FAQs for our products below, you'll see examples of how Microsoft indicates this.

Starting with the February 2018 Windows Update cycle, we noticed that if a system didn't have the registry key present that lets you install Meltdown and Spectre patches, you couldn't get any further security updates. According to this article from ZDNet, eventually, Microsoft may relax that requirement, but they are waiting until they have enough evidence that the risks involving machine blue screens, boot loops, etc with the first patches are gone or low enough.  So this means if you want to get even other security updates from February or later, perhaps skipping the January 2018 Meltdown & Spectre updates, you still must make the registry key entry that Microsoft talks about in their technical advisory!  All the more reason why it's a good time to be methodical, logical, and use your IT resources to insure you meet your uptime requirements, manage risk, and manage security.

Software Toolbox cannot advise you on OS support or provide OS support but as you can see here, we're happy to share what we are seeing and let you know where you need to be careful and provide workarounds or solutions for our offerings.

FAQs for Software Toolbox Products

In this section, as this story develops, we'll link to relevant product FAQs that we add to provide workarounds or advise on how to deal with the Windows OS "patch to a patch to a patch" situations that you may encounter.

What next?

This is a developing story and moving target. We will continue to do our part, you should do yours. We'll update this advisory as we have more useful information to share.

If you have questions that we haven't answered here, please click on "Ask A Question" at the top of this advisory and we will assist our clients who have purchased licenses from us or are considering purchasing licenses. Software Toolbox clients on active support & maintenance agreements get first priority.

If you would like to be notified when this FAQ is updated, you can click on "Notify Me" at the bottom and subscribe to future update notifications.   

References

These links are to non-Software Toolbox resources, and they contain links to the research papers that show how these vulnerabilities work in extreme detail for those that are interested.

Revision History

23 January 2018:  Based on reports from Intel that they have pulled their firmware patches, added link Intel article on the matter and added to our reference links and updated our advisory notes.  Also added more clarity on what we are seeing in our experience with patching of Microsoft systems.  Also added that testing we have done shows performance hits haven't been as bad as all were worried, but at this time, there are still no product updates we can or need to make.  This remains an OS and CPU hardware and firmware level issue, below our products and their code. 

14 February 2018: We have learned about some issues that affect any software using COM objects that the Microsoft January updates with KB numbers starting with KB405689# where "#' is a number that varies by OS. Updated the indirect effects section of this FAQ accordingly and added a "FAQs for Software Toolbox Products" section.

20 February 2018: With the February 2018 updates, it became clear that Microsoft at this point isn't going to deliver other security updates to systems until those systems have the registry key in place that allows for the January 2018 Meltdown/Spectre patches to be installed.  Added notes and external source reference for this to the Microsoft patches section.