Huawei cryptographic keys embedded in Cisco’s firmware
Things happen when they happen. And when developers use third-party or open source libraries in their own product, they may not be aware of potential security issues. Testing firmware for vulnerabilities is time consuming, yet absolutely necessary for compliance with established security standards and legal requirements. That’s why we developed IoT Inspector: to automate security analyses of firmware and to assure a security baseline at scale.
We are constantly improving IoT Inspector’s analysis capabilities. To test new features and capabilities we analyze firmware images from various vendors regularly. One of the more recent analysis results caught us by surprise…
Who is Gary, and why are his keys embedded in Cisco’s firmware?
The starting point was a firmware image for a Cisco SG250 Smart Switch device, which was downloaded from the Cisco download portal and uploaded to IoT Inspector. The analysis results were strange. The firmware contained a few certificates and a corresponding private key. The location of the files in question (/root/.ssh/) is usually intended for SSH keys, not certificates.
The certificates themselves were issued by a Gary from the organization Futurewei Technologies, which is a US-based subsidiary of Huawei Technologies. Just to make sure, we double-checked the IoT Inspector results. They were reliable, a manual analysis confirmed the automated results. So, how does a certificate from a Huawei employee end up in a Cisco firmware image?
IoT Inspector results for a certificate issued by Gary Wu
IoT Inspector directory browser
Given the ongoing political controversy around Huawei, we did not want to speculate any further and decided to hand over all our information to Cisco. Cisco PSIRT immediately acknowledged our submission, started an internal investigation, and kept us up to date during their progress. It took them only a few days to complete a thorough analysis and share detailed results with us.
As it turns out, the certificates and private key in question were part of the OpenDaylight GitHub open source package, which is used in some Cisco products. All Cisco 250/350/350X/550X Series Switches are affected. Developers used the certificates for testing the Cisco FindIT feature. The certificates ended up in the shipped versions of various products due to a simple oversight.
According to Cisco, no attack vectors have been identified as the certificates are not actually being used by shipped versions of firmware. Cisco has released a firmware cleared of the certificates and has published a security advisory today. Furthermore, Cisco acted on other issues discovered by IoT Inspector too. Among them empty password hashes, unneeded software packages, and multiple vulnerabilities in third-party software (TPS) components. We want to thank Cisco for the good cooperation on these issues and their rapid and professional handling of the process.
As a vendor who builds software in-house, even more so if you are cooperating with third parties, you want to know exactly what ends up in a firmware before you ship it to your customers. This is where automated firmware security analysis solutions like IoT Inspector come into play. IoT Inspector finally provides transparency as to what is inside a firmware image, be it cryptographic keys, hard coded passwords, information leakage, outdated third party components and various other issues.
