Our Blog
Your questions answered
Your questions answered

The Terrible, Terrible State of VOIP Security, Part III

>>> This is part tree of a three-part article <<<

>>> Part I

>>> Part II

VII. Cisco Again

Cisco Opens Itself

It came to our attention that an option to support third-party root certificates — that was even present in earlier versions of phone firmware but never worked — finally started working and allowed for third party certificates usage. This was very good news in light of changes within 3CX system, and we plunged into a new investigation.

While we prepared and rearranged our lab, clouds descended on one of the fundamental hash algorithms used in PKI[23] of all sorts — SHA1[24]. No, it is not yet 2017 where first collisions are reported[25], but it is the preparation by browser vendors and certificate authorities: Basically, it was no longer possible to get a SHA1-signed certificate issued — the new algorithm to be used was SHA256[26]. It was (and is) cryptographically much more stable and the newest Cisco firmware allegedly even supported SHA256-signed certificates.

The Culprit

Now remember that Small Business SPA product line consist of 300 and 500-series phones. And within that 500-series there is one model — an SPA-525G2 — that does not share the firmware codebase with the rest of the family. There is no apparent reason for that fact, and although it has many more functions than its siblings, it still is a phone with many aspects in common with them. Now guess what? This was the only phone that did not work with that new cryptography! The whole line worked fine: the 300s, the 500s — but not the 525!

Time to open a new case with Cisco. First thing we learn is that support engineers are not aware of the fact that Cisco now supports third party certificates and at first, they just refuse to take the case, saying that we have to use only Cisco-signed certificates. We insist and the case gets escalated.

The second level is unable to reproduce the case and seems to be quite reluctant to probe any further, since they do not understand what is happening. Time goes on without any progress until we push our contact at Cisco and the case finally gets assigned to a new engineer. The engineer cannot reproduce and asks us to send him network traces.

In Search of the Secret

Now in order to obtain decrypted TLS traces, you need a decryptable handshake on the wire. This becomes quite rare, as more and more nodes negotiate the keys using Diffie-Hellman[27] algorithms. These algorithms do not rely on pre-defined keys (otherwise obtainable from the key/certificate pair), but instead negotiates an ephemeral key that is not stored anywhere and not accessible to our deciphering tool (Wireshark). Thus, in order to obtain the traces, we had to disable cypher suites that used Diffie-Hellman key negotiation in favour of RSA[28] keys that we could use for decryption. Luckily, this change had no effect on the phones' behaviour, so we collected the traces from both a working phone (a 300-series) and non-working one (the 525er) and send them both to Cisco.

Next day Cisco's supporter is back, though not with findings, but with more riddles. He tells us that our server closes the connection after the phone (the 525er) presents its client certificate within the handshake and that Cisco cannot be held accountable for our server's behaviour. Then he gives us crucial information: two traces that we sent to him from a working phone and from our problem phone show two different TLS versions: v1.0 in the first and v 1.2 in the second, not working case.

Microsoft Giggles from Around the Corner

So off we go to dive into the trace and decipher individual messages. It turns out that TLS 1.2, according to the RFC, assumes the hashes in the cryptographical signatures for the presented client certificates to be at least SHA1, unless specified otherwise by the sender in a special field during the TLS handshake.

It also turns out that Cisco phone's client certificates use MD5[29] for their hashes and that the phones indeed specified this algorithm in that particular special field during the handshake as stated in the RFC.

An excerpt from a network trace showing an MD5 signature for the presented client certificate.

So, everything should have been fine as all the steps were carried out according to the big plan — the RFC. What was wrong? We were so close to the resolution, and yet something was still missing.

We carried out two additional tests and were able to see that SIP/TLS indeed worked fine — only the secure provisioning over HTTPS was not working. Something was obviously wrong with our web server. Now our web server was a standard IIS[30] on a Windows Server 2012 R2, and we were not expecting any surprises.

It is important to know that IIS itself does not handle TLS, but rather outsources this functionality to another subsystem called SChannel — Microsoft's SSL/TLS-engine[31]. After some further examination we find out that SChannel has a flaw in TLS 1.2! And this flaw concerns exactly our scenario: The server, while talking TLS 1.2, ignores the said field for hash algorithm specification and, having received an MD5-hash instead of expected SHA1, decides that this is a violation of its TLS contact and closes the connection.

The SIP flow that was not governed by SChannel — but rather by OpenSSL — worked fine, since OpenSSL most probably was RFC compliant in this issue.

Many cryptographic forums were full of questions and outcries on this subject. So we consider ourselves lucky for understanding what had happened — others were just poking around in the dark.

The same setup with TLS 1.2 disabled worked just fine (the phone and the server would negotiate TLS 1.0 in that case), and for the time being it could serve as a compromise. But in the long run the problem would need to be solved. Hopefully Microsoft patches this flaw at some point.

Even if think MD5 is a thing of the past, it still has to be supported — only in order to support those old devices that got their certificates issued when MD5 was still OK to use or to support the devices from devices from slower vendors with older root certificates.

VIII. Conclusion

What have we got at the end of the road?

Let us see:

  • Asynchronous communication as such is complicated
  • Telephony as such is complicated
  • Interoperability between vendors as such is complicated
  • Cryptography is in many ways (very) complicated

So, why on earth did we expect asynchronous encrypted telephone system comprised of nodes from different vendors to be uncomplicated? This assumption might have been our biggest mistake — to assume and just hope that it will work.

But it was not just our bad luck: We learnt in that lengthy experiment that implementation details (and errors) are crucial. You cannot just expect two devices that promised to talk the same protocol to work flawlessly together. Manufacturers make assumptions and predictions and they do it with good intentions or to save money. But sometimes it backfires.

Had not Cisco imposed restrictions on their PKI we would have been able to use trusted certificates from the very beginning. Had 3CX implemented the integration with OpenSSL correctly we would have been able to use Cisco phones from the very beginning. Had Snom implemented the PKI in their phones properly we might have switched to Snom from the very beginning. Had Cisco used the same code-base for the whole product line we would have not lost so much time scratching our heads over why SPA525 was the only funny phone among the rest of the family. And finally, had Microsoft implemented TLS 1.2 properly all this would finally just work as intended — especially if we think about the fact that the life of TLS 1.0 is coming to an end.

And as if this was not enough, I am curious just how Cisco is going to replace the Client Certificates in the myriad of phone devices deployed around the world? All these certificates bear MD5 hashes and are signed by some (now) obscure certificate authorities that bear the names of two VOIP vendors that Cisco bought in the course of time: Linksys and Sipura. These root certificates expire in 2035. Other Cisco root certificates expire later and have SHA1 hashes — but as far as we know, SHA1 is practically broken, so the next problem is just around the corner.

If you think about the last point made, you will have to realize that this problem is actually fundamental for all devices with security features: As security evolves and features get deprecated, devices may just sit there, un-updated, and are unable to get updated, thus posing a potential threat to network environments.

It seems that complicated setups like the one described would never just work— or not at least until standards are strictly enforced and vendor self-discipline rises above today's level.

Let us hope that this day comes soon.

Jump back to Part I or

Jump back to Part II or

Read the whole article as one document


[23] Private Key Infrastructure, a set of roles, policies, and procedures needed to create, manage, distribute, use, store, and revoke digital certificates and manage public-key encryption. (Cited from Wikipedia)

[24] SHA1 is a cryptographical hash function (irreversible transformation) that generates a 20-byte hash-digest.

[25] First successful collision attacks were reported in 2017: https://phys.org/news/2017-02-cwi-google-collision-industry-standard.html

[26] A replacement for SHA1 is SHA2, a family of hash functions with different bit length, e.g. SHA265 uses 256 bits for the digest calculation andproduces a 32-byte value.

[27] A key-exchange protocol that does not rely on a pre-defined secret: https://en.wikipedia.org/wiki/ Diffie%E2%80%93 Hellman_key_exchange

[28] https://en.wikipedia.org/wiki/RSA_(cryptosystem)

[29] Message Digest Algorithm 5, an old and now deprecated hash function, generating 16-byte digests. The algorithm is described in RFC 1321: https://www.ietf.org/rfc/rfc1321.txt

[30] Internet Information Server, a standard part of Microsoft Windows Server

[31] https://technet.microsoft.com/library/ aa380123.aspx

Roman Kuznetsov @ 10.07.2017

Other Posts