The Terrible, Terrible State of VOIP Security, Part II
>>> This is part two of a three-part article <<<
With all that fuzzy setup in mind we embarked on the journey to provide our existing and new DaaS-customers with telephony services.
We had to provide them with robust telephony devices and a robust service as well. So basically, we had two choices to make: a scalable SIP-server and reliable vendor of IP-phones.
We chose Cisco as phone vendor, since phones themselves were of quite good quality and relatively easily customizable. We are talking about Cisco Small Business SPA500 series —originally Linksys products before Cisco bought them. This fact becomes important once we start implementing encryption on these phones.
For the server side our choice of vendor fell on 3CX, a European company with a sound product with close to being non-proprietary in their implementation of SIP. Hence, further along the lines we will be assuming 3CX as our server-side component.
Although 3CX provided phone provisioning to automate phone setup on the customers' side, initially that provisioning was done over HTTP without encryption — so we also wanted to make it really secure by providing an HTTPS-based provisioning and client-certificate-based authentication, thus closing the circle and leaving no chance for an attacker to eavesdrop.
We first made sure that our setup worked in general, and only then did we start to transform it so that it embraced encryption.
And this is where the actual fun began.
Further comparing an HTTPS session with a SIP/TLS session, we have a phone, which can be regarded as a browser and the SIP-server as a web server. As a web browser requests a web page over a secure connection, so does the phone, making a SIP request over a TLS-connection. An established (and in our case, authenticated) TLS session assumes that both sides have exchanged their cryptographically signed certificates and that both are mutually trusted. On a computer running your web browser, you would import any root certificate to the appropriate store to make a foreign certificate trusted. At the time of writing, this is not a huge problem with Cisco, but it was indeed at the time of implementation.
Cisco phones came with a pre-installed client certificate and a small list of root certificates that could be trusted. That meant that we would have needed to obtain a certificate for our provisioning server and for our SIP server, signed by Cisco. Indeed, there was a known way to do so, well described on a Cisco support site. It basically said that you would need to generate a certificate request and submit it to Cisco to be signed. It even provided an email-address to send the requests to, but the submission could only be initiated by an official Cisco distributor. We contacted two distributors regarding the task and none of them was even able to understand what we wanted from them, since they had never heard about Cisco SPA phones in the context of secure SIP.
We opened a case with Cisco partner support and, after a week or two emailing back and forth we were able to find a person responsible. A few days later we finally got a signed certificate and were able to install it on the web server (to ensure secure provisioning) and on the SIP server (for SIP/TLS to work). Although SIP/TLS did not work right away (more to it in part IV), things started moving.
Now, at this stage you have to remember that SIP encrypting is separate from RTP encryption and we had to provide that, too. After switching on SRTP on the phones, we realized that it was not working — the phones were not even trying to establish a secure RTP session. Support forums were full of questions regarding SRTP-encryption — most with no answers.
Further reading revealed that SPA phones required the encryption key pair to be generated for them and installed as part of provisioning process. But there was no specification of the format of that data. Two obscure settings were provided for that: SRTP-key and a "mini-certificate". These were mentioned on support pages, too, with a reference to a command-line tool that would generate that data. But all links that used to point to that tool were broken. We saw a community attempt to recreate the tool, but could not find a working copy of it. Only another support request provided us with the tool that would generate the necessary data for the provisioning.
It is not entirely clear why Cisco chose that complexity in implementing SRTP, but jumping ahead, it is worth mentioning that other vendors did not require any special setup in order to be able to generate SRTP keys.
After installing the server certificates and the obscure SRTP keys we thought that the troubles would be over, but the setup just did not work. This time it was the 3CX server that did not play nice.
As SIP/TLS sessions between our Cisco phones and 3CX would not get established, we started to collect network traces. Since we were in the possession of the private key of the server certificate we were able to decrypt the otherwise encrypted communication between the phone and the server (3CX). What we saw there was that after the phone presented its protocol specification against the server during a TLS negotiation (so-called handshake), the server would just close the connection.
We sent decrypted traces to 3CX support and got an answer, saying that our "phone is trying to talk SSL 2.0 and we do not support that old protocol". Being relatively new to the field of applied cryptography, we were inclined just to believe that 3CX blaming Cisco's choice of an "old protocol" was justifiable. So, we went on and filed another case with Cisco support, citing 3CX and the "old protocol" claim.
The answer came relatively quickly this time, referring us to the TLS RFC (2246 and 5246), where we could read that a client wishing to support an older protocol (SSL 2.0 in our phone's case), but also being able to support a new one (TLS 1.0 in our case), may start talking the old protocol, indicating the support for the new one, thus allowing the server to decide which protocol to use. The server in our case — once registering that the client, while initiating an SSL 2.0 connection also supports TLS 1.0 — would have needed to answer with the new protocol and the phone would have forgotten the SSL 2.0 and started communicating in TLS. But instead, the server closed the connection. That was against the RFC and we returned to 3CX with our findings.
The screenshot above depicts a part of an SSL handshake with an indication of SSL 2.0 as an initial protocol specification (underlined in red) and a supported TLS version as an alternative (underlined in blue).
By now we knew that both Cisco and 3CX used OpenSSL for their implementation of TLS. We looked at how OpenSSL handles these types of requests and saw that it was a matter of linkage options in order for it to start working. It took us some effort to persuade 3CX to investigate the issue, but it took 3CX almost half a year, including QA, to implement the change. Although thankfully, we were provided with a private patch that had the needed functionality and were able to carry on with our findings.
It was almost a success, until suddenly, 3CX, who seemingly stroke a strong partnership with Snom, tells us that SPA phones should not be considered anymore, that Cisco will likely stop producing them and that we should switch to Snom phones, that have everything we needed out of the box. By that time we had deployed SPAs in the field already, and so we are not immediately convinced, but still, we decide to give it a shot.
Snom is a German-based company with good quality phones, a sound roadmap, and German engineering. Their phones do look good and robust, although I personally still prefer the Cisco finishing to Snom's.
We presented our long case to Snom engineers in full detail and were told that their telephones were able to complete our task with ease and that TLS works out-of-the box with their phones. We received 3 different models and started testing.
To our big surprise, the out-of-the-box functionality was nowhere to be found. As soon as we imported Cisco's root certificates into the phone the certificate chains became broken and TLS would stop working altogether. After filing a support case — now with Snom — we were told that the current firmware has a bug and so it would not be possible to use other root certificates than those already installed in the phones. New firmware was in the making, but there was no ETA for the release, so we stowed the phones for a later date.
On a side note, by this time I started realizing that most vendors — whether phone or software — often backed off once encryption was being considered and it started getting complicated. Snom phones did work fine without encryption, but so did Cisco's. But with encryption enabled, all sorts of problems began to creep in, and we could not find anyone capable of handling them.
A year or two have passed and we decided to revisit the Snom case and took the phones off the shelf. After updating to the latest firmware, we saw that only two out of three phones worked. Another case was opened. The engineers at first are perplexed, because surprisingly "the hardware platform is the same across all three models" — but then suddenly it turns out that the smallest model is indeed different and that the difference would even prevent it from working properly in our setup!
The phones were sent back to Snom, the TLS case was closed, though not completely resolved for us. But by now we had a surprise waiting for us from our SIP server vendor, 3CX.
Software vendors update their product from time to time. This is obviously a good practice, because bugs need to be killed and new features have to be added — the software evolves. Along the way they sometimes make architectural changes that might break loosely coupled nodes within your infrastructure.
That was our case with a new version from 3CX. The system now assumed that its web interface, TLS endpoints for SIP and everything else that had to do with SSL was governed by one and the same certificate and was to be publicly trusted if we wanted everything to work fine. This was new, since previously we had a certificate signed by Cisco and it was not publicly trusted, since Cisco's Certificate Authority was a private one and the certificates it signed were normally not for open publication.
So, we had to obtain a new certificate that was (a) publicly trusted, (b) trusted by Cisco and (c) was wildcard-capable, because we needed to repurpose it for many different things within our VOIP infrastructure. This normally would not be an accomplishable task — at least during our first experiments — since Cisco phones did not support any third-party certificates. All had to be signed by Cisco. For reasons that now can only cause speculation, and that even Snowden might have an idea about the purpose of such a policy, Cisco suddenly had a pleasant surprise for us in their pipeline.
Read the whole article as one document
 Desktop as a Service: http://en.wikipedia.org/wiki/Desktop_virtualization
 Secure Socket Layer: a predecessor to TLS.
 Request for Comments, a publication type from the InternetEngineering Task Force, defining standards for the Internet: https://en.wikipedia.org/wiki/Request_for_Comments