Part 2 is here, click that now (it will open in another window) so you can continue the fun when you are finished here

The following image is a screen shot I took of a webpage in Chrome. If you view that same website, you will get a different experience because Telkom is using the same techniques as a man-in-the-middle attack to edit the code which the website uses, to serve Telkom’s own content.

The important bit which has been added, is the piece in the lower right hand corner of the page, which Telkom has added to the website without the permission of the website owner or the customer of the Telkom ISP account.

Admittedly this is a relatively benign addition, in fact it maybe seen as useful, and I can see it being sold that way to non-technical managers and executives. Do not be fooled though, even this simple addition can cause major issues for you.

Breaking of websites

It is impossible for Telkom to know what this addition will do to every website on the web. I will cover below the technicality of how it works below but in short they are adding JavaScript (the code of the web pages) to each page and that additional code could interfere with the existing code and web pages in unforeseen ways and ultimately can break a web page. The sheer size and complexity of the internet says that it is impossible for them to know for sure that they are not break a single website.

Exposing of security risks

The second issue is that they are exposing users to the potential of massive security risks. They may argue that their code is secure now and that maybe true, however that is a point in time view.

At the moment they are using a JavaScript library (a library being reusable code web devs use to make it easier to build websites) called jQuery and while the version they use has nothing reported regarding security issues, that doesn’t mean that in the future it will still be as secure. If a vulnerability is found in that library or in the additional code they wrote, it could be used to launch an attack against their users. You need to ask yourself, do you trust that the developers at Telkom who built this so much they could not have introduced a security issue? Do you trust that they are that good?

A second issue around the security, is that they need to host their content & systems which do this attack somewhere. If that server/s is compromised – either from an external attack or from a disgruntled employee they could use that for XSS attacks or key sniffing attack! As this “feature” becomes more widely known, you can be assured that those servers will become the target of more attacks because of their value of leveraging them to launching attacks on users will be very big. The reality is that it is just a matter of time before those servers are compromised.

Taking away your freedom to privacy

While it will always be possible for ISPs to see what the traffic you are sending & receiving is, it shouldn’t be easily accessible to the staff and people at the ISP. Having a server that can do this manipulation means that they are enabling a very easy point for someone to capture traffic and see what you are doing on the internet in such a way that it will be hidden from you.

While I am sure they will tell you they take security very seriously and that they do not allow that type of access to employees what is stopping an executive at a later stage from using this to prevent adverts from MTN showing up or causing web pages that support EFF or the DA to not load at all? Nothing and they have that power to do that, without oversight and without your permission. Do you trust Telkom enough to not abuse that?

Telkom doing this is violating a founding principal of the Internet: Net neutrality

Performance

Finally, and this is last because it is the least of the issues, they are making you download more 84.8Kb of extra code (see here for the data) and ~120Kb of extra images plus the manipulation of the web page slows down rendering. In short, they are making the web slower for you and helping use more of your bandwidth. They are also adding in extra servers and network hops which add even more performance concerns.

Technical details

How are they able to do this? From my testing they only do this to HTTP traffic, not HTTPS traffic, as that traffic is encrypted. While the encryption should help, it is not impossible to silently break and inject content as shown with the recent Lenono Superfish disaster which is basically the same thing as Telkom is doing except on an ISP level. It isn't easy or likely reasonable for them to do that, and I am not implying they are, just that they could and maybe motivated to add this to secure websites.

The first step in understanding this attack, is that Telkom is not changing the HTML (i.e. the web page) that is sent to your machine, that remains untouched (you can see the HTML for the above page at the time I got that injection of HTML here). So where is the new content coming from?  Telkom is very cleverly intercepting certain calls and redirecting them, so that unless you are actively looking for this, it appears transparent to the website and the user! What they are doing is watching for JavaScript files to be requested, and then appending additional code into those files. The additional code being jQuery 2.0.1 and then their own JavaScript which then is used to manipulate the web page.

To explain this it might help to know what a normal process of getting a file looks like:

1. You web browser requests a file that is needed for the website to load the web page
2. It hits the Telkom proxy, which is meant to help speed up the Internet by providing content to you from a closer location. They may not have a proxy, many ISPs do this so that they can provide a pure & secure experience.
3. If the proxy doesn’t have the content, it should ask the web server for it.
4. The web server provides the content to the proxy.
5. The proxy returns it to the browser unaltered.

The way this attack works is as follows:

1. You web browser requests a file that is needed for the website to load
2. It hits the Telkom proxy, which now checks to see if it is JavaScript or not that is requested.
3. If it is JavaScript, the request is changed to another server to handle.
4. That server then requests the original JavaScript from the website.
5. And the website provides it.
6. The Telkom server now appends its content to the JavaScript.
7. The resulting modified code is sent back to the proxy
8. The proxy returns the request to the browser.

Disclaimer: This is a rough view of it and without inside knowledge I cannot guarantee it is exactly like that. It maybe that there are changes but the principal will be the same.

If you sniff the traffic you can pick up the manipulation as the URLs for the JavaScript is being requested is being changed and you get URL’s that look like this:

http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js?q=Pb4KQL0xYJ…
http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js?q=RRNoU9x7uZ…

If we break down that URL, it is the server portion and a JavaScript file that Telkom generates [http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js]. The URL has two parameters, one a unique ID [q=Pb4KQL0xYJ%2F53kOn%2FSnvQA%3D%3D%7C1435397545%7C02dfd5fbe207b80c1859ed08333cd16d] to prevent caching (I assume) and the second the original file it is changing [url=http%3A%2F%2Fajax.googleapis.com%2Fajax%2Flibs%2Fjquery%2F1.7.1%2Fjquery.min.js].

Note: these URLs do not work outside the Telkom ADSL network.

FAQ

Are you sure your OS / Browser isn’t attacked? Are you sure this is Telkom?

In the image above it is Chrome on Windows 10. I have been able to reproduce this on my Windows Phone and also using Microsoft’s new Edge browser. This is on the network layer

And here is it again on the jQuery website using Microsoft Edge:

 

How are they showing content without changing the HTML?

JavaScript has the ability to edit and inject HTML. Here is the HTML for a page that is attacked and below is an image of the HTML which the browser has rendered. The differ because the HTML has been injected using JavaScript:

How can you be sure that this isn’t an agreement with the web site owner?

Because they are doing this everywhere. I was able to easily reproduce this with the programming site, StackOverflow (see image below). In this case they manipulated the jQuery JavaScript which that website uses. This is a great case because they load all their content from a CDN and you can compare what you got to the CDN and it shows a difference. StackOverflow will confirm they have never agreed to this.

Do you have examples of the JavaScript?

Yup. Here is the modified Modernizer from the PowerBase website: http://pastebin.com/1p4m5LnM

Here is the modified jQuery from StackOverflow: http://pastebin.com/XXY6b3n7 

I am on Telkom ADSL and I am not seeing this, you are wrong!

• You need to have Telkom as your ADSL & ISP. If I switch my ISP to FNB, for example, this doesn’t happen.
• Second it doesn’t happen on HTTPS, or at least I’ve never seen it. StackOverflow is a good example where it would always happen on HTTP, but when I switched to HTTPS (they support both) it stopped.
• It isn’t constant. I first picked this up a few days ago and then it stopped until the weekend.
• It also doesn’t happen on every website. I am assuming there is some requirements in regards to the JavaScript that they need to be able append to it.

What can I do to protect myself?

You can’t block those servers as that will break the websites where Telkom is injecting content into. Telkom has a web page which allows you to opt-out. I haven't been able to confirm it works yet but I will be keeping an eye out for it. Another possible tool to help protect you, you could use the HTTPS Everywhere extension to increase the amount of websites you access with HTTPS, thus lowering the amount of this type of nonsense.

Ultimately, you need to trust your ISP. They can see your data and manipulate it. You need to trust they won't. If they show they are unwilling to live up to the principals of the Internet or violate your trust, you only option is to change ISPs.

What if Telkom lets me opt-out, that is good enough right?

Opting out means that they will still have this system. The system will just check the database to see if you get the original files or the modified files. You still are slowed down for the check (versus performance of compared no check at all because this system is taken out). In the best case (i.e. just a DB lookup and then a redirect), once opted out that performance difference will be so marginal it doesn't matter. In the worst case, they could still direct you through the same (potentially vulnerable servers) and you could get a lot more overhead due to that, bad code etc....

The only approach which introduces no further security issues, no violation of net neutrality and no performance impact is for Telkom not to do this at all.

Updates

As I am totally happy to admit when I am wrong, any updates will be listed here

12h05 10 July 2015 - I wrote many ISPs do not use proxies by mistake. I meant many do, but only for performance reasons. Not for changing of content and added more info & clarification on the HTTPS breaking.

15h00 29 June 2015 - it was pointed out to me it isn't just the JS you download, it is the MASSIVE image too. So rather than 80ish Kb it is almost 200Kb in total! I've updated the article to reflect that.

15h30 29 June 2015 - Added points reflecting that this is a violation of net neutrality and the HTTPS everywhere extension which might help some people.

18h25 20 June 2015 - Added information about opt-out option. Added information around performance and how it may or maybe impacted by opt-in out. All of that had some rewording to the "What can I do to protect myself?" section.