This is a follow up to my post on the man-in-the-middle attack that Telkom continues to use, as well as to the Telkom response in the awesome TechCentral article and new thoughts brought about by the Reddit post.

A real threat

Shortly after I posted my article I was contacted by someone (let us call them Person X ) who went snooping based on my post and found that Telkom did indeed have a major security flaw in the system. We agreed to hold on sharing the information publically until Telkom fixed it or enough time had elapsed to show they didn’t care. Telkom appear to have fixed it, as far as I can tell, so let us dig into this major attack vector.

Recapping the content that is served to you is made up of three pieces:

• JavaScript – this is appended to other JavaScript to ensure it runs and the entire modified JavaScript is returned from a specific server. This server is only visible on the Telkom ISP network.
• HTML – bit of HTML is loaded, once again from the same server as above.
• Image – the graph showing you the usage is a static image served from a server which is (oddly) available everywhere: http://images.telkomsa.net/

Person X, realised that the images came from a folder ibn and that folder had directory listing enabled, which isn’t a good practise but, normally, not a major security concern. What the directory listing showed, besides the files, is the version number of the server which was (at the time) 2.0.52.

Apache 2.0.52 was released in November of 2004… 11 years ago and has NUMEROUS security flaws. There is no reason to run this version at all, it is insecure and points out a major security procedure flaw in Telkom. Using one of the flaws, it was possible to replace the specific image on the server with anything you wanted. For example, you could swop the image file with a flash file that would enable you to use one of the recent zero day attacks against flash and leveraging Telkom’s system to deliver said flash file to the user for you thus allowing you to own the target machine.

Remember this popup only shows to people who meet the following three requirements:

1. Telkom ISP client
2. At or near their limit
3. Have not opted out

I would assume that the tech savvy of Telkom's users have opted out, and those near their limit are soft capped, meaning that security patches are slower to get to them. In short, it is the most vulnerable group who would be targeted. Scary right?! Let us hope Telkom fixed all the issues with that server and all the others.

Comments on Telkom’s choice wording

I am really happy Telkom has responded and is fixing things, that is all I wanted. That said, their choice of wording and delay on commenting until they fixed the issue is interesting. Let us break down their response and I am avoiding nit-picking since I could do a lot of that too.

In technical terms, we refer to it as an HTTP redirect

That is true. It is an HTTP redirect, but the manner it is used in is not the traditional sense of an HTTP redirect where the destination server tells the client to go somewhere else. Here a man in the middle (see what I did there ) is doing the telling, and thus I chose the term MITM attack as the description. It is not an exact description, but it is a description that describes the entire scenario and not just one choice aspect.

HTTP redirect is a common mechanism used in service provider networks for content caching and to optimise video streaming

True again. That said, in those scenarios the service providers are not changing the content merely the destination to be better for the user. Telkom is changing the JavaScript content, there is a fundamental difference there.

does not alter the Web service content

In my previous post, I showed they are changing the JavaScript content, so that is pretty wrong.

is not a security risk

See above and then realise it was only true when it was said, not the day before. 

will not ‘break’ a website

The web  is a big place, it is impossible to know that. I also wonder why break is in quotes – is there more than one definition of break?

Performance impact

How bad is the download size issue really? It is 200k of data, it is all local so latency is low and the image URLs are constant so caching can happen to them. Also if Telkom is smart, then they are not counting this towards your cap. You will likely find that if the JavaScript is loaded in the header of the HTML page it could slow down rendering more than it could slow down network I/O. My gut says that it is insignificant in the grand scheme of things and I don’t believe I got that across well before. I personally feel, Telkom as an ISP, should be squeezing every drop of performance out and this would be a great start. Enough drops will fill a bucket.

I have been asked recently about the ES6 support in VS and I haven’t had a good answer about how much of it is supported right now (i.e. VS 2015 RC) and what is supported. The general feeling from the askers is that VS is far behind in this space, so the only reasonable thing to do is for me to test this and let you know.

Using Luke Hoban’s awesome page on ES6 features gave me a great point to kick off from around the different features. I then tested each feature in VS 2015 RC and also in VS Code (our new lightweight, cross-platform, free IDE) and in the end Visual Studio has 70% implemented and Code has 94% implemented. What does implemented mean though? It means no errors & and the IDE does the right IntelliSense stuff for you. There is on-going work with both IDE’s so this will improve but as a baseline for discussion it is useful.

The way I worked it out is to assign a 1 (works) or zero (doesn’t work) to each feature. In some cases I assigned a middle value because it kinda worked and the footnotes will explain that. The only missing one on here is the reflect API. I don’t have an example I can work with around that yet, so I did not include it. If you find any errors here, please let me know and I will gladly update!

	VS	Code
Arrows	1	1
Classes	0,51	1
Enhanced Object Literals	1	1
Template Strings	1	1
Destructuring	0	1
Default + Rest + Spread	0,52	1
Let + Const	1	0,83
Iterators + For..Of	1	1
Generators	0	1
Unicode	0	1
Modules	0	1
Module Loaders	0,54	0
Map + Set + WeakMap + WeakSet	1	1
Proxies	1	1
Symbols	1	1
Subclassable Built-ins	0,55	1
Math + Number + String + Array + Object APIs	1	1
Binary and Octal Literals	1	1
Promises	1	0,96
Tail Calls	1	1
Total	70%	94%

Footnotes:

1. No issues in the IDE are raised but the IntelliSense is lacking.
2. No issues in the IDE are raised but the IntelliSense is lacking.
3. There is an odd issue where an extra warning is raised in the wrong place. It doesn’t break anything but isn’t ideal.
4. No IntelliSense on the loaders.
5. Again lack of IntelliSense is the issue.
6. Weird warning appearing, which looks like a bug.

In February this year I was contacted by the team at PACKT Publishing about being a technical reviewer for a book which was underway, in exchange I would get a free copy of the book and be listed in the book as one of the reviewers. I have toyed with the idea of writing a book for ages so this felt like a good idea to see what happens behind the scenes without committing to actually writing a book. I said yup to them, and today that book got published! You can get it at: http://bit.ly/json_cookbook (I am really excited about this)

The book* itself is a very interesting mix of content, from the very basics of JSON to introduction to MongoDB and storing data in it. It very much hits the nail on the head with the description “Quick answers to common problems”. What I think is really awesome about it, is that it really tries to cover a lot of languages & tools. So there is .NET, Java, Node.js, Android, ObjectiveC and more. The examples are really great as all work on a variety of OS’s too, so you can quickly try stuff out. I don’t think it is a book you It isn’t a book you would read end to end but rather give you guidance on where to start with problems related to JSON.

The experience itself of reviewing was interesting, each chapter took a few hours of reading and trying out the code and responding with details of issues found. The people I worked with at PACKT made it really pleasant. I never heard from the author or other reviewers, which in hindsight is odd but I think that maybe helped keep my responses focused and not have a bias in them.

If you do pick it up, let me know what you thought of it!

*Note: I’ve only see the content I reviewed, I haven’t seen the final book which maybe different.

Vorlon.js is an amazing tool for web developers as it brings the browser developer tools, aka F12, out of the limitations of a single browser and to the cloud. While you can run it on your machine, I believe to really get the full power of it, you need to put it on a dev server somewhere. The broader the reach, the more use. So with that mindset here is how to run it on an Ubuntu server and for this example I am use Azure to host that VM. I am using the Basic A2 (2 Cores, 3.5 GB) machine which is more than enough for Vorlon and means I can run the server pretty cheaply.

Step 1: Get NPM

Vorlon uses NPM, Node Package Manager, for distribution so you need that to get started. Before you begin, make sure you are up to date:

sudo apt-get update

Once that is done you can run the following to install NPM:

sudo apt-get install npm

If you didn’t follow my advice on doing the update first, or you ran into issues, you may need to run the command again with extra parameters:

Step 2: Get Node.js

Vorlon is built with Node.js, so you need to get that too. This took me surprisingly long to figure out, but thanks to this page I came right. The commands to run are:

curl --silent --location https://deb.nodesource.com/setup_0.12 | sudo bash -
sudo apt-get install --yes nodejs

Step 3: Get Vorlon

Now that the machine is setup correctly, you can get a copy of  Vorlon using NPM with the next command:

sudo npm i -g vorlon

Step 4: Run Vorlon

Running Vorlon is easy, just type: vorlon

For Windows users, make sure it is all lower case. It takes about 30secs and you will see the command output pop up

Once that shows up you can access it!

/usr/bin/env: node: No such file or directory

Every time I ran Vorlon originally I got the following error message: /usr/bin/env: node: No such file or directory

This is because I didn’t have Node.js installed. I had npm, which I assumed brought Node.js with it but it doesn’t and so Vorlon won’t run. In hindsight it is obvious, and even the error message tells me that but at the time – well it was an hour to figure out :/

Azure – don’t forget those endpoints

If you are on Azure, you will also need to setup the endpoints to allow access to the VM. To do this, go to the portal and open you VM. Click All Settings, then Endpoints, then Add. From there you need to add an endpoint that maps the public to the private. I keep both the same (1337) but you could make the public something different.

You want to code in TypeScript. You want to code with Sublime Text. Today, that is really really easy to do, thanks to Package Control and Microsoft, who is now providing a first class plug-in for Sublime Text to enable & light up TypeScript.

Step 1: Install Package Control

The installation instructions for it can be found here, but in summary you press Ctrl+` and then paste the following code (for Sublime Text 2, for 3 see their website) into the text box and press enter

import urllib2,os,hashlib; h = 'eb2297e1a458f27d836c04bb0cbaf282' + 'd0e7a3098092775ccb37ca9d6b2e4b7d'; pf = 'Package Control.sublime-package'; ipp = sublime.installed_packages_path(); os.makedirs( ipp ) if not os.path.exists(ipp) else None; urllib2.install_opener( urllib2.build_opener( urllib2.ProxyHandler()) ); by = urllib2.urlopen( 'http://packagecontrol.io/' + pf.replace(' ', '%20')).read(); dh = hashlib.sha256(by).hexdigest(); open( os.path.join( ipp, pf), 'wb' ).write(by) if dh == h else None; print('Error validating download (got %s instead of %s), please try manual install' % (dh, h) if dh != h else 'Please restart Sublime Text to finish installation')

It takes a few seconds where nothing happens and then, without much fanfare, the message window will say “Please restart Sublime Text to finish installation” – so do that.

Step 2: Add the TypeScript package

Press Ctrl+Shift+P to bring up the command palette and type in Package Control: Install Package and press enter.

You can use the arrow keys to navigate the list quicker and press enter on the right item rather than typing everything.

In the next command palette window type TypeScript and hit enter.

In the status bar it will start to download and install TypeScript:

Once that is done, it is done and you will get glorious TypeScript in Sublime!

Part 2 is here, click that now (it will open in another window) so you can continue the fun when you are finished here

The following image is a screen shot I took of a webpage in Chrome. If you view that same website, you will get a different experience because Telkom is using the same techniques as a man-in-the-middle attack to edit the code which the website uses, to serve Telkom’s own content.

The important bit which has been added, is the piece in the lower right hand corner of the page, which Telkom has added to the website without the permission of the website owner or the customer of the Telkom ISP account.

Admittedly this is a relatively benign addition, in fact it maybe seen as useful, and I can see it being sold that way to non-technical managers and executives. Do not be fooled though, even this simple addition can cause major issues for you.

Breaking of websites

It is impossible for Telkom to know what this addition will do to every website on the web. I will cover below the technicality of how it works below but in short they are adding JavaScript (the code of the web pages) to each page and that additional code could interfere with the existing code and web pages in unforeseen ways and ultimately can break a web page. The sheer size and complexity of the internet says that it is impossible for them to know for sure that they are not break a single website.

Exposing of security risks

The second issue is that they are exposing users to the potential of massive security risks. They may argue that their code is secure now and that maybe true, however that is a point in time view.

At the moment they are using a JavaScript library (a library being reusable code web devs use to make it easier to build websites) called jQuery and while the version they use has nothing reported regarding security issues, that doesn’t mean that in the future it will still be as secure. If a vulnerability is found in that library or in the additional code they wrote, it could be used to launch an attack against their users. You need to ask yourself, do you trust that the developers at Telkom who built this so much they could not have introduced a security issue? Do you trust that they are that good?

A second issue around the security, is that they need to host their content & systems which do this attack somewhere. If that server/s is compromised – either from an external attack or from a disgruntled employee they could use that for XSS attacks or key sniffing attack! As this “feature” becomes more widely known, you can be assured that those servers will become the target of more attacks because of their value of leveraging them to launching attacks on users will be very big. The reality is that it is just a matter of time before those servers are compromised.

Taking away your freedom to privacy

While it will always be possible for ISPs to see what the traffic you are sending & receiving is, it shouldn’t be easily accessible to the staff and people at the ISP. Having a server that can do this manipulation means that they are enabling a very easy point for someone to capture traffic and see what you are doing on the internet in such a way that it will be hidden from you.

While I am sure they will tell you they take security very seriously and that they do not allow that type of access to employees what is stopping an executive at a later stage from using this to prevent adverts from MTN showing up or causing web pages that support EFF or the DA to not load at all? Nothing and they have that power to do that, without oversight and without your permission. Do you trust Telkom enough to not abuse that?

Telkom doing this is violating a founding principal of the Internet: Net neutrality

Performance

Finally, and this is last because it is the least of the issues, they are making you download more 84.8Kb of extra code (see here for the data) and ~120Kb of extra images plus the manipulation of the web page slows down rendering. In short, they are making the web slower for you and helping use more of your bandwidth. They are also adding in extra servers and network hops which add even more performance concerns.

Technical details

How are they able to do this? From my testing they only do this to HTTP traffic, not HTTPS traffic, as that traffic is encrypted. While the encryption should help, it is not impossible to silently break and inject content as shown with the recent Lenono Superfish disaster which is basically the same thing as Telkom is doing except on an ISP level. It isn't easy or likely reasonable for them to do that, and I am not implying they are, just that they could and maybe motivated to add this to secure websites.

The first step in understanding this attack, is that Telkom is not changing the HTML (i.e. the web page) that is sent to your machine, that remains untouched (you can see the HTML for the above page at the time I got that injection of HTML here). So where is the new content coming from?  Telkom is very cleverly intercepting certain calls and redirecting them, so that unless you are actively looking for this, it appears transparent to the website and the user! What they are doing is watching for JavaScript files to be requested, and then appending additional code into those files. The additional code being jQuery 2.0.1 and then their own JavaScript which then is used to manipulate the web page.

To explain this it might help to know what a normal process of getting a file looks like:

1. You web browser requests a file that is needed for the website to load the web page
2. It hits the Telkom proxy, which is meant to help speed up the Internet by providing content to you from a closer location. They may not have a proxy, many ISPs do this so that they can provide a pure & secure experience.
3. If the proxy doesn’t have the content, it should ask the web server for it.
4. The web server provides the content to the proxy.
5. The proxy returns it to the browser unaltered.

The way this attack works is as follows:

1. You web browser requests a file that is needed for the website to load
2. It hits the Telkom proxy, which now checks to see if it is JavaScript or not that is requested.
3. If it is JavaScript, the request is changed to another server to handle.
4. That server then requests the original JavaScript from the website.
5. And the website provides it.
6. The Telkom server now appends its content to the JavaScript.
7. The resulting modified code is sent back to the proxy
8. The proxy returns the request to the browser.

Disclaimer: This is a rough view of it and without inside knowledge I cannot guarantee it is exactly like that. It maybe that there are changes but the principal will be the same.

If you sniff the traffic you can pick up the manipulation as the URLs for the JavaScript is being requested is being changed and you get URL’s that look like this:

http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js?q=Pb4KQL0xYJ…
http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js?q=RRNoU9x7uZ…

If we break down that URL, it is the server portion and a JavaScript file that Telkom generates [http://196.25.211.41/n/50fc073bb429a4eb303a6420235e9381.js]. The URL has two parameters, one a unique ID [q=Pb4KQL0xYJ%2F53kOn%2FSnvQA%3D%3D%7C1435397545%7C02dfd5fbe207b80c1859ed08333cd16d] to prevent caching (I assume) and the second the original file it is changing [url=http%3A%2F%2Fajax.googleapis.com%2Fajax%2Flibs%2Fjquery%2F1.7.1%2Fjquery.min.js].

Note: these URLs do not work outside the Telkom ADSL network.

FAQ

Are you sure your OS / Browser isn’t attacked? Are you sure this is Telkom?

In the image above it is Chrome on Windows 10. I have been able to reproduce this on my Windows Phone and also using Microsoft’s new Edge browser. This is on the network layer

And here is it again on the jQuery website using Microsoft Edge:

 

How are they showing content without changing the HTML?

JavaScript has the ability to edit and inject HTML. Here is the HTML for a page that is attacked and below is an image of the HTML which the browser has rendered. The differ because the HTML has been injected using JavaScript:

How can you be sure that this isn’t an agreement with the web site owner?

Because they are doing this everywhere. I was able to easily reproduce this with the programming site, StackOverflow (see image below). In this case they manipulated the jQuery JavaScript which that website uses. This is a great case because they load all their content from a CDN and you can compare what you got to the CDN and it shows a difference. StackOverflow will confirm they have never agreed to this.

Do you have examples of the JavaScript?

Yup. Here is the modified Modernizer from the PowerBase website: http://pastebin.com/1p4m5LnM

Here is the modified jQuery from StackOverflow: http://pastebin.com/XXY6b3n7 

I am on Telkom ADSL and I am not seeing this, you are wrong!

• You need to have Telkom as your ADSL & ISP. If I switch my ISP to FNB, for example, this doesn’t happen.
• Second it doesn’t happen on HTTPS, or at least I’ve never seen it. StackOverflow is a good example where it would always happen on HTTP, but when I switched to HTTPS (they support both) it stopped.
• It isn’t constant. I first picked this up a few days ago and then it stopped until the weekend.
• It also doesn’t happen on every website. I am assuming there is some requirements in regards to the JavaScript that they need to be able append to it.

What can I do to protect myself?

You can’t block those servers as that will break the websites where Telkom is injecting content into. Telkom has a web page which allows you to opt-out. I haven't been able to confirm it works yet but I will be keeping an eye out for it. Another possible tool to help protect you, you could use the HTTPS Everywhere extension to increase the amount of websites you access with HTTPS, thus lowering the amount of this type of nonsense.

Ultimately, you need to trust your ISP. They can see your data and manipulate it. You need to trust they won't. If they show they are unwilling to live up to the principals of the Internet or violate your trust, you only option is to change ISPs.

What if Telkom lets me opt-out, that is good enough right?

Opting out means that they will still have this system. The system will just check the database to see if you get the original files or the modified files. You still are slowed down for the check (versus performance of compared no check at all because this system is taken out). In the best case (i.e. just a DB lookup and then a redirect), once opted out that performance difference will be so marginal it doesn't matter. In the worst case, they could still direct you through the same (potentially vulnerable servers) and you could get a lot more overhead due to that, bad code etc....

The only approach which introduces no further security issues, no violation of net neutrality and no performance impact is for Telkom not to do this at all.

Updates

As I am totally happy to admit when I am wrong, any updates will be listed here

12h05 10 July 2015 - I wrote many ISPs do not use proxies by mistake. I meant many do, but only for performance reasons. Not for changing of content and added more info & clarification on the HTTPS breaking.

15h00 29 June 2015 - it was pointed out to me it isn't just the JS you download, it is the MASSIVE image too. So rather than 80ish Kb it is almost 200Kb in total! I've updated the article to reflect that.

15h30 29 June 2015 - Added points reflecting that this is a violation of net neutrality and the HTTPS everywhere extension which might help some people.

18h25 20 June 2015 - Added information about opt-out option. Added information around performance and how it may or maybe impacted by opt-in out. All of that had some rewording to the "What can I do to protect myself?" section.

I try not to rant on my blog and mostly I succeed. Today though I wanted to complain to Telkom about them inserting content into web pages on Twitter so I head to their website to find their Twitter account and I don’t find one, I find THREE different Twitter accounts and a situation so bad I cannot keep shut!

It isn’t ideal to have three and I can understand different components need it, in fact they look like a business account (1), their cellphone brand (2) and a general one (3). The part that baffles is what happens when I checked them.

The business account

This account has been taken over by spammers :facepalm:

The mobile account

That account is suspended :facepalm:

In fact only the general account works! Shit happens, people mess up and accounts get suspended or lost – it isn’t ideal or professional and you can mitigate the damage by removing it from your website. The fact Telkom can’t/won’t speaks volumes to their inability to care. There is no acceptable reason for those bad links to be up there and who ever runs their website should hang up their HTML spurs and find a new career.

Let them know

Want to help let them know this isn’t acceptable? Here is the form for the webmaster and here is a handy link to Tweet to them quickly.

Yesterday in Johannesburg we had an amazing //Build Tour event! We had top international speakers like Pete Brown join us and wow the audience with Windows 10, IoT, Edge and more! Unfortunately for the international speakers, they had flight issues and we had a hour and half gap before we could kick off, so a bunch of the local Microsoft people got together to provide a bit of local flavour in a pre-keynote event. I was very excited to share about Visual Studio 2015 and it got cut at the last minute as the presenters arrived! Part of my prep was to create a video of my talk and you can watch this content now:

You can get the demo code on GitHub.

With an agile team you should meet daily for a quick meeting. Normally that takes the form of each person answering a three question status update (What did I achieve, what will I achieve, what is blocking me). For a team working together it means these steps:

1. Team arrives at the shared team board
2. Team goes round and gives status update
3. Team goes back to work

Just because your team is distributed doesn’t mean this shouldn’t happen – it just means that you’ll find a few extra steps:

1. Team members who work together go to booked meeting room (you need a meeting room so those remote can hear over the normal work environment noise)
2. Be surprised to find some one in the room
3. Explain that you booked the room
4. Wait for them to leave the room
5. Connect the projector
6. Realise you left your microphone in your laptop bag, return to desk get it, return to meeting room
7. Plug it in
8. Open the digital scrum board
9. Start meeting software and join meeting
10. See one remote team member is already there – apologise for being late
11. wait for response
12. wait for response
13. realise they can’t hear you.
14. Both the remote team member and yourself try to figure out what is not working
15. Figure out that there is a mute hardware button that today (and today only) is turned itself on
16. Try again – success, they can hear you
17. Wait for the product owner to join.
18. Give up waiting and kick off meeting (everyone is seated now cause no ones legs last that long)
19. Go around the team and give status update
20. Product owner joins during second last team members status update
21. Product owner starts sharing a bunch of bugs and task updates that someone else is now capturing live
22. Finish status updates
23. Team decides to use this time that we are all here to discuss technical info around the new stuff from the product owner
24. Product owner is late for meeting and has to disconnect
25. Someone walks into the meeting room and asks if this is the meeting for organising the year end party
26. silence
27. No one answers
28. silence
29. Scrum master answers
30. Team discusses technical issues. Half are marked as needing more info and will be discussed with product owner in tomorrows meeting
31. Call ends
32. Team goes to get coffee (they need it) and says how glad they are using agile cause they save so much time with less meetings
33. two hours later
34. other team member joins call alone because the time zone he is in changed to daylight savings today

Repeat daily

Community night is an initiative from Microsoft South Africa to help everyone have a platform to engage and grow, through learning and networking in board communities. That sounds awesome on a press release, but what is it?!  It means that on the second Tuesday of the month, Microsoft in Johannesburg opens it’s meeting rooms up for user groups to use to meet, basically giving a safe venue and infrastructure that groups need for face to face events. Microsoft also sponsors food and drinks for the groups so that you can have those amazing hallway conversations. There are a few groups that take part in community night, but it does change based on the groups – so here are some of the regulars:

• Developer - http://developerug.org.za/
• Information Worker - https://www.facebook.com/InformationWorkerZA
• GameDev - http://www.makegamessa.com/
• Xamarin User’s Group - http://xamarin.meetup.com/
• SQL - http://sqlusergroup.co.za/
• Lean StartUp Circle - http://www.meetup.com/Lean-Startup-Circle-Joburg/

if you are interested in getting your group involved, please reach out and I will help put you in touch with the right folks.