13.08.2014 18:08
(Updated 2014/8/13) Syria offline - initial analysis of BGP (and explanation)
This blog post evolved over time - initially it was a mere scratchpad for notes during our initial research between 2012/11/29 and 11/30. Later, after Syria was back online again, I added a summary and some potential explanations of what might have happened at the end of this blog post.UPDATE 2014/8/13: It seems it was the NSA that hacked a router, according to Snowden. Scroll to the end for links
Next, we looked at Renesys' claim that 92% of all announcements were offline. However, the situation seems to be worse: at the time of this writing we did not find a single BGP prefix which was announced via STE AS29386. There still might be some other carrier which announces Syrian IP spaces but none of them went through Syria Telecom. Next, we looked at the list of known netblocks which are announced via AS29386 based on RIPE's stat.ripe.net service: https://stat.ripe.net/as29386#tabId=routing. This list was compared against our BGP table. We indeed could confirm that none of the networks were visible. UPDATE 00:45 a.m.: Cloudflare has a good analysis of the Internet blackout. Summary: a fiber cable cut seems unlikely according to cloudflare.UPDATE 2012/11/30, 11:00 a.m.: The last night was busy with figuring out some details. I could independently confirm that even the IP range, which was active in the Finfisher malware was not reachable anymore. This is interesting since this IP range was actually attributed to the Syrian government. So, even they are definitely offline. This is strange. If I were the Syrian government and I wanted to block rebels from communicating via the Internet, I wouldn't turn off my own connectivity as well. What happened? UPDATE: Renesys has an updated Analysis. There seems to be some traffic flowing out, but via other carriers. We could not confirm this. However what we could find out was...
Why is this traceroute so special? It was smuggled out via a VSAT connection!! So this is to the best of my knowledge a unique view of the Internet blockage as seen from inside Syria at the time of when it happened. Indeed, there seem to be many VSATs in Syria. The New York times confirmed.According to the same sources, regular landline phones still work in Damaskus. Also, accessing *.SY domains / servers from within Syria seems to still work (at least yesterday night). It's just the international wired links which are down. Somehow this proves my point that I made in an interview with "DerStandard". Still, VSATs and modems are slow - but at least some folks still had some connectivity.Speaking of DNS:
In addition there seem to be land cables to turkey - again according to Mikko.UPDATE 2012/11/30 14:45: Otmar and me cross checked the BGP announcements via a different mechanism. Since our initial analysis only included any network which was announced via STE, we thought we should better double check again and look if we had forgotten any Syrian network which was not going through STE. Therefore, we took a different approach: we extracted the syrian IP ranges from the maxmind DB, fetched the ASNs, compared against the full BGP feed and found that our last hope, a traceroute path to Syria via the indian carrier TATA terminates in Paris. So even though we chose a different path for our analysis, we end up with the same result: Syria is indeed offline.UPDATE 2012/12/2: it seems that the land line via Turkey mentioned above was planned but was not in operation during the Internet blackout. This makes it very hard to attribute the cause of the blackout to either the rebels or the government. As Renesys said in their updated blog:
If there was some physical damage to these lines or the landing site, then most probably the carrier operators should know about that. Did someone ask them already what happened? If they don't know of any damage, then something must have happened on the software level or further inland.Telecomix posted pictures of the Tarassul data center (note the Blue-coat (and the power cord that you could trip over ;-) ):
. Again, just to be sure - this is some info from the Internet. No idea how much solid evidence this picture is.
(source: http://xkcd.com/705/)Author: L. Aaron Kaplan
The blackout
2012/11/29: as Renesys and others pointed out, Syria seems to be offline since 10:26 UTC. Michael Kafka and me verified this fact via looking at BGP routing tables and looked at the potential damage to the Internet as a global system. Here is what we can confirm / figured out:AS29386 is the Syrian Telecom (STE). It seems to be the central hub for all connections to/from Syria. So, we started first with this network. This is how STE is connected:Next, we looked at Renesys' claim that 92% of all announcements were offline. However, the situation seems to be worse: at the time of this writing we did not find a single BGP prefix which was announced via STE AS29386. There still might be some other carrier which announces Syrian IP spaces but none of them went through Syria Telecom. Next, we looked at the list of known netblocks which are announced via AS29386 based on RIPE's stat.ripe.net service: https://stat.ripe.net/as29386#tabId=routing. This list was compared against our BGP table. We indeed could confirm that none of the networks were visible. UPDATE 00:45 a.m.: Cloudflare has a good analysis of the Internet blackout. Summary: a fiber cable cut seems unlikely according to cloudflare.UPDATE 2012/11/30, 11:00 a.m.: The last night was busy with figuring out some details. I could independently confirm that even the IP range, which was active in the Finfisher malware was not reachable anymore. This is interesting since this IP range was actually attributed to the Syrian government. So, even they are definitely offline. This is strange. If I were the Syrian government and I wanted to block rebels from communicating via the Internet, I wouldn't turn off my own connectivity as well. What happened? UPDATE: Renesys has an updated Analysis. There seems to be some traffic flowing out, but via other carriers. We could not confirm this. However what we could find out was...
How does it look from inside of Syria?
I received a traceroute from within Syria via a friend of a friend of a friend. So, please be aware of this and treat it merely as a hint. However, to me it looks real. The first IP addresses are intentionally obfuscated to protect the individual who made the traceroute. It is interesting to see that the traceroute ends at:% Information related to '82.137.192.0 - 82.137.199.255'inetnum: 82.137.192.0 - 82.137.199.255 netname: SY-STE-PDN-NETWORK descr: STE PDN Backbone Network country: SY admin-c: WS1833-RIPE tech-c: MS4418-RIPE tech-c: WS1833-RIPE status: ASSIGNED PA mnt-by: STEMNT-1 mnt-lower: STEMNT-1 remarks: ----------------------------------------------------------- remarks: INFRA-AW remarks: ANY Problems Like SPAM, Hacking etc.. remarks: Send Email to wmsalem@gmail.com remarks: ----------------------------------------------------------- source: RIPE # Filteredperson: Mostafa Sawan address: Syria phone: +963 21 5229803 fax-no: +963 21 52288992 nic-hdl: MS4418-RIPE mnt-by: STEMNT-1 source: RIPE # Filteredperson: Weam Salem address: PDN- STE phone: +963-11-4461475 fax-no: +963-11-4466892 nic-hdl: WS1833-RIPE mnt-by: WS-MNT source: RIPE # Filtered% Information related to '82.137.192.0/18AS29256'route: 82.137.192.0/18 descr: STE Public Data Network Backbone and LIR origin: AS29256 mnt-by: STEMNT-1 source: RIPE # Filtered% Information related to '82.137.192.0/18AS29386'route: 82.137.192.0/18 descr: STE Public Data Network Backbone and LIR origin: AS29386 mnt-by: STEMNT-1 source: RIPE # FilteredIn other words, we are observing the path that data pakets would take from within Syria to google (8.8.8.8). However, the network where everything stops is STE (precisely the STE PDN Backbone Network). It will be very interesting to compare this traceroute later when the internet connectivity is restored again. Then we will be able to see exactly at which point data pakets were dropped.Here is a screenshot of the traceroute:
Why is this traceroute so special? It was smuggled out via a VSAT connection!! So this is to the best of my knowledge a unique view of the Internet blockage as seen from inside Syria at the time of when it happened. Indeed, there seem to be many VSATs in Syria. The New York times confirmed.According to the same sources, regular landline phones still work in Damaskus. Also, accessing *.SY domains / servers from within Syria seems to still work (at least yesterday night). It's just the international wired links which are down. Somehow this proves my point that I made in an interview with "DerStandard". Still, VSATs and modems are slow - but at least some folks still had some connectivity.Speaking of DNS:
$ dig -t ns sy; <<>> DiG 9.7.6-P1 <<>> -t ns sy ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26318 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 0;; QUESTION SECTION: ;sy. IN NS;; ANSWER SECTION: sy. 86400 IN NS ns1.tld.sy. sy. 86400 IN NS ns2.tld.sy. sy. 86400 IN NS pch.anycast.tld.sy. sy. 86400 IN NS sy.cctld.authdns.ripe.net.;; Query time: 18 msec ;; SERVER: x x x x #53(x x x x ) ;; WHEN: Fri Nov 30 12:49:46 2012 ;; MSG SIZE rcvd: 125This means that ripe.net is still a secondary authoritative DNS server for *.sy domains. You can still query *.sy domains from there in case you know what you are searching for. This fits together with claims that *.sy domains which are hosted in the US are still reachable.
Physical connections
Syrian sea cables (source: Mikko Hipponen):In addition there seem to be land cables to turkey - again according to Mikko.UPDATE 2012/11/30 14:45: Otmar and me cross checked the BGP announcements via a different mechanism. Since our initial analysis only included any network which was announced via STE, we thought we should better double check again and look if we had forgotten any Syrian network which was not going through STE. Therefore, we took a different approach: we extracted the syrian IP ranges from the maxmind DB, fetched the ASNs, compared against the full BGP feed and found that our last hope, a traceroute path to Syria via the indian carrier TATA terminates in Paris. So even though we chose a different path for our analysis, we end up with the same result: Syria is indeed offline.UPDATE 2012/12/2: it seems that the land line via Turkey mentioned above was planned but was not in operation during the Internet blackout. This makes it very hard to attribute the cause of the blackout to either the rebels or the government. As Renesys said in their updated blog:
The restoration was achieved just as quickly and neatly as the outage: like a switch being thrown. Does that mean that we believe the government (or the opposition) threw the switch? Frankly, the data available just don't support attribution at this point, despite all the speculation.Where do the sea cables come in? Close to Tartous (Tartus) (and by the way, Tartus hosts the russian naval base).Greg's cable map site has a nice list of lines arriving at Tartus:
If there was some physical damage to these lines or the landing site, then most probably the carrier operators should know about that. Did someone ask them already what happened? If they don't know of any damage, then something must have happened on the software level or further inland.Telecomix posted pictures of the Tarassul data center (note the Blue-coat (and the power cord that you could trip over ;-) ):
. Again, just to be sure - this is some info from the Internet. No idea how much solid evidence this picture is.
Post-fact analysis - what actually happened?
Hard to tell. Personally I still believe it was caused by a "network maintenance" event. That is - someone got a phone call and had to disable BGP announcements. Or there was some upgrade to DPI systems. However, the fact that all networks were gone (including the government networks and the network range which was hosting the famous FinFisher command & control server) speaks exactly against this theory. If we apply reasoning (which might be the wrong thing in a war(?)), then we should assume that these government networks should have been up and running. But they were not as we could independently confirm. Also there are multiple sea cables going into Syria. But they seem to come together at one point in Tartus. A single point of failure.UPDATE 2014/8/13 - what really happened!!
As of 2014/8/13 some new Snowden documents - according to Mikko Hypponen (who quotes Wired, see also ArsTechnica) - seem to reveal that actually the NSA really broke the Syrian 'net. I might have to update this whole blog now!Lessons learned
In summary I can only conclude that we don't know for sure what really happened, but we know for sure that it is really a bad idea to have a single point of failure.More specifically, I believe we need:physical redundancy
It makes sense to have many fiber lines. If we were to believe the official story that rebels blew up a communications cable, then redundant connections would have avoided any blackout.organizational redundancy
Having one organization and one technician administering vital systems which are a single point of failure is a bad idea. Humans make errors. Or they can get bribed. Who knows what really happened in Syria? Maybe the technician had to upgrade the DPI system and the upgrade did not work? Having multiple organizations / ISPs minimizes these risks. Nevertheless...they have guns!
All of the above is of course only a recommendation which works in a democratic nation at peace. If there are multiple SWAT teams with guns entering multiple ISPs' office at the same time and ordering a nationwide shutdown, there is little you can do but to shut down everything.UPDATE 2014/8/13: they have guns and hackers!
I guess even a foreign state can enter routers/ISPs remotely and disconnect a whole country. This is scary. Nevertheless, I am very happy that Austria at least is well connected with multiple redundant links and with multiple ISPs. But even here we should learn the lesson from Syria: build redundancy! It's important! Srsly.UPDATE 2014/8/13: and build your own routers and test test test , pentest and hack them some more before others hack them. On a lighter note, I was wondering why I personally got so excited about the subject. Well, I guess xkcd answered it for me:
(source: http://xkcd.com/705/)Author: L. Aaron Kaplan