F-Root DNS server moved to Beijing
Systems such as DNS (root) servers often rely on anycast technology to improve availability and response time. The idea behind anycast is that the same prefix is announced from multiple geographically separated systems. As a result the client should always end-up at the closest (as seen from a BGP perspective) server.

Often these anycast nodes are deployed to only serve specific network and/or regional areas, making them only reachable from the networks where they are deployed. In the case of the F-root server, operated by ISC, there are quite a few local servers. See this page for a detailed list: https://www.isc.org/community/f-root/sites One of these servers is the F-root located in China.

This weekend for about 25 hours this particular root server was reachable not only from China, but from several places in the world when using IPv6. Normally this shouldn’t be a real problem, as the root servers return the same DNS response regardless of their geographical location. However in the case of China, all traffic typically passes through the great firewall where DNS responses for requests such as facebook.com are often rewritten to different false IP addresses. Please note that I am not aware of any data to confirm if this is what happened in this particular case!

Technical details
This weekend between 01 Oct 2011 17:56:12 and 02 Oct 2011 19:01:09 GMT a number of providers thought that the best IPv6 path to the F-root server (AS3557), was to the F-root in China.

The prefix for the F-root server is 2001:500:2f::/48 and is originated by the following AS numbers:
3557 ISC-F-ROOT Internet Systems Consortium, Inc
1280 ISC-AS1280 Internet Systems Consortium, Inc.
30132 ISC-AMS1 Internet Systems Consortium, Inc

By looking at the next hop AS number we can get a good feeling of where the server for this particular announcement resides. When analyzing the announcement for 2001:500:2f::/48 we saw one new AS we had not seen before outside of China, AS55439.

When looking at all announcements for the F-root in China, we saw the following constant in the ASpath:
6939 23911 18344 37944 24151 55439 3557

In this case AS55439 is the AS for ISC-PEK2 Internet Systems Consortium, (Beijing, China). Next the announcement was passed along to AS24151 (China Internet Network Information Center), AS37944 (CHINA SCIENCE AND TECHNOLOGY NETWORK), AS18344 (CNCGROUP-BACKBONE-NORTH), it then ended up at the AS23911, the China Next Generation Internet Beijing IX. That’s where it was learned by AS6939 Hurricane Electric, the world’s largest IPv6 provider. Hurricane electric then advertised it to its customers.

The dataset I used showed that numerous Hurricane electric customers in countries worldwide, including for example Germany, South Africa, Brazil and the US selected this path to China. Some of the large providers include AT&T, Telia and Interoute.

What does all this mean
It’s likely that DNS queries over IPv6 to the F-root server from clients worldwide were answered by the F-root in China. Although this is in itself is not necessarily a problem, it does expose this traffic to the Great Chinese Firewall with the risk of altering the responses.

This is not the first time this happens, last year Renesys reported about a similar event involving the I-root server and China.

The only way to make sure that DNS responses are not rewritten is to use DNSSEC, this will allow the resolver to verify the authenticity of the response.
With regards to BGP, in this case all that seems to have happened was a BGP leak. Things like this are typically the result of an operator mistake. Best practice is to monitor for these kinds of things, so you’ll be alerted as soon as it happens and as a result allowing operators to limit the effect of such an event. BGPmon.net provides users with several flexible ways to monitor BGP announcement and to check if they match your BGP policies.

The unrest in Syria continues and as of this morning it seems that the Syrian government has shutdown about

of all Syrian networks.

The Internet in Syria is dominated by “The Syrian Telecommunications Establishment”, which routes its networks from AS29256 and AS29386. Besides these providers there are AS6453 – Tata communications which routes 6 Syrian prefixes and AS39154 The Syrian Higher Education Network.

As of this morning only 19 of the normally 56 Syrian prefixes are routed.  Interestingly the prefixes that are no longer routed are normally  originated by AS29256 and AS29386, “The Syrian Telecommunications Establishment”.  The 6 Tata communication prefixes as well as the Syrian Higher Education Network have not been affected.

The table below show how many prefixes are normally routed by these providers, and how it has changed over the last hours.

The pie charts below show how the Syrian network are distributed over the different providers. Clearly visible are the reduced number of prefixes announced by “The Syrian Telecommunications Establishment”, AS29256 and AS29386.

Prefix Distribution June 1

Prefix Distribution June 3

Update June 4th, 2011
As of this morning (approximately 08:00 UTC) all Syrian networks routed by “The Syrian Telecommunications Establishment” are back online! Some prefixes came back earlier this morning. Also see Google’s Transparency report here.

Many of you remember the story of about a year ago, when we reported that a Chinese network was announcing a significant part of the prefixes on the Internet.  Networks affected by this incident included big names such as dell.com and cnn.com as well as U.S. government (.gov) and military (.mil) sites, including those for the Senate.
This week a similar story appeared. Although the number of networks involved was low, it did affect one of the world most popular websites, Facebook!

Barrett Lyon reported on his blog that he noticed that AT&T was routing traffic to Facebook through a Chinese network (China telecom AS4134). In this blog post we will take a closer look at what exactly happened.

The raw data
By analyzing the raw data we can indeed see BGP announcements for several of Facebook prefixes with rather long and odd looking ASpaths. We also see several US based providers, such as AT&T, that selected a path through SK Telecom (Hanaro Telecom South Korea) and China Telecom.

It seems to all have started on Tuesday March 22, at 07:15:02 GMT. That’s when we see the first announcement for one of Facebook’s networks, routed through Korean provider SK Telecom. The last announcement via SK Telecom was at Tuesday, 22 Mar 2011 16:11:02 GMT, so about 9 hours in total.

The impact of this incident varied per provider. Some networks didn’t use the path through Korea, some only for a few of Facebook’s prefixes as some for (almost) all of Facebook’s prefixes.

Two Japanese providers, who peer directly with SK Telecom (Korea) routed the following Facebook networks trough AS9318, SK Telecom (Hanaro Telecom South Korea)

• 204.15.20.0/22
• 66.220.144.0/21
• 66.220.152.0/21
• 69.171.224.0/20
• 69.171.239.0/24
• 69.171.240.0/20
• 69.171.255.0/24
• 69.63.176.0/21
• 69.63.184.0/21
• 74.119.76.0/22

The providers affected by this were:
Internet Initiative Japan Inc. (IIJ)   ASpath:  2497 9318 32934 32934 32934
KDDI   ASpath: 7660 2516 9318 32934 32934 32934

Several other large providers such as Savis, AT&T, Tinet, KPN, Telia, Qwest, AboveNet and Telecom Italia (note this list is not inclusive, there are several more) were also affected, however in these cases only two prefixes were affected.

69.171.224.0/20
69.171.255.0/24

All of these providers learned this announcement via AS4134, which is China telecom. In all cases the Facebook prefixes were prepended three times. Different peers on different locations saw the announcements, the common part in the ASpath was:

4134 9318 32934 32934 32934

This proves that the announcements were not spoofed by China telecom, as others learned it directly from SK Telecom as well.

What exactly did AT&T see at the time of the incident?

This is a snapshot of the AT&T routing table at March 22^nd,  8AM, showing Facebook networks only. Note that only 2 networks were routed through Korea and China.

As can be seen from the snapshot above, the majority of the prefixes (as seen from AT&T) are routed through Level3 (3356). So what happened to the other 2 prefixes? Why was the path to those networks not through Level3. To answer that question we need to know the path to these networks from Level3′s perspective. When looking at the same datasource we see the following for 69.171.224.0/20:

This is interesting; it shows that Level3 learns the Facebook prefix with a rather long ASpath, Facebook’s AS is prepended 5 times. This explains why AT&T chose the path via China and Korea. Even though it’s long, it was shorter than the heavily prepended path via Level3. As a result providers who peer with China Telecom and had this shorter alternative, would have chosen the path via China.
Global Crossing, another large global Transit provider showed the same behavior as Level3, an ASpath with 5 times Facebook’s AS32934.

What could be the reason?

For some reason the path via providers such as Level3 or Global Crossing, Transit networks that are commonly used by the affected providers, were advertised with a very long ASpath. And as a result the providers that have a peering agreement with China Telecom (such as AT&T) started to use the shorter path via China and Korea.
SK Telecom normally is not one of Facebook’s transit providers, but likely just a normal peer.  SK Telecom then announced these Facebook prefixes to its peers (IIJ, KDDI and  China Telecom) which started to use them.

China Telecom did not filter these Facebook announcements learned via SK Telecom and re-announced them happily to its peers. As a result traffic from several major global providers started using the path through China and Korea to reach some of Facebook’s networks.

Why the providers such as Level3 and Global Crossing did not have a shorter path to these two Facebook networks is unknown. But it’s likely that it had to do with the Facebook BGP setup / infrastructure.

Keep in mind….
It’s likely that both China Telecom and SK Telecom have a presence in the US. And as such it’s not necessarily the case that traffic was actually routed through China.  If someone has a traceroute from AT&T or any of the other networks taken at the time of the incident, please post this in the comments, this can helps us determining how the actual traffic flowed.

Facebook has a lot of data and does all kinds of CDN (content delivery) tricks to get data to its users.  Depending on your location you get a different DNS result and send to a different server. This might have been the reason why there were no reports of widespread outage.

A few moments ago I noticed the first signs of life from the previously unreachable Egyptian networks. We saw the first BGP announcements for Egypt come in at at 09:30:48 UTC. And as of 10:52 UTC the website of Noor data networks was reachable again. It looks like the majority of the providers are now back.

Your prefix: 81.21.104.0/24:
Prefix Description: Egypt:egypt.gov.eg eGOV Project_2
Update time: 2011-02-02 09:48 (UTC)
Detected by #peers: 14
Detected prefix: 81.21.104.0/24
Announced by: AS31065 (MCIT -- Egyptian Ministry of Communications and Information Technology)
Upstream AS: AS24863 (LINKdotNET-AS)


One of the blocks that did not came back immediately was 193.227.1.0/24. This block hosts FRCU.EUN.eg, one of the primary name servers for the EG domain. 193.227.1.0/24 is normally announced by AS2561 Egyptian Universities Network (EUN), which before these events announced 17 networks. Currently I only see 6 of these networks, some more specifics seem to be missing. I do see 193.227.0.0/18 which is an aggregate for this netblock. Reports are that the server is reachable again from most places, however I personally can’t reach it from the research network in Canada right now, from my Level3 server in Amsterdam it does work however. So I assume that the Egyptian Universities Network is only partly restored. This could result in slower DNS responses for Egyptian domain names for some users.

Both Ripe and Renesys have some other nice graphs visualizing Egypt coming back online.

Egypt has been offline for 5 days, this is truly unprecedented in these modern days.  It’s been interesting to see how alternative ways of electronic communications have been used and how Ad hoc Internet connections have been made available. I’ve read about amateur radio sending out morse code as well as lot’s of dial-up numbers that were distributed for Egyptians to use.

Click for latest updates:  Last updated at January 31, 21:00 UTC

Different media are reporting that Internet and other forms of electronic communications are being disrupted in Egypt.  Presumably after a government order in response to the protests. Looking at BGP data we can confirm that according to our analysis 88% of the ‘Egyptian Internet’ has fallen of the Internet. In this post I’ll  share some observations I made with regards to the reachability of Egyptian networks and providers.

What’s different in this case as compared to other ‘similar’ cases is that all of the major ISP’s seem to be almost completely offline. Whereas in other cases, social media sites such as facebook and twitter were typically blocked.  In this case the government seems to be taking a shotgun approach by ordering ISP’s to stop routing all networks.

Networks affected

Egyptian Routes

When looking at the data it’s clear that many Egyptian networks have fallen off the Internet. Let’s start by looking at a quick summary. Yesterday there were 2903 Egyptian networks, originated from 52  ISP’s. Transit was provided via 45 unique isp’s.
Today at 2am UTC, the numbers look quite different, there were only 327 Egyptian networks left on the Internet. These were originated 26 by ISP’s.
So 88% of the Egyptian networks is unreachable!

Below you’ll find a table with the top 10 providers in Egypt. It shows how many Egyptian networks were announced earlier this week and how many are reachable today.

As you can see in the table below, right now most autonomous systems (ISP’s) are no longer announcing any, or at the very least, significantly less prefixes.

Interestingly the only provider that doesn’t seem to be impacted by this is AS20928  (Noor Data Networks).

Below is the list of providers that are still announcing networks (based on routeviews data):

When did this start?

By looking at some Egyptian websites and looking at when they became unreachable we are able to determine when the problem started.

At this point egypt.gov.eg is offline. This network, 81.21.104.0/24 was withdrawn at January 27^th at  22:28 UTC . Another example is www.ahram.org.eg an Egyptian news paper. This network 196.219.246.0/24, became unreachable at the exact same time, January 27^th at  22:28 UTC.

Update (Jan 28 18:36 PM UTC)
At this point only 239 Egyptian networks are reachable, this means that 91% of the Egyptian routes are unreachable. Noor networks remains the only provider that seems to be unaffected by this.

Vodafone has confirmed on their website, that they have been instructed to shutdown services in parts of the country
http://www.vodafone.com/content/index/press.html

All mobile operators in Egypt have been instructed to suspend services in selected areas. Under Egyptian legislation the authorities have the right to issue such an order and we are obliged to comply with it. The Egyptian authorities will be clarifying the situation in due course .

Update (January 29, 17:48 UTC)
It’s been reported that Mobile phone services have been restored. I found this confirmation on the Vodafone website

Vodafone restored voice services to our customers in Egypt this morning, as soon as we were able.
We would like to make it clear that the authorities in Egypt have the technical capability to close our network, and if they had done so it would have taken much longer to restore services to our customers.
It has been clear to us that there were no legal or practical options open to Vodafone, or any of the mobile operators in Egypt, but to comply with the demands of the authorities.
Moreover, our other priority is the safety of our employees and any actions we take in Egypt will be judged in light of their continuing wellbeing.

There has been no change in the Internet services situation. As of now Egypt has been cut-off the Internet for 36 hours.  In Egypt the work weeks starts tomorrow, so let’s see if the government will bring the services back online perhaps tonight.

Update January 29, 21:00 UTC

I’ve published a list of the Egyptian routes/prefixes that are still being routed. This list of  243 networks can be found here: http://bgpmon.net/egypt-routes-jan29-2011.txt.
This data is extracted from routeviews data (rib.20110129.1800). Note that routed does not necessarily mean that it’s reachable. Format of the list is:
|AS number| Prefix| Prefix description|.
A few examples of routes that are still announced:

• AT-Financial Holding routes
• Bibliothique of Alexandria  (http://www.bibalex.org/)
• Central Bank of Egypt Routes
• Egyptian National Scientific & Technical Information Network routes

Update January 31, 23:30 UTC

As of today thirteen more Autonomous systems (ISP’s) have disappeared. This includes the up to now unaffected Noor Data networks. The networks for Noor data network seemed to have disappeared at 2011-01-31 20:54 UTC.
As these ISP’s disappeared so did more routes. Using a routeviews dump of 22:00 January 31^st,  we now only see 12 origin Autonomous systems and a 134 routes. This means that now only 5% of the Egyptian routes remain reachable.
A list of all remaining routes can be found here:  http://www.bgpmon.net/egypt-routes-jan31-2011.txt

number of advertised egyptian routes

Securing BGP has been on the todo list of the IETF and the community at large for many years. Over the years we’ve seen several proposals, the Resource Public Key Infrastructure (RPKI) is the latest and most successful initiative. RPKI solves one of the most fundemental problems. It allows us to verify whether an Autonomous System (AS) is authorized to announce a specific prefix. This January AfriNIC, LACNIC and RIPE launched their RPKI services, APNIC already started offering this service last year. ARIN should start to offer this service somewhere this year.

As this will become an important technology in securing inter-domain routing, I’ve decided to try to get some hands-on experience with this. In this article I’ll describe my understanding of the current state of the RPKI and introduce the BGPmon ROA validation service.

RPKI building blocks
The main building blocks in the RPKI infrastructure are trust-anchors, ROA’s and validators. Trust-anchors used today are the RIR’s (RIPE, APNIC, LACNIC, AFRINIC). These are the organization that hand out the network resources and therefor are our point of trust. Route Origin Authorization or ROA’s are documents used to link a set of prefixes with an origin ASN. These ROA’s are created by ISP’s, or more accurately the maintainers/administrators of an Autonomous system or Network. The result is signed by the resource holders private key and this signing than creates a chain of trust which allows us to authorize (validate) BGP announcements. Below you’ll find an example ROA, note that a single ROA can contain multiple prefixes and has only one origin AS.

Today, in order to verify BGP announcements, we can only rely on the numerous IRR databases and hope that this information is valid. The RPKI system is not intended to replace the current IRR systems, instead it complements this system.
As you probably know many IRR databases allow you to add any route object to the database. For example you could add a route object for 8.8.8.0/24 with your own origin AS. If your upstream uses the IRR to automatically create prefix-filters, you would now be allowed to announce 8.8.8.0/24. With this new RPKI infrastructure providers, or scripts, can do an additional validations step, to check if the AS in question is authorized to announce this network.

BGP router validation
Eventually the goal is to have BGP capable routers validate prefixes against ROA’s. Based on the result one might change the local-preference of this route add a tag, or whatever you think is best. The key thing is that you’ll have the ability to distinguish authorized announcement from non-authorized routes. So if you have a choice between multiple paths, one with a authorized origin AS and non with a non authorized origin AS you can prefer the path with the authorized origin AS.
Routers won’t do the actual certificate validation, this would take to much resources. Instead they’ll talk to a remote validator that will answer the router’s question:
I just received an announcement for 8.8.8.0/24 with origin AS 15169, is this an authorized announcement?
The remote server will answer this with one of the answers below (as specified in the RFC’s / IETF Drafts):
Code 0: Roa found, validation succeeded
Code 1: No Roa found (resource is not yet signed)
Code 2: Roa found, but validation failed. For example because of an incorrect origin AS or prefixlength.

Validating ROA’s, A proof of concept
In order to gain some experience with RPKI and ROA’s, I’ve decided to develop a ROA validator.
The current implementation relies heavily on the validator written by ripe. This validator fetches all ROA’s from the 4 trust anchors and stores the results locally. As a first implementation I’ve extended the current BGPmon whois service with ROA support. For each query the found prefixes will be checked for a ROA. The output of this whois query will now show and additional line with the results of the ROA validation. See example below:

$ whois -h whois.bgpmon.net 200.7.86.0

Prefix:              200.7.86.0/23
Prefix description:  LACNIC
Country code:        UY
Origin AS:           28001
Origin AS Name:      LACNIC - Latin American and Caribbean IP address
RPKI status:         ROA validation successful

If you would like to check just the ROA or receive more information about a ROA, you can use the -–roa feature. This will return the validation code as specified in the RFC’s / IETF Drafts.
In the case of a valid validation it will also print the details of the ROA. In the case of failed validation (2) it will show why it failed. For example it could be an incorrect prefix length or perhaps an invalid origin AS.
Below an example of how to use the bgpmon whois service to validate ROA’s.

$ whois -h whois.bgpmon.net " --roa 28001 200.7.86.0/23"
0 - Valid
------------------------
ROA Details
------------------------
Origin ASN:       AS28001
Not valid Before: 2011-01-07 02:00:00
Not valid After:  2012-08-05 03:00:00
Trust Anchor:     repository.lacnic.net
Prefixes:         200.7.86.0/23 (max length /24)
                  2001:13c7:7012::/47 (max length /47)
                  200.3.12.0/22 (max length /24)
                  2001:13c7:7002::/48 (max length /48)

I believe that this service will be useful for those who just want to play around with ROA’s and get some experience, but also for those who are be implementing a proof of concept ROA validation in routers or other real time validation software, perhaps even in the IRRToolSet

The next step is to add ROA functionality to the BGPmon.net monitoring service. This will allow BGPmon.net users to monitor in real-time if the ROA for your prefix is still valid and will alert you in case we detect any announcements that causes the ROA validation process to fail.

So are we safe now?
ROA’s will help us make the Internet routing infrastructure more secure. By having the ability to check if the origin AS is really authorized to announce this prefix, we should be able to prevent most of the accidental hijacks. However it’s also important to realize that it’s only a first step. For example it’s still possible to bypass this form of security by spoofing (prepending) the origin AS. This works because complete path attestation against the AS_PATH is not (yet?) possible.

Last but not least, this technology and the available software is still relatively new, so please feel free to play and test with it. Let me know if you find any bugs or if you have any suggestions for improvement.

This is just a quick post to address some of the emails I’ve received today. Quite a bit of BGPmon.net users have received a notification regarding a possible hijack of their address space.

On Friday January 14th AS4761, INDOSAT-INP-AP, started to originate a large number of new prefixes. A quick check show that AS4761 originated approximately 2800 new unique prefixes of 824 unique Autonomous systems. Whereas normally they originate approximately 100 prefixes.
The announcements happened between 12:19 and 12:57 PM UTC. Some prefixes were affected longer than others,

The geographic impact of these announcements varies per prefix. Some were seen by only a few peers, where others were seen by up to 50 peers geographically dispersed all over the world. Some of the networks affected are 8.8.8.0/24 (Google open resolver), a number of AS20940 Akamai prefixes, Amazon prefixes, Cisco, DoD, US Senate, American Express, General Electric and many others.

Wondering if your network was affected by this? Here you’ll find a list of all affected networks.

A number of the transit providers of AS4761 accepted these prefixes. This is the distribution:

Two weeks we published this article, in which we looked at the status of IPv6 deployment worldwide. We saw that by looking at the number of networks (Autonomous Systems) that announce both IPv4 and IPv6 prefixes, the global IPv6 deployment rate is around 8%.
In this article we’ll take a close look at IPv6 deployment in Canada. From the previous article we know that Canada has an IPv6 deployment rate of 8%, which is the same as the global average. It is however significantly lower than for example New Zealand, Japan and many European countries.

Residential IPv6 services in Canada

Today TekSavy, an independent Canadian service provider, is the only ISP in Canada offering experimental IPv6 services to customers. This is pretty much a global trend, as we see most IPv6 deployments and service offerings are mainly offered by large carriers, down to regional ISP’s, business providers and lastly residential users.

The reason for this is easy, high costs.  Customer premise equipment (CPE), i.e. cable/ DSL modems provided by residential providers, are typically low end devices. They do not support newer technologies such as IPv6 or even the use DNSSEC.  As a result, in order to provide IPv6 services to residential users all of the routers/modems at the customer have to be replaced. This of course is a significant investment for the providers.  In the core of these provider networks, more high-end and typically more modern equipment is used, the majority of these routers have supported IPv6 for quite a while.

So if IPv6 development mainly happens in the core, let’s take a look at the Canadian IP Transit market and how IPv6 has change the market relationships. We’ll start by taking a look at the IPv4 market followed by more detailed analyses of the IPv6 market.

Top 10 IPv4 providers
Let’s first take a look at the view of the Canadian transit market with our IPv4 glasses on. For those familiar with the Canadian market, the list below contains few surprises. The list consists of major Canadian Telco’s and Cable providers as well as most of the global Tier1 providers.

According to the table below Cogent the leads the pack, by providing transit to 159 Canadian networks (autonomous systems).  However we should probably take into account that AS577 (Bell Canada) and AS6539  (GT-BELL) are both owned by Bell.  After GT, Group Telecom, was bought by Bell both networks operated as separate networks, but they’re all Bell customers. Having said that, the real market leader is bell with 197 transit customers.

From the list below the only Transit provider that’s not a Telco / Cable or Tier1 provider is Peer1.  Peer1 is a hosting / transit provider based out of Vancouver and provides transit to 49 Canadian Networks.

The table below shows a list of providers that provide IPv4 Transit to Canadian IPv4 networks.

Top IPv6 providers

Now let’s take a look at the Canadian transit market with our IPv6 glasses on. The situation now looks quite different.  The first thing to notice is that non of the Canadian Telco’s or Cable providers are represented in this list. With the exception of Peer1 and Canarie this list has no Canadian providers at all.

Hurricane Electric is the market leader in the Canadian IPv6 Market. No real surprise, as they are the global leader.  Globally, Hurricane Electric provides transit to three times as many networks as the runner up Global Crossing. Other major players are AS3257 Tinet (formerly known as Tiscali), Tata, Global Crossing, Level3 and NTT America.

The only Canadian Transit networks on this list are Peer1 and AS6509, Canarie. Canarie is the Canadian Research and Education (R&E) network, comparable to for example Internet2 in the US and Geant/Dante in Europe. Although TATA communication has some Canadian roots (it used to be TeleGlobe) It’s fair to say that the Canadian IPv6 transit market is dominated by non Canadian companies, specifically large global carriers.

In fact most of the Canadian transit providers that are market leaders in IPv4 do not provide transit to any IPv6 network (Telus, Bell, Allstream).  The exceptions are Peer1 (providing transit to 271/BCNET, 11290/ Cogeco Cable and 36483/ gossamer Threads) and Shaw that provides transit to AS271/BCNET.

The table below shows a list of providers that provide IPv6 Transit to Canadian IPv6 networks.

Research and Education
Research and Education (R&E) networks globally have been early adaptors and in Canada Canarie is no exception as they have been running IPv6 for many years. If we zoom in a little more into the Canadian R&E community we see that in the world of IPv4, fourteen Canadian networks appear behind AS6509 (Canarie).  In the world of IPv6 that’s seven networks. This would make for a score of 50% in our IPv6 deployment scale.  Significantly better than the average Canadian Ipv6 deployment score of 8%.

Please keep in mind that these results are based on BGP data, this means that if the AS is not visible in the data,  it’s considered as not IPv6 ready. If a network uses provider aggregated address space, the actual more specific might not be visible because of aggregation. In those cases the networks will be considered as not IPv6 ready.

Conclusion

The good news for Canada is that compared to the rest of the world, Canada’s IPv6 deployment ratio of 8% is on par with the global average. It’s however significantly lower than countries it normally likes to compare itself with.
Traditionally the IP Transit market in Canada was heavily dominated by Canadian Companies. However these companies have missed the boat in the new world of IPv6. Most of the IPv6 ready Canadian networks are now forced to by transit from the larger global carriers. As more and more transit RFP’s have IPv6 as a mandatory requirement, this could very well result in loss of IPv4 customers as well.

We are nearing the end of 2010 and while we’re all sitting around the Christmas tree something else is nearing its end. We’re just a few months away from IPv4 exhaustion; the end of the IPv4 lifetime is insight. This will have many organizations rush to implement IPv6 in their networks.  So how far are we with IPv6 deployment? Time to take a look at the global IPv6 deployment rate.  In this article I will publish the deployment rate per country. I’m planning to zoom into some countries in the next blog posts.

Earlier reports
I’ve published similar reports earlier, one in October 2009, and one in April 2010. In 2009 we saw that the global deployment rate was around 4,4%, by April 2010 this had grown to 5%. We can see similar growth by just looking at the growth of the IPv6 routing table.

In the graph below we see the number of IPv6 prefixes in the global routing table over time. It’s clearly visible that over the last year the growth has increased quite a bit as compared to earlier years.

IPv6 awareness
Over the last year the push for IPv6 has become more apparent.  I believe that within the global IT industry it has gained more awareness. Likely because of the attention from industry media and mailing lists. Every now and then IPv6 related stories even make it into the mainstream media.

In addition, a number of countries have started IPv6 working groups and some have been brave enough to set IPv6 mandates.  The US government for example set mandates quite a while ago, while India has recently set some mandatory IPv6 goals.
How successful these mandates are remains to be seen, but if anything it increases awareness and a push to implement IPv6.

IPv6 deployment ratio.
Each network in the global Internet has a unique Autonomous System (AS) number. An Autonomous System can be an Internet Service Provider (ISP), Enterprise network, content provider or any other sort of network. Each AS number announces one or more prefixes.  By using ‘Geo IP’ libraries we are able to determine a country for each prefix. This in turn allows us to determine the unique number of networks (AS numbers) per country.   Doing this for both IPv4 as well as IPv6 will result in the IPv4/IPv6 deployment ratio.

Assuming that at some point the majority of the networks will be dualstack, the eventual results should be around 80 to 90%. We’re not there quite yet.

Global IPv6 deployment December 2010

IPv6 Deployment worldwide

The global IPv6 deployment rate increased from 5% in April to 7.95 % December 2010.
If we look at the results for individual countries, we see that Greenland is the winner, with a deployment rate of a 100%. According to our statistics there’s exactly one ASn active in Greenland (Tele Greenland) and it’s announcing both an IPv4 as well as IPv6 prefix.  Second place is shared by Cuba and Fiji. Both with a deployment rate of 75%. In both cases 3 out of 4 networks (ASns) doing business in Cuba and Fiji announce both an IPv4 as well as an IPv6 prefix.

If we look further in this list we can see that the top 10 is taken by relatively small (measured in Internet availability) countries with a low diversity of ASn’s, making it relatively easy to score high.

So I decided to zoom to the countries that have at least one hundred autonomous systems announcing IPv4 address space. The table below shows the results.

IPv6 deployment - 100+ Networks

Clearly visible is that European countries together with Japan, New Zealand and Taiwan are populating the top is this list.  In North America we see Canada leading the pack, with 8% pretty much equal to the global IPv6 deployment rate. The US and Mexico both score 5%. Interested in how your country scored? The complete list of results per country can be found here.

Are we on the right track?

The short answer for this is no… At this point only 8 out of a hundred autonomous systems (8%) are announcing an IPv6 prefix. Which really is only the first step in successfully deploying IPv6. It’s only going to be a number of weeks before IANA will hand out the last to /8’s to the lucky RIR(s). After that the last 5 will automatically be distributed between all 5 RIRs. At that point the global pool of IPv4 addresses will be empty. After that we’ll likely have a year or so before the RIR pool dries up and then it’s really gone.  So we still have a long way to go.

I guess there is some good news for “IPv6 consultants” as there will be lots of demand for last minute IPv6 implementations.  There are also opportunities for Transit and Hosting providers, as in this world the availability of IPv6 is not always widely available, so it’s a great way of distinguishing your product from your competitors.

Who are leading the way into the IPv6 era
So can we say something about the types of networks that are deploying IPv6? Are these mainly Transit providers, small ISP’s or stub networks. Who are leading the way into the IPv6 era?

To answer this question I’ve categorized the Autonomous Systems into the following 6 categories:

• Stub networks are network do not provide transit to other Autonomous Systems.
• tiny transit ISP’s are networks that provide transit to less than 5 (IPv4) Autonomous Systems (<5)
• small transit ISP’s are networks that provide transit to less than 10 (IPv4) Autonomous Systems (>4 && <10)
• Medium transit ISP’s are networks that provide transit to less than fifty (IPv4) Autonomous Systems (>9 && <50)
• Large transit ISP’s are networks that provide transit to less then hundred Autonomous Systems (>49 && <100)
• Super Large transit ISP’s are networks that provide transit to 100 or more Autonomous Systems (>99)

According to our analysis it’s primarily the larger ISP’s that are originating IPv6 networks and hence are taking the lead in IPv6 deployments. The distribution is actually quite nice, as can be seen in the table below. It seems that the larger the network, the higher the chance that they’ve deployed IPv6.

Everyone who follows Internet security just a little bit will have seen an article this week talking about the Chinese BGP hijack in April of this year. I’ve seen articles on Fox, BBC, CBC, Slashdot and the nationaldefensemagazine.org. Apparently Wolf Blitzer talked about it on CNN and the BBC on one of its radio channels (starts at 1h:34min). All of these stories are in response to a report to the US congress from the ‘US China Economic and Security Review Commission’. The actual report can be found here , details about this specific incidents can be found on page 243. This report references BGPmon.net as the source of its data for this incident.

This event again shows how vulnerable the BGP routing infrastructure is. Some of the media have done a good job of describing what could happen in case an attack like this is intentional. Data could be stored, altered or just be thrown away. The Internet community has been working on securing the routing system for a while, but progress is slow. In the mean time all we can do is keep a good eye on our networks by monitoring carefully.

BGP hijacks happen every day, the majority of them don’t affect a large geographic region and only are noticed a small number of users.
Every now and then however we see an event that affects many users, either because of the geographic scale or simply because of the specific prefix that is affected. The latter happened this Sunday for 7 minutes, when the prefix 8.8.8.0/24 was ‘hijacked’.

8.8.8.0/24 is the prefix that serves one of Google’s Open DNS servers, which is available at 8.8.8.8.
A few hours ago 8.8.8.0/24 was announced by AS30890 (EVOLVA Evolva Telecom s.r.l.), a provider from Romania.

This ‘Hijack’ lasted for about 7 minutes, and was detected by 14 RIS peers in 4 unique countries. The majority of these networks learned this announcement through AS6939.

This is the second time in a month that Google is affected by a hijack. Last month on July 9th, AS42473 (ANEXIA) a provider from Austria announced a more specific of one of Google’s prefixes.
The prefix 74.125.127.0/24 was announced by AS42473. This is a more specific of 74.125.126.0/23, a prefix that hosts many of Google’s public services.
This announcement was later identified as a copy paste mistake, and quickly resolved after the engineers of AS42473 detected the mistake.

This is yet another example of how easy it is to ‘accidentally’ mess with the reachability of prefixes. There’s not a lot we can do about this today, except for strict filtering on the edges and monitoring using services such as BGPmon.net.
Luckily there’s some good progress being made on the Resource Certificate Public Key Infrastructure (RPKI) initiative.
Hopefully RPKI related tools will become available soon, so that it will be easy for operators to deploy this. And although this will not be a full proof mechanism for preventing BGP hijacks, it will prevent us from most of the ‘fat finger’ incidents we see on regular basis.

This Friday a number of BGPmon.net users have received an alert message informing them that their AS was announcing a new IPv6 prefix.
I too got an alert email and was surprised to when I saw the prefix that was detected, as the prefix detected was an ‘invalid’ IPv6 prefix.
This is the message I received:

====================================================================
New prefix for AS271 (Code: 60)
====================================================================
Detected new prefix:  f006:9000::/24
Update time:          2010-06-11 19:18 (UTC)
Detected by #peers:   4
Announced by:         AS271 (BCNET-AS - BCnet)
Upstream AS:          AS13768 (PEER1 - Peer 1 Network Inc.)
ASpath:               1280 174 3257 13768 271
Alert details:        http://bgpmon.net/alerts.php?details&alert_id=9019544
Add to my prefixes:   http://bgpmon.net/fp.php?aid=9019544

Looking at this message it seemed odd, although the prefix was very strange, the ASpaths seemed to make perfect sense. After some more digging I noticed that many other BGPmon.net users had also received an alert like this.

BGPmon.net keeps a list of bogon announcements, in this list you can seen many of the detected bogon announcements of yesterdat.
This list can be found here:
http://www.bgpmon.net/showbogons.php?inet=6

Looking at the large number of AS numbers, I found it hard to believe that all these ASn’s are actually announcing these prefixes. This would mean that about 100 networks at the same time decided to announce a bogon prefix, this is very unlikely so there must be something else.

Assuming that these prefixes are not originated by what seems to be the origin AS (based on ASpath), this would mean that the announcements are originated by another AS, which seems to spoof (AS prepends) the ASpath with these AS numbers.

More details
From what I can quickly see these ‘strange’ announcements are seen with at least about 100 different origin ASn’s.

Initially I though this was an issue with the BGPmon.net software, but after reviewing the data at the RIPE RIS website I see the same results.

These are some of examples of observed bogon prefixes:
0:1f00::/24
2800::/8
4000::/8
4800::/8
5810::/12
And many more

The announcements are detected by the following RIS peers at the AMS-IX – Amsterdam, Netherlands, MIX – Milan Internet Exchange and the PAIX – Palo Alto, United States.

Who announced this?
As the administrator of one these ASns I don’t believe these announcement really come from origin the AS as defined in the ASpath, i.e. the AS on the right hand side of the ASpath.
Looking more closely at all the ASpaths of all these bogon announcements, they all have 2 ASns in common,

174 3257 

Which are Cogent (174) & Tiscali (3257), so we should probably focus on those two.

Spoofed ASpath

All of these RIS peers above have a IPv6 relationship with Tiscali AS3257. It’s fair to assume that they also have an IPv6 peering with AS174 (Cogent) as that’s how they learned these announcements.

Because the RIS peers that detected this have a peering with both Cogent as well as Tiscali, it’s surprising that non of them reported a shorter path directly via AS3257. Instead the paths went through AS174 and then to AS3257.

Another observation is that AS3257 is a RIS peer, and as a result one of the peers that BGPmon.net uses to analyze data. However non of the bogon updates we’re detected by AS3257 (or more specifically, non were sent to the RIS collecter from AS3257).

Assuming that AS3257 never saw these updates, that would indicate that that is part of the spoofed AS path and makes 174 the source of these announcements.

Another possibly useful clue is that updates contain two AS174 communities.
174:21100 which according the the whois data means: peer route learned in EU
174:22005 no explanation available

What does this mean
Please note that the above are assumptions, as I have not had contact with either Tiscali or Cogent I can only report on the observations described earlier.
I have no idea what the purpose of these announcements were. In the past we’ve see ‘spoofed’ AS paths as part of a research project. But they have also been used in BGP man in the middle attacks.
Maybe one of you have an idea?

This morning many BGPmon.net users received an alert regarding a possible prefix hijack by a Chinese network. AS23724 is one of the Data Centers operated by China Telecom, China’s largest ISP. Normally AS23724 CHINANET-IDC-BJ-AP IDC, China Telecommunications Corporation only originates about 40 prefixes, however today for about 15 minutes they originated about ~37,000 unique prefixes that are not assigned to them. This is what we typically call a prefix hijack.
This incident follows another concerning incident from China 2 weeks ago.

Although it seems they have leaked a whole table, only about 10% of these prefixes propagated outside of the Chinese network. These include prefixes for popular websites such as dell.com, cnn.com, www.amazon.de, www.rapidshare.com and www.geocities.jp.
A large number of networks impacted this morning were actually Chinese networks. These include some popular Chinese website such as
www.joy.cn , www.pconline.com.cn , www.huanqiu.com, www.tianya.cn and www.chinaz.com
A list of all prefixes that were announced/hijacked can be found here

The event has been detected globally by peers in The Netherland, UK, Rusia, Italy, Sweded USA, Japan and Brazil. However not all individual prefix ‘hijacks’ were detected globally, many only by a few peers, in one or 2 countries, but some by more.

Some details
All announcement had part of the AS path in common. The common part in the ASpath is (note the prepend).
4134 23724 23724

Which are:
AS4134 CHINANET-BACKBONE No.31,Jin-rong Street
AS23724 CHINANET-IDC-BJ-AP IDC, China Telecommunications Corporation

ASns peering with AS4134 seem to have picked this up and propagated that to their customers.
Some of these ASns include:
AS9002 RETN-AS ReTN.net Autonomous System
AS12956 TELEFONICA Telefonica Backbone Autonomous System
AS209 ASN-QWEST – Qwest Communications Company, LLC
AS3320 DTAG Deutsche Telekom AG
AS3356 LEVEL3 Level 3 Communications
AS7018 ATT-INTERNET4 – AT&T WorldNet Services

All RIS peers that detected this where behind (transit/peer) one of those ANS’s.

AS2914 NTT-COMMUNICATIONS-2914 – NTT America, Inc. customers
Looking at more routing information it seems that AS2914 saw more then just the 10% mentioned above. So the impact for NTT America customers might have been bigger.

Impact
28% of the RIS collectors used by BGPmon.net have detected these events. This means that quite a number of networks were impacted by this. The first announcement was detected at 2010-04-08 17:54:31 (UTC), the last ‘hijack’ announcement was at 2010-04-08 18:10:14.
Most ‘alerts’ have now been cleared, they typically lasted a few minutes.

Probably more then the 51 peers mention above would have detected the prefix, but not have chosen this as the best path. Most likely due to the ASpath length or other policies. I believe it’s fair to assume that the impact in China and probably Asia was far bigger then the rest of the world.

Possible Cause
I have not spoken with engineers from AS23724, so I can only speculate. Given the large number of prefixes and short interval I don’t believe this is an intentional hijack.
Most likely it’s because of configuration issue, i.e. fat fingers. But again, this is just speculation.

Prefix distribution
Most prefixes impacted by this were prefixes from the US and China. Below you’ll find the top countries impacted:

Country => number of prefixes hijacked by AS23724
US => 10547
CN => 10298
KR => 2857
AU => 1650
MX => 885
IN => 719
JP => 604
BR => 592
FR => 508
RU => 471
CA => 425
TH => 372
ID => 369
IT => 338
CO => 328
GB => 322
CL => 302
SE => 281
HK => 276
EC => 272
DE => 227

Example alert message

====================================================================
Possible Prefix Hijack (Code: 10)
====================================================================
Your prefix: 203.190.56.0/21:
Prefix Description: www.infoseek.co.jp
Update time: 2010-04-08 16:09 (UTC)
Detected by #peers: 4
Detected prefix: 203.190.56.0/21
Announced by: AS23724 (CHINANET-IDC-BJ-AP IDC, China Telecommunications Corporation)
Upstream AS: AS4134 (CHINANET-BACKBONE No.31,Jin-rong Street)
ASpath: 8331 9002 9002 4134 23724 23724
Alert details: http://bgpmon.net/alerts.php?details&alert_id=6617721
Mark as false alert: http://bgpmon.net/fp.php?aid=6617721

This week it was announced that IANA has allocated 1.0.0.0/8 to APNIC. This prefix must look familiar to many as we see it often in examples and documentation. And let’s be honest haven’t you used 1.1.1.1 on one of your test routers to quickly test something?
Receiving a prefix from this range might result in some issues in regards to duplicate announcements and duplicate address usages.

Duplicate announcements
If multiple networks announce the same prefix it might result in traffic being routed to the wrong network. This problem becomes even worse if someone else starts to announce a more specific of this network. Normally these ‘hijacks’ are not all that common, but with prefixes from this range it might be a bigger issue due to the nature of this prefix.
To try to quantify this I decided to take a look in the BGPmon.net database in which we have a complete collection of bogon announcements since May 2009. Any announcement in the 1.0.0.0/8 range in the last 9 months is recorded in this database.

In this 9 month period we detected 364 unique announcements for in prefix in the 1.0.0.0/8 range. If we group those announcements by origin AS and announced prefix we see 23 unique announcements.

+----------------+----------+-----------------------------------------------------------------------------+
| prefix         | OriginAS | AS_name                                                                     |
+----------------+----------+-----------------------------------------------------------------------------+
| 1.0.0.0/9      | AS24785  | JOINTTRANSIT-AS Open Peering BV trading as Joint Transit                    |
| 1.1.0.0/16     | AS47377  | KPNBE T2 Belgium NV                                                         |
| 1.1.0.0/24     | AS3549   | GBLX Global Crossing Ltd.                                                   |
| 1.1.1.0/24     | AS8300   | Test-AS --  Swisscom Ltd                                                    |
| 1.1.1.0/24     | AS30733  | GLOBUS-AS GLOBUS-TELECOM Autonomous System                                  |
| 1.1.1.0/24     | AS6503   | Axtel, S.A.B. de C. V.                                                      |
| 1.1.1.0/24     | AS34695  | E4A-AS E4A Primary AS                                                       |
| 1.1.1.0/24     | AS8218   | NEO-ASN AS Confederation of Neotelecoms, euNetworks AG and Upstreamnet gmbh |
| 1.1.1.0/24     | AS3549   | GBLX Global Crossing Ltd.                                                   |
| 1.1.1.0/24     | AS45899  | VNPT-AS-VN VNPT Corp                                                        |
| 1.1.1.0/24     | AS16735  | Companhia de Telecomunicacoes do Brasil Central                             |
| 1.1.1.0/30     | AS38091  | HELLONET-AS-KR CJ-CABLENET                                                  |
| 1.1.1.0/31     | AS8359   | COMSTAR COMSTAR-Direct global network                                       |
| 1.1.1.1/32     | AS45400  | NICNET Korea Telecom-PUBNET                                                 |
| 1.1.1.10/31    | AS8359   | COMSTAR COMSTAR-Direct global network                                       |
| 1.1.2.0/30     | AS3313   | INET-AS I.NET S.p.A.                                                        |
| 1.1.88.0/24    | AS4645   | ASN-HKNET-AP HKNet Co. Ltd                                                  |
| 1.1.88.0/24    | AS39386  | STC-IGW-AS Saudi Telecom Company                                            |
| 1.120.0.0/13   | AS23148  | TERREMARK Terremark                                                         |
| 1.2.3.0/24     | AS19151  | WVFIBER-1 - WV FIBER                                                        |
| 1.20.23.178/32 | AS26592  | Dominio BR Consultoria em Informatica Ltda                                  |
| 1.40.0.0/13    | AS23148  | TERREMARK Terremark                                                         |
| 1.80.0.0/13    | AS23148  | TERREMARK Terremark                                                         |
+----------------+----------+-----------------------------------------------------------------------------+

A complete list of bogon announcements can be found here:
As you can see the 1.1.1.0/24 prefix is the most popular prefix, so we can only hope APNIC won’t allocate this prefix. Except maybe for a nice honeynet project.

Duplicate address usage
Duplicate announcements are not the only thing networks in the 1.0.0.0/8 prefix have to worry about. As it turns out a number of organizations have used this prefix as an alternative for the RFC1918 prefixes. With the reasoning that many people already use 192.168.0.0, 10.0.0.0 or 172.16.0.0 , so chances of collisions are reasonable. So these bright minds came up with the idea of using a unallocated prefix as an alternative, such as for example 1.0.0.0/8

AnoNet
AnoNet is a private friend-to-friend network built using VPNs and software BGP routers. anoNet works by making it difficult to learn the identities of others on the network allowing them to anonymously host content and IPv4 services. Also see http://en.wikipedia.org/wiki/AnoNet
The prefix they use for this network is 1.0.0.0/8.
Apparently AnoNet is planning to do the same for their IPv6 initiative, as according to their website:
“Services are gradually being migrated to dual-stack. It is all in the de00::/8 range”
de00::/8 is a unallocated range, just as 1.0.0.0/8 used to be….

WIANA
Wiana is The Wireless Internet Assigned Numbers Authority, provides IP addresses for wireless devices from the 1.0.0.0/8 prefix.
Ironical WIANA claims to have been formed to meet the that need network policies are upheld.
According to their FAQ the reason for this prefix is that several protocols used already utilize the 10.x.x.x network for unregistered addresses during handshaking. Another class A network was required. Unfortunately for WIANA (and the future legitimate holder of this prefix) soon, the prefix they choose will no longer be unqiue.

Receiving a prefix from the 1/8 range
The role of the RIRS is to make sure prefixes are allocated to one organization only and as a result should be unique. With prefixes from the 1.0.0.0/8 prefixes this can no longer be guaranteed. Not because of multiple allocations by the RIR, but in this case by other organizations that thought it would a smart idea to choose a random unallocated prefix.
In order to prevent issue’s with BGP announcements, looking at the bogon announcements it’s probably a good idea to (at least not yet) allocate prefixes in the 1.1.0.0/16 range as these seem to be leaked the most.

As Alain Durand mentioned on Nanog: “Who said the water at the bottom of the barrel of IPv4 addresses will be very pure? We ARE running out and the global pain is increasing.

Just like many you, I use rancid to track changes in configurations of our routers and switches. This week BGPmon.net released a new feature called ‘routing report’, you can think of this as a Rancid for your BGP routing table. Every day BGPmon compares the announcements by your ASN with those of the day before. Any differences in announcements or reachability will be sent to you in an email.

I’ve been using this feature for a few weeks now and found it very useful.
The things it will report on is differences in announced prefixes, including a breakdown of which prefixes are reachable via which upstream / peer AS. This will help you detect if a prefix is no longer reachable via one or all peers or upstreams.
The routing report will also report on upstream and downstream ASns. At the end of this blog post you’ll find an example report. Please note that each report also contains a summary report of your AS, comparable to the cidr report.

[Read the rest of this entry...]

Lately I have received quite some questions with regard to connecting BGPmon.net with existing monitoring software. As well as requests for making more data available for developers.
I’m happy to announce that these things should now be possible trough the BGPmon.net Web Services API. This  API  allows you to access your BGPmon.net alert as well as other features from your own program running anywhere on the Internet.
I will use the rest of this Blog to briefly explain how the API works and what it can be used for. More  detailed information regarding the API and how to use this, can be found on the  Web Services API page.

Web Services
The BGPmon.net API  provide a number of Web services  for developers. It allows you to query our database and let’s you integrate this in your own software.
As we are using SOAP  as the web services protocol. All the web services are described in a WSDL file, which can be found here.
Currently the following functions are available:

1. getIPInfo()
2. GetAlerts()
3. GetAlertDetails()
4. GetASname()
5. GetASPrefixes()
6. GetCountry()

More information about these function can be found here

Integrating BGPmon.net with your software
SOAP is a programming language independent protocol and is supported by all popular languages.  It allows you to access the BGPmon API over HTTP.
Some of the functions are specific for your BGPmon.net account and thus allow you to authenticate, others are currently publicly accessible.

Let’s get started with two examples
On the BGPmon.net API page you will find two examples of SOAP clients that use the the BGPmon.net SOAP API, a perl and a php example.
The examples are for BGPmon.net users to use, as well as to learn from in case you want to write your own application using our API.

Perl Nagios plugin example
The nagios plugin is written in perl and allows you to monitor your prefixes in Nagios by utilizing the BGPmon.net API.
The script expects a number of arguments, these allow you to filter on the type of alerts that will be returned.
The source code for this plugin example can be found here.

PHP RSS feed example
This is an example of utilizing the BGPmon.net SOAP API in PHP. The PHP RSS example retrieves all the alarms and represents the results in a RSS feed.
You will need to edit the PHP script and change the email and password variable to your own BGPmon email and password.
You can then call the php script using your browser or any other RSS reader. Optionally you can change the days, maxalert and active aruguments,
An example using the demo account can be found here:
http://www.bgpmon.net/rssclient.php?days=100&active=0&maxcode=100
The source code can be found here

Developing your own programs
Should you decide that you want to write your own program and you have any questions please leave a comment on this blog.
Also if you have created your own program and are willing to share this, please sent it to me so I can add it the the examples page.

Keep in mind that the Web Services API is a new feature, if you find any bugs please let me know and I will try to solve them asap.
Happy programming!

Last week I finally received a new CPU for the BGPmon.net server. This new quad core 2.66GHz CPU replaces a 1.8GHz single core CPU. This upgrades follows the memory upgrade from a few weeks ago when memory was upgraded from 2GB to 8GB.
Over the course of this week I saw significant improvement in execution times and average system load. So with this faster CPU and more memory I’m pretty confident the software should be able to perform as you may expect.

Now the time has come to release some more new features. Some of these features have been ready for quite a while but I wanted to wait for the new hardware to be installed first. You can expect announcement for these exciting new features soon.

As you can see the the original hardware was very low-end, it was actually bought before BGPmon.net was born. It was never really intended for this purpose. However, with the new hardware I’m confident I can support the ever growing BGPmon.net user base and growing feature set.

Just a final reminder, this project is 100% funded by me. If you would like to support this project and help recover some of the costs, please consider making a donation here.

IPv6 slowly seems to become more mainstream, we hear about IPv6 more and more and it seems that at least some Service providers and governments understand that there is a sense of urgency.  Regularly we see the statements of networks that are planning to roll out IPv6 and vendors that are promising to make their products IPv6 ready.

But talk is cheap and the question remains, how far are we actually with rolling out IPv6 deployment?  We tried to answer that question by looking at the Internet Routing tables.

IPv6 deployment ratio.
Each network in the global Internet has a unique Autonomous System (AS) number. An Autonomous System can be an Internet Service Provider (ISP), Enterprise network, content provider or any other sort of network. Each AS number announces one or more prefixes.  By using Geo IP libraries we are able to determine a country for each prefix. This in turn allows us to determine the unique number of networks (AS numbers) per country.   Doing this for both IPv4 as well as IPv6 will result in the IPv4/IPv6 deployment ratio.

Let’s look at for example at Canada. There are 816 Autonomous Systems that originate a prefix registered as in use in Canada.  If we look at the IPv6 routing tables we see that 50 Autonomous Systems announce a Canadian IPv6 prefix.  This results in an IPv6 deployment percentage of 6.1%. Meaning that 6.1% of the networks doing business in Canada are currently actively deploying IPv6.

Results
If we look at the global statistics, i.e. comparing all IPv4 Autonomous Systems with all IPv6 Autonomous Systems we see that the global IPv6/IPv4 deployment ratio is 5.26%. This is slightly higher than the 4.4% we measured in April 2009.

And the winner is
Jersey , a small country between England and France, is the only country scoring a 100% deployment ration.  IPv4 and IPv6 prefixes registered to Jersey are only announced by one provider, AS8681 Jersey Telecom; resulting in a 100% ratio.

Jersey is followed by Cuba (75%), Oman, Monaco, Holy See (Vatican City State) and Fiji all scoring 50%. If we look at the bigger countries, i.e countries with at least a 100 (IPv4) networks we see that Czech Republic (19%), New Zealand(18%), Japan (17%) and The Netherlands (17%) are leading.

Are we on the right track?
Ideally the IPv6 deployment percentage should be around ~100%. Globally today we score a 5% ratio. Although this is one percent higher than half a year ago it’s still very low. Never the less, it’s positive to see that some individual countries such as Tunisia and Uruguay score surprisingly high. And also Europe and parts of Asia seem to be on the right track.

Top 10%
Below an overview of all countries scoring higher than 10%.  A complete list with the results for for all countries can be found here: IPv6 deployment statistics October 2009

Friday morning around 07:22:08 UTC AS9035 (Wind Telecomunicazioni) started to announce approximately 85.000 prefixes with an invalid origin AS. The origin AS was set to AS9035 while these prefixes did not belong to AS9035. The impact was local to a number of Italian providers, all Telecom Italia customers. The incident was resolved in about ~2 minutes after the first announcement.

Many of you have received alert messages for this event, informing you of the ‘possible hijack’.  I would like to take a bit of time explaining how to interpreted these message so it’s easy to determine the impact of such an event.
The first thing to look for is the number of peers that detected this prefix. In this case the event was detected by 2 peers, this gives you an indication that this event did not have a significant widespread impact.
The next thing to do is to login to BGPmon and check the details of this alert, a direct link to this the detailed info page is now included in the email messages.
Here you’ll quickly see again the number of peers that detected this as well as the geographic location of these peers. In this case both peers were located in Italy, indicating that it’s a fairly local event.  The global impact is also visible on the world map, making it easy to determine the geographical impact.

The same detailed info page also shows the BGP messages that are relevant for this alert. This will give you some more detailed information about the exact BGP announcements. In case the alarm is cleared you will see the exact time this happened. An alarm is cleared when the peer that detected this alert saw  a new valid update for this prefix or a withdrawal. It will also display the exact duration of the event per peer.

BGPmon notification time
BGPmon alert messages are normally sent out a few minutes (<5min average) after we received the updates from the RIPE RIS collectors.
Yesterday, some of you, received the alert messages later then usual.  I apologize for this and am currently working on a solution for this in order to prevent the delay in notification in cases of ‘mass hijacks/leaks’ like we saw yesterday.  A significant part of the solution is  to upgrade some of the hardware components of the  BGPmon.net server.   If you or your company would like to support this project, please consider making a donation. For more information please see this page.

Ok, ok… It happens all the time, people leaking private ASN’s. But this one caught my attention. It seems that the US Department of defense (DoD) is leaking a private ASN  65401 for their prefixes 140.21.18.0/23 and 140.21.15.0/24. These prefixes are normally announced by AS5802 but now appear to be originating from the private AS 65401.

The reason that this caught my attention is that although leaks like this happen all the time in general they appear to happen by the less clueful. The DoD I think should not be considered in this category.  However this ‘leak’  has been happening for a few weeks (starting 2009-08-27).  This leads me to believe that the DoD does not monitor their own announcements…. Maybe they should open a BGPmon.net account?

I’m happy to announce that this week a I released the new version of BGPmon.net into production. There are several new features that many of you have asked for. In addition there are some significant changes in the database backend. This is largely to improve the performance of the webinterface and some soon to be announced exciting new features. In this blog post I will describe some of the features that are new in this release. A small screencast describing these new features is also available here.

Improved overview of alerts gui

If you go to the ‘My Alerts’ overview in the BGPmon.net portal, you’ll notice that it looks somewhat different. These differences are mainly to improve the user friendliness and are based on feedback received from many of you the last view months. One of the features many of you asked for is the ability to select many alerts and delete those. The layout of this page has been changed a bit as well, it will now give you a bit more filtering options. Please try the mouse overs where ever you see the (i) information sign, you’ll see that many fields contain extra information. The new gui uses uses a different database table. ‘All’ recent alerts (maximal 1300) alerts have been migrated to this new table. All the alerts in the old database table are also still available using the old interface here.

Map the geographical  impact of an alert
When clicking on an alert you will see more details for this alert. You’ll see a list of all the peers that saw this alert including some of the BGP attributes. In addition it will render a short summary of unique number of peers that saw this alert including the unique number of countries in which those peers reside. These results are also rendered on a world map. This will give you a quick overview of the geographical impact of this alert.

False positive handling
There’s an updated version of the false positive handling interface. Now when marking an alert as a false positive it will allow you to delete all historic alerts related to the false positive. This will help you keep your alert list nice and clean.

Prefix description
If you received a BGPmon email alert in the last few days you might have noticed that there’s a an extra line called description for each alert. This is a description for the prefix you’re monitoring and will help you identify what the prefix is being used for. The prefix description is also shown in the my alerts page. By default BGPmon will use the description of a matching route object. You can change the description in the my prefixes page.

Must match feature
This feature is specifically build for users who would like to monitor small pieces of a large aggregate. The ‘must match’ value can be a prefix or ip address. Now before an alarm is raised an additional check is performed to verify of the announced prefix contains the ‘must match’ value. If this is not the case no alarm is raised. Let’s look at an example of when and how you would use this. For example I have some servers at my hosting provider in the by them to me allocated address space 10.10.0.0/24. This prefix is not visible in the global routing table. In stead my hosting provider, AS10, announces the prefix is 10.0.0.0/8. So I added AS10 & 10.0.0.0/8 to BGPmon and soon I found out that I receive a number of alerts for example 10.200.200.0/24 was announced by AS666, this raised an alert. As this is probably for a different customer and not relevant for me I am not really interested in this. I am however interested in alerts for 10.0.0.0/8 or any more specifics that match my assigned prefix. In this case I add 10.10.0.0/24 as the ‘must match’ value. Now I will only receive alerts that are relevant for me.

Improved cleared alarm detection
Each announcement (or withdrawal) can either raise or clear an alert. All valid announcements or a new invalid announcement can clear any active alert. As the popularity of BGPmon increased and as a result the number of users and open alerts, the clearing algorithm proofed to be inefficient and required a lot of the computing resources. The clearing functionality in this new version has been rewritten and is many times faster and no longer a burden for the systems resources.

Deprecated alarm codes
I decided to deprecate both code 12 and code 23 alarm codes, as I believe that these are confusing and are all ready covered by different alarm codes. The following alerts codes have been deprecated: Code 12, meant a more specific AND different upstream AS. These events will still be detected however from now on just as a code 22, more specific. Code 23 meant withdraw of more specific. This alarm generated quite some alarms. Now that the alarm clearing algorithm is improved there’s no use to generate a separate alert for this. Withdraws for more specifics are still recorded as they are used to mark the previously announced more specific as cleared.

The last few weeks I have been working on implementing some more notification options. A number of users have asked for sending out notification email to more than just one email address, as well as being able to send notifications to pagers.
The latest release of BGPmon.net supports these features.  You can find the configuration for these new options in your setting page. In this post I will highlight a few of the new  features.

Additional email address
After a number of requests this feature is now available.  The additional email address will be cc’d  on all your notification emails.  This allows you to share these alarms with your colleague or noc.

Pager / SMS notification
This feature is also based on input I received from a number of users who would like to have BGPmon.net notifications sent to their pager or cell phone.  Some of you have email to SMS gateways that takes an email as input and use the content of that email to sent a text message to your pager or cell phone. By adding a pager/sms email address a short messages of maximal 160 characters will be sent to this email address.  This allows you to receive alerts on you pager or cell phone as well.

Conservative notification
For those of you who monitor many prefixes and have dynamic networks (i.e. many new prefixes, upstreams) will receive quite a number of alerts. BGPmon.net used to send out notifications for each suspicious update it detected. If you your filters were not accurate this could result in several hundreds of emails a week or day.
In cases like this less is more. A new feature called ‘conservative mode’ will suppress recurring alarms and only notify you 3 times a day for each unique event.  This is only for recurring alarms and not for new alerts or new prefix & origin combinations.
For example, if you have a new upstream and you did not add this to your upstream list, BGPmon.net will sent out notifications for that each time it sees an update for your prefix containing this upstream. After it sent out 3 notifications for this prefix/upstream event it will ignore this for 24 hours.
Another example: ASx has hijacked your prefix. They are announcing a more specific for one of your prefixes. Conservative notifications do not apply for this kind of alert, as it’s not a recurring alert. In fact it’s a complete new prefix / Origin AS combination.

The conservative notification feature will make BGPmon.net less ‘chatty’. Conservative ignore is enabled by default, you can disable it by setting your notification mode to active. In this case it will notify you for each event over and over. It’s not recommended to run in active mode and in normal circumstances you should not need to do this.

I want to stress that this feature is not needed when you keep your filters up to date. Whenever you see a false alert please click the false positive link in the email so we make sure we don’t sent you notifications for this event again.

New Prefix detection
The last new feature implemented in the latest release is new prefix detection.  When BGPmon.net detects a new prefix for one of your ASns it will sent you a notification (code 60).
This will help you to verify if your new prefix is seen in the global routing table. Or in the case of accidental leaks you’ll be quickly notified.
You will only receive one notification email per prefix & origin AS combination. You can disable this feature in the setting page, in case you do not want BGPmon.net to monitor for new prefixes.

Bug fixes
The new version also contains a few minor bug fixes, regarding IPv6 prefix syntax checking, ignoring more specifics as well as some performance related improvements.

Community feedback
BGPmon.net relies heavily on community feedback. All of the above features are based on input received from BGPmon.net users. If you have a feature request, idea for improvement or found a bug, please let me know.  Together we can continue to improve this project!

This morning there was a discussion about a possible prefix hijack by AS13214 on the Nanog list. Cyclops users received a notification email notifying them that AS13214 was announcing their prefix.  I just went trough some of the raw data and this is what I found.
It seems it was picked up by the route-views4 collector only. Non of the RIS peers seem to have seen this.  This is also the reason why BGPmon.net users did not get notified, as BGPmon.net uses the RIS resources for BGP updates.

Looking at the raw BGP data from routeviews4 it seems that:
AS13214 leaked a full table (~266294 prefixes) with 13214  as OriginAS to AS48285 which is a routeviews4 peer. Routeviews4 saw these announcements as:
ASpath 48285 13214.

It seems to  have happened twice:
~ 11:03:45 GMT to 12:16:31 GMT (here AS48285 start announcing a valid path to routeviews again).
then a few seconds later again:
~ 12:16:36 GMT to 12:18:14 GMT
After that AS48285 announced ‘normal’ ASpath to routeviews again.

So looks like it wasn’t a global hijack, it was only seen by one routeview peer.  This is a very similar event as the one we saw in November 2008:  Prefix hijack by AS16735 or (Brazil Leak: If a tree falls in the rainforest).

This again shows that it’s hard to determine if an event is a ‘real’ hijack or not. Some will say it’s irrelevant some want to be notified in all cases. Based on received feedback regarding the November 11 event, BGPmon.net implemented peer thresholds.

Did you ever wonder what the current status of IPv6 deployment is and which country is taking the lead?  A number of sites provide information about IPv6 deployment; each one uses a different way of measuring the usage. From traffics measurements, to the reach ability of popular websites over IPv6, to the growth of the IPv6 routing table such as for example the bgp weathermap. I recently did a different analysis, by comparing the number of ASN’s per country that are announcing IPv6 prefixes as compared to IPv4 prefixes. This will show us how many ASns per country are deploying IPv6.

Methodology
Using Geo IP libraries we are able to determine a country for each prefix. For each country it was determined how many unique Autonomous Systems (ASns) are announcing an IPv4 prefix. The same was done for IPv6 prefixes. Based on these two numbers we came up with a percentage indicating the how many organizations (ASns) have started to deploy IPv6.
Let’s for example take a look at the global statistics, there are currently 31.844 unique  ASns  announcing at least 1 IPv4 prefix.  There are 1.393 unique ASns announcing at least one IPv6 prefix. Resulting in a global deployment percentage of 4.4%.
The same analysis can be done on a per country basis, by only looking at prefixes that are registered to a specific country.  Let’s for example take a look at Canada. There are currently 818 autonomous systems announcing at least 1 Canadian prefix. As compared to IPv6, there are currently 36 unique autonomous systems announcing a Canadian IPv6 prefix, resulting in a deployment percentage of 4.4%.

The results
The analysis as explained above was done for all countries. Ideally each AS currently announcing an IPv4 prefix should eventually also announce a IPv6 prefix. So in a few years from now the IPv6 deployment ratio should be around 100%.
Below you’ll find the countries that had a deployment score higher than 10%. Country’s with more then 10 IPv4 ASN’s are indicated in bold. Results for all countries can be found here: http://bgpmon.net/IPv6_deployment_statistics.txt

One of the first things to notice is that there are quite a few of the smaller (in Internet terms that is) countries that have a high score.  For example The Vatican, has one ASn doing business for them (AS8978), that same ASn is also advertising an IPv6 prefix, hence the restults is 100%.   For the larger countries it’s much harder to get a good score.

Mapping the score

Global IPv6 deployment

If we visualize the score on the world map, it becomes clear that some parts of the world have a much better score then others.  Especially Europe and the East Asia seem to score high.  Another interesting observation is that parts of Africa scored particularly high.

And the winner is…
As could be seen in the result overview, the country with the highest score is  the Vatican. If we look at the (from a networking perspective) bigger countries, with at least 10 ASNs, Uruguay in Latin America is the winner. There are currently 26 ASN’s originating an IPv4 prefix that is used in Uruguay. As compared to 9 ASN that have IPv6 prefixes, resulting in a score of 35%. When we look at both the relative number as well as the absolute number we see that countries such as Japan,  New Zealand, Czech Republic, and The Netherlands are the leaders.

One of the ‘losers’ is the leader in the IPv4 world, The Unites States. The US has the highest number of Origin ASns in the IPv4 world as well as in the IPv6 world. However the percentage (316./ 13280) of 2% is well below the global average.

Are we on the right track?
Considering that the ideal score would be around ~100%,  and the global score is 4%, we still have quite some work to do.  Never the less, it’s promising to see that some individual countries such as Egypt and Uruguay score surprisingly high. And also Europe and parts of Asia seem to be on the right track.

Since NANOG45  I was filled with inspiration for new features in BGPmon.net. Many of you have sent me your feedback and whenever possible I implemented the smaller feature request and bug fixes. Today the newest version went live; this version contains some of bigger changes and improvements.

In this Blog post I will go over some of the major changes. These changes include the features listed below:

1. Support for multiple Origin ASN’s.
2. Support for multiple upstream ASN’s.
3. Improved BGP Man in the Middle (BGP MITM) attack detection.
4. Ignoring certain prefixes
5. Ignoring certain ASpaths
6. Changed email layout
7. False positive handling

[Read the rest of this entry...]