Implementing Provider-Provisioned VPNs using Route Reflectors

MPLS/BGP Provider-Provisioned VPNs, such as those proposed in RFC 4364 (formerly RFC 2547) or draft-kompella variants, suffer from some scalability issues due to the fact that all PE routers are required to have a full iBGP mesh in order to exchange VPN-IPv4 NLRI and associated VPN label information.  In a modern network consisting of a large number of PE devices, it becomes readily apparent that this requirement can quickly become unmanageable.

The formula to compute the number of sessions for an iBGP full mesh is n * (n-1)/2.  10 PE devices would only require a total of 45 iBGP sessions (10 * (9)/2 = 45).  However, by simply adding 5 additional PEs into this environment your total number of sessions increases exponentially to 105.  Scalability issues arise because maintaining this number of iBGP sessions on each PE is an operational nightmare; similarly control plane resources are quickly exhausted.

An alternative to this that has gained widespread adoption is to utilize Route Reflectors to reflect the VPN-IPv4 NLRI and associated VPN label between PE devices.  However, several issues arise when using Route Reflectors in such an environment.  In a normal environment without the use of Route Reflectors, MPLS tunnels exist between each PE router such that when the VPN-IPv4 NLRI and associated VPN label are received, a PE router can recurse through its routing table to find the underlying MPLS tunnel used to reach the remote BGP next-hop within the VPN-IPv4 NLRI.  In the Route Reflection model, the Route Reflector typically doesn’t have an MPLS tunnel to each PE for which it is receiving VPN-IPv4 NLRI.  Therefore, these routes never become active and are therefore not candidates for reflection back to other client and non-client peers.

A few methods have been developed which circumvent this issue.  One method is to simply define MPLS tunnels from the Route Reflector to each PE.  This solves the problem by allowing the Route Reflector to find a recursive match (i.e. MPLS tunnel) in order to reach the remote PE.  However, this approach suffers from the drawback in that it requires a whole bunch of MPLS tunnels to be configured which only serve to allow the received VPN-IPv4 NLRI to be considered active.  Remember, these tunnels are completely useless in that they will never be used for the actual forwarding of data, they are only used within the control plane to instantiate routes.

An alternative and much more graceful solution to this problem is to configure the Route Reflector with a static discard route within the routing table which is used to reference BGP next-hops in MPLS environments (inet.3 for example in JUNOS).  This static discard route only serves to function as a recursive match when incoming VPN-IPv4 NLRI are received for the express purpose of making these routes active and therefore candidates for redistribution.  In JUNOS, one can accomplish this using the following configuration:

With the above, any VPN-IPv4 NLRI received from a PE router is immediately made active due to the fact that a static route has been created in inet.3 which is the routing table used in JUNOS to recurse for BGP next-hops in MPLS environments.

An excellent whitepaper entitled “BGP Route Reflection in Layer 3 VPN Networks” expands upon this and describes the benefits of using Route Reflection in such environments. It also builds the case for using a distributed Route Reflection design to further enhance scalability and redundancy.

One thing to keep in mind is that with the Route Reflector approach, we have merely moved the problem set from that of the PE device to that of the Route Reflector.  Although it minimizes the number of iBGP sessions required on PE devices, the Route Reflector must be capable of supporting a large number of iBGP sessions and in addition, must be able to store all of the VPN-IPv4 NLRI for all of the VPNs for which it is servicing.  It is highly recommended that adequate amounts of memory are in place on the Route Reflector in order to store this large amount of routing information.

Finally, while using Route Reflectors is an acceptable solution in the interim to addressing scaling concerns with Provider-Provisioned VPNs, it is not clear if this approach is sufficient for the long term.  There are several other options being examined, with some of them outlined in a presentation entitled 2547 L3VPN Control Plane Scaling” given at APRICOT in Kyoto, Japan in 2005.

Book Review :: IS-IS: Deployment in IP Networks

IS-IS_DeploymentIS-IS: Deployment in IP Networks
by Russ White, Alvaro Retana
Hardcover: 320 pages
Publisher: Pearson Education
ISBN-13: 978-0201657722

2starsBetter off choosing an alternative selection

As IS-IS is one of the more esoteric protocols, understood only by a few people in large scale ISP environments, I thought this book would be a welcome addition to my library as there isn’t much else on the market covering this protocol. There are of course ISO 10589 and RFC 1195 which covers these protocols, but seeing as this is a short book I thought it might be able to shed some light on an otherwise complex protocol.

In reviewing this book I’ve come up disappointed in general. There are certainly a few golden nuggets and I give the book a couple of stars just for attempting to bridge the gap between the purely theoretical and the purely vendor specific. However, the book comes up short on most other points. Often times I found myself wanting to scrap this book in favor of some of the other selections on the market, but since I have respect for these authors I read the whole book hoping that they might be able to redeem themselves by the time I finished.

Obviously the authors have a great deal of knowledge about the subject, and I don’t fault them entirely. The quality of the editing is poor with many grammatical and syntactical errors littered throughout the text. There are abundant instances throughout the book where the diagrams used do not match the text describing them. I was rather disappointed because I usually find that Addison-Wesley publishes some of the best texts on the market.

All in all, I thought this book could have been a lot better than it was. After all, these authors have several other titles under their belt, most notably “Advanced IP Network Design”. But in this case, I would say that you are better off looking for other similar titles available on the market, such as Jeff Doyle’s “Routing TCP/IP Volume 1” or “The Complete IS-IS Routing Protocol” by Hannes Gredler and Walter Goralski.

Book Review :: MPLS-Enabled Applications: Emerging Developments and New Technologies

MPLS Enabled Applications

MPLS-Enabled Applications: Emerging Developments and New Technologies
by Ina Minei, Julian Lucek
Paperback: 526 pages
Publisher: Wiley
ISBN-13: 978-0470986448

5starsExcellent coverage of VPLS, and Multicast over Layer 3 VPNs

Recently I had to work on a project which involved demonstrating Multicast over Layer 3 VPN interoperability between Cisco and Juniper. I spent several days reading through all the RFCs and working-group drafts which pertained to this subject matter, after which I still had many unanswered questions. In order to round out my understanding, I decided to order the Second Edition of ‘MPLS-Enabled Applications’. Looking back, I wish I had read this book instead of wasting my time reading the various RFCs and working-group drafts. This book answered all of my questions and went above and beyond to give me a solid understanding of the concepts and their application. As other reviewers have pointed out, often one needs to read a book to understand the technology basics, and then refer to RFCs or working-group drafts in order to keep abreast of the latest changes. Not so with this book… In fact, this book is so current that reading the working-group drafts is largely unnecessary. It is incredibly comprehensive, concise, and gives the reader a thorough understanding of the business drivers. Furthermore, it illustrates the various ways in which MPLS services can be offered and outlines the pros and cons of each approach so that the network designer can make intelligent decisions with regards to implementation.

In addition to the great coverage that was provided by the First Edition, the Second Edition has updated the text to reflect newer trends and applications such as the transport of IPv6 over an IPv4 MPLS core, and detailed coverage of end-to-end and local protection schemes in MPLS networks. Likewise, the chapter previously called “Point-to-Multipoint LSPs” has now been renamed to “MPLS Multicast”, with much more detailed coverage of the P2MP hierarchy and the forwarding-plane and control-plane operation. The biggest value for me was the addition of a completely new chapter on “Multicast over Layer 3 VPNs” which provided comprehensive coverage of this emerging technology and fully illustrates the full gamut of operation of either the PIM/GRE approach, or the NG-VPN approach utilizing BGP and P2MP LSPs. Finally, the addition of a chapter on “MPLS in Access Networks” was well deserved seeing as Ethernet is quickly becoming the access technology of choice and MPLS will likely be utilized as an overlay in order to realize the full potential of Ethernet in these environments.

This book has earned a spot on my bookshelf as one of my most coveted resources, and I refer to it quite often to refresh my memory on the myriad workings of various functions within MPLS. I wish I could give this book a rating higher than five stars! I can’t overemphasize how exceptional this book is. If you are in the market for a book covering MPLS and emerging applications offered on MPLS networks, this single book should be at the top of your list!

The Case for Juniper Press

When was the last time you went into a bookstore and found a book on the bookshelf covering Juniper technologies?  I’ve managed to rarely, if ever, find more than one or two titles on JUNOS or any sort of Juniper related technology.  On the other hand, you’d be hard pressed to miss the bevy of Cisco related titles covering a vast array of technologies and platforms.  When I teach Juniper classes, or I’m on a consulting gig pitching Juniper technologies, one of the things I hear most often is that there aren’t enough resources available to people who want to learn more about JUNOS.  Often the answer is “Well, there’s the Fast Track Program”, or “Well, you can download the free books by Joe Soricelli or Harry Reynolds” (which by the way, it’s high time we get those books back in print – they are excellent books, well written, and in my opinion, second to none when preparing for the Service Provider track).  But often this isn’t enough for the folks who want to learn more.  While Juniper’s Fast Track Program is an extremely valuable resource, it’s coverage is limited to only a few select technologies.  Furthermore, beyond a handful of books on Amazon, there aren’t many additional resources outside of reading Juniper’s technical documentation or various white papers for those students who truly wish to expand their knowledge base.  Sure, the JNCIA-M/T, JNCIS-M/T, JNCIP-M/T, and JNCIE-M/T study guides are freely available, but more often than not people would prefer to have these in hardcopy format rather than in a PDF and printing a PDF doesn’t provide the reader with quite the same experience.  I don’t care what anyone thinks, electronic formats aren’t going to be replacing printed books anytime soon.  Just ask yourself when was the last time you read an entire book on a computer screen?  NOTE TO AUTHORS: A Kindle version would be nice and this would be a good first step, but then again, how many people do you know that actually own a Kindle?

In 1996, when Cisco Press was formed as a joint publishing partnership with Pearson Education, Cisco realized that providing educational materials to meet the growing demand for networking technologies was going to be a cornerstone to their success.  In 1997, “Internet Routing Architectures” by Sam Halabi rolled off the printing press and was an instant best seller.  The rest was pretty much history.  Cisco understood early on that there was a vacuum in the market for high-quality technical books covering the networking arena, and more specifically Cisco technologies.  They met that demand and incidently watched their market share grow to the behemoth they are today.  Since its inception, Cisco Press has published well over 400 titles.  It’s no surprise that given a wealth of educational resources at their disposal, engineers who understand a given technology are more apt to recommend those technologies to their peers, and more importantly, to the people who often end up buying the technology. One only needs to look at the 20,000+ CCIEs in the world today to understand thats a pretty big propaganda machine to silence.

I’ve worked with both Cisco and Juniper technologies for a long time, and I’m convinced that Juniper’s products are infinitely superior, but if Juniper is serious about being a major competitor, especially in the Enterprise market or other market segments outside of the Service Provider arena, it’s high time they think about creating a publication arm which can compete with the likes of Cisco Press.  It might not be the most profitable business, and in fact, if you’re simply looking at dollars and cents with a microscopic view, it might even be a losing proposition.  But the demand is out there, and if they meet it, I am fairly certain Juniper will see their marketshare increase in a really big way.

An Overview of BGP FlowSpec

I have given this presentation a few times in the last year and was asked to make this available for public consumption. Essentially, this is a brief overview of RFC 5575, entitled “Dissemination of Flow Specification Rules”, written by Danny McPherson, Jared Mauch, and others.

This standard had somewhat of a rocky beginning as there was limited vendor support, but as of recently it appears to have picked up quite a bit of steam with Cisco announcing support for the protocol in the very near future.

The benefit of BGP Flow Spec is that it allows BGP speakers to use a new BGP NLRI defining flow filter information which can then be advertised to upsteam neighbors via BGP. The primary and immediate motivation of this protocol is to provide intra and inter provider distribution of traffic filtering rules to filter DoS and DDoS attacks, however it can be used for a wide variety of applications in which filtering information must be dynamically distributed throughout a network.

I will probably make additional modifications to these slides as the protocol gains more significant foothold throughout the vendor community and as Service Providers gain more practical deployment experience. As with my other presentations, I will eventually add a voice-over to turn this into a slide-cast.

 

IP Routing Tutorial

I’ve given this tutorial quite a few times now and several people have asked me to make it publicly available. This is very much geared towards non-technical folks who would like to have a better understanding of how routing in the Internet works.

It covers a brief history of the Internet and evolution of dynamic routing protocols, as well as high-level coverage of link-state vs. distance vector IGPs in addition to discussing EGPs and their role in the exchange of routing information between Autonomous Systems. It also has a few slides on QoS, MPLS, and IPv6.

I would still like to make some modifications to a few of the slides as well as adding more content around MPLS and the future of IP.  I will also eventually add a voice-over to turn this into a true slide-cast.  In the meantime, I’ve put it up on my slideshare account for those of you who would like to have access to it.

 

Hardening DNS Against Reflection Attacks

Generally, the most prevalent types of attacks and the ones which are most likely to target a DNS infrastructure are reflection/amplification attacks (which generally use a DNS servers resources against other third-parties).  Understanding the attack-vector employed in most reflection attacks is necessary in order to understand how to harden an environment against these types of attacks.

In a typical DNS reflection attack, attackers send a large number of queries from a spoofed source address.  The address which is spoofed is typically that of the victim.  When the requests are received by the nameserver, all ensuing responses to these queries are directed back towards the spoofed IP of the victim.  The amplification factor comes into play in these types of attacks because the attacker will typically query for a record of a large size, typically that of a root record which by it’s very nature is very large in size.  By simply sending in small queries, on the order of around 90 Bytes, an attacker can typically get a multiplication factor of five times that of the original query.  This allows an attacker to use a smaller number of hosts in the botnet and cause considerably more impact than would otherwise be possible if these devices were sending traffic directly towards the victim.

Due to the fact that the purpose of this attack is to reflect packets back towards the victim, all of the source IPs of the DNS queries contain the same address.  This makes it fairly easy to spot a reflection attack utilizing your infrastructure.  A simple observation of a spike in DNS queries from a singular IP is a clear indication that this is going on.  One would think that these types of attacks can be mitigated fairly easily by implementing simple ACLs on routers to prevent the incoming queries from those spoofed IP, and in fact that is a method commonly used by network administrators to protect against these types of attacks.  However, most security experts suggest that the filtering should actually be implemented on the nameserver itself – in fact this has been considered an industry best practice for quite some time now.  The reason for this is that implementing an ACLs on a router is largely a reactive countermeasure which can only be deployed after the fact.  An administrator will still need to identify the target of the attack before filters can be put in place; furthermore these types of filters only serve to cause a legitimate Denial of Service when that particular victim actually attempts to query anything for which the nameserver is actually authoritative for.  As an alternative to ACLs, some proposed configurations below can be used to eliminate this problem in it’s entirety.

At the very onset of your investigation into the vulnerabilities of your DNS infrastructure and potential remedies one of the very first things a network administrator must determine is if the nameservers allow anyone on the outside world to query for root (.).  Using the example below, one can easily check to see if their nameserver responds with a root-referral when queried for root.  If you see something along these lines, you can be fairly certain your nameserver is responding with a root-referral:

Normally, most Internet-facing authoritative nameservers should not respond to recursive third-party queries for root.  If an authoritative nameserver responds to queries for root with a root-referral, attackers will likely use your nameservers as an amplification vector to launch attacks.  It is better to remove the temptation altogether, by not allowing these in the first place.  Furthermore, instead of responding with the root-referral, nameservers should be configured to respond with REFUSED or SERVFAIL or another similar type of message.  The reason for this is simple – a REFUSED message is only on the order of around 50 Bytes.  If a countermeasure such as this is not employed, attackers can send in a relatively small spoofed query and will get roughly a 512 Byte response.  However, if we respond with a REFUSED message, the amplification factor is on the order of 1:1.  From an efficiency standpoint this does not provide much if any amplification, and therefore the attackers will simply look for other providers whom don’t apply such stringent security measures.

NOTE: Of course if you are in the business of providing recursive DNS service, that is an entirely different story – if that is the case, network administrators should take extra precautions by strictly enabling this function on the Recursive DNS servers (not the Authoritatives) and combining firewall-filters to limit recursive service to only IP blocks of paying customers.

While we’re on the subject, another current best practice in the industry is to apply a similar methodology to requests for records for which a given DNS Server is not authoritative for.  Some resolvers may respond to these types of requests by providing a root-referral, and in the worst cases a resolver may actually perform a recursive query on behalf of the original source.  An authoritative resolver should respond to any non-existent domain requests with either the REFUSED message or other similar type of message, rather than providing a root referral or performing a recursive query.  In fact, BCP 140 (Preventing Use of Recursive Nameservers in Reflector Attacks, RFC 5358) looked at this problem and concluded that sending REFUSED was the best general guidance that can be given.  While BCP 140 applies guidance to recursive servers, returning REFUSED to queries which are not within the namespace served by authoritative servers is entirely consistent with BCP 140.  You can generally find out if your nameservers allows for recursion or if it responds with a root-referral, by using a command such as the following:

If you see a response which looks like the large root-referral response above, or some other type of response other than a REFUSED or SERVFAIL, you should take steps to harden your resolver.  One can also look for an “RA” entry in the “Flags” section of the DNS response which should give you some indication as to whether the resolver allows for recursion.

In conclusion, there are several such steps which allow administrators to prevent from being used as an unwitting pawn in an attack against third-parties.  Firewall filters are effective, but are reactive in nature.  Therefore, it is recommended to follow some of the steps below in order to effectively harden your DNS infrastructure against reflection attacks.

SYNOPSIS – Steps to prevent a DNS infrastructure from being used for reflection/amplification type attacks:

  1. Disable querying for root on Authoritative resolvers, return REFUSED
  2. Filter queries to Recursives from only paying customers, via ACLs
  3. Apply BCP 140 or similar rules to cause any non-existent domain queries to respond with a REFUSED.

An excellent whitepaper entitled “Anatomy of Recent DNS Reflector Attacks from the Victim and Reflector Point of View” by Ken Silva, Frank Scalzo and Piet Barber covers this topic in more detail.