Mapping Interconnection in the Internet: Colocation, Connectivity and Congestion

As the global Internet expands to satisfy the demands and expectations of an ever-increasing fraction of the world's population, profound changes are occurring in its interconnection structure, traffic dynamics, and the economic and political power of different players in the ecosystem. These changes not only impact network engineering and operations, but also present broader challenges for technology investment, future network design, public policy, and scientific study of the Internet itself. And yet, from both scientific and policy perspectives, the evolving ecosystem is largely uncharted territory. Two related transformations of the Internet ecosystem motivate this project: the emergence of Internet exchanges (IXes) as anchor points in the mesh of interconnection, and content providers and Content Delivery Networks (CDNs) as major sources of traffic flowing into the Internet. By some accounts over half the traffic volume in North America now comes from just two content distributors (Youtube and Netflix). This shift constitutes the rise of a new kind of hierarchy in the ecosystem, bringing new constraints on existing players who need to manage traffic on their networks to minimize congestion, resulting in tussles among commercial players as well as between the private sector and regulatory bodies, at the expense of users suffering degraded quality of experience.

The goal of this project is to characterize the changing nature of the Internet's topology and traffic dynamics, and to investigate the implications of these changes on network science, architecture, operations, and public policy. The first task is to construct a new type of semantically rich Internet map, which will elucidate the role of IXes in facilitating robust and geographically diverse but complex interdomain connectivity. This map will guide the second task: a measurement study of traffic congestion dynamics induced by evolving peering and traffic management practices of CDNs and ISPs, including methods to detect and localize the congestion to specific points in the network. Data used will include measurements gathered using CAIDA's global active measurement infrastructure Archipelago (Ark), as well as data contributed by industry players for use in validation and refinement of methods. This project will advance the scientific community's understanding of the evolving Internet by developing and applying a more sophisticated model of the physical Internet topology. The inquiry is structured to pursue new methods of analysis and insights into the interaction between CDN dynamics, interconnection at IXes, and network performance. Results, including the sharing of collected and curated data sets, will also inform related fields, including traffic modeling, infrastructure resiliency and protection, network economics, industrial organization, and future network architecture design. Results will be disseminated via publications, conferences, industry meetings, online lectures, web pages. CAIDA and CSAIL will jointly host annual international interdisciplinary workshops to close the loop between empirical research results, theoretical modeling activities, and policymaking. By providing unbiased data about the evolving Internet ecosystem, the proposed research has the potential to be of great value to Internet users, network operators, and governments.