In today’s digital entertainment and streaming services, only a few names resonate as strongly as Netflix.
With million of unique users daily, Netflix is not only known for its impressive content library, but also for the quality of the delivering of this content. As network engineers, we easily understand that smooth streaming to viewers around the globe is a huge challenge. Netflix representing (as of today), 15% of the total Internet traffic, it is a challenge not only for Netflix, but also for all ISPs connecting Netflix users.
At the heart of this delivery system lies Netflix CDN, or Content Delivery Network, known as Open Connect.
This article will not go in deep technical details as other posts in my blog would do, but will still provide you with a lot of information about the way Netflix managed to build one of the most optimized CDN for their specific needs.
The history of Netflix’s CDN
The need for a robust and highly scaled Content Delivery Network mainly started when Netflix switched from DVD renting to video streaming (yes, for those of you who weren’t aware, the initial business of Netflix was to ship DVDs for a limited amount of time to their customers. See below a capture of the Netflix website in 2006).
At this time, the Netflix website was using commercial CDNs such as Akamai, as they were not having any specific requirements linked to video streaming.
Starting 2007 and the switch to video streaming, the bandwidth and storage needs of Netflix was exponentially growing, which prompted their engineering team to think about a custom, proprietary CDN fully adapted to this specificity.
This move was driven by the need for more control, flexibility, and cost-effectiveness in managing their content delivery operations, which eventually led to the development of Open Connect, Netflix’s proprietary CDN platform, which cames to live in 2010.
The birth and evolution of Netflix Open Connect
In order to answer Netflix’s specific requirements, their teams had to create a CDN :
– at a better cost than other third-party CDNs
– with better quality (or, at least, more adapted to their incredibly high bandwidth needs)
– highly scalable, for global growth
– running only with Open Source or home-made applications and services
– which is not only a common caching CDN, but a proactive caching CDN
More technically speaking, Open Connect is designed to be deployed at ISP or IX locations, for partners who explicitly ask for a deployment. Each deployment is delivered turnkey (pre-configured, pre-loaded) by Netflix, while still allowing the ISP to control the traffic answered by the Open Connect deployment, entirely managed by the Open Connect control plane.
The first production OCA (Open Connect Appliance) was 4U in height, had a 500W power consumption, was embarking +100TB of storage and a 10GB SFP port. Running Intel Xeon processors, with 128 or 256 GB of RAM, it was able to deliver up to 7 Gbps of encrypted traffic bandwidth.
These OCAs were deployed in strategic locations to maximize content availability, often in regional data centers or at key interconnection points within ISPs. They were featured higher-capacity storage drives, capable of holding a significant portion of Netflix’s content library. They focused on ensuring that popular and regionally relevant content was always available locally.
Driven by the advent of HD/4K and the growing number of users, the OCA have been redefined into different models, adapted according to their positioning and function. They have also unfortunately lost their very recognizable red color.
While it is quite difficult to obtain precise and accurate information about the latest OCA hardware specifications, here’s an estimate of what it actually seems to be.
Global appliances
Lower cost appliances used for smaller ISP partners and emerging markets. This appliance is designed for 10GE attached content delivery. Those are the most used OCA in the Netflix Open Connect CDN.
– 2U rack size
– AC or DC power supply, ~270 W peak consumption
– Up to 120 TB storage (HDD)
– Capacity (throughput) : ~18 Gbps
Storage appliances
Focused on reliable dense storage and cost effective throughput. This appliance is used to hold the Netflix catalog in many IX locations around the world and embedded at larger ISP partner locations.
– Large storage capacity
– 2U for rack efficiency
– AC or DC power supply, ~650 W peak consumption
– Up to 360 TB storage, mix of spinning disks and flash
– Network connectivity : 6 x 10 GB or 2 x 100 GB
– Capacity (throughput) : ~96 Gbps
Flash storage appliances
Used for very large deployments (in conjunction with storage appliances), these 2U appliances help scale network delivery for large sites up to Terabits per second.
– 2U rack size
– AC or DC power supply, ~400 W peak consumption
– Up to 24 TB storage (full flash)
– Capacity (throughput) : ~190 Gbps
On big deployments, storage appliances are used for full catalog offload, while flash appliances are used for high speed popular content serving.
A typical scale for such a deployment (used on main IX or ISP locations) is composed of 10 storage OCA + 30 flash OCA. However, most deployments around the globe consists of a single (or a few) global OCA.
All appliances runs on FreeBSD OS, and all content is delivered using NGINX.
Deployment process and benefits
Open Connect purpose, as any other CDN, is to bring Netflix content closer to users by hosting popular shows and movies directly within ISP (Internet Service Provider) networks or at Internet Exchange Points (IXPs).
Regarding the amount of bandwidth Netflix represents for the global Internet backbone, and more generally for all popular ISPs, the OCA deployment model is a “give and take” partnership between Netflix and ISPs/IXs, where both parties mutually benefit from the arrangement.
Netflix provides ISPs and IXs with Open Connect Appliances (OCAs) at no cost, allowing to deliver its content more efficiently and limiting bandwidth consumption, gaining significant network and cost benefits.
In return, ISPs and IXs host and maintain the appliances within their infrastructure, and are eventually requested for basic intervention (console connection / reboot / hardware part replacement).
For Netflix, deploying OCAs close to users reduces the strain on their upstream infrastructure and ensures faster, higher-quality streaming. For ISPs, OCAs drastically reduce the volume of Netflix traffic traversing their upstream internet links, which leads to lower bandwidth costs, reduced congestion, and improved network efficiency.
The deployment process is actually quite simple :
- After they initiate an Open Connect deployment request, Netflix works closely with the requesting ISP or IX to determine the best location for the OCA deployment based on user traffic volume, regional demand, and network infrastructure. They agree together to validate the deployment need and determine the adapted deployment type and scale
- Once an agreement is reached, Netflix ships the pre-configured OCA hardware to the ISP or IXP. The hardware is designed to integrate seamlessly with the existing network infrastructure, requiring the partner to provide compatible network switches and necessary bandwidth. Configuration templates for the most popular switches vendors are provided on the Open Connect deployment guide
- As soon as the OCA is powered up and connected to the partner network and the BGP peering is established, the OCA is able to join the Open Connect control plane (fully running on AWS), and is visible in the customer portal, ready for provisioning. It will wait for the next fill window (see later) to synchronise and get the latest content catalog (OCA are shipped pre-loaded with the proper titles list depending of their destination, so that they don’t have to download the full catalog once deployed). This step can actually take up to 2 days.
- The ISP / IX responsible administrator has then full visibility on its managed appliances through the Open Connect partner portal, and is able to have a view on its status, BGP learned routes (we’ll see why this is an important parameter later), and to eventually tune some time windows at which the OCA synchronise its content catalog on a daily basis. It also receives notifications about potential failures requiring manual intervention (SFP or disk replacement, loss of connectivity, etc).
The strategic positioning of the Open Connect deployments drastically improves user experience, reduces Internet bandwidth usage and throttling, and lowers transfer costs.
It has been estimated that Open Connect deployment in conjunction with codec optimization reduced ISPs costs between USD1 billion and USD1.25 billion for 2021 (source).
Sites and OCAs naming
Each OCA has an hostname assigned, based on its geographical location, and the ISP / IX provider hosting it.
The geographical part of the hostname is based on the IATA airport code.
Site name assignment : each Open Connect site has an identifier, from which the OCA hostnames are derived.
The site name is in the following format : <IATA-airport-code>YYY.<ISP-Name>.<partner-type> where YYY is a zero-padded integer to give each site a unique number when multiple sites use the same IATA airport code.
The <ISP-Name> is optional (can be avoided if this is a single implicit Netflix partner name, ie: if the partner-type is IX).
The <partner-type> is thus either ISP or IX.
For example, the main site for Paris north region IX could be : CDG001.IX
The first site for Toulouse, and ISP Bouygues Telecom could be : TLS001.bouyguestelecom.ISP
OCA hostname : the Open Connect Appliance hostname is then derived from the site name. Each appliance gets a unique ID (incrementing) on its site pool.
The hostname is in the following format : cXXX-<site-name>.1.oca.nflxvideo.net
XXX is a zero-padded unique integer assigned to each appliance in the given site.
(Note that the “.” in the site name are replaced by “-” in the appliance hostname).
For example, the 9th appliance, in the main Toulouse site for Bouygues Telecom could be : c0009-tls001-bouyguestelecom-isp.1.oca.nflxvideo.net
Each appliance then gets two derived FQDN, for its IPv4 and IPv6 IP addresses : the appliance hostname is simply prepended with either “ipv4-” or “ipv6-“.
Example : ipv6-c009-tls001-bouyguestelecom-isp.1.oca.nflxvideo.net
Routing and content-steering : BGP as the main tool
As soon as they are connected to the ISP / IXP network, OCAs establishes a BGP peering with a pre-configured neighbor (provided during the OCA request by the partner).
This peering is established using BGP ASN 40027 (for the most curious among you !).
Netflix uses BIRD for the BGP implementation on their FreeBSD system.
Not only this BGP peering is used to announce “outbound” routes to the OCA (eventually a default route, or at least the necessary AWS-hosted ranges used to reach the OpenConnect control plane), but it is also used by the Netflix partner (ISP or IX) to announce the IP ranges that each OCA will be expected to serve.
Those announced prefixes received by the OCA BGP process are synchronized with the Open Connect control plane, which is working with the Netflix frontend to steer users to the best and closest OCA when they start streaming content.
This is one of the very smarts features which permits to hosting ISPs and IXs to perform traffic engineering and management to the appliances they host for Netflix.
In the example above, ISP announces prefix 176.128.0.0/11 prefix to the OCA appliance.
This appliance reports this received route to the Open Connect control plane, which is then used for the best-matching logic presented below. Of course, the appliance not only reports the learned routes, but is continuously reporting (every 5 minutes) the full state of the catalog of content it have stored, connectivity status, hardware metrics, etc.
When a Netflix user needs to retrieve any media content from the Netflix front-end page (either a movie trailer, a full movie show episode, or any other kind of media), the Netflix front end requests the Open Connect control plane to get the best matching appliance based on the following matching criteria :
1 – Filter on OCAs reporting availability status “OK” for the requested media
2 – The appliance that receives the most-specific route to the client’s prefix.
3 – The appliance that receives the route to the client’s IP with the shortest AS path
4 – The appliance that receives the route to the client’s IP with the lowest multi-exit discriminator (MED)
5 – The geographically closest appliance, based on client IPs, whose location is then compared to the latitude and longitude of nearby OCAs to determine the closest available system.
We’ll see later that some bandwidth stability / efficiency reports are also used in conjunction with this matching logic to find the best OCA over time.
Netflix also offers direct peering capabilities on some big IX locations, using PNI (private intercos) on ASN 2906.
In such a situation, the Open Connect control plane not only learns routes from the ISP appliances, but also directly from the peering. As routes reported by the OCA have an AS_Path length of 1 (interco ASN 40027 is ignored) while peering routes are learned with a length of 2, ISP Open Connect appliances are still preferred by the Open Connect control plane matching logic for the clients requests direction.
Also, using direct peering, Netflix adds 50 to the MED value received from the prefixes on their peering ASN, while adding 100 to the MED value for prefixes received from Internet. This helps promoting the use of the direct peering for OCA communications.
By capturing / decrypting flows to the front-end API while browsing www.netflix.com, we can easily see the information contained in the JSON exchanges, providing close ISP OCA references for popular content, while providing IX OCA pointers for less popular titles.
Clustering and filling
As evoked previously, OCA are sent pre-filled with the last Netflix content catalog for the target location.
Of course, they need to be regularly updated with the latest content. Those updates are not done haphazardly, and are based on pro-active caching, which is one of the mains difference between third-party CDNs and Open Connect.
Indeed, Netflix can very accurately predict what most of their users will watch and what time of day they will watch it, allowing Open Connect to make use of non-peak periods to download most of the content updates to the OCAs during these time windows. By reducing disk reads (content serving) while performing disk writes (adding new content to the OCAs), disk efficiency is optimized by avoiding read/write contention. The predictability of off-peak traffic patterns permits to find the best time every day to get the content to be positioned at the proper locations before user traffic starts to ramp up, making all of the OCA capacity available for content serving.
Below is an extract of the Open Connect partner guide which shows some information about the filling time-window wide ranges (notice that the ISP/IXP administrator can shift this timeframe +- 2 hours).
Of course, the full time window is not used for updates download, but the exact appropriate time is defined by the Open Connect control plane during this wide time range, depending on learned traffic patterns and update size need.
Content pre-loading is done on a very organized and hierarchical way.
Based on Netflix user habits, bandwidth management, etc, the Open Connect control plane builds a source manifest file for each OCA, which are eventually grouped into a manifest cluster (a group of appliances at the same location).
The source manifest file is built depending of the content region (the group of countries that are expected to stream content from the cluster), a particular popularity feed (which in simplified terms is an ordered list of titles, based on previous data about their popularity), and takes into account the number of copies the manifest cluster should host for each title : if the manifest cluster is composed of multiple appliances, around 40% of the storage space of each appliance will be occupied with the same content, so that Open Connect can deliver this most accessed content to an higher amount of users.
During their regular exchanges with the Open Connect control plane, each OCA gets this updated fill manifest file, and compares its current catalog of hosted content with what the Open Connect control plane expects it to host. Any file which is not part of the manifest file is deleted, while any missing file is planned to be downloaded during the next filling window.
At the same time, the OCA also continuously reports its current catalog status (available titles) to the Control Plane, so that users would not be redirected to a given OCA to start streaming their last TV show episode while this title is not available on the target appliance.
At the root of the “filling process” is an AWS S3 bucket : yes, all the Netflix catalog is hosted in AWS S3.
However, accessing and downloading files from S3 costs money, and having thousands of OCA appliances around the world accessing a central S3 bucket to update their catalog every day could also be a bottleneck, and cause high bandwidth usage on ISP connections.
To mitigate this behavior, a hierarchical tree is built, which permits OCA not to download their updated content from the data source, but from the closest availability point, which can be another OCA. This is part of the filling escalation policy, which is included in the filling manifest file for each title, and instructs the OCA where to download the new content.
To properly understand the way filling escalation policies works, it is important to understand that BGP is still an important brick of the content updates propagation : using the announced BGP routes to each OCA and applying both BGP best-path calculation and geographical location analysis, the Open Connect control plane is able to include in its filling policy for each OCA, informations about the closest possible information source, the next closest, etc.
This filling escalation policy defines :
– How many hops away an OCA can go to download content, and how long it should wait before doing so (making sure this location has had enough time to download the title before making it available to its peers)
– Whether the OCA can go to the entire Open Connect network (beyond the hops defined above), and how long it should wait before doing so
– Whether the OCA can go to S3 (root of the content propagation tree), and how long it should wait before doing so
The control plane elects some OCAs as masters for a given title asset.
The filling escalation policies that are applied to masters typically allow them to reach farther with less delay in order to grab that content and then share it with close non-masters.
To better understand the necessity of having such propagation methods for the content updates, it is important to note that the average amount of updated catalog every day is around 7.5TB.
Title liveness : once a new media has been successfully propagated to a required percentage of Open Connect Appliances, it is tagged as “Live” on the Netflix internal management tools, which then allows to trigger its appearance on the users main page for streaming.
Each new title is thus propagated (proactively cached) to Open Connect several days before the availability date announced to the users.
Performance improvements used on OCA
Beyond designing and deploying their own CDN to meet their specific needs, Netflix went a step further by optimizing the hardware and software of their Open Connect Appliances (OCA).
These custom-built servers are not only strategically placed to ensure efficient content delivery but are also packed with advanced performance improvements that make Netflix’s streaming service faster, more reliable, and highly scalable.
Zero-Copy Networking: Maximizing Throughput
One of the key enhancements is zero-copy.
Traditionally, data passes through multiple layers of the operating system, when copied from hardware (disk) to a network socket. Not only it results on more memory buffers usage, but also on much more context switches.
Netflix engineers implemented (and participated in the improvement of) the ZeroCopy (sendfile()) facility on FreeBSD, which allows the OCA to send data directly from disk to network without extras back-and-forth context switching between user-space and kernel-space, and limits memory buffers usage.
This significantly reduces CPU usage and increases the throughput, allowing Netflix to serve millions of simultaneous streams without bottlenecks.
If you want to learn more about this technique, I already wrote another blog post about it !
Optimized HDD Title Positioning
Netflix takes an ingenious approach to hard drive management.
On OCA using HDD (mechanical disks), the most demanded titles are stored on the outer edges of hard drive platters.
This method leverages the fact that the outer regions of a disk can be read faster due to seek time and rotational latency. By placing popular content in these high-performance zones, Netflix ensures that users experience faster load times for the most-watched shows and movies.
SSL Offloading to Network Cards
To ensure secure streaming while minimizing the performance overhead of encryption, Netflix uses SSL offloading.
SSL (Secure Sockets Layer) encryption is critical for protecting user data and maintaining privacy, but handling SSL encryption and decryption operations can be computationally expensive for OCA CPUs.
Netflix mitigates this by offloading SSL processing to specialized network cards designed to handle cryptographic functions : while the TLS session handshake is handled by the OS, as soon as the TLS sessions keys have been calculated and exchanged, they are offloaded to the network card chipset, which is in turn responsible for traffic encryption, the OCA OS only sending now clear (not encrypted) traffic to the network card.
Give me a CDN, I’ll make it a bandwidth test tool
I think we can say that Netflix managed to create one of the most robust and performant CDN (still taking into account that this is 100% designed to answer their own needs, and is not intended to replace any third-party commercial CDNs we are used to use).
With around 12k Open Connect Appliances deployed in 150 countries, some estimates give a quantity of traffic delivered by Open Connect around 265 Tbps (TeraBytes per Second) at peak hours. Amazing, isn’t it ?
But this massive infrastructure isn’t just limited to delivering movies and TV shows : in 2016, Netflix harnessed the power of Open Connect to introduce Fast.com, a simple yet highly effective bandwidth testing tool. Unlike other speed tests that connect to random servers on the internet, Fast.com utilizes the same Netflix Open Connect servers responsible for streaming video. This means that when users tests their internet speed on Fast.com, they’re actually testing the real-world performance of their connection to Netflix Open Connect, by downloading 25MB of a random chunk of video data from one of the closest OCA.
This tool can be used by accessing Fast.com (or using the Fast mobile app), but it is also used internally by the Netflix app to report bandwidth information to the Open Connect Control Plane, which will be able to adjust accordingly the “routing” of its users.
This service makes uses of the Open Connect control plane on AWS to provide the list of the closest OCA, and then download the 25MB of data necessary to perform the test from those appliances.
Below is an explanation of the process when the Fast service is used to start a speed test, from netflixtechblog.com :
What may interest you here (among the fact to have a speed test result using the Netflix Open Connect network !) is to have an idea of the OCA you use from your location.
This is possible by using the https://api.fast.com/netflix/speedtest URL, which unfortunately requires a token to be requested (sorry if you already clicked the link !).
The good thing is that we can get an active token from the Fast.com website javascript files.
By using the command below, you’ll be able to extract the latest token from the website, and to use it to request the Open Connect API for a list of the OCA “hierarchy” you use based on your location :
token=$(curl -s https://fast.com/app-ed402d.js|egrep -om1 'token:"[^"]+'|cut -f2 -d'"'); curl -s "https://api.fast.com/netflix/speedtest?https=true&token=$token" | jq(Remove the ” | jq” if you don’t have it installed, or even better : install it, muggle !)
The answer you’ll get is an ordered list of the OCA appliances you probably use when watching your latest TV show episode on Netflix.
[
{
"url": "https://ipv6-c009-tls001-bouyguestelecom-isp.1.oca.nflxvideo.net/speedtest?c=fr&n=5410&v=130&e=1727304567&t="
},
{
"url": "https://ipv6-c069-cdg001-ix.1.oca.nflxvideo.net/speedtest?c=fr&n=5410&v=217&e=1727304567&t="
},
{
"url": "https://ipv6-c182-cdg001-ix.1.oca.nflxvideo.net/speedtest?c=fr&n=5410&v=216&e=1727304567&t="
},
{
"url": "https://ipv6-c118-arn001-ix.1.oca.nflxvideo.net/speedtest?c=fr&n=5410&v=31&e=1727304567&t="
},
{
"url": "https://ipv6-c105-arn001-ix.1.oca.nflxvideo.net/speedtest?c=fr&n=5410&v=198&e=1727304567&t="
}
]Conclusion
Netflix’s Open Connect is a shining example of how such big techs are able to deploy highly scalable systems designed specifically for their own requirements. Again : this CDN is not better than its third-party equivalents. It is just perfectly adapted to deliver pro-actively cached media content all around the world, to the 278 millions users of Netflix with unprecedented response time and performance.
Netflix has implemented a series of cutting-edge performance optimizations that make Open Connect incredibly efficient. Features like zero-copy networking, HDD title positioning, and SSL offloading to network cards allow Netflix to maximize throughput and reduce CPU load while ensuring secure and fast delivery of content.
But what makes it even more powerful, is that it is also designed for ease of deployment. Netflix makes it simple for ISPs and IXs to get OCAs, which are offered for free to hosting partners. By placing these appliances within their partners infrastructures, Netflix minimizes bandwidth consumption across backbone networks while improving streaming quality for end-users. This makes it a win-win for both Netflix and the ISPs/IXs, as it reduces operational costs and enhances customer satisfaction.
This was (as of now) the longest post on my blog. I hope it was interesting for you, and that you had a good time reading it 🙂