pull through cache
Overview
This post shows how to bypass Docker Hub rate limit using Pull through cache.
Background
Docker Hub returns a 429 Too Many Requests error when you exceed the rate limit for image downloads.
Here is the pull rate limit for each user type:
| User Type | Pull Rate Limit |
|---|---|
| Anonymous (not logged in) | 100 docker pull requests per 6 hours |
| Logged in account | 200 docker pull requests per 6 hours |
| Pro or higher (logged in) | No limit |
Here are some ways to avoid Docker Hub rate limit:
- Use imagePullSecret based on Kubernetes secrets to pull public images.
- Manually pull images and store them in Private ECR.
- Use Pull through cache to cache public images in Private ECR.
- Use Harbor's Proxy Cache to cache public images in Harbor.
From my experience with all these methods, Harbor Proxy is the most convenient and vendor-neutral. It doesn't lock you into a specific cloud, and if you already use Harbor, you just need to create a Proxy Cache project. This post covers option 3.
Setup
Before Pull Through Cache
I needed to pull 15 images for an Airbyte version upgrade.
I didn't know about Pull Through Cache at first. So I manually cached all 15 public images to Private ECR using this tedious workflow:
After Pull Through Cache
With ECR's Pull Through Cache, you only need to run a single pull command. This makes caching multiple public images from different registries much easier and saves pull costs.
Private ECR repositories are created automatically and cache the public images. The best part is that it caches both the container image manifest and image files from public registries, so it works for all CPU architectures.
You need to set up these ECR settings first:
- Create Registry Template: Auto-define Permission, Tag, and Lifecycle Policy for Private ECR repositories created by Pull Through Cache.
- Create Pull Through Cache Rule: Define credentials for the public registry you want to cache.
Conclusion
Pull Through Cache simplifies caching public images to Private ECR. Once set up, just run a single pull command and ECR handles the rest. If you want to avoid vendor lock-in and run a self-hosted registry, consider Harbor Proxy Cache as an alternative.