For almost two months my media delivery path looked like this:
Browser
↓
working server
↓
Nginx
↓
local proxy_cache
↓ cache MISS
Bunny StorageImages, video and audio lived in Bunny Storage. Users never contacted Bunny directly. My Nginx requested objects from [https://storage.bunnycdn.com](https://storage.bunnycdn.com), added the server-side AccessKey, cached the response on local SSD and served the public URL itself.
I want to make the scope explicit from the beginning: this is not a review of Bunny CDN. I did not use Bunny CDN for this experiment. Bunny's Storage Zone plus Pull Zone/CDN architecture is a different delivery model, and I still consider Bunny CDN a very strong product. What I tested was specifically Bunny Storage as a runtime origin for my own Nginx cache.
Why the original architecture looked reasonable
The workload contains a large number of immutable media objects: AVIF, JPEG, PNG, MP4, WebM, audio and other static files, ranging from small images to much larger videos. I did not want the entire dataset on the application server, but I did want my own Nginx to control public URLs, cache policy, redirects, missing-file behavior, Range handling and media fallbacks.
┌── HIT ── local SSD
│
Browser → Nginx cache
│
└── MISS ── Bunny StorageOn a HIT, the design was excellent. Nginx served the object straight from local disk, so Bunny was not involved in that request at all. That success actually hid the important part of the system for a long time: the behavior of a cold MISS.
Popular files hid the real origin latency
A frequently requested image is likely to remain in cache. A rarely opened older file is much more likely to have been evicted. That is why older pages became an accidental origin benchmark for me: they triggered the path to Bunny Storage more often.
One July snapshot showed roughly 36 GB of Bunny media cache, more than 355,000 cache files, a configured cache size around 35 GB and a root filesystem about 93% full. Other nearby snapshots approached 400,000 cache entries. A cache of that size was still finite, so cold content inevitably existed.
The key lesson was simple: a fast HIT does not prove a fast origin. It proves that local cache is fast.
Production logs showed failures before the file body was transferred
The most useful evidence came from Nginx error logs. I repeatedly saw messages such as:
upstream timed out
while connecting to upstreamand:
upstream timed out ... while SSL handshaking to upstreamThose errors are different from a large file simply taking a long time to download. In these cases Nginx was still trying to establish the upstream connection or complete TLS before normal object transfer could begin.
In one July 30 diagnostic snapshot, the last 5,000 lines of the media error log contained 317 upstream-timeout matches. A snapshot from the previous day contained 578. Those are log matches, not 317 or 578 unique users, but they clearly show that this was not a single isolated request.
The bad cases were not tied to one storage IP
During that period the storage hostname resolved to addresses including:
109.61.89.53
109.61.89.54
109.61.89.55
109.61.89.57
79.127.226.193I observed failures across different addresses from that set. From my server's point of view, the problem was therefore the external origin path as a whole rather than one permanently bad address.
I cannot prove from my logs whether the underlying cause was a storage backend, routing, peering, my provider's path to Bunny, balancing behavior or another network factor. What I can prove is narrower and more useful: some cold requests could not establish the Bunny Storage upstream connection within the expected time.
One measurement exposed the long tail
I also tested the resolved storage addresses individually. Four paths looked normal:
109.61.89.53 total ≈ 29.8 ms
109.61.89.54 total ≈ 28.5 ms
109.61.89.57 total ≈ 42.7 ms
79.127.226.193 total ≈ 43.6 msOne measured path was completely different:
109.61.89.55
TCP 1.017782 s
TLS 1.048306 s
TOTAL 1.054474 sThis does not mean Bunny Storage was always one second slow. Most of the measured paths were in the tens of milliseconds. The important part was variance: the same origin hostname could lead to a connection path that was more than a second slower before useful object transfer was even considered.
For media pages, tail latency matters more than the average
If 79 images load in 30–50 ms and one takes a second, the arithmetic average can still look respectable. The user does not see the average. The user sees one empty rectangle that has not loaded yet.
That changed the way I evaluate media infrastructure. For galleries and other pages containing many independent objects, p95, p99 and the slowest cold MISS can matter more to perceived performance than a good mean response time.
Video exposed a second, completely different problem
Images are usually fetched as complete objects. Video is different because browsers can use HTTP byte ranges. A player may request:
Range: bytes=0-1048575and later, after a seek:
Range: bytes=50000000-51048575A correct server can answer with 206 Partial Content, allowing playback and seeking without downloading every preceding byte.
I wanted to preserve that behavior while also caching efficiently in Nginx.
My first Range cache fragmented one physical video into many cache keys
An early configuration forwarded the browser's Range and included it in the cache key:
proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;
proxy_cache_key "$scheme|$host|$request_uri|range=$http_range";
proxy_cache_valid 200 206 301 302 30d;This prevented different partial responses from overwriting one another, but browser ranges are arbitrary. One client can request bytes=0-1048575, another bytes=0-999999, another can start at an unrelated offset after a seek. One physical MP4 could therefore create many different cache entries.
I did not measure exactly how many gigabytes this fragmentation consumed, so I will not invent a number. The mechanism, however, was present directly in the configuration.
I switched to normalized 1 MB slices
Nginx has a dedicated Slice module for this problem. Instead of caching arbitrary browser ranges, Nginx can normalize a large resource into fixed pieces:
0–1 MB
1–2 MB
2–3 MB
...The relevant configuration became:
slice 1m;
proxy_set_header Range $slice_range;
proxy_cache_key "$scheme|$host|$uri|$slice_range";
proxy_cache_valid 200 206 30d;This was much more attractive for reusable cache entries. Then production started emitting a very specific error:
etag mismatch in slice response while reading response header from upstreamThe MP4 slices were failing on ETag consistency
On July 30, repeated slice subrequests for one MP4 produced etag mismatch in slice response while requests were reaching storage addresses including 109.61.89.53, 109.61.89.57, 79.127.226.193 and 109.61.89.55. On July 31 the same class of error appeared again for another MP4, with subrequests moving across several addresses from the same storage pool.
This was the issue I initially described informally as pieces having different “signatures.” That description was wrong. The relevant value was ETag, not the Storage AccessKey, not a signed URL and not a cryptographic signature.
Why Nginx refuses to assemble slices with mismatched ETags
An ETag is an HTTP validator for a particular representation of a resource. Conceptually:
slice 1 → bytes 0–1 MB → ETag A
slice 2 → bytes 1–2 MB → ETag Ais consistent. But:
slice 1 → ETag A
slice 2 → ETag Bmeans Nginx can no longer safely assume both pieces belong to the same representation. Blindly combining them could assemble bytes from different versions of a file. The Nginx Slice behavior is therefore protective rather than arbitrary. The Nginx article on byte-range caching describes this validator check.
My logs prove that Nginx saw incompatible ETags between slice responses for the same MP4. They do not prove the precise internal reason why those validators differed, because I did not record the literal ETag value of every subrequest. The changing upstream addresses make backend-response inconsistency a plausible explanation, but I keep that as an inference rather than a proven cause.
This does not mean Bunny Storage cannot serve Range requests
Range requests themselves worked, and I was explicitly caching 206 Partial Content. The narrower problem was my combination of Nginx Slice, multiple storage responses and the ETag consistency Nginx required in order to assemble one safe representation.
Video could also suffer from the first class of problem at the same time: my logs contained MP4 requests that simply hit upstream timed out while connecting to upstream. So I had two distinct video failure modes:
- origin connection and long-tail latency;
- slice consistency and ETag mismatch.
I eventually stopped using Slice for MP4
A later MP4 configuration returned to ordinary browser byte ranges:
proxy_set_header Range $http_range;
proxy_set_header If-Range $http_if_range;
proxy_cache_key "$scheme|$host|$uri";
proxy_no_cache $http_range;
proxy_cache_valid 200 30d;Partial responses were deliberately not stored as the complete object. That removed the problematic MP4 slice assembly, but it introduced a trade-off: a cold Range request could once again depend directly on the external origin.
At that point I was choosing between increasingly complicated cache strategies rather than simplifying the actual source of the data.
I realized I was building part of a CDN in front of Storage
Over time my Nginx accumulated cache locking, stale responses, background updates, Range handling, 206, Slice, normalized chunks, keepalive, TLS session reuse, retries, timeout tuning and custom cache keys.
Each feature made sense individually. Together they made me ask a simpler question: what did I actually need from the origin?
The answer was boring: store immutable files and return their bytes.
The replacement origin is deliberately simple
I moved the media dataset to a dedicated server whose job is essentially Nginx plus SSD. For this class of static-only workload, a small machine around one vCPU, roughly 1 GB of RAM and hundreds of gigabytes of SSD is a reasonable starting point to consider, provided the network and disk are adequate.
That is not a universal sizing rule. The exact requirements depend on bandwidth, object sizes, concurrency, IOPS and cache-miss rate. The server used for my current measurements has more resources, so I do not claim that my measured peak is a benchmark of exactly a 1-vCPU/1-GB machine.
The new architecture
Browser
↓
working server
↓
Nginx + local media cache
│
├── HIT
│
└── MISS
↓
WireGuard
↓
media origin
↓
Nginx
↓
SSDThe media origin is private. The working server talks to one fixed private address. User-facing HTTPS still terminates on the working server, while the origin can use plain HTTP inside the already encrypted WireGuard tunnel.
What disappeared from the cold path
The old MISS path included public DNS, a public route, one of several storage addresses, TCP, TLS and the Storage API request. The new path is essentially:
working server
↓
WireGuard
↓
fixed private IP
↓
Nginx
↓
SSDThat does not make the network disappear. It removes variables that were unnecessary for this particular one-server-to-one-origin relationship.
Cold requests became cheap enough that HIT and MISS felt similar
After the migration I deliberately opened rarely visited pages to trigger cold media. Small real-file Range tests from the working server to the origin were typically around:
CONNECT ≈ 9.5–12.5 ms
TTFB ≈ 19–23 ms
TOTAL ≈ 19–23 msSequential and parallel checks completed successfully: 10/10, 20/20 sequential and 20/20 parallel in the tests I ran. On the origin itself, the same small request was typically answered in roughly 0.5–0.9 ms.
These are small Range/TTFB tests. They do not mean a full large video downloads in 20 ms. Full transfer time still depends on file size and available throughput. The useful result is that the cold connection path became both short and stable.
The old bad case and the new cold path are different by an order of magnitude
A normal old Bunny connection in my measurements was often around 29–44 ms. The bad observed path was about 1.054 seconds. The new tiny cold request is around 20 ms.
Mathematically, comparing that particular 1.054-second outlier with 20 ms is roughly a 52× latency difference. That is not a claim that my server is “52× faster than Bunny Storage.” It is a comparison of two concrete observed network paths. The important point is that the long tail disappeared from the tests I am running now.
MP4 is simpler now too
For MP4 I can use a different caching strategy: on a cold fill, the working Nginx can request the complete object from my origin, store one complete cache entry, then serve arbitrary byte ranges locally to browsers. Upstream Range can be cleared during the fill, while proxy_force_ranges allows local byte-range responses to clients.
cold MISS
↓
fetch complete MP4 from private origin
↓
one cached file
↓
client Range requests served locally as 206This removes the need to assemble one MP4 from slices obtained through different remote storage responses.
Full-object MP4 caching has a cost
If a cold MP4 is 500 MB and the first user only wants ten seconds, downloading the full object to fill the cache can transfer much more data between the two servers than a pure Range fetch would have used.
For my file sizes and access patterns that trade-off is acceptable because subsequent Range requests reuse one simple local cache entry. For multi-gigabyte, mostly cold video libraries I would consider a stable sliced cache, HLS/DASH, a video CDN or a service such as Bunny Stream instead.
Fixed slices still make sense for other large media
I do not consider Nginx Slice a bad feature. With an immutable object and a stable origin, fixed slices remain useful. The difference now is that every piece comes from one origin I control and one filesystem representation I control.
A smaller cache can feel faster when MISS is cheap
The old cache reached roughly 35–36 GB and hundreds of thousands of entries. After the move, my new cache settled around 16 GB because I deliberately kept roughly 8 GB free with min_free.
Despite being much smaller, the new cache produced a better experience on old pages. The reason is that cache size was solving the wrong problem before:
old:
HIT = fast
MISS = sometimes painful
new:
HIT = fast
MISS = also fast enoughA large cache reduces the number of misses. A good origin reduces the cost of every miss. For this workload, the second property turned out to matter more than I expected.
Real production traffic confirmed that the origin was doing only origin work
In one 60-second production snapshot the working server sent about 96.84 Mbit/s to users while receiving about 7.07 Mbit/s from the media origin. No new 502, 503, 504 or upstream timeout appeared during that measurement window.
In a later 30-second snapshot, public TX was 61.36 Mbit/s while WireGuard RX from the origin was only 1.40 Mbit/s. The byte ratio was 2.28% origin traffic relative to public TX in that window.
I do not call the inverse of that number a precise cache-hit rate: public traffic contains more than just media, and byte ratios are not request ratios. It does show that after warm-up the origin was serving mostly cold fills rather than duplicating the entire public traffic stream.
What I would monitor now
For media I would log at least:
$upstream_addr
$upstream_connect_time
$upstream_header_time
$upstream_response_time
$upstream_cache_status
$request_time
$statusThen I would compare HIT and MISS separately, especially p95 and p99 MISS latency.
For video I would test at least:
- the first range, for example
bytes=0-1048575; - a range from the middle;
- a suffix range from the end;
- multiple non-sequential ranges to simulate seeking;
- cold and warm behavior separately;
206,Content-Range,Content-Length,Accept-RangesandETag.
If Slice is used, I would explicitly verify that validators remain consistent across multiple slices. A single successful Range request is not enough to prove that sliced video caching is healthy.
One cold URL and twenty different cold URLs are different tests
proxy_cache_lock is useful when many requests want the same cold cache key. It does not combine twenty different files into one origin request. A page containing fifty distinct cold images can therefore be a much harder and more realistic origin test than a benchmark that requests one URL repeatedly.
Self-hosting gives control, not free reliability
A private media server makes the critical path easier for me to understand, but it moves operational responsibility to me: backups, disk health, free space, updates, firewalling, monitoring, restore procedures and potentially redundancy.
A single origin is also a single point of failure for uncached objects. Cached files may continue to work, but new MISS requests need a healthy origin. High availability requires another layer of engineering.
When I would still choose Bunny
If I needed global user-facing delivery, managed redundancy, rapidly growing storage or minimal infrastructure administration, I would absolutely consider Bunny again. I would specifically evaluate the architecture Bunny documents for Storage plus a Pull Zone/CDN rather than assuming my old Storage-API-behind-Nginx design is the only way to use the platform.
My test does not show that one VPS is better than Bunny CDN. It does not show that Bunny Storage is always slow. It does not show that Bunny cannot serve Range requests. It shows that, for my workload, using Bunny Storage directly as the runtime origin of my own Nginx cache produced cold-path and sliced-video behaviors I no longer wanted to manage.
The rule I use now
During those two months I optimized cache size, cache keys, Range behavior, 206, Slice, 1 MB chunks, keepalive, TLS reuse, retries, stale responses and timeouts. All of those were legitimate engineering tools.
The strongest improvement was simpler: I changed the origin.
I no longer start by asking how high I can make the cache-hit rate. I first ask:
What happens when the file the user wants is not in cache?
If the answer is a stable origin, predictable TTFB, correct Range handling, consistent validators and understandable failure modes, then cache is an optimization. If a good user experience depends on never encountering a MISS, the cache is hiding a deeper architectural problem.
For my workload, a small, boring private media origin turned out to be the better fit. It is not a CDN, it is not magic and it does not remove operational responsibility. It simply made the cold path boring. After two months of debugging images, connection timeouts, video Range behavior and etag mismatch in slice response, boring is exactly what I wanted.