rsync

Open to downstream mirror operators. Please pull at most two modules at a time and cap yourself at 50 MB/s — there is no hard limit configured, and we would rather keep it that way.

Modules

Listing and pulling

rsync rsync://pkg-mirror.bulsan.site/

rsync -avH --delete --delete-delay --delay-updates \
      --bwlimit=50000 --timeout=600 \
      rsync://pkg-mirror.bulsan.site/alpine/ /srv/mirror/alpine/

--delay-updates matters: it keeps the visible tree consistent while the transfer is in flight, which is the whole reason a half-synced mirror does not hand out a broken index.

Two-stage pull

What we run against upstream, and what we suggest you run against us — packages first, metadata second, so clients never see an index pointing at files that have not landed yet:

#!/bin/sh
set -e
M=/srv/mirror/fedora
U=rsync://pkg-mirror.bulsan.site/fedora/

flock -n /var/lock/mirror-fedora.lock true || exit 0

# stage 1 — everything but the metadata
rsync -aH --delete --exclude 'repodata/*' --bwlimit=50000 "$U" "$M"
# stage 2 — metadata last, atomically
rsync -aH --delete --delay-updates "$U" "$M"

date -u +%Y-%m-%dT%H:%M:%SZ > "$M/lastsync.txt"

Timestamps

Each tree carries a lastsync.txt at its root with the UTC time of the last completed stage 2. Monitor that file rather than the HTTP Last-Modified of a directory listing — the latter changes on any touch.