mirror of
https://gitea.com/gitea/tea.git
synced 2026-09-10 07:26:33 -04:00
ci(goreleaser): mirror release artifacts to Cloudflare R2
Adds a parallel upload of every release artifact to a Cloudflare R2 bucket, alongside the existing AWS S3 upload, for the duration of the migration away from S3. Once the migration is complete, the `blobs:` block and the AWS_* secrets can be dropped independently of this change. A second `blobs:` entry can't be used for R2 because the blob pipe only authenticates from the global AWS_* environment, with no per-entry credentials. `publishers:` is used instead: it supports per-entry `env:`, so scripts/upload-r2.sh (invoked once per artifact) can be handed its own R2_* credentials without touching the S3 ones. A `check R2 configuration` step is added early in both release workflows, before any build or publish work happens: custom publishers run as the very last step of goreleaser's publish pipeline, so without a preflight check a missing R2_* secret would only be discovered after the Gitea release has already been created and every artifact already uploaded to S3. Unlike the equivalent change in the gitea.com/gitea/runner repository, the R2 publisher here also sets `signature: true` in addition to `checksum: true`, so that checksums.txt.sig (produced by the existing `signs:` block) is mirrored to R2 as well; without it, artifacts downloaded from the R2 mirror would have no signature to verify against. Requires 4 new secrets to be configured on the repository: R2_ENDPOINT, R2_BUCKET, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY. --- Generated by Codet
This commit is contained in:
parent
bade1de792
commit
2492228946
|
|
@ -12,6 +12,18 @@ jobs:
|
|||
with:
|
||||
fetch-depth: 0
|
||||
- run: git fetch --force --tags
|
||||
# Custom publishers (the R2 mirror below) run as the very last
|
||||
# step of goreleaser's publish pipeline, after the Gitea release
|
||||
# has already been created and every artifact already uploaded
|
||||
# to S3. Fail here instead, before anything is built or
|
||||
# published, if the R2 secrets are missing.
|
||||
- name: check R2 configuration
|
||||
run: sh scripts/upload-r2.sh --check-config
|
||||
env:
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
- uses: actions/setup-go@v6
|
||||
with:
|
||||
go-version-file: "go.mod"
|
||||
|
|
@ -38,6 +50,10 @@ jobs:
|
|||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
S3_REGION: ${{ secrets.AWS_REGION }}
|
||||
S3_BUCKET: ${{ secrets.AWS_BUCKET }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
GORELEASER_FORCE_TOKEN: 'gitea'
|
||||
GPGSIGN_PASSPHRASE: ${{ secrets.GPGSIGN_PASSPHRASE }}
|
||||
GPG_FINGERPRINT: ${{ steps.import_gpg.outputs.fingerprint }}
|
||||
|
|
|
|||
|
|
@ -13,6 +13,18 @@ jobs:
|
|||
with:
|
||||
fetch-depth: 0
|
||||
- run: git fetch --force --tags
|
||||
# Custom publishers (the R2 mirror below) run as the very last
|
||||
# step of goreleaser's publish pipeline, after the Gitea release
|
||||
# has already been created and every artifact already uploaded
|
||||
# to S3. Fail here instead, before anything is built or
|
||||
# published, if the R2 secrets are missing.
|
||||
- name: check R2 configuration
|
||||
run: sh scripts/upload-r2.sh --check-config
|
||||
env:
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
- uses: actions/setup-go@v6
|
||||
with:
|
||||
go-version-file: 'go.mod'
|
||||
|
|
@ -39,6 +51,10 @@ jobs:
|
|||
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
|
||||
S3_REGION: ${{ secrets.AWS_REGION }}
|
||||
S3_BUCKET: ${{ secrets.AWS_BUCKET }}
|
||||
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
|
||||
R2_BUCKET: ${{ secrets.R2_BUCKET }}
|
||||
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
|
||||
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
|
||||
GORELEASER_FORCE_TOKEN: 'gitea'
|
||||
GPGSIGN_PASSPHRASE: ${{ secrets.GPGSIGN_PASSPHRASE }}
|
||||
GPG_FINGERPRINT: ${{ steps.import_gpg.outputs.fingerprint }}
|
||||
|
|
|
|||
|
|
@ -86,6 +86,45 @@ blobs:
|
|||
- glob: ./**.xz
|
||||
- glob: ./**.sha256
|
||||
|
||||
# Mirrors the S3 `blobs:` upload above into Cloudflare R2 during the
|
||||
# parallel S3+R2 period (S3 will be removed once migration completes).
|
||||
# A second `blobs:` entry is impossible here since the blob pipe
|
||||
# authenticates from the global AWS_* env with no per-entry
|
||||
# credentials; `publishers:` supports per-entry `env:` instead, so
|
||||
# it's used to invoke scripts/upload-r2.sh once per artifact. Custom
|
||||
# publishers inherit almost nothing from the environment, hence the
|
||||
# explicit R2_* forwarding below.
|
||||
#
|
||||
# This publisher fires more than once per distinct key because
|
||||
# goreleaser's release pipe already registers `release.extra_files`
|
||||
# (./**.xz and ./**.xz.sha256, see the `release:` block below) as
|
||||
# UploadableFile artifacts, and `internal/exec`'s filterArtifacts
|
||||
# appends this block's own extra_files with no de-duplication. It
|
||||
# can't be globbed away, since gobwas/glob (via goreleaser/fileglob)
|
||||
# has no substring-exclusion matcher. It's harmless: PUT is
|
||||
# idempotent, and the `./**.xz` glob below is kept deliberately so
|
||||
# this publisher declares its own complete file set rather than
|
||||
# implicitly depending on the `release:` block's globs.
|
||||
#
|
||||
# checksum: true mirrors goreleaser's generated checksums.txt;
|
||||
# signature: true additionally mirrors checksums.txt.sig, which the
|
||||
# `signs:` block below produces by GPG-signing that checksum file.
|
||||
# Without signature: true, artifacts downloaded from the R2 mirror
|
||||
# would have no signature file to verify against.
|
||||
publishers:
|
||||
- name: cloudflare-r2
|
||||
checksum: true
|
||||
signature: true
|
||||
extra_files:
|
||||
- glob: ./**.xz
|
||||
- glob: ./**.sha256
|
||||
cmd: sh scripts/upload-r2.sh {{ abs .ArtifactPath }} tea/{{ .Version }}/{{ .ArtifactName }}
|
||||
env:
|
||||
- R2_ENDPOINT={{ index .Env "R2_ENDPOINT" }}
|
||||
- R2_BUCKET={{ index .Env "R2_BUCKET" }}
|
||||
- R2_ACCESS_KEY_ID={{ index .Env "R2_ACCESS_KEY_ID" }}
|
||||
- R2_SECRET_ACCESS_KEY={{ index .Env "R2_SECRET_ACCESS_KEY" }}
|
||||
|
||||
archives:
|
||||
- format: binary
|
||||
name_template: "{{ .Binary }}"
|
||||
|
|
|
|||
112
scripts/upload-r2.sh
Executable file
112
scripts/upload-r2.sh
Executable file
|
|
@ -0,0 +1,112 @@
|
|||
#!/bin/sh
|
||||
# Copyright 2026 The Gitea Authors. All rights reserved.
|
||||
# SPDX-License-Identifier: MIT
|
||||
#
|
||||
# upload-r2.sh uploads a single local file to a single object key in a
|
||||
# Cloudflare R2 bucket, using curl's built-in AWS SigV4 signer (R2 is
|
||||
# S3-API compatible).
|
||||
#
|
||||
# This is the R2 half of the release process's parallel S3+R2 upload
|
||||
# period: goreleaser's `blobs:` pipe still uploads every release
|
||||
# artifact to AWS S3, and this script is invoked once per artifact
|
||||
# (via a goreleaser `publishers:` entry) to mirror the same artifact
|
||||
# into R2. Once the migration away from S3 is complete, the `blobs:`
|
||||
# block and the AWS_* secrets can be dropped without touching this
|
||||
# script.
|
||||
#
|
||||
# Usage:
|
||||
# upload-r2.sh <local-file> <remote-key>
|
||||
# upload-r2.sh --check-config
|
||||
#
|
||||
# The second form only validates that the required environment
|
||||
# variables below are set (it does not touch the network or the
|
||||
# filesystem beyond that), and is meant to be run as an early
|
||||
# preflight step in CI: goreleaser custom publishers run as the very
|
||||
# last step of the publish pipeline, so without a preflight check a
|
||||
# missing R2_* secret would only be discovered after the Gitea release
|
||||
# has already been created and every artifact already uploaded to S3.
|
||||
#
|
||||
# Required environment variables:
|
||||
# R2_ENDPOINT Base URL of the R2 endpoint, e.g.
|
||||
# https://<account>.r2.cloudflarestorage.com
|
||||
# R2_BUCKET Destination bucket name.
|
||||
# R2_ACCESS_KEY_ID R2 access key id.
|
||||
# R2_SECRET_ACCESS_KEY R2 secret access key.
|
||||
|
||||
set -eu
|
||||
|
||||
# check_env validates that all required R2_* environment variables are
|
||||
# set and non-empty, printing a single "missing required environment
|
||||
# variable(s): ..." message and exiting non-zero otherwise. Used by
|
||||
# both the normal upload mode and --check-config, so the validation
|
||||
# logic only exists in one place.
|
||||
check_env() {
|
||||
missing=""
|
||||
|
||||
if [ -z "${R2_ENDPOINT:-}" ]; then
|
||||
missing="$missing R2_ENDPOINT"
|
||||
fi
|
||||
if [ -z "${R2_BUCKET:-}" ]; then
|
||||
missing="$missing R2_BUCKET"
|
||||
fi
|
||||
if [ -z "${R2_ACCESS_KEY_ID:-}" ]; then
|
||||
missing="$missing R2_ACCESS_KEY_ID"
|
||||
fi
|
||||
if [ -z "${R2_SECRET_ACCESS_KEY:-}" ]; then
|
||||
missing="$missing R2_SECRET_ACCESS_KEY"
|
||||
fi
|
||||
|
||||
if [ -n "$missing" ]; then
|
||||
echo "upload-r2.sh: missing required environment variable(s):$missing" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
if [ "$#" -eq 1 ] && [ "$1" = "--check-config" ]; then
|
||||
check_env
|
||||
echo "upload-r2.sh: R2 configuration OK"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
if [ "$#" -ne 2 ]; then
|
||||
echo "usage: upload-r2.sh <local-file> <remote-key>" >&2
|
||||
echo " upload-r2.sh --check-config" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
local_file="$1"
|
||||
remote_key="$2"
|
||||
|
||||
if [ ! -f "$local_file" ]; then
|
||||
echo "upload-r2.sh: local file not found: $local_file" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
check_env
|
||||
|
||||
# Strip a single trailing slash from the endpoint, if present, so that
|
||||
# building the path-style URL below never produces a double slash.
|
||||
endpoint="${R2_ENDPOINT%/}"
|
||||
url="$endpoint/$R2_BUCKET/$remote_key"
|
||||
|
||||
# Credentials are passed to curl through a config file read from
|
||||
# stdin rather than as a command-line argument, so they never show up
|
||||
# in `ps` output.
|
||||
#
|
||||
# --fail-with-body (instead of plain --fail) still exits non-zero on
|
||||
# HTTP errors, but also prints R2's XML error body, which is where the
|
||||
# actual error code lives (SignatureDoesNotMatch, NoSuchBucket,
|
||||
# AccessDenied, ...); with plain --fail that body is discarded and the
|
||||
# failure is silent. --retry 3 (without --retry-all-errors) still
|
||||
# retries the transient cases (5xx, 408, 429, connection failures);
|
||||
# --retry-all-errors would additionally retry permanent 4xx responses
|
||||
# three times with backoff, which only delays an inevitable failure.
|
||||
printf 'user = "%s:%s"\n' "$R2_ACCESS_KEY_ID" "$R2_SECRET_ACCESS_KEY" | curl \
|
||||
--config - \
|
||||
--fail-with-body \
|
||||
--silent \
|
||||
--show-error \
|
||||
--retry 3 \
|
||||
--aws-sigv4 "aws:amz:auto:s3" \
|
||||
--upload-file "$local_file" \
|
||||
"$url"
|
||||
Loading…
Reference in a new issue