Initial commit: merged geosite build pipeline
This commit is contained in:
Payroll3042
+12
@@ -0,0 +1,12 @@
|
|||||||
|
FROM golang:1.25
|
||||||
|
|
||||||
|
# go.mod for v2fly/domain-list-community requires go >= 1.25.12.
|
||||||
|
# Using this version directly avoids relying on Go's automatic toolchain
|
||||||
|
# download at build time.
|
||||||
|
|
||||||
|
WORKDIR /work
|
||||||
|
COPY build.sh /work/build.sh
|
||||||
|
COPY data-custom /work/data-custom
|
||||||
|
RUN chmod +x /work/build.sh
|
||||||
|
|
||||||
|
ENTRYPOINT ["/work/build.sh"]
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
FROM golang:1.25
|
||||||
|
|
||||||
|
# git comes with the full (non-slim) golang image; cron does not.
|
||||||
|
RUN apt-get update -qq \
|
||||||
|
&& apt-get install -y -qq --no-install-recommends cron \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
|
WORKDIR /app
|
||||||
|
COPY auto-build-push.sh /app/auto-build-push.sh
|
||||||
|
COPY data-custom /app/data-custom
|
||||||
|
RUN chmod +x /app/auto-build-push.sh
|
||||||
|
|
||||||
|
# Default schedule: 03:10 UTC daily, same time hydraponique rebuilds their
|
||||||
|
# own source repo, so this picks up same-day changes. Override by mounting
|
||||||
|
# a different crontab or changing CRON_SCHEDULE and regenerating (see README).
|
||||||
|
RUN echo "10 3 * * * root /app/auto-build-push.sh >> /proc/1/fd/1 2>> /proc/1/fd/2" > /etc/cron.d/geosite-build \
|
||||||
|
&& chmod 0644 /etc/cron.d/geosite-build \
|
||||||
|
&& crontab /etc/cron.d/geosite-build
|
||||||
|
|
||||||
|
# Run once immediately on container start (so you don't wait until 03:10
|
||||||
|
# UTC to see it work the first time), then hand off to cron in foreground.
|
||||||
|
CMD /app/auto-build-push.sh; cron -f
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
# Merged geosite.dat build
|
||||||
|
|
||||||
|
Combines:
|
||||||
|
- **v2fly/domain-list-community** (base) — ~1520 stock categories, already
|
||||||
|
covers openai, anthropic, netflix, whatsapp, discord, signal, paypal,
|
||||||
|
linkedin, reddit, spotify, facebook, instagram, twitter, roblox, binance,
|
||||||
|
bbc, archive, viber, rutracker, speedtest, notion, figma, jetbrains, adobe,
|
||||||
|
intel, autodesk, cursor, vercel, github, google, youtube, telegram.
|
||||||
|
- **hydraponique/roscomvpn-geosite** (overlay) — RU-specific categories:
|
||||||
|
category-ru, whitelist, category-geoblock-ru, apple, google-play,
|
||||||
|
google-deepmind, microsoft, epicgames, riot, escapefromtarkov, faceit,
|
||||||
|
twitch, twitch-ads, pinterest, steam, category-ads, win-spy, private,
|
||||||
|
torrent, github, telegram, youtube (last three merge/union with v2fly's
|
||||||
|
stock versions of the same name — this is intended compiler behavior,
|
||||||
|
not a conflict).
|
||||||
|
- **data-custom/** — the only 4 services confirmed missing from both:
|
||||||
|
replicate, avast, cointelegraph, snapchat.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Go >= 1.25.12 (v2fly's go.mod hard-requires this)
|
||||||
|
- git
|
||||||
|
|
||||||
|
## Manual one-off build
|
||||||
|
|
||||||
|
Docker (recommended, no local Go/git needed):
|
||||||
|
```
|
||||||
|
docker build -t geosite-builder .
|
||||||
|
docker run --rm -v "$(pwd)/output:/work/output" geosite-builder
|
||||||
|
```
|
||||||
|
|
||||||
|
Or locally if you already have Go 1.25+ and git:
|
||||||
|
```
|
||||||
|
./build.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Output: `output/geosite.dat`
|
||||||
|
|
||||||
|
## Deploying as a scheduled service (Gitea + Portainer)
|
||||||
|
|
||||||
|
This runs the build daily and pushes the result straight to your own
|
||||||
|
Gitea repo, so it's served the exact same way `rules.yaml` already is —
|
||||||
|
`https://git.5838.ru/<owner>/<repo>/raw/branch/main/geosite.dat`.
|
||||||
|
|
||||||
|
### 1. Create the target repo on git.5838.ru
|
||||||
|
|
||||||
|
Create an empty repo (e.g. `geosite-build`) under your account. It can
|
||||||
|
start completely empty — the container's first run will commit the
|
||||||
|
initial `geosite.dat`.
|
||||||
|
|
||||||
|
### 2. Create a Gitea access token
|
||||||
|
|
||||||
|
Gitea → Settings → Applications → Generate New Token, scope: repo
|
||||||
|
read/write. Copy it once, you won't see it again.
|
||||||
|
|
||||||
|
### 3. Push these build files to that repo yourself
|
||||||
|
|
||||||
|
I can't reach git.5838.ru from here, so this part's on you:
|
||||||
|
```
|
||||||
|
cd geosite-build
|
||||||
|
git init
|
||||||
|
git remote add origin https://git.5838.ru/<owner>/geosite-build.git
|
||||||
|
git add .
|
||||||
|
git commit -m "Initial commit: merged geosite build pipeline"
|
||||||
|
git push -u origin main
|
||||||
|
```
|
||||||
|
(If you'd rather keep the pipeline files in a separate repo from the
|
||||||
|
generated `geosite.dat` output, that's fine too — just point `GITEA_REPO`
|
||||||
|
in the compose file at wherever the *output* should land, which can be
|
||||||
|
a different repo than the one holding this Dockerfile/scripts.)
|
||||||
|
|
||||||
|
### 4. Deploy in Portainer
|
||||||
|
|
||||||
|
- Portainer → Stacks → Add stack
|
||||||
|
- Name it (e.g. `geosite-build`)
|
||||||
|
- Paste the contents of `docker-compose.yml`
|
||||||
|
- Under **Environment variables**, add `GITEA_TOKEN` with the token from
|
||||||
|
step 2 — don't put it directly in the compose file, use Portainer's env
|
||||||
|
var field so it isn't stored in plain text in the stack definition
|
||||||
|
- Deploy
|
||||||
|
|
||||||
|
The container runs the build immediately on startup, then again daily at
|
||||||
|
03:10 UTC (same time hydraponique rebuilds their own source, so you pick
|
||||||
|
up same-day upstream changes). Check Portainer's container logs to
|
||||||
|
confirm the first run pushed successfully.
|
||||||
|
|
||||||
|
### 5. Point your configs at the new file
|
||||||
|
|
||||||
|
Once you've confirmed a successful push, update:
|
||||||
|
- Happ's `Geositeurl` in `RU-routing.json`
|
||||||
|
- Clash Verge's geo-data source, if you want the same merged categories there
|
||||||
|
|
||||||
|
to `https://git.5838.ru/<owner>/<repo>/raw/branch/main/geosite.dat`.
|
||||||
|
|
||||||
|
## Adding more custom domains later
|
||||||
|
|
||||||
|
If you find another service missing after the fact, add a new file under
|
||||||
|
`data-custom/` (one domain per line, `domain:example.com` for subdomain
|
||||||
|
matching, `full:example.com` for exact matches — same format v2fly itself
|
||||||
|
uses), commit it to whichever repo holds these pipeline files, and the
|
||||||
|
next scheduled run picks it up automatically.
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# auto-build-push.sh — builds the merged geosite.dat and pushes it to a
|
||||||
|
# Gitea repo, so it can be served the same way rules.yaml already is
|
||||||
|
# (https://<gitea>/<owner>/<repo>/raw/branch/<branch>/geosite.dat).
|
||||||
|
#
|
||||||
|
# Required environment variables (set these in the Portainer stack, not
|
||||||
|
# baked into the image):
|
||||||
|
# GITEA_URL e.g. https://git.5838.ru
|
||||||
|
# GITEA_REPO e.g. Payroll3042/geosite-build (must already exist, empty is fine)
|
||||||
|
# GITEA_TOKEN a Gitea personal access token with repo write access
|
||||||
|
# GIT_USER_NAME e.g. geosite-bot
|
||||||
|
# GIT_USER_EMAIL e.g. geosite-bot@5838.ru
|
||||||
|
# Optional:
|
||||||
|
# GITEA_BRANCH defaults to main
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
: "${GITEA_URL:?GITEA_URL is required}"
|
||||||
|
: "${GITEA_REPO:?GITEA_REPO is required, e.g. Payroll3042/geosite-build}"
|
||||||
|
: "${GITEA_TOKEN:?GITEA_TOKEN is required}"
|
||||||
|
: "${GIT_USER_NAME:?GIT_USER_NAME is required}"
|
||||||
|
: "${GIT_USER_EMAIL:?GIT_USER_EMAIL is required}"
|
||||||
|
GITEA_BRANCH="${GITEA_BRANCH:-main}"
|
||||||
|
|
||||||
|
WORKDIR=/tmp/build
|
||||||
|
OUTDIR=/tmp/output
|
||||||
|
CUSTOM_DIR=/app/data-custom
|
||||||
|
|
||||||
|
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) ==> Starting build"
|
||||||
|
|
||||||
|
rm -rf "$WORKDIR" "$OUTDIR"
|
||||||
|
mkdir -p "$WORKDIR" "$OUTDIR"
|
||||||
|
cd "$WORKDIR"
|
||||||
|
|
||||||
|
echo "==> Cloning v2fly/domain-list-community"
|
||||||
|
git clone --depth 1 -q https://github.com/v2fly/domain-list-community.git v2fly
|
||||||
|
|
||||||
|
echo "==> Cloning hydraponique/roscomvpn-geosite"
|
||||||
|
git clone --depth 1 -q https://github.com/hydraponique/roscomvpn-geosite.git roscomvpn-src
|
||||||
|
|
||||||
|
echo "==> Merging data directories"
|
||||||
|
mkdir -p v2fly/data/roscomvpn v2fly/data/custom
|
||||||
|
cp -r roscomvpn-src/data/. v2fly/data/roscomvpn/
|
||||||
|
if [ -d "$CUSTOM_DIR" ]; then
|
||||||
|
cp -r "$CUSTOM_DIR/." v2fly/data/custom/
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Compiling merged geosite.dat"
|
||||||
|
cd v2fly
|
||||||
|
go run main.go
|
||||||
|
|
||||||
|
if [ ! -f dlc.dat ]; then
|
||||||
|
echo "ERROR: build did not produce dlc.dat"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cp dlc.dat "$OUTDIR/geosite.dat"
|
||||||
|
echo "==> Build produced $(wc -c < "$OUTDIR/geosite.dat") bytes"
|
||||||
|
|
||||||
|
echo "==> Cloning target Gitea repo"
|
||||||
|
GITEA_HOST="$(echo "$GITEA_URL" | sed -e 's~https\?://~~')"
|
||||||
|
PUSH_URL="https://${GITEA_TOKEN}@${GITEA_HOST}/${GITEA_REPO}.git"
|
||||||
|
|
||||||
|
cd /tmp
|
||||||
|
rm -rf repo
|
||||||
|
git clone -q --branch "$GITEA_BRANCH" "$PUSH_URL" repo || git clone -q "$PUSH_URL" repo
|
||||||
|
cd repo
|
||||||
|
|
||||||
|
git config user.name "$GIT_USER_NAME"
|
||||||
|
git config user.email "$GIT_USER_EMAIL"
|
||||||
|
|
||||||
|
cp "$OUTDIR/geosite.dat" ./geosite.dat
|
||||||
|
|
||||||
|
if git diff --quiet -- geosite.dat 2>/dev/null && [ -f geosite.dat ]; then
|
||||||
|
echo "==> No changes since last build, skipping commit"
|
||||||
|
else
|
||||||
|
git add geosite.dat
|
||||||
|
git commit -q -m "Auto-build $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||||
|
git push -q origin "HEAD:${GITEA_BRANCH}"
|
||||||
|
echo "==> Pushed updated geosite.dat to ${GITEA_URL}/${GITEA_REPO}"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) ==> Done"
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# build.sh — merges v2fly/domain-list-community, Loyalsoldier's overlay,
|
||||||
|
# hydraponique/roscomvpn-geosite, and your own custom data into one geosite.dat
|
||||||
|
#
|
||||||
|
# Run inside the provided Docker image (recommended, no local Go/git needed):
|
||||||
|
# docker build -t geosite-builder .
|
||||||
|
# docker run --rm -v "$(pwd)/output:/work/output" geosite-builder
|
||||||
|
#
|
||||||
|
# Or locally if you already have Go + git installed:
|
||||||
|
# ./build.sh
|
||||||
|
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
WORKDIR="$(pwd)/work"
|
||||||
|
OUTDIR="$(pwd)/output"
|
||||||
|
CUSTOM_DIR="$(pwd)/data-custom"
|
||||||
|
|
||||||
|
echo "==> Preparing workspace"
|
||||||
|
rm -rf "$WORKDIR"
|
||||||
|
mkdir -p "$WORKDIR" "$OUTDIR"
|
||||||
|
cd "$WORKDIR"
|
||||||
|
|
||||||
|
echo "==> Cloning v2fly/domain-list-community (base compiler + ~1500 stock categories -"
|
||||||
|
echo " already covers openai, anthropic, netflix, whatsapp, discord, signal, paypal,"
|
||||||
|
echo " linkedin, reddit, spotify, facebook, instagram, twitter, roblox, binance, bbc,"
|
||||||
|
echo " archive, viber, rutracker, speedtest, notion, figma, jetbrains, adobe, intel,"
|
||||||
|
echo " autodesk, cursor, vercel, github, google, youtube, telegram, and more)"
|
||||||
|
git clone --depth 1 https://github.com/v2fly/domain-list-community.git v2fly
|
||||||
|
|
||||||
|
echo "==> Cloning hydraponique/roscomvpn-geosite (RU-specific categories)"
|
||||||
|
git clone --depth 1 https://github.com/hydraponique/roscomvpn-geosite.git roscomvpn-src
|
||||||
|
|
||||||
|
echo "==> Merging data directories"
|
||||||
|
mkdir -p v2fly/data/roscomvpn v2fly/data/custom
|
||||||
|
|
||||||
|
cp -r roscomvpn-src/data/. v2fly/data/roscomvpn/
|
||||||
|
|
||||||
|
# Only genuinely missing from v2fly stock (verified by testing): replicate,
|
||||||
|
# avast, cointelegraph, snapchat. Everything else is already covered upstream.
|
||||||
|
if [ -d "$CUSTOM_DIR" ]; then
|
||||||
|
cp -r "$CUSTOM_DIR/." v2fly/data/custom/
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "==> Building merged geosite.dat"
|
||||||
|
cd v2fly
|
||||||
|
go run main.go
|
||||||
|
|
||||||
|
if [ ! -f dlc.dat ]; then
|
||||||
|
echo "ERROR: build did not produce dlc.dat — check output above for errors"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
cp dlc.dat "$OUTDIR/geosite.dat"
|
||||||
|
|
||||||
|
echo "==> Done. Output: $OUTDIR/geosite.dat"
|
||||||
|
ls -lh "$OUTDIR/geosite.dat"
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
domain:cointelegraph.com
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
services:
|
||||||
|
geosite-build:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile.scheduled
|
||||||
|
container_name: geosite-build
|
||||||
|
restart: unless-stopped
|
||||||
|
environment:
|
||||||
|
GITEA_URL: "https://git.5838.ru"
|
||||||
|
GITEA_REPO: "Payroll3042/geosite-build" # change if your repo name differs
|
||||||
|
GITEA_TOKEN: "${GITEA_TOKEN}" # set in Portainer's stack env vars, not here
|
||||||
|
GIT_USER_NAME: "geosite-bot"
|
||||||
|
GIT_USER_EMAIL: "geosite-bot@5838.ru"
|
||||||
|
GITEA_BRANCH: "main"
|
||||||
Reference in New Issue
Block a user