Initial commit: merged geosite build pipeline
This commit is contained in:
Payroll3042
+12
@@ -0,0 +1,12 @@
|
||||
FROM golang:1.25
|
||||
|
||||
# go.mod for v2fly/domain-list-community requires go >= 1.25.12.
|
||||
# Using this version directly avoids relying on Go's automatic toolchain
|
||||
# download at build time.
|
||||
|
||||
WORKDIR /work
|
||||
COPY build.sh /work/build.sh
|
||||
COPY data-custom /work/data-custom
|
||||
RUN chmod +x /work/build.sh
|
||||
|
||||
ENTRYPOINT ["/work/build.sh"]
|
||||
@@ -0,0 +1,22 @@
|
||||
FROM golang:1.25
|
||||
|
||||
# git comes with the full (non-slim) golang image; cron does not.
|
||||
RUN apt-get update -qq \
|
||||
&& apt-get install -y -qq --no-install-recommends cron \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
COPY auto-build-push.sh /app/auto-build-push.sh
|
||||
COPY data-custom /app/data-custom
|
||||
RUN chmod +x /app/auto-build-push.sh
|
||||
|
||||
# Default schedule: 03:10 UTC daily, same time hydraponique rebuilds their
|
||||
# own source repo, so this picks up same-day changes. Override by mounting
|
||||
# a different crontab or changing CRON_SCHEDULE and regenerating (see README).
|
||||
RUN echo "10 3 * * * root /app/auto-build-push.sh >> /proc/1/fd/1 2>> /proc/1/fd/2" > /etc/cron.d/geosite-build \
|
||||
&& chmod 0644 /etc/cron.d/geosite-build \
|
||||
&& crontab /etc/cron.d/geosite-build
|
||||
|
||||
# Run once immediately on container start (so you don't wait until 03:10
|
||||
# UTC to see it work the first time), then hand off to cron in foreground.
|
||||
CMD /app/auto-build-push.sh; cron -f
|
||||
@@ -0,0 +1,101 @@
|
||||
# Merged geosite.dat build
|
||||
|
||||
Combines:
|
||||
- **v2fly/domain-list-community** (base) — ~1520 stock categories, already
|
||||
covers openai, anthropic, netflix, whatsapp, discord, signal, paypal,
|
||||
linkedin, reddit, spotify, facebook, instagram, twitter, roblox, binance,
|
||||
bbc, archive, viber, rutracker, speedtest, notion, figma, jetbrains, adobe,
|
||||
intel, autodesk, cursor, vercel, github, google, youtube, telegram.
|
||||
- **hydraponique/roscomvpn-geosite** (overlay) — RU-specific categories:
|
||||
category-ru, whitelist, category-geoblock-ru, apple, google-play,
|
||||
google-deepmind, microsoft, epicgames, riot, escapefromtarkov, faceit,
|
||||
twitch, twitch-ads, pinterest, steam, category-ads, win-spy, private,
|
||||
torrent, github, telegram, youtube (last three merge/union with v2fly's
|
||||
stock versions of the same name — this is intended compiler behavior,
|
||||
not a conflict).
|
||||
- **data-custom/** — the only 4 services confirmed missing from both:
|
||||
replicate, avast, cointelegraph, snapchat.
|
||||
|
||||
## Requirements
|
||||
|
||||
- Go >= 1.25.12 (v2fly's go.mod hard-requires this)
|
||||
- git
|
||||
|
||||
## Manual one-off build
|
||||
|
||||
Docker (recommended, no local Go/git needed):
|
||||
```
|
||||
docker build -t geosite-builder .
|
||||
docker run --rm -v "$(pwd)/output:/work/output" geosite-builder
|
||||
```
|
||||
|
||||
Or locally if you already have Go 1.25+ and git:
|
||||
```
|
||||
./build.sh
|
||||
```
|
||||
|
||||
Output: `output/geosite.dat`
|
||||
|
||||
## Deploying as a scheduled service (Gitea + Portainer)
|
||||
|
||||
This runs the build daily and pushes the result straight to your own
|
||||
Gitea repo, so it's served the exact same way `rules.yaml` already is —
|
||||
`https://git.5838.ru/<owner>/<repo>/raw/branch/main/geosite.dat`.
|
||||
|
||||
### 1. Create the target repo on git.5838.ru
|
||||
|
||||
Create an empty repo (e.g. `geosite-build`) under your account. It can
|
||||
start completely empty — the container's first run will commit the
|
||||
initial `geosite.dat`.
|
||||
|
||||
### 2. Create a Gitea access token
|
||||
|
||||
Gitea → Settings → Applications → Generate New Token, scope: repo
|
||||
read/write. Copy it once, you won't see it again.
|
||||
|
||||
### 3. Push these build files to that repo yourself
|
||||
|
||||
I can't reach git.5838.ru from here, so this part's on you:
|
||||
```
|
||||
cd geosite-build
|
||||
git init
|
||||
git remote add origin https://git.5838.ru/<owner>/geosite-build.git
|
||||
git add .
|
||||
git commit -m "Initial commit: merged geosite build pipeline"
|
||||
git push -u origin main
|
||||
```
|
||||
(If you'd rather keep the pipeline files in a separate repo from the
|
||||
generated `geosite.dat` output, that's fine too — just point `GITEA_REPO`
|
||||
in the compose file at wherever the *output* should land, which can be
|
||||
a different repo than the one holding this Dockerfile/scripts.)
|
||||
|
||||
### 4. Deploy in Portainer
|
||||
|
||||
- Portainer → Stacks → Add stack
|
||||
- Name it (e.g. `geosite-build`)
|
||||
- Paste the contents of `docker-compose.yml`
|
||||
- Under **Environment variables**, add `GITEA_TOKEN` with the token from
|
||||
step 2 — don't put it directly in the compose file, use Portainer's env
|
||||
var field so it isn't stored in plain text in the stack definition
|
||||
- Deploy
|
||||
|
||||
The container runs the build immediately on startup, then again daily at
|
||||
03:10 UTC (same time hydraponique rebuilds their own source, so you pick
|
||||
up same-day upstream changes). Check Portainer's container logs to
|
||||
confirm the first run pushed successfully.
|
||||
|
||||
### 5. Point your configs at the new file
|
||||
|
||||
Once you've confirmed a successful push, update:
|
||||
- Happ's `Geositeurl` in `RU-routing.json`
|
||||
- Clash Verge's geo-data source, if you want the same merged categories there
|
||||
|
||||
to `https://git.5838.ru/<owner>/<repo>/raw/branch/main/geosite.dat`.
|
||||
|
||||
## Adding more custom domains later
|
||||
|
||||
If you find another service missing after the fact, add a new file under
|
||||
`data-custom/` (one domain per line, `domain:example.com` for subdomain
|
||||
matching, `full:example.com` for exact matches — same format v2fly itself
|
||||
uses), commit it to whichever repo holds these pipeline files, and the
|
||||
next scheduled run picks it up automatically.
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/bin/sh
|
||||
# auto-build-push.sh — builds the merged geosite.dat and pushes it to a
|
||||
# Gitea repo, so it can be served the same way rules.yaml already is
|
||||
# (https://<gitea>/<owner>/<repo>/raw/branch/<branch>/geosite.dat).
|
||||
#
|
||||
# Required environment variables (set these in the Portainer stack, not
|
||||
# baked into the image):
|
||||
# GITEA_URL e.g. https://git.5838.ru
|
||||
# GITEA_REPO e.g. Payroll3042/geosite-build (must already exist, empty is fine)
|
||||
# GITEA_TOKEN a Gitea personal access token with repo write access
|
||||
# GIT_USER_NAME e.g. geosite-bot
|
||||
# GIT_USER_EMAIL e.g. geosite-bot@5838.ru
|
||||
# Optional:
|
||||
# GITEA_BRANCH defaults to main
|
||||
|
||||
set -eu
|
||||
|
||||
: "${GITEA_URL:?GITEA_URL is required}"
|
||||
: "${GITEA_REPO:?GITEA_REPO is required, e.g. Payroll3042/geosite-build}"
|
||||
: "${GITEA_TOKEN:?GITEA_TOKEN is required}"
|
||||
: "${GIT_USER_NAME:?GIT_USER_NAME is required}"
|
||||
: "${GIT_USER_EMAIL:?GIT_USER_EMAIL is required}"
|
||||
GITEA_BRANCH="${GITEA_BRANCH:-main}"
|
||||
|
||||
WORKDIR=/tmp/build
|
||||
OUTDIR=/tmp/output
|
||||
CUSTOM_DIR=/app/data-custom
|
||||
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) ==> Starting build"
|
||||
|
||||
rm -rf "$WORKDIR" "$OUTDIR"
|
||||
mkdir -p "$WORKDIR" "$OUTDIR"
|
||||
cd "$WORKDIR"
|
||||
|
||||
echo "==> Cloning v2fly/domain-list-community"
|
||||
git clone --depth 1 -q https://github.com/v2fly/domain-list-community.git v2fly
|
||||
|
||||
echo "==> Cloning hydraponique/roscomvpn-geosite"
|
||||
git clone --depth 1 -q https://github.com/hydraponique/roscomvpn-geosite.git roscomvpn-src
|
||||
|
||||
echo "==> Merging data directories"
|
||||
mkdir -p v2fly/data/roscomvpn v2fly/data/custom
|
||||
cp -r roscomvpn-src/data/. v2fly/data/roscomvpn/
|
||||
if [ -d "$CUSTOM_DIR" ]; then
|
||||
cp -r "$CUSTOM_DIR/." v2fly/data/custom/
|
||||
fi
|
||||
|
||||
echo "==> Compiling merged geosite.dat"
|
||||
cd v2fly
|
||||
go run main.go
|
||||
|
||||
if [ ! -f dlc.dat ]; then
|
||||
echo "ERROR: build did not produce dlc.dat"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cp dlc.dat "$OUTDIR/geosite.dat"
|
||||
echo "==> Build produced $(wc -c < "$OUTDIR/geosite.dat") bytes"
|
||||
|
||||
echo "==> Cloning target Gitea repo"
|
||||
GITEA_HOST="$(echo "$GITEA_URL" | sed -e 's~https\?://~~')"
|
||||
PUSH_URL="https://${GITEA_TOKEN}@${GITEA_HOST}/${GITEA_REPO}.git"
|
||||
|
||||
cd /tmp
|
||||
rm -rf repo
|
||||
git clone -q --branch "$GITEA_BRANCH" "$PUSH_URL" repo || git clone -q "$PUSH_URL" repo
|
||||
cd repo
|
||||
|
||||
git config user.name "$GIT_USER_NAME"
|
||||
git config user.email "$GIT_USER_EMAIL"
|
||||
|
||||
cp "$OUTDIR/geosite.dat" ./geosite.dat
|
||||
|
||||
if git diff --quiet -- geosite.dat 2>/dev/null && [ -f geosite.dat ]; then
|
||||
echo "==> No changes since last build, skipping commit"
|
||||
else
|
||||
git add geosite.dat
|
||||
git commit -q -m "Auto-build $(date -u +%Y-%m-%dT%H:%M:%SZ)"
|
||||
git push -q origin "HEAD:${GITEA_BRANCH}"
|
||||
echo "==> Pushed updated geosite.dat to ${GITEA_URL}/${GITEA_REPO}"
|
||||
fi
|
||||
|
||||
echo "$(date -u +%Y-%m-%dT%H:%M:%SZ) ==> Done"
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/bin/sh
|
||||
# build.sh — merges v2fly/domain-list-community, Loyalsoldier's overlay,
|
||||
# hydraponique/roscomvpn-geosite, and your own custom data into one geosite.dat
|
||||
#
|
||||
# Run inside the provided Docker image (recommended, no local Go/git needed):
|
||||
# docker build -t geosite-builder .
|
||||
# docker run --rm -v "$(pwd)/output:/work/output" geosite-builder
|
||||
#
|
||||
# Or locally if you already have Go + git installed:
|
||||
# ./build.sh
|
||||
|
||||
set -eu
|
||||
|
||||
WORKDIR="$(pwd)/work"
|
||||
OUTDIR="$(pwd)/output"
|
||||
CUSTOM_DIR="$(pwd)/data-custom"
|
||||
|
||||
echo "==> Preparing workspace"
|
||||
rm -rf "$WORKDIR"
|
||||
mkdir -p "$WORKDIR" "$OUTDIR"
|
||||
cd "$WORKDIR"
|
||||
|
||||
echo "==> Cloning v2fly/domain-list-community (base compiler + ~1500 stock categories -"
|
||||
echo " already covers openai, anthropic, netflix, whatsapp, discord, signal, paypal,"
|
||||
echo " linkedin, reddit, spotify, facebook, instagram, twitter, roblox, binance, bbc,"
|
||||
echo " archive, viber, rutracker, speedtest, notion, figma, jetbrains, adobe, intel,"
|
||||
echo " autodesk, cursor, vercel, github, google, youtube, telegram, and more)"
|
||||
git clone --depth 1 https://github.com/v2fly/domain-list-community.git v2fly
|
||||
|
||||
echo "==> Cloning hydraponique/roscomvpn-geosite (RU-specific categories)"
|
||||
git clone --depth 1 https://github.com/hydraponique/roscomvpn-geosite.git roscomvpn-src
|
||||
|
||||
echo "==> Merging data directories"
|
||||
mkdir -p v2fly/data/roscomvpn v2fly/data/custom
|
||||
|
||||
cp -r roscomvpn-src/data/. v2fly/data/roscomvpn/
|
||||
|
||||
# Only genuinely missing from v2fly stock (verified by testing): replicate,
|
||||
# avast, cointelegraph, snapchat. Everything else is already covered upstream.
|
||||
if [ -d "$CUSTOM_DIR" ]; then
|
||||
cp -r "$CUSTOM_DIR/." v2fly/data/custom/
|
||||
fi
|
||||
|
||||
echo "==> Building merged geosite.dat"
|
||||
cd v2fly
|
||||
go run main.go
|
||||
|
||||
if [ ! -f dlc.dat ]; then
|
||||
echo "ERROR: build did not produce dlc.dat — check output above for errors"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
cp dlc.dat "$OUTDIR/geosite.dat"
|
||||
|
||||
echo "==> Done. Output: $OUTDIR/geosite.dat"
|
||||
ls -lh "$OUTDIR/geosite.dat"
|
||||
@@ -0,0 +1 @@
|
||||
domain:cointelegraph.com
|
||||
@@ -0,0 +1,14 @@
|
||||
services:
|
||||
geosite-build:
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile.scheduled
|
||||
container_name: geosite-build
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
GITEA_URL: "https://git.5838.ru"
|
||||
GITEA_REPO: "Payroll3042/geosite-build" # change if your repo name differs
|
||||
GITEA_TOKEN: "${GITEA_TOKEN}" # set in Portainer's stack env vars, not here
|
||||
GIT_USER_NAME: "geosite-bot"
|
||||
GIT_USER_EMAIL: "geosite-bot@5838.ru"
|
||||
GITEA_BRANCH: "main"
|
||||
Reference in New Issue
Block a user