arr/blocklists
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct consumer model: Cleanuparr is the only direct consumer

CodeX
2026-04-07 02:12:52 +02:00
parent d4fdeb66ac
commit 5382b10eb9
2 changed files with 132 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Consumers.md
+115 -98
@@ -1,19 +1,25 @@
# Consumers # Consumers
The blocklists are consumed by two tools in the ARR stack: The blocklists are consumed by [Cleanuparr](https://github.com/Cleanuparr/Cleanuparr).
qBittorrent itself does not read any URL from this repository -- the only
direct consumer is Cleanuparr, which exposes two independent features that
each pull a different file:
| Tool | Role | File consumed | Mode | | Cleanuparr feature | File consumed | What it does |
|---|---|---|---| |---|---|---|
| qBittorrent | Download client | `blacklist` | Excluded file names | | Blocklist Sync | `blacklist` | Pushes the full pattern set to qBittorrent's "Excluded file names" via the qBittorrent Web API |
| Cleanuparr | Media cleanup / malware blocker | `blacklist` or `whitelist` | Blacklist or whitelist mode | | Malware Blocker | `whitelist` (whitelist mode) or `blacklist` (blacklist mode) | Inspects torrents in the Sonarr/Radarr queue and removes ones that violate the rules |
Both tools read a remote text file over HTTPS, one glob pattern per line. The two features are configured independently in Cleanuparr's settings,
They refresh on their own schedule (qBittorrent on restart or manual each with its own URL and refresh schedule. Use both, either, or neither.
refresh; Cleanuparr on its configured interval). This repository is set up so that **both** can be used together: Blocklist
Sync against `blacklist`, Malware Blocker against `whitelist` in whitelist
mode.
## Raw URLs ## Raw URLs
Point consumers at the raw file URLs, not the Gitea blob viewer URLs: Point Cleanuparr's features at the raw file URLs, not the Gitea blob viewer
URLs:
``` ```
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
@@ -24,129 +30,128 @@ The `raw/branch/main/` path serves the file contents directly with the
correct `text/plain` content type. Using `src/branch/main/` instead serves correct `text/plain` content type. Using `src/branch/main/` instead serves
the HTML viewer page and will break the consumer. the HTML viewer page and will break the consumer.
## qBittorrent ## Blocklist Sync (push to qBittorrent)
qBittorrent has an **excluded file names** feature that skips files Cleanuparr's **Blocklist Sync** fetches a blocklist URL on its configured
matching any of the configured glob patterns when downloading a torrent. interval, then pushes the full pattern list into qBittorrent's "Excluded
There is no "included file names" or whitelist mode -- qBittorrent only file names" setting via the qBittorrent Web API. The user does not
supports exclusion. This is why it consumes the merged `blacklist` and not configure qBittorrent's exclusion list directly -- everything happens
the `whitelist`. through Cleanuparr.
### What it supports
- **Blacklists only.** There is no whitelist mode for Blocklist Sync.
- **Full replacement** of qBittorrent's exclusion list on every push. Any
patterns set manually in qBittorrent will be overwritten.
- **All connected qBittorrent instances** receive the same list (one push
per configured download client in Cleanuparr).
### Configuration ### Configuration
1. Open **Options** (Tools -> Options, or Ctrl+,). In Cleanuparr's settings, enable Blocklist Sync and point it at:
2. Go to **Downloads**.
3. Scroll to **Excluded file names**.
4. Enable the checkbox.
5. Set the URL to:
``` ```
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
``` ```
qBittorrent fetches the list on startup and whenever you click **Reload** Cleanuparr fetches on its configured interval, then writes the patterns
next to the field. There is no automatic refresh interval -- a restart or to qBittorrent. Check Cleanuparr's logs to confirm the push succeeded
manual reload is required to pick up changes. after each sync.
### What qBittorrent does with the list ### How qBittorrent uses the pushed list
When a torrent is added, qBittorrent iterates the files inside it and Once Cleanuparr has pushed the patterns, qBittorrent treats them exactly
checks each filename against the excluded patterns. Matching files are the same as if they had been set manually. When a torrent is added,
marked as "do not download" and will not be written to disk. The rest of qBittorrent iterates the files inside it and checks each filename against
the torrent downloads normally. the excluded patterns. Matching files are marked as "do not download" and
will not be written to disk. The rest of the torrent downloads normally.
This means the list operates at the **file level within a torrent**, not This means the list operates at the **file level within a torrent**, not
the torrent level. A torrent containing `movie.mkv` and `movie.nor.srt` the torrent level. A torrent containing `movie.mkv` and `movie.nor.srt`
would download both files if `*.srt` is in the whitelist (and thus not in downloads both files because `*.srt` is in the whitelist (and therefore
the blacklist), or just `movie.mkv` if `*.srt` were in the blacklist. not in the pushed pattern list).
### Refreshing after a whitelist change ### Why this URL
qBittorrent does not auto-refresh the list. After updating `whitelist`: The `blacklist` file in this repository is the upstream Cleanuparr blacklist
with the locally-maintained `whitelist` already subtracted. Anything in
`whitelist` is therefore *not* pushed to qBittorrent, so subtitles, AV1
webm releases, and other extensions you want kept will not be excluded
when qBittorrent downloads a torrent.
1. Wait for the next sync run (or dispatch the workflow manually). ## Malware Blocker (Sonarr/Radarr queue)
2. In qBittorrent, open the excluded file names setting and click
**Reload**, or restart qBittorrent.
3. New torrents added from this point on will use the updated list.
Torrents already in the client are not retroactively changed.
## Cleanuparr Cleanuparr's **Malware Blocker** inspects torrents in the Sonarr and Radarr
download queues, examines the files they contain, and removes any torrent
that violates the configured rules. It runs independently from Blocklist
Sync and uses its own URL and refresh schedule.
Cleanuparr supports two modes for its Malware Blocker and Blacklist Sync ### What it supports
features. The repository provides files suitable for both.
### Blacklist mode - **Whitelist mode** -- only torrents whose contained files all match the
whitelist patterns are kept. Everything else is removed from the queue.
- **Blacklist mode** -- torrents containing any file matching the blacklist
patterns are removed.
- **Sonarr and Radarr queues** -- both supported, configured per-arr in
Cleanuparr's settings.
In blacklist mode, Cleanuparr deletes any file matching a pattern in the ### Recommended mode and URL
configured list.
Point it at the same URL as qBittorrent: This repository is configured for **whitelist mode**: a small explicit list
of allowed extensions is easier to reason about than the much larger
upstream blacklist, especially when the goal is "only let through legitimate
video and subtitle files."
``` Point Malware Blocker at:
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
```
Because the whitelist has already been subtracted, this file will not
cause Cleanuparr to delete anything you have marked as "keep" in the
whitelist. Consistent behaviour between the two tools without any
per-tool customisation.
### Whitelist mode
In whitelist mode, Cleanuparr keeps only files matching a pattern in the
configured list and deletes everything else.
Point it at:
``` ```
https://git.hisp.no/arr/blocklists/raw/branch/main/whitelist https://git.hisp.no/arr/blocklists/raw/branch/main/whitelist
``` ```
This is the more conservative choice: only the extensions explicitly In whitelist mode, only torrents whose contained files all have whitelisted
listed (video containers and subtitles) are allowed. Anything else -- extensions are allowed to remain in the queue. A torrent with `movie.mkv`
including extensions that upstream has not yet flagged as malicious -- and `movie.nor.srt` passes (both extensions are whitelisted). A torrent
is deleted. with `movie.mkv` and `setup.exe` is removed from the queue before it
reaches qBittorrent.
### Which mode to use If you would rather use blacklist mode, point Malware Blocker at the
`blacklist` URL instead -- the same one Blocklist Sync uses. Both modes
use the same data files; only the matching logic differs.
| Use case | Mode | Why | ### Why two layers are useful
|---|---|---|
| You trust upstream Cleanuparr's coverage and want to keep everything except known-bad | Blacklist | Lets through unusual-but-legitimate file types (e.g. exotic subtitle formats) |
| You only want a strict set of video + subtitle files on disk | Whitelist | Much stricter; deletes anything not explicitly listed |
| You want behaviour consistent with qBittorrent | Blacklist | Same source file, same semantics |
Blacklist mode is the recommended default because it matches the Blocklist Sync and Malware Blocker operate at different layers:
qBittorrent side and avoids unexpected deletions of legitimate but
non-listed files.
## Keeping both consumers in sync - **Malware Blocker (queue layer)** stops bad torrents from being added to
qBittorrent in the first place. A torrent that fails the whitelist check
never reaches the download client.
- **Blocklist Sync (file layer)** keeps qBittorrent from writing individual
junk files (samples, NFOs, mixed-in executables) to disk while still
allowing the legitimate files in a release through.
Both consumers ultimately read the whitelist (directly in Cleanuparr Used together, the two layers give you a queue-level allow list and a
whitelist mode, indirectly via subtraction in blacklist mode and in file-level deny list, both maintained from the same `whitelist` source of
qBittorrent). This means maintenance is centralised: truth.
1. Add a line to `whitelist`.
2. Wait for the next sync run (or dispatch manually).
3. Both consumers honour the change after their next refresh.
There is no per-tool configuration drift because there is no per-tool
configuration to drift.
## Troubleshooting ## Troubleshooting
### A file I whitelisted is still being blocked / deleted ### A file I whitelisted is still being blocked
Check each layer in order: Check each layer in order:
1. **Sync ran successfully?** Open the Gitea Actions page for the 1. **Sync ran successfully?** Open the Gitea Actions page for the repository
repository and verify the most recent run is green and newer than and verify the most recent run is green and newer than your whitelist
your whitelist commit. commit.
2. **Blacklist was updated?** Read `blacklist` in Gitea and confirm your 2. **Blacklist was updated?** Read `blacklist` in Gitea and confirm your
whitelisted entry is not present. whitelisted entry is no longer present.
3. **Consumer refreshed?** qBittorrent requires a manual reload or 3. **Cleanuparr refreshed?** Blocklist Sync runs on its configured interval,
restart. Cleanuparr refreshes on its own interval -- check its logs not on demand. Check Cleanuparr's logs to confirm it picked up the new
to confirm it picked up the new file. file. Trigger a manual run from Cleanuparr's UI if available.
4. **Exact string match?** Whitelist entries must match blacklist entries 4. **Cleanuparr pushed to qBittorrent?** After fetching the new file,
Cleanuparr writes it to qBittorrent. Check qBittorrent's
Options -> Downloads -> Excluded file names and confirm the new list
is in place.
5. **Exact string match?** Whitelist entries must match blacklist entries
exactly. `*.srt` in whitelist does not strip `*sample.srt` from exactly. `*.srt` in whitelist does not strip `*sample.srt` from
blacklist. See [Lists](Lists) for pattern semantics. blacklist. See [Lists](Lists) for pattern semantics.
@@ -155,11 +160,11 @@ Check each layer in order:
Check whether the pattern is in the blacklist at all: Check whether the pattern is in the blacklist at all:
1. Open `blacklist` in Gitea and search for the extension. 1. Open `blacklist` in Gitea and search for the extension.
2. If it is not there, upstream does not block it either. You can add 2. If it is not there, upstream Cleanuparr does not block it. You can add
it to `blacklist` directly (manual local addition, preserved by the it to `blacklist` directly (manual local addition, preserved by the
three-way merge) or file an upstream issue. three-way merge) or file an upstream issue.
### Consumer returns 404 ### Cleanuparr returns 404 for the blocklist URL
Verify the URL uses `raw/branch/main/`, not `src/branch/main/`: Verify the URL uses `raw/branch/main/`, not `src/branch/main/`:
@@ -174,8 +179,20 @@ https://git.hisp.no/arr/blocklists/src/branch/main/blacklist
Also check the repository name and branch are correct Also check the repository name and branch are correct
(`arr/blocklists`, `main`). (`arr/blocklists`, `main`).
### Cleanuparr deletes subtitle files ### Malware Blocker is removing legitimate torrents
Cleanuparr is running in whitelist mode against `blacklist`, which is Make sure Malware Blocker is in whitelist mode and pointed at `whitelist`,
the wrong combination. Either switch it to blacklist mode (keep the URL), not `blacklist`. In blacklist mode it uses the much larger upstream
or keep whitelist mode and point it at `whitelist` instead. blacklist and may remove a torrent that contains a single junk file
inside an otherwise legitimate release.
### Blocklist Sync did not push to qBittorrent
Check Cleanuparr's connection to qBittorrent:
1. Cleanuparr settings -> Download clients -- verify the qBittorrent host,
port, username, and password.
2. Cleanuparr logs -- look for Blocklist Sync errors after the most recent
scheduled run. Connection failures are logged at error level.
3. qBittorrent Web API enabled -- the push uses the Web API, which must
be enabled in qBittorrent's options under Web UI.
Lists.md
+17 -13
@@ -15,8 +15,9 @@ the three-way merge baseline used by the sync script. Never edit it.
## `blacklist` ## `blacklist`
The blacklist is the output file consumed by qBittorrent and (optionally) The blacklist is the output file consumed by Cleanuparr's Blocklist Sync
Cleanuparr. It is regenerated on every sync as: feature, which pushes the patterns to qBittorrent's excluded file names.
It is regenerated on every sync as:
``` ```
upstream_new | custom_local_additions - whitelist upstream_new | custom_local_additions - whitelist
@@ -137,8 +138,10 @@ You want qBittorrent to download them, not strip them.
*.srt *.srt
``` ```
After the next sync, `*.srt` is gone from `blacklist`. qBittorrent now After the next sync, `*.srt` is gone from `blacklist`. Cleanuparr's
accepts `.srt` files from within torrents. `*sample.srt` remains blocked. Blocklist Sync pushes the new list to qBittorrent, which then accepts
`.srt` files from within torrents. `*sample.srt` remains blocked because
exact-string subtraction does not touch it.
### Supporting AV1 in `.webm` containers ### Supporting AV1 in `.webm` containers
@@ -150,8 +153,9 @@ currently blocked because the upstream blacklist treats `*.webm` as junk.
*.webm *.webm
``` ```
After the next sync, `*.webm` is gone from `blacklist`. `.webm` torrents After the next sync, `*.webm` is gone from `blacklist`. Cleanuparr's
download normally. `*sample.webm` remains blocked. Blocklist Sync pushes the updated list to qBittorrent and `.webm`
torrents download normally. `*sample.webm` remains blocked.
### Adding a site-specific junk extension ### Adding a site-specific junk extension
@@ -194,13 +198,13 @@ check the file in Gitea for the current state.
## Consumer consequences ## Consumer consequences
Changes to either file affect what consumers see: Changes to either file affect what Cleanuparr's two features see:
| Change | Effect on qBittorrent | Effect on Cleanuparr (blacklist mode) | Effect on Cleanuparr (whitelist mode) | | Change | Effect on Blocklist Sync (pushed to qBittorrent) | Effect on Malware Blocker (whitelist mode) |
|---|---|---|---| |---|---|---|
| Add to `whitelist` | Stops blocking this extension | Stops deleting this extension | Starts allowing this extension | | Add to `whitelist` | Stops being pushed to qBittorrent's excluded list | Starts allowing this extension in queue |
| Remove from `whitelist` | Resumes blocking (if upstream has it) | Resumes deleting (if upstream has it) | Stops allowing this extension | | Remove from `whitelist` | Resumes being pushed (if upstream has it) | Stops allowing this extension in queue |
| Add to `blacklist` directly | Starts blocking this extension | Starts deleting this extension | No effect | | Add to `blacklist` directly | Starts being pushed to qBittorrent's excluded list | No effect (Malware Blocker reads `whitelist`) |
| Remove from `blacklist` directly | No effect (sync re-adds) | No effect (sync re-adds) | No effect | | Remove from `blacklist` directly | No effect (sync re-adds from upstream) | No effect |
See [Consumers](Consumers) for configuration details. See [Consumers](Consumers) for configuration details.