arr/blocklists
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Correct consumer model: Cleanuparr is the only direct consumer

CodeX
2026-04-07 02:12:52 +02:00
parent d4fdeb66ac
commit 5382b10eb9
2 changed files with 132 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Consumers.md
+115 -98
@@ -1,19 +1,25 @@
# Consumers
The blocklists are consumed by two tools in the ARR stack:
The blocklists are consumed by [Cleanuparr](https://github.com/Cleanuparr/Cleanuparr).
qBittorrent itself does not read any URL from this repository -- the only
direct consumer is Cleanuparr, which exposes two independent features that
each pull a different file:
| Tool | Role | File consumed | Mode |
|---|---|---|---|
| qBittorrent | Download client | `blacklist` | Excluded file names |
| Cleanuparr | Media cleanup / malware blocker | `blacklist` or `whitelist` | Blacklist or whitelist mode |
| Cleanuparr feature | File consumed | What it does |
|---|---|---|
| Blocklist Sync | `blacklist` | Pushes the full pattern set to qBittorrent's "Excluded file names" via the qBittorrent Web API |
| Malware Blocker | `whitelist` (whitelist mode) or `blacklist` (blacklist mode) | Inspects torrents in the Sonarr/Radarr queue and removes ones that violate the rules |
Both tools read a remote text file over HTTPS, one glob pattern per line.
They refresh on their own schedule (qBittorrent on restart or manual
refresh; Cleanuparr on its configured interval).
The two features are configured independently in Cleanuparr's settings,
each with its own URL and refresh schedule. Use both, either, or neither.
This repository is set up so that **both** can be used together: Blocklist
Sync against `blacklist`, Malware Blocker against `whitelist` in whitelist
mode.
## Raw URLs
Point consumers at the raw file URLs, not the Gitea blob viewer URLs:
Point Cleanuparr's features at the raw file URLs, not the Gitea blob viewer
URLs:
```
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
@@ -24,129 +30,128 @@ The `raw/branch/main/` path serves the file contents directly with the
correct `text/plain` content type. Using `src/branch/main/` instead serves
the HTML viewer page and will break the consumer.
## qBittorrent
## Blocklist Sync (push to qBittorrent)
qBittorrent has an **excluded file names** feature that skips files
matching any of the configured glob patterns when downloading a torrent.
There is no "included file names" or whitelist mode -- qBittorrent only
supports exclusion. This is why it consumes the merged `blacklist` and not
the `whitelist`.
Cleanuparr's **Blocklist Sync** fetches a blocklist URL on its configured
interval, then pushes the full pattern list into qBittorrent's "Excluded
file names" setting via the qBittorrent Web API. The user does not
configure qBittorrent's exclusion list directly -- everything happens
through Cleanuparr.
### What it supports
- **Blacklists only.** There is no whitelist mode for Blocklist Sync.
- **Full replacement** of qBittorrent's exclusion list on every push. Any
patterns set manually in qBittorrent will be overwritten.
- **All connected qBittorrent instances** receive the same list (one push
per configured download client in Cleanuparr).
### Configuration
1. Open **Options** (Tools -> Options, or Ctrl+,).
2. Go to **Downloads**.
3. Scroll to **Excluded file names**.
4. Enable the checkbox.
5. Set the URL to:
In Cleanuparr's settings, enable Blocklist Sync and point it at:
```
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
```
qBittorrent fetches the list on startup and whenever you click **Reload**
next to the field. There is no automatic refresh interval -- a restart or
manual reload is required to pick up changes.
Cleanuparr fetches on its configured interval, then writes the patterns
to qBittorrent. Check Cleanuparr's logs to confirm the push succeeded
after each sync.
### What qBittorrent does with the list
### How qBittorrent uses the pushed list
When a torrent is added, qBittorrent iterates the files inside it and
checks each filename against the excluded patterns. Matching files are
marked as "do not download" and will not be written to disk. The rest of
the torrent downloads normally.
Once Cleanuparr has pushed the patterns, qBittorrent treats them exactly
the same as if they had been set manually. When a torrent is added,
qBittorrent iterates the files inside it and checks each filename against
the excluded patterns. Matching files are marked as "do not download" and
will not be written to disk. The rest of the torrent downloads normally.
This means the list operates at the **file level within a torrent**, not
the torrent level. A torrent containing `movie.mkv` and `movie.nor.srt`
would download both files if `*.srt` is in the whitelist (and thus not in
the blacklist), or just `movie.mkv` if `*.srt` were in the blacklist.
downloads both files because `*.srt` is in the whitelist (and therefore
not in the pushed pattern list).
### Refreshing after a whitelist change
### Why this URL
qBittorrent does not auto-refresh the list. After updating `whitelist`:
The `blacklist` file in this repository is the upstream Cleanuparr blacklist
with the locally-maintained `whitelist` already subtracted. Anything in
`whitelist` is therefore *not* pushed to qBittorrent, so subtitles, AV1
webm releases, and other extensions you want kept will not be excluded
when qBittorrent downloads a torrent.
1. Wait for the next sync run (or dispatch the workflow manually).
2. In qBittorrent, open the excluded file names setting and click
**Reload**, or restart qBittorrent.
3. New torrents added from this point on will use the updated list.
Torrents already in the client are not retroactively changed.
## Malware Blocker (Sonarr/Radarr queue)
## Cleanuparr
Cleanuparr's **Malware Blocker** inspects torrents in the Sonarr and Radarr
download queues, examines the files they contain, and removes any torrent
that violates the configured rules. It runs independently from Blocklist
Sync and uses its own URL and refresh schedule.
Cleanuparr supports two modes for its Malware Blocker and Blacklist Sync
features. The repository provides files suitable for both.
### What it supports
### Blacklist mode
- **Whitelist mode** -- only torrents whose contained files all match the
whitelist patterns are kept. Everything else is removed from the queue.
- **Blacklist mode** -- torrents containing any file matching the blacklist
patterns are removed.
- **Sonarr and Radarr queues** -- both supported, configured per-arr in
Cleanuparr's settings.
In blacklist mode, Cleanuparr deletes any file matching a pattern in the
configured list.
### Recommended mode and URL
Point it at the same URL as qBittorrent:
This repository is configured for **whitelist mode**: a small explicit list
of allowed extensions is easier to reason about than the much larger
upstream blacklist, especially when the goal is "only let through legitimate
video and subtitle files."
```
https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist
```
Because the whitelist has already been subtracted, this file will not
cause Cleanuparr to delete anything you have marked as "keep" in the
whitelist. Consistent behaviour between the two tools without any
per-tool customisation.
### Whitelist mode
In whitelist mode, Cleanuparr keeps only files matching a pattern in the
configured list and deletes everything else.
Point it at:
Point Malware Blocker at:
```
https://git.hisp.no/arr/blocklists/raw/branch/main/whitelist
```
This is the more conservative choice: only the extensions explicitly
listed (video containers and subtitles) are allowed. Anything else --
including extensions that upstream has not yet flagged as malicious --
is deleted.
In whitelist mode, only torrents whose contained files all have whitelisted
extensions are allowed to remain in the queue. A torrent with `movie.mkv`
and `movie.nor.srt` passes (both extensions are whitelisted). A torrent
with `movie.mkv` and `setup.exe` is removed from the queue before it
reaches qBittorrent.
### Which mode to use
If you would rather use blacklist mode, point Malware Blocker at the
`blacklist` URL instead -- the same one Blocklist Sync uses. Both modes
use the same data files; only the matching logic differs.
| Use case | Mode | Why |
|---|---|---|
| You trust upstream Cleanuparr's coverage and want to keep everything except known-bad | Blacklist | Lets through unusual-but-legitimate file types (e.g. exotic subtitle formats) |
| You only want a strict set of video + subtitle files on disk | Whitelist | Much stricter; deletes anything not explicitly listed |
| You want behaviour consistent with qBittorrent | Blacklist | Same source file, same semantics |
### Why two layers are useful
Blacklist mode is the recommended default because it matches the
qBittorrent side and avoids unexpected deletions of legitimate but
non-listed files.
Blocklist Sync and Malware Blocker operate at different layers:
## Keeping both consumers in sync
- **Malware Blocker (queue layer)** stops bad torrents from being added to
qBittorrent in the first place. A torrent that fails the whitelist check
never reaches the download client.
- **Blocklist Sync (file layer)** keeps qBittorrent from writing individual
junk files (samples, NFOs, mixed-in executables) to disk while still
allowing the legitimate files in a release through.
Both consumers ultimately read the whitelist (directly in Cleanuparr
whitelist mode, indirectly via subtraction in blacklist mode and in
qBittorrent). This means maintenance is centralised:
1. Add a line to `whitelist`.
2. Wait for the next sync run (or dispatch manually).
3. Both consumers honour the change after their next refresh.
There is no per-tool configuration drift because there is no per-tool
configuration to drift.
Used together, the two layers give you a queue-level allow list and a
file-level deny list, both maintained from the same `whitelist` source of
truth.
## Troubleshooting
### A file I whitelisted is still being blocked / deleted
### A file I whitelisted is still being blocked
Check each layer in order:
1. **Sync ran successfully?** Open the Gitea Actions page for the
repository and verify the most recent run is green and newer than
your whitelist commit.
1. **Sync ran successfully?** Open the Gitea Actions page for the repository
and verify the most recent run is green and newer than your whitelist
commit.
2. **Blacklist was updated?** Read `blacklist` in Gitea and confirm your
whitelisted entry is not present.
3. **Consumer refreshed?** qBittorrent requires a manual reload or
restart. Cleanuparr refreshes on its own interval -- check its logs
to confirm it picked up the new file.
4. **Exact string match?** Whitelist entries must match blacklist entries
whitelisted entry is no longer present.
3. **Cleanuparr refreshed?** Blocklist Sync runs on its configured interval,
not on demand. Check Cleanuparr's logs to confirm it picked up the new
file. Trigger a manual run from Cleanuparr's UI if available.
4. **Cleanuparr pushed to qBittorrent?** After fetching the new file,
Cleanuparr writes it to qBittorrent. Check qBittorrent's
Options -> Downloads -> Excluded file names and confirm the new list
is in place.
5. **Exact string match?** Whitelist entries must match blacklist entries
exactly. `*.srt` in whitelist does not strip `*sample.srt` from
blacklist. See [Lists](Lists) for pattern semantics.
@@ -155,11 +160,11 @@ Check each layer in order:
Check whether the pattern is in the blacklist at all:
1. Open `blacklist` in Gitea and search for the extension.
2. If it is not there, upstream does not block it either. You can add
2. If it is not there, upstream Cleanuparr does not block it. You can add
it to `blacklist` directly (manual local addition, preserved by the
three-way merge) or file an upstream issue.
### Consumer returns 404
### Cleanuparr returns 404 for the blocklist URL
Verify the URL uses `raw/branch/main/`, not `src/branch/main/`:
@@ -174,8 +179,20 @@ https://git.hisp.no/arr/blocklists/src/branch/main/blacklist
Also check the repository name and branch are correct
(`arr/blocklists`, `main`).
### Cleanuparr deletes subtitle files
### Malware Blocker is removing legitimate torrents
Cleanuparr is running in whitelist mode against `blacklist`, which is
the wrong combination. Either switch it to blacklist mode (keep the URL),
or keep whitelist mode and point it at `whitelist` instead.
Make sure Malware Blocker is in whitelist mode and pointed at `whitelist`,
not `blacklist`. In blacklist mode it uses the much larger upstream
blacklist and may remove a torrent that contains a single junk file
inside an otherwise legitimate release.
### Blocklist Sync did not push to qBittorrent
Check Cleanuparr's connection to qBittorrent:
1. Cleanuparr settings -> Download clients -- verify the qBittorrent host,
port, username, and password.
2. Cleanuparr logs -- look for Blocklist Sync errors after the most recent
scheduled run. Connection failures are logged at error level.
3. qBittorrent Web API enabled -- the push uses the Web API, which must
be enabled in qBittorrent's options under Web UI.
Lists.md
+17 -13
@@ -15,8 +15,9 @@ the three-way merge baseline used by the sync script. Never edit it.
## `blacklist`
The blacklist is the output file consumed by qBittorrent and (optionally)
Cleanuparr. It is regenerated on every sync as:
The blacklist is the output file consumed by Cleanuparr's Blocklist Sync
feature, which pushes the patterns to qBittorrent's excluded file names.
It is regenerated on every sync as:
```
upstream_new | custom_local_additions - whitelist
@@ -137,8 +138,10 @@ You want qBittorrent to download them, not strip them.
*.srt
```
After the next sync, `*.srt` is gone from `blacklist`. qBittorrent now
accepts `.srt` files from within torrents. `*sample.srt` remains blocked.
After the next sync, `*.srt` is gone from `blacklist`. Cleanuparr's
Blocklist Sync pushes the new list to qBittorrent, which then accepts
`.srt` files from within torrents. `*sample.srt` remains blocked because
exact-string subtraction does not touch it.
### Supporting AV1 in `.webm` containers
@@ -150,8 +153,9 @@ currently blocked because the upstream blacklist treats `*.webm` as junk.
*.webm
```
After the next sync, `*.webm` is gone from `blacklist`. `.webm` torrents
download normally. `*sample.webm` remains blocked.
After the next sync, `*.webm` is gone from `blacklist`. Cleanuparr's
Blocklist Sync pushes the updated list to qBittorrent and `.webm`
torrents download normally. `*sample.webm` remains blocked.
### Adding a site-specific junk extension
@@ -194,13 +198,13 @@ check the file in Gitea for the current state.
## Consumer consequences
Changes to either file affect what consumers see:
Changes to either file affect what Cleanuparr's two features see:
| Change | Effect on qBittorrent | Effect on Cleanuparr (blacklist mode) | Effect on Cleanuparr (whitelist mode) |
|---|---|---|---|
| Add to `whitelist` | Stops blocking this extension | Stops deleting this extension | Starts allowing this extension |
| Remove from `whitelist` | Resumes blocking (if upstream has it) | Resumes deleting (if upstream has it) | Stops allowing this extension |
| Add to `blacklist` directly | Starts blocking this extension | Starts deleting this extension | No effect |
| Remove from `blacklist` directly | No effect (sync re-adds) | No effect (sync re-adds) | No effect |
| Change | Effect on Blocklist Sync (pushed to qBittorrent) | Effect on Malware Blocker (whitelist mode) |
|---|---|---|
| Add to `whitelist` | Stops being pushed to qBittorrent's excluded list | Starts allowing this extension in queue |
| Remove from `whitelist` | Resumes being pushed (if upstream has it) | Stops allowing this extension in queue |
| Add to `blacklist` directly | Starts being pushed to qBittorrent's excluded list | No effect (Malware Blocker reads `whitelist`) |
| Remove from `blacklist` directly | No effect (sync re-adds from upstream) | No effect |
See [Consumers](Consumers) for configuration details.