diff --git a/README.md b/README.md index 8b72c8d..890e54f 100644 --- a/README.md +++ b/README.md @@ -3,20 +3,18 @@ ## Overview -`service-royalroad-chapters` is a FastAPI-based webserver that monitors Royalroad stories for new chapters and sends notifications data to `backend-api-internal`. +`service-royalroad-chapters` is a FastAPI-based webserver that monitors RSS feeds for new entries and sends a notification to every subscribed user through `backend-api-internal`. It supports two feed types: -## Features +- **Royal Road**: new chapters of a story +- **Gitea**: new releases of a repository on `gitea.com/gitea/` -- **RSS Feed Monitoring**: Parses Royal Road RSS feeds to detect new chapter releases -- **Automatic Updates**: Updates database with latest chapter numbers -- **Retry Logic**: Built-in retry mechanisms for network failures with exponential backoff -- **Prometheus Metrics**: Request counters and performance monitoring on port 9000 +Feeds are shared between users. A feed is stored once and polled once, and each user has their own subscription with an optional custom name. ## Prerequisites -- Python 3.10+ -- MySQL database -- Backend notification service (for sending push notifications) +- Python 3.11+ +- MySQL database with the tables described under [Database](database.sql), including the `users` table +- `backend-api-internal` (receives the notifications) - Royal Road story IDs ## Installation @@ -25,6 +23,32 @@ pip install -r requirements.txt ``` +Internal libraries (`lib-db-module`, `lib-secret-manager`, `lib-service-auth`) come from the Gitea package registry. Point pip at it first: + +```bash +export PIP_EXTRA_INDEX_URL=https://git.gansejunge.com/api/packages/notifier/pypi/simple/ +``` + +## Configuration + +Secrets are read from files under `/etc/secrets/` (Kubernetes secrets) and fall back to environment variables for development. The files are deleted after startup. + +| Secret | File | Dev environment variable | +|---|---|---| +| Gateway key | `/etc/secrets/gateway_key` | `GATEWAY_KEY` | +| Scheduler key | `/etc/secrets/scheduler_key` | `SCHEDULER_KEY` | +| Gitea token | `/etc/secrets/gitea_token` | `GITEA_TOKEN` | +| Notification API key | `/etc/secrets/api_key` | (see `lib-secret-manager`) | +| Database user | `/etc/secrets/db_username` | (see `lib-db-module`) | +| Database password | `/etc/secrets/db_password` | (see `lib-db-module`) | + +Caller keys must be at least 32 characters and must differ from each other. Generate them with: + +```bash +python3 -c "import secrets; print(secrets.token_urlsafe(32))" +``` + + ## Running the Service ```bash @@ -35,56 +59,178 @@ This starts two servers: - **Main API**: `http://0.0.0.0:5000` - Handles chapter check requests - **Metrics Server**: `http://0.0.0.0:9000/metrics` - Prometheus metrics endpoint -### API Endpoint +## Authentication -#### Check for New Chapters +Every route on the main API requires a bearer key, and each key belongs to exactly one caller: ```http -GET /royalroad +Authorization: Bearer ``` -**Response:** +| Caller | Key | May call | +|---|---|---| +| Gateway (`backend-api`) | `gateway_key` | `/add/feed`, `/delete/feed` | +| Scheduler (CronJob) | `scheduler_key` | `/royalroad`, `/gitea` | + +A missing or unknown key returns `401`. A valid key used on a route it is not allowed to call returns `403`. + +User routes also require the user the gateway is acting for: + +```http +X-User-Id: 1 +``` + +The value must be a positive integer. + +## API + +### Subscribe to a feed + +```http +POST /add/feed +``` + +**Headers:** `Authorization: Bearer `, `X-User-Id: ` + +**Body:** + ```json { - "status": "checked" + "type": "royalroad", + "rss_feed_url": "royalroad.com/fiction/118891", + "custom_name": "My favourite story" +} +``` + +| Field | Required | Description | +|---|---|---| +| `type` | yes | `royalroad` or `gitea` | +| `rss_feed_url` | yes | Feed or page URL, see [Accepted feed URLs](#accepted-feed-urls) | +| `custom_name` | no | Name used as the notification title. If empty, the title from the feed is used | + +**Response:** + +```json +{ + "status": "ok", + "detail": "Subscribed to feed", + "feed_url": "https://www.royalroad.com/fiction/syndication/65058" } ``` **Behavior:** -1. Fetches all active stories from the database -2. Queries Royal Road RSS feed for each story -3. Compares latest chapter with stored chapter number -4. Updates database if new chapter is detected -5. Sends notification to backend service -**Error Responses:** -- `404`: Feed not found or has no entries -- `500`: Internal server error during processing -- `503`: Notification service unavailable after retries +1. Validates the URL and converts it to the canonical feed URL +2. If the feed is not stored yet, checks that it exists and records its latest entry, so only entries published after subscribing trigger a notification +3. Stores the feed, or reactivates it if it was inactive +4. Creates the user's subscription -## Notification Format +Sending the same request again renames the subscription and reactivates it if the user had unsubscribed. -Messages sent to the backend service: +### Unsubscribe from a feed + +```http +POST /delete/feed +``` + +**Headers:** `Authorization: Bearer `, `X-User-Id: ` + +**Body:** same as `/add/feed`. `custom_name` is ignored. + +**Response:** ```json { - "user_id": 1, - "message": { - "title": "New Life As A Max Level Archmage", - "body": "Chapter 68 has been released", - "link": "https://www.royalroad.com/fiction/118891/new-life-as-a-max-level-archmage/chapter/2683797/68-carve", - "category": "royal-road", - "timestamp": 1729339200 - } + "status": "ok", + "detail": "Unsubscribed from feed", + "feed_url": "https://www.royalroad.com/fiction/syndication/118891" } ``` -**Headers:** +**Behavior:** deactivates the user's subscription. If no active subscribers are left, the feed itself is deactivated and is no longer polled. An unknown feed also returns `ok`. + +### Check for new chapters + ```http -X-API-Key-Internal: -Content-Type: application/json +POST /royalroad ``` +**Headers:** `Authorization: Bearer ` + +### Check for new Gitea releases + +```http +POST /gitea +``` + +**Headers:** `Authorization: Bearer ` + +**Response (both):** + +```json +{ + "checked": 12, + "new": 2, + "errors": 0 +} +``` + +If any feed failed, the same counts are returned as the `detail` of a `500` response. + +**Notification failures:** + +- If every notification for a feed fails, `last_entry` is not updated, the feed counts as an error and it is retried on the next run +- If only some fail, `last_entry` is updated, so the users who were notified are not notified again +- If the notification backend is unavailable, the remaining users of that feed are skipped + +When a new release of `gitea/runner` is detected, the service also triggers the runner build workflow on the Gitea instance, using `gitea_token`. + +## Accepted feed URLs + +The scheme and `www.` are optional. Whatever is submitted is converted to the canonical feed URL. + +| Type | Accepted input | Stored as | +|---|---|---| +| `royalroad` | `royalroad.com/fiction/118891`
`https://www.royalroad.com/fiction/118891/story-title` | `https://www.royalroad.com/fiction/syndication/118891` | +| `gitea` | `gitea.com/gitea/runner`
`https://gitea.com/gitea/runner/releases`
`https://gitea.com/gitea/runner/releases.rss` | `https://gitea.com/gitea/runner/releases.rss` | + +Only repositories under `gitea.com/gitea/` are supported. + +## Error Responses + +Errors are returned as `{"detail": ""}`. + +| Status | Cause | Example `detail` | +|---|---|---| +| `401` | Missing or invalid bearer key | `Unauthorized` | +| `403` | Key is valid but not allowed on this route | `Forbidden` | +| `422` | Invalid `X-User-Id` or body | `type: Input should be 'royalroad' or 'gitea'` | +| `422` | URL does not match the feed type | `A gitea feed must be on gitea.com` | +| `422` | URL has the wrong path | `Unrecognised feed path (expected e.g. royalroad.com/fiction/118891)` | +| `422` | The feed does not exist | `The feed does not exist: ` | +| `502` | The feed could not be checked right now | `Could not verify the feed right now, try again later: ` | +| `500` | Internal error, or a check run with failed feeds | | + + +## Notification Format + +One message is sent to `backend-api-internal` per subscriber: + +```json +{ + "user_id": 1, + "message": { + "title": "New Life As A Max Level Archmage", + "body": "Chapter 68 has been released", + "link": "https://www.royalroad.com/fiction/118891/new-life-as-a-max-level-archmage/chapter/2683797/68-carve", + "category": "royal-road", + "timestamp": 1729339200 + } +} +``` + +Gitea releases use the category `utility`. When a subscription has no custom name, the title is the feed title for Royal Road and the repository path (for example `gitea.com/gitea/runner`) for Gitea. + ## Monitoring ### Prometheus Metrics