Added path to unsubscribe from RSS feeds and exception handler to propagate FeedErrors
This commit is contained in:
florian committed
1 parent
6f1f024e90
commit
73fcd3b53c
1 file changed
+175
-29
@@ -3,20 +3,18 @@
|
||||
|
||||
## Overview
|
||||
|
||||
`service-royalroad-chapters` is a FastAPI-based webserver that monitors Royalroad stories for new chapters and sends notifications data to `backend-api-internal`.
|
||||
`service-royalroad-chapters` is a FastAPI-based webserver that monitors RSS feeds for new entries and sends a notification to every subscribed user through `backend-api-internal`. It supports two feed types:
|
||||
|
||||
## Features
|
||||
- **Royal Road**: new chapters of a story
|
||||
- **Gitea**: new releases of a repository on `gitea.com/gitea/`
|
||||
|
||||
- **RSS Feed Monitoring**: Parses Royal Road RSS feeds to detect new chapter releases
|
||||
- **Automatic Updates**: Updates database with latest chapter numbers
|
||||
- **Retry Logic**: Built-in retry mechanisms for network failures with exponential backoff
|
||||
- **Prometheus Metrics**: Request counters and performance monitoring on port 9000
|
||||
Feeds are shared between users. A feed is stored once and polled once, and each user has their own subscription with an optional custom name.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- MySQL database
|
||||
- Backend notification service (for sending push notifications)
|
||||
- Python 3.11+
|
||||
- MySQL database with the tables described under [Database](database.sql), including the `users` table
|
||||
- `backend-api-internal` (receives the notifications)
|
||||
- Royal Road story IDs
|
||||
|
||||
## Installation
|
||||
@@ -25,6 +23,32 @@
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Internal libraries (`lib-db-module`, `lib-secret-manager`, `lib-service-auth`) come from the Gitea package registry. Point pip at it first:
|
||||
|
||||
```bash
|
||||
export PIP_EXTRA_INDEX_URL=https://git.gansejunge.com/api/packages/notifier/pypi/simple/
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Secrets are read from files under `/etc/secrets/` (Kubernetes secrets) and fall back to environment variables for development. The files are deleted after startup.
|
||||
|
||||
| Secret | File | Dev environment variable |
|
||||
|---|---|---|
|
||||
| Gateway key | `/etc/secrets/gateway_key` | `GATEWAY_KEY` |
|
||||
| Scheduler key | `/etc/secrets/scheduler_key` | `SCHEDULER_KEY` |
|
||||
| Gitea token | `/etc/secrets/gitea_token` | `GITEA_TOKEN` |
|
||||
| Notification API key | `/etc/secrets/api_key` | (see `lib-secret-manager`) |
|
||||
| Database user | `/etc/secrets/db_username` | (see `lib-db-module`) |
|
||||
| Database password | `/etc/secrets/db_password` | (see `lib-db-module`) |
|
||||
|
||||
Caller keys must be at least 32 characters and must differ from each other. Generate them with:
|
||||
|
||||
```bash
|
||||
python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
```
|
||||
|
||||
|
||||
## Running the Service
|
||||
|
||||
```bash
|
||||
@@ -35,36 +59,162 @@ This starts two servers:
|
||||
- **Main API**: `http://0.0.0.0:5000` - Handles chapter check requests
|
||||
- **Metrics Server**: `http://0.0.0.0:9000/metrics` - Prometheus metrics endpoint
|
||||
|
||||
### API Endpoint
|
||||
## Authentication
|
||||
|
||||
#### Check for New Chapters
|
||||
Every route on the main API requires a bearer key, and each key belongs to exactly one caller:
|
||||
|
||||
```http
|
||||
GET /royalroad
|
||||
Authorization: Bearer <key>
|
||||
```
|
||||
|
||||
**Response:**
|
||||
| Caller | Key | May call |
|
||||
|---|---|---|
|
||||
| Gateway (`backend-api`) | `gateway_key` | `/add/feed`, `/delete/feed` |
|
||||
| Scheduler (CronJob) | `scheduler_key` | `/royalroad`, `/gitea` |
|
||||
|
||||
A missing or unknown key returns `401`. A valid key used on a route it is not allowed to call returns `403`.
|
||||
|
||||
User routes also require the user the gateway is acting for:
|
||||
|
||||
```http
|
||||
X-User-Id: 1
|
||||
```
|
||||
|
||||
The value must be a positive integer.
|
||||
|
||||
## API
|
||||
|
||||
### Subscribe to a feed
|
||||
|
||||
```http
|
||||
POST /add/feed
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||
|
||||
**Body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "checked"
|
||||
"type": "royalroad",
|
||||
"rss_feed_url": "royalroad.com/fiction/118891",
|
||||
"custom_name": "My favourite story"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `type` | yes | `royalroad` or `gitea` |
|
||||
| `rss_feed_url` | yes | Feed or page URL, see [Accepted feed URLs](#accepted-feed-urls) |
|
||||
| `custom_name` | no | Name used as the notification title. If empty, the title from the feed is used |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"detail": "Subscribed to feed",
|
||||
"feed_url": "https://www.royalroad.com/fiction/syndication/65058"
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
1. Fetches all active stories from the database
|
||||
2. Queries Royal Road RSS feed for each story
|
||||
3. Compares latest chapter with stored chapter number
|
||||
4. Updates database if new chapter is detected
|
||||
5. Sends notification to backend service
|
||||
|
||||
**Error Responses:**
|
||||
- `404`: Feed not found or has no entries
|
||||
- `500`: Internal server error during processing
|
||||
- `503`: Notification service unavailable after retries
|
||||
1. Validates the URL and converts it to the canonical feed URL
|
||||
2. If the feed is not stored yet, checks that it exists and records its latest entry, so only entries published after subscribing trigger a notification
|
||||
3. Stores the feed, or reactivates it if it was inactive
|
||||
4. Creates the user's subscription
|
||||
|
||||
Sending the same request again renames the subscription and reactivates it if the user had unsubscribed.
|
||||
|
||||
### Unsubscribe from a feed
|
||||
|
||||
```http
|
||||
POST /delete/feed
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||
|
||||
**Body:** same as `/add/feed`. `custom_name` is ignored.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"detail": "Unsubscribed from feed",
|
||||
"feed_url": "https://www.royalroad.com/fiction/syndication/118891"
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior:** deactivates the user's subscription. If no active subscribers are left, the feed itself is deactivated and is no longer polled. An unknown feed also returns `ok`.
|
||||
|
||||
### Check for new chapters
|
||||
|
||||
```http
|
||||
POST /royalroad
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||
|
||||
### Check for new Gitea releases
|
||||
|
||||
```http
|
||||
POST /gitea
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||
|
||||
**Response (both):**
|
||||
|
||||
```json
|
||||
{
|
||||
"checked": 12,
|
||||
"new": 2,
|
||||
"errors": 0
|
||||
}
|
||||
```
|
||||
|
||||
If any feed failed, the same counts are returned as the `detail` of a `500` response.
|
||||
|
||||
**Notification failures:**
|
||||
|
||||
- If every notification for a feed fails, `last_entry` is not updated, the feed counts as an error and it is retried on the next run
|
||||
- If only some fail, `last_entry` is updated, so the users who were notified are not notified again
|
||||
- If the notification backend is unavailable, the remaining users of that feed are skipped
|
||||
|
||||
When a new release of `gitea/runner` is detected, the service also triggers the runner build workflow on the Gitea instance, using `gitea_token`.
|
||||
|
||||
## Accepted feed URLs
|
||||
|
||||
The scheme and `www.` are optional. Whatever is submitted is converted to the canonical feed URL.
|
||||
|
||||
| Type | Accepted input | Stored as |
|
||||
|---|---|---|
|
||||
| `royalroad` | `royalroad.com/fiction/118891`<br>`https://www.royalroad.com/fiction/118891/story-title` | `https://www.royalroad.com/fiction/syndication/118891` |
|
||||
| `gitea` | `gitea.com/gitea/runner`<br>`https://gitea.com/gitea/runner/releases`<br>`https://gitea.com/gitea/runner/releases.rss` | `https://gitea.com/gitea/runner/releases.rss` |
|
||||
|
||||
Only repositories under `gitea.com/gitea/` are supported.
|
||||
|
||||
## Error Responses
|
||||
|
||||
Errors are returned as `{"detail": "<message>"}`.
|
||||
|
||||
| Status | Cause | Example `detail` |
|
||||
|---|---|---|
|
||||
| `401` | Missing or invalid bearer key | `Unauthorized` |
|
||||
| `403` | Key is valid but not allowed on this route | `Forbidden` |
|
||||
| `422` | Invalid `X-User-Id` or body | `type: Input should be 'royalroad' or 'gitea'` |
|
||||
| `422` | URL does not match the feed type | `A gitea feed must be on gitea.com` |
|
||||
| `422` | URL has the wrong path | `Unrecognised feed path (expected e.g. royalroad.com/fiction/118891)` |
|
||||
| `422` | The feed does not exist | `The feed does not exist: <url>` |
|
||||
| `502` | The feed could not be checked right now | `Could not verify the feed right now, try again later: <url>` |
|
||||
| `500` | Internal error, or a check run with failed feeds | |
|
||||
|
||||
|
||||
## Notification Format
|
||||
|
||||
Messages sent to the backend service:
|
||||
One message is sent to `backend-api-internal` per subscriber:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -79,11 +229,7 @@ Messages sent to the backend service:
|
||||
}
|
||||
```
|
||||
|
||||
**Headers:**
|
||||
```http
|
||||
X-API-Key-Internal: <api_key>
|
||||
Content-Type: application/json
|
||||
```
|
||||
Gitea releases use the category `utility`. When a subscription has no custom name, the title is the feed title for Royal Road and the repository path (for example `gitea.com/gitea/runner`) for Gitea.
|
||||
|
||||
## Monitoring
|
||||
|
||||
|
||||
Reference in new issue
Block a user