Feature/add or remove #7
No files matched your search
@@ -3,20 +3,18 @@
|
|||||||
|
|
||||||
## Overview
|
## Overview
|
||||||
|
|
||||||
`service-royalroad-chapters` is a FastAPI-based webserver that monitors Royalroad stories for new chapters and sends notifications data to `backend-api-internal`.
|
`service-royalroad-chapters` is a FastAPI-based webserver that monitors RSS feeds for new entries and sends a notification to every subscribed user through `backend-api-internal`. It supports two feed types:
|
||||||
|
|
||||||
## Features
|
- **Royal Road**: new chapters of a story
|
||||||
|
- **Gitea**: new releases of a repository on `gitea.com/gitea/`
|
||||||
|
|
||||||
- **RSS Feed Monitoring**: Parses Royal Road RSS feeds to detect new chapter releases
|
Feeds are shared between users. A feed is stored once and polled once, and each user has their own subscription with an optional custom name.
|
||||||
- **Automatic Updates**: Updates database with latest chapter numbers
|
|
||||||
- **Retry Logic**: Built-in retry mechanisms for network failures with exponential backoff
|
|
||||||
- **Prometheus Metrics**: Request counters and performance monitoring on port 9000
|
|
||||||
|
|
||||||
## Prerequisites
|
## Prerequisites
|
||||||
|
|
||||||
- Python 3.10+
|
- Python 3.11+
|
||||||
- MySQL database
|
- MySQL database with the tables described under [Database](database.sql), including the `users` table
|
||||||
- Backend notification service (for sending push notifications)
|
- `backend-api-internal` (receives the notifications)
|
||||||
- Royal Road story IDs
|
- Royal Road story IDs
|
||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
@@ -25,6 +23,32 @@
|
|||||||
pip install -r requirements.txt
|
pip install -r requirements.txt
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Internal libraries (`lib-db-module`, `lib-secret-manager`, `lib-service-auth`) come from the Gitea package registry. Point pip at it first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
export PIP_EXTRA_INDEX_URL=https://git.gansejunge.com/api/packages/notifier/pypi/simple/
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Secrets are read from files under `/etc/secrets/` (Kubernetes secrets) and fall back to environment variables for development. The files are deleted after startup.
|
||||||
|
|
||||||
|
| Secret | File | Dev environment variable |
|
||||||
|
|---|---|---|
|
||||||
|
| Gateway key | `/etc/secrets/gateway_key` | `GATEWAY_KEY` |
|
||||||
|
| Scheduler key | `/etc/secrets/scheduler_key` | `SCHEDULER_KEY` |
|
||||||
|
| Gitea token | `/etc/secrets/gitea_token` | `GITEA_TOKEN` |
|
||||||
|
| Notification API key | `/etc/secrets/api_key` | (see `lib-secret-manager`) |
|
||||||
|
| Database user | `/etc/secrets/db_username` | (see `lib-db-module`) |
|
||||||
|
| Database password | `/etc/secrets/db_password` | (see `lib-db-module`) |
|
||||||
|
|
||||||
|
Caller keys must be at least 32 characters and must differ from each other. Generate them with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
## Running the Service
|
## Running the Service
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
@@ -35,56 +59,178 @@ This starts two servers:
|
|||||||
- **Main API**: `http://0.0.0.0:5000` - Handles chapter check requests
|
- **Main API**: `http://0.0.0.0:5000` - Handles chapter check requests
|
||||||
- **Metrics Server**: `http://0.0.0.0:9000/metrics` - Prometheus metrics endpoint
|
- **Metrics Server**: `http://0.0.0.0:9000/metrics` - Prometheus metrics endpoint
|
||||||
|
|
||||||
### API Endpoint
|
## Authentication
|
||||||
|
|
||||||
#### Check for New Chapters
|
Every route on the main API requires a bearer key, and each key belongs to exactly one caller:
|
||||||
|
|
||||||
```http
|
```http
|
||||||
GET /royalroad
|
Authorization: Bearer <key>
|
||||||
```
|
```
|
||||||
|
|
||||||
**Response:**
|
| Caller | Key | May call |
|
||||||
|
|---|---|---|
|
||||||
|
| Gateway (`backend-api`) | `gateway_key` | `/add/feed`, `/delete/feed` |
|
||||||
|
| Scheduler (CronJob) | `scheduler_key` | `/royalroad`, `/gitea` |
|
||||||
|
|
||||||
|
A missing or unknown key returns `401`. A valid key used on a route it is not allowed to call returns `403`.
|
||||||
|
|
||||||
|
User routes also require the user the gateway is acting for:
|
||||||
|
|
||||||
|
```http
|
||||||
|
X-User-Id: 1
|
||||||
|
```
|
||||||
|
|
||||||
|
The value must be a positive integer.
|
||||||
|
|
||||||
|
## API
|
||||||
|
|
||||||
|
### Subscribe to a feed
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /add/feed
|
||||||
|
```
|
||||||
|
|
||||||
|
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||||
|
|
||||||
|
**Body:**
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"status": "checked"
|
"type": "royalroad",
|
||||||
|
"rss_feed_url": "royalroad.com/fiction/118891",
|
||||||
|
"custom_name": "My favourite story"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
| Field | Required | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `type` | yes | `royalroad` or `gitea` |
|
||||||
|
| `rss_feed_url` | yes | Feed or page URL, see [Accepted feed URLs](#accepted-feed-urls) |
|
||||||
|
| `custom_name` | no | Name used as the notification title. If empty, the title from the feed is used |
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "ok",
|
||||||
|
"detail": "Subscribed to feed",
|
||||||
|
"feed_url": "https://www.royalroad.com/fiction/syndication/65058"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Behavior:**
|
**Behavior:**
|
||||||
1. Fetches all active stories from the database
|
|
||||||
2. Queries Royal Road RSS feed for each story
|
|
||||||
3. Compares latest chapter with stored chapter number
|
|
||||||
4. Updates database if new chapter is detected
|
|
||||||
5. Sends notification to backend service
|
|
||||||
|
|
||||||
**Error Responses:**
|
1. Validates the URL and converts it to the canonical feed URL
|
||||||
- `404`: Feed not found or has no entries
|
2. If the feed is not stored yet, checks that it exists and records its latest entry, so only entries published after subscribing trigger a notification
|
||||||
- `500`: Internal server error during processing
|
3. Stores the feed, or reactivates it if it was inactive
|
||||||
- `503`: Notification service unavailable after retries
|
4. Creates the user's subscription
|
||||||
|
|
||||||
## Notification Format
|
Sending the same request again renames the subscription and reactivates it if the user had unsubscribed.
|
||||||
|
|
||||||
Messages sent to the backend service:
|
### Unsubscribe from a feed
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /delete/feed
|
||||||
|
```
|
||||||
|
|
||||||
|
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||||
|
|
||||||
|
**Body:** same as `/add/feed`. `custom_name` is ignored.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"user_id": 1,
|
"status": "ok",
|
||||||
"message": {
|
"detail": "Unsubscribed from feed",
|
||||||
"title": "New Life As A Max Level Archmage",
|
"feed_url": "https://www.royalroad.com/fiction/syndication/118891"
|
||||||
"body": "Chapter 68 has been released",
|
|
||||||
"link": "https://www.royalroad.com/fiction/118891/new-life-as-a-max-level-archmage/chapter/2683797/68-carve",
|
|
||||||
"category": "royal-road",
|
|
||||||
"timestamp": 1729339200
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Headers:**
|
**Behavior:** deactivates the user's subscription. If no active subscribers are left, the feed itself is deactivated and is no longer polled. An unknown feed also returns `ok`.
|
||||||
|
|
||||||
|
### Check for new chapters
|
||||||
|
|
||||||
```http
|
```http
|
||||||
X-API-Key-Internal: <api_key>
|
POST /royalroad
|
||||||
Content-Type: application/json
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||||
|
|
||||||
|
### Check for new Gitea releases
|
||||||
|
|
||||||
|
```http
|
||||||
|
POST /gitea
|
||||||
|
```
|
||||||
|
|
||||||
|
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||||
|
|
||||||
|
**Response (both):**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"checked": 12,
|
||||||
|
"new": 2,
|
||||||
|
"errors": 0
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
If any feed failed, the same counts are returned as the `detail` of a `500` response.
|
||||||
|
|
||||||
|
**Notification failures:**
|
||||||
|
|
||||||
|
- If every notification for a feed fails, `last_entry` is not updated, the feed counts as an error and it is retried on the next run
|
||||||
|
- If only some fail, `last_entry` is updated, so the users who were notified are not notified again
|
||||||
|
- If the notification backend is unavailable, the remaining users of that feed are skipped
|
||||||
|
|
||||||
|
When a new release of `gitea/runner` is detected, the service also triggers the runner build workflow on the Gitea instance, using `gitea_token`.
|
||||||
|
|
||||||
|
## Accepted feed URLs
|
||||||
|
|
||||||
|
The scheme and `www.` are optional. Whatever is submitted is converted to the canonical feed URL.
|
||||||
|
|
||||||
|
| Type | Accepted input | Stored as |
|
||||||
|
|---|---|---|
|
||||||
|
| `royalroad` | `royalroad.com/fiction/118891`<br>`https://www.royalroad.com/fiction/118891/story-title` | `https://www.royalroad.com/fiction/syndication/118891` |
|
||||||
|
| `gitea` | `gitea.com/gitea/runner`<br>`https://gitea.com/gitea/runner/releases`<br>`https://gitea.com/gitea/runner/releases.rss` | `https://gitea.com/gitea/runner/releases.rss` |
|
||||||
|
|
||||||
|
Only repositories under `gitea.com/gitea/` are supported.
|
||||||
|
|
||||||
|
## Error Responses
|
||||||
|
|
||||||
|
Errors are returned as `{"detail": "<message>"}`.
|
||||||
|
|
||||||
|
| Status | Cause | Example `detail` |
|
||||||
|
|---|---|---|
|
||||||
|
| `401` | Missing or invalid bearer key | `Unauthorized` |
|
||||||
|
| `403` | Key is valid but not allowed on this route | `Forbidden` |
|
||||||
|
| `422` | Invalid `X-User-Id` or body | `type: Input should be 'royalroad' or 'gitea'` |
|
||||||
|
| `422` | URL does not match the feed type | `A gitea feed must be on gitea.com` |
|
||||||
|
| `422` | URL has the wrong path | `Unrecognised feed path (expected e.g. royalroad.com/fiction/118891)` |
|
||||||
|
| `422` | The feed does not exist | `The feed does not exist: <url>` |
|
||||||
|
| `502` | The feed could not be checked right now | `Could not verify the feed right now, try again later: <url>` |
|
||||||
|
| `500` | Internal error, or a check run with failed feeds | |
|
||||||
|
|
||||||
|
|
||||||
|
## Notification Format
|
||||||
|
|
||||||
|
One message is sent to `backend-api-internal` per subscriber:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"user_id": 1,
|
||||||
|
"message": {
|
||||||
|
"title": "New Life As A Max Level Archmage",
|
||||||
|
"body": "Chapter 68 has been released",
|
||||||
|
"link": "https://www.royalroad.com/fiction/118891/new-life-as-a-max-level-archmage/chapter/2683797/68-carve",
|
||||||
|
"category": "royal-road",
|
||||||
|
"timestamp": 1729339200
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
Gitea releases use the category `utility`. When a subscription has no custom name, the title is the feed title for Royal Road and the repository path (for example `gitea.com/gitea/runner`) for Gitea.
|
||||||
|
|
||||||
## Monitoring
|
## Monitoring
|
||||||
|
|
||||||
### Prometheus Metrics
|
### Prometheus Metrics
|
||||||
|
|||||||
+2
-2
@@ -10,11 +10,11 @@ feedparser==6.0.14
|
|||||||
feedparser-sgmllib==2.1.0
|
feedparser-sgmllib==2.1.0
|
||||||
h11==0.16.0
|
h11==0.16.0
|
||||||
idna==3.20
|
idna==3.20
|
||||||
lib-db-module==0.3.0
|
lib-db-module==0.4.0
|
||||||
lib-secret-manager==0.3.1
|
lib-secret-manager==0.3.1
|
||||||
lib-send-notifications==0.1.0
|
lib-send-notifications==0.1.0
|
||||||
lib-service-auth==0.1.0
|
lib-service-auth==0.1.0
|
||||||
mysql-connector-python==9.4.0
|
mysql-connector-python==26.10.0
|
||||||
opentelemetry-api==1.45.1
|
opentelemetry-api==1.45.1
|
||||||
prometheus_client==0.26.0
|
prometheus_client==0.26.0
|
||||||
pydantic==2.14.1
|
pydantic==2.14.1
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
class FeedError(Exception):
|
||||||
|
"""Base class for every feed problem the API can explain to a client."""
|
||||||
|
status_code = 400
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidFeedUrlError(FeedError, ValueError):
|
||||||
|
"""The URL is malformed or doesn't fit the chosen feed type.
|
||||||
|
|
||||||
|
Subclasses ValueError so Pydantic reports it as a normal 422 validation error.
|
||||||
|
"""
|
||||||
|
status_code = 422
|
||||||
|
|
||||||
|
|
||||||
|
class WrongHostError(InvalidFeedUrlError):
|
||||||
|
def __init__(self, feed_type: str, expected_host: str):
|
||||||
|
super().__init__(f"A {feed_type} feed must be on {expected_host}")
|
||||||
|
|
||||||
|
|
||||||
|
class UnsupportedPathError(InvalidFeedUrlError):
|
||||||
|
def __init__(self, hint: str):
|
||||||
|
super().__init__(f"Unrecognised feed path ({hint})")
|
||||||
|
|
||||||
|
|
||||||
|
class FeedNotFoundError(FeedError):
|
||||||
|
status_code = 422
|
||||||
|
|
||||||
|
def __init__(self, url: str):
|
||||||
|
super().__init__(f"The feed does not exist: {url}")
|
||||||
|
|
||||||
|
|
||||||
|
class FeedCheckFailedError(FeedError):
|
||||||
|
status_code = 502
|
||||||
|
|
||||||
|
def __init__(self, url: str):
|
||||||
|
super().__init__(f"Could not verify the feed right now, try again later: {url}")
|
||||||
+48
-34
@@ -1,11 +1,11 @@
|
|||||||
import time
|
import time
|
||||||
from dataclasses import dataclass
|
from typing import assert_never
|
||||||
from typing import Tuple
|
|
||||||
|
|
||||||
import feedparser
|
import feedparser
|
||||||
import requests
|
import requests
|
||||||
from pydantic import BaseModel
|
|
||||||
|
|
||||||
|
from feed_errors import FeedCheckFailedError, FeedNotFoundError
|
||||||
|
from models import FeedEntryType, FeedUpdate
|
||||||
from simple_logger_handler import setup_logger
|
from simple_logger_handler import setup_logger
|
||||||
|
|
||||||
logger = setup_logger(__name__)
|
logger = setup_logger(__name__)
|
||||||
@@ -14,14 +14,6 @@ _GITEA_INSTANCE_BASE_URL="https://git.gansejunge.com"
|
|||||||
_GITEA_REPOSITORY_OWNER="utility"
|
_GITEA_REPOSITORY_OWNER="utility"
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class RoyalRoadStory(BaseModel):
|
|
||||||
chapter_title: str
|
|
||||||
link: str
|
|
||||||
story_title: str
|
|
||||||
chapter_id: str
|
|
||||||
|
|
||||||
|
|
||||||
def get_rss_feed_data(rss_feed_url: str, max_retries: int = 3) -> feedparser.FeedParserDict:
|
def get_rss_feed_data(rss_feed_url: str, max_retries: int = 3) -> feedparser.FeedParserDict:
|
||||||
"""
|
"""
|
||||||
Fetches information from any RSS feed, with retries on network-related errors.
|
Fetches information from any RSS feed, with retries on network-related errors.
|
||||||
@@ -62,26 +54,28 @@ def get_rss_feed_data(rss_feed_url: str, max_retries: int = 3) -> feedparser.Fee
|
|||||||
logger.debug(f"[Feed] Parsed {len(feed.entries)} entries from {rss_feed_url}")
|
logger.debug(f"[Feed] Parsed {len(feed.entries)} entries from {rss_feed_url}")
|
||||||
return feed
|
return feed
|
||||||
|
|
||||||
def grab_gitea_release_information(rss_feed_url: str) -> Tuple[str, str]:
|
def grab_gitea_release_information(rss_feed_url: str) -> FeedUpdate:
|
||||||
"""
|
"""
|
||||||
Fetches the latest release information from a Gitea releases RSS feed.
|
Fetches the latest release information from a Gitea releases RSS feed.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
A tuple: (release_title, release_link)
|
FeedUpdate:
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ValueError: If the feed has no entries.
|
ValueError: If the feed has no entries.
|
||||||
"""
|
"""
|
||||||
feed = get_rss_feed_data(rss_feed_url)
|
feed = get_rss_feed_data(rss_feed_url)
|
||||||
entry = feed.entries[0]
|
feed_update = FeedUpdate (
|
||||||
title = entry["title"]
|
entry_id = feed.entries[0]["link"],
|
||||||
link = entry["link"]
|
feed_title = feed.feed.title,
|
||||||
|
body_title = feed.entries[0]["title"],
|
||||||
|
link = feed.entries[0]["link"],
|
||||||
|
)
|
||||||
|
logger.info(f"[Feed] Latest release for feed '{rss_feed_url}': {feed_update.feed_title}")
|
||||||
|
logger.debug(f"[Feed] Latest release for feed '{rss_feed_url}': {feed_update.feed_title} -> {feed_update.link}")
|
||||||
|
return feed_update
|
||||||
|
|
||||||
logger.info(f"[Feed] Latest release for feed '{rss_feed_url}': {title}")
|
def grab_royal_road_chapter_information(rss_feed_url: str) -> FeedUpdate :
|
||||||
logger.debug(f"[Feed] Latest release for feed '{rss_feed_url}': {title} -> {link}")
|
|
||||||
return title, link
|
|
||||||
|
|
||||||
def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
|
||||||
"""
|
"""
|
||||||
Fetches the latest chapter information from a Royalroad RSS feed.
|
Fetches the latest chapter information from a Royalroad RSS feed.
|
||||||
|
|
||||||
@@ -89,7 +83,7 @@ def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
|||||||
rss_feed_url: URL of the Royalroad story's RSS feed.
|
rss_feed_url: URL of the Royalroad story's RSS feed.
|
||||||
|
|
||||||
Returns:
|
Returns:
|
||||||
A tuple: (chapter_title, chapter_link, story_title, chapter_id)
|
FeedUpdate
|
||||||
|
|
||||||
Raises:
|
Raises:
|
||||||
ValueError: If the feed has no entries.
|
ValueError: If the feed has no entries.
|
||||||
@@ -97,19 +91,19 @@ def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
|||||||
feed = get_rss_feed_data(rss_feed_url)
|
feed = get_rss_feed_data(rss_feed_url)
|
||||||
latest_chapter_data = feed.entries[0]
|
latest_chapter_data = feed.entries[0]
|
||||||
chapter_title_with_story_title = latest_chapter_data["title"]
|
chapter_title_with_story_title = latest_chapter_data["title"]
|
||||||
chapter_title = chapter_title_with_story_title.removeprefix(feed.feed.title).removeprefix(" - ")
|
body_title = chapter_title_with_story_title.removeprefix(feed.feed.title).removeprefix(" - ")
|
||||||
|
|
||||||
royal_road_story = RoyalRoadStory (
|
feed_update = FeedUpdate (
|
||||||
chapter_title = chapter_title,
|
entry_id = str(latest_chapter_data["id"]),
|
||||||
|
feed_title = feed.feed.title,
|
||||||
|
body_title = body_title,
|
||||||
link = latest_chapter_data["link"],
|
link = latest_chapter_data["link"],
|
||||||
story_title = feed.feed.title,
|
|
||||||
chapter_id = str(latest_chapter_data["id"])
|
|
||||||
)
|
)
|
||||||
|
|
||||||
logger.info(f"[Feed] Latest chapter for story '{royal_road_story.story_title}': {royal_road_story.chapter_title}")
|
logger.info(f"[Feed] Latest chapter for story '{feed_update.feed_title}': {feed_update.body_title}")
|
||||||
logger.debug(f"[Feed] Latest chapter for story '{royal_road_story.story_title}' (RSS URL {rss_feed_url}):"
|
logger.debug(f"[Feed] Latest chapter for story '{feed_update.feed_title}' (RSS URL {rss_feed_url}):"
|
||||||
f"{royal_road_story.chapter_title} (ID {royal_road_story.chapter_id}) -> {royal_road_story.link}")
|
f"{feed_update.body_title} (ID {feed_update.entry_id}) -> {feed_update.link}")
|
||||||
return royal_road_story
|
return feed_update
|
||||||
|
|
||||||
def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
||||||
url = (
|
url = (
|
||||||
@@ -122,7 +116,7 @@ def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
|||||||
payload = {
|
payload = {
|
||||||
"ref": "main",
|
"ref": "main",
|
||||||
"inputs": {
|
"inputs": {
|
||||||
"version": version.lstrip("v"),
|
"version": version.removeprefix("v"),
|
||||||
"revision": "1",
|
"revision": "1",
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
@@ -138,10 +132,30 @@ def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
|||||||
logger.warning(f"[Feed] Failed to run Gitea-Runner build: {e}")
|
logger.warning(f"[Feed] Failed to run Gitea-Runner build: {e}")
|
||||||
return False
|
return False
|
||||||
|
|
||||||
|
def ensure_feed_exists(url: str, timeout: tuple[float, float] = (3.0, 5.0)) -> None:
|
||||||
|
try:
|
||||||
|
with requests.get(url, stream=True, timeout=timeout, allow_redirects=False) as response:
|
||||||
|
status = response.status_code
|
||||||
|
except requests.RequestException as e:
|
||||||
|
raise FeedCheckFailedError(url) from e
|
||||||
|
if status == 404:
|
||||||
|
raise FeedNotFoundError(url)
|
||||||
|
if status != 200:
|
||||||
|
raise FeedCheckFailedError(url)
|
||||||
|
|
||||||
|
def fetch_latest(feed_type: FeedEntryType, url: str) -> FeedUpdate:
|
||||||
|
match feed_type:
|
||||||
|
case FeedEntryType.ROYALROAD:
|
||||||
|
return grab_royal_road_chapter_information(url)
|
||||||
|
case FeedEntryType.GITEA:
|
||||||
|
return grab_gitea_release_information(url)
|
||||||
|
case _:
|
||||||
|
assert_never(feed_type)
|
||||||
|
|
||||||
if __name__ == "__main__":
|
if __name__ == "__main__":
|
||||||
#rsssss="https://gitea.com/gitea/runner/releases.rss"
|
rsssss="https://gitea.com/gitea/runner/releases.rss"
|
||||||
#print(get_rss_feed_data(rsssss))
|
#print(get_rss_feed_data(rsssss))
|
||||||
#print(grab_gitea_release_information(rsssss))
|
print(grab_gitea_release_information(rsssss))
|
||||||
id="118891"
|
id="118891"
|
||||||
rss_feed_url = f"https://www.royalroad.com/fiction/syndication/{id}"
|
rss_feed_url = f"https://www.royalroad.com/fiction/syndication/{id}"
|
||||||
print(grab_royal_road_chapter_information(rss_feed_url))
|
print(grab_royal_road_chapter_information(rss_feed_url))
|
||||||
+143
-20
@@ -4,22 +4,29 @@ from contextlib import asynccontextmanager
|
|||||||
|
|
||||||
import uvicorn
|
import uvicorn
|
||||||
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request
|
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request
|
||||||
|
from fastapi.exceptions import RequestValidationError
|
||||||
|
from fastapi.responses import JSONResponse
|
||||||
|
from requests.exceptions import RequestException
|
||||||
|
|
||||||
from auth_config import Caller, auth
|
from auth_config import Caller, auth
|
||||||
from db_module import close_connection_pool, create_connection_pool, get_db, start_healthcheck_thread
|
from db_module import close_connection_pool, create_connection_pool, get_db, start_healthcheck_thread
|
||||||
|
from feed_errors import FeedCheckFailedError, FeedError
|
||||||
from feed_handler import (
|
from feed_handler import (
|
||||||
build_gitea_runner,
|
build_gitea_runner,
|
||||||
|
ensure_feed_exists,
|
||||||
|
fetch_latest,
|
||||||
grab_gitea_release_information,
|
grab_gitea_release_information,
|
||||||
grab_royal_road_chapter_information,
|
grab_royal_road_chapter_information,
|
||||||
)
|
)
|
||||||
from metrics_server import REQUEST_COUNTER, metrics_api
|
from metrics_server import REQUEST_COUNTER, metrics_api
|
||||||
|
from models import FeedEntry
|
||||||
from secret_manager import cleanup_secret_files, get_secret
|
from secret_manager import cleanup_secret_files, get_secret
|
||||||
from send_notifications import NotificationClient, NotificationError, NotificationUnavailableError
|
from send_notifications import NotificationClient, NotificationError, NotificationUnavailableError
|
||||||
|
from service_auth import UserId
|
||||||
from simple_logger_handler import LOG_LEVEL, setup_logger
|
from simple_logger_handler import LOG_LEVEL, setup_logger
|
||||||
|
|
||||||
logger = setup_logger(__name__)
|
logger = setup_logger(__name__)
|
||||||
|
|
||||||
|
|
||||||
@asynccontextmanager
|
@asynccontextmanager
|
||||||
async def lifespan(app: FastAPI):
|
async def lifespan(app: FastAPI):
|
||||||
logger.info("[FastAPI] Starting application...")
|
logger.info("[FastAPI] Starting application...")
|
||||||
@@ -53,6 +60,31 @@ api = FastAPI(
|
|||||||
dependencies=[Depends(auth.authenticate)],
|
dependencies=[Depends(auth.authenticate)],
|
||||||
)
|
)
|
||||||
|
|
||||||
|
@api.exception_handler(FeedError)
|
||||||
|
async def feed_error_handler(request: Request, exc: FeedError):
|
||||||
|
return JSONResponse(status_code=exc.status_code, content={"detail":str(exc)})
|
||||||
|
|
||||||
|
@api.exception_handler(RequestValidationError)
|
||||||
|
async def validation_error_handler(request: Request, exc: RequestValidationError):
|
||||||
|
errors = exc.errors()
|
||||||
|
|
||||||
|
# 1) If one of our own FeedErrors caused it, use its message as is
|
||||||
|
for err in errors:
|
||||||
|
original = (err.get("ctx") or {}).get("error")
|
||||||
|
if isinstance(original, FeedError):
|
||||||
|
return JSONResponse(
|
||||||
|
status_code=original.status_code,
|
||||||
|
content={"detail": str(original)},
|
||||||
|
)
|
||||||
|
|
||||||
|
# 2) Anything else (missing field, bad enum value, bad header): flatten to one string
|
||||||
|
messages = []
|
||||||
|
for err in errors:
|
||||||
|
field = ".".join(str(p) for p in err["loc"] if p != "body")
|
||||||
|
msg = err["msg"].removeprefix("Value error, ")
|
||||||
|
messages.append(f"{field}: {msg}" if field else msg)
|
||||||
|
return JSONResponse(status_code=422, content={"detail": "; ".join(messages)})
|
||||||
|
|
||||||
@api.middleware("http")
|
@api.middleware("http")
|
||||||
async def prometheus_middleware(request: Request, call_next):
|
async def prometheus_middleware(request: Request, call_next):
|
||||||
logger.debug(f"[FastAPI] Incoming request: {request.method} {request.url.path}")
|
logger.debug(f"[FastAPI] Incoming request: {request.method} {request.url.path}")
|
||||||
@@ -103,13 +135,13 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
|||||||
for feed_id, rss_feed, last_entry in feeds:
|
for feed_id, rss_feed, last_entry in feeds:
|
||||||
results["checked"] += 1
|
results["checked"] += 1
|
||||||
try:
|
try:
|
||||||
release_title, link = grab_gitea_release_information(rss_feed)
|
gitea_release_information = grab_gitea_release_information(rss_feed)
|
||||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={link}")
|
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={gitea_release_information.entry_id}")
|
||||||
|
|
||||||
if link == last_entry:
|
if gitea_release_information.entry_id == last_entry:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
logger.info(f"[FastAPI] Release change for ID {feed_id}: {release_title}")
|
logger.info(f"[FastAPI] Release change for ID {feed_id}: {gitea_release_information.body_title}")
|
||||||
|
|
||||||
with db.cursor() as cursor:
|
with db.cursor() as cursor:
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
@@ -122,20 +154,20 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
|||||||
|
|
||||||
sent = 0
|
sent = 0
|
||||||
failed = 0
|
failed = 0
|
||||||
|
repo = gitea_release_information.link.split("/releases")[0].removeprefix("https://")
|
||||||
for user_id,feed_name in subscribers:
|
for user_id,feed_name in subscribers:
|
||||||
try:
|
try:
|
||||||
repo = link.split("/releases")[0].removeprefix("https://")
|
|
||||||
request.app.state.notifier.send(
|
request.app.state.notifier.send(
|
||||||
title=feed_name or repo,
|
title=feed_name or repo,
|
||||||
body=release_title,
|
body=gitea_release_information.body_title,
|
||||||
link=link,
|
link=gitea_release_information.link,
|
||||||
category="utility",
|
category="utility",
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
)
|
)
|
||||||
sent += 1
|
sent += 1
|
||||||
except NotificationUnavailableError:
|
except NotificationUnavailableError:
|
||||||
failed = len(subscribers) - sent
|
failed = len(subscribers) - sent
|
||||||
logger.error(f"[FastAPI] Notifications backend unavailable, skipping"
|
logger.error(f"[FastAPI] Notifications backend unavailable, skipping "
|
||||||
f"{failed} remaining user(s) for feed_id={feed_id}")
|
f"{failed} remaining user(s) for feed_id={feed_id}")
|
||||||
break
|
break
|
||||||
except NotificationError as e:
|
except NotificationError as e:
|
||||||
@@ -149,14 +181,14 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
|||||||
with db.cursor() as cursor:
|
with db.cursor() as cursor:
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
||||||
(link, feed_id),
|
(gitea_release_information.entry_id, feed_id),
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
logger.debug(f"[FastAPI] Notification sent and database updated for Gitea feed ID {feed_id}")
|
logger.debug(f"[FastAPI] Notification sent and database updated for Gitea feed ID {feed_id}")
|
||||||
results["new"] += 1
|
results["new"] += 1
|
||||||
|
|
||||||
if link.startswith("https://gitea.com/gitea/runner/releases/tag/v"):
|
if gitea_release_information.link.startswith("https://gitea.com/gitea/runner/releases/tag/v"):
|
||||||
gitea_build_success = build_gitea_runner(release_title, request.app.state.gitea_token)
|
gitea_build_success = build_gitea_runner(gitea_release_information.body_title, request.app.state.gitea_token)
|
||||||
logger.info(f"[FastAPI] Gitea build ran successfully:{gitea_build_success}")
|
logger.info(f"[FastAPI] Gitea build ran successfully:{gitea_build_success}")
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
@@ -205,12 +237,12 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
|||||||
results["checked"] += 1
|
results["checked"] += 1
|
||||||
try:
|
try:
|
||||||
story = grab_royal_road_chapter_information(rss_feed)
|
story = grab_royal_road_chapter_information(rss_feed)
|
||||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={story.chapter_id}")
|
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={story.entry_id}")
|
||||||
|
|
||||||
if story.chapter_id == last_entry:
|
if story.entry_id == last_entry:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
logger.info(f"[FastAPI] New chapter detected for story ID {feed_id}: {story.story_title}")
|
logger.info(f"[FastAPI] New chapter detected for story ID {feed_id}: {story.feed_title}")
|
||||||
with db.cursor() as cursor:
|
with db.cursor() as cursor:
|
||||||
cursor.execute (
|
cursor.execute (
|
||||||
"SELECT user_id, feed_name "
|
"SELECT user_id, feed_name "
|
||||||
@@ -225,8 +257,8 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
|||||||
for user_id, feed_name in subscribers:
|
for user_id, feed_name in subscribers:
|
||||||
try:
|
try:
|
||||||
request.app.state.notifier.send(
|
request.app.state.notifier.send(
|
||||||
title=feed_name or story.story_title,
|
title=feed_name or story.feed_title,
|
||||||
body=story.chapter_title,
|
body=story.body_title,
|
||||||
link=story.link,
|
link=story.link,
|
||||||
category="royal-road",
|
category="royal-road",
|
||||||
user_id=user_id,
|
user_id=user_id,
|
||||||
@@ -235,11 +267,11 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
|||||||
except NotificationUnavailableError:
|
except NotificationUnavailableError:
|
||||||
failed = len(subscribers) - sent
|
failed = len(subscribers) - sent
|
||||||
logger.error (f"[FastAPI] Notifications backend unavailable, skipping "
|
logger.error (f"[FastAPI] Notifications backend unavailable, skipping "
|
||||||
f"{failed} remaining user(s) for story={story.story_title}")
|
f"{failed} remaining user(s) for story={story.feed_title}")
|
||||||
break
|
break
|
||||||
except NotificationError as e:
|
except NotificationError as e:
|
||||||
failed += 1
|
failed += 1
|
||||||
logger.error(f"[FastAPI] Failed to send notification to user_id={user_id} for story={story.story_title} "
|
logger.error(f"[FastAPI] Failed to send notification to user_id={user_id} for story={story.feed_title} "
|
||||||
f"status={getattr(e, 'status_code',None)}")
|
f"status={getattr(e, 'status_code',None)}")
|
||||||
if subscribers and sent == 0:
|
if subscribers and sent == 0:
|
||||||
results["errors"] += 1
|
results["errors"] += 1
|
||||||
@@ -248,7 +280,7 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
|||||||
with db.cursor() as cursor:
|
with db.cursor() as cursor:
|
||||||
cursor.execute(
|
cursor.execute(
|
||||||
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
||||||
(story.chapter_id, feed_id),
|
(story.entry_id, feed_id),
|
||||||
)
|
)
|
||||||
db.commit()
|
db.commit()
|
||||||
logger.debug(f"[FastAPI] Notification sent and database updated for story ID {feed_id}")
|
logger.debug(f"[FastAPI] Notification sent and database updated for story ID {feed_id}")
|
||||||
@@ -269,6 +301,97 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
|||||||
logger.info(f"[FastAPI] Royalroad check complete: {results}")
|
logger.info(f"[FastAPI] Royalroad check complete: {results}")
|
||||||
return results
|
return results
|
||||||
|
|
||||||
|
@user_router.post("/add/feed")
|
||||||
|
def add_feed(
|
||||||
|
user_id: UserId,
|
||||||
|
feed_entry: FeedEntry,
|
||||||
|
db = Depends(get_db)
|
||||||
|
):
|
||||||
|
url = feed_entry.rss_feed_url
|
||||||
|
logger.info(f"[FastAPI] Subscribe request: RSS URL={url}, user_id={user_id}")
|
||||||
|
|
||||||
|
with db.cursor() as cursor:
|
||||||
|
cursor.execute("SELECT id FROM rss_feeds WHERE rss_feed = %s", (url,),)
|
||||||
|
known = cursor.fetchone()
|
||||||
|
|
||||||
|
last_entry_id = None
|
||||||
|
if known is None:
|
||||||
|
ensure_feed_exists(url)
|
||||||
|
logger.info(f"[FastApi] Adding new RSS feed, type={feed_entry.type}, rss_feed_url={url}")
|
||||||
|
try:
|
||||||
|
last_entry_id = fetch_latest(feed_entry.type, url).entry_id
|
||||||
|
except (ValueError, RequestException) as e:
|
||||||
|
logger.warning(f"[FastAPI] Could not read latest entry for {url}, storing none")
|
||||||
|
raise FeedCheckFailedError(url) from e
|
||||||
|
try:
|
||||||
|
with db.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"INSERT into rss_feeds (type,rss_feed,last_entry) VALUES (%s,%s,%s) "
|
||||||
|
"ON DUPLICATE KEY UPDATE id = LAST_INSERT_ID(id), active = 1",
|
||||||
|
(feed_entry.type, url, last_entry_id),
|
||||||
|
)
|
||||||
|
feed_id = cursor.lastrowid
|
||||||
|
|
||||||
|
cursor.execute (
|
||||||
|
"INSERT INTO rss_feed_subscriptions (user_id, rss_feed_id, feed_name) "
|
||||||
|
"VALUES (%s, %s, %s) "
|
||||||
|
"ON DUPLICATE KEY UPDATE active = 1, feed_name = VALUES(feed_name)",
|
||||||
|
(user_id, feed_id, feed_entry.custom_name),
|
||||||
|
)
|
||||||
|
db.commit()
|
||||||
|
except Exception as e:
|
||||||
|
db.rollback()
|
||||||
|
logger.exception(f"[FastAPI] Failed to subscribe user_id={user_id} to {url}")
|
||||||
|
raise HTTPException(status_code=500) from e
|
||||||
|
|
||||||
|
logger.info(f"[FastAPI] user_id={user_id} subscribed to feed_id={feed_id}, url={url}")
|
||||||
|
return {"status":"ok","detail":"Subscribed to feed","feed_url":url}
|
||||||
|
|
||||||
|
@user_router.post("/delete/feed")
|
||||||
|
def delete_feed(
|
||||||
|
user_id: UserId,
|
||||||
|
feed_entry: FeedEntry,
|
||||||
|
db = Depends(get_db),
|
||||||
|
):
|
||||||
|
url = feed_entry.rss_feed_url
|
||||||
|
logger.info(f"[FastAPI] Unsubscribe request: RSS URL={url}, user_id={user_id}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with db.cursor() as cursor:
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT id FROM rss_feeds WHERE rss_feed = %s",
|
||||||
|
(url,),
|
||||||
|
)
|
||||||
|
known = cursor.fetchone()
|
||||||
|
|
||||||
|
if known is None:
|
||||||
|
db.rollback()
|
||||||
|
return {"status":"ok","detail":"Unsubscribed to feed","feed_url":url}
|
||||||
|
|
||||||
|
feed_id = known[0]
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE rss_feed_subscriptions SET active = 0 "
|
||||||
|
"WHERE user_id = %s AND rss_feed_id = %s",
|
||||||
|
(user_id, feed_id),
|
||||||
|
)
|
||||||
|
|
||||||
|
cursor.execute(
|
||||||
|
"SELECT 1 from rss_feed_subscriptions "
|
||||||
|
"WHERE rss_feed_id = %s AND active = 1 LIMIT 1",
|
||||||
|
(feed_id,),
|
||||||
|
)
|
||||||
|
if cursor.fetchone() is None:
|
||||||
|
cursor.execute(
|
||||||
|
"UPDATE rss_feeds SET active = 0 WHERE id = %s",
|
||||||
|
(feed_id,),
|
||||||
|
)
|
||||||
|
|
||||||
|
db.commit()
|
||||||
|
except Exception as e:
|
||||||
|
db.rollback()
|
||||||
|
logger.exception(f"[FastAPI] Failed to unsubscribe user_id={user_id} to {url}")
|
||||||
|
raise HTTPException(status_code=500) from e
|
||||||
|
return {"status":"ok","detail":"Unsubscribed to feed","feed_url":url}
|
||||||
api.include_router(user_router)
|
api.include_router(user_router)
|
||||||
api.include_router(scheduler_router)
|
api.include_router(scheduler_router)
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,82 @@
|
|||||||
|
from dataclasses import dataclass
|
||||||
|
from enum import StrEnum
|
||||||
|
from typing import Self
|
||||||
|
from urllib.parse import urlsplit
|
||||||
|
|
||||||
|
from pydantic import BaseModel, Field, field_validator, model_validator
|
||||||
|
|
||||||
|
from feed_errors import InvalidFeedUrlError, UnsupportedPathError, WrongHostError
|
||||||
|
|
||||||
|
REPO_CHARS = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_.-")
|
||||||
|
|
||||||
|
@dataclass(frozen=True)
|
||||||
|
class FeedUpdate():
|
||||||
|
entry_id: str
|
||||||
|
feed_title: str
|
||||||
|
body_title: str
|
||||||
|
link: str | None = None
|
||||||
|
|
||||||
|
class FeedEntryType(StrEnum):
|
||||||
|
ROYALROAD = "royalroad"
|
||||||
|
GITEA = "gitea"
|
||||||
|
|
||||||
|
|
||||||
|
def _path_segments(raw: str, feed_type: FeedEntryType, host: str) -> list[str]:
|
||||||
|
"""Check scheme and host, then return the non-empty path segments."""
|
||||||
|
raw = raw.strip()
|
||||||
|
if "://" not in raw:
|
||||||
|
raw = "https://" + raw # allow "royalroad.com/fiction/118891"
|
||||||
|
try:
|
||||||
|
parts = urlsplit(raw)
|
||||||
|
except ValueError as e:
|
||||||
|
raise InvalidFeedUrlError("The URL could not be parsed") from e
|
||||||
|
if parts.scheme not in ("http", "https"):
|
||||||
|
raise InvalidFeedUrlError("The URL must use http or https")
|
||||||
|
# Comparing the whole netloc also rejects ports and user@host
|
||||||
|
if parts.netloc.lower().removeprefix("www.") != host:
|
||||||
|
raise WrongHostError(feed_type.value, host)
|
||||||
|
return [s for s in parts.path.split("/") if s]
|
||||||
|
|
||||||
|
|
||||||
|
def _royalroad_feed(segments: list[str]) -> str:
|
||||||
|
# fiction/<id>[/<slug>]
|
||||||
|
if len(segments) >= 2 and segments[0] == "fiction":
|
||||||
|
story_id = segments[1]
|
||||||
|
if story_id.isascii() and story_id.isdigit():
|
||||||
|
return f"https://www.royalroad.com/fiction/syndication/{int(story_id)}"
|
||||||
|
raise UnsupportedPathError("expected e.g. royalroad.com/fiction/118891")
|
||||||
|
|
||||||
|
|
||||||
|
def _gitea_feed(segments: list[str]) -> str:
|
||||||
|
# gitea/<repo>[/releases | /releases.rss]
|
||||||
|
if len(segments) >= 2 and segments[0] == "gitea":
|
||||||
|
repo, rest = segments[1], segments[2:]
|
||||||
|
repo_ok = repo[0] not in ".-" and set(repo) <= REPO_CHARS
|
||||||
|
if repo_ok and rest in ([], ["releases"], ["releases.rss"]):
|
||||||
|
return f"https://gitea.com/gitea/{repo}/releases.rss"
|
||||||
|
raise UnsupportedPathError("expected e.g. gitea.com/gitea/runner")
|
||||||
|
|
||||||
|
|
||||||
|
# To support a new feed type: add an enum member, a function above, add a case tofetch_latest and a line here.
|
||||||
|
_RULES = {
|
||||||
|
FeedEntryType.ROYALROAD: ("royalroad.com", _royalroad_feed),
|
||||||
|
FeedEntryType.GITEA: ("gitea.com", _gitea_feed),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
class FeedEntry(BaseModel):
|
||||||
|
type: FeedEntryType
|
||||||
|
rss_feed_url: str = Field(max_length=512)
|
||||||
|
custom_name: str | None = Field(default=None, max_length=255)
|
||||||
|
|
||||||
|
@field_validator("custom_name")
|
||||||
|
@classmethod
|
||||||
|
def blank_name_is_none(cls, v: str | None) -> str | None:
|
||||||
|
return (v or "").strip() or None
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def normalise_feed_url(self) -> Self:
|
||||||
|
host, build_feed_url = _RULES[self.type]
|
||||||
|
segments = _path_segments(self.rss_feed_url, self.type, host)
|
||||||
|
self.rss_feed_url = build_feed_url(segments)
|
||||||
|
return self
|
||||||
Reference in new issue
Block a user