Merge pull request 'Feature/add or remove' (#7) from feature/add-or-remove into main
Reviewed-on: #7
This commit was merged in pull request #7.
This commit is contained in:
commit
bbb4ac5f11
6 files changed
+484
-84
No files matched your search
@@ -3,20 +3,18 @@
|
||||
|
||||
## Overview
|
||||
|
||||
`service-royalroad-chapters` is a FastAPI-based webserver that monitors Royalroad stories for new chapters and sends notifications data to `backend-api-internal`.
|
||||
`service-royalroad-chapters` is a FastAPI-based webserver that monitors RSS feeds for new entries and sends a notification to every subscribed user through `backend-api-internal`. It supports two feed types:
|
||||
|
||||
## Features
|
||||
- **Royal Road**: new chapters of a story
|
||||
- **Gitea**: new releases of a repository on `gitea.com/gitea/`
|
||||
|
||||
- **RSS Feed Monitoring**: Parses Royal Road RSS feeds to detect new chapter releases
|
||||
- **Automatic Updates**: Updates database with latest chapter numbers
|
||||
- **Retry Logic**: Built-in retry mechanisms for network failures with exponential backoff
|
||||
- **Prometheus Metrics**: Request counters and performance monitoring on port 9000
|
||||
Feeds are shared between users. A feed is stored once and polled once, and each user has their own subscription with an optional custom name.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Python 3.10+
|
||||
- MySQL database
|
||||
- Backend notification service (for sending push notifications)
|
||||
- Python 3.11+
|
||||
- MySQL database with the tables described under [Database](database.sql), including the `users` table
|
||||
- `backend-api-internal` (receives the notifications)
|
||||
- Royal Road story IDs
|
||||
|
||||
## Installation
|
||||
@@ -25,6 +23,32 @@
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
Internal libraries (`lib-db-module`, `lib-secret-manager`, `lib-service-auth`) come from the Gitea package registry. Point pip at it first:
|
||||
|
||||
```bash
|
||||
export PIP_EXTRA_INDEX_URL=https://git.gansejunge.com/api/packages/notifier/pypi/simple/
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Secrets are read from files under `/etc/secrets/` (Kubernetes secrets) and fall back to environment variables for development. The files are deleted after startup.
|
||||
|
||||
| Secret | File | Dev environment variable |
|
||||
|---|---|---|
|
||||
| Gateway key | `/etc/secrets/gateway_key` | `GATEWAY_KEY` |
|
||||
| Scheduler key | `/etc/secrets/scheduler_key` | `SCHEDULER_KEY` |
|
||||
| Gitea token | `/etc/secrets/gitea_token` | `GITEA_TOKEN` |
|
||||
| Notification API key | `/etc/secrets/api_key` | (see `lib-secret-manager`) |
|
||||
| Database user | `/etc/secrets/db_username` | (see `lib-db-module`) |
|
||||
| Database password | `/etc/secrets/db_password` | (see `lib-db-module`) |
|
||||
|
||||
Caller keys must be at least 32 characters and must differ from each other. Generate them with:
|
||||
|
||||
```bash
|
||||
python3 -c "import secrets; print(secrets.token_urlsafe(32))"
|
||||
```
|
||||
|
||||
|
||||
## Running the Service
|
||||
|
||||
```bash
|
||||
@@ -35,36 +59,162 @@ This starts two servers:
|
||||
- **Main API**: `http://0.0.0.0:5000` - Handles chapter check requests
|
||||
- **Metrics Server**: `http://0.0.0.0:9000/metrics` - Prometheus metrics endpoint
|
||||
|
||||
### API Endpoint
|
||||
## Authentication
|
||||
|
||||
#### Check for New Chapters
|
||||
Every route on the main API requires a bearer key, and each key belongs to exactly one caller:
|
||||
|
||||
```http
|
||||
GET /royalroad
|
||||
Authorization: Bearer <key>
|
||||
```
|
||||
|
||||
**Response:**
|
||||
| Caller | Key | May call |
|
||||
|---|---|---|
|
||||
| Gateway (`backend-api`) | `gateway_key` | `/add/feed`, `/delete/feed` |
|
||||
| Scheduler (CronJob) | `scheduler_key` | `/royalroad`, `/gitea` |
|
||||
|
||||
A missing or unknown key returns `401`. A valid key used on a route it is not allowed to call returns `403`.
|
||||
|
||||
User routes also require the user the gateway is acting for:
|
||||
|
||||
```http
|
||||
X-User-Id: 1
|
||||
```
|
||||
|
||||
The value must be a positive integer.
|
||||
|
||||
## API
|
||||
|
||||
### Subscribe to a feed
|
||||
|
||||
```http
|
||||
POST /add/feed
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||
|
||||
**Body:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "checked"
|
||||
"type": "royalroad",
|
||||
"rss_feed_url": "royalroad.com/fiction/118891",
|
||||
"custom_name": "My favourite story"
|
||||
}
|
||||
```
|
||||
|
||||
| Field | Required | Description |
|
||||
|---|---|---|
|
||||
| `type` | yes | `royalroad` or `gitea` |
|
||||
| `rss_feed_url` | yes | Feed or page URL, see [Accepted feed URLs](#accepted-feed-urls) |
|
||||
| `custom_name` | no | Name used as the notification title. If empty, the title from the feed is used |
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"detail": "Subscribed to feed",
|
||||
"feed_url": "https://www.royalroad.com/fiction/syndication/65058"
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior:**
|
||||
1. Fetches all active stories from the database
|
||||
2. Queries Royal Road RSS feed for each story
|
||||
3. Compares latest chapter with stored chapter number
|
||||
4. Updates database if new chapter is detected
|
||||
5. Sends notification to backend service
|
||||
|
||||
**Error Responses:**
|
||||
- `404`: Feed not found or has no entries
|
||||
- `500`: Internal server error during processing
|
||||
- `503`: Notification service unavailable after retries
|
||||
1. Validates the URL and converts it to the canonical feed URL
|
||||
2. If the feed is not stored yet, checks that it exists and records its latest entry, so only entries published after subscribing trigger a notification
|
||||
3. Stores the feed, or reactivates it if it was inactive
|
||||
4. Creates the user's subscription
|
||||
|
||||
Sending the same request again renames the subscription and reactivates it if the user had unsubscribed.
|
||||
|
||||
### Unsubscribe from a feed
|
||||
|
||||
```http
|
||||
POST /delete/feed
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <gateway_key>`, `X-User-Id: <user id>`
|
||||
|
||||
**Body:** same as `/add/feed`. `custom_name` is ignored.
|
||||
|
||||
**Response:**
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ok",
|
||||
"detail": "Unsubscribed from feed",
|
||||
"feed_url": "https://www.royalroad.com/fiction/syndication/118891"
|
||||
}
|
||||
```
|
||||
|
||||
**Behavior:** deactivates the user's subscription. If no active subscribers are left, the feed itself is deactivated and is no longer polled. An unknown feed also returns `ok`.
|
||||
|
||||
### Check for new chapters
|
||||
|
||||
```http
|
||||
POST /royalroad
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||
|
||||
### Check for new Gitea releases
|
||||
|
||||
```http
|
||||
POST /gitea
|
||||
```
|
||||
|
||||
**Headers:** `Authorization: Bearer <scheduler_key>`
|
||||
|
||||
**Response (both):**
|
||||
|
||||
```json
|
||||
{
|
||||
"checked": 12,
|
||||
"new": 2,
|
||||
"errors": 0
|
||||
}
|
||||
```
|
||||
|
||||
If any feed failed, the same counts are returned as the `detail` of a `500` response.
|
||||
|
||||
**Notification failures:**
|
||||
|
||||
- If every notification for a feed fails, `last_entry` is not updated, the feed counts as an error and it is retried on the next run
|
||||
- If only some fail, `last_entry` is updated, so the users who were notified are not notified again
|
||||
- If the notification backend is unavailable, the remaining users of that feed are skipped
|
||||
|
||||
When a new release of `gitea/runner` is detected, the service also triggers the runner build workflow on the Gitea instance, using `gitea_token`.
|
||||
|
||||
## Accepted feed URLs
|
||||
|
||||
The scheme and `www.` are optional. Whatever is submitted is converted to the canonical feed URL.
|
||||
|
||||
| Type | Accepted input | Stored as |
|
||||
|---|---|---|
|
||||
| `royalroad` | `royalroad.com/fiction/118891`<br>`https://www.royalroad.com/fiction/118891/story-title` | `https://www.royalroad.com/fiction/syndication/118891` |
|
||||
| `gitea` | `gitea.com/gitea/runner`<br>`https://gitea.com/gitea/runner/releases`<br>`https://gitea.com/gitea/runner/releases.rss` | `https://gitea.com/gitea/runner/releases.rss` |
|
||||
|
||||
Only repositories under `gitea.com/gitea/` are supported.
|
||||
|
||||
## Error Responses
|
||||
|
||||
Errors are returned as `{"detail": "<message>"}`.
|
||||
|
||||
| Status | Cause | Example `detail` |
|
||||
|---|---|---|
|
||||
| `401` | Missing or invalid bearer key | `Unauthorized` |
|
||||
| `403` | Key is valid but not allowed on this route | `Forbidden` |
|
||||
| `422` | Invalid `X-User-Id` or body | `type: Input should be 'royalroad' or 'gitea'` |
|
||||
| `422` | URL does not match the feed type | `A gitea feed must be on gitea.com` |
|
||||
| `422` | URL has the wrong path | `Unrecognised feed path (expected e.g. royalroad.com/fiction/118891)` |
|
||||
| `422` | The feed does not exist | `The feed does not exist: <url>` |
|
||||
| `502` | The feed could not be checked right now | `Could not verify the feed right now, try again later: <url>` |
|
||||
| `500` | Internal error, or a check run with failed feeds | |
|
||||
|
||||
|
||||
## Notification Format
|
||||
|
||||
Messages sent to the backend service:
|
||||
One message is sent to `backend-api-internal` per subscriber:
|
||||
|
||||
```json
|
||||
{
|
||||
@@ -79,11 +229,7 @@ Messages sent to the backend service:
|
||||
}
|
||||
```
|
||||
|
||||
**Headers:**
|
||||
```http
|
||||
X-API-Key-Internal: <api_key>
|
||||
Content-Type: application/json
|
||||
```
|
||||
Gitea releases use the category `utility`. When a subscription has no custom name, the title is the feed title for Royal Road and the repository path (for example `gitea.com/gitea/runner`) for Gitea.
|
||||
|
||||
## Monitoring
|
||||
|
||||
|
||||
+2
-2
@@ -10,11 +10,11 @@ feedparser==6.0.14
|
||||
feedparser-sgmllib==2.1.0
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
lib-db-module==0.3.0
|
||||
lib-db-module==0.4.0
|
||||
lib-secret-manager==0.3.1
|
||||
lib-send-notifications==0.1.0
|
||||
lib-service-auth==0.1.0
|
||||
mysql-connector-python==9.4.0
|
||||
mysql-connector-python==26.10.0
|
||||
opentelemetry-api==1.45.1
|
||||
prometheus_client==0.26.0
|
||||
pydantic==2.14.1
|
||||
|
||||
@@ -0,0 +1,35 @@
|
||||
class FeedError(Exception):
|
||||
"""Base class for every feed problem the API can explain to a client."""
|
||||
status_code = 400
|
||||
|
||||
|
||||
class InvalidFeedUrlError(FeedError, ValueError):
|
||||
"""The URL is malformed or doesn't fit the chosen feed type.
|
||||
|
||||
Subclasses ValueError so Pydantic reports it as a normal 422 validation error.
|
||||
"""
|
||||
status_code = 422
|
||||
|
||||
|
||||
class WrongHostError(InvalidFeedUrlError):
|
||||
def __init__(self, feed_type: str, expected_host: str):
|
||||
super().__init__(f"A {feed_type} feed must be on {expected_host}")
|
||||
|
||||
|
||||
class UnsupportedPathError(InvalidFeedUrlError):
|
||||
def __init__(self, hint: str):
|
||||
super().__init__(f"Unrecognised feed path ({hint})")
|
||||
|
||||
|
||||
class FeedNotFoundError(FeedError):
|
||||
status_code = 422
|
||||
|
||||
def __init__(self, url: str):
|
||||
super().__init__(f"The feed does not exist: {url}")
|
||||
|
||||
|
||||
class FeedCheckFailedError(FeedError):
|
||||
status_code = 502
|
||||
|
||||
def __init__(self, url: str):
|
||||
super().__init__(f"Could not verify the feed right now, try again later: {url}")
|
||||
+48
-34
@@ -1,11 +1,11 @@
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from typing import Tuple
|
||||
from typing import assert_never
|
||||
|
||||
import feedparser
|
||||
import requests
|
||||
from pydantic import BaseModel
|
||||
|
||||
from feed_errors import FeedCheckFailedError, FeedNotFoundError
|
||||
from models import FeedEntryType, FeedUpdate
|
||||
from simple_logger_handler import setup_logger
|
||||
|
||||
logger = setup_logger(__name__)
|
||||
@@ -14,14 +14,6 @@ _GITEA_INSTANCE_BASE_URL="https://git.gansejunge.com"
|
||||
_GITEA_REPOSITORY_OWNER="utility"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class RoyalRoadStory(BaseModel):
|
||||
chapter_title: str
|
||||
link: str
|
||||
story_title: str
|
||||
chapter_id: str
|
||||
|
||||
|
||||
def get_rss_feed_data(rss_feed_url: str, max_retries: int = 3) -> feedparser.FeedParserDict:
|
||||
"""
|
||||
Fetches information from any RSS feed, with retries on network-related errors.
|
||||
@@ -62,26 +54,28 @@ def get_rss_feed_data(rss_feed_url: str, max_retries: int = 3) -> feedparser.Fee
|
||||
logger.debug(f"[Feed] Parsed {len(feed.entries)} entries from {rss_feed_url}")
|
||||
return feed
|
||||
|
||||
def grab_gitea_release_information(rss_feed_url: str) -> Tuple[str, str]:
|
||||
def grab_gitea_release_information(rss_feed_url: str) -> FeedUpdate:
|
||||
"""
|
||||
Fetches the latest release information from a Gitea releases RSS feed.
|
||||
|
||||
Returns:
|
||||
A tuple: (release_title, release_link)
|
||||
FeedUpdate:
|
||||
|
||||
Raises:
|
||||
ValueError: If the feed has no entries.
|
||||
"""
|
||||
feed = get_rss_feed_data(rss_feed_url)
|
||||
entry = feed.entries[0]
|
||||
title = entry["title"]
|
||||
link = entry["link"]
|
||||
feed_update = FeedUpdate (
|
||||
entry_id = feed.entries[0]["link"],
|
||||
feed_title = feed.feed.title,
|
||||
body_title = feed.entries[0]["title"],
|
||||
link = feed.entries[0]["link"],
|
||||
)
|
||||
logger.info(f"[Feed] Latest release for feed '{rss_feed_url}': {feed_update.feed_title}")
|
||||
logger.debug(f"[Feed] Latest release for feed '{rss_feed_url}': {feed_update.feed_title} -> {feed_update.link}")
|
||||
return feed_update
|
||||
|
||||
logger.info(f"[Feed] Latest release for feed '{rss_feed_url}': {title}")
|
||||
logger.debug(f"[Feed] Latest release for feed '{rss_feed_url}': {title} -> {link}")
|
||||
return title, link
|
||||
|
||||
def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
||||
def grab_royal_road_chapter_information(rss_feed_url: str) -> FeedUpdate :
|
||||
"""
|
||||
Fetches the latest chapter information from a Royalroad RSS feed.
|
||||
|
||||
@@ -89,7 +83,7 @@ def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
||||
rss_feed_url: URL of the Royalroad story's RSS feed.
|
||||
|
||||
Returns:
|
||||
A tuple: (chapter_title, chapter_link, story_title, chapter_id)
|
||||
FeedUpdate
|
||||
|
||||
Raises:
|
||||
ValueError: If the feed has no entries.
|
||||
@@ -97,19 +91,19 @@ def grab_royal_road_chapter_information(rss_feed_url: str) -> RoyalRoadStory :
|
||||
feed = get_rss_feed_data(rss_feed_url)
|
||||
latest_chapter_data = feed.entries[0]
|
||||
chapter_title_with_story_title = latest_chapter_data["title"]
|
||||
chapter_title = chapter_title_with_story_title.removeprefix(feed.feed.title).removeprefix(" - ")
|
||||
body_title = chapter_title_with_story_title.removeprefix(feed.feed.title).removeprefix(" - ")
|
||||
|
||||
royal_road_story = RoyalRoadStory (
|
||||
chapter_title = chapter_title,
|
||||
feed_update = FeedUpdate (
|
||||
entry_id = str(latest_chapter_data["id"]),
|
||||
feed_title = feed.feed.title,
|
||||
body_title = body_title,
|
||||
link = latest_chapter_data["link"],
|
||||
story_title = feed.feed.title,
|
||||
chapter_id = str(latest_chapter_data["id"])
|
||||
)
|
||||
|
||||
logger.info(f"[Feed] Latest chapter for story '{royal_road_story.story_title}': {royal_road_story.chapter_title}")
|
||||
logger.debug(f"[Feed] Latest chapter for story '{royal_road_story.story_title}' (RSS URL {rss_feed_url}):"
|
||||
f"{royal_road_story.chapter_title} (ID {royal_road_story.chapter_id}) -> {royal_road_story.link}")
|
||||
return royal_road_story
|
||||
logger.info(f"[Feed] Latest chapter for story '{feed_update.feed_title}': {feed_update.body_title}")
|
||||
logger.debug(f"[Feed] Latest chapter for story '{feed_update.feed_title}' (RSS URL {rss_feed_url}):"
|
||||
f"{feed_update.body_title} (ID {feed_update.entry_id}) -> {feed_update.link}")
|
||||
return feed_update
|
||||
|
||||
def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
||||
url = (
|
||||
@@ -122,7 +116,7 @@ def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
||||
payload = {
|
||||
"ref": "main",
|
||||
"inputs": {
|
||||
"version": version.lstrip("v"),
|
||||
"version": version.removeprefix("v"),
|
||||
"revision": "1",
|
||||
},
|
||||
}
|
||||
@@ -138,10 +132,30 @@ def build_gitea_runner(version: str, gitea_token: str) -> bool:
|
||||
logger.warning(f"[Feed] Failed to run Gitea-Runner build: {e}")
|
||||
return False
|
||||
|
||||
def ensure_feed_exists(url: str, timeout: tuple[float, float] = (3.0, 5.0)) -> None:
|
||||
try:
|
||||
with requests.get(url, stream=True, timeout=timeout, allow_redirects=False) as response:
|
||||
status = response.status_code
|
||||
except requests.RequestException as e:
|
||||
raise FeedCheckFailedError(url) from e
|
||||
if status == 404:
|
||||
raise FeedNotFoundError(url)
|
||||
if status != 200:
|
||||
raise FeedCheckFailedError(url)
|
||||
|
||||
def fetch_latest(feed_type: FeedEntryType, url: str) -> FeedUpdate:
|
||||
match feed_type:
|
||||
case FeedEntryType.ROYALROAD:
|
||||
return grab_royal_road_chapter_information(url)
|
||||
case FeedEntryType.GITEA:
|
||||
return grab_gitea_release_information(url)
|
||||
case _:
|
||||
assert_never(feed_type)
|
||||
|
||||
if __name__ == "__main__":
|
||||
#rsssss="https://gitea.com/gitea/runner/releases.rss"
|
||||
rsssss="https://gitea.com/gitea/runner/releases.rss"
|
||||
#print(get_rss_feed_data(rsssss))
|
||||
#print(grab_gitea_release_information(rsssss))
|
||||
print(grab_gitea_release_information(rsssss))
|
||||
id="118891"
|
||||
rss_feed_url = f"https://www.royalroad.com/fiction/syndication/{id}"
|
||||
print(grab_royal_road_chapter_information(rss_feed_url))
|
||||
+142
-19
@@ -4,22 +4,29 @@ from contextlib import asynccontextmanager
|
||||
|
||||
import uvicorn
|
||||
from fastapi import APIRouter, Depends, FastAPI, HTTPException, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
from fastapi.responses import JSONResponse
|
||||
from requests.exceptions import RequestException
|
||||
|
||||
from auth_config import Caller, auth
|
||||
from db_module import close_connection_pool, create_connection_pool, get_db, start_healthcheck_thread
|
||||
from feed_errors import FeedCheckFailedError, FeedError
|
||||
from feed_handler import (
|
||||
build_gitea_runner,
|
||||
ensure_feed_exists,
|
||||
fetch_latest,
|
||||
grab_gitea_release_information,
|
||||
grab_royal_road_chapter_information,
|
||||
)
|
||||
from metrics_server import REQUEST_COUNTER, metrics_api
|
||||
from models import FeedEntry
|
||||
from secret_manager import cleanup_secret_files, get_secret
|
||||
from send_notifications import NotificationClient, NotificationError, NotificationUnavailableError
|
||||
from service_auth import UserId
|
||||
from simple_logger_handler import LOG_LEVEL, setup_logger
|
||||
|
||||
logger = setup_logger(__name__)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
logger.info("[FastAPI] Starting application...")
|
||||
@@ -53,6 +60,31 @@ api = FastAPI(
|
||||
dependencies=[Depends(auth.authenticate)],
|
||||
)
|
||||
|
||||
@api.exception_handler(FeedError)
|
||||
async def feed_error_handler(request: Request, exc: FeedError):
|
||||
return JSONResponse(status_code=exc.status_code, content={"detail":str(exc)})
|
||||
|
||||
@api.exception_handler(RequestValidationError)
|
||||
async def validation_error_handler(request: Request, exc: RequestValidationError):
|
||||
errors = exc.errors()
|
||||
|
||||
# 1) If one of our own FeedErrors caused it, use its message as is
|
||||
for err in errors:
|
||||
original = (err.get("ctx") or {}).get("error")
|
||||
if isinstance(original, FeedError):
|
||||
return JSONResponse(
|
||||
status_code=original.status_code,
|
||||
content={"detail": str(original)},
|
||||
)
|
||||
|
||||
# 2) Anything else (missing field, bad enum value, bad header): flatten to one string
|
||||
messages = []
|
||||
for err in errors:
|
||||
field = ".".join(str(p) for p in err["loc"] if p != "body")
|
||||
msg = err["msg"].removeprefix("Value error, ")
|
||||
messages.append(f"{field}: {msg}" if field else msg)
|
||||
return JSONResponse(status_code=422, content={"detail": "; ".join(messages)})
|
||||
|
||||
@api.middleware("http")
|
||||
async def prometheus_middleware(request: Request, call_next):
|
||||
logger.debug(f"[FastAPI] Incoming request: {request.method} {request.url.path}")
|
||||
@@ -103,13 +135,13 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
||||
for feed_id, rss_feed, last_entry in feeds:
|
||||
results["checked"] += 1
|
||||
try:
|
||||
release_title, link = grab_gitea_release_information(rss_feed)
|
||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={link}")
|
||||
gitea_release_information = grab_gitea_release_information(rss_feed)
|
||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={gitea_release_information.entry_id}")
|
||||
|
||||
if link == last_entry:
|
||||
if gitea_release_information.entry_id == last_entry:
|
||||
continue
|
||||
|
||||
logger.info(f"[FastAPI] Release change for ID {feed_id}: {release_title}")
|
||||
logger.info(f"[FastAPI] Release change for ID {feed_id}: {gitea_release_information.body_title}")
|
||||
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute(
|
||||
@@ -122,13 +154,13 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
||||
|
||||
sent = 0
|
||||
failed = 0
|
||||
repo = gitea_release_information.link.split("/releases")[0].removeprefix("https://")
|
||||
for user_id,feed_name in subscribers:
|
||||
try:
|
||||
repo = link.split("/releases")[0].removeprefix("https://")
|
||||
request.app.state.notifier.send(
|
||||
title=feed_name or repo,
|
||||
body=release_title,
|
||||
link=link,
|
||||
body=gitea_release_information.body_title,
|
||||
link=gitea_release_information.link,
|
||||
category="utility",
|
||||
user_id=user_id,
|
||||
)
|
||||
@@ -149,14 +181,14 @@ def get_release_information(request: Request, db = Depends(get_db)):
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
||||
(link, feed_id),
|
||||
(gitea_release_information.entry_id, feed_id),
|
||||
)
|
||||
db.commit()
|
||||
logger.debug(f"[FastAPI] Notification sent and database updated for Gitea feed ID {feed_id}")
|
||||
results["new"] += 1
|
||||
|
||||
if link.startswith("https://gitea.com/gitea/runner/releases/tag/v"):
|
||||
gitea_build_success = build_gitea_runner(release_title, request.app.state.gitea_token)
|
||||
if gitea_release_information.link.startswith("https://gitea.com/gitea/runner/releases/tag/v"):
|
||||
gitea_build_success = build_gitea_runner(gitea_release_information.body_title, request.app.state.gitea_token)
|
||||
logger.info(f"[FastAPI] Gitea build ran successfully:{gitea_build_success}")
|
||||
|
||||
except Exception as e:
|
||||
@@ -205,12 +237,12 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
||||
results["checked"] += 1
|
||||
try:
|
||||
story = grab_royal_road_chapter_information(rss_feed)
|
||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={story.chapter_id}")
|
||||
logger.debug(f"[FastAPI] ID:{feed_id},RSS Feed {rss_feed}: db={last_entry}, online={story.entry_id}")
|
||||
|
||||
if story.chapter_id == last_entry:
|
||||
if story.entry_id == last_entry:
|
||||
continue
|
||||
|
||||
logger.info(f"[FastAPI] New chapter detected for story ID {feed_id}: {story.story_title}")
|
||||
logger.info(f"[FastAPI] New chapter detected for story ID {feed_id}: {story.feed_title}")
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute (
|
||||
"SELECT user_id, feed_name "
|
||||
@@ -225,8 +257,8 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
||||
for user_id, feed_name in subscribers:
|
||||
try:
|
||||
request.app.state.notifier.send(
|
||||
title=feed_name or story.story_title,
|
||||
body=story.chapter_title,
|
||||
title=feed_name or story.feed_title,
|
||||
body=story.body_title,
|
||||
link=story.link,
|
||||
category="royal-road",
|
||||
user_id=user_id,
|
||||
@@ -235,11 +267,11 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
||||
except NotificationUnavailableError:
|
||||
failed = len(subscribers) - sent
|
||||
logger.error (f"[FastAPI] Notifications backend unavailable, skipping "
|
||||
f"{failed} remaining user(s) for story={story.story_title}")
|
||||
f"{failed} remaining user(s) for story={story.feed_title}")
|
||||
break
|
||||
except NotificationError as e:
|
||||
failed += 1
|
||||
logger.error(f"[FastAPI] Failed to send notification to user_id={user_id} for story={story.story_title} "
|
||||
logger.error(f"[FastAPI] Failed to send notification to user_id={user_id} for story={story.feed_title} "
|
||||
f"status={getattr(e, 'status_code',None)}")
|
||||
if subscribers and sent == 0:
|
||||
results["errors"] += 1
|
||||
@@ -248,7 +280,7 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"UPDATE rss_feeds SET last_entry = %s WHERE id = %s",
|
||||
(story.chapter_id, feed_id),
|
||||
(story.entry_id, feed_id),
|
||||
)
|
||||
db.commit()
|
||||
logger.debug(f"[FastAPI] Notification sent and database updated for story ID {feed_id}")
|
||||
@@ -269,6 +301,97 @@ def get_chapters(request: Request, db = Depends(get_db)):
|
||||
logger.info(f"[FastAPI] Royalroad check complete: {results}")
|
||||
return results
|
||||
|
||||
@user_router.post("/add/feed")
|
||||
def add_feed(
|
||||
user_id: UserId,
|
||||
feed_entry: FeedEntry,
|
||||
db = Depends(get_db)
|
||||
):
|
||||
url = feed_entry.rss_feed_url
|
||||
logger.info(f"[FastAPI] Subscribe request: RSS URL={url}, user_id={user_id}")
|
||||
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute("SELECT id FROM rss_feeds WHERE rss_feed = %s", (url,),)
|
||||
known = cursor.fetchone()
|
||||
|
||||
last_entry_id = None
|
||||
if known is None:
|
||||
ensure_feed_exists(url)
|
||||
logger.info(f"[FastApi] Adding new RSS feed, type={feed_entry.type}, rss_feed_url={url}")
|
||||
try:
|
||||
last_entry_id = fetch_latest(feed_entry.type, url).entry_id
|
||||
except (ValueError, RequestException) as e:
|
||||
logger.warning(f"[FastAPI] Could not read latest entry for {url}, storing none")
|
||||
raise FeedCheckFailedError(url) from e
|
||||
try:
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"INSERT into rss_feeds (type,rss_feed,last_entry) VALUES (%s,%s,%s) "
|
||||
"ON DUPLICATE KEY UPDATE id = LAST_INSERT_ID(id), active = 1",
|
||||
(feed_entry.type, url, last_entry_id),
|
||||
)
|
||||
feed_id = cursor.lastrowid
|
||||
|
||||
cursor.execute (
|
||||
"INSERT INTO rss_feed_subscriptions (user_id, rss_feed_id, feed_name) "
|
||||
"VALUES (%s, %s, %s) "
|
||||
"ON DUPLICATE KEY UPDATE active = 1, feed_name = VALUES(feed_name)",
|
||||
(user_id, feed_id, feed_entry.custom_name),
|
||||
)
|
||||
db.commit()
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
logger.exception(f"[FastAPI] Failed to subscribe user_id={user_id} to {url}")
|
||||
raise HTTPException(status_code=500) from e
|
||||
|
||||
logger.info(f"[FastAPI] user_id={user_id} subscribed to feed_id={feed_id}, url={url}")
|
||||
return {"status":"ok","detail":"Subscribed to feed","feed_url":url}
|
||||
|
||||
@user_router.post("/delete/feed")
|
||||
def delete_feed(
|
||||
user_id: UserId,
|
||||
feed_entry: FeedEntry,
|
||||
db = Depends(get_db),
|
||||
):
|
||||
url = feed_entry.rss_feed_url
|
||||
logger.info(f"[FastAPI] Unsubscribe request: RSS URL={url}, user_id={user_id}")
|
||||
|
||||
try:
|
||||
with db.cursor() as cursor:
|
||||
cursor.execute(
|
||||
"SELECT id FROM rss_feeds WHERE rss_feed = %s",
|
||||
(url,),
|
||||
)
|
||||
known = cursor.fetchone()
|
||||
|
||||
if known is None:
|
||||
db.rollback()
|
||||
return {"status":"ok","detail":"Unsubscribed to feed","feed_url":url}
|
||||
|
||||
feed_id = known[0]
|
||||
cursor.execute(
|
||||
"UPDATE rss_feed_subscriptions SET active = 0 "
|
||||
"WHERE user_id = %s AND rss_feed_id = %s",
|
||||
(user_id, feed_id),
|
||||
)
|
||||
|
||||
cursor.execute(
|
||||
"SELECT 1 from rss_feed_subscriptions "
|
||||
"WHERE rss_feed_id = %s AND active = 1 LIMIT 1",
|
||||
(feed_id,),
|
||||
)
|
||||
if cursor.fetchone() is None:
|
||||
cursor.execute(
|
||||
"UPDATE rss_feeds SET active = 0 WHERE id = %s",
|
||||
(feed_id,),
|
||||
)
|
||||
|
||||
db.commit()
|
||||
except Exception as e:
|
||||
db.rollback()
|
||||
logger.exception(f"[FastAPI] Failed to unsubscribe user_id={user_id} to {url}")
|
||||
raise HTTPException(status_code=500) from e
|
||||
return {"status":"ok","detail":"Unsubscribed to feed","feed_url":url}
|
||||
api.include_router(user_router)
|
||||
api.include_router(scheduler_router)
|
||||
|
||||
|
||||
@@ -0,0 +1,82 @@
|
||||
from dataclasses import dataclass
|
||||
from enum import StrEnum
|
||||
from typing import Self
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from pydantic import BaseModel, Field, field_validator, model_validator
|
||||
|
||||
from feed_errors import InvalidFeedUrlError, UnsupportedPathError, WrongHostError
|
||||
|
||||
REPO_CHARS = frozenset("abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789_.-")
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class FeedUpdate():
|
||||
entry_id: str
|
||||
feed_title: str
|
||||
body_title: str
|
||||
link: str | None = None
|
||||
|
||||
class FeedEntryType(StrEnum):
|
||||
ROYALROAD = "royalroad"
|
||||
GITEA = "gitea"
|
||||
|
||||
|
||||
def _path_segments(raw: str, feed_type: FeedEntryType, host: str) -> list[str]:
|
||||
"""Check scheme and host, then return the non-empty path segments."""
|
||||
raw = raw.strip()
|
||||
if "://" not in raw:
|
||||
raw = "https://" + raw # allow "royalroad.com/fiction/118891"
|
||||
try:
|
||||
parts = urlsplit(raw)
|
||||
except ValueError as e:
|
||||
raise InvalidFeedUrlError("The URL could not be parsed") from e
|
||||
if parts.scheme not in ("http", "https"):
|
||||
raise InvalidFeedUrlError("The URL must use http or https")
|
||||
# Comparing the whole netloc also rejects ports and user@host
|
||||
if parts.netloc.lower().removeprefix("www.") != host:
|
||||
raise WrongHostError(feed_type.value, host)
|
||||
return [s for s in parts.path.split("/") if s]
|
||||
|
||||
|
||||
def _royalroad_feed(segments: list[str]) -> str:
|
||||
# fiction/<id>[/<slug>]
|
||||
if len(segments) >= 2 and segments[0] == "fiction":
|
||||
story_id = segments[1]
|
||||
if story_id.isascii() and story_id.isdigit():
|
||||
return f"https://www.royalroad.com/fiction/syndication/{int(story_id)}"
|
||||
raise UnsupportedPathError("expected e.g. royalroad.com/fiction/118891")
|
||||
|
||||
|
||||
def _gitea_feed(segments: list[str]) -> str:
|
||||
# gitea/<repo>[/releases | /releases.rss]
|
||||
if len(segments) >= 2 and segments[0] == "gitea":
|
||||
repo, rest = segments[1], segments[2:]
|
||||
repo_ok = repo[0] not in ".-" and set(repo) <= REPO_CHARS
|
||||
if repo_ok and rest in ([], ["releases"], ["releases.rss"]):
|
||||
return f"https://gitea.com/gitea/{repo}/releases.rss"
|
||||
raise UnsupportedPathError("expected e.g. gitea.com/gitea/runner")
|
||||
|
||||
|
||||
# To support a new feed type: add an enum member, a function above, add a case tofetch_latest and a line here.
|
||||
_RULES = {
|
||||
FeedEntryType.ROYALROAD: ("royalroad.com", _royalroad_feed),
|
||||
FeedEntryType.GITEA: ("gitea.com", _gitea_feed),
|
||||
}
|
||||
|
||||
|
||||
class FeedEntry(BaseModel):
|
||||
type: FeedEntryType
|
||||
rss_feed_url: str = Field(max_length=512)
|
||||
custom_name: str | None = Field(default=None, max_length=255)
|
||||
|
||||
@field_validator("custom_name")
|
||||
@classmethod
|
||||
def blank_name_is_none(cls, v: str | None) -> str | None:
|
||||
return (v or "").strip() or None
|
||||
|
||||
@model_validator(mode="after")
|
||||
def normalise_feed_url(self) -> Self:
|
||||
host, build_feed_url = _RULES[self.type]
|
||||
segments = _path_segments(self.rss_feed_url, self.type, host)
|
||||
self.rss_feed_url = build_feed_url(segments)
|
||||
return self
|
||||
Reference in new issue
Block a user