commit a1072dcb977fcc06f3da5a2dfc741c8186427c13 Author: Florian Date: Mon Nov 2 19:55:04 2026 +0100 First version diff --git a/1.getting-started.md b/1.getting-started.md new file mode 100644 index 0000000..04b434c --- /dev/null +++ b/1.getting-started.md @@ -0,0 +1,114 @@ +# Getting Started + + +## 1. Create a Git repository + +Start by creating a repository on GitHub or another Git hosting platform. + +- Give it a descriptive name. +- Add a `README.md` explaining what the project does. +- Add a `.gitignore` to prevent unnecessary files from being committed. Github has a Python template you should use, which prevents cache or build data to be commited +- Commit changes regularly with descriptive commit messages. + +## 2. Recommended project structure + +```text +my-project/ +├── src/ +│ ├── main.py +│ ├── utils.py +│ └── ... +├── .venv/ +├── .gitignore +├── requirements.txt +└── README.md +``` +## 3. requirements.txt + +Use `requirements.txt` to keep track of external Python packages your project depends on. + +Example: + +```text +requests==2.32.5 +beautifulsoup4==4.14.2 +``` + +- One package per line. +- Pin versions to make installations reproducible. +- Update the file whenever you add or remove dependencies. + +Install all dependencies with: + +```bash +pip install -r requirements.txt +``` +# Next step: Set up a virtual environment + +A virtual environment (`venv`) creates an isolated Python environment for your project. + +This means: +- Each project can have its own dependencies. +- Installing packages won't affect your system Python installation. +- Different projects can use different package versions. + +## 1. Check your Python installation + +```bash +python3 --version +``` + + +## 2. Create the virtual environment + +Navigate to your project directory: + +```bash +cd my-project +``` + +Create the environment: + +```bash +python3 -m venv .venv +``` +This creates a `.venv/` directory containing the isolated Python environment. + +## 3. Activate the virtual environment + + +```bash +source .venv/bin/activate +``` +Once activated, your terminal should show something similar to: + +```text +(.venv) user@computer:~/my-project$ +``` + +## 4. Install dependencies + +With the virtual environment activated: + +```bash +pip install -r requirements.txt +``` + +You can now install additional packages: + +```bash +pip install requests +``` + +If you add a new dependency, update `requirements.txt` by running `pip freeze` which lists all installed packages.: + +```bash +pip freeze > requirements.txt +``` +## 5. Deactivate the environment + +When you're finished working or want to change to a different virtual environment: + +```bash +deactivate +``` \ No newline at end of file diff --git a/2.backend-api.md b/2.backend-api.md new file mode 100644 index 0000000..b524a30 --- /dev/null +++ b/2.backend-api.md @@ -0,0 +1,161 @@ + +# Backend API – Getting Started + +## 1. Project overview + +The `backend-api` is a small REST API responsible for receiving, storing and delivering notifications. + +It uses: +- **FastAPI:** Python web framework for building REST APIs. +- **Uvicorn:** Server used to run the FastAPI application. +- **SQLite:** Lightweight database for storing notifications. +- **Pydantic:** Data validation for incoming API requests (included with FastAPI). + +## 2. API endpoints + +The application requires two POST endpoints. POST is used when the application call changes something, GET when it just retrieves information. + +| Method | Endpoint | Description | +|---|---|---| +| POST | `/notifications` | Receive and store a notification. | +| POST | `/notifications/deliver` | Retrieve pending notifications and mark them as delivered. | + +### Receiving notifications + +`POST /notifications` + +Accepts a JSON payload containing the notification information. + +Example request: + +```json +{ + "title": "New notification", + "body": "Something happened!", + "link": "https://example.com", + "timestamp": 1791626400 +} +``` + +The API validates the payload and saves it into the `notifications` database. + +Each notification should have a `delivered` flag, initially set to `false`. + +### Getting notifications + +`POST /notifications/deliver` + +This endpoint retrieves all notifications where `delivered = false`. + +The application should: +1. Query the database for undelivered notifications. +2. Set their `delivered` flag to `true`. +3. Commit the database transaction. +4. Return the retrieved notifications as JSON. + +Example response: + +```json +[ + { + "id": 1, + "title": "New notification", + "body": "Something happened!", + "link": "https://example.com", + "timestamp": 1791626400, + "delivered": true + } +] +``` + +If there are no pending notifications, return an empty array. + +**Important:** The database update should happen before returning the notifications. This prevents the same notifications from being delivered again on subsequent requests. + +## 3. Database structure + +For the initial implementation, use SQLite. + +The database will be stored in a local file called `notifications.db`. + +The `notifications` table should contain: + +| Column | Type | Description | +|---|---|---| +| `id` | INTEGER | Unique notification identifier (primary key). | +| `title` | TEXT | Notification title. | +| `body` | TEXT | Notification content. | +| `link` | TEXT | Optional URL associated with the notification. | +| `timestamp` | INTEGER | Unix timestamp of the notification. | +| `delivered` | BOOLEAN | Indicates whether the notification was delivered. | + +--- + +# 4. Minimal working implementation + +The following implementation gives you a baseline for starting the server + +## requirements.txt + +```text +fastapi +uvicorn +``` + +## src/main.py + +```python +from contextlib import asynccontextmanager +from fastapi import FastAPI +from pydantic import BaseModel #use this to create the Notification class +from db_module import initialize_database, get_database #Implement these yourself with SqLite3 + + +@asynccontextmanager +async def lifespan(app: FastAPI): + initialize_database() + yield + + +app = FastAPI(lifespan=lifespan) + +@app.post("/notifications") +def receive_notification(notification: Notification): + + + +@app.post("/notifications/deliver") +def deliver_notifications(): + +``` + +## 6. Run the application + +Make sure the virtual environment is activated and the dependencies are installed. + +Start the application from the project root: + +```bash +uvicorn src.main:app --host 0.0.0.0 --port 8100 --reload +``` + +Explanation: + +| Argument | Description | +|---|---| +| `src.main:app` | Imports the FastAPI instance called `app` from `src/main.py`. | +| `--host 0.0.0.0` | Makes the server accessible outside localhost. | +| `--port 8100` | Port on which the API listens. | +| `--reload` | Automatically restarts the server when source files change. | + +The `--reload` option should only be used during development. + +## 7. Test the application + +FastAPI automatically generates interactive API documentation. + +Open: + +http://localhost:8100/docs + +You can use the Swagger UI to execute both POST endpoints without installing additional tools. \ No newline at end of file diff --git a/3.youtube-service.md b/3.youtube-service.md new file mode 100644 index 0000000..87cece9 --- /dev/null +++ b/3.youtube-service.md @@ -0,0 +1,25 @@ + +# YouTube Notification Service – Overview + +You need to use an [external Website like this one](https://channelsurfer.tv/tools/youtube-rss-feed/) to get the RSS feed ID for a channel. +## 1. Order of events + +1. **Connect to the database:** Establish a connection to the database containing the subscribed YouTube channels. +2. **Retrieve subscriptions:** Fetch all subscribed YouTube RSS feed URLs from the database. +3. **Fetch RSS feeds:** Use `feedparser` to query each subscribed feed. +4. **Check for updates:** Compare the latest feed entries against the last known entry stored in the database. +5. **Send notifications:** If a new video is found, send its information (`title`, `body`, `link`, `timestamp`) to the `backend-api` using a POST request. +6. **Update database:** After successfully sending the notification, update the stored latest video information to prevent duplicate notifications. +7. **Close database connection:** Close the database connection once all subscriptions have been processed. + +## 2. Dependencies + +- **feedparser:** Parse YouTube RSS feeds and extract video information. +- **requests:** Send HTTP POST requests to the `backend-api`. +- **Database driver:** Connect to and query the database. + +## 3. Things to improve once it works + +- Handle unavailable RSS feeds without stopping the entire service. +- Handle failed API requests so notifications aren't accidentally marked as processed. +- Log errors and processing information using Python's `logging` module. Use a LOG_LEVEL to set it to either debugging or production(info) diff --git a/4.asura-scans-monitoring.md b/4.asura-scans-monitoring.md new file mode 100644 index 0000000..4525a3d --- /dev/null +++ b/4.asura-scans-monitoring.md @@ -0,0 +1,28 @@ + +# AsuraScans Monitoring – Overview + +## 1. Order of events + +1. **Fetch comic URL:** Use `requests.get(baseurl)` to retrieve the current comic page URL. The URL may change frequently, so avoid hardcoding the chapter page URL. +2. **Parse website:** Use `BeautifulSoup` to parse the HTML and extract the chapter information. +3. **Extract chapter data:** The chapter information is stored inside the HTML element's `props` attribute as JSON. + + ```python + soup = BeautifulSoup(response.text, "html.parser") + + element = soup.select_one("[props*='chapters']") + props = json.loads(element.get("props")) + ``` + +4. **Check for updates:** Compare the latest chapter against the last known chapter stored in the database. +5. **Create notification:** If a new chapter is found, extract its title, URL and timestamp. +6. **Send notification:** Convert the chapter information into the notification payload (`title`, `body`, `link`, `timestamp`) and send it to the `backend-api` using a POST request. +7. **Update database:** After successfully sending the notification, update the stored latest chapter information to prevent duplicate notifications. + +## 2. Dependencies + +- **requests:** Fetch the comic website and send HTTP POST requests to the `backend-api`. +- **beautifulsoup4:** Parse the website HTML and extract chapter information. +- **Database driver:** Connect to and query the database. +- **json:** Parse the chapter data stored in the HTML `props` attribute (included in Python's standard library). +- **datetime:** Convert chapter timestamps if necessary (included in Python's standard library). \ No newline at end of file diff --git a/5.Improvements.md b/5.Improvements.md new file mode 100644 index 0000000..9865cee --- /dev/null +++ b/5.Improvements.md @@ -0,0 +1,10 @@ +# 8. Future improvements + +Once the basic setup is working, consider implementing the following: + +- **MySQL:** Replace SQLite with MySQL for persistent production database storage. +- **Authentication:** Protect endpoints using API keys or token-based authentication. The [backend-api](https://git.gansejunge.com/notifier/backend-api/src/branch/main/src/main.py) has an example +- **Logging:** Replace `print()` statements with Python's `logging` module. +- **Error handling:** Handle database connection failures and invalid requests. +- **Configuration:** Move database paths, ports and other settings into environment variables. +- **Deployment** - Use Docker with docker-compose to run self hosted images from \ No newline at end of file diff --git a/README.md b/README.md new file mode 100644 index 0000000..c650620 --- /dev/null +++ b/README.md @@ -0,0 +1,10 @@ +# Guide for Kaash + +- [1. Getting started](1.getting-started.md) + +- [2. The backbone: Backend-API](2.backend-api.md) + +- [3. First Service: Get Youtube updates](3.youtube-service.md) + +- [4. AsuraScans monitoring](4.asura-scans-monitoring.md) +- [5. Improvements for the future](5.Improvements.md) \ No newline at end of file