First version
This commit is contained in:
commit
a1072dcb97
6 files changed
+348
No files matched your search
@@ -0,0 +1,114 @@
|
|||||||
|
# Getting Started
|
||||||
|
|
||||||
|
|
||||||
|
## 1. Create a Git repository
|
||||||
|
|
||||||
|
Start by creating a repository on GitHub or another Git hosting platform.
|
||||||
|
|
||||||
|
- Give it a descriptive name.
|
||||||
|
- Add a `README.md` explaining what the project does.
|
||||||
|
- Add a `.gitignore` to prevent unnecessary files from being committed. Github has a Python template you should use, which prevents cache or build data to be commited
|
||||||
|
- Commit changes regularly with descriptive commit messages.
|
||||||
|
|
||||||
|
## 2. Recommended project structure
|
||||||
|
|
||||||
|
```text
|
||||||
|
my-project/
|
||||||
|
├── src/
|
||||||
|
│ ├── main.py
|
||||||
|
│ ├── utils.py
|
||||||
|
│ └── ...
|
||||||
|
├── .venv/
|
||||||
|
├── .gitignore
|
||||||
|
├── requirements.txt
|
||||||
|
└── README.md
|
||||||
|
```
|
||||||
|
## 3. requirements.txt
|
||||||
|
|
||||||
|
Use `requirements.txt` to keep track of external Python packages your project depends on.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
```text
|
||||||
|
requests==2.32.5
|
||||||
|
beautifulsoup4==4.14.2
|
||||||
|
```
|
||||||
|
|
||||||
|
- One package per line.
|
||||||
|
- Pin versions to make installations reproducible.
|
||||||
|
- Update the file whenever you add or remove dependencies.
|
||||||
|
|
||||||
|
Install all dependencies with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
# Next step: Set up a virtual environment
|
||||||
|
|
||||||
|
A virtual environment (`venv`) creates an isolated Python environment for your project.
|
||||||
|
|
||||||
|
This means:
|
||||||
|
- Each project can have its own dependencies.
|
||||||
|
- Installing packages won't affect your system Python installation.
|
||||||
|
- Different projects can use different package versions.
|
||||||
|
|
||||||
|
## 1. Check your Python installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 --version
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## 2. Create the virtual environment
|
||||||
|
|
||||||
|
Navigate to your project directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd my-project
|
||||||
|
```
|
||||||
|
|
||||||
|
Create the environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv .venv
|
||||||
|
```
|
||||||
|
This creates a `.venv/` directory containing the isolated Python environment.
|
||||||
|
|
||||||
|
## 3. Activate the virtual environment
|
||||||
|
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source .venv/bin/activate
|
||||||
|
```
|
||||||
|
Once activated, your terminal should show something similar to:
|
||||||
|
|
||||||
|
```text
|
||||||
|
(.venv) user@computer:~/my-project$
|
||||||
|
```
|
||||||
|
|
||||||
|
## 4. Install dependencies
|
||||||
|
|
||||||
|
With the virtual environment activated:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
You can now install additional packages:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip install requests
|
||||||
|
```
|
||||||
|
|
||||||
|
If you add a new dependency, update `requirements.txt` by running `pip freeze` which lists all installed packages.:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
pip freeze > requirements.txt
|
||||||
|
```
|
||||||
|
## 5. Deactivate the environment
|
||||||
|
|
||||||
|
When you're finished working or want to change to a different virtual environment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
deactivate
|
||||||
|
```
|
||||||
@@ -0,0 +1,161 @@
|
|||||||
|
|
||||||
|
# Backend API – Getting Started
|
||||||
|
|
||||||
|
## 1. Project overview
|
||||||
|
|
||||||
|
The `backend-api` is a small REST API responsible for receiving, storing and delivering notifications.
|
||||||
|
|
||||||
|
It uses:
|
||||||
|
- **FastAPI:** Python web framework for building REST APIs.
|
||||||
|
- **Uvicorn:** Server used to run the FastAPI application.
|
||||||
|
- **SQLite:** Lightweight database for storing notifications.
|
||||||
|
- **Pydantic:** Data validation for incoming API requests (included with FastAPI).
|
||||||
|
|
||||||
|
## 2. API endpoints
|
||||||
|
|
||||||
|
The application requires two POST endpoints. POST is used when the application call changes something, GET when it just retrieves information.
|
||||||
|
|
||||||
|
| Method | Endpoint | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| POST | `/notifications` | Receive and store a notification. |
|
||||||
|
| POST | `/notifications/deliver` | Retrieve pending notifications and mark them as delivered. |
|
||||||
|
|
||||||
|
### Receiving notifications
|
||||||
|
|
||||||
|
`POST /notifications`
|
||||||
|
|
||||||
|
Accepts a JSON payload containing the notification information.
|
||||||
|
|
||||||
|
Example request:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"title": "New notification",
|
||||||
|
"body": "Something happened!",
|
||||||
|
"link": "https://example.com",
|
||||||
|
"timestamp": 1791626400
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
The API validates the payload and saves it into the `notifications` database.
|
||||||
|
|
||||||
|
Each notification should have a `delivered` flag, initially set to `false`.
|
||||||
|
|
||||||
|
### Getting notifications
|
||||||
|
|
||||||
|
`POST /notifications/deliver`
|
||||||
|
|
||||||
|
This endpoint retrieves all notifications where `delivered = false`.
|
||||||
|
|
||||||
|
The application should:
|
||||||
|
1. Query the database for undelivered notifications.
|
||||||
|
2. Set their `delivered` flag to `true`.
|
||||||
|
3. Commit the database transaction.
|
||||||
|
4. Return the retrieved notifications as JSON.
|
||||||
|
|
||||||
|
Example response:
|
||||||
|
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"id": 1,
|
||||||
|
"title": "New notification",
|
||||||
|
"body": "Something happened!",
|
||||||
|
"link": "https://example.com",
|
||||||
|
"timestamp": 1791626400,
|
||||||
|
"delivered": true
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
If there are no pending notifications, return an empty array.
|
||||||
|
|
||||||
|
**Important:** The database update should happen before returning the notifications. This prevents the same notifications from being delivered again on subsequent requests.
|
||||||
|
|
||||||
|
## 3. Database structure
|
||||||
|
|
||||||
|
For the initial implementation, use SQLite.
|
||||||
|
|
||||||
|
The database will be stored in a local file called `notifications.db`.
|
||||||
|
|
||||||
|
The `notifications` table should contain:
|
||||||
|
|
||||||
|
| Column | Type | Description |
|
||||||
|
|---|---|---|
|
||||||
|
| `id` | INTEGER | Unique notification identifier (primary key). |
|
||||||
|
| `title` | TEXT | Notification title. |
|
||||||
|
| `body` | TEXT | Notification content. |
|
||||||
|
| `link` | TEXT | Optional URL associated with the notification. |
|
||||||
|
| `timestamp` | INTEGER | Unix timestamp of the notification. |
|
||||||
|
| `delivered` | BOOLEAN | Indicates whether the notification was delivered. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
# 4. Minimal working implementation
|
||||||
|
|
||||||
|
The following implementation gives you a baseline for starting the server
|
||||||
|
|
||||||
|
## requirements.txt
|
||||||
|
|
||||||
|
```text
|
||||||
|
fastapi
|
||||||
|
uvicorn
|
||||||
|
```
|
||||||
|
|
||||||
|
## src/main.py
|
||||||
|
|
||||||
|
```python
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
from fastapi import FastAPI
|
||||||
|
from pydantic import BaseModel #use this to create the Notification class
|
||||||
|
from db_module import initialize_database, get_database #Implement these yourself with SqLite3
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
initialize_database()
|
||||||
|
yield
|
||||||
|
|
||||||
|
|
||||||
|
app = FastAPI(lifespan=lifespan)
|
||||||
|
|
||||||
|
@app.post("/notifications")
|
||||||
|
def receive_notification(notification: Notification):
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
@app.post("/notifications/deliver")
|
||||||
|
def deliver_notifications():
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
## 6. Run the application
|
||||||
|
|
||||||
|
Make sure the virtual environment is activated and the dependencies are installed.
|
||||||
|
|
||||||
|
Start the application from the project root:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
uvicorn src.main:app --host 0.0.0.0 --port 8100 --reload
|
||||||
|
```
|
||||||
|
|
||||||
|
Explanation:
|
||||||
|
|
||||||
|
| Argument | Description |
|
||||||
|
|---|---|
|
||||||
|
| `src.main:app` | Imports the FastAPI instance called `app` from `src/main.py`. |
|
||||||
|
| `--host 0.0.0.0` | Makes the server accessible outside localhost. |
|
||||||
|
| `--port 8100` | Port on which the API listens. |
|
||||||
|
| `--reload` | Automatically restarts the server when source files change. |
|
||||||
|
|
||||||
|
The `--reload` option should only be used during development.
|
||||||
|
|
||||||
|
## 7. Test the application
|
||||||
|
|
||||||
|
FastAPI automatically generates interactive API documentation.
|
||||||
|
|
||||||
|
Open:
|
||||||
|
|
||||||
|
http://localhost:8100/docs
|
||||||
|
|
||||||
|
You can use the Swagger UI to execute both POST endpoints without installing additional tools.
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
|
||||||
|
# YouTube Notification Service – Overview
|
||||||
|
|
||||||
|
You need to use an [external Website like this one](https://channelsurfer.tv/tools/youtube-rss-feed/) to get the RSS feed ID for a channel.
|
||||||
|
## 1. Order of events
|
||||||
|
|
||||||
|
1. **Connect to the database:** Establish a connection to the database containing the subscribed YouTube channels.
|
||||||
|
2. **Retrieve subscriptions:** Fetch all subscribed YouTube RSS feed URLs from the database.
|
||||||
|
3. **Fetch RSS feeds:** Use `feedparser` to query each subscribed feed.
|
||||||
|
4. **Check for updates:** Compare the latest feed entries against the last known entry stored in the database.
|
||||||
|
5. **Send notifications:** If a new video is found, send its information (`title`, `body`, `link`, `timestamp`) to the `backend-api` using a POST request.
|
||||||
|
6. **Update database:** After successfully sending the notification, update the stored latest video information to prevent duplicate notifications.
|
||||||
|
7. **Close database connection:** Close the database connection once all subscriptions have been processed.
|
||||||
|
|
||||||
|
## 2. Dependencies
|
||||||
|
|
||||||
|
- **feedparser:** Parse YouTube RSS feeds and extract video information.
|
||||||
|
- **requests:** Send HTTP POST requests to the `backend-api`.
|
||||||
|
- **Database driver:** Connect to and query the database.
|
||||||
|
|
||||||
|
## 3. Things to improve once it works
|
||||||
|
|
||||||
|
- Handle unavailable RSS feeds without stopping the entire service.
|
||||||
|
- Handle failed API requests so notifications aren't accidentally marked as processed.
|
||||||
|
- Log errors and processing information using Python's `logging` module. Use a LOG_LEVEL to set it to either debugging or production(info)
|
||||||
@@ -0,0 +1,28 @@
|
|||||||
|
|
||||||
|
# AsuraScans Monitoring – Overview
|
||||||
|
|
||||||
|
## 1. Order of events
|
||||||
|
|
||||||
|
1. **Fetch comic URL:** Use `requests.get(baseurl)` to retrieve the current comic page URL. The URL may change frequently, so avoid hardcoding the chapter page URL.
|
||||||
|
2. **Parse website:** Use `BeautifulSoup` to parse the HTML and extract the chapter information.
|
||||||
|
3. **Extract chapter data:** The chapter information is stored inside the HTML element's `props` attribute as JSON.
|
||||||
|
|
||||||
|
```python
|
||||||
|
soup = BeautifulSoup(response.text, "html.parser")
|
||||||
|
|
||||||
|
element = soup.select_one("[props*='chapters']")
|
||||||
|
props = json.loads(element.get("props"))
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Check for updates:** Compare the latest chapter against the last known chapter stored in the database.
|
||||||
|
5. **Create notification:** If a new chapter is found, extract its title, URL and timestamp.
|
||||||
|
6. **Send notification:** Convert the chapter information into the notification payload (`title`, `body`, `link`, `timestamp`) and send it to the `backend-api` using a POST request.
|
||||||
|
7. **Update database:** After successfully sending the notification, update the stored latest chapter information to prevent duplicate notifications.
|
||||||
|
|
||||||
|
## 2. Dependencies
|
||||||
|
|
||||||
|
- **requests:** Fetch the comic website and send HTTP POST requests to the `backend-api`.
|
||||||
|
- **beautifulsoup4:** Parse the website HTML and extract chapter information.
|
||||||
|
- **Database driver:** Connect to and query the database.
|
||||||
|
- **json:** Parse the chapter data stored in the HTML `props` attribute (included in Python's standard library).
|
||||||
|
- **datetime:** Convert chapter timestamps if necessary (included in Python's standard library).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# 8. Future improvements
|
||||||
|
|
||||||
|
Once the basic setup is working, consider implementing the following:
|
||||||
|
|
||||||
|
- **MySQL:** Replace SQLite with MySQL for persistent production database storage.
|
||||||
|
- **Authentication:** Protect endpoints using API keys or token-based authentication. The [backend-api](https://git.gansejunge.com/notifier/backend-api/src/branch/main/src/main.py) has an example
|
||||||
|
- **Logging:** Replace `print()` statements with Python's `logging` module.
|
||||||
|
- **Error handling:** Handle database connection failures and invalid requests.
|
||||||
|
- **Configuration:** Move database paths, ports and other settings into environment variables.
|
||||||
|
- **Deployment** - Use Docker with docker-compose to run self hosted images from
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
# Guide for Kaash
|
||||||
|
|
||||||
|
- [1. Getting started](1.getting-started.md)
|
||||||
|
|
||||||
|
- [2. The backbone: Backend-API](2.backend-api.md)
|
||||||
|
|
||||||
|
- [3. First Service: Get Youtube updates](3.youtube-service.md)
|
||||||
|
|
||||||
|
- [4. AsuraScans monitoring](4.asura-scans-monitoring.md)
|
||||||
|
- [5. Improvements for the future](5.Improvements.md)
|
||||||
Reference in new issue
Block a user