First version
This commit is contained in:
commit
a1072dcb97
6 files changed
+348
No files matched your search
@@ -0,0 +1,114 @@
|
||||
# Getting Started
|
||||
|
||||
|
||||
## 1. Create a Git repository
|
||||
|
||||
Start by creating a repository on GitHub or another Git hosting platform.
|
||||
|
||||
- Give it a descriptive name.
|
||||
- Add a `README.md` explaining what the project does.
|
||||
- Add a `.gitignore` to prevent unnecessary files from being committed. Github has a Python template you should use, which prevents cache or build data to be commited
|
||||
- Commit changes regularly with descriptive commit messages.
|
||||
|
||||
## 2. Recommended project structure
|
||||
|
||||
```text
|
||||
my-project/
|
||||
├── src/
|
||||
│ ├── main.py
|
||||
│ ├── utils.py
|
||||
│ └── ...
|
||||
├── .venv/
|
||||
├── .gitignore
|
||||
├── requirements.txt
|
||||
└── README.md
|
||||
```
|
||||
## 3. requirements.txt
|
||||
|
||||
Use `requirements.txt` to keep track of external Python packages your project depends on.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
requests==2.32.5
|
||||
beautifulsoup4==4.14.2
|
||||
```
|
||||
|
||||
- One package per line.
|
||||
- Pin versions to make installations reproducible.
|
||||
- Update the file whenever you add or remove dependencies.
|
||||
|
||||
Install all dependencies with:
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
# Next step: Set up a virtual environment
|
||||
|
||||
A virtual environment (`venv`) creates an isolated Python environment for your project.
|
||||
|
||||
This means:
|
||||
- Each project can have its own dependencies.
|
||||
- Installing packages won't affect your system Python installation.
|
||||
- Different projects can use different package versions.
|
||||
|
||||
## 1. Check your Python installation
|
||||
|
||||
```bash
|
||||
python3 --version
|
||||
```
|
||||
|
||||
|
||||
## 2. Create the virtual environment
|
||||
|
||||
Navigate to your project directory:
|
||||
|
||||
```bash
|
||||
cd my-project
|
||||
```
|
||||
|
||||
Create the environment:
|
||||
|
||||
```bash
|
||||
python3 -m venv .venv
|
||||
```
|
||||
This creates a `.venv/` directory containing the isolated Python environment.
|
||||
|
||||
## 3. Activate the virtual environment
|
||||
|
||||
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
```
|
||||
Once activated, your terminal should show something similar to:
|
||||
|
||||
```text
|
||||
(.venv) user@computer:~/my-project$
|
||||
```
|
||||
|
||||
## 4. Install dependencies
|
||||
|
||||
With the virtual environment activated:
|
||||
|
||||
```bash
|
||||
pip install -r requirements.txt
|
||||
```
|
||||
|
||||
You can now install additional packages:
|
||||
|
||||
```bash
|
||||
pip install requests
|
||||
```
|
||||
|
||||
If you add a new dependency, update `requirements.txt` by running `pip freeze` which lists all installed packages.:
|
||||
|
||||
```bash
|
||||
pip freeze > requirements.txt
|
||||
```
|
||||
## 5. Deactivate the environment
|
||||
|
||||
When you're finished working or want to change to a different virtual environment:
|
||||
|
||||
```bash
|
||||
deactivate
|
||||
```
|
||||
@@ -0,0 +1,161 @@
|
||||
|
||||
# Backend API – Getting Started
|
||||
|
||||
## 1. Project overview
|
||||
|
||||
The `backend-api` is a small REST API responsible for receiving, storing and delivering notifications.
|
||||
|
||||
It uses:
|
||||
- **FastAPI:** Python web framework for building REST APIs.
|
||||
- **Uvicorn:** Server used to run the FastAPI application.
|
||||
- **SQLite:** Lightweight database for storing notifications.
|
||||
- **Pydantic:** Data validation for incoming API requests (included with FastAPI).
|
||||
|
||||
## 2. API endpoints
|
||||
|
||||
The application requires two POST endpoints. POST is used when the application call changes something, GET when it just retrieves information.
|
||||
|
||||
| Method | Endpoint | Description |
|
||||
|---|---|---|
|
||||
| POST | `/notifications` | Receive and store a notification. |
|
||||
| POST | `/notifications/deliver` | Retrieve pending notifications and mark them as delivered. |
|
||||
|
||||
### Receiving notifications
|
||||
|
||||
`POST /notifications`
|
||||
|
||||
Accepts a JSON payload containing the notification information.
|
||||
|
||||
Example request:
|
||||
|
||||
```json
|
||||
{
|
||||
"title": "New notification",
|
||||
"body": "Something happened!",
|
||||
"link": "https://example.com",
|
||||
"timestamp": 1791626400
|
||||
}
|
||||
```
|
||||
|
||||
The API validates the payload and saves it into the `notifications` database.
|
||||
|
||||
Each notification should have a `delivered` flag, initially set to `false`.
|
||||
|
||||
### Getting notifications
|
||||
|
||||
`POST /notifications/deliver`
|
||||
|
||||
This endpoint retrieves all notifications where `delivered = false`.
|
||||
|
||||
The application should:
|
||||
1. Query the database for undelivered notifications.
|
||||
2. Set their `delivered` flag to `true`.
|
||||
3. Commit the database transaction.
|
||||
4. Return the retrieved notifications as JSON.
|
||||
|
||||
Example response:
|
||||
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": 1,
|
||||
"title": "New notification",
|
||||
"body": "Something happened!",
|
||||
"link": "https://example.com",
|
||||
"timestamp": 1791626400,
|
||||
"delivered": true
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
If there are no pending notifications, return an empty array.
|
||||
|
||||
**Important:** The database update should happen before returning the notifications. This prevents the same notifications from being delivered again on subsequent requests.
|
||||
|
||||
## 3. Database structure
|
||||
|
||||
For the initial implementation, use SQLite.
|
||||
|
||||
The database will be stored in a local file called `notifications.db`.
|
||||
|
||||
The `notifications` table should contain:
|
||||
|
||||
| Column | Type | Description |
|
||||
|---|---|---|
|
||||
| `id` | INTEGER | Unique notification identifier (primary key). |
|
||||
| `title` | TEXT | Notification title. |
|
||||
| `body` | TEXT | Notification content. |
|
||||
| `link` | TEXT | Optional URL associated with the notification. |
|
||||
| `timestamp` | INTEGER | Unix timestamp of the notification. |
|
||||
| `delivered` | BOOLEAN | Indicates whether the notification was delivered. |
|
||||
|
||||
---
|
||||
|
||||
# 4. Minimal working implementation
|
||||
|
||||
The following implementation gives you a baseline for starting the server
|
||||
|
||||
## requirements.txt
|
||||
|
||||
```text
|
||||
fastapi
|
||||
uvicorn
|
||||
```
|
||||
|
||||
## src/main.py
|
||||
|
||||
```python
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from pydantic import BaseModel #use this to create the Notification class
|
||||
from db_module import initialize_database, get_database #Implement these yourself with SqLite3
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
initialize_database()
|
||||
yield
|
||||
|
||||
|
||||
app = FastAPI(lifespan=lifespan)
|
||||
|
||||
@app.post("/notifications")
|
||||
def receive_notification(notification: Notification):
|
||||
|
||||
|
||||
|
||||
@app.post("/notifications/deliver")
|
||||
def deliver_notifications():
|
||||
|
||||
```
|
||||
|
||||
## 6. Run the application
|
||||
|
||||
Make sure the virtual environment is activated and the dependencies are installed.
|
||||
|
||||
Start the application from the project root:
|
||||
|
||||
```bash
|
||||
uvicorn src.main:app --host 0.0.0.0 --port 8100 --reload
|
||||
```
|
||||
|
||||
Explanation:
|
||||
|
||||
| Argument | Description |
|
||||
|---|---|
|
||||
| `src.main:app` | Imports the FastAPI instance called `app` from `src/main.py`. |
|
||||
| `--host 0.0.0.0` | Makes the server accessible outside localhost. |
|
||||
| `--port 8100` | Port on which the API listens. |
|
||||
| `--reload` | Automatically restarts the server when source files change. |
|
||||
|
||||
The `--reload` option should only be used during development.
|
||||
|
||||
## 7. Test the application
|
||||
|
||||
FastAPI automatically generates interactive API documentation.
|
||||
|
||||
Open:
|
||||
|
||||
http://localhost:8100/docs
|
||||
|
||||
You can use the Swagger UI to execute both POST endpoints without installing additional tools.
|
||||
@@ -0,0 +1,25 @@
|
||||
|
||||
# YouTube Notification Service – Overview
|
||||
|
||||
You need to use an [external Website like this one](https://channelsurfer.tv/tools/youtube-rss-feed/) to get the RSS feed ID for a channel.
|
||||
## 1. Order of events
|
||||
|
||||
1. **Connect to the database:** Establish a connection to the database containing the subscribed YouTube channels.
|
||||
2. **Retrieve subscriptions:** Fetch all subscribed YouTube RSS feed URLs from the database.
|
||||
3. **Fetch RSS feeds:** Use `feedparser` to query each subscribed feed.
|
||||
4. **Check for updates:** Compare the latest feed entries against the last known entry stored in the database.
|
||||
5. **Send notifications:** If a new video is found, send its information (`title`, `body`, `link`, `timestamp`) to the `backend-api` using a POST request.
|
||||
6. **Update database:** After successfully sending the notification, update the stored latest video information to prevent duplicate notifications.
|
||||
7. **Close database connection:** Close the database connection once all subscriptions have been processed.
|
||||
|
||||
## 2. Dependencies
|
||||
|
||||
- **feedparser:** Parse YouTube RSS feeds and extract video information.
|
||||
- **requests:** Send HTTP POST requests to the `backend-api`.
|
||||
- **Database driver:** Connect to and query the database.
|
||||
|
||||
## 3. Things to improve once it works
|
||||
|
||||
- Handle unavailable RSS feeds without stopping the entire service.
|
||||
- Handle failed API requests so notifications aren't accidentally marked as processed.
|
||||
- Log errors and processing information using Python's `logging` module. Use a LOG_LEVEL to set it to either debugging or production(info)
|
||||
@@ -0,0 +1,28 @@
|
||||
|
||||
# AsuraScans Monitoring – Overview
|
||||
|
||||
## 1. Order of events
|
||||
|
||||
1. **Fetch comic URL:** Use `requests.get(baseurl)` to retrieve the current comic page URL. The URL may change frequently, so avoid hardcoding the chapter page URL.
|
||||
2. **Parse website:** Use `BeautifulSoup` to parse the HTML and extract the chapter information.
|
||||
3. **Extract chapter data:** The chapter information is stored inside the HTML element's `props` attribute as JSON.
|
||||
|
||||
```python
|
||||
soup = BeautifulSoup(response.text, "html.parser")
|
||||
|
||||
element = soup.select_one("[props*='chapters']")
|
||||
props = json.loads(element.get("props"))
|
||||
```
|
||||
|
||||
4. **Check for updates:** Compare the latest chapter against the last known chapter stored in the database.
|
||||
5. **Create notification:** If a new chapter is found, extract its title, URL and timestamp.
|
||||
6. **Send notification:** Convert the chapter information into the notification payload (`title`, `body`, `link`, `timestamp`) and send it to the `backend-api` using a POST request.
|
||||
7. **Update database:** After successfully sending the notification, update the stored latest chapter information to prevent duplicate notifications.
|
||||
|
||||
## 2. Dependencies
|
||||
|
||||
- **requests:** Fetch the comic website and send HTTP POST requests to the `backend-api`.
|
||||
- **beautifulsoup4:** Parse the website HTML and extract chapter information.
|
||||
- **Database driver:** Connect to and query the database.
|
||||
- **json:** Parse the chapter data stored in the HTML `props` attribute (included in Python's standard library).
|
||||
- **datetime:** Convert chapter timestamps if necessary (included in Python's standard library).
|
||||
@@ -0,0 +1,10 @@
|
||||
# 8. Future improvements
|
||||
|
||||
Once the basic setup is working, consider implementing the following:
|
||||
|
||||
- **MySQL:** Replace SQLite with MySQL for persistent production database storage.
|
||||
- **Authentication:** Protect endpoints using API keys or token-based authentication. The [backend-api](https://git.gansejunge.com/notifier/backend-api/src/branch/main/src/main.py) has an example
|
||||
- **Logging:** Replace `print()` statements with Python's `logging` module.
|
||||
- **Error handling:** Handle database connection failures and invalid requests.
|
||||
- **Configuration:** Move database paths, ports and other settings into environment variables.
|
||||
- **Deployment** - Use Docker with docker-compose to run self hosted images from
|
||||
@@ -0,0 +1,10 @@
|
||||
# Guide for Kaash
|
||||
|
||||
- [1. Getting started](1.getting-started.md)
|
||||
|
||||
- [2. The backbone: Backend-API](2.backend-api.md)
|
||||
|
||||
- [3. First Service: Get Youtube updates](3.youtube-service.md)
|
||||
|
||||
- [4. AsuraScans monitoring](4.asura-scans-monitoring.md)
|
||||
- [5. Improvements for the future](5.Improvements.md)
|
||||
Reference in new issue
Block a user