Skip to content

Commit

Permalink
BREAKING CHANGES: Uncached rework + debrid link support, better url, …
Browse files Browse the repository at this point in the history 
…better cache and many mor

- BREAKING: Make sure to wipe your sqlite or postgres comet DB
- BREAKING: Make sure to re add your addon as the encrypted config changed and old encrypted urls wont work u will get invalid config
  - Improved encrypted config. Its even shorter now.
  - Removed unnecessary b64 encoding causing the encrypted string to be base64 encoded twice.
  - Shortened playback url even more if TOKEN env is provided
- Full Uncached Torrents database refactor
- Full Debrid Link Support + DEBRID_TAKE_FIRST
- All missing uncached features implemented
- Fixed issue with multiple uncached Streams not playing.
  - Its now possible to start Caching a batch of episodes and then play E01 as soon as ready without having to wait for the whole container/season to be cached.
  - This depends on provider, real debrid does not support this but debrid link does.
- Fixed multiple comet addons interfering with another for uncached results.
- Major improvement in multi download management using container and torrent ids for uncached torrents.
- Major uncached file selection improvement. Whole system reworked. Now nearly always right file will be selected.
- Fixed FileResponse finally... (Video length was to short and stremio was buggin out + behaviourHints)
- Added Cache Cleanup Background task. Prevents infinite growing of db. In a normal use case db size should normalize at most watch tv shows / movies
  - This will allow for entries to be deleted that where only cached for one person. E.g. Niche shows/movies.
- Added File names to urls
  - Uncached files names might be missing the file extension as it is unknown until the file is cached
- Multidownload of uncached is no longer possible there should be fixed
  - Potentially fixed [issue](#2) multiple downloads
  • Loading branch information
@Zaarrg
Zaarrg committed Nov 29, 2024
1 parent 077adfc commit b1d7cf2
Show file tree
Hide file tree
Showing 17 changed files with 619 additions and 349 deletions.
2 changes: 2 additions & 0 deletions .stack.env
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ FASTAPI_PORT=8111
FASTAPI_WORKERS=1
CACHE_TTL=86400
UNCACHED_TTL=86400 # Time when uncached results that started downloading and never finished or where never watched will be deleted out of cache
CACHE_WIPE=0 # Time interval of whole Cache Cleanup trigger. 0 = disabled
CACHE_WIPE_TTL=86400 # Time after when all entries in the cache will be deleted
DEBRID_TAKE_FIRST=0 # Returns this amount of results straight from debrid then runs through title match check
GET_TORRENT_TIMEOUT=5
PROXY_DEBRID_STREAM_PASSWORD=####
Expand Down
8 changes: 6 additions & 2 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,16 +8,20 @@

# Fork Disclaimer
- This is a work in progress fork of the official [comet repo](https://github.com/g0ldyy/comet) and will be abandoned once finished and fully implemented there
- This fork aims to add primarly uncached support but also enhance and add features.
- This fork aims to add primarily uncached support but also enhance and add features.
- To check the current progress and updates check the [todo.md](https://github.com/Zaarrg/comet-uncached/blob/main/todo.md)
- Stuff might break and not work frequently!

# Fork Features
- Currently only uncached Support for real debrid
- Currently only uncached Support for real debrid and Debrid Link
- Check [todo.md](https://github.com/Zaarrg/comet-uncached/blob/main/todo.md) for support status
- Advance language search. Search in different languages.
- Advance sorting and preferences. Sort by language, rank, seeders, size or completion
- Customize your results shown in streamio. (Result Order)
- Encryption of config via TOKEN env
- Better Url - Shortened url and filename in url included
- Use TOKEN env to get short playback urls aswell
- Use of token env highly recommend even if encryption not needed allows for shortest possible urls improving player compatibility

# Features
- The only Stremio addon that can Proxy Debrid Streams to allow use of the Debrid Service on multiple IPs at the same time on the same account!
Expand Down
118 changes: 57 additions & 61 deletions comet/api/stream.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,21 +3,23 @@
import json
import time
import uuid
from urllib.parse import quote

import aiohttp
import httpx

import os
import orjson

from fastapi import APIRouter, Request, Header
from fastapi import APIRouter, Request
from fastapi.responses import (
RedirectResponse,
StreamingResponse,
Response,
)

from starlette.background import BackgroundTask
from RTN import Torrent
from RTN import Torrent, parse
from starlette.responses import FileResponse

from comet.debrid.manager import getDebrid
from comet.utils.general import (
Expand All @@ -31,7 +33,7 @@
translate,
get_balanced_hashes,
format_title, add_uncached_files, get_localized_titles, get_language_codes, get_client_ip,
language_to_country_code
language_to_country_code, check_completion, short_encrypt
)
from comet.utils.logger import logger
from comet.utils.models import database, rtn, settings
Expand Down Expand Up @@ -149,16 +151,20 @@ async def stream(request: Request, b64config: str, type: str, id: str):

logger.info(f"Cache expired for {log_name}")

# Deletes torrents matching cachekey without torrentid and after UNCACHED_TTL the ones with a torrentid
# Deletes uncached entries if containerId and torrentId empty - meaning download never started
# Deletes uncached entries if TTL expired not matter the status
expiration_timestamp = int(time.time()) - settings.UNCACHED_TTL
await database.execute(
"""
DELETE FROM uncached_torrents
WHERE cacheKey = :cache_key AND
WHERE cacheKey = :cache_key AND
(
(torrentId IS NULL OR TRIM(torrentId) = '')
OR
(TRIM(torrentId) != '' AND timestamp < :expiration_timestamp)
(
(torrentId IS NULL OR TRIM(torrentId) = '') AND
(containerId IS NULL OR TRIM(containerId) = '')
)
OR
(timestamp < :expiration_timestamp)
)
""",
{"cache_key": cache_key, "expiration_timestamp": expiration_timestamp}
Expand Down Expand Up @@ -189,12 +195,23 @@ async def stream(request: Request, b64config: str, type: str, id: str):
"url": "https://comet.fast",
}
)

# Shorten Config for playback url
short_config = b64config
if settings.TOKEN:
short_config = {
"debridApiKey": config["debridApiKey"],
"debridStreamProxyPassword": config["debridStreamProxyPassword"],
"debridService": config["debridService"]
}
short_config = short_encrypt(json.dumps(short_config), settings.TOKEN)
results = []
for resolution, hash_list in balanced_hashes.items():
for hash in hash_list:
if hash in sorted_ranked_files:
hash_data = sorted_ranked_files[hash]
data = hash_data["data"]
url_friendly_file = quote(data["raw_title"].replace('/', '-'), safe='')
results.append(
{
"name": f"[{debrid_extension}⚡] Comet {data['resolution']}",
Expand All @@ -209,7 +226,7 @@ async def stream(request: Request, b64config: str, type: str, id: str):
if "torrent_size" in data
else None
),
"url": f"{request.url.scheme}://{request.url.netloc}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/{b64config}/playback/{hash}/{data['index']}",
"url": f"{request.url.scheme}://{request.url.netloc}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/{short_config}/playback/{hash}/{data['index']}/{url_friendly_file}",
"behaviorHints": {
"filename": data["raw_title"],
"bingeGroup": "comet|" + hash,
Expand Down Expand Up @@ -381,21 +398,22 @@ async def stream(request: Request, b64config: str, type: str, id: str):
if len(torrents) == 0:
return {"streams": []}

torrents_by_hash = {torrent['InfoHash']: torrent for torrent in torrents if torrent['InfoHash'] is not None}
files = await debrid.get_files(
torrents_by_hash,
list({hash[1] for hash in torrent_hashes if hash[1] is not None}),
type,
season,
episode,
kitsu,
config["debridApiKey"]
)
torrents_by_hash = {torrent["InfoHash"]: torrent for torrent in torrents}
logger.info(
f"{len(files)} cached files found on {config.get('debridService', '?')} for {log_name}"
)
# Adds Uncached Files to files, based on config and cached results
allowed_tracker_ids = config.get('indexersUncached', [])
if allowed_tracker_ids:
await add_uncached_files(files, torrents, cache_key, log_name, allowed_tracker_ids, database, season, episode, kitsu)
await add_uncached_files(files, torrents, cache_key, log_name, allowed_tracker_ids, database, season, episode, kitsu, config)

ranked_files = dict()
for hash in files:
Expand Down Expand Up @@ -427,13 +445,14 @@ async def stream(request: Request, b64config: str, type: str, id: str):
)

for hash in sorted_ranked_files: # needed for caching
torrent_name_parsed = parse(torrents_by_hash[hash]["Title"])
sorted_ranked_files[hash]["data"]["title"] = files[hash]["title"]
sorted_ranked_files[hash]["data"]["torrent_title"] = torrents_by_hash[hash]["Title"]
sorted_ranked_files[hash]["data"]["tracker"] = torrents_by_hash[hash]["Tracker"]
sorted_ranked_files[hash]["data"]["size"] = files[hash]["size"]
sorted_ranked_files[hash]["data"]["uncached"] = files[hash]["uncached"]
if files[hash].get("complete"):
sorted_ranked_files[hash]["data"]["complete"] = files[hash]["complete"]
if files[hash].get("complete") is None:
sorted_ranked_files[hash]["data"]["complete"] = torrent_name_parsed.complete or check_completion(torrent_name_parsed.raw_title, season)
if torrents_by_hash[hash].get("Seeders"):
sorted_ranked_files[hash]["data"]["seeders"] = torrents_by_hash[hash].get("Seeders")
torrent_size = torrents_by_hash[hash]["Size"]
Expand Down Expand Up @@ -467,20 +486,29 @@ async def stream(request: Request, b64config: str, type: str, id: str):
"url": "https://comet.fast",
}
)

# Shorten Config for playback url
short_config = b64config
if settings.TOKEN:
short_config = {
"debridApiKey": config["debridApiKey"],
"debridStreamProxyPassword": config["debridStreamProxyPassword"],
"debridService": config["debridService"]
}
short_config = short_encrypt(json.dumps(short_config), settings.TOKEN)
results = []
for resolution, hash_list in balanced_hashes.items():
for hash in hash_list:
if hash in sorted_ranked_files:
hash_data = sorted_ranked_files[hash]
data = hash_data["data"]
url_friendly_file = quote(data["raw_title"].replace('/', '-'), safe='')
results.append(
{
"name": f"[{debrid_extension}⚡] Comet {data['resolution']}",
"title": format_title(data, config),
"torrentTitle": data["torrent_title"],
"torrentSize": data["torrent_size"],
"url": f"{request.url.scheme}://{request.url.netloc}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/{b64config}/playback/{hash}/{data['index']}",
"url": f"{request.url.scheme}://{request.url.netloc}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/{short_config}/playback/{hash}/{data['index']}/{url_friendly_file}",
"behaviorHints": {
"filename": data["raw_title"],
"bingeGroup": "comet|" + hash,
Expand All @@ -490,7 +518,7 @@ async def stream(request: Request, b64config: str, type: str, id: str):
return {"streams": results}


@streams.head("/{b64config}/playback/{hash}/{index}")
@streams.head("/{b64config}/playback/{hash}/{index}/{file_name}")
async def playback(b64config: str, hash: str, index: str):
return RedirectResponse("https://stremio.fast", status_code=302)

Expand All @@ -516,13 +544,13 @@ async def active_connections(request: Request, password: str):
}


@streams.get("/{b64config}/playback/{hash}/{index}")
@streams.get("/{b64config}/playback/{hash}/{index}/{file_name}")
async def playback(request: Request, b64config: str, hash: str, index: str):
config = config_check(b64config)
base_url = str(request.base_url).rstrip('/')
index = index.split('.', 1)[0]
if not config:
return RedirectResponse(f"{base_url}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/assets/invalidconfig.mp4", status_code=302)
return FileResponse("comet/assets/invalidconfig.mp4")

if (
settings.PROXY_DEBRID_STREAM
Expand Down Expand Up @@ -556,14 +584,17 @@ async def playback(request: Request, b64config: str, hash: str, index: str):

if not download_link:
debrid = getDebrid(session, config, ip if (not settings.PROXY_DEBRID_STREAM or settings.PROXY_DEBRID_STREAM_PASSWORD != config["debridStreamProxyPassword"]) else "")
download_link = await debrid.generate_download_link(hash, index)
download_link = await debrid.generate_download_link(hash, index, config["debridApiKey"])

if not download_link:
return RedirectResponse(f"{base_url}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/assets/uncached.mp4", status_code=302)
return FileResponse("comet/assets/uncached.mp4")
# Cleanup uncached Torrent from db if possible
await database.execute(
"DELETE FROM uncached_torrents WHERE hash = :hash",
{"hash": hash}
"""
DELETE FROM uncached_torrents
WHERE hash = :hash AND file_index = :file_index AND debrid_key = :debrid_key
""",
{"hash": hash, "file_index": index, "debrid_key": config["debridApiKey"]}
)

# Cache the new download link
Expand Down Expand Up @@ -592,7 +623,7 @@ async def playback(request: Request, b64config: str, hash: str, index: str):
>= settings.PROXY_DEBRID_STREAM_MAX_CONNECTIONS
for connection in active_ip_connections
):
return RedirectResponse(f"{base_url}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/assets/proxylimit.mp4", status_code=302)
return FileResponse("comet/assets/proxylimit.mp4")

proxy = None

Expand Down Expand Up @@ -659,41 +690,6 @@ async def close(self):
background=BackgroundTask(streamer.close),
)

return RedirectResponse(f"{base_url}{f'{settings.URL_PREFIX}' if settings.URL_PREFIX else ''}/assets/uncached.mp4", status_code=302)
return FileResponse("comet/assets/uncached.mp4")

return RedirectResponse(download_link, status_code=302)


@streams.get("/assets/{filename}")
async def stream_file(filename: str, range: str = Header(None)):
async def file_response(file_path: str, range_header: str = None):
file_size = os.path.getsize(file_path)
start = 0
end = file_size - 1

if range_header:
start, end = range_header.replace("bytes=", "").split("-")
start = int(start)
end = int(end) if end else file_size - 1

chunk_size = 1024 * 1024 # 1MB chunks
headers = {
"Content-Range": f"bytes {start}-{end}/{file_size}",
"Accept-Ranges": "bytes",
"Content-Type": "video/mp4",
"Content-Length": str(end - start + 1),
}

async def file_iterator():
with open(file_path, "rb") as video:
video.seek(start)
while True:
chunk = video.read(chunk_size)
if not chunk:
break
yield chunk

return StreamingResponse(file_iterator(), status_code=206 if range_header else 200, headers=headers)

file_path = f"comet/assets/{filename}"
return await file_response(file_path, range)
Binary file modified comet/assets/invalidconfig.mp4
View file
Binary file not shown.
Binary file modified comet/assets/proxylimit.mp4
View file
Binary file not shown.
Binary file modified comet/assets/uncached.mp4
View file
Binary file not shown.
2 changes: 1 addition & 1 deletion comet/debrid/alldebrid.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ async def get_files(
"title": filename,
"size": file["e"][0]["s"] if pack else file["s"],
"uncached": False,
"complete": torrent_name_parsed.complete or check_completion(torrent_name_parsed.raw_title, season),
"complete": None,
}

break
Expand Down
Loading

0 comments on commit b1d7cf2

Please sign in to comment.