diff --git a/docs/Writerside/kc.tree b/docs/Writerside/kc.tree index 22d6627b..86f6d4f6 100644 --- a/docs/Writerside/kc.tree +++ b/docs/Writerside/kc.tree @@ -7,7 +7,7 @@ - + \ No newline at end of file diff --git a/docs/Writerside/topics/External-access.md b/docs/Writerside/topics/External-access.md new file mode 100644 index 00000000..0055ec4b --- /dev/null +++ b/docs/Writerside/topics/External-access.md @@ -0,0 +1,9 @@ +# External access + + + +In its initial state Knight Crawler will only work on the machine that you host it on, and will not be accessible on +your local network or from the internet. This is a Stremio +limitation ([read Stremio's stance here](https://github.com/Stremio/stremio-features/issues/687)) and is beyond our +control. We provide a guide +how to make Knight Crawler accessible from your local network or the internet [here](). \ No newline at end of file diff --git a/docs/Writerside/topics/Getting-started.md b/docs/Writerside/topics/Getting-started.md index ed18f059..8360b96e 100644 --- a/docs/Writerside/topics/Getting-started.md +++ b/docs/Writerside/topics/Getting-started.md @@ -1,12 +1,7 @@ # Getting started -Knight Crawler is provided as an all-in-one solution. This means that you can get started with very little setup. - -In its initial state Knight Crawler will only work on the machine that you host it on, and will not be accessible on -your local network or from the internet. This is a Stremio -limitation ([read Stremio's stance here](https://github.com/Stremio/stremio-features/issues/687)) and is beyond our -control. We provide a guide -how to make Knight Crawler accessible from your local network or the internet [here](). +Knight Crawler is provided as an all-in-one solution. This means we include all the necessary software you need to get started +out of the box. ## Before you start @@ -16,3 +11,171 @@ Make sure that you have: - [Docker](https://docs.docker.com/get-docker/) and [Compose](https://docs.docker.com/compose/install/) installed - A [GitHub](https://github.com/) account _(optional)_ + +## Download the files + +Installing Knight Crawler is as simple as downloading a copy of the [deployment directory](https://github.com/Gabisonfire/knightcrawler/tree/master/deployment/docker). + +A basic installation requires only two files: +- deployment/docker/.env.example +- deployment/docker/docker-compose.yaml. + + +> Knight Crawler will only be accessible on the machine you run it on, to make it accessible from other machines navigate to [External access](External-access.md). +> +{style="note"} + +For this guide I will be placing them in a directory on my home drive ~/knightcrawler. + +Rename the .env.example file to be .env + +``` +~/ +└── knightcrawler/ + ├── .env + └── docker-compose.yaml +``` + +## Initial configuration + +Below are a few recommended configuration changes. + +Open the .env file in your favourite editor. + +> If you are using an external database, configure it in the .env file. Don't forget to disable the ones +> included in the docker-compose.yaml. + +### Database credentials + +It is strongly recommended that you change the credentials for the databases included with Knight Crawler. This is best done +before running Knight Crawler for the first time. It is much harder to change the passwords once the services have been started +for the first time. + +```Bash +POSTGRES_PASSWORD=postgres +... +MONGODB_PASSWORD=mongo +... +RABBITMQ_PASSWORD=guest +``` + +Here's a few options on generating a secure password: + +```Bash +# Linux +tr -cd '[:alnum:]' < /dev/urandom | fold -w 64 | head -n 1 +# Or you could use openssl +openssl rand -hex 32 +``` +```Python +# Python +import secrets + +print(secrets.token_hex(32)) +``` + +### Your time zone + +```Bash +TZ=London/Europe +``` + +A list of time zones can be found on [Wikipedia](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) + +### Consumers + +```Bash +JOB_CONCURRENCY=5 +... +MAX_CONNECTIONS_PER_TORRENT=10 +... +CONSUMER_REPLICAS=3 +``` + +These are totally subjective to your machine and network capacity. The above default is pretty minimal and will work on +most machines. + +`JOB_CONCURRENCY` is how many films and tv shows the consumers should process at once. As this affects every consumer +this will likely cause exponential +strain on your system. It's probably best to leave this at 5, but you can try experimenting with it if you wish. + +`MAX_CONNECTIONS_PER_TORRENT` is how many peers the consumer will attempt to connect to when it is trying to collect +metadata. +Increasing this value can speed up processing, but you will eventually reach a point where more connections are being +made than +your router can handle. This will then cause a cascading fail where your internet stops working. If you are going to +increase this value +then try increasing it by 10 at a time. + +> Increasing this value increases the max connections for every parallel job, for every consumer. For example +> with the default values above this means that Knight Crawler will be on average making `(5 x 3) x 10 = 150` +> connections at any one time. +> +{style="warning"} + +`CONSUMER_REPLICAS` is how many consumers should be initially started. This is best kept below 10 as GitHub rate limit +how fast we can access a list of torrent trackers. You can increase or decrease the number of consumers whilst the service is running by running the command `docker compose up --scale consumer=`. This value is best increased by 5 at a time. Repeat this process until you have reached the desired level of consumers. + +### GitHub personal access token + +This step is optional but strongly recommended. [Debrid Media Manager](https://debridmediamanager.com/start) is a media library manager +for Debrid services. When a user of this service chooses to export/share their library publicly it is saved to a public GitHub repository. +This is, essentially, a repository containing a vast amount of ready to go films and tv shows. Knight Crawler comes with the ability to +read these exported lists, but it requires a GitHub account to make it work. + +Knight Crawler needs a personal access token with read-only access to public repositories. This means we can not access any private +repositories you have. + +1. Navigate to GitHub settings ([GitHub token settings](https://github.com/settings/tokens?type=beta)): + - Navigate to `GitHub settings`. + - Click on `Developer Settings`. + - Select `Personal access tokens`. + - Choose `Fine-grained tokens`. + +2. Press `Generate new token`. + +3. Fill out the form with the following information: + ```Generic + Token name: + KnightCrawler + Expiration: + 90 days + Description: + + Respository access: + (checked) Public Repositories (read-only) + ``` + +4. Click `Generate token`. + +5. Take the new token and add it to the bottom of the .env file: + ```Bash + # Producer + GITHUB_PAT= + ``` + +## Start Knight Crawler + +To start Knight Crawler use the following command: + +```Bash +docker compose up -d +``` + +Then we can follow the logs to watch it start: + +```Bash +docker compose logs -f --since 1m +``` + +To stop following the logs press Ctrl+C at any time. + +The Knight Crawler configuration page should now be accessible in your web browser at [http://localhost:7000](http://localhost:7000) + +## Stop Knight Crawler + +Knight Crawler can be stopped with the following command: + +```Bash +docker compose down +``` diff --git a/docs/Writerside/topics/Run-Knight-Crawler.md b/docs/Writerside/topics/Run-Knight-Crawler.md deleted file mode 100644 index ba6d6238..00000000 --- a/docs/Writerside/topics/Run-Knight-Crawler.md +++ /dev/null @@ -1,71 +0,0 @@ -# Run Knight Crawler - -To run Knight Crawler you need two files, both can be found in -the [deployment/docker](https://github.com/Gabisonfire/knightcrawler/tree/master/deployment/docker) -directory on GitHub: - -- deployment/docker/.env.example - -- deployment/docker/docker-compose.yaml. - -For this guide I will be placing them in a directory on my home drive ~/knightcrawler. - -Rename the .env.example file to be .env - -``` -~/ -└── knightcrawler/ - ├── .env - └── docker-compose.yaml -``` - -## Configuration - -Before we start the services, we need to change a few things in the .env file. - -> If you are using an external database, configure it in the .env file. Don't forget to disable the ones -> included in the docker-compose.yaml. - -### Your time zone - -```Bash -TZ=London/Europe -``` - -> A list of time zones can be found on [Wikipedia](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) - -### Consumers - -```Bash -JOB_CONCURRENCY=5 -... -MAX_CONNECTIONS_PER_TORRENT=10 -... -CONSUMER_REPLICAS=3 -``` - -These are totally subjective to your machine and network capacity. The above default is pretty minimal and will work on -most machines. - -`JOB_CONCURRENCY` is how many films and tv shows the consumers should process at once. As this affects every consumer -this will likely cause exponential -strain on your system. It's probably best to leave this at 5, but you can try experimenting with it if you wish. - -`MAX_CONNECTIONS_PER_TORRENT` is how many peers the consumer will attempt to connect to when it is trying to collect -metadata. -Increasing this value can speed up processing, but you will eventually reach a point where more connections are being -made than -your router can handle. This will then cause a cascading fail where your internet stops working. If you are going to -increase this value -then try increasing it by 10 at a time. - -> Increasing this value increases the max connections for every parallel job, for every consumer. For example -> with the default values above this means that Knight Crawler will be on average making `(5 x 3) x 10 = 150` -> connections at any one time. -> -{style="warning"} - -`CONSUMER_REPLICAS` is how many consumers should be initially started. This is best kept below 10 as GitHub rate limit -how fast we can access a list of torrent trackers. You can increase or decrease the number of consumers whilst the service is running by running the command `docker compose up --scale consumer=`. This value is best increased by 5 at a time. Repeat this process until you have reached the desired level of consumers. - -### GitHub personal access token