Skip to content

Commit

Permalink
Remove server functionality from media-parser
Browse files Browse the repository at this point in the history
This commit removes all server-related functionalities from the media-parser package. As a result, the package now solely exists for parsing media from social networks. This creates a cleaner codebase and focuses on the core functionality of the package.

Signed-off-by: Jag_k <[email protected]>
  • Loading branch information
jag-k committed May 8, 2024
1 parent f4b058c commit 245876d
Show file tree
Hide file tree
Showing 45 changed files with 1,515 additions and 3,046 deletions.
13 changes: 0 additions & 13 deletions .dockerignore

This file was deleted.

2 changes: 1 addition & 1 deletion .github/FUNDING.yml
Original file line number Diff line number Diff line change
@@ -1 +1 @@
ko_fi: jag_k
github: jag-k
55 changes: 0 additions & 55 deletions .github/workflows/docker-image.yml

This file was deleted.

40 changes: 6 additions & 34 deletions .github/workflows/pypi-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,45 +10,17 @@ jobs:
publish:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2

- uses: actions/setup-python@v2
with:
python-version: "3.11"

- name: Install poetry
uses: abatilo/[email protected]
- uses: actions/checkout@v4
- name: Install Poetry
run: pipx install poetry==1.8.2
- uses: actions/setup-python@v5
with:
poetry-version: "1.6.1"
python-version: "3.12"
cache: "poetry"

- name: Publish to PyPI
env:
PYPI_TOKEN: ${{ secrets.PYPI_TOKEN }}
run: |
poetry config pypi-token.pypi $PYPI_TOKEN
poetry publish --build --no-interaction && echo "published=1" >> "$GITHUB_ENV" || echo "published=0" >> "$GITHUB_ENV"
- name: Check if published
run: |
if [ "$published" = "1" ]; then
echo "Published to PyPI"
else
echo "Failed to publish to PyPI"
exit 1
fi
# - name: Build Dash Docset
# run: |
# poetry run python -m dash_docs.tools.generate_docset

- name: Make release with docset
uses: softprops/action-gh-release@v1
with:
# files: |
# dash-docset.tgz
tag_name: ${{ github.ref }}
# body: |
# Dash Docset for Dash ${{ github.ref }}
draft: false
prerelease: false
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
8 changes: 8 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
.idea/
.vscode/
.fleet/
__pycache__/
*.py[cod]
dist/
Expand All @@ -7,3 +9,9 @@ __pypackages__/

config/
!**/.gitkeep

*.local
*.local.*

*.private
*.private.*
18 changes: 9 additions & 9 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,12 +1,16 @@
# See https://pre-commit.com for more information
# See https://pre-commit.com/hooks.html for more hooks
ci:
skip:
- poetry-lock
- poetry-export
- sync_with_poetry

repos:
- repo: https://github.com/astral-sh/ruff-pre-commit
# Ruff version.
rev: 'v0.4.2'
rev: 'v0.4.3'
hooks:
- id: ruff
args: [ --fix, --exit-non-zero-on-fix ]
- id: ruff-format

- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.6.0
Expand All @@ -16,16 +20,12 @@ repos:
- id: check-yaml
- id: check-added-large-files

- repo: https://github.com/psf/black
rev: "23.10.0"
hooks:
- id: black

- repo: https://github.com/python-poetry/poetry
rev: '1.8.0'
hooks:
- id: poetry-check
- id: poetry-lock
- id: poetry-check
- id: poetry-export
args: [ "--with", "docs", "--without-hashes", "-o", "docs/requirements.txt" ]

Expand Down
38 changes: 0 additions & 38 deletions Dockerfile

This file was deleted.

118 changes: 2 additions & 116 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,129 +15,15 @@ Server for parse Media by URL.
- [x] Instagram
- [x] Twitter
- [x] Reddit
- [ ] VK
- [ ] Pinterest

## Installation and Configuration Server

Use the `docker-compose.yml` file to run the server.

```yaml
version: "3.8"

service:
media-parser:
image: ghcr.io/jag-k/media-parser:latest
ports:
- 8000:8000
environment:
# Sentry integration (optional)
SENTRY_DSN: "https://[email protected]/2"
SENTRY_ENVIRONMENT: "dev"

# Enable sentry user feedback (optional)
SENTRY_ORGANISATION_SLUG: "sentry"
SENTRY_PROJECT_SLUG: "media-parser"
SENTRY_AUTH_TOKEN: "..." # with scope project:write
SENTRY_API_HOST: "https://api.sentry.io/"

# Database
MONGO_URL: "mongodb://mongodb:27017"
MONGO_DATABASE: "test"

volumes:
- ./config:/config

mongodb:
image: mongo:latest
volumes:
- ./data:/data/db
```
### Parsers Configuration
All configs for parsers stored in `config/parsers.json`. JSON Schema for
this: [schemas/parser_schema.json](https://github.com/jag-k/media-parser/blob/main/schemas/parser_schema.json).

To enable parser, you need to add config for this parser.
If parser hasn't config, like `tiktok` set an empty object (`{}`) to enable it.

Example:

```json5
// config/parsers.json
{
"$schema": "https://raw.github.com/jag-k/media-parsers/blob/main/schemas/parser_schema.json",
"instagram": {
// Optional
"instagram_saas_token": "asdasd"
},
"reddit": {
"client_id": "",
"client_secret": "",
// Optional
"user_agent": "video downloader (by u/Jag_k)"
},
"tiktok": {},
"twitter": {
"twitter_bearer_token": "asdasd"
},
"youtube": {}
}
```

Or you can use YAML file like `config/parsers.yaml` or `config/parsers.yml`:

```yaml
# config/parsers.yml
$schema: "https://raw.github.com/jag-k/media-parsers/blob/main/schemas/parser_schema.json"
instagram:
# Optional
instagram_saas_token: "asdasd"
reddit:
client_id: ""
client_secret: ""
# Optional
user_agent: "video downloader (by u/Jag_k)"
tiktok: {}
twitter:
twitter_bearer_token: "asdasd"
youtube: {}
```

## Usage

API documentation available on `/docs` endpoint.

## Clients

### Installation
## Installation

```bash
poetry add media-parser # or pip install media-parser
```

### Usage

```python
from media_parser import Client, FeedbackTypes
client = Client(url="http://localhost:8000")
async def main():
# Get all media
media = await client.parse("https://www.youtube.com/watch?v=9bZkp7q19f0", user="jag-k")
print(media)
# If media is incorrect, you can send feedback
await client.send_feedback(media, "jag-k", FeedbackTypes.wrong_media)
if __name__ == '__main__':
import asyncio
asyncio.run(main())
```

## License

Expand Down
41 changes: 0 additions & 41 deletions docker-compose.yml

This file was deleted.

Loading

0 comments on commit 245876d

Please sign in to comment.