VESSL AI LLMProvider integration

[nsd9696]

Description

• Integrating VESSL AI LLMProvider to use in Llama index.
• VESSL AI provider will be served through vLLM and OpenAI Compatible.
• User can manually serve their own huggingface model with 1) model_name, 2) vessl yaml_file or 3) connect with pre-served vessl llm service endpoint.
• Example

from llama_index.llms.vesslai import VesslAILLM

llm = VesslAILLM()

#1 Serve with hf model name
llm.serve(
    service_name = "llama-index-vesslai",
    model_name = "mistralai/Mistral-7B-Instruct-v0.3",
    hf_token = "HF_TOKEN",
    api_key="openai-api-key"
)
#2 Serve with yaml file
llm.serve(
    service_name = "llama-index-vesslai",
    yaml_path="/users/own/vessl/service.yaml",
    api_key="openai-api-key"
)
#3 Connect with pre-served endpoint
llm.connect(
    served_model_name="mistralai/Mistral-7B-Instruct-v0.3",
    endpoint = "https://model-service-gateway-abc.oregon.google-cluster.vessl.ai/v1",
)
resp = llm.complete("Who is Paul Graham?")

Fixes # (issue)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

• Yes
• No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

• Yes
• No

Type of Change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

Your pull-request will likely not be merged unless it is covered by some form of impactful unit testing.

• I added new unit tests to cover this change
• I believe this change is already covered by existing unit tests

Suggested Checklist:

• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added Google Colab support for the newly added notebooks.
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I ran make format; make lint to appease the lint gods

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[logan-markewich]

Curious about the decision to do serve and connect outside of the __init__() function? Do your users often switch this after the llm object is created? In most llama-index LLMs, you would just do llm = VesslAILLM(...) and then from there you can directly use it

[logan-markewich]

Its fine either way tbh, was just curious

[nsd9696]

Thank you for the feedback. To use VESSL, authentication through configure is required. I wanted to handle this process during initialization and explicitly separate the serving and connection of the llm_provider afterward. Internally at VESSL, we have discussed this flow, and it seems to be fine.

[logan-markewich]

Thoughts on making serve and connect async? Seems like this could possibly be a blocking operation with wait_for_gateway_enabled ?

[logan-markewich]

There's quite a lot of code, is any of it testable? (you'd have to mock out api calls though)

[nsd9696]

Unfortunately, mocking the API for our service is quite complicated at the moment. We will consider adding it after the merge if necessary.