Text Generate REST API schema

[gavrissh]

Propose generate rest api endpoints

/v2/models/{model_name}/versions/${MODEL_VERSION}/generate
/v2/models/{model_name}/versions/${MODEL_VERSION}/generate_stream

Reference - https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/protocol/extension_generate.html#generate-extension

[yuzisun]

This is concatenated text output, we might still want to see the token generated for each iteration.

[gavrissh]

In the Nvidia implementation, each response in returning cumulative set of tokens.

1st json
{
text_output: "Here is"
}
.
.
.
..
subsequent json response
{
text_output: "Here is the output for the prompt"
}

Should we add additional property to display token generated in current response set?

[gavrissh]

@yuzisun Wanted to follow up, if the current state of changes are alright?

[gavrissh]

I have updated with all the recent discussed changes

[cmaddalozzo]

We should probably add the option to return log probabilities in the result. This seems to be fairly common among other APIs. This would comprise a boolean logprobs parameter in the request and a corresponding logprobs property in the response containing an array of objects with keys token and logprob.

[gavrissh]

We should probably add the option to return log probabilities in the result. This seems to be fairly common among other APIs. This would comprise a boolean logprobs parameter in the request and a corresponding logprobs property in the response containing an array of objects with keys token and logprob.

I have updated the PR to support the above items

[yuzisun]

suggest change the naming to Token as it is not just logprob field, see https://github.com/huggingface/text-generation-inference/blob/main/docs/openapi.json#L844.

[yuzisun]

suggest change the name here, in TGI it is called details which includes the tokens, not sure if we should follow the same.
https://github.com/huggingface/text-generation-inference/blob/main/docs/openapi.json#L645

[yuzisun]

in OpenAI it is choices
https://platform.openai.com/docs/api-reference/chat/object

[gavrissh]

in OpenAI logprobs is property under choices.
Even I was not sure here. It is up for any suggestions

Current
Output -> {
text_output,
model_name,
model_version,
logprobs -> List[Token]
}

Token is followed as per TGI - https://huggingface.github.io/text-generation-inference/#/Text%20Generation%20Inference/generate_stream
Token -> {
id,
logprob,
special,
text
}

[yuzisun]

For streaming case it is a single token.

[yuzisun]

should this be part of the GenerateParameters ?

[yuzisun]

Can you add a description for this flag, also I think this should be the details flag as logprob is one of the fields on it.

[yuzisun]

Add a description for this

[yuzisun]

Let's make sure we have descriptions for these fields

[yuzisun]

finish_reason should be an enum

[yuzisun]

both finish_reason and logprobs should be required if details is requested.

[yuzisun]

Thanks @gavrishp !! Great job on getting this going with the initial version.

/lgtm
/approve

[oss-prow-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gavrishp, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [yuzisun]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment