espnet2.sds.end_to_end.mini_omni.litgpt.generate.base.generate

source

espnet2.sds.end_to_end.mini_omni.litgpt.generate.base.generate(model: GPT, input_ids: list, max_returned_tokens: int, , temperature: float = 1.0, top_k: int | None = None, top_p: float = 1.0, eos_id_a: int | None = None, eos_id_t: int | None = None, pad_id: int | None = None, shift: int | None = None, include_prompt: bool = True, generate_text=False) → Tensor

Takes a conditioning sequence (prompt) as input and continues to

generate as many tokens as requested. The implementation of this function is modified from A. Karpathy’s nanoGPT.

• Parameters:

  • model – The model to use.

  • prompt – Tensor of shape (T) with indices of the prompt sequence.

  • max_returned_tokens – The maximum number of tokens to return (given plus generated).

  • temperature – Scales the predicted logits by 1 / temperature.

  • top_k – If specified, only sample among the tokens with the k

  • probabilities. (highest)

  • top_p – If specified, it represents the cumulative

  • process. (probability threshold to consider in the sampling) –

    In top-p sampling, the next token is sampled from the highest probability tokens whose cumulative probability exceeds the threshold top_p. When specified, it must be 0 <= top_p <= 1. Here, top_p=0 is equivalent to sampling the most probable token, while top_p=1 samples from the whole distribution. It can be used in conjunction with top_k and temperature with the following order of application:

    1. top_k sampling
    2. temperature scaling
    3. top_p sampling

    For more details, see https://arxiv.org/abs/1904.09751 or https://huyenchip.com/2024/01/16/sampling.html#top_p

  • eos_id – If specified, stop generating any more token once

  • triggered. (the <eos> token is)

  • include_prompt – If true (default) prepends the prompt (after

  • output. (applying the prompt style ) to the)