How to generate text with SIE

The generate primitive runs text generation on small LLMs you host yourself. Same client, same routing, and the same model catalog as the other primitives, so a generation model sits alongside your encoders and rerankers instead of behind a separate service.

Basic Usage

Pass a model, a prompt, and a token budget. max_new_tokens is required.

Python
TypeScript

from sie_sdk import SIEClient

client = SIEClient("http://localhost:8080")

result = client.generate(
    "Qwen/Qwen3-4B-Instruct-2507",
    "Write a one-sentence summary of vector search.",
    max_new_tokens=128,
)
print(result["text"])

import { SIEClient } from "@superlinked/sie-sdk";

const client = new SIEClient("http://localhost:8080");

const result = await client.generate(
  "Qwen/Qwen3-4B-Instruct-2507",
  "Write a one-sentence summary of vector search.",
  { maxNewTokens: 128 },
);
console.log(result.text);

await client.close();

Common Options

Option	Type	Description
`max_new_tokens`	int	Hard cap on output tokens. Required.
`temperature`	float	Sampling temperature. Defaults to `1.0`.
`top_p`	float	Nucleus sampling cutoff. Defaults to `1.0`.
`stop`	list of strings	Stop sequences that end generation early.

The result carries the generated text, a finish_reason, token usage, and SIE-native timing (ttft_ms, tpot_ms). See the SDK Reference for the full surface.

Streaming

Stream tokens as they arrive with stream_generate. It yields chunks, each carrying a text_delta; the terminal chunk (done) also carries usage and ttft_ms.

Python
TypeScript

for chunk in client.stream_generate(
    "Qwen/Qwen3-4B-Instruct-2507",
    "Write a haiku about vector search.",
    max_new_tokens=64,
):
    print(chunk.get("text_delta", ""), end="", flush=True)
    if chunk.get("done"):
        print(f"\nttft_ms={chunk.get('ttft_ms')}")

for await (const chunk of client.streamGenerate(
  "Qwen/Qwen3-4B-Instruct-2507",
  "Write a haiku about vector search.",
  { maxNewTokens: 64 },
)) {
  process.stdout.write(chunk.text_delta);
  if (chunk.done) console.log(`\nTTFT: ${chunk.ttft_ms}ms`);
}

Models

Generation models are listed in the Model Catalog alongside the encoders and rerankers. Qwen/Qwen3-4B-Instruct-2507 is a solid default; Qwen/Qwen3-0.6B is a smaller, faster option for short prompts.

On a multi-pool cluster, generation models may run on a dedicated GPU pool. Target it with the gpu parameter, for example gpu="rtx6000", so the request lands on a pool that serves the generation bundle.

What’s Next

Model Catalog - all supported models
SDK Reference - full client API
API Reference - HTTP endpoints