standard completions dot org

RT @domenic: In response to requests from users, and partially inspired by the @stdcompletions effort, we're working on adding assistant pr….

first RFC draft is out, for assistant prefixes (prefills). please leave comments on the RFC PR! link in next post.

we now have an RFCs repository! please feel free to open an issue to discuss specific incompatibilities you've encountered between OpenAI-compatible providers. link below!

(the person responsible has been fired)

dms were previously not open, sorry! now are. please DM this account for inquiries about standard completions!.

13/13

If an open standard for OpenAI-compatible completions APIs interests you, give us a follow! And if you're a model provider or router interested in participating in the Standard Completions Working Group, send us a DM or an email (link in bio). 12/.

Obviously, the status quo here is not ideal! At Standard Completions, we're aiming to create a standard for the Completions and Chat Completions APIs that OpenAI-compatible providers can adopt, unifying features like assistant prefixes behind a single interface. 11/.

And on vLLM, some vLLM-based providers, and 🤗Transformers models, use the "continue_final_message" and "add_generation_prompt" flags in `extra_body` 10/

On Deepseek models, use the beta endpoint, and set { "prefix": True } on the trailing assistant message. If you forget the "prefix" flag, assistant prefixes will silently not work as expected😔 9/

On Anthropic and (some) OpenRouter models, you can add an assistant prefix by appending a trailing assistant message to the request. The model will continue that message as if it produced it. 8/

So how do you use assistant prefixes in your own apps? This is where Standard Completions will help, because currently, there are a multitude of standards. Let's go through them. 7/.

Now, only the text between the prefix and stop sequence will be returned—the model will produce clean, parseable TOML! No tips or threats required. 6/

By adding a user-provided prefix to the model's response, we can exploit the autoregressive nature of LLMs to generate just the content we want. Here, I prefix the model's response with "```toml\n", and set "```" as a stop sequence. 5/

If you are running the model locally, you can use a structured generation library like dottxt's Outlines to enforce a desired schema. But what if you're using a model over an API that doesn't support custom grammars? 4/.

Trying to extract structured data from an LLM, you will often run into parsing issues due to the model returning Markdown instead of the desired format: 3/

Often when developing a feature that uses an LLM, you want the model to return data in a specific format, such as JSON or TOML. Sometimes tool calling is enough for this, but what if you need a different format, or a more flexible schema? 2/.

🧵 What are assistant prefixes/prefills, how are they useful, and how will Standard Completions help you use them in your LLM projects? 1/