Can we reconstruct a closed-source LLM tokenizer using only two oracles from the chat API?
英文标题：Can we reconstruct a closed-source LLM tokenizer using only two oracles from the chat API?
From the chat APIs alone, we can extract two useful oracles: 1. Token length oracle: Given any string s, return len(tokenize(s)). Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s). The first oracle (token count) is straightforward
打开原文相关 3 · 机会 2
中文
From the chat APIs alone, we can extract two useful oracles: 1. Token length oracle: Given any string s, return len(tokenize(s)). Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s). The first oracle (token count) is straightforward
中文全文翻译正在补齐，可先查看下方英文原文。
English / 原文
From the chat APIs alone, we can extract two useful oracles: 1. Token length oracle: Given any string s, return len(tokenize(s)). Prefix token oracle: Given a string s and integer n, return the string decoded from the first n tokens of tokenize(s). The first oracle (token count) is straightforward