RubatoCodec: Budgeted Adaptive Speech Tokenization with Streaming Decoding

The paper addresses a trade-off in neural speech codecs: lower frame rates shorten token sequences but force each token to carry more information. Variable-frame-rate codecs adapt token allocation to content, but threshold-based allocation lets speech content determine the token count. The supplied text is truncated before it states the proposed RubatoCodec method and its results, so the approach and its outcomes cannot be fully described.

Key takeaways

• Lower frame rates shorten token sequences but require each token to carry more linguistic and acoustic information.
• Variable-frame-rate codecs adapt token allocation to speech content to manage this trade-off.
• Threshold-based allocation lets content determine the total token count, which the source frames as the problem.

Why it matters

Token count controls the cost and latency of speech generation models, so codecs whose token budget is predictable and adaptive could matter for streaming speech systems.

Read with care

The supplied source text is truncated and omits the proposed method's details and experimental results; source claims not independently checked.