#RWKV is One Dev's Journey to Dethrone Transformers The largest RNN ever (up to 14B). Parallelizable. Fast inference & training. Quantizable. Low vram usage. 3+ years of hard work https://t.co/ebRmfDrQbD Created by @BlinkDL_AI Computation sponsored by @StabilityAI @AiEleuther

Wrapping up: #RWKV was created by @BlinkDL_AI as a project at @AIEleuther - and is now being hosted by @LFAIDataFdn The RWKV wiki can be found at: https://t.co/ebRmfDso1b Our discord can be found at: https://t.co/veH8lO4Kf6 Give the models a try, drop by our discord

Overall we are excited by the progress we look forward to - the step jump in RWKV7-based attention - and the upcoming converted 70B models Once properly tuned, and trained, it will serve as a full drop-in replacement for the vast majority of AI workloads today 🪿

This is in addition to our latest candidate RWKV-7 "Goose" 🪿 architecture. Which we are excited about, as it shows early signs of a step jump from our v6 finch 📷 models. Future QRWKV model for V7 is to be expected as well https://t.co/K1zp2jO76L

Or the latest RWKV-6 Finch 7B World v3 model Which provides the strongest multi-lingual performance at 7B size scale among all our RWKV line of models https://t.co/K6F9OhzrDy

While the Q-RWKV-32B Preview takes the crown as our new Frontier RWKV model, unfortunately, this is not trained on RWKV usual 100+ language dataset For that refer to the other models, such as the RWKV MoE 37B-A11B v0.1 https://t.co/JWmkgT9K7v

This RWKV Frontier model was, trained by the Recursal AI team, by converting a Qwen 32B model with the following process - freeze feedforward - replace QKV attention layers, with RWKV6 layers - train RWKV layers - unfreeze feedforward & train all layers https://t.co/YIThic1rXR

Announcing a flock of RWKV models all arriving with apache2 licensing: available today First up, the strongest and largest linear model to date: QRWKV6-32B-Instruct-Preview Surpassing previous state space & RWKV models matching transformer performance with much lower inference

🚀 RWKV.cpp AI system - is being deployed to 0.5 billion installs globally Making it one of the world's most deployed, truly open-source (apache2) AI solutions out there As it ships with every Windows 11 system today!! Our group's code install count, went from ~100k -> 0.5B

RWKV.cpp project can be found here: https://t.co/XI1dXqlSzd Our RWKV wiki can be found at: https://t.co/ebRmfDso1b Our discord can be found at: https://t.co/veH8lO4Kf6 Give the models a try, drop by our discord, and have fun!

Energy cost - Is critical for a device that aims for long battery life! Variety of language support - is critical for shipping a product to the whole world (beyond English or Chinese) We will eagerly keep watch on this development https://t.co/ZLAepum3OU

What could Microsoft be using this for? Our main guess would be the beta features being tested - local copilot - local memory recall And it makes sense, especially for our smaller models - we support a 100+ languages - we are extremely low in energy cost (vs transformer)

These are real binaries, which you can decompile and inspect them They are a variation of the RWKV.cpp project, which has been built by the community, to run RWKV on nearly anything (even Raspberry Pi's) here: https://t.co/XI1dXqlSzd

You can verify this yourself, by updating to the latest Windows 11 - and going to C:\Program Files\Microsoft Office\root\vfs\ProgramFilesCommonX64\Microsoft Shared\OFFICE16 Alternatively, you could just search system files for rwkv, on any "copilot Windows 11" in an IT store

RWKV.cpp - is now being deployed to half a billion systems worldwide Making it one of the world's most widely deployed, truly open-source (apache2) AI solutions out there As it now ships with every Windows 11 system

Wrapping up: #RWKV was created by @BlinkDL_AI as a project at @AIEleuther - and is now being hosted by @LFAIDataFdn The RWKV wiki can be found at: https://t.co/ebRmfDrQbD Our discord can be found at: https://t.co/veH8lO4cpy Give the model a try, drop by our discord

As previously covered, this model is an upcycled model from our v5 - with an additional 1.4T tokens trained, for the conversion process. The RWKV v6 Finch paper covering the architecture can be found here:

In general, this can be viewed as a jump up from the Eagle v5 7B model, to the Finch v6 7B or 14B model Serving potentially full drop-in replacements for existing RWKV Eagle models across existing tasks like - translation - text summarization - text/code generation

The RWKV v6 Finch lines of models are here Scaling from 1.6B all the way to 14B Pushing the boundary for an Attention-free transformer, and Multi-lingual models. Cleanly licensedm Apache 2, under @linuxfoundation Find out more from the writeup here:

Is AI's insatiable appetite for compute holding back innovation? I recently talked with Eugene Cheah about @RWKV_AI. Eugene and the open-source RWKV community are working to democratize AI by addressing some of the key limitations of transformers.

The RWKV community wiki can be found at: https://t.co/ebRmfDso1b Our discord can be found at: https://t.co/veH8lO4Kf6 Give the model a try, drop by our discord, and provide us feedback on how we can improve the model for the community.