PluralisHQ Profile Banner
Pluralis Research Profile
Pluralis Research

@PluralisHQ

Followers
8K
Following
648
Media
12
Statuses
100

Protocol Learning

Joined July 2024
Don't wanna be here? Send us removal request.
@PluralisHQ
Pluralis Research
2 months
We've reached a major milestone in fully decentralized training: for the first time, we've demonstrated that a large language model can be split and trained across consumer devices connected over the internet - with no loss in speed or performance.
Tweet media one
94
266
934
@PluralisHQ
Pluralis Research
9 days
RT @_AlexanderLong: Thats a wrap for ICML2025. Incredible to watch the space go from "What are you talking about" to "That's impossible" to….
0
4
0
@PluralisHQ
Pluralis Research
17 days
We also made a @3blue1brown style video explaining the paper at a high level.
@PluralisHQ
Pluralis Research
17 days
Pluralis has a main tack paper at ICML this week and the team is in Vancouver running several events. The best will be the Open Source Mixer we're running with @gensynai and @akashnet_ Thursday night. For anyone interested in decentralised training it should be a great evening.
6
11
76
@PluralisHQ
Pluralis Research
17 days
Pluralis has a main tack paper at ICML this week and the team is in Vancouver running several events. The best will be the Open Source Mixer we're running with @gensynai and @akashnet_ Thursday night. For anyone interested in decentralised training it should be a great evening.
11
14
119
@PluralisHQ
Pluralis Research
24 days
RT @gensynai: Connect with fellow open-source developers, engage with leading minds in decentralized AI (DeAI) and machine learning, and en….
Tweet card summary image
lu.ma
Join Akash Network, Gensyn, and Pluralis Research for an Open Bar & Open Source Mixer—a premier networking event for machine learning professionals—on July…
0
7
0
@PluralisHQ
Pluralis Research
1 month
RT @_AlexanderLong: Using beautiful Grafana dashboards for everything internally, so much nicer than Tensorboard. Wandb still good but does….
0
5
0
@PluralisHQ
Pluralis Research
2 months
@SameeraRamasin1 Now this is feasible, we will begin a live run of the first community Protocol Model - Pluralis’s Genesis Run - in the coming days. Registration here: EOT.
Tweet card summary image
docs.google.com
7
13
67
@PluralisHQ
Pluralis Research
2 months
@SameeraRamasin1 We also expect this to also have a major impact on centralized training, reducing the importance of high speed networking, making spot-instance based training feasible, and increasing inference speeds.
1
2
31
@PluralisHQ
Pluralis Research
2 months
@SameeraRamasin1 And it is the most significant milestone in our Protocol Learning Research Program.
Tweet card summary image
blog.pluralis.ai
Two enormous, previously disparate fields converge and a path towards the largest models to ever be trained is opened.
1
0
26
@PluralisHQ
Pluralis Research
2 months
@SameeraRamasin1 This unlocks:. 1. Truly community-created and owned base models. 2. A new path to scaling base models beyond anything we have seen to date. It is a result that we set out to achieve almost exactly one year ago today.
Tweet card summary image
blog.pluralis.ai
Collaborative Training of foundation models is closer to actualization than broadly understood. The popular view that low bandwidth node-to-node connections render this infeasible is incorrect.
1
1
33
@PluralisHQ
Pluralis Research
2 months
@SameeraRamasin1 What this means is, for the first time, individuals can pool globally distributed compute to train models far larger than they could alone. As the model is split across nodes, the only constraint is how much compute the protocol can gather.
1
1
21
@PluralisHQ
Pluralis Research
2 months
This is a very high-level summary. All details are in the pre-print (led by @SameeraRamasin1)
1
1
33
@PluralisHQ
Pluralis Research
2 months
By decomposing the high-rank embedding information, we can enforce a shared subspace across all blocks. We then only need to transmit a small set of coefficients between nodes.
1
1
20
@PluralisHQ
Pluralis Research
2 months
We found this property in EVERY transformer we analysed, across all parameter sizes and architectures. It’s obscured by the recursive addition of positional embeddings via residuals - but once found, it can be used to dramatically decrease communication overhead.
Tweet media one
2
1
29
@PluralisHQ
Pluralis Research
2 months
How is this possible? All transformer models - regardless of size or architecture - have a hidden property: the output projection weights of each block have low stable rank.
Tweet media one
1
1
25
@PluralisHQ
Pluralis Research
2 months
Protocol Models fix this: we place one transformer block on each device and compress the communication between blocks by over 100x without altering training dynamics. This enables multi-participant training, of very large models, over the internet at datacenter speeds.
1
0
29
@PluralisHQ
Pluralis Research
2 months
Why? Because the internet is 100–300× slower than datacenter connections. Compressing to compensate breaks the model's internal communication. Errors build up, training collapses.
Tweet media one
1
1
34
@PluralisHQ
Pluralis Research
2 months
What has been achieved? Today, training happens in datacenters, where models are spread over many GPUs with fast interconnects. Trying to replicate this over the internet causes huge slowdowns.
1
1
31
@PluralisHQ
Pluralis Research
2 months
This work proves a third path is viable between closed models and today’s open-weight releases, which remain centralized and unsustainable. Community-owned models are the only true open-source AI and open a new path to scale.
Tweet card summary image
blog.pluralis.ai
Developing the true open-source AI
1
4
54
@PluralisHQ
Pluralis Research
2 months
Almost exactly 1 year ago today Pluralis set out to solve a very difficult problem. We are pleased to announce an update on that problem. We can now combine small consumer devices, connected via the internet, to train giant models. Full paper release in 72 hours.
Tweet media one
29
27
215