parithosh | 🐼👉👈🐼
@parithosh_j
Followers
8,478
Following
1,332
Media
149
Statuses
2,466
DevOps at the Ethereum Foundation. ENS: parithosh.eth Website: Mastodon: @parithosh @chaos .social
Berlin
Joined June 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Leverkusen
• 421293 Tweets
#Comebacktome
• 267550 Tweets
RM IS COMING
• 209480 Tweets
メイドの日
• 85735 Tweets
Celtics
• 77211 Tweets
Luka
• 69445 Tweets
Maranhão
• 61388 Tweets
WIN CENTRAL FAMILY
• 54610 Tweets
#ดวงใจเทวพรหมSportsDay
• 47589 Tweets
Rangers
• 44119 Tweets
メイドさん
• 40145 Tweets
Cavs
• 37859 Tweets
भगवान परशुराम
• 37670 Tweets
Tatum
• 33394 Tweets
メイド服
• 31495 Tweets
ドラマ化
• 27797 Tweets
Safe Campaign
• 25535 Tweets
भगवान विष्णु
• 22702 Tweets
Canes
• 22600 Tweets
PRISCILLA
• 17931 Tweets
RTTT
• 13805 Tweets
#VileirosNaMansão
• 12652 Tweets
Pumas
• 12443 Tweets
Rex Murphy
• 12403 Tweets
Cruz Azul
• 12333 Tweets
Zavala
• 11690 Tweets
Aleska
• 10859 Tweets
The goerli fork finalized! :D
After the fix was patched in, the validators came back online and the chain started finalizing again. The MEV circuitbreaker automatically disables and mev-blocks have started flowing through as well.
Yay client diversity!
293
433
709
The Goerli testnet will merge in the next ~2 days, this will be the last testnet to merge before mainnet.
If you are running a Goerli node, please update and follow the numerous guides to ensure we can have a smooth merge.
More information here:
211
246
529
Mainnet-Shadow-Fork-4 looks perfect! All clients went through the transition without a hitch. Only erigon missing (unrelated sync issues, we'll know in a bit if its synced to head as expected)
We finalized and just keep chugging!
#TestingTheMerge
170
170
511
We shadow forked mainnet successfully a second time!
#TestingTheMerge
21
75
485
After 4 days of PoW, merge-devnet-5 hit TTD today. The chain continued finalizing post TTD, as expected. This means the merge was successful.
We ran a few zombie miners. Miners ran older versions that don't know about TTD and just continue mining post TTD. No issues were found.
merge-devnet-5 has been launched. The beaconchain genesis happens at 4PM UTC and the PoW chain has already been launched. The TTD has been set to be really high to allow everyone to join pre-merge.
Tooling can be found here:
1
21
99
9
91
424
The merge pandas have arrived!
mainnet-shadow-fork-1 hit TTD ~half an hour ago. We've been finalizing and producing blocks!
We quickly noticed some seemingly minor issues with Nethermind and Besu(triage ongoing). Erigon is currently syncing to head, status update later.
Wondering what the
#TestingTheMerge
team has been up to?
The aim of the Kiln merge testnet was to allow the community to practice running their nodes, deploying contracts, testing infrastructure, etc. We hope it's helping the community get a sense of the post merge world!
11
68
246
28
105
395
Ropsten beacon chain is live!
We've finalized as expected and the chain is chugging along till the un-announced TTD.
Users can join the chain and practice running their nodes!
Initial docs:
If you make docs/vids/etc do tag me and
#TestingTheMerge
21
80
391
Blobs are now flowing in Sepolia :D
Uneventful testnet forks are the best ones!
14
52
304
Shapella is live and finalized on Sepolia
#SepoliaShapella
Next would be Goerli, then Mainnet🦉
9
88
281
The configs for the Kiln testnet are out!
You can find all the information you need here:
Help us with
#TestingTheMerge
!
13
76
281
Wondering what the
#TestingTheMerge
team has been up to?
The aim of the Kiln merge testnet was to allow the community to practice running their nodes, deploying contracts, testing infrastructure, etc. We hope it's helping the community get a sense of the post merge world!
11
68
246
We managed to get a testnet up and running with a total of 12 combinations of execution and consensus clients! Expect some refining and many more testnets till the merge! - wagmi.
Huge props to
@protolambda
and all the client teams for their efforts this week :D
#Ethereum
7
41
235
This was the first shadow fork where every client combination survived the transition and managed to stay in sync afterwards. We're making real progress here!
5
30
237
Sepolia merged successfully!
After some config issues, the chain is finalizing. There were no client related issues. 1 EF node had a hiccup during the transition, but it self healed without intervention. Of a possible ~95% participation rate, we are generally seeing ~95%.
I wanted to share some thoughts on the events of the Sepolia merge. I speak for myself, not necessarily the rest of
@sigp_io
. Needless to say, it was less than ideal and we will be extra vigilant moving forward. I've also seen some understandable concerns from the community:
1/6
4
17
102
7
33
192
The new merge testnet Pithos is now live!
Explorer:
Instructions to join:
60
100
178
Goerli has successfully upgraded to Shapella! 🦉
We had some validators running older versions of clients, once that was fixed the attestation rate went back up and we are currently finalizing!
#GoerliShapella
Mainnet comes next!
4
40
191
Blobs are live on Ethereum Mainnet!
The Dencun fork finalized without issues and the network has been running stably since then. We can see healthy blob propagation statistics (expect a deeper analysis over the next weeks).
Congrats to all the client teams and contributors!
7
25
186
I made my first post about a potential testing path to figure out gas limit increases:
We should be doing this systematically and figure out our constraints, current limits and projected limits before committing to an increase.
8
30
170
As we move closer to shipping Dencun on Ethereum, let's have a look at the latest test results:
Goerli-shadowfork-1 went live recently. Its a 300 node shadowfork (largest public one to date) with almost every client combo as they are on mainnet
10
42
163
New
#TestingTheMerge
milestone!
The Kiln testnet is now processing more transactions than mainnet (for the last days)!
7
30
157
Let's bring
@EFDevcon
to Namma Bengaluru, India ! :D
The proposal for Devcon VII is now live! Check it out and share it if you want to enjoy DevCon with a Dosa and amazing weather!
@devfolio
@ETHIndiaco
16
29
153
Holesky is finalized for Deneb :D
Churn limit looks good so far and blobs are flowing smoothly!
Next stop, Mainnet!
We have blobs on Holesky!
Another un-eventful testnet fork for the books :)
2
7
41
4
27
155
After the insane chaos spread by the
#Kintsugi
testnet fuzzer, the testnet finalizes again!
Another day, another bug! All hail the
#Kintsugi
testnet fuzzer. Thread.
So as you might remember from my last thread, the fuzzer created a block which uncovered two separate issue in the besu and nethermind code for verifying payload as well as the optimistic sync code
10
25
187
2
24
143
Time for devnet updates again! :D
We had a sepolia shadowfork this week, this shadowfork successfully added EIP-4844 support to sepolia with a subset of clients!
11
111
139
We're going to monitor the chain over the next few weeks to make sure no client pairs go out of sync. We'd hope that community members raise alarms if they see dapps/tooling not work as expected! Sync nodes in weird ways to find bugs! Send txs!
Help with
#TestingTheMerge
3
7
134
Top to bottom: Geth, Nethermind,Besu(with lighthouse,teku)
Left to right: Prysm,Lighthouse, Teku, Nimbus,Lodestar
Join us on Kintsugi:
#TestingTheMerge
10
19
133
Not only is Ropsten the first existing testnet to get merged, its also the first testnet where every client team is running an equal part of the network. That's at least a dozen unique ways of setting up a node!
A hearty congratulations to every client team on a great merge! :D
2
5
131
Withdrowls are live and finalizing! Congrats to everyone on a successful upgrade! :D
#ShapellaUpgrade
6
16
132
Goerli forked and blobs are now enabled on the testnet!
We're debugging a client issue right now, we should have more info in a bit. Expect to hear more later today and a deeper dive at ACD tomorrow!
5
26
126
The Holešovice (Holesky) testnet is planned for launch roughly on Sept 15th(Merge day!). We've had 2 coordination calls and the testnet aims to give client teams a platform to test clients at a scale larger than mainnet and for home stakers to test their setups without risks. 1/n
9
35
118
The Shanghai+Capella upgrade is going full steam ahead! The first public withdrawals testnet is launching 1st of Feb at 15 UTC.
All info here:
#TestingTheWithdrurge
The Zhejiang public testnet is going live tomorrow (1st of Feb 15:00 UTC, 2023). Shanghai+Capella will be triggered 6 days later (at epoch 1350). You will be able to deposit validators, practice BLS change and exit without risk. All links are here:
13
81
183
8
30
112
After the fixes were in place, we quickly saw the participation and proposal rates go back up towards ~99%! About what we had pre-merge!
2
3
107
merge-devnet-5 has been launched. The beaconchain genesis happens at 4PM UTC and the PoW chain has already been launched. The TTD has been set to be really high to allow everyone to join pre-merge.
Tooling can be found here:
1
21
99
Best part of the merge being done - deprecating and deleting old configs/infra
6
0
97
The Ropsten testnet hit TTD at ~16:00 UTC (~2h after the planned time, great accuracy for PoW standards!). The chain was at ~99.2% participation rate with healthy proposals and sync committee performance - Pretty much the perfect beacon chain to merge, easy to see what breaks!
1
3
97
All L2s using Goerli can start testing EIP-4844 now, Looking forward to seeing the process unfold!
Interested in estimated timelines of all L2s here. EIP-4844 is hitting L1 testnets soon, and more different L2 testruns are needed.
What would be a safe timeframe between testnets and mainnet L1 4844?
6
21
105
2
23
101
wen merge?
2
3
97
merge-devnet-4 is live! We're still working on getting the tooling links up, but the configs can be found here:
4
29
99
The Ethereum mainnet fork is scheduled for 22:27:35 UTC on Apr. 12, 2023!
Don't forget to update all of your nodes! This includes the EL (geth/nether./erigon/besu..), CL beacon(lighth./nimbus/prysm/lodestar/teku),CL validator and mev-boost!
Update the host OS while you're at it
It's happening 🎊
Shapella is scheduled on mainnet for epoch 194048, scheduled for 22:27:35 UTC on Apr. 12, 2023 📆
Client releases compatible with the upgrade are listed in the announcement below 👇
753
946
3K
2
37
90
Does this mean there were no bugs? Almost..
While we have proposals (vital for consensus), some client pairs timeout while building a block and propose with 0 transactions in them. This would affect the throughput of the chain. We saw this in shadow forks too! Fixes incoming!
2
3
91
Announcing EIP-7665, Adding the EXTSELFDESTRUCT opcode to Ethereum. This would help the fight against state growth and allow us to perform the verkle transition in a matter of minutes.
@lightclients
and I are hoping that the EIP gets included for Electra
11
16
91
Time flies quickly in the crypto sphere. A year ago today
@protolambda
showed me how to setup my first consensus layer testnet, now we've even got merge testnets! If you want to help with merge testnets or any other DevOps tasks, join us at the EF!
12
21
88
Huge shout out to all the client teams who have been working tirelessly! Show the client teams some love :D
5
8
88
Finally, RIP Ropsten!
Feel free to use it short term, but don't expect long term support. Migrate stuff onto Goerli/Sepolia for the long term!
My (personal) 2c wrt
#Ethereum
testnest:
- Fork Ropsten into merge, deprecate (no more forks)
- Fork Sepolia into merge, recommend as new testnet
- Fork Goerli into merge, maintain as legacy testnet
- Not fork Rinkeby into merge, deprecate (no more forks)
8
11
136
2
2
83
merge-devnet-2 was launched yesterday and has been running well! We'd love to have people come test it with us!
@vdWijden
has created an awesome doc with ideas and links to resources here:
3
16
80
We should be releasing information about the kiln testnet in the next few days.
0
5
82
A hearty congratulations to the client teams and everyone helping!
4
1
83
We found 2 non-critical bugs that didn't stop finality, both were patched really quickly by client teams. I'm really glad we're looking at optimization bugs rather than critical bugs.
1
4
80
Holesky is happening soon! We crowdsourced requests for running genesis validators and created a genesis state that's 1,460,000 validators: Roughly 2x mainnet! Genesis will happen in less than a month :D
Holesky public testnet genesis file has been generated, and merged into main branch of .
Launch date Sept/15, 2023, 14:00 UTC
Chain ID-17000 (zip code of Holešovice).
The genesis will have 1,460,000 validators. Our biggest public network to date.
5
26
96
6
19
81
The releases for Goerli, the last testnet merge, are out!
Please make sure your node is up to date and configured properly!
Some config tips and common pitfalls can be found here:
#TestingTheMerge
📢📢📢 Goerli/Prater Merge Announcement 📢 📢📢
Prater will run through the Bellatrix upgrade on August 4th, and merge with Goerli between August 6-12th: if you run a node or validator, this is your last chance to go through the process before mainnet 🚨
306
260
859
2
19
72
Blobs are coming to a node near you! :D
Deneb timeline update🎉
Client release: Feb/22
Mainnet: March/13
6
36
126
1
6
78
Tune in to ACD tomorrow for a full incident report. But I do want to mention that we went from fork -> issue -> triage -> fix -> finality in under 4h. The debugging apparatus has gotten great!
4
5
75
To anyone wondering how to perform local shadowforks, you can do so with a single line command now!
We made some modifications across the stack to make it easier to pull a snapshot, run the genesis tools and get a local shadowfork in record time (for Holesky)
Local fully featured Ethereum devnets in one line, now with shadow forking and node selectors.🟢
Check out the Ethereum package v1.40.0 here:
3
3
37
5
18
76
Its the shadowfork time of the year again :D
We had a great goerli shadowfork yesterday (goerli-sf-0)!
This forked goerli and enabled the Dencun upgrade.
You can find the explorer here:
#TestingTheSurge
10
20
72
We launched a website :D
will now contain blog posts on topics we find interesting, a summary of all the tools we maintain and more(links to services we host is a WIP)!
We'd like to introduce our new website - your gateway to all things EthPandaOps and a way to keep up to date with projects we’re developing! 🌿
1
10
30
1
19
71
Client releases are out! This is the final step before the merge goes live on Ethereum. The first part of The Merge happens on the 6th of Sept in the Beaconchain, called Bellatrix. The full transition will happen ~15th Sept!
Update your nodes now!
This is it: The Merge is coming.
UPGRADE 👏YOUR 👏 CLIENTS!
+ More👇
470
754
2K
3
12
67
Please don't, use sepolia instead!
Tons of resources, faucets here:
And here's a landing page with links:
8
19
70
Next shadow fork will be once everyone is back home and rested,
#TestingTheMerge
won't stop until we merge :D
1
3
70
This was the first time we attempted a mainnet-shadow-fork, we were expecting to learn a lot from the transition! The next week or so would be spent with sync tests against this fork and trying to trigger more edge cases.
We plan on repeating it next week, for advanced users ;)
5
3
71
Thank you for hosting us for a Dencun watch party!
𝗕𝗹𝗼𝗯𝘀 𝗮𝗿𝗲 𝗹𝗶𝘃𝗲 𝗼𝗻 𝗘𝘁𝗵𝗲𝗿𝗲𝘂𝗺! 🐡🐡🐡
Cheers to the EF team for smoothly delivering the long-awaited upgrade. ️
Time to celebrate with some Dencun Donuts! 😋🍩
2
8
99
5
3
68
Are you interested in setting up a robust monitoring system for your Ethereum node?
Have a look at the guide from
@Kiln_finance
:
Most of it is overkill for home stakers, but extremely important for node operators! Either ways, peek behind the curtain :D
9
18
68
With the help of the community, we hit TTD and the participation rate dropped by ~13%. We began looking at the missing indexes and knocked on some doors. A large % of the drop was from a config issue on the Nimbus team's nodes, that was easy fixable.
2
1
64
The last missing percentage points are explained by a bug found by the nethermind team shortly before the ropsten merge. Thankfully a simple restart fixed the issue.
Few nodes might have failed on race condition during the transition. Just before Ropsten merge we were reviewing transition on a call and found out that this can potentially happen. Here is the error log. If it happened to you simple restart fixed the issue on our nodes.
2
1
18
2
1
62
We're happy to open source Attacknet! We've been playing with this tool behind the scenes since Nov and have used it for Dencun testing!
The ability to control network latency in certain network layouts will definitely help us on the path to peerDAS.
Attacknet, a tool we collaborated with
@trailofbits
on is out! It gives us the ability to perform chaos engineering on local devnets, allowing us to simulate edge cases before they happen on mainnet.
Find more info here:
2
7
28
2
13
66
The remaining missing participants are from a config issue in the besu team's nodes that affected just the nimbus CL. The EF nimbus-besu node continues to work fine, reinforcing that it isn't a bigger issue.
2
2
63
The node split on this shadow-fork attempts to mimic mainnet(roughly). Here's a grafana dashboard with how we're doing -> Green=Good, Red=Bad.
1
3
64
We had a misconfiguration in the el genesis file and that led to the holesky network improperly launching. Some validators manually fixed the config and were able to start the chain, but not enough for the network to finalize.
Holesky launch update!
Due to a misconfiguration in the ExtraData field (applied to EL configs but not to CL ones),
#Holesky
didn't initiate properly.
A consensus has been reached among the EF DevOps and client teams to relaunch a week from now.
10
19
61
2
14
61
The underlying PoW testnet had multiple forks, so different beacon node detected different terminal block hashes. The chains however agreed on a block hash and moved ahead as expected, the chain is finalizing and we had no downtime with 0 intervention. Great test overall! :)
2
4
60
Genesis for the Kintsugi testnet is in ~15mins, all links can be found here:
Information on joining has been updated here:
2
17
58
The kaustinen testnet is now live...again!
Kaustinen is a verkle testnet. This time with proofs in blocks that conform to the new consensus spec:
Find tooling links here:
4
16
58
Updates from testing land!
Dencun-devnet-8 is live! Its the next iteration in devnets, aimed to allow clients to test interoperability with the latest spec
8
12
55
Is anyone still reliant on Goerli for testing? If so, please move to a supported testnet!
Reach out if you need help moving.
Sepolia -> Dapps
Holesky -> Large validators
Ephemery -> Short term validator tests
6
21
58
Participation rate did drop, but we are still well above the minimum required for finality. The missing % needs to be 0 before we're going to ship the merge!
5
7
59
Why can't that day be today?
The entire bee movie script is a blob on devnet-10. Next step, the movie itself!
Thanks for
@__flcl
for the tooling ;)
blob morning
can't wait till blobs come out, so I can distribute this via the mempool
7
2
60
8
10
56
Introducing the new testing tool on the block: Assertoor
As the name suggests, it asserts behavior on a network. Imagine an end-to-end integration testing tool thats built to support client interop testing
Our newest blog post introduces Assertoor, a new end-to-end, cross-client integration testing tool that operates at a higher abstraction level than Hive.
Check out how it works and how we're using it to do automated testing on full scale ethereum testnets:
4
12
46
2
10
57
We're seeing a healthy number of blobs submitted to the network. You can expect deeper analysis over the next days. We'll be relying heavily on Xatu for the data:
2
5
56
Client hotfix incoming, seems like they've identified an issue. In the meantime, the network is processing blobs already! We've ingested >1k blobs
Goerli forked and blobs are now enabled on the testnet!
We're debugging a client issue right now, we should have more info in a bit. Expect to hear more later today and a deeper dive at ACD tomorrow!
5
26
126
2
6
53
Another small difference is that this shadow fork used the develop/unstable branch of every client, so we aren't using merge branches anymore. This is the code that will make its way into releases.
1
3
52
We had the
@EthereumOnARM
team taking part in the shadow forks too! So we will have blocks produced post merge by an ARM client.
1
0
53
If you ever wondered how we setup test network, what tools exist or what hidden dragons lie in the darkness, come for our workshop at
@ETHBerlin
:D
Pro tip: We'll make sure to show you a workflow on how you can test your EIPs with a local testnet, takes just one command!
*
@parithosh_j
and
@BarnabasBusa
, both devops engineers at
@ethereum
, will be running a workshop diving deep into the extensive tooling they use for single-host and multi-host testnets, aiming to showcase preferred options for various use cases.
1
3
16
4
6
45
Reminder that the Goerli network is deprecated and will not be supported for the future. Please make preps to move over your testing to Ephemery, Holesky or Sepolia networks.
Ephemery-> small validator testing
Holesky-> long term, large validator testing
Sepolia-> Dapp testing
1
17
49
Estimating TTD for merge devnets is quite hard since we have no idea what hashrate to expect, its a bit easier if we control all the miners.
There seems to be some external miners on Kiln, it'd be great if they could reach out to us so we can work together :)
#TestingTheMerge
1
16
48
The beaconchain genesis will be Friday at 14:00 UTC. The PoW chain has already started and we expect the PoW phase to last ~a week. Deploy your smart contracts now to know how they behave pre and post merge!
3
6
46
We're gonna continue monitoring the network over the week to look for issues, perform sync tests and stress the nodes.
1
2
46
@evan_van_ness
We'll soon have a more public testnet for everyone to play around with withdrawals :D
2
3
47
@dimsomedim
@protolambda
The devnets all forked at <256 epochs, meaning the network was too new to even hit one cycle of SLOTS_PER_HISTORICAL_ROOT. So we never triggered this issue in devnets. Shadowforks are more of an EL stress test, underneath they're just fresh CL devnets - so same reason.
3
3
46
We'd like to tear down the Zhejiang testnet on Wednesday (April 19th). Please migrate any infra that still runs there before then. The testnet was meant to test the Shapella upgrade and is now considered deprecated as the upgrade is live on mainnet.
Thank you for testing!
3
17
44
We still need tools built to help us debug issues more efficiently, there's a non-complete list here:
- Get intermediate state roots
- Trace block(by RLP)
- A way to pause/flush state in a client(Prevent pruning)
- Get Block receipt RLP
- PR in clients to dump dump invalid blocks/non-deserializable stuff received in gossip
- Visualize when merge is successful!!
2
2
25
4
3
44
Earlier shadowforks required us to sync and run a set of scripts to generate the forked network. We realized this wouldn't scale well with Verkle, as shadowforks would be one of the primary testing tools.
We were able to use
@KurtosisTech
to switch to local and automatic sfs
Slow and painful in the making, a shadowfork converting the Holesky state to verkle has finally occurred 🎉
10
26
152
3
7
46
New record by the team, looks like we can pull off mainnet size tests that don't cost an arm and a leg too :D
EF DevOps' largest testnet (in terms of validator count) has launched successfully this morning. The aim was to have a network run at mainnet validator set in order to test BLS key changes. Huge thanks to
@beaconcha_in
guys to help us get up and running with the explorer.
82
111
232
2
6
45
We reused the mainnet deposit contract with a new fork ID. This means that every mainnet deposit needs to be processed and listed as invalid on the shadow fork. This huge computation triggered some edge cases in some clients, the good news is that the network still chugged along.
1
3
42
Interested in testnets and what tools we use? Check out my talk at
#DevConBogota
! I'll try to highlight some of the tooling we built during the merge testing efforts, We built most of it for reuse by the community!: 🧵
6/11
🎥 Watch the
#Devcon
Main Stage talk:
'Quest for the Best Tests' by
@parithosh_j
, part of the DevOps Crew at the
@Ethereum
Foundation. 👇
1
0
1
2
10
39