Senior Game Producer - World of Warcraft - Occasional 'SERVER' - Nothing on this account is endorsed by Blizzard Entertainment, all tweets are personal opinion.
Thankfully WoW's site reliability engineers are awesome and are churning through the work, we're making sure there's no other surprises waiting for us and then we'll get you back in as quickly as we can.
The root of this one is an update to the teeny little service that bootstraps the VMs and delivers OS updates etc. Deployed fine in test, on Live, WoW did *not* approve and we locked up our file stores globally. We're left with a lot of manual cleanup to get things moving again.
Very much aware of the gold farmer issues in Season of Discovery at the moment, radical steps being taken and more action coming asap. In the meantime, remember, friends don't let friends buy gold. Give them a high interest rate loan instead so they can wallow in eternal debt...
Alright, so we’ve been looking at getting Wrath Classic realms condensed down ahead of Cataclysm launch, SoD style mega realms. But, while doing that we realized we have an opportunity that we’re honestly too afraid to take. Thoughts?
Faerlina-US and Benediction-US are fake PvP
We meant it when we said we’re committed to no bad choices for realms for Season of Discovery. Play on any realm you can. If and I mean long shot if, any realm falls short of viability anytime during this season we will get you off, for free.
We identified a gap in our coverage and have so far already closed several thousand accounts involved in RMT in Season of Discovery in both US & EU. By we I mean our awesome Risk team who are working on their weekend to get things back under control.
Since some folks seem to find this fascinating (hey, check out careers in Engineering and Live Ops!) I'll give you one more chunk of behind the scenes/thought process behind some of this. If you're after Classic WoW news you can skip :)
1. Why wasn't the change that caused this
Seen some chatter about faction balance going to crap, alas no, your faction likely just sucks at PvP. Harsh I know.
Lone Wolf US - 3.32% - Horde
Living Flame US - 1.29% Alliance
Crusader Strike US - 0.15% Horde
% is max delta, Horde are catching up on most realms! This wasn't
Season 2 dropping on Thursday, a quick look at the PvP realms in US and EU for faction balance... Aaaah, ok that feels good.
Things that could be improved? For sure! The balance at max level and the balance between people who *actually* engage in PvP is going to be different.
Caveat, there's no announcement of any changes in this, this is all just fascinating info if you're into that sort of thing.
Layering, Sharding, let's dive in.
So, in what we call Mainline (modern game, latest expansion) we use tech called Sharding ever since Warlord of
So that's pretty nifty, we've manage to get every single person wanting to play Season of Discovery in game today, no queues anywhere in US. I've still got some noodling to do to see if we can get more folks in in EU but overall, not bad for some mahahoosive mega realms!
Nothing of value to add, just eugh. Its going to be a bloodbath for the industry this year. Today was my 18th Blizzard anniversary, glad irony didn't take the opportunity to smash me in the face this morning.
Fine fine, I'll sneak you in an RP-PvP realm, don't make me regret this. Happy thanksgiving to all you North Americans, as a Brit expat I still don't entirely understand this holiday, but any excuse to eat more roast turkey is a good one to me.
Ok, for disgruntled Chaos Bolters, first, I absolutely understand and acknowledge your feelings. This is the gross and uncomfortable reality of realm management. Every time someone wades into the realm management discussion on the team this is a great example of the impossible
See some folks a little triggered, note the your faction likely sucks at PvP comment is a joke, this is just how I talk, I am a troll. This is also why I don't forum and am not a community manager.
I hope everyone knows none of what we've planned for SoD realm management is meant to be nothing but net positive. Classic has developed an awful narrative around good and bad realms to the point if the realm isn't queued its viewed as dead. Cont...
Layering Part 2: Design & Engineering Clash!
Dramatic title, not actually any conflict here. So what's the deal with the increase in grumbling around layering in Season of Discovery. Because we've not actually changed anything, Classic, TBC and Wrath all ran the same tech.
The
Aight, made my forum debut! We have some extra spice in store for Season of Discovery, check it out! Fun fact, Fwoibles was my GM name back in 2009, what a long strange trip is been!
One of the more random things I am responsible for from time to time is Realm Names. Season of Discovery is coming up, give me your best name suggestions.
1. Can't be in-use already.
2. Needs to exist within Classic Era 1.15's ecosystem.
3. Ideally no more than two words.
Season of Discovery players, question for you. How has 50/50 PvP realms worked out for you? If you had an exit route to a PvE realm, would you want it?
Things were a bit spicy there, it's been some years since the layering tech took a beating like that. We're back in action now and slowly raising realm caps. All realms are currently 50% larger than a queued Wrath launch realm. All good places to play.
I’ll see if we can rustle up something more official but since I keep seeing it asked, yes the faction balance worked, the average delta between Alliance and Horde is 3.7% globally with no PvP realm worse than 5%. Alliance tried to take off, give ‘en hell Horde!
For anyone wondering, the realm shenanigans aren't *only* trolling. We're just making sure all our ducks are in a row for launch, but a quirk we've never cared enough to fix is that even when a realm is hidden to players (appears Offline to you, Online to us) the realm list...
Lot of chatter about layering and STV perf, I know it feels like lip service but we're well aware and digging. Classic is doing their thing and Server is diving into finding what's driving the poor performance. Believe Classic will have some updates for you later, in the meantime
And if you're the wrong side of the character locked realms, we genuinely feel for you, but its a rock and a hard place. Those realms are stuffed and allowing more people to call them home is just going to bite us harder down the road, we've a long season to go!
Ok I've been up to more anti-botting/gold selling hijinks, sorry we've been quiet on this but we were specifically keeping this one close to the chest to ensure maximum effect. More details on the official forums here -
It bares reminding, this little line in our post on realm expectations for Season of Discovery -
"Please note: we will actively manage the realm sizes at launch, so you might see sporadic queues as the populations spread out and we ensure realm sizes remain roughly equal." 1/2
Seeing chatter about faction balance getting destroyed by FCMs. Just to head that off, there's a reason we're juggling these FCMs so much.
US PvP Faction Balance Days Before FCMs -
US PvP Faction Balance Last Night -
EU did start to drift more than I'd like, pre-FCMs -
Post-FCMs
...still picks it up! So since we're playing our cards close to our chest this launch, it required some temporary names. Though, I am quite fond of A Realm. Maybe not so temporary?
The read between the lines there is if everyone tries their hardest to pile into a single realm, you're going to run head first into a queue that we're not going to do a dang thing about. Only when all realms are equal will be raising realm caps for that ruleset. 2/2
We’re digging into what driving the unexpected “layering” aka transferring party members to other layers unexpectedly. To be clear you should only see your party members move between layers when someone joins or leaves, and only then if the layer you’re on is over populated (and
While it has been fun watching the community desperately analyze anything possible while we tested out some of the Seasonal realms in US today I think I’ve finally found the ultimate solution!
One of the main reasons behind perceived faction imbalance is layering, unlike Warmode in Mainline, it doesn't faction balance so its a race to get the most of a factions players into a layer before it fills ups once the raids start forming.
We routinely see people label realms that have more players than a queued Vanilla realm on them as "dead realms". So it's a real goal of ours this time around to dig our heels in and hold firm in the face of a single realm with a massive queue...
If your realm is "Locked" on the realm selection screen, it means you can't create characters there if you don't already. These realms are beyond full and you'll need to pick a new home. Reminder, there is no realm right now that wouldn't be consider "High" or queued normally.
Remember if you've already installed the Shadowlands PTR or Beta clients, you'll have downloaded 99% of the data already, the client automatically shares the data on disk.
@Zendorx
Even more than in US, EU decided it Living Flame was the only place to play SoD so they crushed one realm at the expense of the others so as we'd said we'd do before launch, we locked it. When we briefly unlocked it, folks started rushing over to it, so we locked it again.
Aside
EU-Living Flame is (temporarily) open for new characters, likely to close at any moment without notice, watching to see who moves in but if you were chomping at the bit to join your friends.
@ItsYent
Connected realms? It’s a bizarre amount of work that causes a ton of fallout for years to come for customer service. Realms have millions of characters on, if 1/100000 has some special problem that occurred in its life it blocks the process and had to be individually dig into.
The wrong "dead" choice? That *only* peaked at 10x the size of a queued Vanilla realm. These Classic WoW realms are MASSIVE compared to Vanilla. Layers were required tech because in Vanilla one realm = one copy of the world...
But taking Hardcore launch as an example, the EU realm Stitches the "winning" realm this time around peaked at a player number that was the equivalent of 20x queued Vanilla WoW realms. The "losing" realm Nek'rosh, that by player standards is...
But the world cannot hold a fraction of these large player numbers. So layering came in and retooled the modern WoW sharding tech to create multiple seamless copies of the world that a realm could use to sustain these much higher modern realm caps...
Going to have another stab at various values, if this seems like alchemy, well it is. It’s algorithms and player behavior uniting in glorious chaos. I could use your feedback though. If you played STV last night, did…
1) Lag feel better?
2) Going your party together was better
Alright first proposal for names sent, we’ll see what everyone thinks. Cheers for all the suggestions! Saved off the best fit future use, there’s always more realms.
@Kirarozu
They probably would if I asked them to! But the forums are still *insanely* larger audience than this and slightly terrify me. The larger community there can do really weird things with everything we post. Like this random tidbit of info would get weirdly twisted or argued into
Aight, more values changed, let me know how (from literally this moment) STV feels, groups should get split less (the code change to avoid this behaviour isn't in yet).
@Aaoogaa
Just isn't tech that exists in Layers as it exists now! It wasn't such an issue until the servers got so large and we now have an incentivized world PvP area. All lessons learnt and new things to improve going forward.
@SardonicCB
CB is not going anywhere, if you want to remain in it as a small community you’re more than welcome to. But we said with launch if a realm started shrinking too fast we’d let people move for free. I have to honor that statement.
@KushimOfUruk
Lava lash is an RP realm, people chose that specific ruleset and there isn’t another RP realm to merge with. We generally leave RP realms alone. RP-PvP tends to be less RP focused, at least in the US, so we treat it more like a PvP realm.
Avg. number of DB queries per second on Beta last week, ~2,500. Avg. number of DB queries per second since I turned on Char Copy... 173,000 and rising. I feel like I just took a lazy coach potato and kicked him out the door to run a marathon. You can do it buddy!
@Dildyjxd
Yep! That change applies to all PvP rulesets. We could have ignored RP-PvP as historically balance is pretty good on those realms, but it felt right that if we're enforcing balance, we should do it everywhere that's PvP.
@kezyn_
We've been opening Free Character Migrations from realms that fall below healthy player numbers for the last three years of FCMs now.
I can't help on the friends issue sadly, the tech isn't there yet to allow everyone on the same realm so multiple realms are still a necessity.
@Dildyjxd
The realm list populations are actually based on the current largest realm, everything is then based on a percentage value of that realms peak. Mega realms make all other realms display 'Medium/Low'. We've chat about changes to this over the years, all have drawbacks.
To temper expectations, beware, this may rapidly descend into a terrible mess. The PTR is not a full production environment. It's OK, we're learning everything we need to on our end. Delicious lovely data.
@sleekhare
Clarification! Words are correct but its the free version of a character transfer, no faction change! The faction discussion is because more of one faction might move than another, leaving the original realm unbalanced.
@RossGuthro
@SardonicCB
By the numbers, new players weren’t rolling there anyway. And leaving it open to new creation just creates a back door to bypass faction balance on Crusader. It’s not perfect but working with the toggles we have.
@El_carpeto
Sadly no, PCTs feel wrong for a seasonal realm and FCMs have a habit of ruining realm health/balance. We are likely (potentially temporarily if everyone hasn't lost their lust for Living Flame-EU) to open character creation their again 1-2 days after Phase 2 launch for those that
@JuceeRob
They always are, unless the STV you’re trying to go to excessively exceeds it’s player cap. Which is what you’re running into. Letting more people into an STV that’s creaking at the seams hurts a thousand people to make a free group experience better. Not unsolvable but it’s a
Watching
@EsfandTV
react to Classic Beta coming back online is probably one of the most wholesome enjoyable things I've seen in a long time. I hope he blows up big with Classic, such a genuinely happy person to watch.
@MartonVarga5
@shirubusutori
@EU
Unfortunate and definitely not the result anyone wanted, this is the kind of WoW service-adjacent change that shouldn't have had any impact on us and due diligence was done testing to make sure but on Live ran into an unexpected bomb. Live Ops is fun \o/