sadservers.com
@sadservers_com
Followers
1,649
Following
450
Media
90
Statuses
538
Troubleshoot and make a Linux server happy!
Joined July 2022
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
BECKY X MAYBELLINE LIVE
• 230469 Tweets
Northern Lights
• 211450 Tweets
オーロラ
• 170154 Tweets
#Auroraborealis
• 92475 Tweets
DeNA
• 61864 Tweets
#baystars
• 61443 Tweets
THE SIGN in MANILA
• 43046 Tweets
京王杯SC
• 22051 Tweets
Fulham
• 21610 Tweets
太陽フレアのせい
• 18177 Tweets
ベイスターズ
• 18129 Tweets
ハマスタ
• 17413 Tweets
バチコン
• 16802 Tweets
ウインマーベル
• 10671 Tweets
Pinned Tweet
- Troubleshoot and make a sad server happy!
"Like LeetCode for Linux"
Capture The Flag challenges. Train and prove your debugging skills.
Practice for your next SRE/DevOps interview.
Get a full remote Linux server with a problem and fix it.
2
31
69
Too many people using SadServers today; we ran out of AWS spot instances
4
0
31
This eBPF documentary is better than most superhero movies! 🍿
Also it has a twist at the end I did not see coming
0
3
19
Another SadServers video thanks Kevan!
0
3
15
Had so many sessions and VMs rows in my tables that queries were slow. After archiving old ones, looking so much better. 💪
2
0
16
"The best thing a human being can do is to help another human being know more.” - Charlie Munger
0
1
15
I love this, finally with 🇧🇷 there are SadServers videos now in English, Spanish, Russian, German, Japanese, Hindi and Portuguese!
0
1
14
The Germans then the French plus Indians created a roller coster of love in the last two weeks!
So far the very basic infrastructure EIASPOF ("Everything is a Single Point of Failure") has been fine (except replay server buckling a couple times). Now the alerts are always on 😅
2
0
14
Ich habe es genossen und konnte es ein bisschen verstehen 😜 kaput!
1
0
13
So I made a thing this weekend: 😀💪
0
3
13
"maybe I should get also the (singular) domain" f*cking squatters lol-sob
1
0
13
Things look stable now. Hitting AWS limits hasn't happened since we were on top of Hacker News last year🤔
I really need to look into Firecracker...
2
0
12
SadServers is hosted in AWS but were not affected by the recent us-east-1 outage.
It took a lot of planning for failure, knowledge of distributed systems and a complex and resilient implementation. Oh and also, we are on us-east-2 😜
1
0
10
Finishing the year with a new feature , now there's only the little detail of writing good troubleshooting guides 😀(docker one probably next).
Tired of seeing the same content being recycled and full of ads.
0
5
11
When the solution to a SadServer scenario is correct, now I’m displaying the time it took (I can add later a comparison to others “you did better than 82% of users who solved the scenario”). 💪
I’m also shutting down the server automatically, bit of extra savings! 💸
0
1
11
I have a nice Grafana board for my SLOs and all that.
Little Schrödinger detail: if I see the graph I'm DoS'ing myself. Fun. Any SREs around? 😅
2
0
11
I like reading post-mortem reports and learn how people deal with incidents and troubleshooting, like this one
0
0
10
Hug of death from Hacker News. Hitting AWS quota, requested bigger quota and temporarily reduced VM life span, thanks for the patience!
1
2
9
Thinking of adding a tag for scenarios that are more “realistic” (also more apt for “interviews”) and one for more “unrealistic” ones (what are other good words for this? like more weird/folly/creative/fantasy)
1
0
9
After giving hints for a couple months around my house, I got this for Father’s Day, I may take it out in September or so
1
0
7
Google search spam is out of hand (this is a top result for a listmonk query, there’s another one comparing it to PHP), in Spain for this we say “you are confusing speed and bacon”
1
1
9
When you leave an ssl certificate renewal to the last minute 🙈 , anyways fixed now (thanks Let’s Encrypt certbot!)
1
0
8
“Open sourced” the 1st scenario with Ansible playbook and Hashicorp Packer template (either works)
Added description and clues too.
This can help in case someone wants to submit their scenario and also for suggesting changes in current scenario wording.
1
0
8
If you also love SQLite (one of my favourite pieces of software) you want to hear this podcast with its creator
0
0
9
New feature coming very soon: stop-and-resume (or destroy) instances
1
1
8
We are on for the Linux World Cup!
This Sunday September 24th and next Sunday October 1st
Details at
0
5
8
@MariuszMichaow3
@mxssl
Nice to see SadServers as the link to “SaaS where users can test their Linux troubleshooting skills” in
0
0
8
Finally caught up and answered all feedback from contact form.
Types:
- 80% Positive comment 😀
- 8% Check Solution not working right sometimes (sorry, will investigate)
- 10% Giving another solution (excellent)
- 2% issues with terminal 🤷🏻♂️
0
0
7
I guess Christmas Eve is as good a time as any to send the SadServers newsletter
0
1
7
@_terpsichora
I have a new hard-ish scenario ready, but I made a lot of code changes I haven’t pushed yet related to a new user dashboard to keep track of what scenarios you’ve done, number of attempts and times.
Hopefully this weekend I can take another review and push to prod 😄
1
1
7
There's likely going to be a lot of new people visiting today. Let's see how the servers and AWS reacts. Please be patient if you are affected.
0
0
7
Traffic sources to SadServers last week 🇫🇷 🇮🇳 🇺🇸 (
#MapsWhereGreenlandIsBiggerThanArgentina
sigh).
Look at all those people coming from Alaska :-)
1
0
7
When the domain registrar “auto renew” had one job :(
On the other hand, happy first birthday to us! 🎂🎉
0
0
7
The repository has reached 1,000 stars, nice 😊
0
0
8
Youtube video of someone showing how to fix the Cape Town (borked nginx) scenario:
1
1
7
@merycodes
Plot twist: it may have never been $120
Udemy has agreed to pay $4,000,000 to settle a case alleging it advertised fake discounts for its courses.
Remember: using fake reference prices to run sales is illegal under the FTC Act and California's laws. 1/
26
100
457
0
2
6
Finally caught up with all email (replied to all comments). Take away:
- I'm killing Santiago and Alice (!), this scenario is confusing and giving a lot of grief.
- I need to troubleshoot (oh the irony) the check solution call.
0
0
6
Spent the Sunday refactoring the database and fighting with Django's ORM. Created new warehousing in the backend with easier to get aggregated data.
In the front the user dashboard is going to be fast
& compact. This will also recover "archived" (older) data into the dashboard
0
1
6
Linux World Cup finals is over, will let the running sessions drain then will calculate the rankings.
Bear with me for about a day or so since I need to review (replay) many sessions (probably this could be crowdsourced).
Thanks all for participating and to the volunteers!
2
1
6
New scenario with haproxy load balancing in front of two nginx servers using Docker, hope you enjoy this one!
0
2
6
Scenario tags are implemented in SadServers now, see for example
Also and should eventually redirect properly.
0
0
6
Re-introducing a previous scenario that I had taken down: “Santiago”: Find the secret combination.
This is an easy task-based scenario (like “Saskatoon”) rather than a troubleshooting one.
0
0
6
People from 126 countries have used SadServers so far! Here's a breakdown of the top countries (with > 1% of total, it's a long tail) 🌎🌍🌏
0
0
6
Added code that will just warn if there are no servers available instead of getting the spinning wheel.
1
0
6
👋🏻 Created a new scenario “Oaxaca”: Close an Open File , not sure how difficult or how useful but non-experts may learn a thing or two 🤔
0
2
6
✅ A ranking page is available now:
⏱️ To do:
- add average time to solve and % solved per scenario
- allow users to enter an “alias” in their dashboard
- link to page (new top bar menu)
1
1
6
Better database optimization, happy queries and it's easier to push changes like this to SadServers (a feature with a lot of requests)
0
0
6
Created a
@sadservers
@mastodon
.social account, now I just need to remember to look at it from time to time 😅
0
0
5
First weekend of (one simple qualifier) is over. Saw no significant issues 😀
Tomorrow I’ll email the users that passed (system should have told them) & can go to the final next weekend (same times).
Want to show best times on 1st try & links to replays.
0
2
5
Now I've been hitting the last increased AWS ec2 quota, requested another increase.
2
0
5
Datadog has been frustrating. Every other line of my logs is how it cannot check the disk of a Docker volume. Tried many things even RTFM, can’t still figure out how to tell it to ignore.
Replay system has been broken because calling asciinema returned also a f datadog message!
2
0
5
After some open heart surgery yesterday, the messaging system is so much better. Pics of the before and after for RabbitMQ, nothing better than seeing an empty queue in a system based on background jobs!
0
0
5
Working on adding users ssh keys or auto generating an ephemeral one for scenario VMs (this opens security concerns, so only for some users).
Also working on logging command history and being able to replay your own or somebody else’s bash session. Pretty much needed for LWC
1
0
5
New Linux troubleshooting scenario "Pokhara": SSH and other sshenanigans 😀 .
Set as “hard” but it’s more like many ssh/login/special surprise guest annoyances (I left out a couple subtle ways to break ssh for maybe a different scenario)
0
1
5
AWS Quota issue fixed, so any other issues would be on me ;-)
1
0
4
Added more restrictions today to the number of scenarios that can be used in a row and number of total servers as a quick measure to deal with the heavy load usage today while I work on more stable solutions.
0
0
4
New Linux challenge "Roseau": Hack a Web Server
This is a fun scenario (I think) where we have to break passwords. No superuser this time, so you can’t “cheat” (other than reading the solution in the provided clues) 😀 let me know if you find another way!
2
3
4
@marcelpociot
This is what I use. It doesn’t look original but it’s simple and it works (I don’t even use jquery but plain JS).
I just want to offer a clean functional site, don’t want to learn a new framework every six months. 🤷🏻♂️
0
0
3
10 Linux (Docker, web servers, networking...) scenarios now 😀
0
2
4
New scenario , a word histogram one (counting word frequency). Typical programming exercise (map reduce anyone?) but can be done with Bash/Linux tools.
0
0
4
I’d be very thankful if somebody who has done some of the hard scenarios wants to test the Linux World Cup challenges instead of competing 😀
Please DM me here (I lost track of who can do this now here) or email info
@sadservers
.com
0
1
4
Oops, made big changes last night and forgot to test unauthenticated users, site was broke for them. Just fixed it.
Also for some reason with the packages upgrades looks like Sentry stopped working and didn’t warn me
OTOH public instances and personal keys are GA for VIP users
1
0
4
After some code optimization, system is performing better and is more stable, cheers!
0
0
4
New scenario Melbourne dealing with WSGI and Gunicorn , enjoy!
0
2
4
I simplified the logic for the VM lifecycle management and things look much better. The downside is that now all VMs are started directly from AMIs so about 50 seconds start time instead of about 30.
0
0
3
Let’s have some fun. If there are coding competitions why not a Linux competition?
Sign up if you want to play or collaborate 🐧🌍🏆
0
3
4
Starting the year with a new scenario Lisbon , it involves SSL certificate (tried to make it trickier but it is what it is, taking suggestions) and etcd (although no knowledge of it is needed)
0
1
4
@TheJobfather__
Excellent list. If only there was a site to practice Linux troubleshooting under time constraints as well! 😅
0
0
4