Brian Scanlan
@brian_scanlan
Followers
2,465
Following
1,784
Media
253
Statuses
4,562
Working at @intercomeng . Runs a bit. He/him.
Swords, Dublin, Ireland
Joined May 2009
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
オーロラ
• 492250 Tweets
Joost
• 334472 Tweets
#الهلال_الحزم
• 85607 Tweets
Fulham
• 81859 Tweets
#SixTONESANN
• 78237 Tweets
Man City
• 62845 Tweets
自分これ
• 61859 Tweets
Gvardiol
• 54232 Tweets
Ali Koç
• 47760 Tweets
Dremo
• 41197 Tweets
太陽フレアのせい
• 39467 Tweets
Burnley
• 33746 Tweets
SEE YOU NEXT MONTH JIN
• 31685 Tweets
دوري روشن
• 30597 Tweets
الدوري الاقوي
• 28769 Tweets
Granada
• 25210 Tweets
روشن السعودي
• 24591 Tweets
Luton
• 22315 Tweets
Haley
• 22196 Tweets
やまとなでしこ
• 14763 Tweets
الدوري التاريخي
• 14541 Tweets
#الهلال_بطل_الدوري_التاريخي
• 11927 Tweets
Trying out the new "Least Outstanding Requests" algorithm on AWS ALBs on Intercom's busiest endpoint
8
59
252
I really like my team's use of variable names in this monitoring configuration - the consequences are very clear.
2
206
240
We’ve been retiring cloud dev environments in favour of laptop based development at Intercom over the last few months. Modern Macs are serious machines, and the ergonomics are hugely better.
If laptops keep getting faster, cloud based developer tooling is going to become less appealing. These machines are so powerful that most of the value add can run locally. Maybe we'll get a hybrid model, SaaS running locally, but it's clear where things are heading.
161
213
2K
16
19
231
At
@intercomeng
we just went three weeks without any out of hours pages. Plenty of stuff alarmed when we were in the office, but it's still very satisfying that our product and infrastructure doesn't rely on constantly interrupting engineers' lives to keep it working well.
9
21
213
52,937 RPS on Intercom's Rails monolith at 14:01 UTC today. We're coming for you GitHub! 😀
9
29
187
No big deal - just my life partner getting awarded a Fulbright scholarship for research into diversity in technology. She'll be working in
@UCBerkeley
this time next year!
25
2
152
@zackkanter
I'm not a Musk fanboi, but there's a vast distance between the two. From what I've read, Theranos was built on very little real science. Tesla and SpaceX were ambitious for sure, but were built on top of incremental progress based largely in reality.
10
1
137
Last week, Intercom shipped code to production 1073 times.
5
37
131
Last week Intercom took the week off as a company holiday. I helped cover oncall and incident command, and we had to deal with the grand total of zero pages and incidents.
5
11
127
Doing work in an actual office for the first time in 873,264 days.
1
0
93
@IntercomEng
To get to this state we:
- build on reliable AWS services and re-use patterns and infrastructure where possible
- Alarm on customer facing metrics not symptoms
- Practice continuous improvement, reviewing every alarm that pages somebody and take action to ensure their value
2
7
88
The data is in! We're now spending $18k a year on public IPv4 addresses due to the new AWS charge.
10
12
89
Me and my wife don't celebrate Valentines, because I *broke* it many years ago. I bought her a toaster when we lived in Barcelona. We didn't have a toaster. She likes toast. IT 👏 WAS 👏 A 👏 GOOD 👏 TOASTER.
11
2
87
Moving to PlanetScale is going to result in a lot of red diffs from our code base 😍
4
7
84
@IntercomEng
Here's a writeup I did of our out of hours oncall setup if you're interested in learning a bit more about it :)
1
21
85
AWS's product strategy of having a cloud implementation for every manpage in Linux is going well.
1
21
79
Decided to head over to the Boston Marathon instead of running parkrun this morning.
9
0
81
Boston was unbelievable. I got way too excited and ran aggressively for 16 miles, then my quads detonated on the hills. I still smiled my way through most of the rest despite… mild… discomfort 😰
8
0
81
I spoke at Rails World earlier today about the history of Intercom's architecture. Here is a highly accurate diagram of the evolution of said architecture that I used in the talk.
3
3
80
@Cathal_Dennehy
Those trees will play havoc with his Garmin! I hope he gets a pace band at the expo.
1
2
73
A week in the life for the Intercom monolith Rails application. Each colour change is a new version in production. The area under each colour corresponds to the amount of requests processed.
3
17
73
I got my 7th intercomic today, celebrating my 7th(!) year at Intercom. If you've been in a meeting with me since the whole pandemic thing, the background should look pretty familiar. I've only used my pink hoodie sparingly this year but it's still notorious.
4
0
74
This month I ran every day, covering 290 miles including two PBs at 5k (18:07) and 10k (38:18). Probably my best ever month of running since I started around this time 5 years ago! 🏃🏻♂️😅
7
0
72
I’m gonna be talking about “Monolith-ifying Microservices” at Rails World!
The
#RailsWorld
speaker lineup is live! Meet all of the speakers coming to Amsterdam this October to help shape the future of Rails:
0
39
160
5
5
70
Now we have c6i.24xlarge hosts. Interestingly it looks like the same CPUs as m6i.24xlarge. The cost savings for one of Intercom's services means $150k/year drop in our AWS bill, we immediately saw close to 20% fewer instances running after putting them in service.
Real world performance of m6i.24xlarge vs. m5.24xlarge running Intercom's largest API workload. This is nginx request rate, which is the best measure as we're using Least Outstanding Connections algorithm. They're doing just under 20% more work.
3
2
29
4
9
70
Rails doesn't scale, etc.
Bit of stuff I'm working on got me curious so I dug around in Intercom's US data hosting region and found we're using 900+ instances, 34 different instance types and it takes 25,000 vCPUs and 65TB of memory to serve our customers. We deploy to all that in 6 minutes 🤯
5
5
74
6
6
66
Dealing with significant technical debt right now - untangling the Xmas lights. Why did we rush putting them away last year?
3
5
63
I published a writeup of this morning's outage of Intercom's Europe region. Not exactly fun to write these, but getting them out publicly and quickly feels good.
We published a postmortem on our November 22nd incident:
0
0
0
3
1
63
For the sake of transparency, here's the guidance I gave to Intercom's engineering Slack channel last week (STRIKE is the name of our out-of-hours virtual oncall team...). It's the same policy as we've had for years.
Need to bootstrap a good 🎄holiday🤶deploy🎅policy🎄?
@IntercomEng
has an extremely sane one. Borrow it! 🍷🎁🎉
0
5
27
1
10
62
I'm speaking at Re:Invent this Tuesday evening!
If you are going to AWS Re:Invent in December, I'm giving a talk at it about Intercom's engineering principals and how they apply to our use of the cloud. Here it is in the session catalog:
1
3
30
2
3
61
I love the brutally minimalist and simple technology used to count Irish elections. Also the whole "proportional representation" thing. I am not being ironic or sarcastic.
7
4
59
A good example of documentation being kept up to date 👏🏻
1
2
57
The current spate of tech layoffs remind me of when I was laid off from The Irish Times in 2003, and
@moranjustin
very kindly and immediately offered to meet for a pint, which I took. Seemingly small gestures can matter a lot, do take some time to mind each other.
4
7
56
I recently had a chat with
@Lethain
and it just got published on Intercom's podcast! We mostly talked about technical leadership as an engineer, a topic that Will recently wrote a book about (see ). Check the podcast out here:
0
11
56
I wrote up yesterday's Intercom outage. Hoping to have fewer outages of course, but also trying out being more public about the details in the future when they do happen.
We published a postmortem on our October 12th incident:
0
0
7
3
0
56
Ok! Here's a thread about one of Intercom's biggest outages 🧵
1
8
56
I bought a pressure washer today. It's a very middle-aged thing to buy, but omg there's so much childish fun in water spraying the shit out of everything.
6
0
55
My twitter feed is a rollercoaster of third doses and positive tests.
2
1
51
Proud to announce that, almost entirely due to the global pandemic, I finished 311th in the world in the 50k running distance in 2020 😹
8
0
54
Today on Intercom's blog I wrote about some good and bad technical strategies, mostly about use of the cloud. I originally spoke about these at the Dublin AWS Community Day a while ago, but they make for a pretty fun and opinionated read too:
2
11
53
@houlihan_rick
@QuinnyPig
@travisci
To nobody’s surprise, a company purchased by a private capital investor rots to its core.
1
1
53
A few months ago I gave the best talk I've been part of. It's about oncall, with one of the best people I've ever worked with, and I've finally figured out how to link directly to it in vimeo:
4
8
51
There's a custom emoji in Intercom's Slack that is quite cynical about what "green-i" really means.
6
6
52
We're working on building a version of Intercom that will be hosted in the EU, and it's almost time to bring in real people to start using it. Anybody interested in trying it out? Sign up below or DM me.
2
14
51
I just published what happened today:
Battle hardening never really stops I guess.
In addition to checks for primary key space, I now also recommend checking for all references to those primary keys.
3
0
15
7
5
50
macOS Catalina doesn't support 32-bit applications, so I'm using that as an excuse to call it a day on my Football Manager 2011 game where over 69 seasons I took lowly Sant Andreu to be the greatest club in the history of European football
7
2
48
The beta went well, so today we released Intercom’s European data hosting. A huge infrastructure project, with a very simple outcome - you can use Intercom with the data stored in Europe.
We're working on building a version of Intercom that will be hosted in the EU, and it's almost time to bring in real people to start using it. Anybody interested in trying it out? Sign up below or DM me.
2
14
51
4
4
49
@mipsytipsy
At
@intercom
we killed the 24x7 ops team, and replaced it with a transient team of volunteers from across the org:
Teams still do oncall for their stuff during office hours - which is actually when things tend to break anyway :)
1
8
49
First post-pandemic and post-stupid knee injury race this afternoon. One of the best races around. Looking to run it 4 minutes slower than in 2020 😭
1
0
48
The trend of databases being able to query data in S3 is super interesting. The latest is
@MongoDB
:
@honeycombio
's journey to S3 & Lambda architecture is fascinating:
AWS have Athena and Redshift able to query S3 too.
3
8
46
This book is worth its weight in gold for the opening chapter alone, which is about how to make technology choices. The AWS guidance is excellent too. Recommended!
THE GOOD PARTS OF AWS
Now available:
Almost everything we know about AWS, packed into a short 2-part book.
Part 1: The Good Parts
- The Default Heuristic
- DynamoDB
- S3
- EC2
- EC2 Auto Scaling
- Lambda
- ELB
- CloudFormation
- Route 53
- SQS
- Kinesis
37
148
913
1
6
45
AWS released “Capacity Rebalancing” for EC2 spot instances yesterday, and I quickly turned on at Intercom!
2
6
45
omg
@rails
tweeted me
We often see engineering teams breaking services out of monoliths...at
@Intercom
they did the opposite. In his talk at
#RailsWorld
@brian_scanlan
shares how they moved multiple services back into their Rails monolith for productivity and reliability wins:
1
3
63
4
0
44
Younger kid supports Man City for the same reason that I support Liverpool I guess, so here we are.
3
0
44
It's interesting to think about things S3 got objectively "wrong", yet has been so spectacularly successful.
- "US Standard" used to span both the West and East coasts of the USA.
- The bittorrent interface.
- Eventual consistency in general.
- The ease of data leakage.
4
4
43
Paul was one of my dearest friends. RIP.
Paul Byrne was a lot of things to a lot of people. A talented comedy director and stand up fixer, he was loved by so many.
But to me, he was my pain-in-the-arse little brother and I will miss him so much.
RIP Paul Byrne 1977-2022
2K
442
42K
13
0
43
I am 100% certain that this expiry implementation has been the cause of at least one production bug.
1
0
43
1
8
35
We all know that the Irish covid tracking app was open-sourced before it was released, but I didn't realise that this included the terraform setup for the backend in AWS 🤯
1
6
38
Here are some things we've written about how Intercom goes about this type of work:
Out Of Hours Oncall:
Incident Management:
Incident Reviews:
2
3
38
I still have my office fob in my wallet, a curious mix of optimism, nostalgia and lazyness.
8
0
38
I love that this quote from
@mipsytipsy
in this github issue template used at
@intercom
for when an alarm fires 😃
2
4
38
Looks like I'm in, just booked my flights for Boston!
For the first time since 2013, the “cut-off” time is 0 minutes & 0 seconds. We’re proud to share that all qualified athletes who applied Nov. 8-12 will be accepted into the 126th
#BostonMarathon
, provided their qualifying time is verified.
#Boston126
27
118
852
6
0
37
S3 Intelligent Tiering in practice with one of our largest buckets - I moved it from a lifecycle policy that put everything into Infrequent Access around December 1st, paid a penalty for a month but now it's ~50% cheaper.
2
4
36
10 years ago at Intercom,
@dehora
wrote "contrail", a java based data processing system. It ended up being used for a far narrower set of use-cases than originally envisaged (Rails got good in the meantime!) but it has basically run with only very minor wobbles for 10 years.
6
2
35
Our flawless household covid-free streak got shattered today. One kid in isolation so far...
17
1
35
Proud to announce that I am making my stand-up debut in the Laughter Lounge in Dublin on Thursday week. I'll be talking about EC2 cost optimisation at this AWS Meetup:
4
1
34
For years I've been thinking about building an alternative to nmap called "ScanLAN". Maybe this is the excuse I need!
It seems it hasn't received any wider attention that nmap is no longer GPL, and the new license has some hard to interpret clauses which sound like they're made with the purpose of creating a legal minefield?
7
55
115
3
0
33
@danluu
Intercom is largely a monolith, and we’re continually monolithifying stuff that’s out of the monolith.
1
1
32
Just got a booster appointment from my GP, they're texting their list a
@Calendly
link to book in, pretty cool!
2
1
33
The one weird trick that worked really well here and got us to this place was years of consistent focus on operational health.
1
0
31
If you are going to AWS Re:Invent in December, I'm giving a talk at it about Intercom's engineering principals and how they apply to our use of the cloud. Here it is in the session catalog:
1
3
30
I did a talk at
@datadoghq
’s Dash conference about oncall at
@intercom
and
@lizthegrey
did a great tweet thread of it that’s better than the talk itself :) Thanks Liz!
1
7
31
Getting to work with Ciaran has been a highlight of my career. He's probably the most positive and ambitious person I've ever worked with, and a world-class engineer too. Thank you Ciaran! ♥️
Last Friday was my final day at
@intercom
, the company I cofounded 10 years ago along with
@eoghan
@destraynor
and
@davebarrett
. It has been such a privilege to have been part of the adventure for so long!
62
4
402
1
0
31
My kids figured out that the fastest way to skip a Youtube ad is to mark it as inappropriate. They are scarily fast at doing this. Pretty sure Youtube has my account categorised as an unreliable crank.
2
1
30
@mipsytipsy
At Intercom we:
- Ask folks to take their time off! Most parts of the company shuts down. Why ship?
- Don’t block deploys. CI/CD works as normal.
- Tell folks to take more responsibility and make more noise than usual. Tell us if you’re deploying and keep an eye on things!
1
4
29
My
@ServerlessChats
podcast is now online!
@jeremy_daly
and
@beccaodelay
were lovely to chat to and are super pro at what they do. Thanks to
@mipsytipsy
for the intro too 😀
Episode
#119
: Scaling your Startup with Brian Scanlan (
@brian_scanlan
) is now available! Listen online or with your favorite podcast app! 🎧
#serverless
0
3
2
1
6
30
I've seen this around the place, and Jon is the original source! I think that its popularity is down to the great advice, along with the exciting colour scheme.
1
3
30
Me at my annual review in work.
Still the best Roy Keane interview ever! 🔴
122
969
7K
1
1
30
Real world performance of m6i.24xlarge vs. m5.24xlarge running Intercom's largest API workload. This is nginx request rate, which is the best measure as we're using Least Outstanding Connections algorithm. They're doing just under 20% more work.
🆕 Amazon EC2 M6i Instances Powered by the Latest-Generation Intel Xeon Scalable Processors are now available!
M6i instances offer up to 15% improvement in price/performance versus comparable fifth-generation instances.
3
15
50
3
2
29
@SageCanaday
I don’t think most people understand the difference between a sub-4 marathon and a sub-3 marathon. Most sub-3 marathoners understand the gap between them and 2:30 marathoners though!
3
0
29
