I ended my time at
@Meta
as a director.
But I started as an engineer on FB Chat.
Everything about it was broken — we had to rewrite it.
And while the effort to fix it is one the projects that led to
@reactjs
, the most important fix was far simpler...
Here’s the full story:
—
I managed big software systems at Facebook and Robinhood.
Both were hypergrowth environments, but the difference between "reliability" in social media vs. finance was staggering ↓
I’m starting as VP of Engineering for
@RobinhoodApp
this week. I am super psyched! My
#1
factor for choosing any role is the people, and the Robinhood team is awesome. Here are three more reasons why I joined:
1/ I’m psyched to join ⚡️
@ElectricCapital
as an engineer.
Before I was VP Eng
@RobinhoodApp
, I led teams that worked on
@reactjs
at Facebook.
It's stormy 🌩 skies over cryptoland these days, but I want to tell a story about the early days of React that's not all ☀️ and 🌈
I’ve managed software engineers for 20 years.
I was VP of engineering at
@Robinhoodapp
and before that I was a director at
@Meta
.
The best part about my career?
The people.
Here's my love letter to software engineers:
(And their strange, skeptical and nerdy ways)
—
When I
Well, in the words of the great
@leeb
, you can't fix it if you don't measure it.
We quantified our reliability in terms of these conventional measures:
* MTTD (mean-time-to-detection)
* MTTR (mean-time-to-remediation)
* MTBF (mean-time-between-failures)
Junior engineer: Coding is hard and extra stuff is harmless, so don't delete this. We might need it later.
Senior engineer: Coding is easy and complexity makes stuff hard, so remove this. We can always put it back later.
Applies to code, data, and systems.
3/ Role. Robinhood’s whole engineering team is the size of the group of specialists I managed at Facebook. I enjoyed going deep in front-end technologies like
@reactjs
and
@grapql
, but I've always considered myself a generalist. I’m excited for a bigger role on a smaller team.
1/ Mission. Until now, I was ambivalent about the finance industry, although I found it interesting. It has always seemed like a rigged game to me. Robinhood’s mission to democratize access to financial markets resonates strongly with me.
This project to move our clearing in-house was what got me really excited about joining Robinhood. At first it will enable small changes like reduced fees, but over time it will change everything. It's also an exciting and complicated engineering project. Very proud of this team.
Introducing Clearing by Robinhood. It’s the only clearing system in the last decade to be built from scratch.
For you, this means:
▪️ Fewer fees
▪️ Improved customer support
▪️ A better overall experience
Read more about Clearing by Robinhood here:
Wow, the new React hooks API is lovely. I was lucky to get a glimpse of this before I left Facebook, and it's impressive to see how well it works as a fleshed-out design. I really admire the way the React team continues to find ways to improve what's already such a great library.
Very excited for
@RobinhoodApp
's first big open source release. I think streaming is the end-state for any scaled system, and Faust brings the hotness of Python 3's asyncio to this problem. Highly concurrent stream processing as a straightforward python iteration? Yes please!
We’re open sourcing Faust, a Python distributed stream processing library that processes large amounts of data in real-time, and makes it easier to design and deploy traditionally complex streaming architectures.
I love how, here at
@RobinhoodApp
, we work to make complex financial interactions more accessible by shipping them primarily as a mobile phone experience. Our mission to democratize access to the financial markets starts with world-class design informed by thorough user research
"We make use of simple colors to remove as much information as possible, so that it’s clear to the user what’s happening."
Alex Bond, our senior designer, spoke to
@Adobe
&
@99U
about our design philosophy. Read more on her approach to designing with us.
@rsms
@Meta
@reactjs
I'll never forget hacking on one of the earliest implementations of ~React Native at the IPO hackathon with you.
You showed me your hobby language of the moment and opened *my* eyes to what was possible with clang.
1/ I had the good fortune to organize
@ElectricCapital
's first hackathon last week. We had a ton of fun hacking with new tools like GPT and the team built some amazing public goods.
There were too many great projects to cover them all, but here are some highlights ⚡️
Great talk from
@schrockn
about the development of GraphQL, and how a GraphQL layer can function as a centralized alternative to a fully distributed microservices architecture. Anyone who worked at Facebook knows that centralized repositories and monolithic artifacts do scale.
If you have found product market fit as a financial team, you must now learn operational excellence. Start by tracking outages and quantifying their severity.
Once you find the lever for improving your reliability, you will be unstoppable. You will have nothing to fear.
1/ I helped organize the first
@ElectricCapital
Founder Hackathon last week.
I can't think of a better party than one where everyone stays up late writing code and it gets a little weird. We had so much fun!
There were a ton of great projects. Here are some highlights:
2/ Challenge. Robinhood poses a very pure engineering problem: using commodity cloud infrastructure to deliver financial services. Success will require Robinhood engineering to move faster and be more efficient than the competition.
@acdlite
I think reactive programming is mostly a dead-end. It generally trades initialization speed for update speed, which is mostly the opposite of what you want i UIs.
The hardest part of React's fake reactivity is lining up the data sources so screens only initialize once.
It was so fun to be back at React Conf this year! The progress on React and Relay are amazing, and I learned a ton from the brilliant speakers and attendees. Here's my little talk about how the Flux pattern applies to distributed systems
Our Engineering team uses Airflow to manage key workflows across Robinhood. Very proud of the work we've done to improve our Airflow usage as we continue to scale.
@dan_abramov
The alternative is to let debris and kindling collect around a part of your codebase until it catches fire. Then you can do a sexy, high-impact project to rewrite that part. Either way, the code you least want to change is probably the code that most needs change.
.
@ptarjan
and I first worked together at Facebook almost 10 years ago(!) I'm so excited to announce that now he's bringing his passion for productivity enabled by technology to help us streamline our operations here at Robinhood. Welcome to the team Paul!
Developing against production is convenient, but it's not an option for financial firms like Robinhood. Apollo gives every one of our engineers their own version of our backend to use for development. It's also great for integration testing.
@jlongster
I think this is what makes gradual typing so great. I agree that there's no sense in preventing unreachable states in a version of a program that will only live for 30 seconds in my dev environment. But you can "allow implicit any" at first and tighten the screws later.
At Robinhood, it was completely different. Every mistake had to be recorded and potentially remediated.
No one thought it was funny when customers lost their money.
At Robinhood, we have a fairly fragmented service architecture. In general, a given service manages all the writes to a single DB. However, dependent services need to read that data. We see two alternatives:
After some of the most exciting and challenging years of my career, that’s the end of my time at Robinhood. I’m still processing everything I learned, but I’m looking forward to sharing some of these lessons as they crystalize. I’ll be blogging here:
I’m currently working
@ElectricCapital
as an engineer.
But I also love to cook.
So while I’ll be posting more insights and stories from my career and time as VP Eng
@RobinhoodApp
and
@Meta
…
I might also share the occasional cooking post.
I know a lot of software engineers
Environments like dev, ci, and staging just didn't have the fidelity to catch the kind of cascading failures we saw in production.
That and half the incidents were caused by a config change, amirite? 😉
One time, when Facebook went down, people called the cops.
We felt bad about breaking, but also kind of proud that people relied on us so much. We mostly thought it was funny.
I could not disagree more. I've personally seen so many huge projects that started from hackathons.
Even discounting the very real ROI, the energy and camaraderie that hackathons generate is invaluable.
Hackathons are mostly a waste of time. Basically nothing created there goes on to become a success (with undoubtedly a handful of exceptions).
If there’s value there, it’s merely introducing future co-founders to each other, with the hackathon being a selection filter.
This year's developer report has some really interesting data. The future is bright for web3.
It's also a big project that requires meaty software engineering, data collection, design and analysis. It reflects a ton of hard work by my teammates at Electric Capital. Enjoy!
0/ We are delighted to release the 5th annual
@ElectricCapital
Developer Report
We analyzed a record 485 million code commits across 818k repositories—20x more code than our first report from 2019!
356+ people across the community contributed to making this report.
Long,
@jarredsumner
I've wanted this for package.json, but I'm afraid this feature being only in Bun will further fragment the already insane JS project landscape.
@sfchronicle
I unsubscribed last year and haven't regretted it for a moment. This is a completely unhinged take. SF is off the rails and we are trying to take it back with commonsense policies.
Optimizing MTTR is harder. It requires teamwork and some 💦
Get better at communicating. Stay cool under pressure.
Improve your runbooks. Clarify your escalation paths. Invest in your SEV process.
I couldn't disagree more. These mental gymnastics are only necessitated by the modern performance process.
Someone who helps everybody else be more awesome is also awesome. The fact that you can't easily measure their contribution is a shortcoming of the process, not the person.
Interesting comment on our High Performer post - what’s wrong with High Performers who just help people all day?
The problem is first that you can’t really quantify that behavior. Often the claims of how helpful someone was are different than if you ask
Since the beginning of the year, Robinhood has seen unprecedented growth of our platform. We are looking for world-class Site Reliability engineers to help us meet the demanding requirements of our industry. Interested in stability at scale? Apply today!
My main takeaway from
#reactconf
: progress on Relay makes a convincing argument that the future of GraphQL is bright. Data-driven dependencies, fetch-before-render, React Suspense integration 🔥🔥🔥
So happy to share our mobile development tooling platform, Sonar, in open source. It has some great functionality out of the box, but it also gives framework builders an interactive way to communicate with their users, beyond APIs and documentation.
@davidfowl
We do this in initContainers in prod.
I'm surprised at all the answers that suggest running migrations from CD. This creates undesirable coupling IMO and also leaves the potential for can't-go-forward/can't-go-back.
I'm working on a team with a medium sized React app. We want to monitor how fast our screens initialize in production. What's the best way to measure this? How do you know when a screen is initialized?
Jacques Pépin is one of the people I admire most.
A true culinary master, a skier, and a bit of a perfectionist.
He’s also one of my greatest programming inspirations.
I doubt he’s ever written a line of code.
But his core principles have influenced how I see creativity in
@marktenenholtz
What sucks about Django is that you quickly realize that the thing you are building has no value.
With languages like Go and Rust, you can write code for months without having to confront this.
Hey
#reactconf
! Come join me,
@leeb
and the rest of the
@RobinhoodApp
crew after the conference ends today for a little happy hour we are hosting down the street. RSVP here:
It's not lost on me that the historical documents "Silicon Valley" ends with the filming of a documentary about that crazy time.
Working on React in the early days was absurd, frenetic, and mostly, a lot of fun.
@Scott_Wiener
Although I was open to these kind of programs as a progressive way of dealing with drug over-use, I don't think it's working. It seems to me that e.g. needle exchange in SF has led to more use and abuse and deterioration of the civic commons.
@dan_abramov
IMO, the problem with ORM is that it conflates reads and writes. The model you get back from a query has almost nothing to do with how you write or update it. I think Redux and its ilk confirm that CQRS is a superior design. I'm giving a talk about this at FinJS NYC this week!
@housecor
I don't really see the point of this. I trust the coder to implement what they say. IMO, the purpose of the code review is to look for bad code patterns (like circular references or unnecessary caching,) poor variable names, missing comments and confusing factoring.
At first I was all team MTBF. I mean why get better at fixing things if we can just prevent them in the first place?
But prevention is a research project. How do you improve at it?
Ultimately, the answer was testing in production, but I'll save that story for another day.
It's 2023. Web browsers support multitouch and GPU, but they still won't let you run a local JavaScript file.
People wonder why we have all this framework bloat, but these days it's easier to start a new hosted project than to set up a simple local dev environment.
This is huge. The traditional financial system runs on trust, but we have seen that trust betrayed over and over again.
This is a small but important step in establishing a new system, one built on transparency from end-to-end.
Announcement: Today the Bitwise Bitcoin ETF (BITB) becomes the first U.S. bitcoin ETF to publish the bitcoin addresses of its holdings.
Now anyone can verify BITB's holdings and flows directly on the blockchain.
Onchain transparency is core to Bitcoin's ethos. We're proud to
@dan_abramov
I think it's different when you can, for example, accidentally leak secrets to your client because they share a codebase.
It can be difficult to reason about what data will make it to the client or guarantee that the client has the necessary permissions to write.
@ryanflorence
I think a big difference is that <?php echo> writes to stdout. Everything about PHP sits very solidly on top of UNIX primitives, down to requests running in a POSIX thread.
The new React abstractions are based on shifting sands of JavaScript. The abstractions leak like a sieve.
On the other hand MTTD is easy to optimize. You build realtime dashboards and alerting. You track your SLAs.
If will find out tomorrow what happened yesterday, you are going to lose this game.
When I started at Robinhood, the team already had a lot of great practices around production incidents. But we’ve also found that this is an area that benefits from continual reinvestment. Today, we’re excited to share some of what we’ve learned.
3/ But the audience hated it.
They had been brainwashed by bullshit like "don't mix your presentation with your business logic."
Everyone on the team felt like the public launch of React was a failure.
The difference between using Vim and using an IDE is the difference between being a creator and being a user.
Surrendering to the constraints of a tool like VSCode makes you powerless.
Be the master of your tools, and become unstoppable!
People like Vim because they've learned to be very productive in it after a long struggle which feels like more of an accomplishment than using a GUI where you can be immediately productive.
New blog post: Hipster Software 🧔
I am loving the combination of Rust and JavaScript. I know there was a little bit of backlash against some of the finer points of Rust concurrency last week, but overall I think it's a great environment and ecosystem.
Transitioning from large companies to startups can be both thrilling and horrifying at the same time. What to watch out? How to handle chaos? What to unlearn? Join us tomorrow to find out. RSVP:
2/Looking back, it may seem like React was a guaranteed success, but it sure felt like a failure at first.
When we demoed React
@JSConfUS
in 2013, we thought it was a slam dunk. I mean people were still writing -for- loops in their templates!
9/ Obviously, crypto is different from Web 2.0, but they share this arc about collaborating on software in the open.
I’m here for the big ideas. Software that can't be censored or blocked. True privacy. A web that connects every human.
@ElonBachman
I was astonished to discover that "a chicken in every pot" referred to a depression era hope that every family would be able to afford a chicken dinner.
8/ Where I come from, everyone wins a hackathon.
Hackathons are about more than greasy food and staying up late. They're about remembering how much fun work can be. They're about the joy of making things and the energy that gives you.
Let's build! 🔨
Whew, this article brought back some memories: 😅
If you've never worked on an app like Facebook, it can be hard to imagine the kind of engineering that goes in to performance at scale. Most iOS developers never deal with dylibs at all.
Really happy to see this success story with React Native. Very interesting that the team chose to rewrite rather than migrate. Perhaps this is the key to successfully adopting RN.
This year we've successfully upgraded our applications to React Native. In this blog we explore the engineering challenges and lessons for building React products at scale.
@dan_abramov
One of the worst things about Python is that it does not generally have this property because you need to adjust block indents when moving code around.
The Electric Capital developer report has been a bellwether for crypto. This year, we're excited to launch a website to go with it. You can dive into the data yourself!
Thanks in advance for your bug reports and feature requests.
6/ We had a stated goal of turning haters into advocates.
We found that to change someone’s mind, we had to risk changing our minds.
We had to welcome feedback even when it was harsh or ambiguous.
one thing that helped
@reactjs
get adoption early on was we were really nice to people who were mean on the internet, and then they turned into nice people
@thepanta82
@matthewcp
I used to make this argument, but now I think it's over-rated and I agree with the OP.
It's just too hard to make every form do the right thing with respect to updates vs. creates. Also, it's rare that retrying an update is really what you want.