This was a huge headache in the early days of
@MosaicML
, so we built our tooling to seamlessly handle GPU failures. Our platform will detect a faulty node, pause training, cordon the node, sub in a spare, and resume from the most recent checkpoint. All w/o any human intervention
Hardware failures are common while training the largest machine learning models across thousands of GPUs. It is similar to the elder days of computers, when a vacuum tube burning out during your batch computation was a real issue.
52
102
2K
3
5
90
Replies
There's also a pile of smoldering A100s in the alley behind the office, but I don't think it's polite to post gore on twitter
0
0
9
1
0
7
@leavittron
@MosaicML
Don’t forget that deterministic resumption because of our streaming datasets!
0
0
7