@_arohan_ At high performance computing conferences we have an artifact evaluation to provide all of the above information and badges are being awarded based on that - () might potentially also be a recipe for machine learning conferences?

@ludgerpaehler This is a great initiative and idea, hoping ML conferences appreciate the engineering aspects :)

@_arohan_ wouldn't a seed with the init code be enough for 0th ?

@vlordier No because frameworks dont agree on type of init or it’s parameterization.

@_arohan_ This may help, while also uncovering a slew of differences due to low-level software stack (like this one from pre-GPU era )

@_arohan_ I think for a stochastic process, the entire trajectory is required. Seed, initialized weights and 1st opt step is good for gd or adam?

@_arohan_ Also a checksum of the data.

@_arohan_ Data can be the trickiest part for reproducibility.

@_arohan_ @weights_biases has entered the chat 😎

@_arohan_ And the data shuffling order?

@_arohan_ Another helpful one is releasing all the model predictions on the final dataset; not just an average metric.

@_arohan_ and random seed.

@_arohan_ Yup yup! Great idea.