Institutional Data Initiative
@instdin
Followers
145
Following
3
Media
5
Statuses
14
A research center at Harvard working to strengthen society’s connection to knowledge by advancing our access to and understanding of the data that shapes AI.
Joined August 2024
RT @leppert: This Monday, @instdin will host @petrknoth to share his experience leading CORE ("The world’s largest collection of open acces….
0
2
0
RT @leppert: Tomorrow, it's our pleasure to host @ayahbdeir to talk about the power of data in building an AI ecosystem that's open, transp….
0
2
0
We hope Institutional Books will be the beginning of a process that makes millions more books accessible to the public for a variety of uses. We welcome feedback as we continue to expand this dataset, refine its contents, and sharpen our process.
0
0
2
We look forward to growing Institutional Books through community. We welcome collaboration from researchers and model makers as we:.- Evaluate the dataset’s impact on model outputs.- Continuing to refine our OCR pipelines. View the dataset on Hugging Face:
1
0
3
As part of our refinement work, we supplemented the original OCR-extracted text with a post-processed version that utilizes line detection to reassemble the text according to the line type.
1
0
0
We included extensive volume-level metadata with both original and generated components, such as results from text-level language detection.
1
0
0
We analyzed the dataset’s coverage across time, topic, and language and found:.- 40% of English text + long tail of 254 languages.- 20 clear topical tranches.- Largely published in the 19th and 20th centuries. Technical report here:
1
0
1
Today we released Institutional Books 1.0, a 242B token dataset from Harvard Library's collections, refined for accuracy and usability. 🧵
4
13
38
