Happy to share our paper on language modelling with pixels has been accepted to ICLR‘23 (notable-top-5% / oral) 🎉. Big thanks and congrats to Team-PIXEL @jonasflotz @ebugliarello @esalesk @mdlhx @delliott and looking forward to presenting in Kigali! 🌍 #ICLR2023.

I will be presenting this work in-person at ACL🇹🇭 this week. Drop by if you'd like to chat!. Oral: Today (Monday) 16:30.Poster: Tuesday (Tomorrow) 10:30 - 12:00.

This project is a collaboration with my amazing peers and mentors during my internship @AIatMeta: Bowen Shi, @skylrwang, @ncihancamgoz @j_maillard. ⭐. 🧵(9/9).

For more experiments and all the details, check out our arXiv preprint linked above. We are working on releasing our code and data, so stay tuned! 👨‍💻. 🧵(8/9).

We also highlight the importance of pretraining on longer video clips to learn long-range spatio-temporal dependencies 🎬➡️🧠. Even when controlling for the number of video tokens seen, we observe a large boost in performance by scaling from 16 to 128 frames 🚀. 🧵(7/9)

Face blurring incurs a loss of linguistic information in sign languages, leading to performance degradation. We show that such information, when lost during anonymized pretraining, can largely be recovered during finetuning. An effective privacy-performance trade-off ⚖️!. 🧵(6/9)

Our best models outperform the prior SOTA for ASL-to-English translation performance on How2Sign by over 3 BLEU in both the finetuned and zero-shot settings 🥇. 🧵(5/9)

🌐 Optionally, an intermediate language-supervised pretraining (LSP) objective can help bridge the modality gap between sign language video inputs and text outputs. 🧵(4/9)

Our method, SSVP-SLT, consists of:. 🎥 Self-supervised video pretraining (SSVP) on anonymized, unannotated videos to learn high-quality continuous sign language representations. 🎯 Supervised finetuning on a curated SLT dataset to learn translation-specific information. 🧵(3/9)

Training data scarcity and privacy risks are huge issues in sign language translation (SLT). Our approach is designed to be.🚀 scalable (by enabling training on unlabeled data).🎭 privacy-aware (through anonymization). 🧵(2/9)

Introducing “Towards Privacy-Aware Sign Language Translation at Scale”. We leverage self-supervised pretraining on anonymized videos, achieving SOTA ASL-to-English translation performance while mitigating risks arising from biometric data. 📄: 🧵(1/9)

RT @gaotianyu1350: New preprint "Improving Language Understanding from Screenshots" w/ @zwcolin @AdithyaNLP @danqi_chen. We improve langua….

RT @delliott: In PHD: Pixel-Based Language Modeling of Historical Documents with @NadavBorenstein @rust_phillip and @IAugenstein, we apply….

RT @delliott: In Text Rendering Strategies for Pixel Language Models with @jonasflotz @rust_phillip and @esalesk, we design new text render….

RT @yoavgo: anon policy survey is out:.

RT @MetaAI: Introducing SeamlessM4T, the first all-in-one, multilingual multimodal translation model. This single model can perform tasks….

RT @delliott: 📢 I am hiring a postdoc to join our project on pixel-based natural language processing. The position is based in Copenhagen….

RT @delliott: Thrilled to receive a grant from @VILLUMFONDEN to carry out blue-skies research on tokenization-free NLP .

RT @coastalcph: CoAStaL [ will hire several 1-3y postdocs next year. Topics: fair ASR, historical handwriting, expl….

RT @delliott: Are you looking to do a Ph.D/Postdoc in multimodal learning? I'm supporting applications to join my group with a fully-funded….