Excited to share a new position paper I wrote on a recent exciting trend in generative AI: autonomous AI agents. These are are capable of accomplishing tasks entirely on their own and we at
@SFResearch
call them LAMs—Large Action Models. For more details:
1/2 Salesforce Research's open source programming language model (CodeT5) is selected by Forbes among the best 5 AI tools that can generate code to help programmers! Congrats team
@SFResearch
!
When it comes to choosing an LLM, there are two paths to take: open-source or not. This interesting article explores the benefits of going with an open-source LLM.
#SalesforceAI
#AI
Excited to introduce a new generative large multi-modal model for image generation and editing using natural language! Great collaboration Salesforce AI Research
@SFResearch
and Stanford U!
Web:
ArXiv:
Code:
AI Agents can autonomously plan and execute complex tasks. AI Agents are the next GenAI product frontier and Salesforce AI Research (
#SalesforceAI
) is leading the way (see recent blog) -- 1/2
The discussion on how small foundation models are reshaping the landscape of enterprise continues! Read more in a recent Financial Times’ piece (where I am quoted) and my own blog:
#SalesforceAI
#AI
#ML
Very proud of our Stanford team
@cvgl
@zamir_ar
Will Shen, Sasha Sax and collaborators for their amazing work on taskonomy! — aka how learn a taxonomy of visual tasks for solving general computer vision and A.I. problems. Best paper award at cvpr 18. Well done!
Enabling large language models with planning and execution capabilities is key for unlocking automation in critical enterprise-level use cases. A new approach from
#SalesforceAI
sheds more lights into solving this challenging problem:
Excited to share that our newest multi modal foundation model xGen-mm is out and it’s open source! It’s small (<5B models) and shining in both pre-trained and fine-tuned benchmarks.
Check it out 👉
@huggingface
:
@SFResearch
#SalesforceAI
#AI
#ML
Very excited to share this new research
@SFResearch
on using language to generate executable code. Learn more about this breakthrough in conversational AI programming!
Paper:
Blog:
Code:
Time-series forecasting methods perform poorly on long sequences when data changes over time. DeepTime overcomes this issue by using forecasting-as-meta-learning on deep time-index models. Result: state-of-the-art performance and a highly efficient model.
[1/2] Excited to announce a new version of our simulation environment, iGibson! More functionalities and scenes to develop and train robots in interactive tasks in large virtual environments
website:
SAIL blogpost:
Very proud and excited that this initiative (that I have been co-leading with collaborator Jayesh Govindarajan) finally came to light today! Big shout out to our
@SFResearch
for building the models under the hood!
New exciting research to make long and complex documents more accessible and simpler to read. Brought to you by Salesforce AI Research
@SFResearch
!
🔗Paper:
🔗Github:
Thank you,
@Benioff
, for championing open source innovation. MINT-1T is poised to accelerate multimodal AI breakthroughs. I'm looking forward to the diverse applications our community develops.
#AICollaboration
🚀 Introducing MINT-1T! We've just launched the first trillion-token open-source multimodal interleaved dataset. This groundbreaking resource scales up data diversity and size, enabling the training of larger, more capable multimodal models. Perfect for research and innovation in
Small foundation models are reshaping the landscape of enterprise AI and point to a more inclusive future for AI where cost-to-serve, training efficiency, sustainability and accuracy are key assets. Read more in my latest blog:
#SalesforceAI
#AI
#ML
"Shared imagination" in AI isn’t a mere curiosity. Our new research reveals that LLMs agree on hallucinated content up to 86% of the time. This could fundamentally reshape our understanding of AI cognition:
#AIResearch
#AIsafety
Introducing ULIP-2, our latest work scaling multimodal pre-training for 3D understanding without the need for any manual annotations. Check out our code & released large-scale tri-modal datasets!
Arxiv:
Blog:
The future of work isn't just about AI performing tasks. It's about AI evolving alongside us, learning our organizational DNA. Excited to share my thoughts on the future of autonomous AI agents in business.
#FutureOfWork
#AutonomousAgents
How will autonomous AI systems revolutionize enterprise operations? Our Chief Scientist
@silviocinguetta
breaks down the potential and challenges ahead. Explore the future of collaborative AI.
#AIInnovation
"
#FutureofAI
New collaborative research effort between Salesforce AI Research
@SFResearch
and XLang Lab leading OPEN LEMUR -- a state of the art pre-trained LLM for code generation.
Excited to see that recent work by
@SFResearch
on multi-modal generative AI is gaining strong momentum: Diffusion-DPO is a new method to align diffusion models to human preferences by directly optimizing on human feedback.
#SalesforceAI
#AI
Brushing up on your Italian? 🇮🇹 Check out
@silviocinguetta
on CNBC's (
@classcnbc
) latest AI Special where he talks about
#GenerativeAI
and its impact on both business and society while keeping trust and ethics front and center.
Link:
Congratulations on Didi for starting their new R&D Lab in the Bay Area! I helped cutting the ribbon w/ Didi CTO Bob Zhang, major Ken Rosenberg and other tech VIPs :-)
Salesforce AI Research
@SFResearch
is helping shape the future of development and unleash developer productivity. More details are in this exciting blog by
@yingbozhou_ai
The ability to perform tasks across modalities (text, image, video, audio, and 3D) is the next frontier of LLM research. X-InstructBLIP
@SFResearch
is a new approach showing the emergence of cross-modality capabilities combined with efficient training.
Excited to share our work on understanding the relationship between environmental complexity, evolved morphology, and the learnability of intelligent control.
Paper:
Video:
w/
@silviocinguetta
@SuryaGanguli
@drfeifei
It's exciting to hear that our work
@CVGL
at Stanford on Real-World Perception for Embodied Agents (GIBSON) has been recipient of the 2018 NVIDIA research award (handed in by Nvidia CEO Jensen Huang in person!). Congrats to
@zamir_ar
, Fei, Jerry, Sasha for their terrific effort!
Long sequences can be the Achilles' heel of LLMs. The ThinK method's 20% memory reduction without performance loss redefines our approach to memory efficiency. A must-read for anyone working with LLMs:
@SFResearch
&
@Mila_Quebec
announce the AI for Global Climate Cooperation working group & competition.
Help the world by building climate change solutions, using AI to design negotiation protocols & climate agreements. Join us!
@AI4ClimateCoop
Learn more
Very excited about this project by Salesforce research on accelerating multi-agent reinforcement learning. Check Out blog and related info about Warpdrive here: Blog:
OpenSource (include Colab tutorials):
Very thrilled that my SVL Stanford colleagues
@jcniebles
@drfeifei
and myself are contributing to a new exciting cycle of AI /vision research at CVPR 18!
Can
#AI
language models learn from evolution to design proteins? Learn how Salesforce is taking a step towards enabling solutions to cure disease and clean our planet.
Blog:
Paper:
That’s right,
@Benioff
. Atlas also achieves an unprecedented 95%+ faithfulness and 90%+ relevance. Our semantic consistency breakthroughs usher in the next era of trustworthy and powerful agentic capabilities for enterprise AI.
#AIScience
#TrustedAI
@SFResearch
Salesforce's newest version of Agentforce, code named Atlas, resolves 90%+ of all customer inquiries for our top healthcare, financial services, payments, travel & entertainment clients—over double the success rate of competing Agents. State-of-the-art results. See it at
With our 1B and 7B parameter
#SLM
series xLAM outperforming larger models on specific tasks, we're proud to be leaders in the path toward
#AgenticAI
. Great piece on
#Salesforce
's vision from
@TheDrum
It was a honor to have the chance to join a discussion panel on AI and ethics with Ravi Prasad — IT minister of India
@rsprasad
and other Stanford colleagues.
It was an extra-ordinary visit to Stanford University. Had a wonderful exchange with its brilliant faculty on use of technology for human development and the application of AI and the challenge it poses. The faculty was deeply impressed with India's story of digital inclusion.
I’ve been reflecting on why it’s so important to put every-day reliability before novelty when it comes to testing the effectiveness of LLMs. I’m proud of our recent LLM benchmark breakthrough. For details, see:
#LLM
#TrustedA
Happy to see our team's hard work come to fruition. The xLAM family of models represents a huge leap in AI capabilities for function calling, planning and reasoning—fit-for-purpose for varied needs of modern business. Eager to see where its application takes us!
#AIInnovation
Introducing the full xLAM family, our groundbreaking suite of Large Action Models! 🚀 From the 'Tiny Giant' to industrial powerhouses, xLAM is revolutionizing AI efficiency!
#AIResearch
#AIEfficiency
🤗 Hugging Face Collection:
🤩 Research Blog
Introducing JackRabbot 2. Like its predecessor, the intentionally cute "social" robot is learning to navigate safely through spaces occupied by people, following the rules of human etiquette.
Thrilled to open-source MINT-1T today—a huge leap in multimodal data. I'm eager to see how the AI community uses this to advance research in multimodal reasoning and generation:
Breaking news! ➡️➡️➡️ We just released the MINT-1T 🍃dataset! One trillion tokens. Multimodal. Interleaved. Open-source. Perfect for training multimodal models and advancing their pre-training. Try it today!
Blog:
Dataset:
Totally agree with these concerns. I have been following the situation of North Italy: saturated ICUs and zero capacity at hospitals are creating a serious healthcare disaster
The overload and then collapse of regional healthcare system is my major concern - this has happened in Wuhan, Iran and N. Italy. This is why some regional fatality number is so much higher. Every local/national government should try hard to prevent such tragedy from happening.
Very excited that Einstein GPT is coming to life and proud that Apex (and more) generative capabilities will be powered by large foundation model developed in-house by
@SFResearch
!
Get ready to be wowed by Salesforce EinsteinGPT! It generates leads, closes deals, writes Apex, and even makes coffee (just kidding, but wouldn't that be amazing?)
#EinsteinGPT
#TDX23
💼☕️😎
Stanford today soft launches our Human-Centered AI Initiative. I’m very excited to be co-directing this initiative with former provost John Etchemendy. Our mission: To advance
#AI
research, education, policy, and practice to benefit humanity.
@Stanford
Using both natural and artificial abilities, the human relationship with tools has drastically evolved. The best tools are powerful because they’re easy to use. This is where our skill of language and AI meet.
Learn more on how conversation can power AI >
Excited to share that SF Research
@SFResearch
launched an AI for Global Climate Cooperation to explore how
#AI
& economics can help combat climate change. Our work was featured by Financial Times
@FT
. Congrats to the team and our collaborators at Mila!!
Large Action Models (xLAM) from
@SFResearch
continue to outperform on newly updated LLM Benchmark for CRM. We just added evaluation for AI agent use cases, assessing 19 top models on function calling, argument accuracy, and responses. Try it!
#AgenticAI
Salesforce AI Research and AI Frontier launched the world's first LLM benchmark for CRM. Customers can now select the right LLM for their tasks based on metrics that matter – accuracy, cost, speed, trust & safety. Check it out!
#SalesforceAI
#AI
#LLM
Don't miss upcoming episode “Salesforce AI Research: Shaping the Future of CRM” today at 4:30pm PT! I’ll be presenting along with
@CaimingXiong
,
@VenaLi14
, and Vera Serdiukova.
Save your (virtual) front-row seat 🎬🍿
#DF21
Are you overwhelmed by the never-ending slack messages? We’ve got you covered! Learn how Salesforce AI Research is transforming the way we work.
View our episode "Salesforce AI Research: Shaping the Future of CRM" tomorrow at 4:30pm PT!
#DF21
After 3+ years, today is the day that my book “The Worlds I See” gets to see the world itself. It is a science memoir of the intertwining histories of me becoming an
#AI
scientist, and the making of the modern AI itself. All versions are now on Amazon 1/
Fine-tuned as a specialized AI agent, our “tiny giant” xLAM-1B can run on-device or local servers while making decisions, executing tasks, and interfacing with systems more efficiently than larger models. Open source version coming soon!
#AgenticAI
#SML
Interesting project that sheds insights on how state-of-the-art Gen AI still falls behind humans when it comes to creative writing:
Great collaboration between
@SFResearch
and Columbia U.
#NLProc
#HCI
Great to hear that our work "GONet" has been selected as the Best Paper Award Finalist on Safety, Security, and Rescue
#Robotics
at
#IROS
2018. Congrats to Noriaki Hirose,
@marynel_vazquez
, Patrick Goebel,
@silviocinguetta
for the terrific team effort!
Hi, we are the
@Stanford
Institute for Human-Centered Artificial Intelligence. AI has the potential to transform our world – how will we ensure it improves life for all of us? Join us in our work to explore this dream of a better future.
#StanfordHumanAI
Big congrats to Salesforce
@SFResearch
for earning top ten on
@HuggingFace
's list of datasets tonight. As the largest open source multimodal dataset, MINT-1T will advance your multimodal training. We're eager to see what you do with it!
#multimodal
#AI
The power of diversity in AI is undeniable. Our recent work
@SFResearch
shows that integrating diverse AI software engineer agents as code reviewers can more than double problem-solving capabilities. It mirrors the benefits of diversity in human teams!
Today's release of xLAM-1B on HuggingFace exemplifies our strategic focus on compact, powerful language models. Another step forward in our mission to make AI more accessible and impactful for all.
Just in! Our “Tiny Giant”, xLAM-1B-fc, has officially arrived on
@huggingface
with a few friends!🎉
Check out for our suite of small agentic models, including xLAM-1B-fc and xLAM-7B-fc with mobile-ready, quantized versions now!⚡️
#LAM
#AIModels
#AI
🤗:
What type of data and perceptual problems need to be solved to bring robots to navigate and interact in human environments? Come to figure out to our morning workshop at
#iccv2019
Room 307A! Great speakers and our new dataset from JackRabbot, the JRDB:
Expanding the power of generative AI from text to images and now to 3D! Check out what
#SalesforceAI
's ULIP-2 is doing and how it is bringing us closer to a future where AI can truly connect us with the 3D world: at
@SFResearch
Imagine if machines could comprehend 3D objects the way we do. That's what
#SalesforceAI
's ULIP-2 is doing: redefining 3D understanding.
Read how
@SFResearch
is bringing us closer to a future where AI can truly understand our world:
How can novel architectures take dev productivity beyond tooling to true human-AI collaboration? CodeGenie's a great start. A new blog outlines the research roadmap for 24/7 pair programmers.
#AIResearch
#SoftwareEngineeringAI
CodeGenie's impact: 2M+ lines of code accepted, 500K+ chat questions answered. But the real breakthrough? Our vision for a 24/7 AI pair programmer. We’re transforming software engineering, exploring continuous learning and adaptive coding assistance. Check out the technical
Exciting work from Salesforce research on AI for software. Check out CodeT5 —AI driven tools for software development.
Blog:
Paper:
GitHub: …
#codeintelligence
Meet CodeT5 - the first code-aware encoder-decoder pre-trained model that achieves SoTA on 14 sub-tasks in CodeXGLUE! Learn how it’s disrupting software development.
Blog:
Paper:
GitHub:
#codeintelligence
I wrote a blog post about how we can build increasingly intelligent robots & deploy them in the real world, outside of factories. It cites a bunch of awesome research from a lot of really smart friends/colleagues!
Part 1
Part 2
Getting help from LLMs when editing a document can involve lots of manual copy-pasting. What if LLMs could suggest one-click edits directly in a text editor as you
review it? Check details in this amazing work by Philippe Laban and et al.
@SFResearch
#AI
)
Excited to share MobileAIBench-our new open source framework for assessing mobile-readiness of LLMs. Benchmarks spanning NLP, multimodality, trust and safety. iOS app included! Paper: Code:
#AI
#MobileAI
#Benchmarking
#NLP
#LLMs
Excellent thread from
@togethercompute
on LlamaRank. Our novel RAG reranker's 92.9% hit rate and linear scoring bring new levels of performance and interpretability to AI systems. Grateful for this partnership to advance
#ExplainableAI
!
Introducing the Together Rerank API: A new serverless endpoint for enterprise search and RAG systems.
We're excited to be the exclusive launch partner for
@salesforce
@SFResearch
LlamaRank – a new reranker model that outperforms Cohere Rerank in document and code ranking tasks.
Happy to share my latest blog on "AI Balancing Act: How Companies Can Scale LLMS to Improve Performance, Cut Costs, and Help the Environment":
@SFResearch
#SalesforceAI
#AI
#ML
#BLIP
-2 is accepted to
#ICML2023
! BLIP-2 and our open-source library LAVIS have driven many interesting multimodal applications, including miniGPT4. Looking forward to see more BLIP-2 use cases!
Blog:
Code:
Trevor Standley did a great job in presenting our latest work
@Stanford
Vision Learning Lab on estimating the weight of generic objects from images! That's a critical step for enabling robots to effectively interact with the environment.
Want to build bots better? Try Converse: a new Task-Oriented Dialogue System that simplifies chatbot building while handling complex tasks and conversations.
#NLP
#AI
Code:
Paper:
Blog:
Interested in state-of-the-art AI tools for understanding and managing relationships between cause and effect in complex systems? Check out Causal AI -- a new library created by Salesforce AI Research:
GitHub:
Blog:
#AI
@SFResearch
Mark your calendars! For the 1st Colloquium on AI4AEC: "Is AI ready for the building industry? (and vice-versa)". We lined up a great set of speakers from both AEC and AI - join us online on Aug 19th and 20th!
@ir0armeni
@fischermartin
@silviocinguetta