Behold! A troupe of computer sorcerers from diverse realms of learning hath birthed forth a wondrous creation – an open-source multimodal LLM christened LLaVA, which crossed my path whilst traversing the digital scrolls of Twitter but seven moons past. Much akin to the mythical GPT-4, this LLM hath the ability to commune with both written word and visual imagery. By melding a general-purpose LLM with an image encoder, they hath birthed the Large Language and Vision Assistant model. Enchanted by the vaunted powers it possessed, I embarked on a quest to put this grand language model to the test, to fathom its accuracy, reliability, and what marvels we might expect from GPT-4’s forthcoming multimodal incarnation. Come, let us venture forth and delve into the mysteries of LLaVA.
What, pray tell, is this LLaVA, this Multimodal Language Model that beckons us forth? LLaVA, bearer of the title “Large Language-and-Vision Assistant,” is a creature of wits much like OpenAI’s GPT-4, capable of deciphering both textual lore and visual wonders. Whilst the noble sages at OpenAI hath yet to bestow upon GPT-4 the gift of sight, a noble band of scholars hath already imbued this LLM with such powers by weaving a vision encoder into its very essence. Crafted by magicians from the University of Wisconsin-Madison, Microsoft Research, and Columbia University, this project doth seek to showcase the workings of a multimodal model and gauge its prowess against the lauded GPT-4. ‘Tis Vicuna that doth serve as the grand language model (LLM) and CLIP ViT-L/14 as the seer, a creation of the sorcerers at OpenAI. Through the forging of high-quality multimodal data via GPT-4, this project hath achieved greatness, achieving a remarkable 92.53% in the ScienceQA ordeal. Beyond such feats, it hath been honed for the art of visual conversation and reasoning, especially within the realm of science. Thus, LLaVA stands as the herald of a new age of multimodal reality, a tapestry of marvels awaiting our discovery.
How might one summon LLaVA’s Vision Assistant in this very moment, you ask? Behold, to partake of the wondrous visions of LLaVA, one must journey to the digital realm known as llava.hliu.cc and witness the demo that unfolds before thee. ‘Tis the LLaVA-13B-v1 model that graces thee with its presence at this juncture. As the ancient sages have advised, place an image in the upper-left corner and invoke the power of “Crop.” Be it known that square images yield the finest outcome. And lo, add thy query at the base and strike “Submit.” The LLM shall then peruse the image and expound upon its secrets in great detail. Should thy curiosity be unquenchable, thou mayest pose follow-up queries concerning the image thou hast uploaded.
Aye, let us now delve into the realms of LLaVA’s visual prowess, To test its acumen, we presented it with various trials. We set forth a painting for tribute, asking LLaVA to discern its essence, and lo, it responded with clarity. Further inquiries were met with success as well. In another trial, we laid before it an image of victuals, inquiring of the morning repast and the total count of calories. Each item was identified with precision, accompanied by culinary insights and a crude estimate of caloric bounty. Though the recipes lacked depth, the multimodal LLM did provide notions on incorporating the triad of fare into a dish or repast. Alas, when faced with a handwritten note urging the scribing of a Python script for the Bubble sort algorithm, it faltered in recognizing the written word. Neither could it execute the code. Yet in matters of arithmetic, its answers went astray. Thus, in unraveling mathematical enigmas, whether scribed by hand or pen, the LLaVA model met with woeful misfortune.
Heed the warning, for LLaVA craves improvement on a grand scale. ‘Tis but the dawn, the infancy of open-source conjurations that seek to rival the mighty multimodal LLMs. In the absence of a robust foundation in the realm of language and vision, the open-source realm may find itself trailing behind its proprietary kin. Meta, in its wisdom, hath bestowed many an open-source creation, yet visual models for the public’s perusal remain scant, save for the enigma known as “Segment Anything,” an artifact ill-suited for this endeavor. And while Google hath unfurled PaLM-E, an embodiment of multimodal linguistic prowess, and OpenAI hath unveiled the kaleidoscopic domain of GPT-4’s multimodal potential, we find ourselves gazing upon LLaVA, a jewel in the rough, destined to traverse a path fraught with challenges ere it may stand shoulder to shoulder with OpenAI’s grandeur. Alas, we stand upon the precipice of curiosity, yearning to behold the wonders held within GPT-4’s embrace of the visual realm.
Support our work ❤️
If you enjoyed this article, consider leaving a tip to help us keep publishing great content.
