We Have No Moat:
The Open Source AI Revolution

Google and OpenAI are losing ground to open source artificial intelligence

It’s no secret that Google and OpenAI have been in a heated competition to advance artificial intelligence. However, recent developments reveal a new contender outpacing both companies: the open source community.

The open source community has made significant strides in AI capabilities. From running large language models on phones to finetuning personalized AIs on laptops, their achievements are nothing short of remarkable. The quality gap between open-source models and those developed by Google and OpenAI is shrinking rapidly.

In March, Meta’s LLaMA (a foundation model) was leaked to the public, sparking a wave of innovation within the open-source community. New variants, including multimodal models, have emerged in just weeks, and the barrier to entry for training and experimentation has dropped dramatically.

The democratization of AI, fueled by low-cost, efficient models, has ignited a new era of innovation, forever transforming the landscape of artificial intelligence and unlocking the potential for countless real-world applications.

The open source community’s success also raises questions about OpenAI’s closed policy. With the rapid advancement of open source AI, OpenAI may need to reevaluate its approach and adopt a more collaborative stance.

In conclusion, the open source AI revolution is transforming the landscape of artificial intelligence. As Google and OpenAI face increasing competition from this flourishing community, the key to success may lie in embracing open-source innovation and becoming leaders in this new era of AI development.

What is a LoRA model?

A Low Rank Adaptation (LoRA) model is a technique used in fine-tuning large-scale pre-trained language models. LoRA is based on the idea of approximating large weight matrices with low-rank matrices. This approach helps in reducing the number of parameters required for fine-tuning, which in turn reduces computational resources, time, and cost.

Benefits of LoRA:

  1. Reduced computational complexity: By using low-rank matrices, LoRA reduces the number of trainable parameters, leading to faster fine-tuning and lower memory requirements.
  2. Cost-efficiency: With fewer parameters to fine-tune, LoRA models can be trained with a lower budget, making it accessible to researchers and developers with limited resources.
  3. Enhanced adaptability: LoRA models can be easily adapted to new tasks or domains, making them more flexible and versatile.

Example: In the provided timeline, Eric Wang’s alpaca-lora repository is an example of using the Low Rank Adaptation technique. By applying LoRA to the LLaMA model, the fine-tuning process became more efficient, enabling training “within hours on a single RTX 4090.” This allowed researchers and developers to quickly adapt LLaMA to various tasks and share the low-rank updates independently from the original weights, making them free from the original license restrictions imposed by Meta.

Let’s talk!

If our project resonates with you and you see potential for a collaboration, we would 💙 to hear from you.

get in touch

Timeline

Feb 24, 2023 – LLaMA is Launched Meta launches LLaMA, open sourcing the code but not the weights. LLaMA is a small model with parameters ranging from 7B to 65B, not yet instruction or conversation tuned.

March 3, 2023 – The Inevitable Happens LLaMA is leaked to the public, and despite commercial use restrictions, the AI community starts experimenting and innovating rapidly.

March 12, 2023 – Language models on a Toaster Artem Andreenko gets LLaMA working on a Raspberry Pi, although it runs slowly due to memory constraints, prompting minification efforts.

March 13, 2023 – Fine Tuning on a Laptop Stanford releases Alpaca, adding instruction tuning to LLaMA. Eric Wang’s alpaca-lora repo enables low-budget fine-tuning projects using low rank updates, independent of Meta’s original license.

March 18, 2023 – Now It’s Fast Georgi Gerganov uses 4-bit quantization to run LLaMA on a MacBook CPU, creating the first practical “no GPU” solution.

March 19, 2023 – A 13B model achieves “parity” with Bard A cross-university collaboration releases Vicuna, using GPT-4-powered eval for qualitative comparisons. They leverage ChatGPT data by sampling impressive dialogues, bypassing API restrictions.

March 25, 2023 – Choose Your Own Model Nomic creates GPT4All, an ecosystem that gathers various models (including Vicuna) in one place, with a training cost of $100.

March 28, 2023 – Open Source GPT-3 Cerebras trains the GPT-3 architecture using optimal compute schedules and scaling, outperforming existing GPT-3 clones and reducing dependency on LLaMA.

March 28, 2023 – Multimodal Training in One Hour LLaMA-Adapter uses Parameter Efficient Fine Tuning (PEFT) to introduce instruction tuning and multimodality with only 1.2M learnable parameters, achieving a new SOTA on multimodal ScienceQA.

April 3, 2023 – Real Humans Can’t Tell the Difference Between a 13B Open Model and ChatGPT Berkeley launches Koala, a dialogue model that often performs comparably to ChatGPT in human preference tests, with a training cost of $100.

April 15, 2023 – Open Source RLHF at ChatGPT Levels Open Assistant releases a model and dataset for Alignment via RLHF, achieving near-ChatGPT human preference scores. The dataset’s public availability makes RLHF more accessible for small experimenters.

Trying to predict the future

We asked ChatGPT4 to extend this timeline on May 8th 2023. Here’s what it came up with.

 

May 22, 2023 – Multilingual Models Emerge
Researchers begin to focus on creating multilingual versions of LLaMA and its derivatives, further broadening their applicability and impact on the global AI community.

June 10, 2023 – Improved Safety Measures
As language models become more powerful and accessible, new safety measures and guidelines are introduced to mitigate potential misuse and harmful outputs from the AI systems.

July 15, 2023 – Democratization of AI Continues
The rise of low-cost, efficient models leads to an increase in AI startups, research projects, and AI-powered applications, resulting in a more diverse ecosystem and fostering innovation in various domains.

August 29, 2023 – Real-Time Applications Flourish
Advancements in model efficiency enable real-time applications such as instant translation, voice assistants, and content generation to thrive, as they now require fewer resources and less latency.

October 6, 2023 – Collaborative AI Platforms
Major AI companies and organizations launch collaborative platforms to share research, models, and resources, further accelerating the growth and development of the AI field.

November 18, 2023 – AI in Education
AI-powered tools, such as adaptive learning platforms and personalized tutoring systems, become more widespread and accessible, revolutionizing the way students learn and teachers teach.

Please note that this is a speculative extension of the timeline and cannot guarantee the accuracy of these predictions.

ChatGPT4 definitely paints a future of collaboration between tech firms. Let’s hope it’s right.

Keep reading

;