Model Optimization: Enhancing Language Models for Efficient Coding Tasks
The University of Hong Kong has unveiled two open-source language models, Lemur and Lemur-Chat, designed to strike a balance between conversational and coding abilities. The project aims to promote further research into multi-purpose agents by open-sourcing these models.
Lemur, a neural network dataset and framework, is designed to support automated machine learning (AutoML), benchmarking, and standardized model evaluation. It offers built-in support for natural language processing, image classification, segmentation, and object detection, and is resource-efficient, making it suitable for deployment on edge devices.
Lemur's pretraining corpus encompasses diverse textual data from Wikipedia, news, webpages, and books, as well as programming content from code repositories like GitHub. This extensive data range allows Lemur to ground itself in technical contexts, a crucial aspect for planning and executing actions in messy real-world environments.
After pretraining, Lemur undergoes instruction tuning to effectively execute directions in free-form written or spoken language, resulting in Lemur-Chat. This conversational agent has demonstrated remarkable performance, excelling in 12 out of 13 agent evaluations, outperforming specialized conversational (LLaMA) and coding (Codex) counterparts.
Lemur-Chat has shown particular prowess in leveraging Python interpreters and Wikipedia to enhance reasoning, and in utilizing error messages to fix and refine code. It has also matched or exceeded the performance of commercial models like GPT-3.5 across most benchmarks.
The integration of Lemur with edge devices and AutoML suggests adaptability to varied deployment scenarios, from cloud to on-device applications. This flexibility sets Lemur-Chat apart from many existing language models, which often prioritise language or coding exclusivity over synergy, hindering their versatility as agents.
While specialized models may outperform Lemur-based ones for outright performance in either domain, Lemur excels in reproducibility, benchmarking, and integration with AutoML pipelines for diverse environments. This makes it an attractive choice for scenarios where a balance between conversational and coding abilities is needed.
As AI systems evolve from conversational bots to fully-fledged agents that can get things done, models like Lemur and Lemur-Chat are poised to play a significant role. They represent a step towards unifying natural and programming language abilities within a single open-source language model, a move that could unlock greater versatility for these agents in the future.
[1] The University of Hong Kong. (n.d.). Lemur: A General-Purpose AutoML Platform for Large Language Models. Retrieved from https://github.com/hku-ai/lemur [2] Li, C., Huang, Y., Liu, Y., Zhang, Y., & Zhang, Y. (2022). Lemur: A General-Purpose AutoML Platform for Large Language Models. arXiv preprint arXiv:2203.05170. [3] Brown, J. L., Ko, D., Lee, K., Luong, M. D., Radford, A., Strubell, E., ... & Zettlemoyer, L. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33786-33805.
- The University of Hong Kong's open-source language model, Lemur, incorporates artificial-intelligence capabilities, showcasing its adaptability to diverse tasks including natural language processing and programming, thereby making it an ideal candidate for further exploration in artificial-intelligence research.
- As technology advances and AI systems evolve, the unification of natural language and programming abilities within a single open-source language model, such as Lemur and Lemur-Chat, may hold significant promise for artificial-intelligence systems, potentially leading to more versatile agents in the future.
