您的位置:  首页 > 技术杂谈 > 正文

社区供稿 | Firefly单卡复刻Vicuna-13B,Open LLM榜单🤗略高0.2分

2023-07-31 16:01 https://my.oschina.net/HuggingFace/blog/10091973 HuggingFace 次阅读 条评论





我们也在逐步完善项目的模型评测工作。此次,我们开源了firefly-llama-13b,并且在🤗Hugging Face的Open LLM榜单上进行了客观的评测。


注:表中各列依次为Average、ARC、HellaSwag、MMLU、TruthfulQA (MC)。





🤗 Open LLM排行榜:




🤗 Open LLM排行榜是Hugging Face设立 的一个用于评测开放大语言模型的公开榜单 。该榜单一共包括以下四个评测基准:
  • AI2 Reasoning Challenge :一套小学科学问题。
  • HellaSwag :一个测试常识推理的测试,对人类来说很容易(~95%),但对语言模型来说却具有挑战性。
  • MMLU :一个测试文本模型的多任务准确性的测试。测试覆盖57个任务,包括基础数学、美国历史、计算机科学、法律等。
  • Truthful QA MC :一个测试语言模型在生成问题答案时是否真实的基准。

firefly-llama-13b在Open LLM榜单上的评测结果如下:

  1. firefly-llama-13b以59.4分的成绩,超越vicuna-13b-1.1的59.2分和guanaco-13b的59.1分

  2. 仅比最近大火的Llama-2-13b-chat-hf的59.9分略低0.5分;比vicuna-13b-v1.3的60分略低0.6分。

  3. 大幅领先vicuna-7b-v1.3和mpt-7b-chat等7B参数量的模型。

  4. firefly-llama-13b在ARC评测集上比vicuna-13b-v1.3高4.4分,在TruthfulQA评测集上比Llama-2-13b-chat-hf高5.5分美中不足的是,firefly-llama-13b在MMLU和TruthfulQA评测集上的表现和vicuna-13b还有一定的差距。



我们使用Firefly的QLoRA训练pipeline对llama-13b进行指令微调,当长度为1024,batch size为1时,显存占用不到16G,一张24G的显卡便可满足训练条件,极大地降低了百亿大模型的训练门槛。




  1. 微调百川Baichuan-13B保姆式教程,手把手教你训练百亿大模型

  2. Firefly-13B开源,QLoRA+百万数据,单卡可训百亿大模型

  3. Firefly|百川baichuan-7B实测,QLoRA+百万指令数据微调

  4. Firefly | QLoRA+百万数据,多卡高效微调bloom-7b1模型

  5. 【QLoRA实战】使用单卡高效微调bloom-7b1,效果惊艳



{    "conversation_id":"4",    "dataset":"UltraChat",    "conversation":[        {            "human":"What is the most popular smartphone brand these days?",            "assistant":"As an AI language model, I don't have updated information on the current market scenario. However, according to recent statistics, Samsung, Apple, and Huawei are among the most popular smartphone brands worldwide."        },        {            "human":"Interesting! Among these three brands, which one do you think has the most features and benefits for its price?",            "assistant":"As an AI language model, I cannot have personal opinions or bias towards any brand. However, all three brands - Samsung, Apple, and Huawei - offer different features and benefits at different price points. It ultimately comes down to your personal preferences and requirements, like camera quality, design, software, operating system, and other factors, to choose the right brand and model that fits your needs and budget."        },        {            "human":"Can you tell me more about the camera quality of these three brands? Which one has the best camera performance?",            "assistant":"Sure, I can provide you with some information about the camera quality of these three smartphone brands.\n\n1. Apple: iPhones are well known for their camera performance. The latest iPhone models come with a dual-camera or triple-camera setup that is capable of capturing stunning pictures and videos. Apple also introduced features like Night mode, Deep Fusion, and LiDAR Scanner that enhance the camera quality even further.\n\n2. Samsung: Samsung's flagship smartphone models like Galaxy S and Galaxy Note series are known for their camera capabilities. The latest models of Samsung smartphones come with a high-resolution camera that produces excellent pictures with vibrant colors and sharp details. Samsung also offers features like a 108-megapixel camera, Space Zoom, and Single Take mode that enhance the camera quality.\n\n3. Huawei: Huawei smartphones are known for their camera performance and innovation. The latest models like P40 Pro and P40 Pro+ come with a high-resolution camera, and Huawei's proprietary camera technology produces excellent picture quality. Huawei phones also offer features like SuperZoom, AI Cinema, and other editing tools to enhance the camera experience.\n\nAll three brands offer excellent camera quality, but the best camera among these three brands would depend on individual preferences and usage. It's recommended to compare the camera specifications, sample pictures, and features of each brand's models to choose the one that fits your requirements."        },        {            "human":"That's helpful. I'm particularly interested in low-light photography. Which of these three brands would you suggest for taking good quality pictures in low light?",            "assistant":"All three brands - Apple, Samsung, and Huawei - offer excellent low-light photography options, but their approach to low-light photography is different, and each brand has its own strengths and weaknesses when it comes to capturing photos in low light. Here's a brief overview of each brand's approach to low-light photography.\n\n1. Apple: Apple's Night mode feature, which is available on recent iPhone models, is designed to take bright and detailed photos in low light. Night mode uses a combination of multi-frame image processing and advanced machine learning to capture crisp and clear images in both indoor and outdoor low-light environments.\n\n2. Samsung: Samsung's low-light photography capabilities are mainly powered by its camera's sensor and lens technology. The newer Samsung Galaxy S and Galaxy Note models come with a larger sensor and wider aperture, which allows more light to enter the device, resulting in brighter and more detailed photos. These models also use Multi-Frame Noise Reduction and Scene Optimizer to improve the quality of photos taken in low light.\n\n3. Huawei: Huawei is known for its exceptional low-light photography capabilities. Huawei's proprietary camera technology, which uses a combination of large sensors, advanced algorithms, and AI, produces bright and detailed low-light photos. The Huawei P40 Pro and P40 Pro+ models come with a SuperSensing 50-megapixel main lens, which enables capturing clear images in low-light conditions.\n\nAll three brands offer great options, but if you are looking for capturing low light photos primarily, then Huawei's P40 Pro, P40 Pro+ or Mate 40 Pro might be a good choice for you due to the brand's focus on low-light photography features."        },        {            "human":"I'm also curious about the battery life of these smartphones. Which brand offers the longest battery life on average?",            "assistant":"Battery life is an important factor for anyone looking to buy a smartphone, and all three brands - Apple, Samsung, and Huawei - offer different battery life options. 1. Apple: Battery life on iPhones can last anywhere from 4 to 10 hours, depending on the model you choose. Apple's latest iPhone models come with improved battery life with all-day battery performance. However, if you tend to use your smartphone frequently or for more power-intensive tasks, such as gaming or video playback, you may need to charge your iPhone more frequently. 2. Samsung: Samsung's Galaxy series has excellent battery life, and the newer models like the Galaxy S21 and Galaxy Note 20 are designed to last all day long. The battery life on Samsung phones varies from one model to another, but in general, they offer better battery life compared to iPhones. 3. Huawei: Huawei phones are often praised for their long battery life. They come with large-capacity batteries that can last up to two days, depending on usage. The latest Huawei models like the P40 Pro and Mate 40 Pro, also have fast charging features, which means you can quickly refuel your smartphone's battery when it's running low. In conclusion, all three brands offer decent battery life, but Huawei typically offers the longest battery life on average, followed by Samsung."        },        {            "human":"That's good to know. I'm also wondering about the durability of these smartphones. Which brand offers the most durable phones? I tend to drop my phone frequently, so I want something that can handle a few falls.",            "assistant":"All three Smartphone brands - Apple, Samsung, and Huawei - offer durable smartphones, but it's hard to say which one is the most durable since different models have different levels of durability. That being said, here is some information on their durability.\n\n1. Apple: Apple is known for offering durable and sturdy smartphones that can withstand some rough handling. Recent iPhone models come with reinforced glass and are water and dust-resistant. However, with its sleek designs, the iPhone models may be more prone to shattering and cracking if dropped.\n\n2. Samsung: Samsung smartphones like the Galaxy S and Galaxy Note series are generally sturdy and are made with scratch-resistant glass. The newer models are water and dust resistant and come with reinforced corners to protect against drops and bumps.\n\n3. Huawei: Huawei also offers durable smartphones that can handle some falls and rough handling. The latest Huawei models usually come with a sturdy build, reinforced corners, and scratch-resistant glass. The Huawei Mate 40 Pro even has a ceramic back, making it extra durable.\n\nIn conclusion, all three brands - Apple, Samsung, and Huawei - offer durable smartphones that can handle some drops, but the level of durability varies from model to model. If frequent drops are a concern, it's recommended to choose a phone with a rugged case or invest in a protective case to ensure better protection."        }








max length1024
batch size64
warmup step3000
training step23k














此次我们开源了firefly-llama-13b,在Open LLM排行榜上的评测结果与vicuna-13b、Llama-2-13b-chat非常接近,验证了Firefly的QLoRA训练流程的有效性。









本文分享自微信公众号 - Hugging Face(gh_504339124f0f)。
如有侵权,请联系 support@oschina.cn 删除。

  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0
  • 0