minimind最小开源大模型，可以自己训练自己的大模型
yzj54643.59MB需要积分:1资源文件列表:
minimind-master.zip 大约有261个文件 
minimind-master/
minimind-master/0-eval_pretrain.py 5.72KB - minimind-master/1-pretrain.py 6.17KB
- minimind-master/2-eval.py 6.18KB
- minimind-master/3-full_sft.py 6.95KB
- minimind-master/4-lora_sft.py 5.17KB
- minimind-master/5-dpo_train.py 2.21KB
- minimind-master/CODE_OF_CONDUCT.md 5.08KB
- minimind-master/LICENSE 11.09KB
- minimind-master/README.md 48.78KB
- minimind-master/README_en.md 50.21KB
- minimind-master/ceval/
- minimind-master/ceval/ceval-exam/
- minimind-master/ceval/ceval-exam/dev/
- minimind-master/ceval/ceval-exam/dev/accountant_dev.csv 3.26KB
- minimind-master/ceval/ceval-exam/dev/advanced_mathematics_dev.csv 6.79KB
- minimind-master/ceval/ceval-exam/dev/art_studies_dev.csv 1.33KB
- minimind-master/ceval/ceval-exam/dev/basic_medicine_dev.csv 1.71KB
- minimind-master/ceval/ceval-exam/dev/business_administration_dev.csv 3.02KB
- minimind-master/ceval/ceval-exam/dev/chinese_language_and_literature_dev.csv 1.78KB
- minimind-master/ceval/ceval-exam/dev/civil_servant_dev.csv 4.4KB
- minimind-master/ceval/ceval-exam/dev/clinical_medicine_dev.csv 1.83KB
- minimind-master/ceval/ceval-exam/dev/college_chemistry_dev.csv 3.46KB
- minimind-master/ceval/ceval-exam/dev/college_economics_dev.csv 3.52KB
- minimind-master/ceval/ceval-exam/dev/college_physics_dev.csv 3.67KB
- minimind-master/ceval/ceval-exam/dev/college_programming_dev.csv 2.75KB
- minimind-master/ceval/ceval-exam/dev/computer_architecture_dev.csv 2.66KB
- minimind-master/ceval/ceval-exam/dev/computer_network_dev.csv 2.23KB
- minimind-master/ceval/ceval-exam/dev/discrete_mathematics_dev.csv 1.89KB
- minimind-master/ceval/ceval-exam/dev/education_science_dev.csv 2.95KB
- minimind-master/ceval/ceval-exam/dev/electrical_engineer_dev.csv 2.05KB
- minimind-master/ceval/ceval-exam/dev/environmental_impact_assessment_engineer_dev.csv 2.36KB
- minimind-master/ceval/ceval-exam/dev/fire_engineer_dev.csv 2.08KB
- minimind-master/ceval/ceval-exam/dev/high_school_biology_dev.csv 2.04KB
- minimind-master/ceval/ceval-exam/dev/high_school_chemistry_dev.csv 2.45KB
- minimind-master/ceval/ceval-exam/dev/high_school_chinese_dev.csv 5.1KB
- minimind-master/ceval/ceval-exam/dev/high_school_geography_dev.csv 1.97KB
- minimind-master/ceval/ceval-exam/dev/high_school_history_dev.csv 2.3KB
- minimind-master/ceval/ceval-exam/dev/high_school_mathematics_dev.csv 3.41KB
- minimind-master/ceval/ceval-exam/dev/high_school_physics_dev.csv 2.14KB
- minimind-master/ceval/ceval-exam/dev/high_school_politics_dev.csv 4.56KB
- minimind-master/ceval/ceval-exam/dev/ideological_and_moral_cultivation_dev.csv 1.19KB
- minimind-master/ceval/ceval-exam/dev/law_dev.csv 3.98KB
- minimind-master/ceval/ceval-exam/dev/legal_professional_dev.csv 6.75KB
- minimind-master/ceval/ceval-exam/dev/logic_dev.csv 5.45KB
- minimind-master/ceval/ceval-exam/dev/mao_zedong_thought_dev.csv 3.2KB
- minimind-master/ceval/ceval-exam/dev/marxism_dev.csv 2.02KB
- minimind-master/ceval/ceval-exam/dev/metrology_engineer_dev.csv 2.35KB
- minimind-master/ceval/ceval-exam/dev/middle_school_biology_dev.csv 4.16KB
- minimind-master/ceval/ceval-exam/dev/middle_school_chemistry_dev.csv 3.71KB
- minimind-master/ceval/ceval-exam/dev/middle_school_geography_dev.csv 2.03KB
- minimind-master/ceval/ceval-exam/dev/middle_school_history_dev.csv 1.89KB
- minimind-master/ceval/ceval-exam/dev/middle_school_mathematics_dev.csv 3.05KB
- minimind-master/ceval/ceval-exam/dev/middle_school_physics_dev.csv 3.38KB
- minimind-master/ceval/ceval-exam/dev/middle_school_politics_dev.csv 3.54KB
- minimind-master/ceval/ceval-exam/dev/modern_chinese_history_dev.csv 2.84KB
- minimind-master/ceval/ceval-exam/dev/operating_system_dev.csv 2.42KB
- minimind-master/ceval/ceval-exam/dev/physician_dev.csv 1.91KB
- minimind-master/ceval/ceval-exam/dev/plant_protection_dev.csv 3.57KB
- minimind-master/ceval/ceval-exam/dev/probability_and_statistics_dev.csv 6.55KB
- minimind-master/ceval/ceval-exam/dev/professional_tour_guide_dev.csv 1.65KB
- minimind-master/ceval/ceval-exam/dev/sports_science_dev.csv 4.02KB
- minimind-master/ceval/ceval-exam/dev/tax_accountant_dev.csv 4.11KB
- minimind-master/ceval/ceval-exam/dev/teacher_qualification_dev.csv 3.07KB
- minimind-master/ceval/ceval-exam/dev/urban_and_rural_planner_dev.csv 3.02KB
- minimind-master/ceval/ceval-exam/dev/veterinary_medicine_dev.csv 2.24KB
- minimind-master/ceval/ceval-exam/test/
- minimind-master/ceval/ceval-exam/test/accountant_test.csv 162.73KB
- minimind-master/ceval/ceval-exam/test/advanced_mathematics_test.csv 45.05KB
- minimind-master/ceval/ceval-exam/test/art_studies_test.csv 33.48KB
- minimind-master/ceval/ceval-exam/test/basic_medicine_test.csv 24.13KB
- minimind-master/ceval/ceval-exam/test/business_administration_test.csv 69.71KB
- minimind-master/ceval/ceval-exam/test/chinese_language_and_literature_test.csv 26.79KB
- minimind-master/ceval/ceval-exam/test/civil_servant_test.csv 167.53KB
- minimind-master/ceval/ceval-exam/test/clinical_medicine_test.csv 36.59KB
- minimind-master/ceval/ceval-exam/test/college_chemistry_test.csv 39.81KB
- minimind-master/ceval/ceval-exam/test/college_economics_test.csv 105.69KB
- minimind-master/ceval/ceval-exam/test/college_physics_test.csv 50.42KB
- minimind-master/ceval/ceval-exam/test/college_programming_test.csv 74.42KB
- minimind-master/ceval/ceval-exam/test/computer_architecture_test.csv 35.24KB
- minimind-master/ceval/ceval-exam/test/computer_network_test.csv 30.67KB
- minimind-master/ceval/ceval-exam/test/discrete_mathematics_test.csv 31.85KB
- minimind-master/ceval/ceval-exam/test/education_science_test.csv 48.29KB
- minimind-master/ceval/ceval-exam/test/electrical_engineer_test.csv 64.3KB
- minimind-master/ceval/ceval-exam/test/environmental_impact_assessment_engineer_test.csv 76.3KB
- minimind-master/ceval/ceval-exam/test/fire_engineer_test.csv 75.23KB
- minimind-master/ceval/ceval-exam/test/high_school_biology_test.csv 49.98KB
- minimind-master/ceval/ceval-exam/test/high_school_chemistry_test.csv 41.91KB
- minimind-master/ceval/ceval-exam/test/high_school_chinese_test.csv 103.67KB
- minimind-master/ceval/ceval-exam/test/high_school_geography_test.csv 36.2KB
- minimind-master/ceval/ceval-exam/test/high_school_history_test.csv 50.77KB
- minimind-master/ceval/ceval-exam/test/high_school_mathematics_test.csv 36.69KB
- minimind-master/ceval/ceval-exam/test/high_school_physics_test.csv 56.34KB
- minimind-master/ceval/ceval-exam/test/high_school_politics_test.csv 77.54KB
- minimind-master/ceval/ceval-exam/test/ideological_and_moral_cultivation_test.csv 30.55KB
- minimind-master/ceval/ceval-exam/test/law_test.csv 72.86KB
- minimind-master/ceval/ceval-exam/test/legal_professional_test.csv 114.43KB
- minimind-master/ceval/ceval-exam/test/logic_test.csv 136.26KB
- minimind-master/ceval/ceval-exam/test/mao_zedong_thought_test.csv 50.38KB
- minimind-master/ceval/ceval-exam/test/marxism_test.csv 33.65KB
- minimind-master/ceval/ceval-exam/test/metrology_engineer_test.csv 41.37KB
- minimind-master/ceval/ceval-exam/test/middle_school_biology_test.csv 41.76KB
- minimind-master/ceval/ceval-exam/test/middle_school_chemistry_test.csv 42.22KB
- minimind-master/ceval/ceval-exam/test/middle_school_geography_test.csv 20.28KB
- minimind-master/ceval/ceval-exam/test/middle_school_history_test.csv 41.23KB
- minimind-master/ceval/ceval-exam/test/middle_school_mathematics_test.csv 28.41KB
- minimind-master/ceval/ceval-exam/test/middle_school_physics_test.csv 43.57KB
- minimind-master/ceval/ceval-exam/test/middle_school_politics_test.csv 66.53KB
- minimind-master/ceval/ceval-exam/test/modern_chinese_history_test.csv 45.2KB
- minimind-master/ceval/ceval-exam/test/operating_system_test.csv 26.32KB
- minimind-master/ceval/ceval-exam/test/physician_test.csv 77.67KB
- minimind-master/ceval/ceval-exam/test/plant_protection_test.csv 26.57KB
- minimind-master/ceval/ceval-exam/test/probability_and_statistics_test.csv 52.17KB
- minimind-master/ceval/ceval-exam/test/professional_tour_guide_test.csv 34.2KB
- minimind-master/ceval/ceval-exam/test/sports_science_test.csv 27.63KB
- minimind-master/ceval/ceval-exam/test/tax_accountant_test.csv 160.36KB
- minimind-master/ceval/ceval-exam/test/teacher_qualification_test.csv 95.84KB
- minimind-master/ceval/ceval-exam/test/urban_and_rural_planner_test.csv 98.31KB
- minimind-master/ceval/ceval-exam/test/veterinary_medicine_test.csv 33.79KB
- minimind-master/ceval/ceval-exam/val/
- minimind-master/ceval/ceval-exam/val/accountant_val.csv 18.01KB
- minimind-master/ceval/ceval-exam/val/advanced_mathematics_val.csv 4.83KB
- minimind-master/ceval/ceval-exam/val/art_studies_val.csv 3.75KB
- minimind-master/ceval/ceval-exam/val/basic_medicine_val.csv 2.16KB
- minimind-master/ceval/ceval-exam/val/business_administration_val.csv 8.28KB
- minimind-master/ceval/ceval-exam/val/chinese_language_and_literature_val.csv 2.87KB
- minimind-master/ceval/ceval-exam/val/civil_servant_val.csv 19.74KB
- minimind-master/ceval/ceval-exam/val/clinical_medicine_val.csv 3.59KB
- minimind-master/ceval/ceval-exam/val/college_chemistry_val.csv 3.83KB
- minimind-master/ceval/ceval-exam/val/college_economics_val.csv 12.91KB
- minimind-master/ceval/ceval-exam/val/college_physics_val.csv 5.6KB
- minimind-master/ceval/ceval-exam/val/college_programming_val.csv 8.58KB
- minimind-master/ceval/ceval-exam/val/computer_architecture_val.csv 3.6KB
- minimind-master/ceval/ceval-exam/val/computer_network_val.csv 3.3KB
- minimind-master/ceval/ceval-exam/val/discrete_mathematics_val.csv 3.01KB
- minimind-master/ceval/ceval-exam/val/education_science_val.csv 4.75KB
- minimind-master/ceval/ceval-exam/val/electrical_engineer_val.csv 7.31KB
- minimind-master/ceval/ceval-exam/val/environmental_impact_assessment_engineer_val.csv 8.29KB
- minimind-master/ceval/ceval-exam/val/fire_engineer_val.csv 9.08KB
- minimind-master/ceval/ceval-exam/val/high_school_biology_val.csv 5.56KB
- minimind-master/ceval/ceval-exam/val/high_school_chemistry_val.csv 5.08KB
- minimind-master/ceval/ceval-exam/val/high_school_chinese_val.csv 9.83KB
- minimind-master/ceval/ceval-exam/val/high_school_geography_val.csv 3.49KB
- minimind-master/ceval/ceval-exam/val/high_school_history_val.csv 6.04KB
- minimind-master/ceval/ceval-exam/val/high_school_mathematics_val.csv 4.68KB
- minimind-master/ceval/ceval-exam/val/high_school_physics_val.csv 6.69KB
- minimind-master/ceval/ceval-exam/val/high_school_politics_val.csv 8.31KB
- minimind-master/ceval/ceval-exam/val/ideological_and_moral_cultivation_val.csv 2.76KB
- minimind-master/ceval/ceval-exam/val/law_val.csv 7.41KB
- minimind-master/ceval/ceval-exam/val/legal_professional_val.csv 11.43KB
- minimind-master/ceval/ceval-exam/val/logic_val.csv 14.72KB
- minimind-master/ceval/ceval-exam/val/mao_zedong_thought_val.csv 4.84KB
- minimind-master/ceval/ceval-exam/val/marxism_val.csv 3.74KB
- minimind-master/ceval/ceval-exam/val/metrology_engineer_val.csv 5.45KB
- minimind-master/ceval/ceval-exam/val/middle_school_biology_val.csv 4.68KB
- minimind-master/ceval/ceval-exam/val/middle_school_chemistry_val.csv 5.09KB
- minimind-master/ceval/ceval-exam/val/middle_school_geography_val.csv 2.33KB
- minimind-master/ceval/ceval-exam/val/middle_school_history_val.csv 5.37KB
- minimind-master/ceval/ceval-exam/val/middle_school_mathematics_val.csv 4.4KB
- minimind-master/ceval/ceval-exam/val/middle_school_physics_val.csv 4.75KB
- minimind-master/ceval/ceval-exam/val/middle_school_politics_val.csv 6.69KB
- minimind-master/ceval/ceval-exam/val/modern_chinese_history_val.csv 4.57KB
- minimind-master/ceval/ceval-exam/val/operating_system_val.csv 2.81KB
- minimind-master/ceval/ceval-exam/val/physician_val.csv 7.42KB
- minimind-master/ceval/ceval-exam/val/plant_protection_val.csv 3.07KB
- minimind-master/ceval/ceval-exam/val/probability_and_statistics_val.csv 5.31KB
- minimind-master/ceval/ceval-exam/val/professional_tour_guide_val.csv 3.77KB
- minimind-master/ceval/ceval-exam/val/sports_science_val.csv 3KB
- minimind-master/ceval/ceval-exam/val/tax_accountant_val.csv 17.4KB
- minimind-master/ceval/ceval-exam/val/teacher_qualification_val.csv 10.97KB
- minimind-master/ceval/ceval-exam/val/urban_and_rural_planner_val.csv 11.48KB
- minimind-master/ceval/ceval-exam/val/veterinary_medicine_val.csv 3.97KB
- minimind-master/ceval/ceval_result/
- minimind-master/ceval/ceval_result/accountant_val_result.csv 18.14KB
- minimind-master/ceval/ceval_result/advanced_mathematics_val_result.csv 4.94KB
- minimind-master/ceval/ceval_result/art_studies_val_result.csv 3.87KB
- minimind-master/ceval/ceval_result/basic_medicine_val_result.csv 2.26KB
- minimind-master/ceval/ceval_result/business_administration_val_result.csv 8.42KB
- minimind-master/ceval/ceval_result/chinese_language_and_literature_val_result.csv 2.99KB
- minimind-master/ceval/ceval_result/civil_servant_val_result.csv 19.87KB
- minimind-master/ceval/ceval_result/clinical_medicine_val_result.csv 3.71KB
- minimind-master/ceval/ceval_result/college_chemistry_val_result.csv 3.95KB
- minimind-master/ceval/ceval_result/college_economics_val_result.csv 13.05KB
- minimind-master/ceval/ceval_result/college_physics_val_result.csv 5.71KB
- minimind-master/ceval/ceval_result/college_programming_val_result.csv 8.71KB
- minimind-master/ceval/ceval_result/computer_architecture_val_result.csv 3.71KB
- minimind-master/ceval/ceval_result/computer_network_val_result.csv 3.41KB
- minimind-master/ceval/ceval_result/discrete_mathematics_val_result.csv 3.12KB
- minimind-master/ceval/ceval_result/education_science_val_result.csv 4.88KB
- minimind-master/ceval/ceval_result/electrical_engineer_val_result.csv 7.44KB
- minimind-master/ceval/ceval_result/environmental_impact_assessment_engineer_val_result.csv 8.44KB
- minimind-master/ceval/ceval_result/fire_engineer_val_result.csv 9.2KB
- minimind-master/ceval/ceval_result/high_school_biology_val_result.csv 5.67KB
- minimind-master/ceval/ceval_result/high_school_chemistry_val_result.csv 5.2KB
- minimind-master/ceval/ceval_result/high_school_chinese_val_result.csv 9.94KB
- minimind-master/ceval/ceval_result/high_school_geography_val_result.csv 3.6KB
- minimind-master/ceval/ceval_result/high_school_history_val_result.csv 6.15KB
- minimind-master/ceval/ceval_result/high_school_mathematics_val_result.csv 4.8KB
- minimind-master/ceval/ceval_result/high_school_physics_val_result.csv 6.8KB
- minimind-master/ceval/ceval_result/high_school_politics_val_result.csv 8.43KB
- minimind-master/ceval/ceval_result/ideological_and_moral_cultivation_val_result.csv 2.88KB
- minimind-master/ceval/ceval_result/law_val_result.csv 7.51KB
- minimind-master/ceval/ceval_result/legal_professional_val_result.csv 11.54KB
- minimind-master/ceval/ceval_result/logic_val_result.csv 14.83KB
- minimind-master/ceval/ceval_result/mao_zedong_thought_val_result.csv 4.95KB
- minimind-master/ceval/ceval_result/marxism_val_result.csv 3.84KB
- minimind-master/ceval/ceval_result/metrology_engineer_val_result.csv 5.57KB
- minimind-master/ceval/ceval_result/middle_school_biology_val_result.csv 4.8KB
- minimind-master/ceval/ceval_result/middle_school_chemistry_val_result.csv 5.21KB
- minimind-master/ceval/ceval_result/middle_school_geography_val_result.csv 2.44KB
- minimind-master/ceval/ceval_result/middle_school_history_val_result.csv 5.49KB
- minimind-master/ceval/ceval_result/middle_school_mathematics_val_result.csv 4.52KB
- minimind-master/ceval/ceval_result/middle_school_physics_val_result.csv 4.86KB
- minimind-master/ceval/ceval_result/middle_school_politics_val_result.csv 6.8KB
- minimind-master/ceval/ceval_result/modern_chinese_history_val_result.csv 4.69KB
- minimind-master/ceval/ceval_result/operating_system_val_result.csv 2.92KB
- minimind-master/ceval/ceval_result/physician_val_result.csv 7.55KB
- minimind-master/ceval/ceval_result/plant_protection_val_result.csv 3.18KB
- minimind-master/ceval/ceval_result/probability_and_statistics_val_result.csv 5.43KB
- minimind-master/ceval/ceval_result/professional_tour_guide_val_result.csv 3.9KB
- minimind-master/ceval/ceval_result/sports_science_val_result.csv 3.11KB
- minimind-master/ceval/ceval_result/tax_accountant_val_result.csv 17.54KB
- minimind-master/ceval/ceval_result/teacher_qualification_val_result.csv 11.11KB
- minimind-master/ceval/ceval_result/test.log 4.34KB
- minimind-master/ceval/ceval_result/urban_and_rural_planner_val_result.csv 11.62KB
- minimind-master/ceval/ceval_result/veterinary_medicine_val_result.csv 4.09KB
- minimind-master/chat_openai_api.py 1.42KB
- minimind-master/data_process.py 6.62KB
- minimind-master/eval_ceval.py 6.87KB
- minimind-master/export_model.py 2.07KB
- minimind-master/fast_infenence.py 4.2KB
- minimind-master/images/
- minimind-master/images/1-wiki.png 136.21KB
- minimind-master/images/2-eval.png 109.16KB
- minimind-master/images/2-wiki.png 72.93KB
- minimind-master/images/3-wiki.png 229.81KB
- minimind-master/images/4-wiki.png 104.37KB
- minimind-master/images/5-wiki.png 239.4KB
- minimind-master/images/LLM-structure-moe.png 153.43KB
- minimind-master/images/LLM-structure.png 133.48KB
- minimind-master/images/fastgpt.png 72.41KB
- minimind-master/images/gpt3_config.png 65.63KB
- minimind-master/images/logger.png 81.37KB
- minimind-master/images/logo.png 402KB
- minimind-master/images/streamlit.png 38.81KB
- minimind-master/model/
- minimind-master/model/LMConfig.py 2.25KB
- minimind-master/model/__pycache__/
- minimind-master/model/__pycache__/LMConfig.cpython-310.pyc 1.49KB
- minimind-master/model/__pycache__/dataset.cpython-310.pyc 3.58KB
- minimind-master/model/__pycache__/model.cpython-310.pyc 15.57KB
- minimind-master/model/dataset.py 4.23KB
- minimind-master/model/minimind_tokenizer/
- minimind-master/model/minimind_tokenizer/merges.txt 56.41KB
- minimind-master/model/minimind_tokenizer/tokenizer.json 255.29KB
- minimind-master/model/minimind_tokenizer/tokenizer_config.json 1.57KB
- minimind-master/model/minimind_tokenizer/vocab.json 93.75KB
- minimind-master/model/model.py 16.53KB
- minimind-master/my_openai_api.py 15.03KB
- minimind-master/requirements.txt 425B
- minimind-master/train_tokenizer.py 5.26KB
资源介绍:
minimind最小开源大模型，可以自己训练自己的大模型
[](https://github.com/jingyaogong/minimind/stargazers)
[](LICENSE)
[](https://github.com/jingyaogong/minimind/commits/master)
[](https://github.com/jingyaogong/minimind/pulls)
[](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
"大道至简"
中文 | [English](./README_en.md)
* 本开源项目旨在完全从0开始，最快仅用3小时！即可训练出仅为26M大小的微型语言模型**MiniMind**。
* **MiniMind**极其轻量，体积约是 GPT3 的 $\frac{1}{7000}$，力求做到最普通的个人GPU也可快速推理甚至训练。
* **MiniMind**改进自DeepSeek-V2、Llama3结构，项目包含整个数据处理、pretrain、sft、dpo的全部阶段，包含混合专家(MoE)模型。
* 这是一个既是开源项目，又是入门LLM教程，同时也是一个初具雏形的开源模型，希望能起到抛砖引玉的作用。
---
https://github.com/user-attachments/assets/88b98128-636e-43bc-a419-b1b1403c2055
[Bilibili视频链接](https://www.bilibili.com/video/BV12dHPeqE72/?share_source=copy_web&vd_source=670c2504f88726f8cf4a21ef6147c0e8)
# 📌 Introduction
大语言模型（LLM）领域，如 GPT、LLaMA、GLM 等，虽然它们效果惊艳，
但动辄10 Bilion庞大的模型参数个人设备显存远不够训练，甚至推理困难。
几乎所有人都不会只满足于用Lora等方案fine-tuing大模型学会一些新的指令，
这约等于在教牛顿玩21世纪的智能手机，然而，这远远脱离了学习物理本身的奥妙。
此外，卖课付费订阅的营销号漏洞百出的一知半解讲解AI的教程遍地，
让理解LLM的优质内容雪上加霜，严重阻碍了学习者。
因此，本项目的目标是把上手LLM的门槛无限降低，
直接从0开始训练一个极其轻量的语言模型。
> [!TIP]
> （截至2024-9-17）minimind训练了3个型号模型，最小仅需26M（0.02B），即可具备流畅的对话能力！
| 模型 (大小) | tokenizer长度 | 推理占用 | release | 主观评分（/100） |
|-------------------------|-------------|--------|------------|------------|
| minimind-v1-small (26M) | 6400 | 0.5 GB | 2024.08.28 | 50' |
| minimind-v1-moe (4×26M) | 6400 | 1.0 GB | 2024.09.17 | 55' |
| minimind-v1 (108M) | 6400 | 1.0 GB | 2024.09.01 | 60' |
> 该分析在一个带有Torch 2.1.2、CUDA 12.2和Flash Attention 2的RTX 3090 GPU上运行。
项目包含：
- 公开MiniMind模型代码（包含Dense和MoE模型）、Pretrain、SFT指令微调、LoRA微调、DPO偏好优化的全过程代码、数据集和来源。
- 兼容`transformers`、`accelerate`、`trl`、`peft`等流行框架。
- 训练支持单机单卡、单机多卡(DDP、DeepSpeed)训练。训练过程中支持在任意位置停止，及在任意位置继续训练。
- 在Ceval数据集上进行模型测试的代码。
- 实现Openai-Api基本的chat接口，便于集成到第三方ChatUI使用（FastGPT、Open-WebUI等）。
希望此开源项目可以帮助LLM初学者快速入门！
### 👉**最近更新**
2024-09-17 (new🎉)
- 更新minimind-v1-moe模型 - 为了防止歧义，不再使用mistral_tokenizer分词，全部采用自定义的minimind_tokenizer作为分词器。2024-09-01
- 更新minimind-v1 (108M)模型，采用minimind_tokenizer，预训练轮次3 + SFT轮次10，更充分训练，性能更强。 - 项目已部署至ModelScope创空间，可以在此网站上体验： - [ModelScope在线体验](https://www.modelscope.cn/studios/gongjy/minimind)2024-08-27
- 项目首次开源
Hugging Face
[MiniMind (HuggingFace)](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5)
[MiniMind (ModelScope)](https://www.modelscope.cn/models/gongjy/minimind-v1)
项目已部署至ModelScope创空间，可以在此网站上体验：
[ModelScope在线体验](https://www.modelscope.cn/studios/gongjy/minimind)
# 📌 Quick Start
* 0、环境安装
```bash
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
```
* 1、克隆项目代码
```text
git clone https://github.com/jingyaogong/minimind.git
```
* 2、如果你需要自己训练
* 2.1 下载[数据集下载地址](#数据集下载地址)放到`./dataset`目录下
* 2.2 `python data_process.py`处理数据集，例如pretrain数据提前进行token-encoder、sft数据集抽离qa到csv文件
* 2.3 在`./model/LMConfig.py` 中调整model的参数配置
* 2.4 `python 1-pretrain.py` 执行预训练
* 2.5 `python 3-full_sft.py` 执行指令微调
* 2.6 `python 4-lora_sft.py` 执行lora微调（非必须）
* 2.7 `python 5-dpo_train.py` 执行DPO人类偏好强化学习对齐（非必须）
* 3、测试模型推理效果
* 确保需要使用的，训练完成的参数权重位于`./out/`目录下
* 也可以直接去[训练完成的模型权重](#训练完成的模型权重)下载使用我训练好的
```text
out
├── multi_chat
│ ├── full_sft_512.pth
│ ├── full_sft_512_moe.pth
│ └── full_sft_768.pth
├── single_chat
│ ├── full_sft_512.pth
│ ├── full_sft_512_moe.pth
│ └── full_sft_768.pth
├── pretrain_768.pth
├── pretrain_512_moe.pth
├── pretrain_512.pth
```
* `python 0-eval_pretrain.py`测试预训练模型的接龙效果
* `python 2-eval.py`测试模型的对话效果
🍭 【Tip】预训练和全参微调pretrain和full_sft均支持多卡加速
* 单机N卡启动训练(DDP)
```bash
torchrun --nproc_per_node N 1-pretrain.py
# and
torchrun --nproc_per_node N 3-full_sft.py
```
* 单机N卡启动训练(DeepSpeed)
```bash
deepspeed --master_port 29500 --num_gpus=N 1-pretrain.py
# and
deepspeed --master_port 29500 --num_gpus=N 3-full_sft.py
```
# 📌 Data sources
- 🤖 分词器：nlp中的Tokenizer类似于词典，将单词从自然语言通过“词典”映射到0,1,36这样的数字，可以理解为数字就代表了单词在“词典� 
2401_89288642
yangyin1998