ZIPminimind最小开源大模型，可以自己训练自己的大模型 3.59MB

yzj5464

资源文件列表:

minimind-master.zip 大约有261个文件
  1. minimind-master/
  2. minimind-master/0-eval_pretrain.py 5.72KB
  3. minimind-master/1-pretrain.py 6.17KB
  4. minimind-master/2-eval.py 6.18KB
  5. minimind-master/3-full_sft.py 6.95KB
  6. minimind-master/4-lora_sft.py 5.17KB
  7. minimind-master/5-dpo_train.py 2.21KB
  8. minimind-master/CODE_OF_CONDUCT.md 5.08KB
  9. minimind-master/LICENSE 11.09KB
  10. minimind-master/README.md 48.78KB
  11. minimind-master/README_en.md 50.21KB
  12. minimind-master/ceval/
  13. minimind-master/ceval/ceval-exam/
  14. minimind-master/ceval/ceval-exam/dev/
  15. minimind-master/ceval/ceval-exam/dev/accountant_dev.csv 3.26KB
  16. minimind-master/ceval/ceval-exam/dev/advanced_mathematics_dev.csv 6.79KB
  17. minimind-master/ceval/ceval-exam/dev/art_studies_dev.csv 1.33KB
  18. minimind-master/ceval/ceval-exam/dev/basic_medicine_dev.csv 1.71KB
  19. minimind-master/ceval/ceval-exam/dev/business_administration_dev.csv 3.02KB
  20. minimind-master/ceval/ceval-exam/dev/chinese_language_and_literature_dev.csv 1.78KB
  21. minimind-master/ceval/ceval-exam/dev/civil_servant_dev.csv 4.4KB
  22. minimind-master/ceval/ceval-exam/dev/clinical_medicine_dev.csv 1.83KB
  23. minimind-master/ceval/ceval-exam/dev/college_chemistry_dev.csv 3.46KB
  24. minimind-master/ceval/ceval-exam/dev/college_economics_dev.csv 3.52KB
  25. minimind-master/ceval/ceval-exam/dev/college_physics_dev.csv 3.67KB
  26. minimind-master/ceval/ceval-exam/dev/college_programming_dev.csv 2.75KB
  27. minimind-master/ceval/ceval-exam/dev/computer_architecture_dev.csv 2.66KB
  28. minimind-master/ceval/ceval-exam/dev/computer_network_dev.csv 2.23KB
  29. minimind-master/ceval/ceval-exam/dev/discrete_mathematics_dev.csv 1.89KB
  30. minimind-master/ceval/ceval-exam/dev/education_science_dev.csv 2.95KB
  31. minimind-master/ceval/ceval-exam/dev/electrical_engineer_dev.csv 2.05KB
  32. minimind-master/ceval/ceval-exam/dev/environmental_impact_assessment_engineer_dev.csv 2.36KB
  33. minimind-master/ceval/ceval-exam/dev/fire_engineer_dev.csv 2.08KB
  34. minimind-master/ceval/ceval-exam/dev/high_school_biology_dev.csv 2.04KB
  35. minimind-master/ceval/ceval-exam/dev/high_school_chemistry_dev.csv 2.45KB
  36. minimind-master/ceval/ceval-exam/dev/high_school_chinese_dev.csv 5.1KB
  37. minimind-master/ceval/ceval-exam/dev/high_school_geography_dev.csv 1.97KB
  38. minimind-master/ceval/ceval-exam/dev/high_school_history_dev.csv 2.3KB
  39. minimind-master/ceval/ceval-exam/dev/high_school_mathematics_dev.csv 3.41KB
  40. minimind-master/ceval/ceval-exam/dev/high_school_physics_dev.csv 2.14KB
  41. minimind-master/ceval/ceval-exam/dev/high_school_politics_dev.csv 4.56KB
  42. minimind-master/ceval/ceval-exam/dev/ideological_and_moral_cultivation_dev.csv 1.19KB
  43. minimind-master/ceval/ceval-exam/dev/law_dev.csv 3.98KB
  44. minimind-master/ceval/ceval-exam/dev/legal_professional_dev.csv 6.75KB
  45. minimind-master/ceval/ceval-exam/dev/logic_dev.csv 5.45KB
  46. minimind-master/ceval/ceval-exam/dev/mao_zedong_thought_dev.csv 3.2KB
  47. minimind-master/ceval/ceval-exam/dev/marxism_dev.csv 2.02KB
  48. minimind-master/ceval/ceval-exam/dev/metrology_engineer_dev.csv 2.35KB
  49. minimind-master/ceval/ceval-exam/dev/middle_school_biology_dev.csv 4.16KB
  50. minimind-master/ceval/ceval-exam/dev/middle_school_chemistry_dev.csv 3.71KB
  51. minimind-master/ceval/ceval-exam/dev/middle_school_geography_dev.csv 2.03KB
  52. minimind-master/ceval/ceval-exam/dev/middle_school_history_dev.csv 1.89KB
  53. minimind-master/ceval/ceval-exam/dev/middle_school_mathematics_dev.csv 3.05KB
  54. minimind-master/ceval/ceval-exam/dev/middle_school_physics_dev.csv 3.38KB
  55. minimind-master/ceval/ceval-exam/dev/middle_school_politics_dev.csv 3.54KB
  56. minimind-master/ceval/ceval-exam/dev/modern_chinese_history_dev.csv 2.84KB
  57. minimind-master/ceval/ceval-exam/dev/operating_system_dev.csv 2.42KB
  58. minimind-master/ceval/ceval-exam/dev/physician_dev.csv 1.91KB
  59. minimind-master/ceval/ceval-exam/dev/plant_protection_dev.csv 3.57KB
  60. minimind-master/ceval/ceval-exam/dev/probability_and_statistics_dev.csv 6.55KB
  61. minimind-master/ceval/ceval-exam/dev/professional_tour_guide_dev.csv 1.65KB
  62. minimind-master/ceval/ceval-exam/dev/sports_science_dev.csv 4.02KB
  63. minimind-master/ceval/ceval-exam/dev/tax_accountant_dev.csv 4.11KB
  64. minimind-master/ceval/ceval-exam/dev/teacher_qualification_dev.csv 3.07KB
  65. minimind-master/ceval/ceval-exam/dev/urban_and_rural_planner_dev.csv 3.02KB
  66. minimind-master/ceval/ceval-exam/dev/veterinary_medicine_dev.csv 2.24KB
  67. minimind-master/ceval/ceval-exam/test/
  68. minimind-master/ceval/ceval-exam/test/accountant_test.csv 162.73KB
  69. minimind-master/ceval/ceval-exam/test/advanced_mathematics_test.csv 45.05KB
  70. minimind-master/ceval/ceval-exam/test/art_studies_test.csv 33.48KB
  71. minimind-master/ceval/ceval-exam/test/basic_medicine_test.csv 24.13KB
  72. minimind-master/ceval/ceval-exam/test/business_administration_test.csv 69.71KB
  73. minimind-master/ceval/ceval-exam/test/chinese_language_and_literature_test.csv 26.79KB
  74. minimind-master/ceval/ceval-exam/test/civil_servant_test.csv 167.53KB
  75. minimind-master/ceval/ceval-exam/test/clinical_medicine_test.csv 36.59KB
  76. minimind-master/ceval/ceval-exam/test/college_chemistry_test.csv 39.81KB
  77. minimind-master/ceval/ceval-exam/test/college_economics_test.csv 105.69KB
  78. minimind-master/ceval/ceval-exam/test/college_physics_test.csv 50.42KB
  79. minimind-master/ceval/ceval-exam/test/college_programming_test.csv 74.42KB
  80. minimind-master/ceval/ceval-exam/test/computer_architecture_test.csv 35.24KB
  81. minimind-master/ceval/ceval-exam/test/computer_network_test.csv 30.67KB
  82. minimind-master/ceval/ceval-exam/test/discrete_mathematics_test.csv 31.85KB
  83. minimind-master/ceval/ceval-exam/test/education_science_test.csv 48.29KB
  84. minimind-master/ceval/ceval-exam/test/electrical_engineer_test.csv 64.3KB
  85. minimind-master/ceval/ceval-exam/test/environmental_impact_assessment_engineer_test.csv 76.3KB
  86. minimind-master/ceval/ceval-exam/test/fire_engineer_test.csv 75.23KB
  87. minimind-master/ceval/ceval-exam/test/high_school_biology_test.csv 49.98KB
  88. minimind-master/ceval/ceval-exam/test/high_school_chemistry_test.csv 41.91KB
  89. minimind-master/ceval/ceval-exam/test/high_school_chinese_test.csv 103.67KB
  90. minimind-master/ceval/ceval-exam/test/high_school_geography_test.csv 36.2KB
  91. minimind-master/ceval/ceval-exam/test/high_school_history_test.csv 50.77KB
  92. minimind-master/ceval/ceval-exam/test/high_school_mathematics_test.csv 36.69KB
  93. minimind-master/ceval/ceval-exam/test/high_school_physics_test.csv 56.34KB
  94. minimind-master/ceval/ceval-exam/test/high_school_politics_test.csv 77.54KB
  95. minimind-master/ceval/ceval-exam/test/ideological_and_moral_cultivation_test.csv 30.55KB
  96. minimind-master/ceval/ceval-exam/test/law_test.csv 72.86KB
  97. minimind-master/ceval/ceval-exam/test/legal_professional_test.csv 114.43KB
  98. minimind-master/ceval/ceval-exam/test/logic_test.csv 136.26KB
  99. minimind-master/ceval/ceval-exam/test/mao_zedong_thought_test.csv 50.38KB
  100. minimind-master/ceval/ceval-exam/test/marxism_test.csv 33.65KB
  101. minimind-master/ceval/ceval-exam/test/metrology_engineer_test.csv 41.37KB
  102. minimind-master/ceval/ceval-exam/test/middle_school_biology_test.csv 41.76KB
  103. minimind-master/ceval/ceval-exam/test/middle_school_chemistry_test.csv 42.22KB
  104. minimind-master/ceval/ceval-exam/test/middle_school_geography_test.csv 20.28KB
  105. minimind-master/ceval/ceval-exam/test/middle_school_history_test.csv 41.23KB
  106. minimind-master/ceval/ceval-exam/test/middle_school_mathematics_test.csv 28.41KB
  107. minimind-master/ceval/ceval-exam/test/middle_school_physics_test.csv 43.57KB
  108. minimind-master/ceval/ceval-exam/test/middle_school_politics_test.csv 66.53KB
  109. minimind-master/ceval/ceval-exam/test/modern_chinese_history_test.csv 45.2KB
  110. minimind-master/ceval/ceval-exam/test/operating_system_test.csv 26.32KB
  111. minimind-master/ceval/ceval-exam/test/physician_test.csv 77.67KB
  112. minimind-master/ceval/ceval-exam/test/plant_protection_test.csv 26.57KB
  113. minimind-master/ceval/ceval-exam/test/probability_and_statistics_test.csv 52.17KB
  114. minimind-master/ceval/ceval-exam/test/professional_tour_guide_test.csv 34.2KB
  115. minimind-master/ceval/ceval-exam/test/sports_science_test.csv 27.63KB
  116. minimind-master/ceval/ceval-exam/test/tax_accountant_test.csv 160.36KB
  117. minimind-master/ceval/ceval-exam/test/teacher_qualification_test.csv 95.84KB
  118. minimind-master/ceval/ceval-exam/test/urban_and_rural_planner_test.csv 98.31KB
  119. minimind-master/ceval/ceval-exam/test/veterinary_medicine_test.csv 33.79KB
  120. minimind-master/ceval/ceval-exam/val/
  121. minimind-master/ceval/ceval-exam/val/accountant_val.csv 18.01KB
  122. minimind-master/ceval/ceval-exam/val/advanced_mathematics_val.csv 4.83KB
  123. minimind-master/ceval/ceval-exam/val/art_studies_val.csv 3.75KB
  124. minimind-master/ceval/ceval-exam/val/basic_medicine_val.csv 2.16KB
  125. minimind-master/ceval/ceval-exam/val/business_administration_val.csv 8.28KB
  126. minimind-master/ceval/ceval-exam/val/chinese_language_and_literature_val.csv 2.87KB
  127. minimind-master/ceval/ceval-exam/val/civil_servant_val.csv 19.74KB
  128. minimind-master/ceval/ceval-exam/val/clinical_medicine_val.csv 3.59KB
  129. minimind-master/ceval/ceval-exam/val/college_chemistry_val.csv 3.83KB
  130. minimind-master/ceval/ceval-exam/val/college_economics_val.csv 12.91KB
  131. minimind-master/ceval/ceval-exam/val/college_physics_val.csv 5.6KB
  132. minimind-master/ceval/ceval-exam/val/college_programming_val.csv 8.58KB
  133. minimind-master/ceval/ceval-exam/val/computer_architecture_val.csv 3.6KB
  134. minimind-master/ceval/ceval-exam/val/computer_network_val.csv 3.3KB
  135. minimind-master/ceval/ceval-exam/val/discrete_mathematics_val.csv 3.01KB
  136. minimind-master/ceval/ceval-exam/val/education_science_val.csv 4.75KB
  137. minimind-master/ceval/ceval-exam/val/electrical_engineer_val.csv 7.31KB
  138. minimind-master/ceval/ceval-exam/val/environmental_impact_assessment_engineer_val.csv 8.29KB
  139. minimind-master/ceval/ceval-exam/val/fire_engineer_val.csv 9.08KB
  140. minimind-master/ceval/ceval-exam/val/high_school_biology_val.csv 5.56KB
  141. minimind-master/ceval/ceval-exam/val/high_school_chemistry_val.csv 5.08KB
  142. minimind-master/ceval/ceval-exam/val/high_school_chinese_val.csv 9.83KB
  143. minimind-master/ceval/ceval-exam/val/high_school_geography_val.csv 3.49KB
  144. minimind-master/ceval/ceval-exam/val/high_school_history_val.csv 6.04KB
  145. minimind-master/ceval/ceval-exam/val/high_school_mathematics_val.csv 4.68KB
  146. minimind-master/ceval/ceval-exam/val/high_school_physics_val.csv 6.69KB
  147. minimind-master/ceval/ceval-exam/val/high_school_politics_val.csv 8.31KB
  148. minimind-master/ceval/ceval-exam/val/ideological_and_moral_cultivation_val.csv 2.76KB
  149. minimind-master/ceval/ceval-exam/val/law_val.csv 7.41KB
  150. minimind-master/ceval/ceval-exam/val/legal_professional_val.csv 11.43KB
  151. minimind-master/ceval/ceval-exam/val/logic_val.csv 14.72KB
  152. minimind-master/ceval/ceval-exam/val/mao_zedong_thought_val.csv 4.84KB
  153. minimind-master/ceval/ceval-exam/val/marxism_val.csv 3.74KB
  154. minimind-master/ceval/ceval-exam/val/metrology_engineer_val.csv 5.45KB
  155. minimind-master/ceval/ceval-exam/val/middle_school_biology_val.csv 4.68KB
  156. minimind-master/ceval/ceval-exam/val/middle_school_chemistry_val.csv 5.09KB
  157. minimind-master/ceval/ceval-exam/val/middle_school_geography_val.csv 2.33KB
  158. minimind-master/ceval/ceval-exam/val/middle_school_history_val.csv 5.37KB
  159. minimind-master/ceval/ceval-exam/val/middle_school_mathematics_val.csv 4.4KB
  160. minimind-master/ceval/ceval-exam/val/middle_school_physics_val.csv 4.75KB
  161. minimind-master/ceval/ceval-exam/val/middle_school_politics_val.csv 6.69KB
  162. minimind-master/ceval/ceval-exam/val/modern_chinese_history_val.csv 4.57KB
  163. minimind-master/ceval/ceval-exam/val/operating_system_val.csv 2.81KB
  164. minimind-master/ceval/ceval-exam/val/physician_val.csv 7.42KB
  165. minimind-master/ceval/ceval-exam/val/plant_protection_val.csv 3.07KB
  166. minimind-master/ceval/ceval-exam/val/probability_and_statistics_val.csv 5.31KB
  167. minimind-master/ceval/ceval-exam/val/professional_tour_guide_val.csv 3.77KB
  168. minimind-master/ceval/ceval-exam/val/sports_science_val.csv 3KB
  169. minimind-master/ceval/ceval-exam/val/tax_accountant_val.csv 17.4KB
  170. minimind-master/ceval/ceval-exam/val/teacher_qualification_val.csv 10.97KB
  171. minimind-master/ceval/ceval-exam/val/urban_and_rural_planner_val.csv 11.48KB
  172. minimind-master/ceval/ceval-exam/val/veterinary_medicine_val.csv 3.97KB
  173. minimind-master/ceval/ceval_result/
  174. minimind-master/ceval/ceval_result/accountant_val_result.csv 18.14KB
  175. minimind-master/ceval/ceval_result/advanced_mathematics_val_result.csv 4.94KB
  176. minimind-master/ceval/ceval_result/art_studies_val_result.csv 3.87KB
  177. minimind-master/ceval/ceval_result/basic_medicine_val_result.csv 2.26KB
  178. minimind-master/ceval/ceval_result/business_administration_val_result.csv 8.42KB
  179. minimind-master/ceval/ceval_result/chinese_language_and_literature_val_result.csv 2.99KB
  180. minimind-master/ceval/ceval_result/civil_servant_val_result.csv 19.87KB
  181. minimind-master/ceval/ceval_result/clinical_medicine_val_result.csv 3.71KB
  182. minimind-master/ceval/ceval_result/college_chemistry_val_result.csv 3.95KB
  183. minimind-master/ceval/ceval_result/college_economics_val_result.csv 13.05KB
  184. minimind-master/ceval/ceval_result/college_physics_val_result.csv 5.71KB
  185. minimind-master/ceval/ceval_result/college_programming_val_result.csv 8.71KB
  186. minimind-master/ceval/ceval_result/computer_architecture_val_result.csv 3.71KB
  187. minimind-master/ceval/ceval_result/computer_network_val_result.csv 3.41KB
  188. minimind-master/ceval/ceval_result/discrete_mathematics_val_result.csv 3.12KB
  189. minimind-master/ceval/ceval_result/education_science_val_result.csv 4.88KB
  190. minimind-master/ceval/ceval_result/electrical_engineer_val_result.csv 7.44KB
  191. minimind-master/ceval/ceval_result/environmental_impact_assessment_engineer_val_result.csv 8.44KB
  192. minimind-master/ceval/ceval_result/fire_engineer_val_result.csv 9.2KB
  193. minimind-master/ceval/ceval_result/high_school_biology_val_result.csv 5.67KB
  194. minimind-master/ceval/ceval_result/high_school_chemistry_val_result.csv 5.2KB
  195. minimind-master/ceval/ceval_result/high_school_chinese_val_result.csv 9.94KB
  196. minimind-master/ceval/ceval_result/high_school_geography_val_result.csv 3.6KB
  197. minimind-master/ceval/ceval_result/high_school_history_val_result.csv 6.15KB
  198. minimind-master/ceval/ceval_result/high_school_mathematics_val_result.csv 4.8KB
  199. minimind-master/ceval/ceval_result/high_school_physics_val_result.csv 6.8KB
  200. minimind-master/ceval/ceval_result/high_school_politics_val_result.csv 8.43KB
  201. minimind-master/ceval/ceval_result/ideological_and_moral_cultivation_val_result.csv 2.88KB
  202. minimind-master/ceval/ceval_result/law_val_result.csv 7.51KB
  203. minimind-master/ceval/ceval_result/legal_professional_val_result.csv 11.54KB
  204. minimind-master/ceval/ceval_result/logic_val_result.csv 14.83KB
  205. minimind-master/ceval/ceval_result/mao_zedong_thought_val_result.csv 4.95KB
  206. minimind-master/ceval/ceval_result/marxism_val_result.csv 3.84KB
  207. minimind-master/ceval/ceval_result/metrology_engineer_val_result.csv 5.57KB
  208. minimind-master/ceval/ceval_result/middle_school_biology_val_result.csv 4.8KB
  209. minimind-master/ceval/ceval_result/middle_school_chemistry_val_result.csv 5.21KB
  210. minimind-master/ceval/ceval_result/middle_school_geography_val_result.csv 2.44KB
  211. minimind-master/ceval/ceval_result/middle_school_history_val_result.csv 5.49KB
  212. minimind-master/ceval/ceval_result/middle_school_mathematics_val_result.csv 4.52KB
  213. minimind-master/ceval/ceval_result/middle_school_physics_val_result.csv 4.86KB
  214. minimind-master/ceval/ceval_result/middle_school_politics_val_result.csv 6.8KB
  215. minimind-master/ceval/ceval_result/modern_chinese_history_val_result.csv 4.69KB
  216. minimind-master/ceval/ceval_result/operating_system_val_result.csv 2.92KB
  217. minimind-master/ceval/ceval_result/physician_val_result.csv 7.55KB
  218. minimind-master/ceval/ceval_result/plant_protection_val_result.csv 3.18KB
  219. minimind-master/ceval/ceval_result/probability_and_statistics_val_result.csv 5.43KB
  220. minimind-master/ceval/ceval_result/professional_tour_guide_val_result.csv 3.9KB
  221. minimind-master/ceval/ceval_result/sports_science_val_result.csv 3.11KB
  222. minimind-master/ceval/ceval_result/tax_accountant_val_result.csv 17.54KB
  223. minimind-master/ceval/ceval_result/teacher_qualification_val_result.csv 11.11KB
  224. minimind-master/ceval/ceval_result/test.log 4.34KB
  225. minimind-master/ceval/ceval_result/urban_and_rural_planner_val_result.csv 11.62KB
  226. minimind-master/ceval/ceval_result/veterinary_medicine_val_result.csv 4.09KB
  227. minimind-master/chat_openai_api.py 1.42KB
  228. minimind-master/data_process.py 6.62KB
  229. minimind-master/eval_ceval.py 6.87KB
  230. minimind-master/export_model.py 2.07KB
  231. minimind-master/fast_infenence.py 4.2KB
  232. minimind-master/images/
  233. minimind-master/images/1-wiki.png 136.21KB
  234. minimind-master/images/2-eval.png 109.16KB
  235. minimind-master/images/2-wiki.png 72.93KB
  236. minimind-master/images/3-wiki.png 229.81KB
  237. minimind-master/images/4-wiki.png 104.37KB
  238. minimind-master/images/5-wiki.png 239.4KB
  239. minimind-master/images/LLM-structure-moe.png 153.43KB
  240. minimind-master/images/LLM-structure.png 133.48KB
  241. minimind-master/images/fastgpt.png 72.41KB
  242. minimind-master/images/gpt3_config.png 65.63KB
  243. minimind-master/images/logger.png 81.37KB
  244. minimind-master/images/logo.png 402KB
  245. minimind-master/images/streamlit.png 38.81KB
  246. minimind-master/model/
  247. minimind-master/model/LMConfig.py 2.25KB
  248. minimind-master/model/__pycache__/
  249. minimind-master/model/__pycache__/LMConfig.cpython-310.pyc 1.49KB
  250. minimind-master/model/__pycache__/dataset.cpython-310.pyc 3.58KB
  251. minimind-master/model/__pycache__/model.cpython-310.pyc 15.57KB
  252. minimind-master/model/dataset.py 4.23KB
  253. minimind-master/model/minimind_tokenizer/
  254. minimind-master/model/minimind_tokenizer/merges.txt 56.41KB
  255. minimind-master/model/minimind_tokenizer/tokenizer.json 255.29KB
  256. minimind-master/model/minimind_tokenizer/tokenizer_config.json 1.57KB
  257. minimind-master/model/minimind_tokenizer/vocab.json 93.75KB
  258. minimind-master/model/model.py 16.53KB
  259. minimind-master/my_openai_api.py 15.03KB
  260. minimind-master/requirements.txt 425B
  261. minimind-master/train_tokenizer.py 5.26KB

资源介绍:

minimind最小开源大模型，可以自己训练自己的大模型
<div align="center"> ![logo](./images/logo.png) </div> <div align="center"> ![visitors](https://visitor-badge.laobi.icu/badge?page_id=jingyaogong/minimind) [![GitHub Repo stars](https://img.shields.io/github/stars/jingyaogong/minimind?style=social)](https://github.com/jingyaogong/minimind/stargazers) [![GitHub Code License](https://img.shields.io/github/license/jingyaogong/minimind)](LICENSE) [![GitHub last commit](https://img.shields.io/github/last-commit/jingyaogong/minimind)](https://github.com/jingyaogong/minimind/commits/master) [![GitHub pull request](https://img.shields.io/badge/PRs-welcome-blue)](https://github.com/jingyaogong/minimind/pulls) [![Collection](https://img.shields.io/badge/🤗-MiniMind%20%20Collection-blue)](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5) </div> <div align="center"> <h3>"大道至简"</h3> </div> <div align="center"> 中文 | [English](./README_en.md) </div> * 本开源项目旨在完全从0开始，最快仅用3小时！即可训练出仅为26M大小的微型语言模型**MiniMind**。 * **MiniMind**极其轻量，体积约是 GPT3 的 $\frac{1}{7000}$，力求做到最普通的个人GPU也可快速推理甚至训练。 * **MiniMind**改进自DeepSeek-V2、Llama3结构，项目包含整个数据处理、pretrain、sft、dpo的全部阶段，包含混合专家(MoE)模型。 * 这是一个既是开源项目，又是入门LLM教程，同时也是一个初具雏形的开源模型，希望能起到抛砖引玉的作用。 --- <div align="center"> https://github.com/user-attachments/assets/88b98128-636e-43bc-a419-b1b1403c2055 [Bilibili视频链接](https://www.bilibili.com/video/BV12dHPeqE72/?share_source=copy_web&vd_source=670c2504f88726f8cf4a21ef6147c0e8) </div> # 📌 Introduction 大语言模型（LLM）领域，如 GPT、LLaMA、GLM 等，虽然它们效果惊艳， 但动辄10 Bilion庞大的模型参数个人设备显存远不够训练，甚至推理困难。 几乎所有人都不会只满足于用Lora等方案fine-tuing大模型学会一些新的指令， 这约等于在教牛顿玩21世纪的智能手机，然而，这远远脱离了学习物理本身的奥妙。 此外，卖课付费订阅的营销号漏洞百出的一知半解讲解AI的教程遍地， 让理解LLM的优质内容雪上加霜，严重阻碍了学习者。 因此，本项目的目标是把上手LLM的门槛无限降低， 直接从0开始训练一个极其轻量的语言模型。 > [!TIP] > （截至2024-9-17）minimind训练了3个型号模型，最小仅需26M（0.02B），即可具备流畅的对话能力！ | 模型 (大小) | tokenizer长度 | 推理占用 | release | 主观评分（/100） | |-------------------------|-------------|--------|------------|------------| | minimind-v1-small (26M) | 6400 | 0.5 GB | 2024.08.28 | 50' | | minimind-v1-moe (4×26M) | 6400 | 1.0 GB | 2024.09.17 | 55' | | minimind-v1 (108M) | 6400 | 1.0 GB | 2024.09.01 | 60' | > 该分析在一个带有Torch 2.1.2、CUDA 12.2和Flash Attention 2的RTX 3090 GPU上运行。 项目包含： - 公开MiniMind模型代码（包含Dense和MoE模型）、Pretrain、SFT指令微调、LoRA微调、DPO偏好优化的全过程代码、数据集和来源。 - 兼容`transformers`、`accelerate`、`trl`、`peft`等流行框架。 - 训练支持单机单卡、单机多卡(DDP、DeepSpeed)训练。训练过程中支持在任意位置停止，及在任意位置继续训练。 - 在Ceval数据集上进行模型测试的代码。 - 实现Openai-Api基本的chat接口，便于集成到第三方ChatUI使用（FastGPT、Open-WebUI等）。 希望此开源项目可以帮助LLM初学者快速入门！ ### 👉**最近更新** <details close> <summary> <b>2024-09-17 (new🎉)</b> </summary> - 更新minimind-v1-moe模型 - 为了防止歧义，不再使用mistral_tokenizer分词，全部采用自定义的minimind_tokenizer作为分词器。 </details> <details close> <summary> <b>2024-09-01</b> </summary> - 更新minimind-v1 (108M)模型，采用minimind_tokenizer，预训练轮次3 + SFT轮次10，更充分训练，性能更强。 - 项目已部署至ModelScope创空间，可以在此网站上体验： - [ModelScope在线体验](https://www.modelscope.cn/studios/gongjy/minimind) </details> <details close> <summary> <b>2024-08-27</b> </summary> - 项目首次开源 </details> # 📌 Environment 仅是我个人的软硬件环境配置，自行酌情更改： * Ubuntu == 20.04 * Python == 3.9 * Pytorch == 2.1.2 * CUDA == 12.2 * [requirements.txt](./requirements.txt) # 📌 Quick Inference & Test <div align="center" style="font-size: 1.5em; font-weight: bold;"> <img src="https://huggingface.co/front/assets/huggingface_logo-noborder.svg" alt="Hugging Face Logo" style="vertical-align: middle; height: 30px;" /> Hugging Face [MiniMind (HuggingFace)](https://huggingface.co/collections/jingyaogong/minimind-66caf8d999f5c7fa64f399e5) <img src="https://g.alicdn.com/sail-web/maas/1.15.0/static/modelscopeIcon.cd89353f.svg" alt="Hugging Face Logo" style="vertical-align: middle; height: 30px;" /> [MiniMind (ModelScope)](https://www.modelscope.cn/models/gongjy/minimind-v1) </div> ```bash # step 1 git clone https://huggingface.co/jingyaogong/minimind-v1 ``` ```bash # step 2 python 2-eval.py ``` 或者启动streamlit，启动网页聊天界面 ```bash # or step 3, use streamlit streamlit run fast_inference.py ``` ![](./images/streamlit.png) <div align="center"> 项目已部署至ModelScope创空间，可以在此网站上体验： [ModelScope在线体验](https://www.modelscope.cn/studios/gongjy/minimind) </div> # 📌 Quick Start * 0、环境安装 ```bash pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple ``` * 1、克隆项目代码 ```text git clone https://github.com/jingyaogong/minimind.git ``` * 2、如果你需要自己训练 * 2.1 下载[数据集下载地址](#数据集下载地址)放到`./dataset`目录下 * 2.2 `python data_process.py`处理数据集，例如pretrain数据提前进行token-encoder、sft数据集抽离qa到csv文件 * 2.3 在`./model/LMConfig.py` 中调整model的参数配置 * 2.4 `python 1-pretrain.py` 执行预训练 * 2.5 `python 3-full_sft.py` 执行指令微调 * 2.6 `python 4-lora_sft.py` 执行lora微调（非必须） * 2.7 `python 5-dpo_train.py` 执行DPO人类偏好强化学习对齐（非必须） * 3、测试模型推理效果 * 确保需要使用的，训练完成的参数权重位于`./out/`目录下 * 也可以直接去[训练完成的模型权重](#训练完成的模型权重)下载使用我训练好的 ```text out ├── multi_chat │   ├── full_sft_512.pth │   ├── full_sft_512_moe.pth │   └── full_sft_768.pth ├── single_chat │   ├── full_sft_512.pth │   ├── full_sft_512_moe.pth │   └── full_sft_768.pth ├── pretrain_768.pth ├── pretrain_512_moe.pth ├── pretrain_512.pth ``` * `python 0-eval_pretrain.py`测试预训练模型的接龙效果 * `python 2-eval.py`测试模型的对话效果 ![2-eval](./images/2-eval.png) 🍭 【Tip】预训练和全参微调pretrain和full_sft均支持多卡加速 * 单机N卡启动训练(DDP) ```bash torchrun --nproc_per_node N 1-pretrain.py # and torchrun --nproc_per_node N 3-full_sft.py ``` * 单机N卡启动训练(DeepSpeed) ```bash deepspeed --master_port 29500 --num_gpus=N 1-pretrain.py # and deepspeed --master_port 29500 --num_gpus=N 3-full_sft.py ``` # 📌 Data sources - 🤖 分词器：nlp中的Tokenizer类似于词典，将单词从自然语言通过“词典”映射到0,1,36这样的数字，可以理解为数字就代表了单词在“词典�
100+评论
captcha
    相关资源
    类型标题大小时间
    ZIPRDTU H071 (2023-11-23 16-57-05).zip5.46MB5月前
    ZIP点餐系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip15.97MB5月前
    ZIP宿舍管理小程序 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip18.27MB5月前
    ZIP学生宿舍管理系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip20.59MB5月前
    ZIP自助点餐系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip34.04MB5月前
    ZIP中国各地美食推荐平台 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip28.77MB5月前
    ZIP在线选课系统 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip37.35MB5月前
    ZIP展柜设计公司平面布置小程序 微信小程序+SpringBoot毕业设计 源码+数据库+论文+启动教程.zip21.23MB5月前