首页下载资源人工智能lm-dataformat-master.zip

ZIPlm-dataformat-master.zip

m0_6100655252.93KB需要积分:1

资源文件列表:

lm_dataformat-master.zip 大约有15个文件
  1. lm_dataformat-master/
  2. lm_dataformat-master/.gitignore 21B
  3. lm_dataformat-master/.travis.yml 210B
  4. lm_dataformat-master/LICENSE 1.02KB
  5. lm_dataformat-master/README.md 775B
  6. lm_dataformat-master/lm_dataformat/
  7. lm_dataformat-master/lm_dataformat/__init__.py 12.28KB
  8. lm_dataformat-master/setup.py 592B
  9. lm_dataformat-master/test/
  10. lm_dataformat-master/test/blns.jsonl.zst.tar 26.5KB
  11. lm_dataformat-master/test/blns.txt 29.12KB
  12. lm_dataformat-master/test/blns.txt.tar.gz 10.69KB
  13. lm_dataformat-master/test/blns.txt.zip 10.24KB
  14. lm_dataformat-master/test/test_dat_archive_reader.py 4.17KB
  15. lm_dataformat-master/test/testtarfile.tar 7KB

资源介绍:

lm-dataformat-master.zip
# LM_Dataformat [![Build Status](https://travis-ci.org/leogao2/lm_dataformat.svg?branch=master)](https://travis-ci.org/leogao2/lm_dataformat) [![Coverage Status](https://coveralls.io/repos/github/leogao2/lm_dataformat/badge.svg?branch=master)](https://coveralls.io/github/leogao2/lm_dataformat?branch=master) Utilities for storing data for LM training. ## Basic Usage To write: ``` ar = Archive('output_dir') for x in something(): # do other stuff ar.add_data(somedocument, meta={ 'example': stuff, 'someothermetadata': [othermetadata, otherrandomstuff], 'otherotherstuff': True }) # remember to commit at the end! ar.commit() ``` To read: ``` rdr = Reader('input_dir_or_file') for doc in rdr.stream_data(): # do something with the document ```
100+评论
captcha