colin
/
Witllm
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Benchmark single 4090D 15fps 

cd /wit
python train.py
This commit is contained in:
Colin 2025-02-18 21:40:49 +08:00
parent f8480678d8
commit c92db47135
1 changed files with 0 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
wit/dataset/dataset.py
View File

@ -47,7 +47,6 @@ def InitDataset(config):
print(f"INFO: Load dataset end")
else:
raw_dataset = MeaningDataset(start, start + size, vocab, None, conf.level_ratio, conf.min_subitem)
print("INFO: raw_dataset.token_frequency" + raw_dataset.token_frequency())
raw_dataset.set_mask(conf.mask_level, conf.mask_idx)
train_dataset, val_dataset = raw_dataset.split(0.9)
torch.save(train_dataset, trainfile)