phyer/avante.nvim
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
avante.nvim/crates/avante-tokenizers/README.md

2 lines
54 B
Markdown
Raw Normal View History

feat: tokenizers (#429) * feat: tokenizers This reverts commit d5a4db8321d232a1b9c0d86fc38e8dd516d15776. * fix(inputs): #422 Signed-off-by: Aaron Pham <contact@aarnphm.xyz> --------- Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
2024-08-31 13:39:50 -04:00
A simple crate to unify hf/tokenizers and tiktoken-rs
Reference in New Issue Copy Permalink