0557deeab7
* feat: tokenizers This reverts commit d5a4db8321d232a1b9c0d86fc38e8dd516d15776. * fix(inputs): #422 Signed-off-by: Aaron Pham <contact@aarnphm.xyz> --------- Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
54 B
54 B
A simple crate to unify hf/tokenizers and tiktoken-rs