Toke(n)icer v0.0.3

Qubitium released this 21 Feb 07:18

· 5 commits to main since this release

b0b2591

What's Changed

Now tokenicer instance dynamically inherits the native tokenizer.__class__ of tokenizer passed in or loaded via our Tokenicer.load() api.

fix mpt pad token bug by @CL-ModelCloud in #24
fix model_config bugs by @CL-ModelCloud in #25
test code clean up by @CL-ModelCloud in #26
Inherits PretrainedTokenizer by @Qubitium in #28

Full Changelog: v0.0.2...v0.0.3

Contributors

Qubitium and CL-ModelCloud

Assets 2