About Tokenizer
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
Tokenizer 1.3.3
Contains Typescript and C# implementation of byte pair encoding (BPE) tokenizer for OpenAI LLMs, it is based on open sourced rust implementation in the OpenAI tiktoken.
- By Microsoft Corporation
- Open Source
- Portable
Technical details
| Publisher | Microsoft Corporation |
|---|---|
| Version | 1.3.3 |
| License | MIT |
| Installer type | zip |
| Architecture | x64 |
| Package ID | Microsoft.Tokenizer |
Install Tokenizer from the command line
- Install with Windows Package Manager
winget install --id=Microsoft.Tokenizer -e- Update to the latest version
winget upgrade --id=Microsoft.Tokenizer -e- Uninstall
winget uninstall --id=Microsoft.Tokenizer -e
Frequently asked questions
Is Tokenizer free?
Yes. Tokenizer is free and open-source software released under the MIT license.
What is the latest version of Tokenizer?
The latest version is 1.3.3.
Is it safe to download Tokenizer?
The download button links to the official installer published on github.com, the same file Windows Package Manager installs. You can verify the downloaded file by comparing its SHA-256 checksum with e0f039c6521fb7a4e3013386cd60ed64fb44ad9cab74bbe7d744b60d1afd5ce4.
Which Windows versions does Tokenizer support?
Tokenizer runs on Windows and is available for 64-bit (x64) systems.
How do I uninstall Tokenizer?
Open Settings › Apps › Installed apps, find Tokenizer and select Uninstall. From the command line, run winget uninstall --id=Microsoft.Tokenizer -e.
Is there a portable version of Tokenizer?
Yes. Tokenizer is available as a portable ZIP archive (Tokenizer.zip, 64-bit (x64)) that runs without installation. Copy it to any folder or USB drive and run it from there.
Alternatives to Tokenizer
Other Web development programs for Windows from different developers.
VSCodium 1.143.17183
VSCodium is a community-driven, freely-licensed and telemetry-free rebuilt of Microsoft's Visual Studio Code.
Deno 2.9.8
A modern runtime for JavaScript and TypeScript
NSwagStudio 14.7.1
The OpenAPI/Swagger API toolchain for .NET and TypeScript
Threema For Desktop 2.5.1
Unofficial desktop client for Threema
RunJS 4.1.0
RunJS is an JavaScript playground that auto-evaluates your code as you type and outputs the results to a console pane.
Biome 2.6.3
Biome is a toolchain for web projects, aimed at providing functionalities to maintain them. It offers a formatter and linter, usable via CLI and LSP.
More from Microsoft Corporation
All 391 programs »
Microsoft Edge 156.0.4314.50
Microsoft Edge is a cross-platform web browser built for modern life.
Windows Terminal 1.25.2733.0
The new Windows Terminal, a tabbed command line experience for Windows.
Microsoft SQL Server Management Studio 20.2.1
SQL Server Management Studio (SSMS) is an integrated environment for managing any SQL infrastructure
PowerToys 0.101.2362.0
Microsoft PowerToys is a set of utilities for power users to tune and streamline their Windows experience for greater productivity.