3 Commits

Author SHA1 Message Date
dependabot[bot]
dc5c7108e4 Bump oxc-transform from 0.44.0 to 0.82.3
Bumps [oxc-transform](https://github.com/oxc-project/oxc/tree/HEAD/napi/transform) from 0.44.0 to 0.82.3.
- [Release notes](https://github.com/oxc-project/oxc/releases)
- [Changelog](https://github.com/oxc-project/oxc/blob/main/napi/transform/CHANGELOG.md)
- [Commits](https://github.com/oxc-project/oxc/commits/crates_v0.82.3/napi/transform)

---
updated-dependencies:
- dependency-name: oxc-transform
  dependency-version: 0.82.3
  dependency-type: direct:development
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
2025-08-21 01:46:56 +00:00
Geoff Seemueller
c2a01f9b62 Update README.md
spelling error

Signed-off-by: Geoff Seemueller <28698553+geoffsee@users.noreply.github.com>
2025-07-20 18:51:05 -04:00
Geoff Seemueller
e287652c25 Update README.md
Signed-off-by: Geoff Seemueller <28698553+geoffsee@users.noreply.github.com>
2025-07-20 18:49:04 -04:00
2 changed files with 8 additions and 24 deletions

View File

@@ -1,25 +1,21 @@
# toak # code-tokenizer
it's no joke
[![npm version](https://img.shields.io/npm/v/toak)](https://www.npmjs.com/package/toak) [![npm version](https://img.shields.io/npm/v/toak)](https://www.npmjs.com/package/toak)
![Tests](https://github.com/seemueller-io/toak/actions/workflows/tests.yml/badge.svg) ![Tests](https://github.com/seemueller-io/code-tokenizer/actions/workflows/tests.yml/badge.svg)
[![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0.html) [![License: AGPL v3](https://img.shields.io/badge/License-AGPL_v3-blue.svg)](https://www.gnu.org/licenses/agpl-3.0.html)
## Overview ## Overview
`toak` is an intentionally simple yet powerful tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer. `toak` is a cli tool, named for phonetics, that processes git repository files, cleans code, redacts sensitive information, and generates a `prompt.md` with token counts using the Llama 3 tokenizer.
```shell ```shell
$ cd your-git-repo $ cd your-git-repo
$ npx toak $ npx toak
``` ```
![toak](https://github.com/seemueller-io/toak/blob/471c2a359e342c0103d2074650afe1f1b2b5f71d/toak.jpg?raw=true) ![toak](https://github.com/seemueller-io/code-tokenizer/blob/471c2a359e342c0103d2074650afe1f1b2b5f71d/toak.jpg?raw=true)
## Philosophy
1. _Human-first_ technologies for a better future.
2. If you don't like the name...good.
---
## Features ## Features
@@ -29,14 +25,10 @@ $ npx toak
- Redacts sensitive information (API keys, tokens, JWT, hashes) - Redacts sensitive information (API keys, tokens, JWT, hashes)
- Counts tokens using llama3-tokenizer-js - Counts tokens using llama3-tokenizer-js
- Supports nested .toak-ignore files - Supports nested .toak-ignore files
### Token Cleaning
- Removes single-line and multi-line comments - Removes single-line and multi-line comments
- Strips console.log statements - Strips console.log statements
- Removes import statements - Removes import statements
- Cleans up whitespace and empty lines - Cleans up whitespace and empty lines
### Security Features
- Redacts API keys and secrets - Redacts API keys and secrets
- Masks JWT tokens - Masks JWT tokens
- Hides authorization tokens - Hides authorization tokens
@@ -45,16 +37,8 @@ $ npx toak
## Requirements ## Requirements
- Node.js (>=14.0.0) - npm/bun/yarn/pnpm
- Git repository
- Bun runtime (for development)
## Installation
```bash
npm install toak
```
## Usage ## Usage
### CLI ### CLI

View File

@@ -64,7 +64,7 @@
"bun-plugin-isolated-decl": "^0.1.10", "bun-plugin-isolated-decl": "^0.1.10",
"eslint": "^9.24.0", "eslint": "^9.24.0",
"globals": "^15.15.0", "globals": "^15.15.0",
"oxc-transform": "^0.44.0", "oxc-transform": "^0.82.3",
"prettier": "^3.5.3", "prettier": "^3.5.3",
"typescript": "^5.8.3" "typescript": "^5.8.3"
}, },