geoffsee/code-tokenizer
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update README

Browse Source
This commit is contained in:
Geoff Seemueller
2024-12-01 13:00:42 -05:00
parent b76313b57e
commit 0bbb04e8e1
2 changed files with 7 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files

8
README.md
View File

@@ -1,6 +1,8 @@
# code-tokenizer-md # code-tokenizer-md
> Created to push creative limits. Processes git repository files into markdown with token counting and sensitive data redaction. ![code-tokenizer-md](./code-tokenizer-md.jpg)
> Push the limits of possible
## Quick Start ## Quick Start
```bash ```bash
@@ -12,6 +14,10 @@ $ npx code-tokenizer-md
`code-tokenizer-md` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer. `code-tokenizer-md` is a tool that processes git repository files, cleans code, redacts sensitive information, and generates markdown documentation with token counts using the Llama 3 tokenizer.
## Philosophy
> Human-first technologies for a better tomorrow.
```mermaid ```mermaid
graph TD graph TD
Start[Start] -->|Read| Git[Git Files] Start[Start] -->|Read| Git[Git Files]

BIN
code-tokenizer-md.jpg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 638 KiB