7 Commits

Author	SHA1	Message	Date
geoffsee	296d4dbe7e	add root dockerfile that contains binaries for all services	2025-09-04 14:54:20 -04:00
geoffsee	fb5098eba6	fix clippy errors	2025-09-04 13:53:00 -04:00
geoffsee	c1c583faab	run cargo fmt	2025-09-04 13:45:25 -04:00
geoffsee	1e02b12cda	fixes issue with model selection	2025-09-04 13:42:30 -04:00
geoffsee	ff55d882c7	reorg + update docs with new paths	2025-09-04 12:40:59 -04:00
geoffsee	d04340d9ac	update docs	2025-08-28 12:54:09 -04:00
geoffsee	8338750beb	Refactor apply_cached_repeat_penalty for optimized caching and reuse, add extensive unit tests, and integrate special handling for gemma-specific models. ... Removed `test_request.sh`, deprecated functionality, and unused imports; introduced a new CLI tool (`cli.ts`) for testing inference engine and adjusted handling of non-streaming/streaming chat completions. - Add CPU fallback support for text generation when primary device is unsupported - Introduce `execute_with_fallback` method to handle device compatibility and shape mismatch errors - Extend unit tests to reproduce tensor shape mismatch errors specific to model configurations - Increase HTTP timeout limits in `curl_chat_stream.sh` script for reliable API testing chat completion endpoint functions with gemma3 (no streaming) Add benchmarking guide with HTML reporting, Leptos chat crate, and middleware for metrics tracking	2025-08-27 16:15:01 -04:00