diff options
Diffstat (limited to '_log')
| -rw-r--r-- | _log/lex.md | 261 | ||||
| -rw-r--r-- | _log/lex/lex-email.png | bin | 0 -> 187842 bytes | |||
| -rw-r--r-- | _log/lex/rpi5.jpg | bin | 0 -> 90173 bytes |
3 files changed, 261 insertions, 0 deletions
diff --git a/_log/lex.md b/_log/lex.md new file mode 100644 index 0000000..adebd49 --- /dev/null +++ b/_log/lex.md @@ -0,0 +1,261 @@ +--- +title: 'Lex: LLM dictionary on RPi5' +date: 2026-09-19 +layout: post +--- + +Year of our Lord 2026. LLMs are everywhere. + +Built Lex, a context-aware dictionary. + +<table style="width: 100%; border-collapse: collapse;"> + <tr> + <td> + <img src="rpi5.jpg" alt="RPi5" style="max-height: 450px; width: auto; max-width: 100%;"> + </td> + <td> + <img src="lex-email.png" alt="Email" style="max-height: 450px; width: auto; max-width: 100%;"> + </td> + </tr> +</table> + +Components: lex-cgi, lex-llm, lex-mailer. User emails word and context. +lex-cgi receives webhook, runs lex-llm, writes definition to file. lex-mailer +emails past entries daily, timed for spaced repetition. + +Tried local inference using llama.cpp + Llama 3.1 8B (GGUF) first: + +``` +llama_backend_init(); +mparams = llama_model_default_params(); +mparams.n_gpu_layers = 0; /* force all layers onto CPU */ +model = llama_model_load_from_file(MODEL_PATH, mparams); + +cparams = llama_context_default_params(); +cparams.n_ctx = N_CTX; /* 1024 */ +llama_init_from_model(model, cparams); +``` + +Generation loop synthesizes definition: + +``` +for (i = 0; i < MAX_TOKENS; i++) { + new_token_id = llama_sampler_sample(smpl, ctx, -1); + llama_sampler_accept(smpl, new_token_id); + if (llama_vocab_is_eog(vocab, new_token_id)) + break; + + char buf[128]; + n = llama_token_to_piece(vocab, new_token_id, buf, sizeof(buf), 0, false); + if (n > 0) { + fwrite(buf, 1, (size_t)n, out); + fflush(out); + } + + batch = llama_batch_get_one(&new_token_id, 1); + if (llama_decode(ctx, batch) != 0) + break; +} +``` + +Build what's needed: + +``` +cmake -S $(LLAMA_SRC_DIR) -B $(LLAMA_BUILD) \ + -DBUILD_SHARED_LIBS=OFF \ + -DGGML_OPENMP=OFF \ + -DGGML_VULKAN=OFF \ + -DLLAMA_BUILD_SERVER=OFF \ + -DLLAMA_BUILD_UI=OFF + # + misc size-reduction flags, omitted + +cmake --build $(LLAMA_BUILD) --config Release -j$$(sysctl -n hw.ncpu) +``` + +Functional, but requires a $40/mo VPS (8 GB RAM). Benchmarked alternative +Q4_K_M models on T490 (i7-10510U, OpenBSD 7.9, CPU only): + +<pre class="pre-no-style"> ++--------------+--------+-------------+---------------------+----------+ +| Model | Tokens | Tok/s (μ±σ) | Time s (Med/Range) | RAM (MB) | ++--------------+--------+-------------+---------------------+----------+ +| Qwen 2.5 3B | 114.0 | 5.22 ± 0.31 | 21.79 (19.39-25.20) | 3,235.22 | +| Qwen 2.5 7B | 104.5 | 2.09 ± 0.07 | 49.27 (45.50-52.98) | 7,335.18 | +| Phi 3.5 3.8B | 128.2 | 2.06 ± 0.10 | 71.87 (48.67-72.37) | 3,922.55 | +| LLaMA 3.1 8B | 115.5 | 2.01 ± 0.08 | 57.10 (53.95-66.09) | 7,873.52 | +| Mistral 7B | 106.6 | 1.67 ± 0.13 | 64.25 (57.85-66.86) | 7,535.61 | ++--------------+--------+-------------+---------------------+----------+ +</pre> + +Larger models caught nuances smaller ones missed, but definitions weren't +consistently better. Gemma 2 9B generated empty responses due to chat template +bug; fixed, but 7.8 GB RAM usage didn't justify re-running benchmarks. Smaller +models handled shorter prompts better. Qwen 2.5 struggled with formality level, +but produced tighter, faster definitions at 3.2 GB RAM. + +Qwen 2.5 3B halves the VPS cost—still wasteful for the occasional prompt. +Decoupling lex-llm from lex-cgi lets lex-llm run on a 4 GB Raspberry Pi 5; rest +could run on a $5/mo 1 GB VPS. + +Refactored lex-llm to communicate over Unix domain sockets. lex-cgi now spools +incoming emails to disk. A shell daemon periodically fetches them via SSH, runs +lex-llm, and syncs output back to VPS. + +``` +while true; do + # fork to avoid resource leakage on unexpected errors + ( + prompt=$(ssh -F "${SSH_CONFIG}" -n "${SSH_HOST}" \ + "cat '${SPOOL_DIR}/${SPOOL_FILE}'") + + response=$(print -r -- "${prompt}" \ + | nc -w 300 -U "${SOCK_PATH}" \ + | awk '# formatting fix, omitted...' + ) + + # validate output, write to tmp_file, + # atomically mv into place, rm spool file—omitted + + print -r -- "${response}" \ + | fold -s -w 72 \ + | ssh -F "${SSH_CONFIG}" "${SSH_HOST}" "${remote_cmd}" + ) +done +``` + +Performance: lex-llm was 8x slower (190 s/req) on the RPi5. Benchmarked +version loaded the model every request. Loading it once at init reduced +execution time to 55 seconds. + +Forcing the model to RAM at init via LLAMA_LOAD_MODE_MLOCK would have been +desirable, but RPi5 lacks RAM to grant the mlock. Between LLAMA_LOAD_MODE_NONE +and LLAMA_LOAD_MODE_MMAP, former yielded better steady-state performance (1.3 +GB less MAX RSS, 1-2 seconds faster). + +RPi5's slower Cortex-A76 processor was still 2x behind benchmarks. +llama_decode() was evaluating system + user prompt every request. System prompt +doesn't change between requests; cached token offsets and decoded only the part +that changed: + +``` +/* drop everything after common prefix. */ +llama_memory_seq_rm(llama_get_memory(llm->ctx), 0, n_common, -1); + +/* decode the diff */ +if (n_common < n_new_tokens) { + batch = llama_batch_get_one(new_tokens + n_common, + n_new_tokens - n_common); + if (llama_decode(llm->ctx, batch) != 0) + return; + + memcpy(llm->cached_tokens + n_common, + new_tokens + n_common, + (size_t)(n_new_tokens - n_common) * sizeof(llama_token)); + + llm->n_cached_tokens = n_new_tokens; +} +``` + +Prefix-caching reduced execution time to 35 seconds—acceptable for an +email-driven async dictionary. + +Note on measurements: 3.4 A supply likely caused CPU throttling on the RPi5 +during measurements. Final version uses the recommended 5 A supply. + +Fidelity: Didn't notice qualitative differences between T490 and RPi5 output +under normal operation. They exist, however. System prompt fragment duplicated +the SYNONYMS instruction inside GENERAL TONE: + +``` +One terse sentence of describing general tone. One terse sentence of +standard conversational synonyms. +``` + +RPi5 echoed the second stray sentence verbatim; T490 did not: + +``` +GENERAL TONE: +The phrase suggests the impending end of life with somber and formal +language. +One terse sentence of standard conversational synonyms: Buried or +surrounded by darkness. +``` + +Possible artifact of non-associative floating-point arithmetic (x86 SIMD vs ARM +NEON instruction ordering). Dropping duplicate sentence from prompt reconciled +the outputs. + +Storage: lex-llm binary is 12.9 MB. GGUF takes up 2 GB. Standard OpenBSD +install recommends 8 GB. SD card I have is 4 GB—3.5 GB usable. + +Flashed OpenBSD onto SD card; tossed man, game, and X file sets. Kept comp set +to build lex-llm. Disabled boot-time library re-linking and purged +/usr/share/relink to reclaim a further 450 MB. OS + lex-llm + model now fits: + +``` +Filesystem Size Used Avail Capacity Mounted on +/dev/sd0a 3.5G 2.8G 477M 86% / +mfs:76650 61.9M 5.0K 58.8M 1% /tmp +mfs:29597 15.4M 87.0K 14.6M 1% /var/run +mfs:35201 30.9M 113K 29.3M 1% /var/log +``` + +Installation went smoothly, except the RPi5 wouldn't boot without a serial +adapter plugged into the JST port. RX pin floated with no internal pull-up +enabled. Resolved by adding an external 10 kΩ pull-up (RX to 3.3 V). + + +Worked well for two weeks. Then the file system corrupted twice within a week. +Disk had 400M+ free space; badblocks revealed no bad sectors. fsck_ffs reported +a partially truncated inode indicating an interrupted write; momentary link +glitch perhaps—no way to know now. + +Relocated /dev (sshd pty allocation) to mfs. Moved cron's log to /var/log and +symlinked ntpd.drift to /var/run/ntpd.drift. Assigned a static IP and disabled +most daemons (dhcpleased, resolvd, slaacd). / is now read-only: + +``` +/dev/sd0a on / type ffs (local, noatime, wxallowed, read-only) +mfs:69255 on /dev type mfs (asynchronous, local, nosuid, size=32768 512-blocks) +mfs:75124 on /tmp type mfs (asynchronous, local, nodev, nosuid, size=131072 512-blocks) +mfs:15499 on /var/log type mfs (asynchronous, local, nodev, nosuid, size=65536 512-blocks) +mfs:14160 on /var/run type mfs (asynchronous, local, nodev, nosuid, size=32768 512-blocks) +``` + +Security: PGP keys authenticate requests. lex-cgi (chrooted), lex-llm, +lex-mailer employ pledge and unveil: + +``` +my $pid = fork(); +if ($pid == 0) { + pledge(qw(stdio proc exec)); + # sendmail +} + +# unveil rules... +unveil($dict_dir, 'r'); +unveil(); # unveil lock + +# main process drops 'exec' +pledge(qw(stdio rpath wpath cpath proc)); +``` + +Packet filter blocks network scans, access to other hosts: + +``` +set skip on lo +block all + +block out on $net_if to $home_net +pass out on $net_if proto { tcp, udp } to $router port domain +pass out on $net_if inet proto tcp from ($net_if) to $vps port ssh +pass out on $net_if proto udp to ! $home_net port ntp +pass out on $net_if proto tcp to ! $home_net port https +pass in on $net_if inet proto tcp from $home_net to ($net_if) port ssh +``` + +End of line. + +Source: <a +href="https://git.asciimx.com/lex/tree/?id=b904341947bfeac99fe91d3c9ace760f7da312fd" +class="external" target="_blank" rel="noopener noreferrer">b904341</a> diff --git a/_log/lex/lex-email.png b/_log/lex/lex-email.png Binary files differnew file mode 100644 index 0000000..40e17de --- /dev/null +++ b/_log/lex/lex-email.png diff --git a/_log/lex/rpi5.jpg b/_log/lex/rpi5.jpg Binary files differnew file mode 100644 index 0000000..5594071 --- /dev/null +++ b/_log/lex/rpi5.jpg |
