I am not a native English speaker; this article was translated by AI.

In the previous post about Harness Engineering, I compressed my default AI coding workflow into a few steps:

1. Read
2. Search
3. Change
4. Verify
5. Record

The easiest one to underestimate is Search.

Many agents fail not because they cannot edit code, but because they read the wrong place first. The user describes a behavior, a bug, or a cross-layer workflow, while the code may not contain a function with the same name. Running rg login, rg upload, or rg session is fast, but it only works when the keyword is already known. If the keyword is unknown, speed just helps the agent drift faster.

So I open-sourced a small layer I have been using recently:

ferstar/ace-wrapper

It does one narrow thing: wrap Augment Context Engine’s filesystem context search as an ace command, so coding agents can run semantic retrieval from the shell before editing.

Why this layer exists

#

The target is concrete: make the search action part of the harness.

I used to see this path often:

flowchart LR
  A[User describes behavior] --> B[Agent guesses keywords]
  B --> C[Reads nearby files]
  C --> D[Edits plausible code]
  D --> E[Verification fails]
  E --> B

The problem with this loop is that, after failure, the agent often keeps circling around the same wrong files. It can edit code; what it needs is a better entry point into candidate files. Put less politely, it is working hard after entering the wrong door.

ace-wrapper is meant to patch this part:

flowchart LR
  A[User describes behavior] --> B[ace semantic retrieval]
  B --> C[Candidate files]
  C --> D[Read returned files]
  D --> E[rg / tests confirm evidence]
  E --> F[Small patch]
  F --> G[Verify]

The important part is the order: ace only finds candidate files. Conclusions still require reading files, exact search, and tests. It is not an answer generator; it just helps the agent waste fewer steps.

Usage is short

#

Install it:

uv tool install ace-wrapper

Install a local development checkout:

uv tool install /path/to/ace-wrapper

Search for a workflow when the exact keyword is unknown:

timeout 60s ace "user uploads an unsupported file and should see skipped-file feedback" -w /repo
rg -n "unsupported|skipped|upload|file" /repo

The first command answers “which files may be relevant.” The second command confirms “which identifiers, events, copy, or tests actually exist in the code.”

I usually put this rule into a project’s AGENTS.md:

Use `timeout 60s ace "" -w ` for semantic codebase discovery.
Treat `ace` results as candidate files.
After it returns results, read the relevant files and use exact search before using them as evidence.

These lines work better than “read more context,” because they give the agent a concrete action and a boundary against false conclusions.

How it works with rg

#

ace and rg work better as consecutive steps.

I intentionally wrote this boundary into the README: ACE returns candidate files, while evidence still has to come from code and tests. That boundary matters.

Semantic retrieval returns “nearby” things. If you ask about a feature that does not exist, it may still find files that look related. If an agent treats “there are results” as “the feature exists,” it starts inventing a story. A conclusion is only defensible after reading an implementation, test, route, config, or call site.

Where it fits in Harness Engineering

#

ace-wrapper is small, and I want it to stay that way. It is closer to a small gear in the harness: it turns open-ended code discovery into a repeatable, constrained command.

I now prefer this project rule:

Read -> Search -> Change -> Verify

Here, Search means choosing the tool by problem type:

• Open-ended behavior and cross-layer workflows: use ace first
• Exact identifiers, errors, routes, and config keys: use rg
• Structural replacements: use ast-grep
• External strategy and industry practice: use web research
• Old decisions and repeated lessons: use memory

The useful part of this split is reduced agent randomness. The agent first uses semantic retrieval to narrow the reading surface, then uses deterministic tools to confirm facts, and only then changes code. The order is a little more verbose, but it is much cheaper than confidently editing the wrong file.

The prompt matters most

#

A good ace query describes behavior and avoids keyword piles:

timeout 60s ace "frontend sends requestId to backend and starts a processing job" -w /repo
timeout 60s ace "用户拖入不支持的文件后应该显示跳过文件提示" -w /repo
timeout 60s ace "how provider config is persisted and restored after app restart" -w /repo

I try to include four kinds of information:

• User action: click, drag, upload, stop generation
• Runtime boundary: frontend to backend, CLI handler to core service
• Expected effect: persist config, abort loop, show skipped-file feedback
• Known fields: sessionId, requestId, files, workspace

This is much more stable than only searching upload or provider. It lets the retrieval system look for behavior and data flow, and it reminds the agent that this step is still semantic retrieval, not evidence by itself.

Why I open-sourced it

#

ace-wrapper has very little code. The core is just FileSystemContext.create(str(workspace)) plus context.search(args.query). I wanted to preserve the workflow constraints around those few lines:

1. If the keyword is unknown, start with semantic retrieval
2. Ask one workflow per query
3. Treat results as candidate files
4. Read the files, then use rg to confirm exact evidence
5. Do not conclude without evidence

Once these rules live in the tool README, skill, and agent prompt, they become much more likely to stick. Otherwise every session depends on a human reminding the agent again, which gets old fast.

The previous post said Harness Engineering means putting an engineering track around AI. ace-wrapper is one small piece of that track: it does not make the agent better at writing code; it just helps the agent read the right place first.

I am not a native English speaker; this article was translated by AI.

This is the written version of an internal team sharing session. The slides are here:

From Vibe Coding to Harness Engineering

For a while I kept looking at one question: can AI really take over most of the coding work?

The answer is mostly settled now. When the project context, quality gates, and verification flow are in place, AI-generated code can enter the engineering workflow reliably. Human time moves from “typing the code” to “holding the line”: breaking down requirements, judging architecture, arranging context, checking boundaries, and handling failures.

Recent practice pushed this one step further. The question is no longer how to make the prompt prettier. It is whether the whole workflow can survive long-running tasks. I have stepped on this rake a few times, especially when I open the laptop in the morning, see that the agent ran all night, and still cannot tell which diff should be kept.

What changed

#

Early Vibe Coding solved the entry problem: describe the requirement clearly, put project rules into AGENTS.md / CLAUDE.md, and let tests, lint, and review catch the model output.

That setup is still useful, but it is closer to single-task engineering. Once a task gets longer, a few problems start showing up:

• Context keeps growing until the model loses the important part
• Repeated retries can push the fix further away from the real issue
• Without external references, strategy becomes guesswork
• After many rounds, it is hard to tell which changes should be kept
• User rejection, permission blocks, and empty output need explicit stop semantics

So I now prefer calling this layer Harness Engineering: put an engineering track around AI so tasks are executable, results are verifiable, and failures are recoverable. The name sounds a bit grand. In practice, it just means trusting “it will figure it out” a little less and adding a few guardrails.

flowchart LR
  A[Task scope] --> B[Context route]
  B --> C[Agent loop]
  C --> D[Verification gate]
  D --> E[Recovery / memory]
  D -->|failed| F[Patch harness]
  F --> C

The four things I manage first

#

The first thing is task boundaries.

Before a medium-sized task starts, I want at least done when, out of scope, the change surface, and the verification command. This does not need to be a long document. Five lines are often enough. The point is to let the executor know when to stop, instead of drifting into “while I am here” changes.

The second thing is context routing.

AGENTS.md should not become an encyclopedia. It works better as an index: project rules, entry points, verification commands, things that must not be touched, and where to read the next layer of docs. Long context should be opened on demand, not dumped into the session. When the context gets too full, the model behaves a bit like me with too many browser tabs open: it looks busy, but the focus is gone.

The third thing is the verification loop.

My default order is now:

1. Read: read README, AGENTS, older notes, and key implementation files
2. Search: use ace, rg, ast-grep, nmem, and Exa to find evidence
3. Change: apply a small patch and avoid drive-by refactors
4. Verify: run narrow checks first, then expand by risk
5. Record: write repeated lessons back into rules, tests, or memory

This order is boring in a good way. Reading and searching first reduce model guesswork. Narrow verification avoids one giant change where nobody knows which step broke.

The fourth thing is failure handling.

After a failure, I classify it first: stop, retry, patch the harness, or record it.

I used to treat many failures as “try again.” Now I am more careful: only retry failures that are actually retryable, and stop when the situation says stop. Letting an agent push forward from a wrong premise usually just creates more diff for a human to clean up.

Where external research fits

#

In this workflow, Exa or similar web search tools also have a clearer place.

I usually do not search for broad trends. I search for concrete engineering questions:

• What timeout should be used?
• Should this failure be retried?
• How should the default strategy be split?
• What boundaries do mainstream tools provide?
• What failure samples show up in real issues?

I still do not copy external answers directly. External material gives me a reference frame, and the final decision has to fit the current repo. Useful conclusions should land in specs, project rules, tests, or scripts. Otherwise I will search for the same thing again next time, which is a very small but reliable way to waste time.

Autoresearch and Ralph Loop

#

Autoresearch works best for long loops with a clear metric. Give the agent a goal, a guard, and a verification command first. Each round should allow only one rollback-friendly change. If it drifts, the damage is still contained.

I currently treat Ralph Loop as persistent single-owner execution. The same owner keeps driving the work. PRD and test spec come first, then the agent runs the long task. It cares more about preserving context, judgment, and verification clues than about adding more agents early. Fewer people in the loop can sometimes make ownership much clearer.

Both patterns share the same idea: define the track before letting the agent run. The track needs metrics, boundaries, verification, and rules for what to keep or discard.

Three steps worth copying first

#

If this needs to move into a team workflow, I would not start with platform work. Three steps are enough to copy tomorrow:

1. Write done when and out of scope for every medium-sized task
2. Ask the agent to list files, evidence, and the change surface before allowing edits
3. After one failure, patch tests, rules, or scripts before letting the agent continue

Once these three steps are in place, AI coding moves a bit from “it can produce output” toward “it can be shipped.” Autoresearch, Ralph Loop, team workers, and memory become easier to reason about after that.

I am not a native English speaker; this article was translated by AI.

What happened

#

On January 21, 2026, the target cluster started alerting.

Prometheus first showed a clock drift warning: Warning: Error fetching server time: Detected 38.116000175476074 seconds time difference between your browser and the server.

38 seconds does not sound like much. For Prometheus queries, it is enough to break charts. For something like Teleport, with security checks in the handshake path, it is even worse. target02 was already unreachable at that point: Teleport handshake failed, and SSH was gone with it.

Checking the network first

#

The first suspect was still the network. I tested reachability from target01 to the jump host:

# TCP scan, OK
for port in 22 80 8888; do nc -zv -w 2 100.64.0.5 $port; done

# UDP scan, looks OK at first glance, but no packets come back
nc -uvz -w 2 100.64.0.5 8888

TCP was fine. UDP looked suspicious. nc -u can easily make this look “connected” when nothing useful actually reached the other side, so I had to capture packets.

I started tcpdump on the jump host and sent one UDP packet from target01:

# jump-host
sudo tcpdump -i any udp port 8888 -n

# target01
nc -u 100.64.0.5 8888 <<< "test"

The jump host saw nothing. That was enough to confirm UDP was blocked somewhere in the isolated network.

I also checked port 8888 while I was there:

curl -v http://100.64.0.5:8888
# < Proxy-Agent: gost/2.12.0

That port was a gost proxy, mostly for TCP tunnels. UDP was not wired through it, so it was not going to keep NTP alive.

It came down to clock drift

#

After all that, the actual issue was still clock drift:

• target01 was about 38 seconds behind, enough to make Prometheus queries and charts go weird.
• target02 had drifted further, and Teleport rejected the handshake outright.

NTP could not get out of the isolated environment. chrony was running, but only in the sense that it was repeatedly failing. The normal path was dead, so I needed something reachable inside the environment that could still act as a time reference.

The jump host’s port 80 was reachable, and HTTP responses have a Date header.

Using HTTP Date to recover first

#

I stopped chrony first so it would not keep fighting the manual correction:

systemctl stop chronyd
systemctl disable chronyd

Then I pulled Date from the jump host’s HTTP header and fed it straight into date -s:

HTTP_DATE=$(curl -sI http://100.64.0.5 | grep -i "^Date:" | cut -d" " -f2-)
[ -n "$HTTP_DATE" ] && date -s "$HTTP_DATE"

It is crude. Precision is only seconds. But the goal at that moment was not elegance; it was getting Teleport back within handshake tolerance. After running it, the Prometheus clock drift warning cleared, and target02 became reachable again.

Yes, it is 2026 and I am still syncing clocks with curl + date -s. The last time I remember doing this was probably on OpenWrt.

Making it stick with crontab

#

After the quick recovery, I still needed to stop the nodes from drifting again. I put the same sync into crontab and ran it hourly:

(crontab -l 2>/dev/null; echo '0 * * * * HTTP_DATE=$(curl -sI http://100.64.0.5 | grep -i "^Date:" | cut -d" " -f2-) && [ -n "$HTTP_DATE" ] && date -s "$HTTP_DATE"') | crontab -

It is not pretty, but it works well enough in this environment. It only needs HTTP. As long as the jump host is reachable, the nodes stay within an acceptable time window.

Notes for next time

#

A few things worth writing down:

1. Do not assume NTP works in isolated networks. Once UDP 123 is blocked, chrony can look alive while syncing nothing.
2. Do not trust nc -u too much. UDP has no handshake, so a success message does not mean the remote side received anything.
3. Prometheus and Teleport are both sensitive to clock drift. A few dozen seconds can easily create symptoms that look like network failures.

If I deploy into a similar environment again, I will probably turn this HTTP Date sync into a small systemd timer. crontab is fine for firefighting; long term, it deserves something slightly less feral.

I am not a native English speaker; this article was translated by AI.

This round was about a KVM VM on a GPU host: pass through 8 H800 GPUs and 4 NVSwitches, then run single-node K8s, GPU Operator, and Prometheus inside the VM.

The process was not fancy, but the traps were very familiar: black screen, not enough PCI resources, DHCP lease mismatch, and a ServiceMonitor label that did not line up. I am writing the path down in the order I debugged it, mostly so I do not have to dig through logs from scratch next time.

Environment and goal

#

• Host: GPU node
• VM: ubuntu_gpu (KVM/libvirt, UEFI/OVMF)
• Goal: 8x H800 + 4x NVSwitch passthrough, GPU scheduling in single-node K8s, and DCGM metrics collected by Prometheus

Rough troubleshooting path

#

flowchart TD
  A[VM boot + GPU passthrough] --> B[Black screen / some GPUs fail to init]
  B --> C[Switch to non-Secure Boot OVMF]
  C --> D[Increase PCI hole64]
  D --> E[NUMA + vCPU pinning]
  E --> F[8 GPUs + NVSwitch OK]
  F --> G[K8s + GPU Operator + Prometheus]
  G --> H[DCGM metrics verified]

1. Black screen and PCI resource shortage

#

The first symptom looked scary: after GPU passthrough, VNC showed nothing, and the driver logs contained PCI I/O region invalid. For this kind of issue, I would not blame the driver too early. The 64-bit PCI hole on Q35 can be too small, especially with multiple GPUs plus NVSwitch.

I changed two things:

• switched OVMF to the non-secure version, to remove Secure Boot from the equation
• increased pci-hole64-size to 2048G

 readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.cc.fd



   value='-global'/>
   value='q35-pcihost.pci-hole64-size=2048G'/>

After that, GPU initialization and topology detection became normal. This option is easy to forget because most VMs never need it, but for multi-GPU passthrough it is one of the first places worth checking.

2. IP unreachable: just DHCP binding

#

There was also a very boring, very time-wasting network issue: the VM IP was unreachable. In the end, the NIC MAC had changed, so the DHCP binding no longer matched.

Restoring the old MAC brought the address back to 192.168.122.146.

# old MAC
52:54:00:a9:a2:11

Logs are not always generous for this kind of problem. I had to compare the libvirt XML, DHCP lease, and routes by hand.

3. Memory, NUMA, and vCPU pinning

#

The VM memory was set to 256GB:

 unit='KiB'>268435456
 unit='KiB'>268435456

vCPUs were split into two groups and pinned to matching NUMA nodes. GPUs were placed according to NUMA locality as well. This does not always decide whether the VM can boot, but it does matter once real workloads start running, so I prefer to get it right early.

4. Verifying GPU, K8s, and monitoring

#

On the GPU side, check the devices, topology, and NVLink state:

nvidia-smi -L
nvidia-smi topo -m
nvidia-smi nvlink -s

On the K8s side, check the node and pods:

kubectl get nodes -o wide
kubectl get pods -A

For Prometheus, query one DCGM metric directly:

curl http://127.0.0.1:9090/api/v1/query?query=DCGM_FI_DEV_SM_CLOCK

If data comes back, the DCGM Exporter to Prometheus path is basically working.

5. Key config snippets

#

containerd proxy

#

This machine cannot reach the public internet directly, so containerd needs a proxy for image pulls:

# /etc/systemd/system/containerd.service.d/http-proxy.conf
[Service]
Environment="HTTP_PROXY=http://100.64.0.5:8888"
Environment="HTTPS_PROXY=http://100.64.0.5:8888"
Environment="NO_PROXY=127.0.0.1,localhost,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,100.64.0.0/10"

GPU Operator and Prometheus

#

helm upgrade --install gpu-operator nvidia/gpu-operator \
  -n gpu-operator --create-namespace \
  --set driver.enabled=false --set dcgmExporter.enabled=true

helm upgrade --install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace

DCGM metrics into Prometheus

#

The easy-to-miss bit is the ServiceMonitor label. Prometheus usually only selects objects with the matching release label, so add it explicitly:

# ServiceMonitor must match Prometheus release label
kubectl -n gpu-operator label servicemonitor nvidia-dcgm-exporter release=kube-prometheus-stack --overwrite

Notes for next time

#

• For multi-GPU passthrough failures, check OVMF and pci-hole64-size early
• Turn off Secure Boot while validating, otherwise there are too many variables
• If DHCP suddenly behaves strangely, verify whether libvirt changed the MAC
• Having DCGM Exporter running is not enough; the Prometheus release label must match

Most of the issues here were the annoying kind where one small config mismatch makes the whole thing look broken. Once fixed, the 8 GPUs, NVSwitches, single-node K8s, and monitoring path all stayed stable.

I am not a native English speaker; this article was translated by AI.

This is a plain migration note: moving from an old DigitalOcean host (Ubuntu 20.04) to a new one (Debian 13 + XanMod).

The reason was also plain. My old $6/mo instance (1GB RAM / 20GB DISK / 1TB BW) was idle most of the time, and the blog had already moved to Cloudflare Pages, the reliable “cyber bodhisattva” in the corner. So I downgraded the Droplet to the $4/mo plan (512MB RAM / 10GB DISK / 500GB BW). Saving two dollars a month is not life-changing, but for this kind of tinkering, saving anything still feels like a win.

After the downgrade, this little VPS mainly runs a backup proxy and a half-asleep WeChat public account backend. 512MB RAM is not generous, so I also cleaned up the kernel, memory settings, Nginx port multiplexing, and certificate renewal while migrating.

1. Basic environment

#

• Source Host: DigitalOcean Ubuntu 20.04 (IP hidden)
• Target Host: DigitalOcean Debian 13 Trixie (IP hidden)
• Reserved IP: Attached to the new host for DNS resolution.
• Hostname / PTR: ferstar.org (automatically triggered by renaming the DigitalOcean Droplet).

2. Kernel and memory tuning (Kernel 6.18+)

#

2.1 Upgrade to XanMod Edge

#

For BBRv3 and newer scheduler features, I went straight to XanMod Edge:

wget -qO - https://dl.xanmod.org/archive.key | gpg --dearmor | tee /usr/share/keyrings/xanmod-archive-keyring.gpg > /dev/null
echo 'deb [signed-by=/usr/share/keyrings/xanmod-archive-keyring.gpg] http://deb.xanmod.org releases main' | tee /etc/apt/sources.list.d/xanmod-kernel.list
apt update && apt install linux-xanmod-edge-x64v3 -y

linux-xanmod-edge-x64v3 requires x86-64-v3 CPU support. If the VPS does not support it, use linux-xanmod-edge-x64v2 or linux-xanmod-edge-x64 instead. No need to be heroic here.

2.2 The small-memory set: zswap + MGLRU + KSM

#

With only 512MB RAM, there is not much headroom, so the kernel needs a little help. I enabled MGLRU, KSM, and the zswap shrinker. Some of these parameters are awkward to persist through sysctl, so I put them in crontab with @reboot. Crude, but easy to inspect later.

Persistence commands (crontab -e):

# Enable all MGLRU optimization tiers to significantly reduce OOM risk under low memory
@reboot echo 7 > /sys/kernel/mm/lru_gen/enabled

# Enable KSM memory page merging to reduce duplicate memory usage between Docker containers
@reboot echo 1 > /sys/kernel/mm/ksm/run

# Enable zswap shrinker to allow the kernel to balance data more aggressively between the compressed area and physical Swap
@reboot echo Y > /sys/module/zswap/parameters/shrinker_enabled

Other settings:

• zswap: Enabled memory compression cache, currently using the lzo algorithm.
• Swap: 1GB physical file as a fallback.

3. Port 443 multiplexing (SNI proxy)

#

Fewer exposed ports usually means less daily noise. Here I use the stream module from Nginx 1.29.4 (Mainline) to route traffic by SNI. Unknown traffic falls back to SSH.

3.1 Nginx global config (/etc/nginx/nginx.conf)

#

stream {
    map $ssl_preread_server_name $stream_map {
        api.ferstar.org api;   # WeChat Bot -> Forward to local 8444
        fm.ferstar.org  fm;    # File Server -> Forward to local 8445
        default         ssh;   # Non-SSL or unknown domain defaults to local 22 (SSH)
    }
    upstream ssh  { server 127.0.0.1:22; }
    upstream api  { server 127.0.0.1:8444; }
    upstream fm   { server 127.0.0.1:8445; }
    
    server {
        listen 443 reuseport;
        proxy_pass $stream_map;
        ssl_preread on;
    }
}

The benefits are straightforward:

• the firewall only needs 80/443 exposed, so the rules stay clean
• SSH is not directly exposed on port 22, which cuts down scan noise
• HTTPS, SSH, and proxy services can share one entry point, which is handy on awkward networks

3.2 Async IO tuning

#

File serving still runs on this box, so I enabled thread-pool async IO in the http block to keep large file reads and writes from blocking workers:

• aio threads;
• thread_pool default threads=32 max_queue=65536;
• directio 4m;

4. Firewall config (UFW)

#

Inbound 22 is closed and handled through the 443 stream fallback. The rules are intentionally boring:

ufw reset
ufw default deny incoming
ufw default allow outgoing
ufw allow 80/tcp
ufw allow 443/tcp
ufw allow 443/udp
ufw allow 18443:18445/udp  # For proxy services
ufw enable

5. Let’s Encrypt wildcard certificate

#

5.1 DNS validation (Cloudflare)

#

The wildcard certificate (*.ferstar.org) is renewed through the dns-cloudflare plugin.

• Credential File: /root/certbot-creds.ini (contains CF API Token).
• Plugin Installation: apt install python3-certbot-dns-cloudflare -y.

5.2 Post-renewal hook

#

The renewal config lives at /etc/letsencrypt/renewal/ferstar.org.conf. After a certificate update, reload Nginx and restart the containers that use the certificate:

post_hook = systemctl reload nginx && docker restart hysteria hysteria2 tuic-server

6. Application config templates

#

These templates are not trying to be a clever abstraction. They are here so the next migration has something concrete to compare against.

6.1 Hysteria v1

#

docker-compose.yml

services:
  hysteria:
    image: tobyxdd/hysteria:v1.3.5
    container_name: hysteria
    logging:
      driver: "json-file"
      options: {max-size: "10m", max-file: "3"}
    restart: unless-stopped
    command: ["-config", "/etc/config.json", "server"]
    volumes:
      - ./config.json:/etc/config.json
      - /etc/letsencrypt:/etc/letsencrypt:ro
    ports: ["18443:443/udp"]
    sysctls: {net.ipv4.tcp_congestion_control: bbr}

config.json

{
  "listen": ":443",
  "cert": "/etc/letsencrypt/live/DOMAIN/fullchain.pem",
  "key": "/etc/letsencrypt/live/DOMAIN/privkey.pem",
  "auth": { "mode": "passwords", "config": ["PASSWORD"] },
  "up_mbps": 1000,
  "down_mbps": 1000
}

6.2 Hysteria v2

#

docker-compose.yml

services:
  hysteria2:
    image: tobyxdd/hysteria:latest
    container_name: hysteria2
    restart: unless-stopped
    command: ["server", "-c", "/etc/hysteria/config.yaml"]
    logging:
      driver: "json-file"
      options: {max-size: "10m", max-file: "3"}
    volumes:
      - ./config.yaml:/etc/hysteria/config.yaml:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro
    ports: ["18445:443/udp"]
    sysctls: {net.ipv4.tcp_congestion_control: bbr}

config.yaml

listen: :443
tls:
  cert: /etc/letsencrypt/live/DOMAIN/fullchain.pem
  key: /etc/letsencrypt/live/DOMAIN/privkey.pem
auth:
  type: password
  password: "PASSWORD"
bandwidth:
  up: 1 gbps
  down: 1 gbps

6.3 TUIC v5

#

docker-compose.yml

services:
  tuic:
    image: ghcr.io/itsusinn/tuic-server:1.4.5
    container_name: tuic-server
    logging:
      driver: "json-file"
      options: {max-size: "10m", max-file: "3"}
    restart: always
    ports: ["18444:443/udp"]
    volumes:
      - ./config.json:/etc/tuic/config.json:ro
      - /etc/letsencrypt:/etc/letsencrypt:ro

config.json

{
    "server": "[::]:443",
    "users": { "UUID": "PASSWORD" },
    "certificate": "/etc/letsencrypt/live/DOMAIN/fullchain.pem",
    "private_key": "/etc/letsencrypt/live/DOMAIN/privkey.pem",
    "alpn": ["h3"],
    "udp_relay_ipv6": true,
    "zero_rtt_handshake": false,
    "dual_stack": true,
    "log_level": "warn"
}

6.4 Filebrowser

#

docker-compose.yml

services:
  filebrowser:
    image: filebrowser/filebrowser:latest
    container_name: filebrowser
    logging:
      driver: "json-file"
      options: {max-size: "10m", max-file: "3"}
    user: 0:0
    ports: ["127.0.0.1:1122:80"]
    volumes:
      - /root/fm/filebrowser/srv:/srv
      - /root/fm/filebrowser/database.db:/database.db
    command: ["--address", "0.0.0.0", "--port", "80", "--database", "/database.db", "--root", "/srv"]
    restart: unless-stopped

Closing notes

#

There is nothing deep about this migration. The main idea was to rethink what the old VPS still needed to do: let Cloudflare Pages handle the blog, keep only the small services on the VPS, expose fewer ports, and squeeze memory where it is safe to do so. 512MB is still tight, but for these lightweight jobs it is enough.

I am not a native English speaker; this article was translated by AI.

This post is basically a renovation note for the blog. A few hours earlier, the site was still Chinese-only: static/_redirects had a pile of old rules from several migrations, and many posts with decent traffic still had no description.

I first used Google Search Console (GSC) to find where the pain was, then cleaned up redirects, and finally added the bilingual structure and writing rules. In this round, AI mostly handled repetitive work and cross-checking. The calls on what to keep and what to delete were still mine. Fair enough, since the model is not going to take the blame if I break the site.

GSC poured some cold water on me first

#

The data was simple, but not pleasant:

• US market: 13,000+ impressions, only 1.46% CTR
• Chinese market: CTR stayed above 6%

I used to think technical posts would find readers as long as the content was solid. In reality, when a fully Chinese page shows up in search results, many overseas readers will just skip it. Going bilingual is not about looking international; it is about letting people who already found the post actually read it.

Redirects look small, but they can quietly kill indexed URLs

#

After moving the blog from Farbox to Bitcron and then to Hugo, static/_redirects was not pretty. My first lazy idea was to replace spaces with -, collapse repeated dashes, and call it done.

Hugo corrected me pretty quickly. Filenames with & can generate slugs like google-search-tips--tricks, with two dashes in the middle. If a script helpfully collapses -- into -, already-indexed Google URLs go straight to 404.

I ended up using a dumber but safer approach:

• Keep both the single-dash normalized path and the original multi-dash variants (--, ---, etc.), all pointing exactly to the new slug
• Replace fuzzy wildcards (like /post/*) with explicit mappings, so debugging later does not become guesswork

This pass cleaned up 300+ legacy redirects into something I am comfortable keeping long-term. The file is still not short, but at least every rule has a reason to exist.

Bilingual does not mean appending a short English summary

#

I did not want a half-finished setup with the Chinese post plus a tiny English abstract, so I set a few rules for translations:

1. Code parity: shell commands, Java hooks, and kernel configs must match character-for-character
2. Diagram parity: Mermaid diagrams must render on the English pages too
3. Do not sand off the details: traps, trade-offs, and performance numbers should stay, not turn into lukewarm prose

The workflow was: AI drafted, then AI cross-checked the Chinese and English versions; I edited paragraph by paragraph and cut or rewrote anything that sounded too templated. The first batch covered 10+ high-traffic posts. The older debt can be paid down slowly.

Put the rules in AGENTS.md, because memory is unreliable

#

After the cleanup, my biggest worry was not that the code would break. It was that I would forget why I made these choices a few months later. So I added AGENTS.md to the repo and wrote down the ground rules:

• Bilingual alignment: after a Chinese update, the English version must be updated with high fidelity
• SEO description: descriptions follow [Pain Point] + [Solution] + [Result]
• Directory convention: keep content/ and content.en/ separate, and handle links as a multilingual site
• No redirect rollback: static/_redirects should use explicit rules only, no lazy /post/*

It is not complicated, but it helps. Next time, whether I come back myself or ask an agent to continue, there is a runnable set of rules instead of “probably close enough”.

Closing

#

I write blog posts to preserve how problems were solved. SEO and bilingual pages just smooth the path to the door, making those notes easier to find and easier to finish reading.

For me, AI is more like a power tool: checklists, bulk edits, cross-checks — it is fast at those. But the final trade-offs, tone, and responsibility are still mine. That is probably for the best. If something breaks later, I know exactly who to blame.

I am not a native English speaker; this article was translated by AI.

There are several DGX H800 nodes in a private network. They are not public-facing, but vendors occasionally need remote access for troubleshooting. The requirements sound simple, until network paths and auditing get involved:

1. Vendors need to connect and do their work.
2. Sessions must be fully recorded, but the audit entry points should not be exposed to them.
3. Our own team still needs a direct path, without going through the bastion every single time.

The final setup is Teleport for bastion access and auditing, plus Tailscale for internal connectivity and an ops bypass path. These are the bits that tripped me up, recorded here so I do not step on the same rake again.

The rough shape

#

flowchart LR
  subgraph External[Public Users]
    Vendor[Vendor
Web/SSH]
    AppUser[App User
Web]
  end
  subgraph Ops[Internal Ops]
    Admin[Admin
SSH Direct]
  end
  subgraph Edge[Public Edge]
    Nginx[Nginx 443]
    Teleport[Teleport Proxy 3080]
    FW[DOCKER-USER Firewall]
    Logs[(Session Recording/Logs)]
    ExitNode[Tailscale Exit Node]
  end
  subgraph Intranet[GPU Cluster]
    GPU[GPU Nodes]
    AppUI[App UI 32000]
  end

  Vendor -- HTTPS/SSH --> Nginx --> FW --> Teleport --> GPU
  AppUser -- HTTPS --> Nginx --> FW --> Teleport --> AppUI
  Admin -- SSH 22 --> TSUser[Internal Tailscale Node]
  TSUser -- Tailscale Tunnel --> ExitNode --> GPU
  Teleport --> Logs

First, collapse the ports

#

Teleport uses a range of ports by default, roughly 3022-3080. That is fine in a small test, but once Nginx, Docker, and cloud firewall rules join the party, it becomes annoying fast.

Multiplexing lets HTTPS, SSH, and tunnel traffic share port 3080:

# teleport.yaml
auth_service:
  proxy_listener_mode: multiplex

It uses ALPN to distinguish traffic types. The practical win is boring but useful: only one public-facing port to reason about.

Installing the Agent in an isolated network: put the proxy in the repo file

#

The GPU nodes cannot reach the internet directly, so installing the Teleport Agent needs a proxy. I first configured the system proxy and still got certificate errors from yum/dnf. The missing piece was setting the proxy explicitly in the repo file:

# /etc/yum.repos.d/teleport.repo
[teleport]
proxy=http://[Bastion_VIP]:8888

This is not a glamorous problem, but it eats time. The error looks like TLS trouble, while the real issue is that the package manager is not using the proxy path you think it is using.

Web Terminal disconnects immediately? Check Nginx WebSocket handling

#

When Nginx reverse-proxied the Teleport Web Terminal, the page loaded but the terminal disconnected as soon as it opened. These settings ended up being the minimum stable set:

proxy_pass https://127.0.0.1:3080;  # MUST use https
proxy_ssl_verify off;                # For self-signed certs
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";

The https in proxy_pass is easy to overlook. The UI loading successfully does not mean the terminal channel is healthy.

Docker mapped ports may skip the INPUT chain you trusted

#

This was the scary one. I had written restrictions in the INPUT chain and assumed 3080 was internal-only. Docker-published ports enter through DOCKER-USER, so those old rules were mostly useless. Port 3080 was still exposed to the public internet.

Note: -F below flushes the DOCKER-USER chain. Do not blindly paste this into production. Back it up first, or use -I to insert rules in the right place.

My fixed version looked like this:

iptables -F DOCKER-USER
# Allow only localhost and Tailscale network
iptables -A DOCKER-USER -s 127.0.0.1 -p tcp --dport 3080 -j ACCEPT
iptables -A DOCKER-USER -s 100.64.0.0/10 -p tcp --dport 3080 -j ACCEPT
iptables -A DOCKER-USER -p tcp --dport 3080 -j DROP

Do not point Nginx at the VPN IP unless you enjoy circular failures

#

At first I set the Nginx upstream to the Tailscale VPN IP. It looked tidy, but it created a circular dependency: if the VPN blipped, the management page went down too.

Changing it to 127.0.0.1 fixed that. Even if Tailscale is down, the public management entry still works, which is exactly what you want when you need to repair Tailscale.

Permission split

#

The RBAC trick is to remove audit from the vendor’s restricted-dev role, so they cannot see recordings or log pages.

One caveat: removing audit only changes visibility of recordings/logs. It does not guarantee that every recording banner disappears. Teleport behavior depends on version and configuration, so test this yourself before relying on the word “silent”.

Two extra constraints are worth adding:

1. Vendors access only through Teleport Web/SSH; they do not get Tailscale.
2. Vendor roles restrict login users and visible nodes, with labels for isolation.

# Example: node labels (vendor name redacted)
labels:
  vendor: vendor-x
  env: prod

# Example: role constraints (vendor login + visible nodes)
kind: role
spec:
  allow:
    logins: ["vendor-user"]
    node_labels:
      vendor: "vendor-x"
  deny:
    logins: ["root"]

Core config snippets

#

docker-compose.yml

services:
  teleport:
    ports:
      - '3080:3080'
      - '127.0.0.1:3025:3025'  # Management API local only

teleport.yaml

proxy_service:
  public_addr: [teleport.example.com:443, 100.64.0.x:3080]

App Access came later

#

Expose apps (example)

app_service:
  enabled: "yes"
  apps:
  - name: app-ui
    uri: http://10.120.0.0:32000/
    public_addr: app-ui.example.com

Proxy environment causing 503

If the node has HTTP_PROXY, Teleport may try to reach internal apps through that proxy and return 503. Adding NO_PROXY in systemd is safer:

# /etc/systemd/system/teleport.service
Environment="NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,100.64.0.0/10"

Nginx Host passthrough header

For multi-subdomain access, do not forget the Host header:

proxy_set_header Host $host;

With this setup, vendors work through the Web UI, internal ops connect directly over Tailscale, session recordings are kept, and the GPU cluster does not need to sit on the public internet.

Not elegant, but steady enough. For this kind of temporary-but-sensitive remote support, steady matters more than pretty.

NOTE: I am not responsible for any expired content.
Created at: 2026-01-03T03:58:20+08:00
Updated at: 2026-01-03T05:55:29+08:00
Origin issue: https://github.com/ferstar/blog/issues/95

I am not a native English speaker; this article was translated by AI.

Based on the Vibe Coding Skeleton engineering practice. Note: For specific implementation details, refer to CLAUDE.md, .pre-commit-config.yaml, ast-grep/, sgconfig.yml, and justfile in this repository.

TL;DR: It is not about “letting AI write code,” but “using engineering to ensure AI writes the right code.”

I. A Hard Fact

#

1.1 The Data Shock

#

Consider a real project (anonymized), iterated over ~8 months with 1k+ commits.

Aside from requirement decomposition, business analysis, architectural adjustments, and final acceptance— the vast majority of the code was generated by AI, while humans focused on decision-making and verification.

Today, we aren’t debating if AI can write code. We are discussing: How to make AI write the code correctly.

Project Overview:

Commit Type Distribution:

Development Phase Evolution:

1.2 The Role Shift

#

Traditional Perception vs. Reality:

The New Workflow:

flowchart TB
    subgraph Human's Work
        A[Decomposition] --> B[Business Analysis]
        B --> C[Architectural Decisions]
        C --> D[Context Preparation]
        D --> E[Verification & Review]
    end

    subgraph AI's Work
        F[Controller]
        G[Service]
        H[Schema]
        I[Test]
        J[Migration]
        K[Config]
        L[Commit Message]
    end

    D --> F & G & H & I & J & K & L
    F & G & H & I & J & K & L --> E

Role Definitions:

II. Why It Works

#

2.1 Context is King

#

The Core Formula:

AI Coding Level = Your Context Provision Level

The Complete Version:

AI Delivery Quality = Context Quality × Automated Verification × Task Alignment

Without verification, output is just a draft. Without task alignment, verification costs explode.

The Three-Layer Context Model:

graph TB
    subgraph Context Pyramid
        P["Project Context
Tech Stack / Directory Structure / Naming Specs / Prohibitions
CLAUDE.md"]
        T["Task Context
Requirements / Boundaries / Acceptance Criteria / Domain Terms
Prompt"]
        S["Session Context
Chat History / Intermediate Results
Auto-accumulated"]
    end

    P --> T --> S

    style P fill:#4ecdc4,stroke:#333,stroke-width:2px
    style T fill:#f38181
    style S fill:#fce38a

Investment Return:

ROI: Project >> Task >> Session

Project: 8h investment → Benefits all tasks (Compound Interest)
Task: 5-10min → Current task only
Session: 0 → Current conversation only

Conclusion: Project Context is an “invest once, benefit continuously” asset. Spending time on a perfect CLAUDE.md is the highest ROI move.

2.2 The Bottleneck Shift

#

My Time Allocation:

pie title Time Allocation (Empirical)
    "Requirement/Arch Design" : 20
    "Preparing Context" : 15
    "Waiting for AI" : 5
    "Verification + Fixes" : 40
    "Paying Technical Debt" : 20

Key Insight: 40% of time is spent “verifying,” not “writing.” The bottleneck has shifted from generation to validation.

III. How It’s Done

#

3.1 Project-level Context: CLAUDE.md

#

This project uses a ~300-line CLAUDE.md as an “AI Onboarding Manual.”

Note: Different AI tools prefer different filenames (e.g., CLAUDE.md, .cursorrules, AGENTS.md). The essence is the same: translate your project rules into machine-readable specs.

Core Structure:

# CLAUDE.md

## Project Overview        # What the project is for
## Technology Stack        # Litestar + PostgreSQL + Redis + OpenDAL + Celery
## Basic Rules             # Activate venv first, manage deps with uv
## Development Commands    # just lint, just test
## Architecture Overview   # DDD layers, business grouping under src/domain/
## Coding Style            # ruff formatting, 150 char line width
## Testing Guidelines      # pytest (fail-fast, warnings-as-errors)
## CLI Commands            # litestar database upgrade, litestar run

Case Study: How ast-grep Intercepts Bad AI Habits

#

Requirement: Write a function to process a temporary PDF file.

Typical AI Output:

import tempfile
from pathlib import Path

def process_pdf(content: bytes) -> str:
    # Using delete=False for subsequent processing
    with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as tmp:
        tmp_path = Path(tmp.name)
        tmp.write(content)

    try:
        result = parse_pdf(tmp_path)
        return result
    finally:
        tmp_path.unlink(missing_ok=True)  # Manual cleanup

The Issues:

• delete=False forces manual cleanup logic, increasing maintenance cost.
• Cleanup logic is easily missed or written incorrectly (e.g., forgetting try/finally), leading to file leaks.

ast-grep Interception:

$ git commit -m "feat: add pdf processor"
error[no-tempfile-delete-false]: NamedTemporaryFile with delete=False is forbidden
  --> src/utils/pdf.py:6:10
   |
 6 |     with tempfile.NamedTemporaryFile(suffix=".pdf", delete=False) as tmp:
   |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Fixed Version:

def process_pdf(content: bytes) -> str:
    with tempfile.NamedTemporaryFile(suffix=".pdf") as tmp:  # default delete=True
        tmp_path = Path(tmp.name)
        tmp.write(content)
        tmp.flush()  # Ensure data is on disk

        result = parse_pdf(tmp_path)
        return result
    # Automatic cleanup on block exit, zero leak risk.

The Point: Don’t rely on AI “voluntarily obeying” your rules. Use ast-grep to force compliance.

3.2 Automated Validation Workflow

#

AI-generated code must pass through a quality pipeline. If the gate fails, the code doesn’t enter the main branch.

The Pipeline:

flowchart LR
    A[Code Change] --> B[pre-commit: ruff format]
    B --> C[pre-commit: ruff check]
    C --> D[pre-commit: ast-grep scan]
    D --> E[git commit]
    E --> F[commit-msg: Conventional Commits]
    F --> G[git push]
    G --> H[pre-push: pytest]

    style D fill:#ff6b6b,stroke:#333,stroke-width:2px
    style H fill:#4ecdc4,stroke:#333,stroke-width:2px

ast-grep Rules (Selected):

IV. Boundaries and Stop-Loss

#

4.1 Sweet Spot vs. Quagmire

#

It’s not about whether AI can write it—AI can write anything. It’s about whether the verification cost is acceptable.

flowchart TB
    subgraph High Verification Cost
        subgraph Q1["❌ Quagmire"]
            Q1A["Complex Business"]
            Q1B["Performance Optimization"]
            Q1C["Algorithm Design"]
        end
    end
    subgraph Low Verification Cost
        subgraph Q3["✅ Sweet Spot"]
            Q3A["CRUD APIs"]
            Q3B["Unit Tests"]
            Q3C["Config Files"]
            Q3D["DB Migrations"]
        end
    end

    style Q3 fill:#4ecdc4,stroke:#333
    style Q1 fill:#ff6b6b,stroke:#333

Case Study: The Correct Way to Optimize Performance

#

Scenario: N+1 Query Optimization

Problem: A list API response time spikes from 100ms to several seconds.

Wrong Way — Letting AI “guess” without data:

Me: “This API is slow, optimize it.” AI: “Add caching / Add indexes / Preload…” (Often ineffective shots in the dark).

Correct Way — Feed the AI “Runtime Facts”:

1. Human: Execute EXPLAIN (ANALYZE, BUFFERS) in test env.
2. AI (Analytical): Locate bottleneck based on the plan (e.g., duplicated scans in a subquery).
3. Human: Confirm the rewrite strategy + required migration.
4. AI (Executive): Generate code + migration scripts + tests.
5. Human: Run just test and compare load test metrics.

4.2 The Abandonment Threshold

#

V. The Elephant in the Room

#

5.1 Is your concern valid?

#

“If AI writes the code, what am I?”

• Is it just “KPI Marketing”? No. Prompt engineering, requirement decomposition, and verification are programming, just in a different language.
• How do Juniors learn? By reading high-quality open-source code and reviewing AI mistakes, rather than mindlessly typing CRUD.
• Reviewing AI is tiring. Yes. It takes 40% of the time because AI code “looks” correct but may hide subtle bugs.

5.2 A Reality Check

#

You aren’t the AI’s secretary; the AI is your intern.

• The intern can work, but you must tell them what to do.
• The intern makes mistakes; you must review them.
• If the intern does well, the credit is yours.

The difference? This intern doesn’t sleep, doesn’t complain, and you can manage ten of them at once.

Has the barrier to entry lowered? No. It has shifted—from “knowing how to type code” to “knowing how to understand systems + verify them.”

In the AI era, the requirements for engineers are actually HIGHER, not lower.

VI. Action List (TL;DR)

#

• Create CLAUDE.md with your tech stack and rules.
• Solidify a zero-cost validation loop (e.g., just lint, just test).
• Configure pre-commit hooks (ruff + ast-grep).
• Identify “Sweet Spot” tasks for AI delegation.

Tools change, but our value in solving problems remains. Be the driver of AI, not the passenger.

Created: 2025-12-31 *Updated: 2026-01-03"

I am not a native English speaker; this article was translated by AI.

The Problem

#

During a cross-regional GPU cluster deployment, two layers of tunnels were established between Site-A and Site-B: WireGuard + IPsec. Due to security compliance, ICMP was disabled site-wide—ping and traceroute were useless.

Then came a strange symptom:

• A → B: 50Mbps, full speed, perfect.
• B → A: SSH connects, but scp of large files drops to exactly 30KB/s.

This is a classic “one-way street” symptom. The link isn’t broken, but a protocol-level ceiling is being hit.

Troubleshooting: How to play without ICMP?

#

Normally, you’d use ping -s -M do to detect Path MTU. But with ICMP blocked, there’s no feedback for “Packet Too Big.”

I had to rely on TCP behavioral fingerprints. I conducted experiments with iperf3:

1) Default TCP Test

#

Bandwidth locked at ~0.4Mbps with massive retransmissions (Retr).

2) UDP Control Test

#

20Mbps worked normally with controllable loss. The link itself was fine.

3) TCP MSS Stepping (The Breakthrough)

#

iperf3 -c  -t 10 -M 1400  # Bandwidth collapses, heavy retrans
iperf3 -c  -t 10 -M 1350  # Instant recovery to ~49Mbps

Case Closed: A classic PMTUD Black Hole triggered in a “blind” network environment. This is a typical “expert’s blind spot”: you have tcpdump and wireshark running, looking for complex protocol issues, while ignoring the most fundamental link parameters.

The Theory: Who is “Murdering” the Packets?

#

MTU/MSS Breakdown

#

TCP MSS is the maximum payload of a single TCP segment, limited by the Path MTU.

• IPv4 approx: MSS ≈ MTU - 20(IP) - 20(TCP) - Options
• Standard Ethernet MTU 1500 results in ~1460 MSS.

In multi-layer tunnels (WireGuard + IPsec), the encapsulation headers continuously “eat” the available MTU. The default large MSS becomes an “overloaded packet.”

Why the Black Hole?

#

1. The sender sends a large packet based on default MSS.
2. An intermediate node finds the packet too large and tries to return an ICMP (Fragmentation Needed).
3. But ICMP is blocked, so this signal is lost.
4. The sender never receives the notice, assumes the packet was dropped due to congestion, and retransmits the same oversized packet indefinitely.
5. Throughput is killed by retransmissions.

Solutions

#

A. The Cure (Ideal)

#

Allow specifically required ICMP types:

• IPv4: ICMP Type 3 Code 4 (Fragmentation Needed)
• IPv6: ICMPv6 Type 2 (Packet Too Big)

B. Workaround 1: Lower Interface MTU

#

Lower the MTU on the WireGuard or IPsec interfaces to stop the bleeding.

C. Workaround 2: MSS Clamping (My Implementation)

#

Since ICMP is unreliable, force the MSS rewrite on the transit gateway:

#!/bin/bash
set -euo pipefail

# --- Config ---
PHY_IF="ens3"
REMOTE_NET="10.0.0.0/24"

# 1. Physical Layer Optimization (increases CPU overhead, enable as needed)
ip link set dev "$PHY_IF" mtu 1400
ethtool -K "$PHY_IF" tso off gso off gro off

# 2. MSS Clamping
# For Forwarded traffic: 1280 (Conservative)
iptables -t mangle -C FORWARD -p tcp -d "$REMOTE_NET" --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1280 2>/dev/null || \
iptables -t mangle -I FORWARD -p tcp -d "$REMOTE_NET" --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1280

# For Local traffic: 1350 (Aggressive but functional in this case)
iptables -t mangle -C OUTPUT -p tcp -d "$REMOTE_NET" --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1350 2>/dev/null || \
iptables -t mangle -I OUTPUT -p tcp -d "$REMOTE_NET" --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 1350

Summary

#

1. MTU is the Foundation: In VPN/Overlay scenarios, the default 1500 is a trap.
2. Asymmetric Thinking: One-way speed limits are often MTU issues, not bandwidth caps.
3. Better Safe than Sorry: Set a conservative MSS. Sacrificing a bit of payload efficiency is worth a 100x gain in stability.

I am not a native English speaker; this article was translated by AI.

Background

#

When deploying a VPN relay service on a 50Mbps cloud host, I encountered the classic Bufferbloat problem: whenever a large download saturated the bandwidth, SSH latency spiked from 50ms to over 200ms, severely degrading the interactive experience.

The traditional fq + bbr combo has limited effectiveness on narrow bandwidth links because fq only ensures fair queuing—it doesn’t actively control queuing delay.

The Solution: DualPI2 + BBR

#

Linux 6.17 introduced the sch_dualpi2 scheduler, designed specifically for low latency:

Configuration Steps

#

1. Requirements

#

Linux Kernel 6.17 or higher (includes the sch_dualpi2 module).

2. sysctl Configuration

#

Edit /etc/sysctl.conf:

# --- Core Scheduling Scheme ---
net.core.default_qdisc = dualpi2
net.ipv4.tcp_congestion_control = bbr

# --- Latency Optimization ---
net.ipv4.tcp_ecn = 1              # Essential for dualpi2
net.ipv4.tcp_fastopen = 3
net.ipv4.tcp_recovery = 1         # RACK for fast reordering fixes
net.ipv4.tcp_notsent_lowat = 8192 # Lower BBR kernel buffering

# --- Buffer Settings (Preventing Bloat on 50Mbps) ---
net.core.rmem_max = 4194304
net.core.wmem_max = 4194304
net.ipv4.tcp_rmem = 4096 16384 4194304
net.ipv4.tcp_wmem = 4096 16384 4194304

# --- Stability Optimization ---
net.core.somaxconn = 2048
net.ipv4.tcp_max_syn_backlog = 2048
net.mptcp.enabled = 1

3. Auto-load Module

#

echo "sch_dualpi2" | sudo tee /etc/modules-load.d/dualpi2.conf

4. Persistence (udev Rule)

#

Create /etc/udev/rules.d/99-ens3-dualpi2.rules:

ACTION=="add", SUBSYSTEM=="net", NAME=="ens3", RUN+="/sbin/tc qdisc replace dev ens3 root dualpi2"

Replace ens3 with your network interface name.

5. Apply Configuration

#

sudo sysctl -p
sudo tc qdisc replace dev ens3 root dualpi2

Verification Methods

#

Check DualPI2 Status

#

tc -s qdisc show dev ens3
# Observe the target value, default is 5ms

Stress Testing (Solution for ICMP-Disabled Servers)

#

Traditional ping fails on servers with ICMP blocked. Use kernel RTT statistics instead:

# Client: Saturate the bandwidth
iperf3 -c  -t 60

# Server: Observe Kernel RTT
ss -tni
# Focus on rtt:X/Y — X is mean RTT, Y is jitter (mdev)

Results

#

Under full 50Mbps load:

SSH latency remained only 6.4ms above the physical limit during full downloads, with 2ms jitter. Bufferbloat is completely eliminated.

I am not a native English speaker; this article was translated by AI.

Android has matured significantly, but for power users, rooting remains a prerequisite for total control over the hardware and software experience. Here is why rooting still matters in 2025.

1. Bleeding Edge Kernel Features

#

Manufacturers often use older, stable kernels. Rooting allows me to leverage modern features:

• BBRv2 Congestion Control: Significantly improves speeds on cellular networks (noticeable on China Unicom 4G/5G).
• F2FS Optimization: Paired with UFS 3.1, it offers noticeable gains in random I/O performance.
• Zstd Compressed ZRAM: Offers ~20% better compression than lz4.
• Custom CPU Schedulers: Schedulers like EAS (Energy Aware Scheduling) can reduce daily power consumption by ~15%.

I’ve compiled custom kernels for xaga and RMX3888 devices, enabling many features disabled by OEMs. KernelSU has proven to be more stable than Magisk for these scenarios.

Tools: Franco Kernel Manager / UKMM

2. Killing “Cloud Control” and OEM Bloat

#

Manufacturers often push unwanted “optimizations” or ads silently via cloud control. My approach:

# Freeze system updates
pm disable com.xxx.ota

# Block cloud control domains via hosts
echo "127.0.0.1 cloud.manufacturer.com" >> /system/etc/hosts

Essential Modules: CorePatch (bypass signature verification), Shamiko (hide root).

3. Bypassing Ecosystem Locks

#

Some banking apps or region-locked games restrict access based on device models. Technically, this involves hooking the android.os.Build class:

XposedHelpers.setStaticObjectField(Build.class, "MODEL", "SM-G9980");
XposedHelpers.setStaticObjectField(Build.class, "MANUFACTURER", "samsung");

Modules: Device Faker / Hide My Applist.

4. Automated Bookkeeping (The LSPosed Way)

#

Accessibility services and OCR are too unstable for automated bookkeeping. LSPosed hooks are the gold standard—hooking directly into the payment success callback:

// Hooking WeChat Pay success callback
findAndHookMethod("com.tencent.mm.plugin.wallet...", "onPaySuccess", 
    new XC_MethodHook() {
        protected void afterHookedMethod(MethodHookParam param) {
            // Extract amount and merchant info directly into the database
            saveTransaction(param);
        }
    });

It achieves 100% accuracy and never gets killed by the system because it runs within the app process itself.

Reference: AutoAccounting (Open Source).

5. System-Wide Ad Blocking

#

Beyond DNS-based blocking, root allows for:

• AdAway: The classic hosts-based blocker.
• ReVanced: The essential YouTube and streaming app enhancer.
• iptables Rules: Forcefully rejecting ad-domain traffic at the kernel level.

# Reject ad domains via iptables
iptables -A OUTPUT -d ad.xxx.com -j REJECT

6. Underclocking and Undervolting for Longevity

#

Every SoC is different. On my Snapdragon 8 Gen2:

Method: Modify the kernel device tree voltage table or use Franco Kernel Manager for real-time tuning.

7. Bypassing Thermal Throttling & Charging Mods

#

Thermal Control

#

OEMs are often too conservative, throttling at 45°C. I increase the threshold to 55°C:

# Modify thermal trip points
echo 55000 > /sys/class/thermal/thermal_zone0/trip_point_0_temp

Smart Charging

#

For devices always plugged in, I limit the charging current and capacity:

# Limit charging current to 2A
echo 2000000 > /sys/class/power_supply/battery/constant_charge_current_max

# Stop charging at 80%
echo 80 > /sys/class/power_supply/battery/charge_control_limit

Module: ACC (Advanced Charging Controller).

8. Termux: The Full Linux Experience

#

Without root, Termux is isolated. With root, it becomes a complete mobile server:

# Install a full Debian distro
proot-distro install debian

# Network packet analysis
tcpdump -i wlan0 -w capture.pcap

# Flash custom kernels directly
dd if=/sdcard/custom_boot.img of=/dev/block/by-name/boot

Conclusion

#

In 2025, rooting is still a necessity for me. It’s about performance extraction, privacy protection, and functional expansion. While there are risks like warranty loss or app detection, the journey of tuning the system is where the learning and fun reside.

My strategy: Keep the primary device conservative, but never stop tinkering with the secondary ones.

I am not a native English speaker; this article was translated by AI.

As a long-time Linux enthusiast, I usually install a Linux distro as soon as I get a new laptop. That makes the preinstalled Windows almost useless, but I still couldn’t delete it — I needed it to run vendor BIOS updaters. Keeping a ~100GB Windows partition just for the occasional BIOS update is pretty annoying. Sure, something like Windows To Go exists, but that doesn’t really change the situation: you’re still relying on Windows.

Right before the end of 2024, I came across someone mentioning a pure-Linux BIOS update workflow on Twitter: https://x.com/felixonmars/status/1876646199207604351

It was only a few lines, but it lit the tinkering fire in my head: this might work — let’s do it.

All my laptops are Lenovo ThinkBooks. I started with one machine at home (A). Specs:

OS: Ubuntu oracular 24.10 x86_64
Host: 21D0 (ThinkBook 14 G4+ ARA)
Kernel: Linux 6.12.9-bigv
Uptime: 1 day, 53 mins
Packages: 2800 (dpkg), 7 (flatpak)
Shell: zsh 5.9
Display (AUOC391): 2880x1800 @ 90 Hz in 14"
DE: GNOME
WM: Mutter (X11)
WM Theme: Yaru
Theme: Yaru [GTK2/3/4]
Icons: Yaru [GTK2/3/4]
Font: Cantarell (11pt) [GTK2/3/4]
Cursor: DMZ-White (48px)
Terminal: tmux 3.4
CPU: AMD Ryzen 7 6800H (16) @ 4.79 GHz
GPU: AMD Radeon 680M [Integrated]
Memory: 12.95 GiB / 29.55 GiB (44%)
Swap: 5.25 MiB / 32.00 GiB (0%)
Disk (/): 312.72 GiB / 414.32 GiB (75%) - btrfs
Local IP (wlan0): 192.168.31.157/24
Battery (AP16L5J): 77% [AC Connected]
Locale: en_US.UTF-8

1. Download Framework’s UEFI Shell flashing tool. We only need H2OFFT-Sx64.efi. Save it to /boot/efi: https://downloads.frame.work/bios/Framework_Laptop_13_13th_Gen_Intel_Core_BIOS__3.05_EFI.zip
2. Download UEFI Shell. Save shellx64.efi to /boot/efi: https://github.com/pbatard/UEFI-Shell/releases/tag/24H2
3. Download the BIOS updater for your exact laptop model from Lenovo’s driver page. It’s usually named something like j6cn50ww.exe.
4. Try extracting the exe with 7z: 7z e j6cn50ww.exe. You may see output like: Comments: This installation was built with Inno Setup.
5. Use innoextract to unpack it: innoextract -e j6cn50ww.exe. You’ll get a new exe — mine was J6CN50WW.exe.
6. Extract again with 7z: 7z e J6CN50WW.exe. This time you should get the actual BIOS firmware, WinJ6CN50WW.fd.
7. Save WinJ6CN50WW.fd to /boot/efi.
8. Add a UEFI Shell boot entry. Edit /etc/grub.d/40_custom, paste the following, then run sudo update-grub:

   menuentry "UEFI Shell" {
       insmod part_gpt
       insmod chain
       set root='(hd0,gpt1)'
       chainloader /shellx64.efi
   }

9. Reboot into UEFI Shell. Type FS0: and press Enter to switch to the EFI partition, then run H2OFFT-Sx64.efi WinJ6CN50WW.fd and press Enter.
10. You should see the familiar BIOS flashing UI. Wait for it to complete.

Finally, I can throw the bundled Windows partition into the trash ✌️

Note: For some newer Lenovo models, 7z may be able to extract the BIOS firmware directly — usually named xxx.bin — so innoextract is not required. Also, 7z isn’t mandatory either. You can use any tool that can unpack an exe (for example bsdtar).

NOTE: I am not responsible for any expired content.
Created at: 2025-01-12T16:58:10+08:00
Updated at: 2025-01-12T17:04:21+08:00
Origin issue: https://github.com/ferstar/blog/issues/83

I am not a native English speaker; this article was translated by AI.

After switching to Linux, I’ve been missing the smooth three-finger drag experience from macOS. Given that recent Windows laptops have significantly improved touchpad size and responsiveness, I decided it was time to tackle touchpad gestures on Linux.

Research and Selection

#

There are basically two approaches to implementation:

1. Parsing libinput debug-events output and using tools like xdotool to send keystrokes or mouse clicks.
2. Calling the libinput API directly, which offers the best performance.

Implementation

#

I chose to fork and merge two Rust projects to create a unified solution. One project handles general gestures, and the other focuses on the three-finger drag effect at the API level.

Architecture

#

graph TD
    A[Touchpad Event] --> B{libinput}
    B --> C[ferstar/gestures 
Rust Engine]
    C --> D{Display Server}
    D -- X11 --> E[libxdo API]
    D -- Wayland --> F[ydotool daemon]
    E --> G[Smooth Drag / Key Stroke]
    F --> G
    
    style C fill:#f96,stroke:#333,stroke-width:2px
    style G fill:#4ecdc4,stroke:#333,stroke-width:2px

The project is now at v0.8.1, with major improvements including:

• Dual Platform Support: Works on both X11 and Wayland (auto-detection).
• Performance Optimization:
  • Direct libxdo API calls for X11 (minimum latency).
  • Optimized ydotool integration for Wayland with 60 FPS throttling.
  • Thread pooling to prevent PID exhaustion.
  • Regex and config caching for speed.

Performance Metrics

#

1. Low CPU Usage

   • Even under intense three-finger dragging, this implementation uses less than 1% CPU.
   • Competing Python or Ruby implementations often exceed 20%.

2. Resource Efficiency

   • Memory usage: < 5MB.
   • Binary size: < 2MB.
   • Zero unnecessary dependencies.

Installation & Usage

#

Dependencies

#

Ubuntu/Debian:

sudo apt install libudev-dev libinput-dev libxdo-dev xdotool
# For Wayland
sudo apt install ydotool

Arch Linux:

sudo pacman -S libinput xdotool
# Wayland
yay -S ydotool

Install Binary

#

Option 1: Pre-compiled Binary (Recommended)

wget https://github.com/ferstar/gestures/releases/latest/download/gestures
chmod +x gestures
sudo mv gestures /usr/local/bin/

Option 2: Install via Cargo

cargo install --git https://github.com/ferstar/gestures.git

Configuration

#

The config uses the KDL format. Example:

// Three-finger drag (X11 & Wayland)
gesture "drag" swipe any {
    fingers 3
    acceleration 1.0      // Drag speed
    mouse_up_delay 500    // Delay after lifting fingers (ms)
}

// Four-finger swipe up to switch workspace
gesture "switch-workspace-up" swipe up {
    fingers 4
    exec "xdotool" "key" "super+Page_Up"
}

Running the Program

#

Systemd Service

#

# Install service file
gestures install-service

# Enable and start
systemctl --user enable --now gestures

Common Issues

#

Permission Denied

#

You need to add your user to the input group:

sudo usermod -aG input $USER

Links

#

• GitHub: https://github.com/ferstar/gestures
• Issues: https://github.com/ferstar/gestures/issues

Enjoy!

I am not a native English speaker; this article was translated by AI.

The root process and ShellClash installation for this router are well-documented here: https://qust.me/post/ax3600_shellclash/

I won’t repeat those steps. Instead, I’ll focus on how to replace the Clash-Meta core to support modern proxy protocols like Hysteria 2 and TUIC.

1. Download Clash-Meta

#

Download the appropriate version (arm64): Clash.Meta Releases. I personally use the Alpha version for the latest features.

2. The Storage Challenge

#

Here’s the problem: The unzipped Clash-Meta binary is nearly 20MB, but the AX3600’s root partition usually has less than 8MB left after installing ShellClash.

Solution: UPX Compression Download UPX and compress the binary: upx -9 clash

This reduces the size to under 7MB, making it easy to fit into the router’s storage.

root@XiaoQiang:~# du -sh /data/clash/clash
6.6M    /data/clash/clash
root@XiaoQiang:~# /data/clash/clash -v
Clash Meta alpha-096bb8d linux arm64 with go1.19.5 Mon Jan 23 06:08:40 UTC 2023

3. Cleaning Up Logs

#

The default firmware has a habit of filling up /data/usr/log with junk files, which can cause network lag. It’s best to set up a cron job to clear it regularly.

Add this to your crontab (crontab -l):

*/3 * * * * rm -rf /data/usr/log/*  # Clear junk logs every 3 minutes
30 3 * * 1~7 /data/clash/start.sh restart >/dev/null 2>&1 # Restart Clash daily at 3:30 AM

4. Performance Tip

#

Since Hysteria can be CPU-intensive, I recommend limiting the downlink speed (e.g., to 100 Mbps) to keep the router load manageable.

Created: 2023-01-24 Updated: 2026-01-04

I am not a native English speaker; this article was translated by AI.

Note: Technical audit performed in 2026. While protocols evolve, the underlying logic of Cloudflare’s edge acceleration remains robust.

The Principle

#

Use Cloudflare Tunnel to map local services to Cloudflare’s edge network. VPS with poor connectivity (like Oracle Free Tier) can gain a significant speed boost through CDN edge acceleration.

graph LR
    User((Client)) -- "WSS Request" --> CF[Cloudflare Edge]
    subgraph "Your Home/VPS"
        CFT[cloudflared] -- "Persistent Tunnel" --> CF
        Brook[Brook WSServer] -- "Local Loopback" --> CFT
    end
    style User fill:#f9f,stroke:#333,stroke-width:2px
    style CF fill:#f96,stroke:#333,stroke-width:2px
    style Brook fill:#4ecdc4,stroke:#333,stroke-width:2px

Setup Guide

#

1. Preparation:

   • Cloudflare Zero Trust Dashboard: Create a Tunnel directly in the dashboard—it’s much easier than managing certificates via CLI.
   • Brook wsserver: A very simple and logical proxy tool.

2. Start the Server: brook wsserver --listen 127.0.0.1:8888 --password your_pass

3. Configure the Tunnel: In the Cloudflare dashboard, point your domain (Public Hostname) to http://localhost:8888.

4. Crucial Setting (Must Do): Ensure WebSocket support is enabled in Cloudflare’s “Network” settings; otherwise, the WSS connection will be terminated immediately.

5. Client Connection: Select WSS mode on your client. Traffic now flows through the encrypted tunnel.

Pros and Cons

#

Pros:

• Stealth: Your VPS IP is never exposed. Cloudflare handles all traffic, significantly reducing the risk of IP blocking.
• Penetration: Works in IPv6-only or NAT/internal network environments.
• Free: Cloudflare’s free tier for personal users is more than sufficient.

Cons:

• Latency: The extra hops add latency (typically in the hundreds of milliseconds), making it unsuitable for gaming.
• UDP: Support for non-HTTP UDP traffic is limited; recommended for TCP/WSS scenarios.

Further Reading

#

• Hysteria 2: A performance monster for non-CDN scenarios.
• Cloudflare WARP: For auxiliary optimization.

Created: 2022-03-16 Updated: 2026-01-04