AdGuard Home: Taming the DNS Loop

Installing AdGuard Home (AGH) via the iStoreOS app store was straightforward. The initial setup wizard (:3000) guided me through setting ports. I changed AGH’s DNS port to :5353 to avoid conflict with iStoreOS’s built-in dnsmasq (running on :53).

The goal: Clients -> iStoreOS (dnsmasq :53) -> AdGuard Home (:5353) -> Public DNS (e.g., 1.1.1.1).

Configuration involved:

1. dnsmasq: Set it to not forward upstream queries found in /etc/resolv.conf, and explicitly forward all queries to AGH listening on the loopback address: server=127.0.0.1#5353.
2. AdGuard Home: Set its upstream DNS servers to Cloudflare/Google.
3. DHCP: Configure iStoreOS’s DHCP server (Option 6) to advertise the router’s IP (10.0.0.1) as the DNS server for all clients.

Initially, this resulted in DNS resolution failures and context deadline exceeded errors in the AGH logs. Diagnosis: a DNS loop. Dnsmasq was forwarding to AGH, but AGH, possibly due to a misconfiguration or startup order issue, was somehow trying to query dnsmasq back.

Adding to the confusion was a “ghost configuration.” I had initially tried installing AGH via a curl script before using the app store version. This seemed to leave conflicting settings. The breakthrough came almost accidentally: running the AdGuard Home binary directly from the command line with the -s run flag (./AdGuardHome -s run) triggered some kind of self-reset or reinitialization. After this, restarting the service properly allowed the correct DNS chain (dnsmasq -> AGH -> Public) to establish, and DNS resolution started working beautifully across my network.

The Network Speed Crisis: A Virtualization Bottleneck

With DNS filtering active, I ran a speed test. Disaster! My gigabit fiber connection, which previously delivered ~950 Mbps, was now capped at a measly 400 Mbps.

The troubleshooting began:

• Disable AdGuard Home: No change. DNS filtering wasn’t the bottleneck.
• Check Physical Links: Used ethtool eth0 and ethtool eth1 within the iStoreOS VM to confirm the virtual NICs were negotiating at 2500 Mbps with the PVE bridges. Links were fine.
• Enable Software Flow Offloading: Toggled this common OpenWrt optimization. No significant change.

The problem had to lie deeper, likely within the virtualization layer itself. Research pointed towards a common issue with high-speed networking in VMs: the CPU struggling to handle the sheer volume of network packets being processed by a single core for the virtual NIC.

The Solution: Multiqueue

Proxmox VE (and the underlying KVM/QEMU) supports a feature called Multiqueue VirtIO-Net. This allows the network packet processing load for a virtual NIC to be spread across multiple vCPU cores assigned to the VM.

1. Shut down the iStoreOS VM.
2. In the PVE web UI, go to the VM’s “Hardware” tab.
3. Select the VirtIO network device corresponding to the LAN interface (net0 in my case, connected to vmbr1).
4. Click “Edit”.
5. Check the “Multiqueue” box.
6. Set the “Queues” number equal to the number of vCPU cores assigned to the VM (e.g., if you gave it 4 cores, set Queues to 4).
7. Repeat steps 3-6 for the WAN network device (net1, connected to vmbr0).
8. Start the iStoreOS VM.

Ran the speed test again. Success! Speeds jumped right back up to the ~950 Mbps range. The virtualization bottleneck was eliminated.

Tailscale Triumph: Conquering Remote Access

My final major goal was setting up remote access, primarily to manage services like my NAS (planned OMV VM) and potentially the router itself when away from home. I opted for Tailscale due to its ease of use and availability in the iStoreOS app store.

Installation was simple. Following the command-line instructions (tailscale up --advertise-routes=10.0.0.0/24) connected the router to my Tailnet and advertised my home LAN subnet. However, while the router appeared online in the Tailscale admin console, I couldn’t access any devices on my home LAN (like 10.0.0.1) from my phone connected via Tailscale.

Troubleshooting Round 1: Firewall

Pinging 10.0.0.1 from my phone resulted in “Destination Port Unreachable.” This screamed firewall. iStoreOS hadn’t automatically created a firewall zone or rules for the new tailscale0 interface.

Troubleshooting Round 2: Interface & Zone

Following guidance from GitHub issues, I manually edited /etc/config/network to define the tailscale0 interface:

config interface 'tailscale'
        option proto 'none'
        option device 'tailscale0'

Then, in the LuCI Firewall settings (Network -> Firewall), I:

1. Created a new firewall zone named tailscale.
2. Set its Input/Output/Forward policies to ACCEPT.
3. Assigned the tailscale0 network interface to this zone.
4. Crucially, created a Forwarding rule allowing traffic from the tailscale zone to the lan zone.

Saved and applied. Tried connecting again. The “Port Unreachable” error was gone, but now attempts to access 10.0.0.1just… hung. Loading indefinitely.

Troubleshooting Round 3: MTU/MSS

This new symptom – indefinite loading – strongly suggested an MTU (Maximum Transmission Unit) mismatch problem, common with VPNs and tunnels like Tailscale. Packets were likely getting fragmented or dropped because their size exceeded what the tunnel could handle. The standard fix is MSS Clamping (Maximum Segment Size), where the router rewrites TCP packet headers to request smaller segment sizes.

The Final Fix: iptables MSS Clamping

Added the following rules via Network -> Firewall -> Custom Rules:

# Tailscale MSS Clamping for IPv4
iptables -t mangle -A FORWARD -o tailscale0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

# Tailscale MSS Clamping for IPv6 (if using IPv6 with Tailscale)
ip6tables -t mangle -A FORWARD -o tailscale0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Restarted the firewall. Tried accessing 10.0.0.1 from my phone via Tailscale one last time.

Victory! The iStoreOS login page loaded instantly. Remote access was finally conquered.

Conclusion: A System Forged in Debugging

This journey, from a simple IPv6 issue to a fully virtualized N100 soft router with custom services and robust remote access, was far more challenging than I initially anticipated. Every step seemed to uncover a new problem, requiring deep dives into networking concepts, Linux internals, and virtualization quirks.

However, each problem solved solidified my understanding and resulted in a final system that is incredibly powerful, stable, and perfectly tailored to my needs. The N100 platform with Proxmox and iStoreOS has proven to be an outstanding foundation. The debugging war is over, and the era of enjoying a truly capable home lab has begun.

Act I: The Struggles with Vanilla OpenWrt

Creating the VM in PVE (giving it VM ID 100) involved several steps:

1. Download Firmware: Get the official OpenWrt x86-64 combined-ext4 image (.img.gz).
2. Upload & Import: Upload the .img.gz file to PVE (e.g., via SCP or the web UI’s storage browser – note: uploading directly as an ISO image type fails). Import the disk image into the VM using qm importdisk 100 openwrt.img.gz local-lvm.
3. VM Hardware Config: Assign CPU cores, RAM. Remove the default SCSI disk. Attach the imported OpenWrt disk as a VirtIO block device. Add two VirtIO network cards – one connected to PVE’s vmbr0 (WAN) and one to vmbr1 (LAN). Set the boot order to use the OpenWrt disk.
4. Disk Resizing: The default image is tiny. Resize the virtual disk in PVE’s hardware tab. Boot the VM with a GParted Live ISO attached, use GParted to expand the OpenWrt partition to fill the larger virtual disk.
5. Initial Network Config: Boot OpenWrt. Access the console (via PVE’s web UI or qm terminal 100 - not qm console). Use the vi editor to modify /etc/config/network, changing the LAN interface’s IP address to 10.0.0.1.

So far, so good. I could access the OpenWrt LuCI web interface at 10.0.0.1. I configured PPPoE on eth1 (WAN, connected to vmbr0) and set up the LAN interface (eth0, connected to vmbr1) with DHCP and IPv6 server settings. Basic routing worked!

The Dependency Hell

The problems began when I tried to customize OpenWrt by installing packages. My goal was simple: install luci-app-adguardhome. But opkg update often failed to connect to the repositories. When it did connect, attempting to install AdGuard Home resulted in a cascade of errors about incompatible architectures or missing dependencies (luci-lua-runtime was a frequent culprit). Even manually downloading .ipk packages failed with similar architecture errors.

I spent hours trying different package versions, compiling dependencies, and scouring forums. I even managed to “force install” some packages, but this led to a broken LuCI interface – the “Services” menu simply wouldn’t appear, despite the package seemingly being installed.

[Placeholder: Add a screenshot of the terminal showing the opkg errors or the LuCI interface missing the Services menu.]

Clearing LuCI caches (rm /tmp/luci-indexcache) did nothing. Log files (logread) showed no obvious errors. File permissions seemed correct. The final nail in the coffin was a detailed error log clearly stating that key LuCI components and their dependencies were fundamentally incompatible with the base system. Trying to manually build a complex setup on vanilla OpenWrt felt like assembling a car from mismatched parts found in a junkyard.

Act II: The Turn Towards iStoreOS

Frustrated and realizing the “DIY package management” route was unsustainable, I revisited earlier advice: use a pre-integrated firmware like iStoreOS. Based on OpenWrt but comes with a user-friendly interface and, crucially, a curated app store where dependencies should just work.

The Migration Plan

The challenge: perform this migration without disrupting my family’s internet access for too long.

1. Preparation: In PVE, create a new VM (e.g., VM ID 101) for iStoreOS. Download the iStoreOS x86 .img.gz firmware.
2. Import & Basic Config: Upload and import the iStoreOS disk into VM 101. Configure its hardware similarly to the OpenWrt VM (CPU, RAM, 2x VirtIO NICs connected to vmbr0 and vmbr1).
3. Offline Config Attempt (Failed): Boot VM 101 with GParted. Try to mount the iStoreOS partition to change the default IP (192.168.100.1) to 10.0.0.1 offline. Failure! GParted couldn’t identify the filesystem. Manual mount commands failed, complaining about unknown filesystem types or missing superblocks. Offline config was impossible.
4. The Switch: Plan B - configure online. Physically disconnect the WAN cable from the N100. Shut down the old OpenWrt VM (100). Start the new iStoreOS VM (101). Set my Mac’s IP statically to 192.168.100.5. Connect my Mac to one of the N100’s LAN ports (connected to vmbr1). Try accessing 192.168.100.1.

The WAN/LAN Inversion Twist

Failure again! 192.168.100.1 was unreachable. In a moment of desperation (or maybe just random plugging), I moved my Mac’s cable to the N100 port I had designated for WAN (connected to vmbr0). And… success! I reached the iStoreOS login page.

Diagnosis: iStoreOS had, for some reason, reversed the network interface identification compared to OpenWrt. It saw eth0 (connected to PVE’s vmbr1, my LAN bridge) as its WAN interface, and eth1 (connected to PVE’s vmbr0, my WAN bridge) as its LAN interface, assigning the 192.168.100.1 address there.

Resolution: Logged into iStoreOS via the “wrong” port. Navigated to Network -> Interfaces. Edited the LAN and WAN interfaces, swapping their assigned physical devices (eth0 <-> eth1). Saved and applied. Moved my Mac’s cable back to a correct LAN port. Set my Mac back to DHCP. Accessed 192.168.100.1 (the default iStore IP). Success! Changed the iStoreOS LAN IP to 10.0.0.1. Plugged the WAN cable back into the N100’s WAN port.

The network migration was complete in under 15 minutes. The foundation was finally stable. iStoreOS provided the pre-packaged environment I needed. Now, it was time to deploy the services that started this whole journey.

Installing PVE: The First Hurdle

Installing PVE itself is relatively straightforward, similar to installing any Linux distribution:

1. Create Installation Media: Download the PVE ISO image. On my Mac, I used Balena Etcher to write the ISO to a USB drive. My first attempt failed – turns out the old USB drive I grabbed was faulty. A newer drive worked perfectly.
2. Boot from USB: Connect a monitor and keyboard to the N100 box, plug in the USB drive, and boot into the BIOS/UEFI settings (usually by pressing DEL or F2 during startup). Set the USB drive as the primary boot device.
3. PVE Installer: Follow the graphical installer prompts. Key settings during installation:
   • Target Hard Disk: Select the Fanxiang NVMe SSD.
   • Location/Timezone: Set appropriately.
   • Hostname: Set to something memorable, like pve.local.
   • Network Configuration: Crucially, assign a static IP address for PVE’s management interface (e.g., 10.0.0.2), along with the gateway (10.0.0.1 - my eventual router VM’s address) and DNS servers. I chose one of the four physical ports (e.g., enp1s0) for this initial management access.

The installation completed smoothly, and PVE rebooted. I could access the command line directly on the N100, and more importantly, access the web UI from my Mac at https://10.0.0.2:8006. Stage one complete!

The Networking Nightmare: Reconfiguring for VMs

PVE’s default network setup uses one physical port (enp1s0 in my case) solely for its own management. However, to allow my virtual machines (like the router VM) to connect to both the internet (WAN) and my home network (LAN), I needed to create Linux Bridges.

My plan:

• vmbr0: Use a different physical port (e.g., enp2s0) for the WAN connection (connecting to my ISP modem).
• vmbr1: Use the remaining two physical ports (e.g., enp3s0, enp4s0) bridged together for the LAN connection (connecting to my home switch).
• Crucially: Move PVE’s own management interface from the physical port enp1s0 onto the newly created LAN bridge (vmbr1). This way, I could manage PVE from any computer on my home network using the 10.0.0.2 address, freeing up enp1s0 perhaps for other uses.

This reconfiguration is done by editing the /etc/network/interfaces file directly via the PVE console (or SSH). I carefully modified the file, creating the bridge definitions and changing the management interface settings.

I applied the changes (systemctl restart networking or a reboot). And then… disaster.

I could no longer access the PVE web UI at 10.0.0.2. Pings failed. It seemed I had completely locked myself out.

Troubleshooting in the Dark

Panic set in. Had I made a typo in the config file? Did the bridge fail to come up? I had to go back to the “direct connection” method: plug a monitor and keyboard directly into the N100 box.

Logging into the PVE console, I ran ip a to check the network interface status. The output revealed the problem: vmbr1 (my intended LAN bridge and new management interface) had no IP address. My configuration changes hadn’t been applied correctly, or something had gone wrong during the network restart.

Carefully re-examining /etc/network/interfaces line by line, I eventually spotted the error – perhaps a typo, a missing auto vmbr1, or incorrect bridge port settings. I corrected the file using the nano editor (a bit friendlier than vi for quick edits).

After saving the corrected file and restarting the networking service again (systemctl restart networking), I ran ip a one more time. Success! vmbr1 now showed the correct 10.0.0.2 address.

Back on my Mac, I refreshed https://10.0.0.2:8006, and the Proxmox login screen reappeared. Access restored!

This harrowing experience was my first real taste of configuring Linux networking and the importance of meticulousness when editing critical system files. With the PVE host networking finally stable, I was ready for the main event: creating the virtual machine that would become my new home router.

What is a Soft Router?

Unlike traditional “hard” routers with proprietary hardware and software, a soft router is essentially a small, dedicated PC running a specialized router operating system (like OpenWrt, pfSense, OPNsense, or RouterOS). The “x86” refers to the standard PC processor architecture (Intel/AMD), offering significantly more power and flexibility than the ARM or MIPS chips found in most consumer routers.

The Allure of x86

The potential benefits were immense:

• Raw Performance: Capable of handling gigabit+ speeds with complex firewall rules, VPNs, and traffic shaping without breaking a sweat.
• Flexibility: Run virtually any router OS or even standard Linux distributions.
• Expandability: Add more RAM, faster storage, or even specialized network cards.
• Virtualization: This was the game-changer. An x86 platform could potentially run multiple operating systems simultaneously using a hypervisor like Proxmox VE (PVE). I could have my router, a NAS, a media server, and more, all running on one efficient box.

Choosing the Platform: N5105/J4125 vs. N100

My research focused on popular low-power x86 chips commonly used in mini PCs and soft routers:

• Older Generation (Jasper Lake N5105, Celeron J4125): Widely available, often cheaper, proven platforms. Decent performance for routing and light server tasks.
• New Generation (Alder Lake-N N100): Significant performance uplift (especially single-core, crucial for routing), much better integrated graphics (useful for PVE console/light desktop tasks), generally more power-efficient, supports faster DDR5 RAM. Often comes paired with newer Intel i226-V 2.5GbE network controllers, known for better driver support and stability compared to the sometimes-problematic Realtek chips found in cheaper units.

Given the relatively small price difference for new units and the significant performance and efficiency gains, the Intel N100 quickly became the clear winner for future-proofing.

Navigating the Hardware Maze

Choosing the N100 platform opened up a dizzying array of options, mostly from Chinese manufacturers like Beikong (Topton/Kingnovy), CWWK, etc., often sold as barebones systems (no RAM/SSD). Key considerations:

• Ports: How many? What speed? The standard was 4x 2.5GbE Intel i226-V ports, perfect for high-speed LAN and future ISP upgrades. Some offered 2x 2.5GbE + 2x 10GbE SFP+ (fiber) – tempting, but overkill and more expensive for my current needs.
• Cooling: Passive (fanless) is preferred for silent operation, requiring a well-designed chassis.
• RAM/SSD: DDR5 SO-DIMM and NVMe M.2 slots were standard.
• Price: Barebones N100 boxes with 4x 2.5GbE ports were readily available.
• Used Market: While tempting for cost savings, diagnosing potential issues on used hardware felt risky after my R8000 experience.

My Final Choice: Beikong H30W (N100)

After much comparison, I settled on a brand new Beikong H30W N100 barebones unit featuring:

• Intel N100 Processor
• 4x Intel i226-V 2.5GbE ports
• Passive cooling chassis
• DDR5 SO-DIMM slot
• NVMe M.2 slot

To complete the build with maximum cost-effectiveness:

• RAM: I salvaged a compatible 16GB DDR5 stick from an old laptop – a huge saving!
• SSD: I purchased a new, budget-friendly but well-regarded Fanxiang S500 Pro 256GB NVMe SSD. Plenty of space for PVE and multiple VMs.

The Grand Vision: Virtualization

With the hardware assembled, the real excitement began. This N100 wasn’t just going to be a router. It was going to be the heart of my home lab. The plan: install Proxmox VE (PVE) as the bare-metal hypervisor, then create virtual machines for:

1. OpenWrt/iStoreOS: My primary router, firewall, and network manager.
2. OpenMediaVault (OMV): A NAS for file storage and backups.
3. Future Possibilities: Home Assistant, AdGuard Home, a web server, maybe even AI experiments?

This N100 box represented a convergence of all my needs and aspirations. The hardware was ready. Now came the software challenge: installing and taming the Proxmox hypervisor.

Operation: Unbrick

Reviving a bricked OpenWrt router usually involves entering Failsafe Mode, a minimal environment allowing command-line access. The process is timing-sensitive and requires precision:

1. Static IP Setup: I configured my computer with a static IP address in the 192.168.1.x range (e.g., 192.168.1.2), subnet mask 255.255.255.0.

2. Direct Connection: Connected the computer directly to one of the R8000’s LAN ports.

3. Power Cycle & Trigger: Powered on the R8000. As soon as a specific LED (usually the Power or WPS light) started blinking rapidly, I repeatedly pressed the WPS button. This triggers Failsafe Mode.

4. SSH Connection: Opened a terminal and attempted to SSH into the router’s default failsafe IP:

   ssh root@192.168.1.1
   

After a few tense tries, catching the blinking light at just the right moment, I got a connection! The familiar OpenWrt command prompt appeared. From there, the standard unbricking procedure followed:

# Mount the root filesystem read-write
mount_root

# Reset configuration to defaults
firstboot

# Force reboot
reboot -f

The Mysterious 10.0.0.1

After the reboot, I expected to access the OpenWrt LuCI web interface at the standard 192.168.1.1. But… nothing. The page wouldn’t load. Puzzled, I opened the terminal again and ran ping 192.168.1.1. No response.

What was going on? I vaguely remembered the OpenWrt firmware I had flashed wasn’t an official build, but a custom one from a forum. Could it have a different default IP? I decided to check my computer’s network settings – had it received an IP via DHCP?

Indeed, it had received 10.0.0.141. The gateway? 10.0.0.1.

A quick ping 10.0.0.1 confirmed it – the router was alive and well, just hiding at a non-standard address! This custom firmware developer had decided to use 10.0.0.1 as the default management IP. Lesson learned: always check DHCP when standard IPs fail.

Configuring the R8000 (Hope Rises)

Logging into the LuCI interface at 10.0.0.1, I proceeded to configure the R8000 as my main router:

• PPPoE: Set up the WAN interface for PPPoE dialing using my ISP credentials. Success – it connected and got an IPv4 address (still CGNAT, of course).
• IPv6: Configured the WAN6 interface for “Native IPv6”. As expected, due to the ONT issue, it didn’t get a delegated prefix, but it did acquire a WAN IPv6 address.
• Interfaces: Mapped the physical ports correctly (identifying eth0.2 as WAN and eth0.1/br-lan as LAN).
• Wi-Fi: Configured the SSIDs and security. Interestingly, I initially set WPA3, but some older devices failed to connect. Downgrading to WPA2-PSK resolved this compatibility issue. Set appropriate channel widths (20MHz for 2.4GHz to reduce interference, 80MHz for 5GHz for speed).

For a moment, things looked promising. The router was routing traffic, Wi-Fi was working (albeit with some WPA3 issues). Could this be the solution?

The Final Failure (Hope Dashed)

My optimism was short-lived. As I started exploring further configuration – trying to set up firewall rules or install necessary packages (opkg update often failed or was incredibly slow) – the R8000 started acting erratically. Sometimes the interface would become unresponsive. Other times, the lights would start flashing indefinitely, requiring a hard reboot.

After hours of troubleshooting, observing the inconsistent behavior, I had to face the harsh reality: this router wasn’t just bricked by configuration before; it likely had an underlying, unrecoverable hardware fault. The flash memory was probably damaged, leading to corrupted data and instability under load.

The rescue attempt had failed. The Netgear R8000 was truly destined for the electronics recycling bin.

It seemed consumer-grade hardware, even powerful ones running flexible firmware like OpenWrt, had reached its limit for my needs. If I wanted reliability, performance, and the ability to truly overcome my network’s challenges, I needed to look towards a different class of hardware altogether: the world of x86 soft routers.

Diving Deep into IPv6 and Network Structure

After some initial head-scratching, research pointed towards a concept called Prefix Delegation (PD). In simple terms, my ISP assigns a block (prefix) of IPv6 addresses to my router, and the router is then responsible for distributing individual addresses from that block to my devices. The symptom – WAN gets an address, LAN doesn’t – strongly suggested PD was failing.

Why? The prime suspect became my ISP’s modem (Optical Network Terminal or ONT). Was it running in router mode instead of bridge mode? If the modem itself was acting as a router and didn’t properly support delegating the prefix downstream, my AX6000 would never receive the necessary block.

I confirmed my network topology: ONT (in router mode) -> Switch -> Multiple Routers (including the AX6000). Unfortunately, due to other devices directly connected to the ONT via the switch, simply switching the ONT to bridge mode wasn’t feasible without restructuring everything.

As a temporary workaround, I enabled NAT6 on the AX6000. Success! My devices could now access IPv6 websites. However, I knew this wasn’t true end-to-end IPv6. NAT6 essentially translates internal IPv6 addresses, similar to how traditional NAT works for IPv4. It solved the immediate browsing issue but didn’t address the underlying PD failure and wouldn’t help with services requiring direct public IPv6 reachability.

Enter the Double NAT Monster

Around the same time, my Xbox started complaining: “Double NAT detected.” This made perfect sense. My network traffic was going through two layers of Network Address Translation: first at the AX6000, and then again at the ISP’s ONT (which was also acting as a router). This configuration is notorious for causing issues with online gaming, peer-to-peer connections, and services requiring open ports.

We discussed potential mitigations like UPnP, manual port forwarding, and placing the AX6000 in the ONT’s DMZ, but none felt like a clean solution.

The CGNAT Revelation

The real blow came when investigating the Double NAT further. By comparing the WAN IP address shown on my ONT’s status page with the public IP address reported by websites like “whatismyip.com”, I confirmed a dreaded reality: I was behind Carrier-Grade NAT (CGNAT). My ISP wasn’t assigning my ONT a unique, public IPv4 address. Instead, I was sharing one with potentially hundreds of other customers.

[Placeholder: Add a screenshot here showing the discrepancy between your ONT’s WAN IP (likely a private 10.x or 100.x address) and your public IP, confirming CGNAT.]

This revelation made two things crystal clear:

1. Solving the Xbox NAT issue reliably via IPv4 was going to be nearly impossible without resorting to complex (and often unreliable) tunneling solutions.
2. Getting native, public IPv6 working correctly was no longer just desirable; it was essential for any kind of future-proof home networking or self-hosting.

Back to the IPv6 Root: The Faulty ONT

With renewed focus, the investigation circled back to the ONT. I confirmed the ONT itself was receiving a /64 or similar public IPv6 prefix starting with 240e:. The problem was solely its inability to delegate this prefix downstream.

Analyzing the ONT’s configuration page revealed a peculiar “Port Binding” setting. Initially, only LAN1 was bound for internet services.

I modified the settings to bind all LAN ports. Hope surged… but was quickly dashed. Even after ensuring all ports were bound, testing with the AX6000 and other routers on different ports yielded the same result: no PD, no LAN-side public IPv6.

The conclusion was unavoidable: the ONT’s Prefix Delegation functionality was faulty, likely due to a hardware limitation or firmware bug. Relying on it was a dead end.

I needed a router powerful and flexible enough to potentially work around the ONT’s limitations, or perhaps even replace its routing functions altogether. This led me to revisit an old piece of hardware sitting in my closet… a supposedly bricked Netgear R8000 running OpenWrt. Could it be resurrected?

Then, immediately after publishing that post, I discovered my “Tags” page was broken in the exact same way.

When I clicked the “Tags” link in my (perfectly functional) menu, it loaded a page that also had no CSS and contained only a giant “#Tags” title, with no actual tags listed—even though my post was correctly tagged and published.

This was the final, definitive clue.

The Unified Bug Theory

This confirmed the problem wasn’t a series of isolated glitches. It was one, single, catastrophic bug. My chosen theme, hugo-paper, had a fatal flaw.

Its “list” templates (like the homepage) worked perfectly. But its “single” template (for the About page) and its “terms” template (for the Tags page) were both fundamentally broken. They contained code that, for some reason, failed only in the Cloudflare Pages production environment, defaulting all asset paths to localhost.

My local build (hugo server -D) worked perfectly, which had masked this problem for days and led me down a debugging rabbit hole of config files, environment variables, and build flags. None of those fixes worked, because the theme’s engine itself was defective.

The Theme Transplant

The solution was obvious. You don’t fix a broken engine; you replace it.

I decided to perform a “theme transplant.” I ripped out hugo-paper completely. This involved:

1. Running git submodule deinit and git rm to purge the old theme from my project.
2. Choosing a new, robust, and extremely popular theme: hugo-PaperMod.
3. Adding it as a new submodule: git submodule add ... themes/PaperMod.

Of course, this came with its own final hurdles. My brand new Hugo version (v0.150.0) immediately threw an error because the new theme’s config used a deprecated paginate key. After fixing that (it’s now [pagination].pagerSize), the Tags page still showed up empty.

One final search revealed the truth: in our quest to create a “clean” config, we had removed the block that activates taxonomies. The final piece of the puzzle was adding this back to hugo.toml:

[taxonomies]
  tag = "tags"
  category = "categories"

My name is Frank, and I’m currently navigating high school while exploring my deep fascination with technology, problem-solving, and the intricate process of building things – both digital and sometimes physical.

This blog serves as my personal notebook and digital playground. It’s where I plan to document my journey as I dive into coding projects, experiment with web development (like the creation and ongoing refinement of this site itself!), tackle interesting technical challenges I encounter, and share insights or concepts I’m learning along the way. You might also find occasional posts about gaming or other topics that capture my interest.

Building this website using Hugo, deploying it globally via Cloudflare Pages, and integrating features like search, comments with Giscus, and even LaTeX rendering has been an incredible learning adventure in itself. It pushed me to understand web infrastructure, Git workflows (CI/CD), and the realities of debugging in a way no textbook could.

Here, you’ll find posts reflecting that journey: projects I’m working on, interesting tech I’m exploring, perhaps breakdowns of problems I’ve solved (or spectacularly failed to solve!), and things I simply find cool.

My hope is that you’ll find something interesting, useful, or perhaps even relatable in these digital pages.

Feel free to explore using the navigation or the search function. Thanks for stopping by!

My very first post should have been the “About” page. It’s the standard, simple, “Hello, this is who I am” page that every blog is supposed to have.

But I deleted it. I killed it before this blog ever saw the light of day.

This isn’t a great start, but it was a necessary one.

This Was Supposed to Be Easy

My goal was simple: set up a personal blog using the “golden stack” of modern static sites: Hugo (as the engine), the hugo-paper theme (for the clean look), Git, and Cloudflare Pages (for fast, free, global hosting).

And to be honest, it worked. Mostly.

Except for one, single, impossibly stubborn page.

The Ghost in the Machine

After I pushed my code, a bizarre problem appeared. My homepage loaded perfectly. The /posts/ list page loaded perfectly. They had the correct styles (CSS) and the correct English site title (“Frank’s Blog”).

But the /about/ page was a ghost.

It rendered as raw, unstyled HTML. Worse, every single link on that one page (the CSS file, the link back to the homepage, the footer links) was broken, pointing to an impossible address: http://localhost:1313/.

To make it even weirder, the footer—which was supposed to pull my site’s English title—instead showed an old, non-existent Chinese title: “Frank 的博客”.

My site was having a split-personality crisis. The homepage knew it was an English site deployed to production, while the About page thought it was a Chinese site running on a local test server.

The Debugging Rabbit Hole

So began an epic debugging saga. An AI assistant and I went down every conceivable rabbit hole to fix this. We tried:

1. Fixing the Config File? We triple-checked hugo.toml. We fixed deprecated keys (paginate), corrected TOML syntax, and scoured the file for invisible whitespace characters. The problem remained.
2. A “Ghost” Config? We hypothesized a “ghost” multilingual file (languages.toml) left over from the theme’s example site was forcing the build into a broken state. We hunted it down and deleted the entire config directory. The problem remained.
3. Build Environment Mismatch? The most maddening clue was that hugo server -D on my local Mac worked perfectly. The About page was beautiful. This suggested a conflict with Cloudflare’s build environment. So, we set the HUGO_VERSION environment variable in Cloudflare to match my exact local version. The problem remained.
4. A Build Command Override? We thought maybe Cloudflare was ignoring the baseURL in the config. So we manually forced it in the build command: hugo --baseURL https://frankblogs.com/. The problem remained.
5. A File Structure Problem? Our final theory. Maybe the theme’s “single page” template was buggy, but its “list page” template (used by the working homepage) was fine. We changed the file structure from content/about.md to content/about/_index.md to “trick” Hugo into using the working template. And still, it failed.

The Verdict

After trying every logical configuration fix, the conclusion was inescapable.

The problem wasn’t my config. It wasn’t Cloudflare. It was a deep, unfixable bug in the hugo-paper theme itself—a flaw in its single-page template that, for some reason, fails catastrophically only when built on the Cloudflare Pages environment, defaulting all its paths to localhost. It was invisible locally, but fatal in production.

So I had a choice: spend the next week learning the Go Template language to debug a third-party theme I didn’t write… or just delete the broken feature.

I chose deletion.

This blog is going to be a record of building things that work. The first lesson is knowing when to cut your losses and pick the pragmatic path.

So, hello, world. There is no “About” page here. There is only this “Posts” page.

Because I know for a fact: this part works.