Forewords

I do a lot of networking in my career life, from telco vendor’s chassis router’s operating system and ethernet switch operating system, and cloud provider’s virtual network’s software appliances when the cloud becomes a thing.

After longstanding back pain and burnout in a stressful work environment, I decided to collab with company in mid of 2021 to replace their existing SDWAN software stack with a new one based on fd.io VPP, the collaboration is hardly to say a business success most due to the financial relation between me and the company which continually failed to keep the promise, but the software stack is quite complete and nowadays it powers one of my free VPN services’s VPN servers, because the stack bypass most most overhead the Linux network using some quirks from Linux kernel(I do not use the DPDK plugin because on small VPS that’s not practical)

I knowed some iOS app stuffs dated to 2017 which is hard to believe is almost 10 years ago when I start writing this article, at that time I start a startup with some friends to build our initial SDWAN B2B business, and we pivoted because it turned out we cannot do managed SDWAN solutions due to the infra cost and lack of sales experience and resource. then we pivoted to build early consumer VPN solution and the iOS vpp is written with help of one of my friends but we decide to cancel it after our initial startup team decide to part and try new stuffs. Even the iOS app is built by my friend, I did learn some stuffs of how to public iOS app to the App Store as a organization.

While I was collaborating with the aforementioned company during my second SDWAN stack implementation, I started to play with tailscale and really like it idea, I use my free time to build a customized version of headscale and build and publish a tailscale based iOS app for domestic users in our country which unable to access tailscale especially the DERP fleet may be blocked usecase.

That app is initial written by Gio which I find from the the android open source tailscale app, and gradually tailscale abandoned the Gio and use native kotlin and compose to build the app and given the iOS tailscale app is not open sourced due to a closed source client on closed source Operating Systems policy 1, but I do learn some SwiftUI at that moment and gradually abandoned Gio too.

After the collaboration is finished, I have been left nowhere, but the desire to build something cool truly independent intrigued me a lot.

And with the iOS app experience I learned during the years, I got an idea, why not trying to become a indie hacker, start to porting the fd.io vpp to apple? and build a new WireGuard client on that platform because at that moment, I see the official wireguard client is not actively updated.

When I am writing this article, I have also publish another app called NovaScale which is also based on tailscale but without requirement the VPN extension, it run tailscale entirely inside the app itself, and the initial WireGuard client also have a pending release that have completely rewrite the fd.io VPP to a rust version using same architecture i learned over the years and also get tailscale integrated as a layer 3 VPN endpoint.

Those are big steps if I look back, and the agentic coding help me a lot and I have to say I also observed the history that software development change drastically. So this article is also for noting this historic changes.

How to Build a fd.io VPP based Wireguard client for iOS

When doing app dev pre agentic age where the word AGE means long time but actually it’s just near 2 years ago.

We need a abstract software architecture design and some basically understand of how a VPN app works in iOS platform.

On the iOS platform, a VPN app run as a Network Extension, it have a few flavors, but for WireGuard that works on layer 3, we actually want to use the packet tunnel provider, which means iOS will sent traffic selected by us by programming the iOS routing table and let us process them, in WireGuard case we need handshake with remote VPN server when needed and encrypt and send the packet to the VPN server and decrypt the reply and sent back to the iOS and let it to deliver to the various Apps.

That’s basically how the most of VPN app works.

Given fd.io VPP already have a WireGuard plugin and I like to port the fd.io VPP to apple platform. So we have too main tasks need to be addressed:

  • How to make fd.io VPP process rx/tx packets from the iOS
  • How to port fd.io VPP to apple platform

Let’s talk about separately below.

How to make fd.io VPP process rx/tx packets from the iOS

When I built the app at that moment, I do not know much FFI mechanism as I know when writing this article, so I think I’d better give a look on how official WireGuard iOS does.

Turned out it also do not use official Packet Tunnel Provider’s Swift API, instead, it do the following stuffs:

  • Enumerate the underlying utun’s corresponding, If one have used any VPN on macOS including the tailscale macOS app, you may know what is utun is), its a Layer 3 VPN virtual interface that macOS/iOS system just like the Linux’s tuntap, one end is the system, one end is a user space program
  • Use standard IO interface to operate on the fd , thanks for the Unix philosophy, it’s a plain good old file handle we can just read and write

That’s the core glue logic, all other part is basically covered by the Wireguard-go open source code.

If you can interesting, you can look at the source code in 2 to see exactly what it does, and I do not know exactly how Jason get the following definition, but he may just dig on the XNU source which is the open source part of Apple’s Darwin component.

#define CTLIOCGINFO 0xc0644e03UL

Given we can find the fd, for fd.io VPP, we just add a dedicate plugin that do async IO by leverage fd.io VPP’s clib_file related APIs.

How to port fd.io VPP to apple platform

This is the most challenging part when I start the project because fd.io VPP is mainly used on Linux, but if we dig it deeply we can see it’s actually OS agnostic from the beginning perhaps Cisco have many CPU/OS combinations on the routers.

And at the moment I noticed that there was a news about fd.io VPP get running on FreeBSD3 and I had dived again on the source and saw it used a project called epoll-shim which use kqueue on FreeBSD to provide a compat interface for high level application and in this scenario its fd.io VPP.

Given I know Apple’s XNU is based on Mach Kernel + userland FreeBSD stack, and XNU provides robust POSIX compatibility, I concluded that there is no major road block to achieve that.

Turned out it take me near 2 month to finish the port and get a lean version fd.io VPP running on macOS and can get packet from the macOS and responding ping from the macOS.

I say lean version because the fd.io VPP does this:

  • A pure plain makefile without the vpp complex build system involved, which allow me to build only the needed feature because fd.io VPP use fixed pool to allocate resource which hard to run given the iOS network extension have strict 50MB limit.
  • 1 master + 1 worker only, this is also for reduce complexity and memory usage

Most of the fd.io VPP works involve the following areas:

  • mmap API compatibility workaround on the macOS
  • vpp co routine’s aarch64 assembly code fix for the Xcode clang syntax

But I have to say, fd.io VPP’s C code quality is quite good and port it is actually not a pain but a joy of learning.

WireGuard feature integration

On the network extension, we need a way to send and receive packet to the internet where the WireGuard VPN server lives.

The network extension run under an sandbox like ordinary Apps on the iOS platform, there are many flavors to do the networking communication with the internet4, given I was using fd.io App, I will just use the BSD socket API.

For the concrete implementation, I wrote a dedicate NetIO(network IO) thread and port the fd.io VPP memif too to allow WireGuard generated packets (handshake and data) to routed to this thread, then I just use some metadata passed along with the packet to index the real endpoint ’s corresponding socket fd and do the IO stuffs.

After some trial and fail, I managed to get the WireGuard plugin handshake with my test WireGuard server, that’s a moment I will never forget as a Engineer because it’s the joy of an abstract idea turned into reality.

UI and Apple store submission

Oh, the boring UI stuffs and Apple store submission, this was really my idea at the moment, I learned a lot on UI/UX after that and I felt how silly I was thinking the UI stuffs is boring.

Luckily, even at first half year of 2025, I can use ChatGPT and Claude web interface to help me to do some UI works, and I managed published the app to App Store after a initial 4.3(Spam) rejection.

The App was named MintFlow NetStack, a bad Name if I look back but what I was thinking its really have a cool network stack on the app powered by fd.io VPP.

The initial version is limited, but it does support multiple active WireGuard interface and support CIDR based split routing.

Road to a non-WireGuard-only VPN client with some HTTP MITM features

After the initial version of MintFlow NetStack live on the App Store, I think it still lack of some fatal feature, notably the following:

  • Domain based split routing
  • Layer 4 proxy support (including proxy to the iOS’s local network)

With those feature, I can only send some domains to the WireGuard VPN/Proxy server while kept other traffic to local for better latency and throughput.

When I thinking about those features, I kept thinking how about to bring rust to the app because fd.io’s memory model already take a portion of the restricted 50MB memory budget, I can’t introduce any go based program to it and I do not want to use C for those stuffs because the velocity is a bit low.

I started to learn rust and use some well know open source project such as hickory-dns5 for the internal DNS server which is core for the domain based split routing, and YtFlowCore 6 which support some well known Layer 4 proxy protocols and have a nice fd.io VPP node like architecture.

The most challenging part is how to connect to the Layer 4 proxy because basically we only got Layer 3 packets from the iOS system, but it’s glad fd.io VPP have a experimental host stack which allow me to transparent reconstructed the Layer 3 packets and then proxy the data to a Layer 4 proxy based on user’s routing rules.

When those two features implemented, I think I finally get the app in good shape and use it daily for my own.

I kept a growing feature list on the App’s web site7, I am glad I kept enhancing it even not gain too much track as my NovaScale App does.

Road to a full rust based 2.0.0 version of the app.

I have witnessed the power of the Agentic coding keep evolving in the last half of 2025 as an Claude code starting to be general available, it start to help me to do some real works, but I do let it to do the fd.io VPP based C initially.

In the end of 2025, I started to subscribe to OpenAI’s codex given it have more flexible plan and some public benchmark shown it achieve some level as Anthropic models.

Some experiment

I noticed AmneziaWG which is a nice alternative WireGuard based VPN protocols which have some cool ideas and also have a kernel module based self hosted solution.

I digged some of its code and find its quite neat and I was wondering why not give codex a change to let it digest the fd.io VPP’s wireguard plugin and add the AmneziaWG as a variant reuse most of its code.

If still remembered correctly, it was Gpt 5.5 model, and it seems take sort of half hour to add the AmneziaWG support given I provided it the AmneziaWG and Kernel module implementation, and I did some review and some additional round to fix the stuffs.

Say goodbye to fd.io VPP with too much Codex resets

The above experiment give me confidence for some more bold work such as rewrite fd.io VPP to solve the two hanging issues:

  • fd.io VPP’s self hosted memory heap maybe good for networking stack but lack of flexible as a iOS VPN app
  • the host stack of fd.io VPP’s still lack of some advanced feature such as QUIC and its keeps evolving but still a large code base for reasoning

I know a bunch of networking stack during the years, so I heard smoltcp which is a rust based network stack, but when I first saw it, I remembered it does not support TCP congestion control, but turned on it made great progress over the years.

And start from the July of 2026, codex give user many weekly resets and also cancelled the 5 hour limit for pro 5x users(yes, I upgraded my subscription to 5x because codex really did a good job in my various works), so while I kept working on the my NovaScale app’s version, I started to wrote some documents to guide a fd.io VPP to rust rewrite.

Given deep knowledge of how fd.io VPP works under the hood and the experience I gained about networking these years, same 2 months passed even before the Gpt 6 astra released, I have to say the rewrite is complete.

The notable things I want to leave on this rewrite:

  • The app does not crashed more due to memory limit
  • Since the rust rewrite still bear vpp’s vectorized packet processing in mind, the performance is quite good and I get nice figured on loopback test with packet over the rust rewrite
  • I get smoltcp based l4stack integrated to the fd.io VPP to reconstruct the sessions for the same reason of do Layer 4 proxy support
  • Given I have more memory budget, I integrated the tailscale as an Layer 3 VPN interface just as WireGuard, with its gVisor based netstack removed and use derived DNS records with the app’s internal DNS server to support most function of MagicDNS without rely on the gVisor as official Tailscale iOS app does

And in the early of this month(Sept. 2026), I got the new 2.0.0 app Approved on the app store and renamed it to a more correct name as Mintflow: MintFlow: VPN & HTTPS Capture , one can download it at AppStore 8

Meantime, I made some adjustment on the free features to make the internal DNS server available for free because it then allow user to use the App to setup DoT/DoH and kept most of traffic to iPhone’s network.

Closing words

Nowadays write a long article seems to be not a easy task, I wrote some portion of this article and found that I dig to much into the details and get lost so I abandon it a bit.

But I finally motivated myself to finished it while I was waiting Codex to finished a project that I have long want to do because the manual operation overhead, and I am glad that I wrote this article quickly and got the flow state again!

And last is my two cents for software engineers on the new Agentic coding era, I do not like the so called Vibe Coding, perhaps its just a bias based on the Mind Washing on the social media about that stuffs, but I still think we may have to gain enough domain knowledge to build some stuffs, I cannot image one does not know c vpp and what does the vector packet processing do can do a vpp rust rewrite successfully; It just like the recent bun rewrite from the zig to rust, if the author does not build using zig and understand nodejs deeply, the rust rewrite will not be that easy (I do not know much of rust and node, so I do not comment on the quality of the Bun rewrite)

FootNotes

Footnotes

  1. refer to point 2 in Open source at Tailscale ↩

  2. check how the tunnelFileDescriptor variable get set in WireGuard apple source ↩

  3. Vpp On FreeBSD ↩

  4. Apple TN3151 ↩

  5. https://github.com/hickory-dns/hickory-dns ↩

  6. https://github.com/YtFlow/YtFlowCore ↩

  7. https://galaxnet.dev/mintflow/roadmap/ ↩

  8. Download MintFlow App on the App Store ↩