🎉 Ebbflow has launched! 🎊
Use coupon code LaunchParty to save 25% off of the base price!

Packaging & Vending Production Rust Software For Windows

A look into vending a Rust project for various OSes and CPU architectures.

Ryan Gorup - Founder

Background: Ebbflow is a multi-cloud load balancer that provisions browser-trusted certificates for your endpoints in addition to providing a client-friendly SSH proxy. Servers can host endpoints from any cloud, on premises, or from your home, all at the same time. It uses nearest-server routing for low latencies and is deployed around the globe.

This post describes how Ebbflow vends its client to Windows users which is written in Rust, describing the tools used to build and ultimately deliver the program to users.

Ebbflow Client 101

Ebbflow's job is to route user traffic to your web server application (or SSH Daemon) on your servers. The central Ebbflow service proxies data between user connections (e.g. browser, SSH client) and the client. The client then proxies the data to your web server or local SSH daemon. Users of Ebbflow install the client on machines that will host endpoints, be SSH-ed to, or both at the same time (which is very useful). The following diagram shows this visually.

Quick note: The client initiates the connection to the central Ebbflow service via an outbound TLS connection, which makes the client extremely firewall friendly. You may completely block all inbound connections but still host websites or be SSH-ed to! Also, this innovation allows the servers to be located in any network, and even change networks without any configuration at all. Ebbflow just receives connections from the general internet, and after authentication and authorization, will allow the server to host the endpoint or be SSH-ed to. Neat!

The client has two parts, the CLI and the background daemon. Both of these programs are 100% Rust, 100% async, and 100% 'safe', and statically linked - the dream of any Rust developer! The CLI is an executable tool named ebbflow that is used to tell the background to host new endpoints, disable or re-enable endpoints or the SSH proxy, and configure other settings.

The background daemon is the second piece to this puzzle, and is the workhorse and is responsible for actually transferring bytes between the central Ebbflow servers and your local web server or SSH daemon. This daemon is just a long-running background executable named ebbflowd.

Windows

Supporting Windows as a first-class server OS is very important to Ebbflow. Taking the two executables & re-compiling them on Windows was simple thanks to Rust's abstractions over the underlying OS and numerous community projects. First things first, the background daemon needs to be changed to work in a Windows world.

Background Daemon / Windows Service

The ebbflow CLI is very simple and is portable in the sense that you could execute the binary on a windows machine without installing it and it will do its job correctly, but ebbflowd is another story. To execute a background program in Windows one uses the Windows Service framework. This framework is needed as the ebbflowd program needs to run when there are no interactive users, start when the system starts, and be watched and restarted in the event of a crash (which has not happened due to Rust's inherit safety and never unwraping).

A sticking point of using the Windows Service framework is that your program needs to implement a specific interface, namely it must provide an entry point, handle signals, and register itself. To help, the windows-service Rust crate (provided by Linus Färnstrand) makes this very simple.

This windows-service crate provides a macro which requires that you implement just a few functions that are effectively main functions. The Ebbflow client uses #[cfg(windows)] to conditionally include all of the necessary code to implement these functions and therefore the Windows Service interface itself! The winlog crate was also helpful for taking log events and adding them to the Windows Event Log system.

Going Static

Rust will dynamically link against the OS's libc implementation when the standard library is used, which is the case for almost all substantial Rust applications. On Windows, this means Rust will compile programs and link them to the system's 'MSVC' (MicroSoft Visual C++) libc implementation. Interestingly, the default Windows 10 image does NOT include this library! This results in Rust code breaking when executed on a new system, something that has come up in testing, but may not come up in development as MSVC would likely get installed at one point or another when setting up a Windows machine for code development.

Statically linking Rust code is simple. To bring along MSVC with your compiled Rust program, you instruct the compiler to statically link MSVC. In my case I set just one environment variable (RUSTFLAGS='-C target-feature=+crt-static') and it simply worked, but you can add a config file entry or pass a flag to compilation as well.

Besides the std library, Rust code may link to other OS libraries most often OpenSSL or libsodium for crypto. The Ebbflow client avoids this by using rustls which uses ring under the hood. Rustls is an ergonomic TLS library for Rust. It is highly performant even compared to OpenSSL and recently underwent a 3rd party security audit which showed no flaws. When you invest in the Rust ecosystem and use Rust-written libraries, you are rewarded with the ability to statically link which is a desirable situation.

Building the .msi using Wix

Windows uses 'installers' in the form of .msi packages to handle which packages are installed on a system. These installers tell Windows what files/executables go where and contain logic to handle upgrades, uninstallations, and configuration management.

Ebbflow uses Wix which is a tool that takes an .xml configuration file and generates .msi packages. Conveniently, a plug-in to Rust's cargo build tool named cargo-wix exists. This tool can generate a skeleton Wix xml file which we then modified to suit our needs. Under the hood, executing cargo wix will compile your program in --release, then package up everything into your .msi using candle.exe to compile your Wix config and light.exe to link it.

Generating and tinkering with the client's Wix file took some time. There are numerous StackOverflow q/a's and other resources to help with writing the Wix file which was helpful, but I found that vending a Windows Service in an installer is more of an edge case. Also, configuring the UI components of the installer is tricky and I never found a great reference implementation or other documentation.

Automated Builds using GitHub Actions

Up to this point, I've produced a statically linked Windows program and packaged it into an installer. For various reasons such as build reproducability & transparency, and hosting the built .msi for free, Ebbflow uses GitHub Actions to build and package up the client. You can see the client's Windows build configuration on GitHub, and here is a snippet:

GitHub provides a Windows image that has various build tools included, importantly Wix. Getting Windows builds set up on GitHub Actions was pretty simple and I had a great experience with the service.

Distribution to Users

After writing the code, making an installer and testing that out, the next step is to figure out how to get this into our customers hands. To solve this, the first thought is to just distribute the .msi file itself and notify customers of upgrades. This generally works and the installer can handle upgrades and uninstallations. However, there are drawbacks. The major one is that any future updates will require the customer to download the new .msi, which requires some custom process. Windows will flag any software that is unsigned and all applications must be signed using certificates derived from trusted certificate authorities. This is fine and secure but to have our .msi work without warnings you must purchase a code signing certificate/key from someone like Digicert or the like which can easily run into the hundreds of dollars. Alternatively you can create your own CA & signing key/certificate, self-sign the msi, then have all customers add your CA to their trusted store. This does not seem too common from my understanding.

These two drawbacks are pretty severe, which is what leads us to our chosen solutions, Chocolatey and Winget.

Chocolatey

Chocolatey is a 3rd party package manager and you can use it to install, update, remove, and/or search for packages. It works great for personal use and will feel very familiar to any Linux users. Chocolatey also provides a multi-computer management solution to enterprise customers so sysadmins could distribute and maintain many Windows computers from afar.

Importantly, Chocolatey packages do not require code-signing and will not trigger any warnings when installed. Adding package to the Chocolatey package store is decently simple. Chocolatey deals in .nupkg-es, which are easily created with the Chocolatey CLI. Execute choco new to generate a base set of files. After modifying them, you package up everything into the .nupkg container using choco pack. Ebbflow's chocolatey repo can be found on GitHub. Be sure to remove configuration entries or files that you don't need. And for referencing the actual installer for download, we just use the GitHub Release artifact that we post, all thanks to our earlier decision to build on GitHub.

Once you submit your package to Chocolatey, it goes through two automated steps of validation and it will spit out some advice that can be required and block your package from being released or it will spit out recommended changes like adding specific links or documentation. If these pass, the package enters 'moderation', and an actual human will review the package before it is released. This can take days to weeks, and is a drawback. Chocolatey does have Business plans, but to my knowledge these do not come with quicker package moderation.

Further, any new versions of a package after the initial moderation will need to be human-moderated. Your package can become 'Trusted' to allow new versions to bypass the human moderation, but this takes several complaint/issue-free versions of your application to be released.

Winget

Windows is working on a 1st party package manager named winget. Its exciting and adding your program to the community repo is simple and will feel familiar if you've worked with MacOS's Homebrew system. You must author a 'manifest' which points to your .msi and includes things like a description and a SHA256 hash. You can view Ebbflow's Winget configuration on GitHub.

Like Chocolatey, you do not need to code-sign your package and will not trigger any warnings on installation. Also like Chocolatey, there is a human moderation/approval process that is needed for each package and subsequent version. I believe this will change in the future when winget adds support for other sources. At that point, I assume you would have full control of your package and users will just need to add your Source to their source list. winget is promising and I'm looking forward to seeing it develop and march towards release in 2021.

Conclusion

The Rust language and 1st party tools facilitate a quick and smooth transition from code running only using cargo run to a fully built and distributed program. The static linking support is simple. The cargo tool is a game changer; it is used for dependency management which is a major headache for most languages. Cargo subcommands are a joy to use and made integration with Wix dead simple. The cargo wix subcommand allows developers to stay within the Rust ecosystem and still build and compile their program idiomatically through cargo.

Vending a package to Windows is overall a nice experience. The tooling works well and there is a good amount of online resources available such as prior-art so to speak on GitHub and other sources. Also, general documentation and StackOverflow are widely available. The biggest sticking points were anything with Wix, working with the 'Service' framework, and waiting for Chocolatey and winget packages to be approved. Non-Windows developers will find it easy to get started building and vending Rust code thanks to the language's built in support and through the numerous helpful community provided projects.

Thanks for taking the time to read this! If you'd like to check out Ebbflow you can use the free trial without providing a credit card! Simply create an account and get started. It takes six minutes to register, install the client, and host your website!