Background: Ebbflow is a multi-cloud load balancer that provisions browser-trusted certificates for your endpoints in addition to providing a client-friendly SSH proxy. Servers can host endpoints from any cloud, on premises, or from your home, all at the same time. It uses nearest-server routing for low latencies and is deployed around the globe.
This post describes how Ebbflow vends its client which is written in Rust to its Linux users, describing the tools used to build the various packages for popular distributions. In a future post, we will discuss how these packages are ultimately vended to users.
Ebbflow vends the client to numerous distributions of Linux based operating systems. Linux is free, flexible, and is widely used in the software community and is being used increasingly in consumer markets. Ebbflow uses Linux for development machines and production servers.
To vend software to Linux users, you must consider the Distribution that the users may be running. For non-Linux users, a distribution can be thought of as a version or flavor of an OS. Each distribution provides package management systems and backround-service management software which may be different, adding complication to our task of vending our client.
This guide will talk about the problems that need to be solved after the core code has been written and all that's left to do is get it into users' hands. So, given that we have our binaries, here are the main tasks that we must work on:
Ebbflow Client 101
Ebbflow's job is to route user traffic to your web server application (or SSH Daemon) on your servers. The central Ebbflow service proxies data between user connections (e.g. browser, SSH client) and the client. The client then proxies the data to your web server or local SSH daemon. Users of Ebbflow install the client on machines that will host endpoints, be SSH-ed to, or both at the same time (which is very useful). The following diagram shows this visually.
Quick note: The client initiates the connection to the central Ebbflow service via an outbound TLS connection, which makes the client extremely firewall friendly. You may completely block all inbound connections but still host websites or be SSH-ed to! Also, this innovation allows the servers to be located in any network, and even change networks without any configuration at all. Ebbflow just receives connections from the general internet, and after authentication and authorization, will allow the server to host the endpoint or be SSH-ed to. Neat!
The client has two parts, the CLI and the background daemon. Both of these programs are 100% Rust, 100%
async, and 100% 'safe', and statically linked - the dream of any Rust developer! The CLI is an executable tool named
ebbflow that is used to tell the background to host new endpoints, disable or re-enable endpoints or the SSH proxy, and configure other settings.
The background daemon is the second piece to this puzzle, and is the workhorse and is responsible for actually transferring bytes between the central Ebbflow servers and your local web server or SSH daemon. This daemon is just a long-running background executable named
ebbflowd needs to run in the background, run without a logged-in user, start on system boot or reboot, and be started again if it crashes. (most) Linux distributions provide a mechanism for this called
systemd, which is a system to manage and execute "services". When
systemd manages a service, it will start it, watch it, and fulfulls all of the needs we have.
systemd uses "unit files" as the means to know how a service should be handled. They can be very simple, and you can see Ebbflow's unit file on GitHub. The unit file points to the executable, declares a dependency that the system's networking services should be initialized first, and that the program should be restarted quickly if it exits. There are many helpful guides for
systemd online from general info to more reference type guides.
Below is the actual
ebbflowd.service unit file used by Ebbflow.
Simple, short, and gets the job done! The daemon will be restarted after one second, and will execute
/usr/sbin/ebbflowd on our users' behalf.
If you are making your own, you can get started quickly by changing
ExecStart to your built program in whatever directory it lies in now. To run the service, we must do the following
- Add the unit to
systemd's known services
sudo cp myprogram.service /etc/systemd/system/
- Enable the unit so
systemd will start it on boot, registering the unit file
sudo systemctl enable myprogram
systemd to start your program
sudo systemctl start myprogram
- Check that its running!
sudo systemctl status myprogram
- Stop the program
sudo systemctl stop myprogram
After writing and testing your unit file you will need to either instruct users to complete the above steps or, preferably, vend your Unit in a proper package so that the users can avoid managing the unit file themselves. Ebbflow's unit file is vended through the built packages which we describe later.
Rust will dynamically link against the OS's
libc implementation when the standard library is used, which is the case for almost all substantial Rust applications. When compiling Rust on a Linux system Rust will dynamically link against
glibc by default.
For background, linking against
glibc is not necessarily a bad thing - you can reduce your binary size and benefit from any performance or security updates without changing your code. However, when vending your project to users, linking against something that is not under you control is not ideal. First off, when linking, you will link against a specific version of
glibc - the version that your build system has. So let's look at a ficticious example where you build on Ubuntu 20.04 (which uses
2.31) and vend to Ubuntu 18.04 (which uses
2.27). If you build on your Ubuntu 20.04 system and
rustc uses some feature of
2.31 that is not present in
2.27, and you execute your code on the Ubuntu 18.04 system, your code may break!
Edit: Per reddit commenter /u/STEROVIS, the above statement is innacurrate, and instead of "breaking", the "program simply will refuse to run at all whether you use a nonexistent feature or not. The dynamic linker will instantly fail complaining about a missing
GLIBC_* version symbol". Still, the program will not work!
You could avoid this scenario by building and linking to a low
glibc version, but how low? How far back should you go? What if you want to vend to users of very very old systems?
For maximum flexibility and to gain ground against our goal of working on many distributions, you can avoid these problems by statically linking
libc implementation that
rustc can include in your built binary so your program can run without any
glibc dependencies. To get started using
MUSL, you add a new target to
rustup by executing the following. Also note that you may need to install a musl package, for example on Debian execute
apt-get install musl.
# Add new MUSL target to rustup
$ rustup target add x86_64-unknown-linux-musl
$ cargo build --target x86_64-unknown-linux-musl
std library, Rust code may link to other OS libraries most often OpenSSL or libsodium for crypto. The Ebbflow client avoids this by using
rustls which uses ring under the hood. Rustls is an ergonomic TLS library for Rust. It is highly performant even compared to OpenSSL and recently underwent a 3rd party security audit which showed no flaws. When you invest in the Rust ecosystem and use Rust-written libraries, you are rewarded with the ability to statically link which is a desirable situation.
To this point, we've taken our project and registered the backround daemon with the OS using
systemd and statically linked our code using
MUSL. The client could be tested to run on a single machine, but there are many installation steps. From here, we will package up our daemon and unit file nicely using packages.
Each Linux distribution provides a Package Manager which is used to manage the installed software, called packages, on the system. Most distributions use the
.rpm package formats.
.deb packages are used by the Debian distribution and its derivitatives such as Ubuntu.
.rpm packages are used by other distributions such as OpenSUSE, Fedora, and CentOS. Each distribution may have a different CLI tool for managing packages, even if they use the same package format, but that problem will be solved in the next blog post.
To create a package, you first create a format-specific specification or configuration file. Once your specification is created you use the format-specific build tool to build the actual packages, in our case a
Building Debian packages for a Rust project is dead simple using the
cargo-deb crate, which is used as a cargo subcommand. This command will create a
.deb package from your Rust project and is highly configurable - check out the README for all of the options.
cargo deb is simple, so simple that you don't actually need to touch any configuration at all if you have a simple Rust project: you can just execute
cargo deb and the tool will infer all of the required settings and build your package!
The Ebbflow client is more complicated, so we will provide configuration through the client's
Cargo.toml file. Here is a snapshot of what that looks like:
There are three interesting components to this. The first is the
conf-files list. In general, configuration files are different than other files that may be vended in a package - they are typically changed after installation, but not changed between versions, and are user-specific. This is opposed to the binary executables which typically change between versions and are not user-specific. Package managers strive for integrity by hashing known files for a package, and checking these hashes during uninstallation and upgrades. If an upgrade occurs and a file has an unexpected hash, the user will be prompted to resolve the issue.
To deal with the fact that configuration files are special, the Debian package specification allows you to specify configuration files which will not be checked for integrity. This avoids bugging the users during upgrades which is desirable. Long story short, if you have configuration files, inform the package manager by listing the file in your package build!
The second interesting item is the
maintainer-scripts item. This points to some scripts which are used to control the installation of the package. Specifically, the scripts will register the
systemd unit of ours, start the service, and stop it when uninstalled. For more information about this, see the discussion of services in the
Lastly, the third intesting item is the
assets section. This simply tells
cargo deb what files should be copied to our final package, to which location, and their respective unix permissions.
cargo deb will build the
.deb and you can
sudo apt install ./path/to/.deb your package on your local system to test it out!
Like Debian packages, it is super easy to create
.rpm packages for Rust projects using a helpful cargo subcommand, in this case,
cargo-rpm. To get started in a new project, execute
cargo rpm init to create a basic
.spec file. Under the hood,
cargo rpm will pass the
rpmbuild which builds the final
.rpm file. A reference for the
.spec file can be found here.
cargo rpm use the
.spec file alongside any changes to your
Cargo.toml when building your package. Writing the
.spec file is a little more intimidating than the
.deb configuration, but quite simple once you understand how it works. The Ebbflow client's
.spec can be found on GitHub, but a snapshot is below. Besides the spec, we will also change our
First, the changes to our
Above, we state that we would like the
cargo rpm tool to use the
--release flag and to target
x86_64-unknown-linux-musl so we statically link our binaries. The next section informs
cargo rpm that we have two binaries that should be taken from the build output, namely
ebbflowd. Lastly, the final section lists some other files that will be brought into our RPM build environment, which will be referenced in the
.spec file. These files are all placed in the
.rpm directory of our Rust project.
Now let's look at the
Much of this was auto-generated from running
cargo rpm init. The major changes needed were to include the
ebbflowd.service file (which we referenced in our changes to
Cargo.toml) and to list our configuration files. Rpm configuration files act like Debian configuration files, which we discussed earlier in the Debian package section.
cargo rpm build will build the
.rpm file based on our various configuration items and spit out the package. Note that this will need to be completed on an RPM based distribution or you will need to figure out how to grab the
rpmbuild tool and execute that on your non-RPM-based distribution.
One more note is guidelines dictate that you may NOT initiate/start your
systemd service on installation if the service requires manual configuration, which the Ebbflow client does. For this reason, Ebbflow informs users to execute
sudo systemctl enable --now ebbflowd after they have initialized/configured the client by executing
sudo ebbflow init.
Going Forward: Automated Builds & Vending to Users
We have the packages, they're statically linked and ready to go, but the final hurdles are to build the packages in a sane way and to deliver the packages through the users' package management systems. These topics are reserved for an upcoming blog post! These will be posted to /r/rust, Hacker News, or feel free to send an email to firstname.lastname@example.org with subject "blog subscribe" (or something similar) to have blog posts be emailed to you!
cargo deb and
cargo rpm building Linux packages is dead simple for Rust projects. These two projects are invaluable for working with
.rpm packages. More documentation would always be lovely for these tools, but thankfully the package formats themselves are old and there are numerous examples and helpful resources online to help with your configuration.
The Ebbflow client was developed over a handful of months and underwent major changes. It was first just a CLI, then re-written to have the daemon and CLI frontent. There was actually little forethought regarding the packaging process during the re-write; the re-write got finished and then we turned to packaging it up. The Cargo Deb crate and Cargo Rpm crates came to the rescue and made packaging very simple. The trickiest bit was getting the systemd files to work and automatically start up, but there are enough resources online, and now the Ebbflow client is another example of how to get that running, but specifically in Rust!
Thanks for taking the time to read this! If you'd like to check out Ebbflow you can use the free trial without providing a credit card! Simply create an account and get started. It takes six minutes to register, install the client, and host your website!