How to write Rust in your day job, as a Python Developer

Everytime I attend the rust meetup in Copenhagen, I talk to a lot of cool people of whom no one actually write rust in their day job. This is my attempt to change this.

I get paid to work as a Data Engineer. In that capacity my clients usually want me to one or more of the following:

• Write notebooks or scripts in python

• Write a bunch of Stored Procedures in SQL

• Use some legacy drag and drop tool, to define data pipelines (I’m looking at you SQL Server Integration Services)

What I’ve never been paid to do is to write rust. Honestly if I start talking about rust with most of my clients, they’ll start thinking of corroding metals, and definitely not the programming language. They might even think about checking their car for rust.

Which is a shame, because I love writing rust. Generally I think there are 3 main reasons why Rust is a great language.

1. Performance: Rust is a low level language, like C and C++. Therefore modules written in rust, will perform a lot better than native Python code. Which is one of the reasons why ruff and uv are so fast

2. Types: Unlike Python, Rust has a great and very expressive static type system. I’m not necessarily one, to hate on the dynamic typing of Python. However for some workloads, the type system, which Rust provides can’t be beat. I imagine that this is one of the reasons that Pydantic is written in rust.

3. Fun: Writing Rust is fun, and sometimes we should just do stuff which we like to do.

So how do I get to write it in my day job. I’m currently building a product which can translate SQL Server Integration Services (SSIS) packages to Python. This product, is primarily written in Python, because it is a language that both I and my technical cofounder are productive in.

As part of that work, I need to find a way to split large T-SQL MERGE Statements into its individual components. For this there is a great rust package called sqlparser, which can take a SQL Statement and return the AST (Abstract Syntax Tree). This is great, because the AST contains all the information we need to split a MERGE statement into its individual components. The challenge is, that the whole program is written in python, and we would not want to rewrite it to rust, just to use this library.

Instead what we can do is to just write this small part which splits the MERGE statement into its individual components in rust, and then expose it as a library which can be imported into our Python program. The great side effect of this is, that I can finally say that I use Rust in my day job.

PyO3 making Rust/Python interoperability easy

Great let’s look into how we can do this. The first thing I did was to head over to PyO3’s user guide. PyO3 is a tool, which can be used build your rust library into a python package. In the user guide they advice you to use the Maturin build system, so that is what I’m going to do.

To install Maturin globally, I run the following command

pip3 install maturin

Once maturin is installed, the setup process, is fairly simple. Just follow these steps:

1. Create a folder where I want to have my package

2. Initialise a python virtual environment

3. Run maturin init and chose the pyo3 template.

As you can see we now have a pyproject.toml file with our python configurations. It should look something like this:

We also get a Cargo.toml file, which looks like this.

We can now get to the important part. Writing our rust library. In this blog post, I won’t go to much into detail the actual code. But if you want to follow along, make sure, the file src/lib.rs has the following content.

As you might have noticed our code looks a lot like your traditional rust library, however there are a few macros, you might not have seen before.

The first you notice is the #[pyfunction] macro. It tells pyO3 that we want this code to exposed as a function in python.

The second is #[pymodule], this sends a module definition to PyO3, and declares which functions should be included.

The last part which you might notice are when we derive the IntoPyObject traits. Because we are sending back our own custom struct, we need to tell PyO3 that it should be converted to a python object.

This, plus our logic of course, is all we need to write our first python package in rust.

Running the maturin develop command is enough to build the project and place an installed package in our virtual environment.

The only thing left is to test if it works. Let’s create a main.py file which uses the new library:

Once it is saved we can inspect the result.

That’s it we’ve now built our first python package, and can finally be part of the cool club (I think).

But how does it work?

When we write a library in rust and compile it, it uses the Rust ABI (Application Binary Interface). An ABI is way for two binary programs to interact with each other. However Python (and a lot of other non rust programs), do not know how to talk to Rust Libraries using the Rust ABI.

What Python does have, is something called the Python/C API, which enables us to extend python with C. So instead what PyO3 does are the following:

1. Compile the library using the local system ABI, by using extern “C”

2. Ensure that the new library abides by the Python/C API, so it acts like any package written in C or C++

This is logic is hidden from us using macros. Running cargo expand on our lib.rs file, can show the code generated by the macro.

Looking at the module definition for our merge parser. We get a better idea of how PyO3 implements this under the hood.

Why did I write this?

I wrote this because, I myself have often been a victim of my own all or nothing mentality. Either I write rust or python. Nonetheless this experience has taught me, to be more open to mix languages in a project. I learned a lot from doing it, and even made my first contribution to open source (read more here: My first experience with open source). If you read all the way to here, I hope you found it interesting. Leave a comment on what you would like me to cover next.