Welcome to SYSTEM•ILLUMINATION! This is the first illumination I have written and the one that prompted me to start this space. This first session tackles several topics as you join me on the journey I took to debug Rustler misbehaving on OmniOS.
I'm a beginner with illumos. This page serves a twofold purpose: to help me document and clarify my learnings as I delve into the illumos/Solaris world. And to shine a bit of light into a system that is fairly obscure and hard to get good info on. However, because I'm also learning it, I may be wrong on many things. Use the information in these pages as a starting point for your own investigations.
From Linux to illumos
Recently I decided I wanted to delve deeper into the illumos ecosystem; I had briefly used it a couple of years ago for a small server I was running. Two weeks ago, I had to decide on and build the final infrastructure for Katarineko a personal project I'm about to release. I had the choice between using my trusted Linux or giving in to the voice in my head that said USE ILLUMOS!. I did give in, maybe not the sanest choice, but it is my choice.
Katarineko is built with Elixir, a language that runs on the Erlang virtual machine. For reasons I will not get into here, I also have a couple of NIFs written in Rust. NIFs are the main way you can call into compiled languages from Erlang/Elixir. The Rustler project is a very straightforward way of integrating NIFs written in Rust, it handles compilation lifecycle, exposing all the right structures and functions, and serialization between Erlang and Rust, including support for Serde. We'll get into how this works a bit further down.
Migrating was quite simple. I used pkgsrc to get Elixir packages, set up some zones, and I was up and running within a day. At first, everything appeared to be running properly: there were no errors, and the page was running smoothly. I shared the beta version with a friend that was testing and he tried to use the Spaced-Repetition system. :erlang.nif_error(:nif_not_loaded) . The Sun gods had provided an opportunity to explore the famous observability tooling of Solaris earlier than I thought. Let's get right to it!
dtrace primer
One reason for moving to illumos was to learn dtrace(8) . dtrace is a dynamic tracing framework to observe the state and evolution of the system and user programs. We will use it extensively during this debugging session, so it is better to become acquainted with its syntax and capabilities. The D language is a mix between awk and C. Let's check an example:
#!/usr/sbin/dtrace -s syscall::open:entry , syscall::openat:entry 1 /pid == $target/ 2 { printf( "target used open syscall on %s
" , copyinstr(arg0)); 3 }
... continue reading