laumann.xyz: How to derive Show

*_Disclaimer_* All the code quoted in this post is extracted from the Rust compiler source code. Most snippets are annotated with the file name relative to @$RUST_ROOT/src@. Rust is an awesome language. In the beginning I had a lot of trouble coming to terms with some of the decisions taken in the language specification. But a little over a year later, I must admit that I am enjoying it. To be honest, it is probably not only the language itself, but the community around it - there are a lot of opinions, but the general tune is "we want to make the best systems programming language possible". And that, of course, will leave some less content and some very content, but all in all I there is a lot of excitement around it. Recently, I have been interested in writing compiler plugins, as I think they will come in handy for my Master's project. The documentation is a little sparse, only grazing the surface, but it's probably for the better as that whole section of the compiler is still marked as unstable. On the other hand, I doubt it will change drastically before 1.0, as quite a few projects make use of it as it is (to great effect I might add). Looking through the Rust source code, trying to learn about macro expansions, I naturally find a definition of the different types of syntax extensions available in Rust (file: @libsyntax/ext/base.rs@), defined as @enum SyntaxExtension@: * @Decorator@: A syntax extension attached to an item, creating new items based on it. * @Modifier@: Syntax extension attached to an item, modifying it in-place. * @MultiModifier@: Same as above, but more flexible (whatever that means) * @NormalTT@: A normal, function-like extension, for example @bytes!@ is one such * @IdentMacroExpander@: As a @NormalTT@, but has an extra @ident@ before the block. * @MacroRulesTT@: Represents @macro_rules!@ itself. How interesting. Then a question popped up: *How is a standard derivable trait such as @Show@ actually derived?* First of all, in the same file, there is a function defining all the basic syntax extensions, @initial_syntax_expander_table()@ in which we find the following lines:

fn initial_syntax_expander_table(ecfg: &expand::ExpansionConfig) -> SyntaxEnv {
    // ...
    let mut syntax_expanders = SyntaxEnv::new();

    // ...
    syntax_expanders.insert(intern("derive"),
                            Decorator(box ext::deriving::expand_meta_derive));
    // ...
}

which tells us that the "derive" functionality is registered as a decorator (which makes sense), and expands to call the function @expand_meta_derive()@. This function is defined in @libsyntax/ext/deriving@ and is not much different from any other syntax extension.

// File: libsyntax/ext/deriving/mod.rs
pub fn expand_meta_derive(cx: &mut ExtCtxt,
                          _span: Span,
                          mitem: &MetaItem,
                          item: &Item,
                          mut push: Box<FnMut(P<Item>)>) {
    // ...
}

First it checks the node type of @mitem@. If it is not a list or an empty list, an error is emitted. Otherwise all the items are inspected in turn. This gives us exactly what @derive@ can derive:

// File: libsyntax/ext/deriving/mod.rs in function expand_meta_derive()
match tname.get() {
    "Clone" => expand!(clone::expand_deriving_clone),

    "Hash" => expand!(hash::expand_deriving_hash),

    "RustcEncodable" => {
        expand!(encodable::expand_deriving_rustc_encodable)
    }
    "RustcDecodable" => {
        expand!(decodable::expand_deriving_rustc_decodable)
    }
    "Encodable" => {
        cx.span_warn(titem.span,
                     "derive(Encodable) is deprecated \
                      in favor of derive(RustcEncodable)");

        expand!(encodable::expand_deriving_encodable)
    }
    "Decodable" => {
        cx.span_warn(titem.span,
                     "derive(Decodable) is deprecated \
                      in favor of derive(RustcDecodable)");

        expand!(decodable::expand_deriving_decodable)
    }

    "PartialEq" => expand!(eq::expand_deriving_eq),
    "Eq" => expand!(totaleq::expand_deriving_totaleq),
    "PartialOrd" => expand!(ord::expand_deriving_ord),
    "Ord" => expand!(totalord::expand_deriving_totalord),

    "Rand" => expand!(rand::expand_deriving_rand),

    "Show" => {
        cx.span_warn(titem.span,
                     "derive(Show) is deprecated \
                      in favor of derive(Debug)");

        expand!(show::expand_deriving_show)
    },

    "Debug" => expand!(show::expand_deriving_show),

    "Default" => expand!(default::expand_deriving_default),

    "FromPrimitive" => expand!(primitive::expand_deriving_from_primitive),

    "Send" => expand!(bounds::expand_deriving_bound),
    "Sync" => expand!(bounds::expand_deriving_bound),
    "Copy" => expand!(bounds::expand_deriving_bound),

    ref tname => {
        cx.span_err(titem.span,
                    &format!("unknown `derive` \
                             trait: `{}`",
                            *tname)[]);
    }
}

Straight from the heart (or kidney) of the beast! Not only do we clearly see that @Show@ is supported, mapping to the function @expand_deriving_show@, we also see that it comes with a deprecation warning, and we should prefer @Debug@ over @Show@. At the moment there is no difference, as they both map to the same function. We are getting close to the end here. Instead of explaining what goes on I am going to quote the entire function @expand_deriving_show@:

// File: libsyntax/ext/deriving/show.rs
pub fn expand_deriving_show<F>(cx: &mut ExtCtxt,
                               span: Span,
                               mitem: &MetaItem,
                               item: &Item,
                               push: F) where
    F: FnOnce(P<Item>),
{
    // &mut ::std::fmt::Formatter
    let fmtr = Ptr(box Literal(Path::new(vec!("std", "fmt", "Formatter"))),
                   Borrowed(None, ast::MutMutable));

    let trait_def = TraitDef {
        span: span,
        attributes: Vec::new(),
        path: Path::new(vec!["std", "fmt", "Debug"]),
        additional_bounds: Vec::new(),
        generics: LifetimeBounds::empty(),
        methods: vec![
            MethodDef {
                name: "fmt",
                generics: LifetimeBounds::empty(),
                explicit_self: borrowed_explicit_self(),
                args: vec!(fmtr),
                ret_ty: Literal(Path::new(vec!("std", "fmt", "Result"))),
                attributes: Vec::new(),
                combine_substructure: combine_substructure(box |a, b, c| {
                    show_substructure(a, b, c)
                })
            }
        ],
        associated_types: Vec::new(),
    };
    trait_def.expand(cx, mitem, item, push)
}

This is beautiful! Deriving @Show@ looks a lot like we had written it by hand. We have a trait definition for @std::fmt::Debug@ with no additional bounds nor generics. There is one method called @fmt@ that takes @&self@ (borrowed explicit self) and a pointer to a @std::fmt::Formatter@ as arguments. The return type is @std::fmt::Result@. This is not the end however, since this does not mention the name of the structure we are trying to derive @Show@ for. This must take place in @trait_def.expand()@. This function expands the trait definition, ensuring that the derived-upon item is either a struct or an enum, taking care of various possible error conditions, juggling lifetimes, generics, @where@ clauses and associated types. All this boils down to the following item creation:

cx.item(
    self.span,
    ident,
    a,
    ast::ItemImpl(ast::Unsafety::Normal,
                  ast::ImplPolarity::Positive,
                  trait_generics,
                  opt_trait_ref,
                  self_type,
                  methods.into_iter()
                         .map(|method| {
                             ast::MethodImplItem(method)
                         }).chain(
                             associated_types.map(|type_| {
                                 ast::TypeImplItem(type_)
                             })
                         ).collect()))

which I will not even pretend to understand. We can conclude that we end up calling @cx.item@, creating a new item in the AST. The @item()@ method is not defined on @ExtCtxt@ itself, but rather declared in a trait @AstBuilder@, which is implemented for @ExtCtxt@.

// File: libsyntax/ext/build.rs
impl<'a> AstBuilder for ExtCtxt<'a> {
    // ...
    fn item(&self, span: Span, name: Ident,
            attrs: Vec<ast::Attribute>, node: ast::Item_) -> P<ast::Item> {
        // FIXME: Would be nice if our generated code didn't violate
        // Rust coding conventions
        P(ast::Item {
            ident: name,
            attrs: attrs,
            id: ast::DUMMY_NODE_ID,
            node: node,
            vis: ast::Inherited,
            span: span
        })
    }
    // ...
}

So there you have it. How @Show@ (or @Debug@) gets derived in Rust. It is a rather long story, with some gaps, but it is very instructive to skip around the compiler infrastructure to see how some of the AST-mangling syntax extensions do their work. If you stuck with it this far, thanks for reading, hope you enjoyed it.