Meta-Programming and parser extensibility in C# and Roslyn

While Roslyn is a fantastic framework to parse/analyse/modify C# code, the lack of extensibility in the parser part is really frustrating in meta-programming scenarios.

This is of course not the entire fault of Roslyn itself, but more a missing feature in C#, but it would be really powerful to be able to extend the language syntax (at several possible - fixed - points into the AST tree), without having to write a full parser on our own. I know that traditional lexer/parser are not always friendly for this kind of extension points, but I have several situations where I would like to provide custom embedded DLR in C# (as it is the case for Linq syntax, or async/await), and this is currently not possible without going through the laborious way of rewriting a whole C# parser/analyser (or using existing Mono as a starting point).

We really need a way to express in C# itself language extension that can be consumed by the parser to generate transformed C# code that will get compiled. Extensions languages like async/await or linq expressions should be able to be expressed into this meta-programming C# language.

Let see how this could be done. Suppose that we would like to introduce a new keyword "mykeyword" which takes a block expression ({...}), and this is supported at the class and struct body level. I would declare this syntax extension like this:

using Microsoft.CSharp.LanguageExtensions;
public static class MyLanguageExtension
{
    /// <summary>
    /// Creates an new syntax "mykeyword + { ... }" at the Class and Struct body level.
    /// </summary>
    /// <returns>A syntax declaration describing the language extension</returns>
    [SyntaxExtension]
    public static SyntaxDeclaration Create()
    {
        return new SyntaxDeclaration {
            // Defines a BNF like syntax
            Syntax = new SyntaxToken("mykeyword") + new SyntaxBlockExpression(), 

            // Defines where this syntax is accepted
            Location = SyntaxLocation.ClassBody | SyntaxLocation.StructBody, 

            // Callback after the AST tree is built, in order to provide code transformation into pure C# code.
            AstHandler = MyKeywordHandler
        };
    }

    public static void MyKeywordHandler(SyntaxDeclaration syntaxDeclaration, CompilerContext context)
    {
        // Implements here modification to the ast through the compiler context that will contain the AST tree
        // This methods typically replace the AST tree node "mykeyword" with some static fields, methods... whatever
        // in the class to translate it into pure C# language.
    }
}

And a typical usage of this new keyword would then be:

public class MyClass
{
    // Use the keyword previously introduced at the class body level
    mykeyword { Console.WriteLine("This is an extension method at the class body level");}
}

The AstHandler would then generate appropriate pure C# code in order to translate this embedded DLR declaration into the C# class itself. There are lots of limit cases to take care, and viable extension points to identify (for example, It would even be nice to be able to add new modifiers, like public/private, for whatever syntax elements...etc), but there is nothing that would prevent this kind of scenario to be provided.

Having this kind of extensibility would be a major enhancement, as It would be possible to leverage on C# to include some embedded DLR, but with having the full parsing experience supported (External tools like Resharper would even be able to consume language extensions and provides them in their syntax analysis/highlighting).

One technical challenge is that the parser should support extensible syntax declaration points. This is absolutely feasible, but most regular parsers (like lex/yacc) generate statics DFA states with little or no extension points, but this should not be a problem to do it, as It is fairly easy to add extension points into a DFA. The only downside is that the parser would have to verify that the language extension is not generating an invalid dynamic DFA at compiling time, but I don't think this is a major issue.

I would love to see this kind of things implemented in C#, as much as I love to work in C#!

Let me know what you think about this.

Alexandre Mutel - SharpDX - NShader - Code4k