2020-11: AST Library Choice
When we first started working on the Azure Service Operator Code Generator, we used the standard go
ast library to create the final Go code.
Unfortunately, we have run into a number of significant limitations with this library.
Comments are often emitted in unexpected places.
They would often appear at the end of the previous line, which conveys an incorrect meaning to a casual reader of the code. As a workaround, we found that prefacing some comments with newline characters would force them onto the next line. While hacking the library in this function worked, we are concerned that it’s a fragile technique that would break in a future version of Go. We also found situations where this technique did not work.
Generated code does not always comply with
For some constructs, the generated code would fail the
go fmtcheck included in our continuous integration (CI) builds. We found a workaround that involved using
astto read in a generated file and immediately rewrite it back out again, immitating the results of
Generated code does not always compile.
For some constructs, we found the comments were emitted at the start of the line of code (not on the previous line), resulting in the code being commented out. We found no workaround for this.
Poor control over whitespace
We were unable to find a reliable technique within
astto introduce blank lines betweens stanza of code, or before comment blocks. As a workaround, we would read in the formatted code, scan it for comments and manually inject blank lines before each comment block.
We have come to the conclusions that the
ast library is intended as a way to make minor modifications to existing well formatted Go files, not for the creation of entirely new Go files from scratch.
A potential alternative is the
dst (Decorated Syntax Tree) package. This package was specifically created to address the kinds of issues described above.
After some informative trials, we have decided to adopt the
Initial change committed in PR#366.
With a very high level of API compatibility, we were able to introduce
dst with a low level of code churn by aliasing the import as
ast in most files. Future changes should remove this aliasing as files are modified.
The DST library requires that any node be used exactly once, and will panic if a node is reused in mulitiple locations. We’ve mitigated this by liberally cloning nodes as we build the desired final syntax tree. (See
astbuilder.Expressions(), introduced in PR#1613 and
astbuilder.Statements(), introduced in PR#427).
Go AST (Abstract Syntax Tree) library
DST (Decorated Syntax Tree) library