Skip to content

Blog

Control Flow Graph for FAST Fortran

A Control Flow Graph analysis for FAST Fortran

Section titled “A Control Flow Graph analysis for FAST Fortran”

Control Flow Graphs (CFG) are a common tool for static analyzis of a computation unit (eg. a method) and find some errors (unreachable code, infinite loops)

It is based on the concept of Basic Block: a sequence of consecutive statements in which flow of control can only enter at the beginning and leave at the end. Only the last statement of a basic block can be a branch statement and only the first statement of a basic block can be a target of a branch.

There are two distinctive basic blocks:

  • Start Block: The entry block allows the control to enter into the control flow graph. There should be only one start block.
  • Final Block: Control flow leaves through the exit block. There may be several final blocks.

The package FAST-Fortran-Analyses in https://github.com/moosetechnology/FAST-Fortran contains classes to build a CFG of a Fortran program unit (a main program, a function, or a subroutine).

We must first create a FAST model of a Fortran program. For this we need an external parser. We currently use fortran-src-extras from https://github.com/camfort/fortran-src-extras.

To run it on a fortran file you do:

fortran-src-extras serialize -t json -v77l encode <fortran-file.f>

This will produce a json AST of the program that we can turn into a FAST-Fortran AST.

If you have fortran-src-extras installed on your computer, all this is automated in FAST-Fortran

<fortran-file.f> asFileReference
readStreamDo: [ :st |
FortranProjectImporter new getFASTFor: st contents ]

This script will create an array of ASTs from the <fortran-file.f> given fortran file. If there are several program units in the file, there will be several FAST models in this array. In the example below, there is only one program, so the list contains only the AST for this program.

We will use the following Fortran-77 code:

PROGRAM EUCLID
* Find greatest common divisor using the Euclidean algorithm
PRINT *, 'A?'
READ *, NA
IF (NA.LE.0) THEN
PRINT *, 'A must be a positive integer.'
STOP
END IF
PRINT *, 'B?'
READ *, NB
IF (NB.LE.0) THEN
PRINT *, 'B must be a positive integer.'
STOP
END IF
IA = NA
IB = NB
1 IF (IB.NE.0) THEN
ITEMP = IA
IA = IB
IB = MOD(ITEMP, IB)
GOTO 1
END IF
PRINT *, 'The GCD of', NA, ' and', NB, ' is', IA, '.'
STOP
END

From the FAST model above, we will now create a Control-Flow-Graph:

<FAST-model> accept: FASTFortranCFGVisitor new

The class FASTFortranCFGVisitor implements an algorithm to compute basic blocks from https://en.wikipedia.org/wiki/Basic_block.

This visitor goes throught the FAST model and creates a list of basic blocks that can be inspected with the #basicBlocks method.

There is a small hierarchy of basic block classes:

  • FASTFortranAbstractBasicBlock, the root of the hierarchy. It contains #statements (which are FAST statement nodes). It has methods to test its nature: isStart, isFinal, isConditional. It defines an abstract method #nextBlocks that returns a list of basic blocks that this one can directly reach. Typically there are 1 or 2 next blocks, but Fortran can have more due to “arithmetic IF”, “computed GOTO” and “assigned GOTO” statements.
  • FASTFortranBasicBlock, a common basic block with no branch statement. If it is final, its #nextBlocks is empty, otherwise it’s a list of 1 block.
  • FASTFortranConditionalBasicBlock, a conditional basic block. It may reach several #nextBlocks, each one associated with a value, for example true and false. The method #nextBlockForValue: returns the next block associated to a given value. In our version of CFG, a conditional block may only have one statement (a conditional statement).

You may have noticed that our blocks are a bit different from the definition given at the beginning of the blog-post:

  • our “common” blocs cannot have several next, they never end with a conditional statement;
  • our conditional blocks can have only one statement.

For the program above, the CFG has 10 blocks.

  • the first block is a common block and contains 2 statements, the PRINT and the READ;
  • its next bloc is a conditional block for the IF. It has 2 next blocs:
    • true leads to a common block with 2 statements, the PRINT and the STOP. This is a final block (STOP ends the program);
    • false leads to the common block after the IF

As a first analysis tool, we can visualize the CFG. Inspecting the result of the next script will open a Roassal visualization on the CFG contained in the FASTFortranCFGVisitor.

FASTFortranCFGVisualization on: <aFASTFortranCFGVisitor>

For the program above, this gives the visualization below.

  • the dark dot is the starting block (note that it is a block and contains statements);
  • the hollow dots are final blocks;
  • it’s not the case here, but a block may also be start and final (if there are no conditional blocks in the program) and this would be represented by a “target”, a circle with a dot inside;
  • a grey square is a comon block;
  • a blue square is a conditional block;
  • hovering the mouse on a block will bring a pop up with the list of its statements (this relies on the FASTFortranExporterVisitor)

"Viualizing the Control Flow Graph"

One can see that:

  • the start block has 2 associated statements (PRINT and READ);
  • there are several final blocks, due to the STOP statements;
  • there is a loop at the bottom left of the graph where the last blue conditional block is “IF (IB.NE.0)” and the last statement of the grey block (true value of the IF), is a GOTO.

There are little analyses for now on the CFG, but FASTFortranCFGChecker will compute a list of unreachableBlocks that would represent dead code.

Control flow graphs may also be used to do more advanced analyses and possibly refactor code. For example, we mentioned the loop at the end of our program implemented with a IF statement and a GOTO. This could be refactored into a real WHILE loop that would be easier to read.

This is left as an exercise for the interested people 😉

Building a control flow graph is language dependant to identify the conditional statements, where they lead, and the final statements.

But much could be done in FAST core based on FASTTReturnStatement and a (not yet existing at the time of writing) FASTTConditionalStatement.

Inspiration could be taken from FASTFortranCFGVisitor and the process is not overly complicated. It would probably be even easier for modern languages that do not have the various GOTO statements of Fortran.

Once the CFG is computed, the other tools (eg. the visualization) should be completely independant of the language.

All hands on deck!

Some tools on FAST models

The package FAST-Core-Tools in repository https://github.com/moosetechnology/FAST offers some tools or algorithms that are running on FAST models.

These tools may be usable directly on a specific language FAST meta-model, or might require some adjustements by subtyping them. They are not out-of-the-shelf ready to use stuff, but they can provide good inspiration for whatever you need to do.

Writing test for FAST can be pretty tedious because you have to build a FAST model in the test corresponding to your need. It often has a lot of nodes that you need to create in the right order with the right properties.

This is where FASTDumpVisitor can help by visiting an existing AST and “dump” it as a string. The goal is that executing this string in Pharo should recreate exactly the same AST.

Dumping an AST can also be useful to debug an AST and checking that it has the right properties.

To use it, you can just call FASTDumpVisitor visit: <yourAST> and print the result. For example:

FASTDumpVisitor visit:
(FASTJavaUnaryExpression new
operator: '-' ;
expression:
(FASTJavaIntegerLiteral new
primitiveValue: '5'))

will return the string: FASTJavaUnaryExpression new expression:(FASTJavaIntegerLiteral new primitiveValue:'5');operator:'-' which, if evaluated, in Pharo will recreate the same AST as the original.

Note: Because FAST models are actually Famix models (Famix-AST), the tools works also for Famix models. But Famix entities typically have more properties and the result is not so nice:

FASTDumpVisitor visit:
(FamixJavaMethod new
name: 'toto' ;
parameters: {
FamixJavaParameter new name: 'x' .
FamixJavaParameter new name: 'y'} ).

will return the string: FamixJavaMethod new parameters:{FamixJavaParameter new name:'x';isFinal:false;numberOfLinesOfCode:0;isStub:false.FamixJavaParameter new name:'y';isFinal:false;numberOfLinesOfCode:0;isStub:false};isStub:false;isClassSide:false;isFinal:false;numberOfLinesOfCode:-1;isSynchronized:false;numberOfConditionals:-1;isAbstract:false;cyclomaticComplexity:-1;name:'toto'.

By definition an AST (Abstract Syntax Tree) is a tree (!). So the same variable can appear several time in an AST in different nodes (for example if the same variable is accessed several times).

The idea of the class FASTLocalResolverVisitor is to relate all uses of a symbol in the AST to the node where the symbol is defined. This is mostly useful for parameters and local variables inside a method, because the local resover only looks at the AST itself and we do not build ASTs for entire systems.

This local resolver will look at identifier appearing in an AST and try to link them all together when they correspond to the same entity. There is no complex computation in it. It just looks at names defined or used in the AST.

This is dependant on the programming language because the nodes using or defining a variable are not the same in all languages. For Java, there is FASTJavaLocalResolverVisitor, and for Fortran FASTFortranLocalResolverVisitor.

The tool brings an extra level of detail by managing scopes, so that if the same variable name is defined in different loops (for example), then each use of the name will be related to the correct definition.

The resolution process creates:

  • In declaration nodes (eg. FASTJavaVariableDeclarator or FASTJavaParameter),a property #localUses will list all referencing nodes for this variable;
  • In accessing nodes, (eg. FASTJavaVariableExpression), a property #localDeclarations will lists the declaration node corresponding this variable.
  • If the declaration node was not found a FASTNonLocalDeclaration is used as the declaration node.

Note: That this looks a bit like what Carrefour does (see /blog/2022-06-30-carrefour), because both will bind several FAST nodes to the same entity. But the process is very different:

  • Carrefour will bind a FAST node to a corresponding Famix node;
  • The local resolver binds FAST nodes together.

So Carrefour is not local, it look in the entire Famix model to find the entity that matches a FAST node. In Famix, there is only one Famix entity for one software entity and it “knows” all its uses (a FamixVariable has a list of FamixAccess-es). Each FAST declaration node will be related to the Famix entity (the FamixVariable) and the FAST use nodes will be related to the FamixAccess-es.

On the other hand, the local resolver is a much lighter tool. It only needs a FAST model to work on and will only bind FAST nodes between themselves in that FAST model.

For round-trip re-engineering, we need to import a program in a model, modify the model, and re-export it as a (modified) program. A lot can go wrong or be fogotten in all these steps and they are not trivial to validate.

First, unless much extra information is added to the AST, the re-export will not be syntactically equivalent: there are formatting issues, indentation, white spaces, blank lines, comments that could make the re-exported program very different (apparently) from the original one.

The class FASTDifferentialValidator helps checking that the round-trip implementation works well. It focuses on the meaning of the program independently of the formatting issues. The process is the follwing:

  • parse a set of (representative) programs
  • model them in FAST
  • re-export the programs
  • re-import the new programs, and
  • re-create a new model

Hopefully, the two models (2nd and last steps) should be equivalent This is what this tool checks.

Obviously the validation can easily be circumvented. Trivially, if we create an empty model the 1st time, re-export anything, and create an empty model the second time, then the 2 models are equivalent, yet we did not accomplish anything. This tool is an help for developers to pinpoint small mistakes in the process.

Note that even in the best of conditions, there can still be subtle differences between two equivalent ASTs. For example the AST for “a + b + c” will often differ from that of “a + (b + c)”.

The validator is intended to run on a set of source files and check that they are all parsed and re-exported correctly. It will report differences and will allow to fine tune the comparison or ignore some differences.

It goes through all the files in a directory and uses an importer, an exporter, and a comparator. The importer generates a FAST model from some source code (eg. JavaSmaCCProgramNodeImporterVisitor); the exporter generates source code from a model (eg. FASTJavaExportVisitor); the comparator is a companion class to the DifferentialValidator that handle the differences between the ASTs.

The basic implementation (FamixModelComparator) does a strict comparison (no differences allowed), but it has methods for accepting some differences:

  • #ast: node1 acceptableDifferenceTo: node2: If for some reason the difference in the nodes is acceptable, this method must return true and the comparison will restart from the parent of the two nodes as if they were the same.
  • #ast: node1 acceptableDifferenceTo: node2 property: aSymbol. This is for property comparison (eg. the name of an entity), it should return nil if the difference in value is not acceptable and a recovery block if it is acceptable. Instead of resuming from the parent of the nodes, the comparison will resume from an ancestor for which the recovery block evaluates to true.

A real example on using tags

Tags can be a powerful tool to visualize things on legacy software and perform analyses. For example, tags can be used to create virtual entities and see how they “interact” with the real entities of the system analyzed. In the article Decomposing God Classes at Siemens we show how tags can be used to create virtual classes and see their dependencies to real classes.

In this post I will show another use of tags: how they can materialize a concept and show its instantiation in a system.

The scenario is that of analysing Corese, a platform to “create, manipulate, parse, serialize, query, reason and validate RDF data.” Corese is an old software that dates back to the early days of Java. Back then, enums did not exist in Java and a good way to implement them was to use a set of constants:

public static final int MONDAY = 1;
public static final int TUESDAY = 2;
public static final int WEDNESDAY = 3;
public static final int THURSDAY = 4;
public static final int FRIDAY = 5;
public static final int SATURDAY = 6;
public static final int SUNDAY = 7;

Those were the days!

As an effort to restructure and rationalize implementation, the developers of Corese wish to replace these sets of constants by real Java enums. This is not something that can be done in any modern IDE even with the latest refactoring tool.

Let us see how Moose can help in the task.

For an analysis in Moose, we need a model of the system, and this starts with getting the source code (https://github.com/corese-stack/corese-core). The model is created using VerveineJ which can be run using docker:

docker run -rm -v src/main/java/:/src ghcr.io/evref-bl/verveinej:latest -alllocals -o corese-core.json

This will create a file corese-core.json in the directory src/main/java/. The command to create the model as an option -alllocals. This is because VerveineJ by default only tracks the uses of variables with non primitive type (variables containing objects). Here the constants are integers and if we want to know where they are used, we need more details.

Let’s import the model in Moose. This can be done simply by dragging-and-dropping the file in Moose.

"Importing the Corese model"

We will study the use of the constants defined in fr.inria.corese.core.stats.IStats:

public interface IStats {
public static final int NA = 0;
public static final int SUBJECT = 1;
public static final int PREDICATE = 2;
public static final int OBJECT = 3;
public static final int TRIPLE = 4;
[...]

To find where the constants are used, we need to find the representation of the constants in the model. For this, we can inspect the model (“Inspect” button in the Model Browser) and look for all “Model Attributes”. The constants are attributes of the interface/class in which they are defined as shown in the listing above). And they are model attributes because they are defined in the source code analysed, as opposed to System.out which may be used in the code but for which we don’t have the source code.

We can then select all the model attributes named PREDICATE: select: [ :each | each name = 'PREDICATE']. (note, the backslash (\) before the square bracket ([) was added by the publishing tool and is not part of the code)

Moose gives us 8 different definitions of PREDICATE (and 9 for OBJECT, and 10 for SUBJECT). The one we are interested in is the 3rd in the list (IStats.PREDICATE).

"All attributes named PREDICATE"

Having the same constants defined multiple times is not good news for the analysis and for the developers. But this kind of thing is fairly common in old systems which evolved during a long time in the hands of many developers. Not all of them had a complete understanding of the system and each had different skills and programming habits.

Looking at the lists of definitions for the 3 main constants (SUBJECT, PREDICATE, OBJECT), we find that there are at least 5 different definitions of these constants:

  • stats.IStats:
public static final int NA = 0;
public static final int SUBJECT = 1;
public static final int PREDICATE = 2;
public static final int OBJECT = 3;
public static final int TRIPLE = 4;
  • kgram.sorter.core.Const:
public static final int ALL = 0;
public static final int SUBJECT = 1;
public static final int PREDICATE = 2;
public static final int OBJECT = 3;
public static final int TRIPLE = 4;
public static final int NA = -1;
  • compiler.result.XMLResult
private static final int TRIPLE = 9;
private static final int SUBJECT = 10;
private static final int PREDICATE = 11;
private static final int OBJECT = 12;
  • kgram.api.core.ExprType
public static int TRIPLE = 88;
public static int SUBJECT = 89;
public static int PREDICATE = 90;
public static int OBJECT = 91;
  • kgram.core.Exp
public static final int ANY = -1;
public static final int SUBJECT = 0;
public static final int OBJECT = 1;
public static final int PREDICATE = 2;

So now we need to track the uses of all these constants in the system to understand how they can be replaced by one enum.

Note: Don’t close the Inspector window yet, we are going to need it soon.

Moose can help us here with tags. Tags are (as the name implies) just labels that can be attached to any entity in the model. Additionally, tags have a color that will help us distinguish them in visualizations.

So let’s tag our constants. We will define 5 tags, one for each set of constants, that is to say one for each of the 5 classes that implement these constants. You can choose whatever name and color you prefer for your tags, as long as you remember which is which. Here I named the tags from the name of the classes that define each set of constant.

"The tags that represent each set of constant"

Now we want to tag all the constants in a set with the same tag. Let’s see how to do it for constants in IStats, the ones listed in the previous section and that were our initial focus.

We select the “IStats” tag in the Tag Browser and go back to the Inspector where we have a list of all definitions of PREDICATE. If we click on the 3rd of these PREDICATE (“fr::inria::corese::core::stats::IStats.PREDICATE”), a new pane appears on the right, focusing on this attribute. There, we can click on its “parentType”, giving yet another pane. (The following screenshot shows the inspector right before we click on “parentType”).

"The inspector while navigating to the set of attributes of IStats".

The right pane now focuses on the IStats Java interface. We can click on “attributes” to get the list of attributes it defines (including PREDICATE from which we started). There are 5 attributes which are the ones listed in the previous section.

So far so good.

To tag these attributes, we will “propagate” them (toolbar button of the Inspector on the right) to all tools that are in “Follow” mode. Note that if you minimized the Tag Browser at some point, it will be in “Freeze” mode like in the screenshot above. You need to put it back in “Follow” (radio toolbar button on the left) before propagating the list of constants.

Once propagated, the list appears in the center pane of the Tag Browser and you can pass it to the right pane with the ”>>>” button. Doing this will effectively tag the entities with the selected tag.

We now have tagged these 5 constants with the “IStats” tag. Ideally we want to find also the usage of these constants. So we would like to also tag the methods that use these constants.

For this you can open a Query Browser, it will start with the same list of 5 attributes that we just propagated. We can create a “Navigation query” and ask for all the “incoming” “accesses” to these attributes as shown below. The result is a list of 6 methods.

"The methods accessing the 5 attributes propagated"

We can now propagate these 6 methods and they will appear in the Tag Browser. We tag them with the same tag as the attributes themselves.

You can repeat the same operations for the 5 sets of constants listed above and the 5 different tags.

All this tagging was to be able to visualize where each set of constant is defined and, most importantly, used. We now turn to the “Architectural Map” which is a fine tool to visualize tags. for example, we could show all the top level packages of Corese and the Architectural Map will give visual clues on which ones contain tagged entities, and what tags. The Architectural Map allows to expand the content of entities which will allow us to deep dive into each package containing tagged entities to understand where exactly the entities is used or defined.

To select all the top level packages, we go back one last time to the Inspector to the very first pane on the left (you may also “Inspect” again the model to open a new Inspector). We select the “Model packages” and enter this query in the “script” at the bottom: self select: [ :each | each parentPackage isNotNil and: [each parentPackage name = 'core'] ]. (Again, ignore the backslashes)

The result is a list of 23 packages that we can propagate. Finally we open an Architectural Map that will start with the 23 packages that we just propagated.

In the following screenchot, I restricted the Architectural Map to the only 5 packages that do use our tags: “stats”, “kgram”, “util”, “sparql”, and “query”. This makes it easier to see the results here. I also expanded “kgram” that is small and contains different tags.

"The packages using the 5 attributes"

The single-color square, on the right of each package name, shows that it contains entities having one uniq tag (of this color). In our case it means that it contains the constants and methods accessing them, all with the same tag. For example, “core” and “util” packages contain entities tagged with only the green tag (which corresponds to the kgram.core.Exp class as previously shown in the Tag Browser screenshot).

When the square is multicolored, it means it contains entities with different tags. For example, we see that the package “kgram” contains at least the green (“Exp”) and the yellow (“Const”) tags.

Note that in this particular case, I added another tag for class kgram.api.core.Node which has its own definition of the OBJECT constant. I wanted to see where it was used also. This is the reason for the multicolored square of class StatsBasedEstimation, in package “stats”, which uses OBJECT from Node and the other constants from IStats.

In the end, the visualization allows to conclude that each package sticks pretty much to its own definition of the constants which is rather reassuring. It also shows where one would have to look if we were to replace the constant by a real enum.

This is not the end of it however because the constant values used in these methods can be passed off to other methods as argument. Here Famix alone (the meta-model used in Moose by default) can no longer help us to follow the flow of usage of the constants because they are just integer being passed around. For a finer analysis, a complete AST model should be used. This could be done with the FAST meta-model (Famix-AST), but it is another story that falls outside the scope of this blog-post.

See you later.

Generating a visitor infrastructure for a given meta-model

This post is part of a serie dedicated to Famix Tools

Once we have a model of a program in Famix, we often find ourselves wanting to ¨ go through it” to apply some systematic analysis or modification. For example one could want to export the model as source-code https://github.com/moosetechnology/FAMIX2Java.

The Visitor design pattern is well adapted for these tasks. Note that the double-dispatch mechanism, which is an integral part of the visitor pattern, may also be useful to have entity specific actions even if one does not want to visit the entire model.

The Visitor pattern requires:

  • an accept: aVisitor method in every entity of the meta-model (eg.: in FamixJavaClass, FamixJavaMethod, FamixJavaAttribute,…)
  • visitFamixXYZ: aFamixXYZ for all entites to be visited, in the visitor class
  • the accept: methods invoke the correct visitFamixXYZ: method of the visitor depending on the class it is implemented in
  • the visitFamixXYZ: by default recursively visits all the “children” of the entity being visited (eg.: visitFamixJavaClass: should trigger the visit of the attributes and methods of the class)

For large meta-models, this can be cumbersome to implement as there may be many kinds of entities (43 for FamixJava) and the work is very repetitive. The tool FamixVisitorCodeGenerator can do all the work for you, automatically.

Taking advantage of the meta-description of the Famix entities and the reflective nature of Pharo, it generates the accept: and visitFamixXYZ: for all entities of a meta-model.

Usage exmaple:

FamixVisitorCodeGenerator new
package: 'Famix-Java-Entities' visitorClass: FamixJavaVisitor .

or for a FAST meta-model:

FamixVisitorCodeGenerator new
package: 'FAST-Java-Entities' visitorClass: FASTJavaVisitor.

The tool needs an empty visitor class (or trait) created by the user (FamixJavaVisitor in the example above), and a Pharo package containing the Famix classes of a meta-model (Famix-Java-Entities in the example above). From this it will:

  • create an accept: method in all the classes in the given Famix package;
  • the accept: methods are created as extensions made by the package of the visitor;
  • the accept: methods invoke the correct visitFamixXYZ: depending on the class they are implemented in.
  • a setter method allows to skip this part: generateAccepts: false
  • the visitFamixXYZ: methods are created in the visitor class (or trait) for a “maximal visit” (see below).

For a friendlier visitor, it is convenient that the visitor methods reproduce the inheritance hierarchy of the entities. For example, if FamixJavaAttribute and FamixJavaParameter both inherit from FamixJavaVariable entity, it is convenient that visitFamixJavaAttribute: and visitFamixJavaParameter: both call visitFamixJavaVariable: so that a common behaviour for the visit can be implemented only once.

Since Famix heavily relies on traits to compose meta-models, the same idea applies to used traits. For example many entities implements FamixTSourceEntity to get a sourceAnchor. For all these entities, and if we need a generic behavior based on the source anchor, it is convenient that all the visitXYZ: methods call visitTSourceEntity: where the common behavior can be implemented.

Finally it might be convenient that the visitXYZ: methods, visit all the entites related to the one we are visiting. For example when visiting a FamixJavaVariable, it might be interesting to recursively visit its declaredType.

All these conditions defines what we call a “maxium” visit of the meta-model.

As described above, the maximum visitXYZ: methods will trigger a resursive visit of many things (super-class visit method, used traits visit method, all entities related to the one being visited).

One down side of this is that a maximum visit is not a viable one in Famix because all relationships are bi-directionnal, causing infinite loops in the visit. For example, if in a FamixJavaVariable we recursively visit its declaredType, then in the FamixJavaTType we will recursively visit the typedEntities including the one we just came from.

There could be various solution to this problem:

  • implement a memory mechanism in the visitor to remember what entities were already visited and do not go back to them.
  • do not visit all relationships of an entity, but only its “children” relationship
  • let the user handle the problem

For now the tool rely on the third solution. If an actual visitor inherits (or use) the generated visitor, it must take care or redefining the visit methods so that it does not enter in an infinite loop (implementing any of the two other solutions)

In this sense, the maximum visit methods can be viewed as cheat sheets, showing all the things that one could do when visiting an entity.

In the future we might implement the second solution (visit only children) as an option of the visitor. For now it is important to remember that the generated visitor cannot be used as is. Methods must be redefined to get a viable visitor.

The natural action would be to define the visitor as aclass from which all required actual visitors will inherit. However, because visitors are so handy to go through the entire model, we discovered that we needed a lot of them (eg. visitors to create a copy of a model, to re-export a model) and they sometimes need to inherit from some other class.

As such, we rather recommend to create the maximal visitor as a trait that can be used by all actual visitors. This make no difference for the FamixVisitorCodeGenerator and it might prove very useful.

Transformation journey (3/3) : Visualize and apply code transformations

Once again, let me welcome you to the final blog post in those three blog posts about code transformations! During the first blog post, we used queries and tools to locate the entities we seek to transform, before implementing the actual transformation logic in the second blog post.

In this third post, we will use a visualisation window allowing us to check the source code of our entity before and after a transformation, and finally apply those transformation on the actual source files, therefore transposing this transformation from Moose to your actual software!

We are still following the scenario presented in the previous blog posts, on our use case, ArgoUML. In this system three classes define and use a method named logError. A class ArgoLogger has been defined that contains a static method logError.

The transformation task is to add a receiver node to each logError method invocation so that the method called is now ArgoLogger.logError().

As this blog post follows what was discussed and built in the first one of the series, a lot of information (how to build the image and model used for our use case, but also the use case itself) is provided in the previous posts.

If you haven’t read it, or if you forgot the content of the post, it is recommended to go back to it, as it will not be repeated in this post.

In this final blog post, we do not have any new tools to import. As long as you still have FAST-Java-Tools, Carrefour and Motion from the two previous posts, you should have no problem following this blogpost!

At this point, we now have in our tool modified FAST trees of the methods we need to transform, all encapsulated in wrapper objects to save all the necessary information to complete the transformation, and apply it where needed. Hovewer, before that, you may feel the need to check the transformation produced by your tool.

To do so, we can use the TransformationEditor tool, a dialog window able to display the code before and after our code transformation, along with some nice and practical features to fulfill this task.

First, let’s add a new method to our transformation tool; displayTransformationOfMethod: aMethodWrapper. We will call this new method in the already existing method transformMethod: aJavaMethodWrapper, as shown here :

transformMethod: aJavaMethodWrapper
| methodInvocationNode codeEditor |
methodInvocationNode := self motionQueryForFastMethod:
aJavaMethodWrapper transformedFastMethod.
methodInvocationNode receiver: self createNewReceiverNode.
aJavaMethodWrapper addTransformedFastNode:
methodInvocationNode receiver.
codeEditor := self displayTransformationOfMethod: aJavaMethodWrapper.
displayTransformationOfMethod: aMethodWrapper
| exportVisitor transformedCode codeEditor |
exportVisitor := FASTJavaExporttVisitor new.
transformedCode := exportVisitor export:
aMethodWrapper transformedFastMethod.
codeEditor := TransformationEditor
openForOriginalText:
aMethodWrapper originalFastMethod sourceText
transformed: transformedCode
originalEntity: aMethodWrapper famixMethod.
^ codeEditor

As you can notice in this new method, we also call another tool, the FASTJavaExportVisitor. This visitor will allow us to convert a FAST Java tree to a string giving us the source code of our method after transformation.

Coupling the result of using this tool to this class method of the TransformationEditor class will display such results :

"Transformation display"

In this new window, you can find on the left the original source code of the entity, before any transformation. And on the right, you can find the source code of the same entity, after applying the transformation.

The first thing we can notice, is how radically different the two methods look at a glance. Hovewer, everything is the same (to the exception of the new ArgoLogger class added with our transformation). They look different because the transformed code is generated from the transformed FAST tree, and the export visitor exports that code with its own formating.

The source code after transformation (on the right pane) is also editable, meaning that you can manually add code to the entity if it is necessary for any reason. Do be careful that the code is syntaxically correct however, as there is no checks (syntax, compilation) on this code in Moose.

We now have a way to view our transformation, but as it is, it is still tough to view properly where are the changes and what they are. And in this example, the transformation is very basic! If the transformation was spread throughout a class rather than a method, or if the transformation was simply more complex, then the tool would fail as a means to review the changes.

But of course, we can fix that. 😉

To do so, we will modify the new method displayTransformationOfMethod: and use a subclass of the export visitor previously used; FASTJavaExportAndHiglightVisitor.

displayTransformationOfMethod: aMethodWrapper
| exportVisitor transformedCode codeEditor |
exportVisitor := FASTJavaExportAndHighlightVisitor new.
exportVisitor entitiesToHighlight:
aMethodWrapper transformedFastNodes.
transformedCode := exportVisitor export:
aMethodWrapper transformedFastMethod.
codeEditor := TransformationEditor
openForOriginalText:
aMethodWrapper originalFastMethod sourceText
transformed: transformedCode
removedHighlights: OrderedCollection new
addedHighlights: exportVisitor highlights
originalEntity: aMethodWrapper famixMethod.
^ codeEditor

Our first edit simply calls the method wrapper to add a “transformed node”, a FAST node added during the transformation. In a similar fashion, you can add an “original node”, a FAST node removed during the transformation (no such node exists in our example, so we don’t use that feature).

This new transformed node is used in our second edit, where we give the transformed nodes collection to the new visitor tool that we are going to use.

This tool behaves just like the original exporter tool, except when it encounters a node present in the transformed nodes list. In this case, the tool save an interval corresponding to the startPos and endPos of the node, to know precisely its position in the generated string.

This allows our editing window to apply an highlight on this interval, therefore showing only the nodes added in the transformation for the user to easily review the work of their transformation tool.

To use this feature in the editing window, simply do as done in the edit, giving the tool a collection of interval for the removedHighlights and / or addedHighlights slots.

"Transformation display with highlighting"

Now that we reviewed and accepted the transformation on our entities, only one step remains. To apply those code changes on the actual source code of the modeled software, thereby completing the transformation task.

Once again, we have a tool ready for this need, in the form of the TransformationApplyerOnSourceAnchor class. Let’s do a final edit on our method transformSourceCodeOfMethod to add it:

transformSourceCodeOfMethod: aMethodWrapper
| exportVisitor transformedCode codeEditor |
exportVisitor := FASTJavaExportAndHighlightVisitor new.
exportVisitor entitiesToHighlight:
aMethodWrapper transformedFastNodes.
transformedCode := exportVisitor export:
aMethodWrapper transformedFastMethod.
codeEditor := TransformationEditor
openForOriginalText:
aMethodWrapper originalFastMethod sourceText
transformed: transformedCode
removedHighlights: OrderedCollection new
addedHighlights: exportVisitor highlights
originalEntity: aMethodWrapper famixMethod.
codeEditor isOk ifTrue: [
| tool |
tool := TransformationApplyerOnSourceAnchor new.
tool
transformMethod: aMethodWrapper famixMethod
withNewSource: codeEditor fetchEditedCode ]

As you can see, all this tool needs in our context is a method and the new source code as a String to complete the operation. You can also see that we called the method fetchEditedCode on our code editor window, which returns the String present in the right window of said editor. Using this method ensures that you get the right data, even if the code was manually edited during its review.

Once this tool has made its job and transformed the source code of your software, do keep in mind that the model of that software is no longer up to date with the source. This means that any analysis that you could run on this model could be flawed, so it is recommended to regularly replace that model with a new one, based from the source after it has been transformed.

Hovewer, the model is slightly updated by the tool after each transformation applyed. The values of startPos and endPos on the source anchor of entities within the file which received a transformation. This means that their source code will still be properly fetched by Moose whenever you need it.

For example, if your transformation tool relies mostly on the FAST models, this allows you to apply some transformations on several entities even if those are located in the same file without having to worry too much about the order in which you make those transformations.

This concludes our blog post series about code transformation!

We now have a tool managing every step of the way to complete a specific transformation case, from finding the necessary informations needed for the transformation, transforming a FAST model and using those transformed FAST models to view, edit and apply the transformations to the source code.

Just like with the two other posts, feel free to now try the tool and methods that we created here in a Playground and experiment with the results, or the methods themselves!

t := LoggerTransformationTool onModel: MooseModel root first.
t findAndTransformAllLogErrorInvocations

"Testing class"

The whole source code that was written on the three blog posts is still available on that repository.

Through the use of tools to query and search through FAST models, transform said models, before generating, reviewing and applying the new code on the source files, we managed to build a fully functionnal and semi-automatic transformation tool for our use case!

This use case was a rather small example. A real transformation in a software would need more queries and FAST edits. But the only real difference would be the scale. Using the methodology and tools in the same way that they were used in this blog post would be enough to apply to many use cases out there. 😄