Blog

Some tools on FAST models

Mar 7, 2025

Nicolas Anquetil

Moose expert

The package FAST-Core-Tools in repository https://github.com/moosetechnology/FAST offers some tools or algorithms that are running on FAST models.

These tools may be usable directly on a specific language FAST meta-model, or might require some adjustements by subtyping them. They are not out-of-the-shelf ready to use stuff, but they can provide good inspiration for whatever you need to do.

Dumping AST

Writing test for FAST can be pretty tedious because you have to build a FAST model in the test corresponding to your need. It often has a lot of nodes that you need to create in the right order with the right properties.

This is where FASTDumpVisitor can help by visiting an existing AST and “dump” it as a string. The goal is that executing this string in Pharo should recreate exactly the same AST.

Dumping an AST can also be useful to debug an AST and checking that it has the right properties.

To use it, you can just call FASTDumpVisitor visit: <yourAST> and print the result. For example:

FASTDumpVisitor visit:
  (FASTJavaUnaryExpression new
    operator: '-' ;
    expression:
      (FASTJavaIntegerLiteral new
        primitiveValue: '5'))

will return the string: FASTJavaUnaryExpression new expression:(FASTJavaIntegerLiteral new primitiveValue:'5');operator:'-' which, if evaluated, in Pharo will recreate the same AST as the original.

Note: Because FAST models are actually Famix models (Famix-AST), the tools works also for Famix models. But Famix entities typically have more properties and the result is not so nice:

FASTDumpVisitor visit:
  (FamixJavaMethod new
    name: 'toto' ;
    parameters: {
      FamixJavaParameter new name: 'x' .
      FamixJavaParameter new name: 'y'} ).

will return the string: FamixJavaMethod new parameters:{FamixJavaParameter new name:'x';isFinal:false;numberOfLinesOfCode:0;isStub:false.FamixJavaParameter new name:'y';isFinal:false;numberOfLinesOfCode:0;isStub:false};isStub:false;isClassSide:false;isFinal:false;numberOfLinesOfCode:-1;isSynchronized:false;numberOfConditionals:-1;isAbstract:false;cyclomaticComplexity:-1;name:'toto'.

Local symbol resolution in an AST

By definition an AST (Abstract Syntax Tree) is a tree (!). So the same variable can appear several time in an AST in different nodes (for example if the same variable is accessed several times).

The idea of the class FASTLocalResolverVisitor is to relate all uses of a symbol in the AST to the node where the symbol is defined. This is mostly useful for parameters and local variables inside a method, because the local resover only looks at the AST itself and we do not build ASTs for entire systems.

This local resolver will look at identifier appearing in an AST and try to link them all together when they correspond to the same entity. There is no complex computation in it. It just looks at names defined or used in the AST.

This is dependant on the programming language because the nodes using or defining a variable are not the same in all languages. For Java, there is FASTJavaLocalResolverVisitor, and for Fortran FASTFortranLocalResolverVisitor.

The tool brings an extra level of detail by managing scopes, so that if the same variable name is defined in different loops (for example), then each use of the name will be related to the correct definition.

The resolution process creates:

• In declaration nodes (eg. FASTJavaVariableDeclarator or FASTJavaParameter),a property #localUses will list all referencing nodes for this variable;
• In accessing nodes, (eg. FASTJavaVariableExpression), a property #localDeclarations will lists the declaration node corresponding this variable.
• If the declaration node was not found a FASTNonLocalDeclaration is used as the declaration node.

Note: That this looks a bit like what Carrefour does (see /blog/2022-06-30-carrefour), because both will bind several FAST nodes to the same entity. But the process is very different:

• Carrefour will bind a FAST node to a corresponding Famix node;
• The local resolver binds FAST nodes together.

So Carrefour is not local, it look in the entire Famix model to find the entity that matches a FAST node. In Famix, there is only one Famix entity for one software entity and it “knows” all its uses (a FamixVariable has a list of FamixAccess-es). Each FAST declaration node will be related to the Famix entity (the FamixVariable) and the FAST use nodes will be related to the FamixAccess-es.

On the other hand, the local resolver is a much lighter tool. It only needs a FAST model to work on and will only bind FAST nodes between themselves in that FAST model.

Round-trip validation

For round-trip re-engineering, we need to import a program in a model, modify the model, and re-export it as a (modified) program. A lot can go wrong or be fogotten in all these steps and they are not trivial to validate.

First, unless much extra information is added to the AST, the re-export will not be syntactically equivalent: there are formatting issues, indentation, white spaces, blank lines, comments that could make the re-exported program very different (apparently) from the original one.

The class FASTDifferentialValidator helps checking that the round-trip implementation works well. It focuses on the meaning of the program independently of the formatting issues. The process is the follwing:

• parse a set of (representative) programs
• model them in FAST
• re-export the programs
• re-import the new programs, and
• re-create a new model

Hopefully, the two models (2nd and last steps) should be equivalent This is what this tool checks.

Obviously the validation can easily be circumvented. Trivially, if we create an empty model the 1st time, re-export anything, and create an empty model the second time, then the 2 models are equivalent, yet we did not accomplish anything. This tool is an help for developers to pinpoint small mistakes in the process.

Note that even in the best of conditions, there can still be subtle differences between two equivalent ASTs. For example the AST for “a + b + c” will often differ from that of “a + (b + c)”.

The validator is intended to run on a set of source files and check that they are all parsed and re-exported correctly. It will report differences and will allow to fine tune the comparison or ignore some differences.

It goes through all the files in a directory and uses an importer, an exporter, and a comparator. The importer generates a FAST model from some source code (eg. JavaSmaCCProgramNodeImporterVisitor); the exporter generates source code from a model (eg. FASTJavaExportVisitor); the comparator is a companion class to the DifferentialValidator that handle the differences between the ASTs.

The basic implementation (FamixModelComparator) does a strict comparison (no differences allowed), but it has methods for accepting some differences:

• #ast: node1 acceptableDifferenceTo: node2: If for some reason the difference in the nodes is acceptable, this method must return true and the comparison will restart from the parent of the two nodes as if they were the same.
• #ast: node1 acceptableDifferenceTo: node2 property: aSymbol. This is for property comparison (eg. the name of an entity), it should return nil if the difference in value is not acceptable and a recovery block if it is acceptable. Instead of resuming from the parent of the nodes, the comparison will resume from an ancestor for which the recovery block evaluates to true.

Tags:

• Famix-tools
• FAST

A real example on using tags

Mar 5, 2025

Nicolas Anquetil

Moose expert

A real example on using tags

Tags can be a powerful tool to visualize things on legacy software and perform analyses. For example, tags can be used to create virtual entities and see how they “interact” with the real entities of the system analyzed. In the article Decomposing God Classes at Siemens we show how tags can be used to create virtual classes and see their dependencies to real classes.

In this post I will show another use of tags: how they can materialize a concept and show its instantiation in a system.

The scenario is that of analysing Corese, a platform to “create, manipulate, parse, serialize, query, reason and validate RDF data.” Corese is an old software that dates back to the early days of Java. Back then, enums did not exist in Java and a good way to implement them was to use a set of constants:

  public static final int MONDAY = 1;
  public static final int TUESDAY = 2;
  public static final int WEDNESDAY = 3;
  public static final int THURSDAY = 4;
  public static final int FRIDAY = 5;
  public static final int SATURDAY = 6;
  public static final int SUNDAY = 7;

Those were the days!

As an effort to restructure and rationalize implementation, the developers of Corese wish to replace these sets of constants by real Java enums. This is not something that can be done in any modern IDE even with the latest refactoring tool.

Let us see how Moose can help in the task.

Where are the constants used?

For an analysis in Moose, we need a model of the system, and this starts with getting the source code (https://github.com/corese-stack/corese-core). The model is created using VerveineJ which can be run using docker:

docker run -rm -v src/main/java/:/src ghcr.io/evref-bl/verveinej:latest -alllocals -o corese-core.json

This will create a file corese-core.json in the directory src/main/java/. The command to create the model as an option -alllocals. This is because VerveineJ by default only tracks the uses of variables with non primitive type (variables containing objects). Here the constants are integers and if we want to know where they are used, we need more details.

Let’s import the model in Moose. This can be done simply by dragging-and-dropping the file in Moose.

We will study the use of the constants defined in fr.inria.corese.core.stats.IStats:

public interface IStats {

    public static final int NA = 0;
    public static final int SUBJECT = 1;
    public static final int PREDICATE = 2;
    public static final int OBJECT = 3;
    public static final int TRIPLE = 4;
[...]

To find where the constants are used, we need to find the representation of the constants in the model. For this, we can inspect the model (“Inspect” button in the Model Browser) and look for all “Model Attributes”. The constants are attributes of the interface/class in which they are defined as shown in the listing above). And they are model attributes because they are defined in the source code analysed, as opposed to System.out which may be used in the code but for which we don’t have the source code.

We can then select all the model attributes named PREDICATE: select: [ :each | each name = 'PREDICATE']. (note, the backslash (\) before the square bracket ([) was added by the publishing tool and is not part of the code)

Moose gives us 8 different definitions of PREDICATE (and 9 for OBJECT, and 10 for SUBJECT). The one we are interested in is the 3rd in the list (IStats.PREDICATE).

Having the same constants defined multiple times is not good news for the analysis and for the developers. But this kind of thing is fairly common in old systems which evolved during a long time in the hands of many developers. Not all of them had a complete understanding of the system and each had different skills and programming habits.

Looking at the lists of definitions for the 3 main constants (SUBJECT, PREDICATE, OBJECT), we find that there are at least 5 different definitions of these constants:

• stats.IStats:

    public static final int NA = 0;
    public static final int SUBJECT = 1;
    public static final int PREDICATE = 2;
    public static final int OBJECT = 3;
    public static final int TRIPLE = 4;

• kgram.sorter.core.Const:

    public static final int ALL = 0;
    public static final int SUBJECT = 1;
    public static final int PREDICATE = 2;
    public static final int OBJECT = 3;
    public static final int TRIPLE = 4;
    public static final int NA = -1;

• compiler.result.XMLResult

    private static final int TRIPLE = 9;
    private static final int SUBJECT = 10;
    private static final int PREDICATE = 11;
    private static final int OBJECT = 12;

• kgram.api.core.ExprType

       public static int TRIPLE    = 88;
       public static int SUBJECT  = 89;
       public static int PREDICATE = 90;
       public static int OBJECT    = 91;

• kgram.core.Exp

    public static final int ANY        = -1;
    public static final int SUBJECT    = 0;
    public static final int OBJECT     = 1;
    public static final int PREDICATE  = 2;

So now we need to track the uses of all these constants in the system to understand how they can be replaced by one enum.

Note: Don’t close the Inspector window yet, we are going to need it soon.

Tagging the constants and their uses

Moose can help us here with tags. Tags are (as the name implies) just labels that can be attached to any entity in the model. Additionally, tags have a color that will help us distinguish them in visualizations.

So let’s tag our constants. We will define 5 tags, one for each set of constants, that is to say one for each of the 5 classes that implement these constants. You can choose whatever name and color you prefer for your tags, as long as you remember which is which. Here I named the tags from the name of the classes that define each set of constant.

Now we want to tag all the constants in a set with the same tag. Let’s see how to do it for constants in IStats, the ones listed in the previous section and that were our initial focus.

We select the “IStats” tag in the Tag Browser and go back to the Inspector where we have a list of all definitions of PREDICATE. If we click on the 3rd of these PREDICATE (“fr::inria::corese::core::stats::IStats.PREDICATE”), a new pane appears on the right, focusing on this attribute. There, we can click on its “parentType”, giving yet another pane. (The following screenshot shows the inspector right before we click on “parentType”).

.

The right pane now focuses on the IStats Java interface. We can click on “attributes” to get the list of attributes it defines (including PREDICATE from which we started). There are 5 attributes which are the ones listed in the previous section.

So far so good.

To tag these attributes, we will “propagate” them (toolbar button of the Inspector on the right) to all tools that are in “Follow” mode. Note that if you minimized the Tag Browser at some point, it will be in “Freeze” mode like in the screenshot above. You need to put it back in “Follow” (radio toolbar button on the left) before propagating the list of constants.

Once propagated, the list appears in the center pane of the Tag Browser and you can pass it to the right pane with the ”>>>” button. Doing this will effectively tag the entities with the selected tag.

We now have tagged these 5 constants with the “IStats” tag. Ideally we want to find also the usage of these constants. So we would like to also tag the methods that use these constants.

For this you can open a Query Browser, it will start with the same list of 5 attributes that we just propagated. We can create a “Navigation query” and ask for all the “incoming” “accesses” to these attributes as shown below. The result is a list of 6 methods.

We can now propagate these 6 methods and they will appear in the Tag Browser. We tag them with the same tag as the attributes themselves.

You can repeat the same operations for the 5 sets of constants listed above and the 5 different tags.

Visualizing the result

All this tagging was to be able to visualize where each set of constant is defined and, most importantly, used. We now turn to the “Architectural Map” which is a fine tool to visualize tags. for example, we could show all the top level packages of Corese and the Architectural Map will give visual clues on which ones contain tagged entities, and what tags. The Architectural Map allows to expand the content of entities which will allow us to deep dive into each package containing tagged entities to understand where exactly the entities is used or defined.

To select all the top level packages, we go back one last time to the Inspector to the very first pane on the left (you may also “Inspect” again the model to open a new Inspector). We select the “Model packages” and enter this query in the “script” at the bottom: self select: [ :each | each parentPackage isNotNil and: [each parentPackage name = 'core'] ]. (Again, ignore the backslashes)

The result is a list of 23 packages that we can propagate. Finally we open an Architectural Map that will start with the 23 packages that we just propagated.

In the following screenchot, I restricted the Architectural Map to the only 5 packages that do use our tags: “stats”, “kgram”, “util”, “sparql”, and “query”. This makes it easier to see the results here. I also expanded “kgram” that is small and contains different tags.

The single-color square, on the right of each package name, shows that it contains entities having one uniq tag (of this color). In our case it means that it contains the constants and methods accessing them, all with the same tag. For example, “core” and “util” packages contain entities tagged with only the green tag (which corresponds to the kgram.core.Exp class as previously shown in the Tag Browser screenshot).

When the square is multicolored, it means it contains entities with different tags. For example, we see that the package “kgram” contains at least the green (“Exp”) and the yellow (“Const”) tags.

Note that in this particular case, I added another tag for class kgram.api.core.Node which has its own definition of the OBJECT constant. I wanted to see where it was used also. This is the reason for the multicolored square of class StatsBasedEstimation, in package “stats”, which uses OBJECT from Node and the other constants from IStats.

In the end, the visualization allows to conclude that each package sticks pretty much to its own definition of the constants which is rather reassuring. It also shows where one would have to look if we were to replace the constant by a real enum.

This is not the end of it however because the constant values used in these methods can be passed off to other methods as argument. Here Famix alone (the meta-model used in Moose by default) can no longer help us to follow the flow of usage of the constants because they are just integer being passed around. For a finer analysis, a complete AST model should be used. This could be done with the FAST meta-model (Famix-AST), but it is another story that falls outside the scope of this blog-post.

See you later.

Tags:

• story

Generating a visitor infrastructure for a given meta-model

Feb 26, 2025

Nicolas Anquetil

Moose expert

This post is part of a serie dedicated to Famix Tools

Once we have a model of a program in Famix, we often find ourselves wanting to ¨ go through it” to apply some systematic analysis or modification. For example one could want to export the model as source-code https://github.com/moosetechnology/FAMIX2Java.

The Visitor design pattern is well adapted for these tasks. Note that the double-dispatch mechanism, which is an integral part of the visitor pattern, may also be useful to have entity specific actions even if one does not want to visit the entire model.

The Visitor pattern requires:

• an accept: aVisitor method in every entity of the meta-model (eg.: in FamixJavaClass, FamixJavaMethod, FamixJavaAttribute,…)
• visitFamixXYZ: aFamixXYZ for all entites to be visited, in the visitor class
• the accept: methods invoke the correct visitFamixXYZ: method of the visitor depending on the class it is implemented in
• the visitFamixXYZ: by default recursively visits all the “children” of the entity being visited (eg.: visitFamixJavaClass: should trigger the visit of the attributes and methods of the class)

For large meta-models, this can be cumbersome to implement as there may be many kinds of entities (43 for FamixJava) and the work is very repetitive. The tool FamixVisitorCodeGenerator can do all the work for you, automatically.

FamixVisitorCodeGenerator

Taking advantage of the meta-description of the Famix entities and the reflective nature of Pharo, it generates the accept: and visitFamixXYZ: for all entities of a meta-model.

Usage exmaple:

FamixVisitorCodeGenerator new
  package: 'Famix-Java-Entities' visitorClass: FamixJavaVisitor .

or for a FAST meta-model:

FamixVisitorCodeGenerator new
  package: 'FAST-Java-Entities' visitorClass: FASTJavaVisitor.

The tool needs an empty visitor class (or trait) created by the user (FamixJavaVisitor in the example above), and a Pharo package containing the Famix classes of a meta-model (Famix-Java-Entities in the example above). From this it will:

• create an accept: method in all the classes in the given Famix package;
• the accept: methods are created as extensions made by the package of the visitor;
• the accept: methods invoke the correct visitFamixXYZ: depending on the class they are implemented in.
• a setter method allows to skip this part: generateAccepts: false
• the visitFamixXYZ: methods are created in the visitor class (or trait) for a “maximal visit” (see below).

Visiting methods

For a friendlier visitor, it is convenient that the visitor methods reproduce the inheritance hierarchy of the entities. For example, if FamixJavaAttribute and FamixJavaParameter both inherit from FamixJavaVariable entity, it is convenient that visitFamixJavaAttribute: and visitFamixJavaParameter: both call visitFamixJavaVariable: so that a common behaviour for the visit can be implemented only once.

Since Famix heavily relies on traits to compose meta-models, the same idea applies to used traits. For example many entities implements FamixTSourceEntity to get a sourceAnchor. For all these entities, and if we need a generic behavior based on the source anchor, it is convenient that all the visitXYZ: methods call visitTSourceEntity: where the common behavior can be implemented.

Finally it might be convenient that the visitXYZ: methods, visit all the entites related to the one we are visiting. For example when visiting a FamixJavaVariable, it might be interesting to recursively visit its declaredType.

All these conditions defines what we call a “maxium” visit of the meta-model.

”Maximum” Visit

As described above, the maximum visitXYZ: methods will trigger a resursive visit of many things (super-class visit method, used traits visit method, all entities related to the one being visited).

One down side of this is that a maximum visit is not a viable one in Famix because all relationships are bi-directionnal, causing infinite loops in the visit. For example, if in a FamixJavaVariable we recursively visit its declaredType, then in the FamixJavaTType we will recursively visit the typedEntities including the one we just came from.

There could be various solution to this problem:

• implement a memory mechanism in the visitor to remember what entities were already visited and do not go back to them.
• do not visit all relationships of an entity, but only its “children” relationship
• let the user handle the problem

For now the tool rely on the third solution. If an actual visitor inherits (or use) the generated visitor, it must take care or redefining the visit methods so that it does not enter in an infinite loop (implementing any of the two other solutions)

In this sense, the maximum visit methods can be viewed as cheat sheets, showing all the things that one could do when visiting an entity.

In the future we might implement the second solution (visit only children) as an option of the visitor. For now it is important to remember that the generated visitor cannot be used as is. Methods must be redefined to get a viable visitor.

class vs. trait visitor

The natural action would be to define the visitor as aclass from which all required actual visitors will inherit. However, because visitors are so handy to go through the entire model, we discovered that we needed a lot of them (eg. visitors to create a copy of a model, to re-export a model) and they sometimes need to inherit from some other class.

As such, we rather recommend to create the maximal visitor as a trait that can be used by all actual visitors. This make no difference for the FamixVisitorCodeGenerator and it might prove very useful.

Tags:

• Famix-tools

Transformation journey (3/3) : Visualize and apply code transformations

Jun 13, 2024

Romain Degrave

Master degree intern in software analysis

Once again, let me welcome you to the final blog post in those three blog posts about code transformations! During the first blog post, we used queries and tools to locate the entities we seek to transform, before implementing the actual transformation logic in the second blog post.

In this third post, we will use a visualisation window allowing us to check the source code of our entity before and after a transformation, and finally apply those transformation on the actual source files, therefore transposing this transformation from Moose to your actual software!

We are still following the scenario presented in the previous blog posts, on our use case, ArgoUML. In this system three classes define and use a method named logError. A class ArgoLogger has been defined that contains a static method logError.

The transformation task is to add a receiver node to each logError method invocation so that the method called is now ArgoLogger.logError().

Preface

As this blog post follows what was discussed and built in the first one of the series, a lot of information (how to build the image and model used for our use case, but also the use case itself) is provided in the previous posts.

If you haven’t read it, or if you forgot the content of the post, it is recommended to go back to it, as it will not be repeated in this post.

Tools to import

In this final blog post, we do not have any new tools to import. As long as you still have FAST-Java-Tools, Carrefour and Motion from the two previous posts, you should have no problem following this blogpost!

A first look on our transformation

At this point, we now have in our tool modified FAST trees of the methods we need to transform, all encapsulated in wrapper objects to save all the necessary information to complete the transformation, and apply it where needed. Hovewer, before that, you may feel the need to check the transformation produced by your tool.

To do so, we can use the TransformationEditor tool, a dialog window able to display the code before and after our code transformation, along with some nice and practical features to fulfill this task.

First, let’s add a new method to our transformation tool; displayTransformationOfMethod: aMethodWrapper. We will call this new method in the already existing method transformMethod: aJavaMethodWrapper, as shown here :

transformMethod: aJavaMethodWrapper

    | methodInvocationNode codeEditor |
    methodInvocationNode := self motionQueryForFastMethod:
        aJavaMethodWrapper transformedFastMethod.

    methodInvocationNode receiver: self createNewReceiverNode.

    aJavaMethodWrapper addTransformedFastNode:
        methodInvocationNode receiver.

    codeEditor := self displayTransformationOfMethod: aJavaMethodWrapper.

displayTransformationOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExporttVisitor new.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        originalEntity: aMethodWrapper famixMethod.

    ^ codeEditor

As you can notice in this new method, we also call another tool, the FASTJavaExportVisitor. This visitor will allow us to convert a FAST Java tree to a string giving us the source code of our method after transformation.

Coupling the result of using this tool to this class method of the TransformationEditor class will display such results :

In this new window, you can find on the left the original source code of the entity, before any transformation. And on the right, you can find the source code of the same entity, after applying the transformation.

The first thing we can notice, is how radically different the two methods look at a glance. Hovewer, everything is the same (to the exception of the new ArgoLogger class added with our transformation). They look different because the transformed code is generated from the transformed FAST tree, and the export visitor exports that code with its own formating.

The source code after transformation (on the right pane) is also editable, meaning that you can manually add code to the entity if it is necessary for any reason. Do be careful that the code is syntaxically correct however, as there is no checks (syntax, compilation) on this code in Moose.

Adding highlights

We now have a way to view our transformation, but as it is, it is still tough to view properly where are the changes and what they are. And in this example, the transformation is very basic! If the transformation was spread throughout a class rather than a method, or if the transformation was simply more complex, then the tool would fail as a means to review the changes.

But of course, we can fix that. 😉

To do so, we will modify the new method displayTransformationOfMethod: and use a subclass of the export visitor previously used; FASTJavaExportAndHiglightVisitor.

displayTransformationOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExportAndHighlightVisitor new.
    exportVisitor entitiesToHighlight:
        aMethodWrapper transformedFastNodes.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        removedHighlights: OrderedCollection new
        addedHighlights: exportVisitor highlights
        originalEntity: aMethodWrapper famixMethod.

    ^ codeEditor

Our first edit simply calls the method wrapper to add a “transformed node”, a FAST node added during the transformation. In a similar fashion, you can add an “original node”, a FAST node removed during the transformation (no such node exists in our example, so we don’t use that feature).

This new transformed node is used in our second edit, where we give the transformed nodes collection to the new visitor tool that we are going to use.

This tool behaves just like the original exporter tool, except when it encounters a node present in the transformed nodes list. In this case, the tool save an interval corresponding to the startPos and endPos of the node, to know precisely its position in the generated string.

This allows our editing window to apply an highlight on this interval, therefore showing only the nodes added in the transformation for the user to easily review the work of their transformation tool.

To use this feature in the editing window, simply do as done in the edit, giving the tool a collection of interval for the removedHighlights and / or addedHighlights slots.

Applying our changes on the source files

Now that we reviewed and accepted the transformation on our entities, only one step remains. To apply those code changes on the actual source code of the modeled software, thereby completing the transformation task.

Once again, we have a tool ready for this need, in the form of the TransformationApplyerOnSourceAnchor class. Let’s do a final edit on our method transformSourceCodeOfMethod to add it:

transformSourceCodeOfMethod: aMethodWrapper

    | exportVisitor transformedCode codeEditor |
    exportVisitor := FASTJavaExportAndHighlightVisitor new.
    exportVisitor entitiesToHighlight:
        aMethodWrapper transformedFastNodes.

    transformedCode := exportVisitor export:
        aMethodWrapper transformedFastMethod.

    codeEditor := TransformationEditor
        openForOriginalText:
        aMethodWrapper originalFastMethod sourceText
        transformed: transformedCode
        removedHighlights: OrderedCollection new
        addedHighlights: exportVisitor highlights
        originalEntity: aMethodWrapper famixMethod.

    codeEditor isOk ifTrue: [
    | tool |
    tool := TransformationApplyerOnSourceAnchor new.
    tool
        transformMethod: aMethodWrapper famixMethod
        withNewSource: codeEditor fetchEditedCode ]

As you can see, all this tool needs in our context is a method and the new source code as a String to complete the operation. You can also see that we called the method fetchEditedCode on our code editor window, which returns the String present in the right window of said editor. Using this method ensures that you get the right data, even if the code was manually edited during its review.

Once this tool has made its job and transformed the source code of your software, do keep in mind that the model of that software is no longer up to date with the source. This means that any analysis that you could run on this model could be flawed, so it is recommended to regularly replace that model with a new one, based from the source after it has been transformed.

Hovewer, the model is slightly updated by the tool after each transformation applyed. The values of startPos and endPos on the source anchor of entities within the file which received a transformation. This means that their source code will still be properly fetched by Moose whenever you need it.

For example, if your transformation tool relies mostly on the FAST models, this allows you to apply some transformations on several entities even if those are located in the same file without having to worry too much about the order in which you make those transformations.

Try it yourself!

This concludes our blog post series about code transformation!

We now have a tool managing every step of the way to complete a specific transformation case, from finding the necessary informations needed for the transformation, transforming a FAST model and using those transformed FAST models to view, edit and apply the transformations to the source code.

Just like with the two other posts, feel free to now try the tool and methods that we created here in a Playground and experiment with the results, or the methods themselves!

t := LoggerTransformationTool onModel: MooseModel root first.
t findAndTransformAllLogErrorInvocations

The whole source code that was written on the three blog posts is still available on that repository.

Conclusion

Through the use of tools to query and search through FAST models, transform said models, before generating, reviewing and applying the new code on the source files, we managed to build a fully functionnal and semi-automatic transformation tool for our use case!

This use case was a rather small example. A real transformation in a software would need more queries and FAST edits. But the only real difference would be the scale. Using the methodology and tools in the same way that they were used in this blog post would be enough to apply to many use cases out there. 😄

Tags:

• transformation

Transformation journey (2/3) : Copying FASTs and creating nodes

Apr 15, 2024

Romain Degrave

Master degree intern in software analysis

Welcome back to the little series of blog posts surrounding code transformation! In the previous post, we started to build a transformation tool using a basic fictional transformation use case on the software ArgoUML. In this post, we will add behavior to the class we created previously, to achieve the actual transformation while making sure that we do not modify the original model in the process. As a reminder, this is the scenario: in the ArgoUML system, three classes define and use a method named logError. A class ArgoLogger has been defined that contains a static method logError. The transformation task is to add a receiver node to each logError method invocation so that the method called is now ArgoLogger.logError.

Preface

As this blog post follows what was discussed and built in the first one of this series, a lot of information (how to build the image and model used for our use case, but also the use case itself) is provided in the previous post. If you haven’t read it, or if you forgot the content of the post, it is recommended to go back to it, as it will not be repeated in this post.

Tools to import

For this blog post, we will have to import a tool library, FAST-Java-Tools, which contains several tools to use on FAST for Java entities. It also contains practical tools to build transformation tools! To load the project, execute this command in a Playground :

Metacello new
  baseline: 'FASTJavaTools';
  repository: 'github://moosetechnology/FAST-Java-Tools:main/src';
  load

Loading it will automatically load the FAST-Java repository. Carrefour and MoTion (which were used in the previous post) are still necessary, so if you are starting from a clean image, you will need them as well.

Copying the (F)ASTs to transform

With our import done, we are ready to get to work! 😄 To create our transformation, we will modify the FAST of each candidate method for our transformation, and this modified FAST will then be generated into source code that we will be able to inject into the actual source files. Therefore, our next step is modifying the FAST of our methods. However, we will begin by making a copy of said FAST, to ensure that our transformation will not modify the actual original model of our method. If that happens, we will still be able to run Carrefour on our method again (using the message generateFastAndBind) but it is better to avoid that scenario. If we have a bug on our tool and the FAST of many methods ends up being modified, then re-calculating and binding every FAST will be a big waste of time. Making a copy allows us to make the same actions, but ensures that our model remains untouched.

To create a copy, we have a visitor that fortunately does all the job for us! Using this simple command will give you a FAST copy :

FASTJavaCopyVisitor new copy: aFASTMethod

However, to make things even easier, we will use an already made wrapper that will contain our candidate method for transformation in every useful “format” for transformation, which are :

• The Famix entity of that method
• The original FAST
• The copied (transformed) FAST

{::comment} This wrapper also has collections to store specific nodes, but patience, we will see how to use these in the next blog post! 😉 {:/comment}

For now, let’s add a method to create that wrapper on a collection of candidate methods, in the class created in the first post :

createWrappersForCandidateMethods: aCollectionOfFamixJavaMethods

  ^ aCollectionOfFamixJavaMethods collect: [ :fmxMeth |
    | fastMethod |
    fmxMeth generateFastIfNotDoneAndBind.
    fastMethod := fmxMeth fast.
    JavaMethodToTransformWrapper
      forFamixMethod: fmxMeth
      andFAST: fastMethod ]

With this, we created wrappers for each method to transform, and each of these wrappers contains a copy of the FAST ready to be transformed. Before getting to the next step, we will define one more method, which will contain the “main behavior” of our tool, so that calling this method is enough to follow each step in order to transform the code.

findAndTransformAllLogErrorInvocations

  | wrappers |
  wrappers := self createWrappersForCandidateMethods:
    self fetchLogErrorMethodInvocationsSenders.

  wrappers do: [ :w | self transformMethod: w ].

If you followed everything so far, you can notice that there are two new methods called here. The first one, fetchLogErrorMethodInvocationsSenders, is just here to make the code easier to read :

fetchLogErrorMethodInvocationsSenders

  ^ self fetchLogErrorMethodInvocations
    collect: [ :mi | mi sender ]

The other one, transformMethod:, is the next step in this transformation journey, creating a new receiver node for our logError invocations and applying it on said invocations.

Create a new node

Therefore, the first thing to do is pretty obvious. We need to build a receiver node, for the ArgoLogger class. This transformation use case is pretty easy, so the node to create is always the same and is a pretty basic node.

We can in this case simply build this node “manually” in a method :

createNewReceiverNode

  ^ FASTJavaIdentifier new
    name: 'ArgoLogger';
    yourself

However, this option is not possible for every use case. Sometimes the node to create will be dependent on the context (the name of an identifier, method or variable might change), or we might have to create more complex nodes, composed of several nodes. In order to create those more complex nodes, one option is to parse the code using the parsing tool from the FAST-Java repository, the JavaSmaCCProgramNodeImporterVisitor, before inspecting the parsed result using the FASTDump view.

Using this view of the inspector then enable us to get the node or tree that we seek for our transformation and copy-paste the code to create it in a method for our tool to use it when needed.

Adding the new receiver node to (F)ASTs

Now that we have created our node and saw different means to do so, all that remains to do is setting it as the receiver of every copied FAST method to complete our transformation! Let us do so right away, by finally implementing our transformMethod: from before:

transformMethod: aJavaMethodWrapper

  | methodInvocationNode |
  methodInvocationNode := self motionQueryForFastMethod:
    aJavaMethodWrapper transformedFastMethod.

  methodInvocationNode receiver: self createNewReceiverNode

There are two things worth saying after making this method. First, and hopefully without boring you by repeating this over and over… This is a very simple transformation use case! As you can see, the only thing we do is using the receiver: setter on the appropriate node. Depending on the kind of edit that you want to apply on a method, you might need to experiment a bit and read the comments and method list of the classes of the nodes you are trying to edit. To experiment, one way is inspecting one of the FAST method that you wish to transform, to find the nodes you need to locate and transform, then go and read the class documentation and method list of the types of those nodes to find the appropriate setters you need to use. Another fun way to do so involves the Playground and using once again the parsing tool used in the previous section to use the FASTDump view of the Moose Inspector.

Using this little code snippet, you can parse a Java method and inspect its FAST. So, copy the method you need to transform, inspect it, then transform the code of this method manually, then parse it and inspect it again! It’s an easy way to start the work towards building a transformation tool. 😄

The second thing to notice, is the use of MoTion and not Carrefour. As we now modify a copy of a FAST, it is easier to run the MoTion query on it to get the node we are looking for right away, rather than go through the original Famix invocation. However, it is still doable, as the original and copied FAST nodes both store their counterpart in an attribute slot. We can see this by inspecting one of the fast methods in a wrapper resulting from our transformation (look at the inspected slots and mooseID of each object).

Try it yourself!

We are now done with the second part of our three blog post journey into code transformation! Our class is now able to create wrappers for each candidate method containing a copy of the FAST to transform, creating the new nodes to insert and finally make the edit on the copied FAST.

Just like with the first post, feel free to now try the tool and methods that we created here in a Playground and experiment with the results, or the methods themselves!

t := LoggerTransformationTool onModel: MooseModel root first.
t findAndTransformAllLogErrorInvocations

The whole source code that was written on this blog post (and the previous one) is also available on that repository.

Conclusion

With the help of some tools and wrappers for FAST-Java, we managed to copy the FASTs of our candidate methods for our small transformation use case, before creating the necessary node for the transformation. In the next and final blog post on code transformation, we will see how to view and edit our transformation before confirming its content, and applying it to the source files. Until next time! 😄

Tags:

• transformation

Newer posts

Older posts