Skip to content

Blog

Carrefour: The bridge between FAMIX and FAST

To analyze software systems, the Famix meta-model provides enough abstraction to understand how models work.

However, when we are interested in details, the FAST (Famix AST) meta-model provides less abstraction and gives us more information about our model (for example expression statements, identifiers etc.).

In some situations, such as modernization/migration projects, we need the binding between the two meta models. And here Carrefour comes in!

Carrefour

Carrefour represents a two-way link between Famix and FAST, it allows one to navigate on the AST and at the same time return to the elements of FAMIX when needed.

In this blog, we are going to use a simple snippet of code to simplify and grasp all necessary concepts we should know about FAST & Carrefour & Famix. Consider the MyClass class and the following methodAB method:

class MyClass {
public int methodAB(int a, int b){
if (a > b) {
a = a + 2;
} else {
b = 1;
}
return b;
}
}

Let’s prepare the ground for using Carrefour by generating the Famix model of the MyClass class using VerveineJ. Open a code editor and create a new MyClass.java file. Inside the Java file, we add the code above.

To generate the Famix Java model we use VerveineJ by running this command in the MyClass java file directory:

Terminal window
/path/to/VerveineJ/verveinej.sh -format json -o MyClass.json -anchor assoc -autocp ./ ./

PS: Note that Carrefour uses entities & associations as source anchor information, so make sure to add the option -anchor assoc.

In this section, we start from the MyClass Famix java model and we build the link between Famix and FAST using Carrefour. First, we need to install Carrefour, for example by running the following script on Moose Playground:

Metacello new
githubUser: 'badetitou' project: 'Carrefour' commitish: 'v3' path: 'src';
baseline: 'Carrefour';
load

Then we import the MyClass model and pick the first class (which is the only class MyClass).

'/path/to/MyClass.json' asFileReference readStreamDo: [ :stream |
model := FamixJavaModel new importFromJSONStream: stream
].
model rootFolder: '/path/to/MyClass/Directory/'.
method := model allModelClasses first.

Now, we call Carrefour to generate the AST (the figure below) and bind the newly created AST with Famix.

method generateFastAndBind

Class Code in left and the generated AST in right

it’s recommended to use generateFastIfNotDoneAndBind instead of generateFastAndBind in complex project and heavy computation when generating AST

To have a complete vision of the meta-models described above, we give the corresponding figures of each meta-model FAST and FAMIX:

Famix & Fast Overview

Once Carrefour has been called and the binding is done, we will have the first links between the meta-models as follows:

Famix & Fast 1st call

As an example, for the condition level variable (a>b) in FAST we would want its correspondence in FAMIX. To do this, we send the #famixVariable message to the FASTJavaVariableExpression object and we get as returned value the corresponding FAMIX variable.

FamixVariable Call

Now we go in the opposite direction, we will access all the matches of the FAMIX variable a in the FAST meta-model.

To do so, we use the #fastAccesses message as in the figure:

fastAccesses Call

Carrefour also provides the #fastDeclaration API to get where a Famix variable has been declared at the FAST meta-model level.

In conclusion, Carrefour allows us to go back and forth between FAST and FAMIX meta-models. The example used in the blog post is not complex but allowed us to see how to navigate between the two meta-models. In large projects, where there are more initialization, invocation, and relationships between entities Carrefour is crucial to perform deep analysis 💪.

Load FAST Pharo/Java model

When we are interested in the migration/modernization of projects we are using models of the project and their meta-models. Moose revolves around a powerful Famix meta-model that allows us to do several operations. For instance, previous posts present how to analyze and query a model, visualize a model with plantUML, or create a model, etc.

FAST is a meta-model that helps us understand source code in a less abstract way. Indeed, FAST is based on AST (Abstract Syntax Tree) which is close to the source code. And as the devil is in the details, FAST contains interesting elements when analyzing programs (for example some specific expression or statement), and effectively this is what makes the difference between FAST and Famix. (Consult this overview about the FAST model).

Abstraction Level

In this blog post we will explain how to load FAST Java and generate a FAST model of Java code. For this we will take ArgoUML, an open-source Java project, as an example.

First of all, we have to understand from where we are going to start and where we are going to end up. As already mentioned, we will take the AgroUML’s java code and the goal is to generate the corresponding AST and do analyses on it. To do this, 3 steps are necessary to have the AST as illustrated in the figure below:

  1. Parse Java to build a Famix model
  2. Load the model into Moose
  3. Generate the AST

Steps for generating FAST Model

Before starting, we must download the source code and the Famix model of the ArgoUML project, step 1 of the diagram above (follow this blog for more details).

Now, we will import the Famix model from the ArgoUML-0-34.json file in the Models Browser. Then, we should know that the FAST meta-model is specific to a gien programming language, i.e for Pharo code we need FAST for Pharo, for X language code we need the FAST meta-model for the X language. Right now, there are two FAST meta-models: FAST Java and FAST Pharo.

In the following, we will generate the AST of a class (or method) for Pharo/Java code in three different ways: directly from some source code, from a method in Pharo, or from a Famix entity.

To install FAST Java you can run the following script on Moose Playground:

Metacello new
githubUser: 'moosetechnology' project: 'FAST-JAVA' commitish: 'v3' path: 'src';
baseline: 'FASTJava';
load: 'all'

To install FAST Pharo use the following script:

Metacello new
baseline: 'FASTPharo';
repository: 'github://moosetechnology/FAST-Pharo:v2/src';
load: 'importer'.

In this case, we will use a specialized importer “FAST-Java importer” to import the AST from a method source code. The complete code of the method to import is between single quote (i.e. a Pharo string) in the following code:

JavaSmaCCProgramNodeImporterVisitor new
parseCodeMethodString: 'public boolean covidTest(Person person) {
if(testCovid(person) == "POSITIVE"){
return true;
} else {
return false;
}
}'

The following script imports the method #collect: of Collection :

FASTSmalltalkImporterVisitor new
runWithSource: (Collection >> #collect:) sourceCode

In this section, we will not proceed as above. Instead, we start from a class/method of the Famix Java model and we will load its FAST representation.

We will add the model to the Playground

Add Model on Playground

We got this:

argoUML034 := MooseModel root at: 1.

We pick any model class from the model:

class := argoUML034 allModelClasses anyOne.

And finally we generate the AST using generateFastJava:

class generateFastJava

One nice way to explore a FAST model is to use the source code and the tree extensions of the inspector. It allows one to navigate in a FAST model and see the code corresponding to each node.

To use it, we start from the Java model loaded above. Then, we select a model method entity. On the right-hand pane of the inspector, select the Tree tab, on the left-hand pane, select the source code extension. The source code is highlighted and the area selected corresponds to the entity selected in the right-hand panel. ( from FAST-Pharo article )

Navigating Through AST

In this post, we saw how to load the AST of a Pharo/Java model using FAST. The FAST model is useful when we need to understand more details about our model (for example identifiers, expression statements .. etc) which are not provided by Famix.

How to build a toolbar with Spec

Let’s build a simple presenter showing a counter. To manage this counter, we will build a toolbar which buttons increase, decrease or reset the value of this counter.

"Counter Presenter"

Our presenter is implemented in CounterPresenter, a subclass of SpPresenter. It will define 3 methods to manage the counter value: #increaseCounter, #decreaseCounter and #resetCounter. We will not consider the building of the presenter itself but we will focus on the toolbar.

Spec provides an API to build toolbars and add dedicated buttons in it. We will use it in the presenter initialization method: #initializePresenters, overriden from SpPresenter, to instantiate a toolbar:

toolbar := self newToolbar

Then, we manually build toolbar buttons and add them into the toolbar:

toolbar
add: (SpToolbarButtonPresenter new
label: 'Increase';
icon: (self iconNamed: #smallAdd);
action: [ self increaseCounter ]);
add: (SpToolbarButtonPresenter new
label: 'Decrease';
icon: (self iconNamed: #remotesManagerRemoveRemote);
action: [ self decreaseCounter ]);
add: (SpToolbarButtonPresenter new
label: 'Reset';
icon: (self iconNamed: #smallUpdate);
action: [ self resetCounter ]).

We also need to add the toolbar to the presenter layout. Since Pharo 10, we can do this instance side:

intializeLayout
self layout: SpBoxLayout newTopToBottom
add: toolbar height: self class toolbarHeight;
"... other widgets ...";
yourself

This version works perfectly. However, the definition of the buttons, their actions, labels and icons are only defined locally and are not easy to reuse. We will extract this behavior to Commands and use them to build our toolbar.

We use Commander, the implementation of the Command pattern in Pharo. Let’s create 3 subclasses of CmCommand, one for each of our buttons. These classses define an instance variable #context. In this case, it will be our presenter.

Each class should define the #execute method according to its expected behavior:

execute
self context increaseCounter

Our commands should also implement the class methods #defaultTitle, and #defaultIconName.

defaultTitle
^ 'Increase'
defaultIconName
^ #smallAdd

To be used in Spec, our commands are decorated as SpCommands via the method #asSpecCommand. We override this method to correctly use the icon name, as follows:

asSpecCommand
^ super asSpecCommand
iconName: self class defaultIconName;
yourself

SpCommand provides an API to be converted as buttons in spec. A first way to do it is to directly add them in the toolbar. Here is a second version of the #initializePresenters method:

toolbar
add: (IncreaseCounterCommand forSpecContext: self) buildPresenter;
add: (DecreaseCounterCommand forSpecContext: self) buildPresenter;
add: (ResetCounterCommand forSpecContext: self) buildPresenter.

Here, we give the presenter as context for the commands before building each button.

At this step, we have a functional toolbar using built with commands. We can still improve this by using commands groups

Spec presenters can define commands groups to be used in toolbars and menus via the class method #buildCommandsGroupWith:forRoot:. We implement it in our presenter, on class side:

buildCommandsGroupWith: aPresenter forRoot: aRootCommandGroup
aRootCommandGroup
register: (IncreaseCounterCommand forSpecContext: aPresenter);
register: (DecreaseCounterCommand forSpecContext: aPresenter);
register: (ResetCounterCommand forSpecContext: aPresenter)

To get this command group, SpPresenter implements #rootCommandsGroup. This method collects the commands defined in #buildCommandsGroupWith:forRoot: and set the current presenter as their context. We call it in the #initializePresenters method.

toolbar fillWith: self rootCommandsGroup.

⚠️ Be careful, #fillWith: will remove all items already present in the toolbar. In this code snippet, aButton will not be in the toolbar:

toolbar
add: aButton; "Will be removed by next line"
fillWith: aCommandGroup.

Instead of defining a toolbar as a subpresenter, it is a good practice to define the toolbar in the window directly. We remove the toolbar instance variable and all the related code in #initializePresenters and #initializeLayout. We then override #initializeWindow: from SpPresenter.

initializeWindow: aMiWindowPresenter
super initializeWindow: aMiWindowPresenter.
aMiWindowPresenter toolbar:
(SpToolbarPresenter new fillWith: self rootCommandsGroup)

Building toolbars in Spec can be done manually. However, by using commands, we separate responsibilities and we can re-use, extend and test these commands. The commands our presenter builds can be used not only in toolbars, but also in menus in a similar manner.

Introducing the new Queries Browser

Let’s say that you want to know which classes of your Moose model are stub. That means: which classes are not defined in your Moose model but are used by some of your defined classes. Those classes are part of your model, although they are not part of your code. Checking those stub classes is easy. You only have to create a Boolean query (assuming that your model contains only classes) with the property isStub. Like:

FQBooleanQuery property: #isStub

However, creating queries programmatically can be a tedious task. Of course, not all the queries are as frivolous as the one in the example. For queries with lots of children, the code is not easy to understand at first sight.

The new Queries Browser was developed to create queries in a more visual way. This is a Moose tool that allows one to create and manipulate queries without the need of knowing the FamixQueries syntax or how to instantiate them. It has a friendly and intuitive user interface.

"The brand new Queries Browser"

If you want to know:

  1. What are all the classes that are not stub.
  2. What are the entities with incoming references and inheritances.
  3. What are the entities that have more than 50 lines of code.
  4. Finally, what is the intersection of those three queries.

This is an easy task for the Queries Browser. First, we need to create a type query that filters all the entities except the classes. To do that, we select Type Query in the queries browser and then select type “Classes”.

"Type Query"

Then, we create a child query from the Type Query. This child is going to be a Boolean Query that has the property isStub.

"The brand new Queries Browser"

Now, we create a Complement Query, a.k.a Negation Query, and choose the Boolean Query to be the query to be negated.

"The brand new Queries Browser"

Now we have the first task completed: All the classes that are not stub.

For the second task, we need to create another query, a Navigation Query, select the Incoming direction, and only select the associations Reference and Inheritance.

The third one is also simple. We only need to create a Numeric Query, select the property number of lines of code, the operator >, and put the value 50.

Now, our Queries Browser looks like this:

"The brand new Queries Browser"

For the final task, we need to create an And Query, a.k.a Intersection Query, click on the ”+” button to add a new query to intersect, and select the three previous queries that we created.

"The brand new Queries Browser"

If we wanted to the the above queries programmatically, the code would have look like this:

(FQComplementQuery queryToNegate:
(FQTypeQuery types: { FamixStClass })
--> (FQBooleanQuery property: #isStub))
& (FQNavigationQuery incoming associations: {
FamixStReference.
FamixStInheritance }) & (FQNumericQuery
property: #numberOfLinesOfCodeWithMoreThanOneCharacter
comparator: #>
valueToCompare: 50)

The code is also shown in the “Current query code tab”: "The brand new Queries Browser"

As you may already noticed, there are two button for saving and loading the queries. The save button saves all queries that are currently present on the queries browser as a STON file. The queries will be saved inside a folder in the same location as the image. The path is determinated in MiSaveQueriesCommand class>>#path.

When loading the queries, the saved queries will be put after the queries that are present in the browser, if any. For example, if we save the queries that we have created above.

"Save queries as"

Then, with an empty browser, we create a new query and then loading the file we get:

"Load queries"

The new Queries Browser can simplify how the Famix Queries are created and make it more visual and understandable. Even if the example in this post is not complex, it made our tasks easier. Analyzing models with real-life examples can lead to very nested queries. Create those queries programmatically can be very tedious and error-prone. The Queries Browser is here to help us in those cases. 😄

Migrating internationalization files

During my Ph.D. migration project, I considered the migration of several GUI aspects:

  • visual
  • behavioral
  • business

These elements are the main ones. When perfectly considered, you can migrate the front-end of any application. But, we are missing some other stuff 😄 For example, how do you migrate i18N files?

In this post, I’ll present how to build a simple migration tool to migrate I18N files from .properties (used by Java) to .json format (used by Angular).

First, let’s see our source and target.

As a source, I have several .properties files, including I18N for a Java project. Each file has a set of key/value and comments. For example, the EditerMessages_fr.properties is as follow:

##########
# Page : Edit
##########
pageTitle=Editer
classerDemande=Demande
classerDiffusion=Diffusion
classerPar=Classer Par

And it’s Arabic version EditerMessages_ar.properties

#########
# Page : Editer
#########
pageTitle=تحرير
classerDemande=طلب
classerDiffusion=بث
classerPar=تصنيف حسب

As a target, I need only one JSON file per language. Thus, the file for the french translation looks like this:

{
"EditerMessages" : {
"classerDemande" : "Demande",
"classerDiffusion" : "Diffusion",
"classerPar" : "Classer Par",
"pageTitle" : "Editer"
}
}

And the Arabic version:

{
"EditerMessages" : {
"classerDemande" : "طلب",
"classerDiffusion" : "بث",
"classerPar" : "تصنيف حسب",
"pageTitle" : "تحرير"
},
}

To perform the transformation from the .properties file to json, we will use MDE. The approach is divided into three main steps:

  1. Designing a meta-model representing internationalization
  2. Creating an importer of properties files
  3. Creating a JSON exporter

I18N files are simple. They consist of a set of key/values. Each value is associated with a language. And each file can be associated with a namespace.

For example, in the introduction example, the namespace of all entries is “EditerMessages”.

I designed a meta-model to represent all those concepts:

meta-model

Once the meta-model is designed, we must create an importer that takes .properties files as input and produces a model.

To produce a model, I first look for a .properties parser without much success. Thus, I decided to create my own parser. Given a correctly formatted file, the parser provides me the I18N entries. Then, by iterating on this collection, I build an I18N model.

To implement the parser, I used the PetitParser2 project. This project aims to ease the creation of new parsers.

First, I downloaded the last version of Moose, and I installed PetitParser using the command provided in the repository Readme:

Metacello new
baseline: 'PetitParser2';
repository: 'github://kursjan/petitparser2';
load.

In my Moose Image, I created a new parser. To do so, I extended the PP2CompositeNode class.

PP2CompositeNode << #CS18NPropertiesParser
slots: { };
package: 'Casino-18N-Model-PropertyImporter'

Then, I defined the parsing rules. Using PetitParser2, each rule corresponds to a method.

First, start is the entry point.

start
^ pairs end

pairs parses the entries of the .properties files.

pairs
^ comment optional starLazy, pair , ((newline / comment) star , pair ==> [ :token | token second ]) star , (newline/comment) star ==> [ :token |
((OrderedCollection with: token second)
addAll: token third;
yourself) asArray ]

The first part of this method (before ==>) corresponds to the rule parsed. The second part (after ==>), to the production.

The first part tries to parse one or several comment. Then, it parses one pair followed by a list of comment, newline, and pair.

This parser is clearly not perfect and would require some improvement. Nevertheless, it does work for our context.

The second part produces a collection (i.e. a list) of the pair.

Now that we can parse one file, we can build a I18N model. To do so, we will first parse every .properties file. For each file, we extract the language and the namespace based on the file name. Thus, EditerMessages_fr.properties is the file for the fr language and the EditerMessages namespace. Then, for each file entry, we instantiate an entry in our model inside the namespace and with the correct language attached.

importString: aString
(parser parse: aString) do: [ :keyValue |
(self model allWithType: CS18NEntry) asOrderedCollection
detect: [ :entry |
"search for existing key in the file"
entry key name = keyValue key ]
ifOne: [ :entry |
"If an entry already exists (in another language for instance)"
entry addValue: ((self createInModel: CS18NValue)
name: keyValue value;
language: currentLanguage;
yourself) ]
ifNone: [
"If no entry exist"
(self createInModel: CS18NEntry)
namespace: currentNamespace;
key: ((self createInModel: CS18NKey)
name: keyValue key;
yourself);
addValue: ((self createInModel: CS18NValue)
name: keyValue value;
language: currentLanguage;
yourself);
yourself ] ]

After performing the import, we get a model with, for each namespace, several entries. Each entry has a key and several values. Each value is attached to the language.

To perform the JSON export, I used the NeoJSON project. NeoJSON allows one to create a custom encoder.

For the export, we first select a language. Then, we build a dictionary with all the namespaces:

rootDic := Dictionary new.
(model allWithType: CS18NNamespace)
select: [ :namespace | namespace namespace isNil ]
thenDo: [ :namespace | rootDic at: namespace name put: namespace ].

To export a namespace (i.e., a CS18NNamespace), I define a custom encoder:

writter for: CS18NNamespace customDo: [ :mapper |
mapper encoder: [ :namespace | (self constructNamespace: namespace) asDictionary
]
].
constructNamespace: aNamespace
| dic |
dic := Dictionary new.
aNamespace containables do: [ :containable |
(containable isKindOf: CS18NNamespace)
ifTrue: [ dic at: containable name put: (self constructNamespace: containable) ]
ifFalse: [ "should be an CS18NEntry"
dic at: containable key name put: (containable values detect: [ :value | value language = language ] ifOne: [ :value | value name ] ifNone: [ '' ]) ] ].
^ dic

The custom encoder consists on converting a Namespace into a dictionary of entries with the entries keys and their values in the selected language.

Once my importer and exporter are designed, I can perform the migration. To do so, I use a little script. It creates a model of I18N, imports several .properties file entries in the model, and exports the Arabic entries in a JSON file.

"Create a model"
i18nModel := CS18NModel new.
"Create an importer"
importer := CS18NPropertiesImporter new.
importer model: i18nModel.
"Import all entries from the <myProject> folder"
('D:\dev\myProject\' asFileReference allChildrenMatching: '*.properties') do: [ :fileRef |
self record: fileRef absolutePath basename.
importer importFile: fileRef.
].
"export the arabian JSON I18N file"
'D:/myFile-ar.json' asFileReference writeStreamDo: [ :stream |
CS18NPropertiesExporter new
model: importer model;
stream: stream;
language: ((importer model allWithType: CS18NLanguage) detect: [ :lang | lang shortName = 'ar' ]);
export
]

The meta-model, importer, and exporter are freely available in GitHub.