Skip to content

Blog

Mixing up analysis of Project Architecture (Famix) and its history (Mining Software Repository)

Famix is the meta-model family of Moose allowing one to analyse the dependencies of a project and its global architecture. But sometimes, you want more, you want to see what are the part of the code that are modified most, are critical, etc. It is for instance what people do when using Sonar, Coverage analysis, and so on.

To explore the history of existing projects, Mining Software Repositories (MSR) is the way to go. Moose also includes a set of tools to explore the history of a project, and to combine it with the Famix analysis. The project is named GitProjectHealth, and the full documentation is available here.

But let’s today perform an example. I have this nice HR application that I want to investigate. It is large, and I want to retrieve information between the architecture and the history of the project.

So let’s do a step by step blog about this story.

I am working with Moose12. But this story should last for future version (I mean, I don’t know how long, but it should not change a lot) First, I clone the project, and then, I install its dependency with maven mvn install.

Then, I build a nice model using the docker version of VerveineJ.

Terminal window
docker run -e JAVA_XMX=-Xmx16G -v /path/to/hr:/src -v /path/to/.m2/repository:/dependency ghcr.io/evref-bl/verveinej:v4.2.6 -anchor assoc -alllocals -format json

I use the version 4.2.6 of VerveineJ, because I am with Moose12, but don’t hesitate to use the latest version of VerveineJ if you are with a more recent version of Moose.

Then, I load the model in Moose, and I can start to explore the architecture of the project. To load the model, I simply drag and drop the json file in Moose, and then I can start to explore the model.

To retrieve the history of the project, I use the GitProjectHealth tool. This project is not included in the Moose distribution by default, but you can download it by executing the following command in a Moose playground:

Metacello new
repository: 'github://moosetechnology/GitProjectHealth:main/src';
baseline: 'GitLabHealth';
onConflict: [ :ex | ex useIncoming ];
onUpgrade: [ :ex | ex useIncoming ];
onDowngrade: [ :ex | ex useLoaded ];
load

Then, because it is a large HR application, I will perform the retrieving of the history locally using git.

rhModel := GLHModel new.
repository := GLHRepository new
cacheAt: #localImporterReference
put: '/path/to/hr' asFileReference;
yourself.
project := GLHProject new.
project repository: repository.
rhModel add: repository.
rhModel add: project.
localImporter := GitLocalModelImporter new.
localImporter withFiles: true.
localImporter withCommitDiffs: true.
localImporter withDiffRanges: false.
localImporter glhModel: rhModel.
localImporter withCommitsSince: 30 years.
localImporter importRepository: repository.
rhModel flush.
rhModel entities do: #flush.

You’ll notice several things in the code above.

  • I created a fake project and repository to store the history of the project. This is to help the local importer and might be improved.
  • I use the withFiles option to retrieve the files of the project, and the withCommitDiffs option to retrieve the diffs of the commits. The withDiffRanges option is set to false because I don’t need it for now.
  • I use the withCommitsSince: 30 years option to retrieve all the commits of the project. You can change this value to retrieve only the last commits if you want.
  • I use the flush method to remove cache that is computed by the local importer, this is for storage efficiency.

Once it is done, I have the full history of the project, but not connected to the exsiting model of the code.

We want now to connect the GitProjectHealth model, with the Famix model. There is a tool dedicated for this usage in GitProjectHealth, it is called GPCGitToFamixConnector.

The project links the types of the Famix model with the files retrieved by the GitProjectHealth model based on the files path. To use it, simply execute the following code:

GPCGitToFamixConnector new
famixModel: myFamixModel;
glhProject: project;
connect.

it is possible that GPCGitToFamixConnector is not loaded in your Moose image. It is is the case, load the missing package from Iceberg, or check in the baseline of GitProjectHealth if it is included in the version you are using, there should be a group named “Famix”

And, that’s it… yeah, it is quite easy to connect the two models, and now you can explore the history of the project in the context of its architecture.

You want to search for the types the most modified? One classic Moose query will do the job:

(myFamixModel allUsing: FamixTType) sorted: [ :type | type appliedDiffs size ] descending

Building an ownership cartography of a system - Part 2

When you share your work with others, such as the previous blog post, you can get feedback and suggestions for improvement. And this is exactly what happened.

And by Distribution Map, we are talking about a research paper accepted at ICSM 2006 So, let’s not reinvent the wheel anymore and look at how to use the already existing Distribution Map in Moose to visualize the ownership of a system.

The first visualisation that comes to mind is to use one color for each developer. Using the Degree-of-Authorship (DOA) formula, we can compute the ownership of each file and retrieve the main developer for each file.

In moose, we can then create a tag that will associate a color to each developer and then use this tag to color the files in the system. We assign a tag to each file that corresponds to the main developer of the file.

((glhModel allWithSubTypesOf: GLHFile)
select: [ :file | file diffs size ~= 0 ])
do: [ :file | file tagWithName: ((file degreeOfAuthorship) keyAtValue: 1.0) name ].

We first search for all the files that have been modified, then we compute the degreeofAuthorship for every possible user, we select the one with the best score, which is 1, and we create a tag with the author name. Once the tag is created for every file, it is possible to propagate the files to the Distribution Map.

To propagate the files to the distribution, we first retrieve the files in our model. Then, we get the parent folder of the files.

appsFolder := (glhModel allWithSubTypesOf: GLHFileDirectory) detect: [ :fi | fi path = 'src' ].
files := (appsFolder allToScope: GLHFileBlob).
"Find parent folder"
folders := (files collect: [ :file | file directoryOwner ] as: Set).
folders

Based on the parent folders, the Moose Distribution Map will be able to recalculate the children, and so the files of interest. We select the folders variable and propagate with Cmd + Shift + M, Cmd + Shift + P.

Then, we open the Distribution map from the Moose Menu.

We first get a visualisation without any color. So, in the settings of the visualisation, we select the tags to display. We select every authors and get the last visualisation.

visu

Again, the visualisation well presents the different packages and the one with major author. The gray square corresponds to file less modified in the system the last two years.

Building an ownership cartography of a system

When managing a software system, it is important to know who is responsible for which part of the system. This information can be useful for various reasons, such as identifying potential bottlenecks, understanding the impact of changes, and improving collaboration among team members.

One major threat could, for example, come from a developer leaving the team. This is often referred to as the “bus factor” or “truck factor”. But how to detect such a situation?

Research have shown that the ownership of a system can be inferred from its version control history. To do so, one approach is to first investigate the ownership of each file.

Two nice papers, not from the Moose community, are:

Looking at the research paper, we can find a formula that can help build a first version of ownership. It consists for each file of a git repository to compute a degree of authorship for each potential author using this formula.

Given a file ff with path fpf_p, the degree-of-authorship of a developer dd whose Git user has been mapped to mdm_d is given by.

DOA(md,fp)=3.293+1.098×FA(md,fp)+0.164×DL(md,fp)0.321×ln(1+AC(md,fp))DOA(m_d, f_p) = 3.293 + 1.098 × FA(m_d, f_p) + 0.164× DL(m_d, f_p)−0.321 ×ln(1 + AC (m_d, f_p))

With the following definitions:

DOA depends on three factors: (i) first authorship (FAFA): if mdm_d originally created ff, FAFA is 1; otherwise it is 0; (ii) number of deliveries (DLDL): number of changes in ff made by mdm_d; and (iii) number of acceptances (ACAC): number of changes in ff made by any developer, except mdm_d.

Looking at the source code of the Truck-Factor tool, we can see thet DLDL corresponds to a commit that modifies a file. The file can be added, modified, and renamed.

AC(md,fp)AC (m_d, f_p) is computed by the total number of modifications of a file minus the number of modifications made by the developer mdm_d.

Moose comes with the GitProject Health project to represent a Git repository and analyze its history. So, I wanted to implement the DOA formula in Moose.

One concern is that GitProject Health is firstly made to analyze Git Social Platform and not only git repository with the truck factor in mind. Thus, it is not optimized for this use case.

The major concern comes when loading the model that rely on several calls to the Git Platform API. In order to avoid this, I decided to implement a companion importer named GitLocalModelImporter that can load a model from a local git repository.

To go faster, it relies on the FFI LibGit2. This project includes a LGitRevwalk, a walker that can iterate over the commits of a git repository. Thus, we can iterate over the commits and retrieve the files, the diffs, the commits, and the authors in a repository.

This version is still not optimized to compute truck factor, but it can be used to do lot more than just computing truck factor. It comes as a new classic importer for GitProject Health and so can be used side by side with other importers for optimization.

To implement the DOA formula, I created a simple but effective class named GLPDegreeOfAuthorshipAnalyzer. This class is now part of the GitProject Health project set of metrics. It reuses the exact same algorithm as the Truck-Factor tool to compute the degree of authorship for each file in a git repository.

We note several additionnal considerations to compute the DOA formula. These considerations are mention because they enable analysis of a project without considering the history of the project, which is long to consider.

  • The first authorship (FAFA) is computed by looking at the first commit that added or modified a file to the repository.
  • GitProject Health does not consider renaming of files yet, so it does not consider file renaming.

Even though not considering rename can create issues, this work has been performed on an industrial project and considering renaming would not drastically change the results of the analysis.

The last step I wanted to investigate is to visualize the ownership of a system. I did not create a vizualization dedicated to this point yet. But let’s investigate a running example for the pharo project.

After cloning the project, I create a Moose image with the GitProject Health project installed. Then, I load the pharo project.

glhModel := GLHModel new.
repository := GLHRepository new
cacheAt: #localImporterReference
put: '/path/to/MSR/pharo' asFileReference;
yourself.
glhModel add: repository.
localImporter := GitLocalModelImporter new.
localImporter withFiles: true.
localImporter glhModel: glhModel.
"I will import the last two years"
localImporter withCommitsSince: 2 years.
localImporter importRepository: repository.

After a possibly long execution, we get the repository loaded. It is thus possible to investigate it using our tools.

We want to build a cartography of the application to identify the coverage of files with an author. To do so, I decided to build a TreeMap using the Roassal framework.

First, I need to load the Roassal TreeMap support

Metacello new
baseline: 'Roassal';
repository: 'github://pharo-graphics/Roassal:Pharo13';
onConflictUseIncoming;
load: #( Full )

Then, start the fun.

We build a dictionary with the degree of authorship for everyfile.

filesAuthorShip := ((glhModel allWithSubTypesOf: GLHFile)
select: [ :file | file diffs size ~= 0 ])
collect: [ :file |
"This uses our analyser behind the scene"
file -> file degreeOfAuthorshipAuthors ]
as: Dictionary.

We look for the folder that is the root of the tree map, and we compute the max number of authorship that will be used for coloring the vizualisation.

appsFolder := (glhModel allWithSubTypesOf: GLHFileDirectory)
detect: [ :fi | fi path = 'src' ].
max := (filesAuthorShip collect: #sum) max.

Finally, we build the tree map.

palette := RSQualitativeColorPalette qualitative set39.
b := RSTreeMap new.
b inset: 2 asPoint.
b boxShape borderColor: Color black.
b boxShape borderWidth: 0.05.
color := NSScale linear domain: { 0. max };
range: #( white red ).
b
leafWeight: [ :f | 1 ];
explore: appsFolder nesting: [ :directory |
directory files select: [ :f | (f isKindOf: GLHFileDirectory) ] ]
leaves: [ :directory | directory files reject: [ :f | f isKindOf: GLHFileDirectory ] ].
b shapes
do: [ :box |
box borderWidth: 1.
box isSLeaf
ifTrue: [ box color: (color scale: (filesAuthorShip at: box model ifPresent: [ :dic | dic sum ] ifAbsent: [ 0 ])) ]
ifFalse: [
| path |
path := (box model path splitOn: '/').
path size > 1 ifTrue: [
box color: (palette scale: (box model path splitOn: '/') second) ] ] ].
b shapes @ (RSPopup text: [:f | f path, String crlf, ((filesAuthorShip at: f ifPresent: [ :dic | dic sum ] ifAbsent: [ 0 ]) printString) ]).
b build

Using this script, we built a vizualisation with different color for each package, and a red scale for know packages. It ouput the following image

"Result"

Looking at the image we see that some package have author from the 2 last year modification whereas other were not modified at all.

More than showing the Truck Factor, this vizualization highlights the code that have been updated by people that gain or conserve knowledge on the codebase.

One improvment could be to highlight the core contributors.

To highlight the core contributors, we first create a color palette for them.

coreContributors := (filesAuthorShip values flatCollect: #keys) asSet.
paletteContributor := RSQualitativeColorPalette qualitative flatui120.

Then, we update the first b shapes statements by:

b shapes
do: [ :box |
box borderWidth: 1.
box isSLeaf
ifTrue: [
filesAuthorShip at: box model ifPresent: [ :dic |
| author |
author := (dic associations sorted: [ :a :b | a value > b value ]) first key.
box color: (paletteContributor scale: author) ] ]
ifFalse: [
| path |
path := (box model path splitOn: '/').
box color: Color white ] ] ].

This attributes the color of the core contributors to each box. We present below a final computed visualization that highlight one contributor who seems to became the expert of the Refactoring package.

Core Contributor

Improving the generation of visitors

Recently, I needed to implement a feature in FAST-Python requiring a visitor for my metamodel. People knowing me knows that I if I can automatize a task to save time and to keep the code coherant with the model, I’ll go for it. Nicolas Anquetil and Clotilde Toullec recently started to implement a visitor generator, so I tried it. Since it was a POC, I encountered some problems.

I’ll explain in this blog post how to use this visitor generator, what problems I encountered and the solutions I proposed.

Once my changes will be integrated, using the visitor generator is as simple as adding one method on the class side of your generator. For example, in FAST-Python:

FASTPythonMetamodelGenerator class>>metamodelToolGenerators
^ { FamixVisitorGenerator }

This will tell the generator to also generate a visitor trait.

It is possible to also customize the package in which the visitor is generated by overridin #packageNameForVisitor:

FASTPythonMetamodelGenerator class>>packageNameForVisitor
^ #'FAST-Python-Visitor'

While using this visitor generator I faced a few problems.

The first one I encountered is that FAST-Python is depending on FAST that is depending on Famix that is depending on MooseQuery.

In order to have the FAST-Python visitor working, all dependencies needed to have a visitor themselves which was not the case. So I had to generate a FAST visitor for example. I could do it since I am part of the moosetechnology organization, but this is a hassle.

The solution I found was to generate standalone visitors. Instead of using the visitors of the sub metamodels, now we generate a visitor containing the visit of all entities of the model and the visit of the remote trait used in the model.

This can seems like a lot of useless code at first, but it has two advantages:

  • Visitors and now standalone
  • This allow to customize the visit of remote traits for the language we are currently working on. This will be important for one of the other problems I encountered.

The second problem I faced was that the generated visitor had infinit loops while I tried to use it. The reason is simple, when visiting an entity, we were visiting all its relations, but Moose is a cyclic graph since we have relations to the contained entities and relations to the containers of the entity.

Since most analysis that we are doing are “top down” (meaning that we start by the top level entities of our project and we visit their children) and we do not often need a “bottom up” visitor (for this we can just iterate on parents), I decided to exclude the visit of parent entities from the visit.

In some cases I noticed the we visited two times the same nodes. This is due to the fact that Famix model mix the usage of superclasses and traits. Here is a simple case to understant what is happening:

Double visit example

Here we see that a FASTPyReturnStatement inherits from FASTPyStatement. This class is using FASTTStatement. FASTPyReturnStatement also uses the trait FASTPyReturnStatement, but this trait also uses FASTTStatement.

Now when we visit FASTPyReturnStatement, we end up visiting two times FASTTStatement. Once via its superclass and once via its trait composition.

The solution I proposed is that when we visit a trait composition, we check the users of the trait and if some of them already visit the trait in their superclasses, we skip the visit of this relation.

With this, #visitFASTPyReturnStatement in the context of FAST-Python become:

visitFASTTReturnStatement: aTReturnStatement
<generated>
"We should visit FASTTStatement but all its users in this language already visit it in their superclasses so we skip the call here.".
self visitEntity: aTReturnStatement expression

In some cases, we need to skip the visit for some entities, but not all of them. For example, with conditional statements. In that case we generate a method like this:

visitFASTTConditionalStatement: aTConditionalStatement
<generated>
"We do not visit all behaviorals because some classes already visit it in their superclasses in this language implementation. Visiting them here also would cause a double visit of this trait."
({ FASTPyIfStatement . FASTPyWhileStatement } includes: aTConditionalStatement class)
ifFalse: [ self visitFASTTStatement: aTConditionalStatement ].
self visitFASTTWithCondition: aTConditionalStatement

This is possible only because we generate standalone visitors now.

With those changes I’m hoping it will be easier to generate and use visitors with Famix. The big advantage is that if we do not need to touch the visitor by hand, it will follow all the evolutions of the metamodel.

Have fun with this :)

Introducing Java initializers

In Java, we can define behavior that is executed exclusively at the initialization of an instance. For now, our metamodel represented these behaviors as methods. This evolution represents them as Initializers.

We consider as initializers the following elements:

  • Constructors: they are called when creating a new instance. When a constructor is called, if no explicit call is defined, it implicitly calls the default no-argument constructor, that calls the no-argument constructor in the superclass. We do not represent implicit constructors and these invocations.
  • Initialization blocks: blocks that are executed when a new instance is created. They are copied by the Java compiler into each constructor and avoid code duplication. We do not represent this implicit invocation.
  • In Famix: the <Initializer> method: we create a method to hold all attribute definitions in a type.

The main motivation for this change is to adapt the metamodel to the needs of building call graphs. Call graphs must be able to create the implicit invocations described above and to distinguish between the 3 types of initializers.

Another motivation is to differentiate between initializers and actual methods. In analyses, we often need to focus on methods and initializers can add noise when treated as actual methods, especially the <Initializer> method.

We introduce FamixJavaInitializer, a subclass of FamixJavaMethod. An Initiliazer has 2 properties:

  • #isInitializationBlock: boolean, false by default.
  • #isConstructor: boolean, derived. In java, a constructor is an initializer with the same name as its parent type, with no declared type (or void as declared type).

We do not merge all initializers as we did before, but we still merge similar initializers, that will always be called together. In a Java model, a type (TWithMethods) can define a maximum of 4 initializers (besides constructors):

  • An instance initialization block, that is the merge of all instance initialization blocks.
  • An instance-side <Initializer> method, similar to the one we created before.
  • A static initialization block, merge of all static initialization blocks (isClassSideis true).
  • A static <Initializer>method (isClassSideis true) for static attributes definition.

When inspecting a Java model, initializers can now be found under Initializers and Model initializers.