When you share your work with others, such as the previous blog post, you can get feedback and suggestions for improvement.
And this is exactly what happened.
And by Distribution Map, we are talking about a research paper accepted at ICSM 2006
So, let’s not reinvent the wheel anymore and look at how to use the already existing Distribution Map in Moose to visualize the ownership of a system.
The first visualisation that comes to mind is to use one color for each developer.
Using the Degree-of-Authorship (DOA) formula, we can compute the ownership of each file and retrieve the main developer for each file.
In moose, we can then create a tag that will associate a color to each developer and then use this tag to color the files in the system.
We assign a tag to each file that corresponds to the main developer of the file.
We first search for all the files that have been modified, then we compute the degreeofAuthorship for every possible user, we select the one with the best score, which is 1, and we create a tag with the author name.
Once the tag is created for every file, it is possible to propagate the files to the Distribution Map.
Based on the parent folders, the Moose Distribution Map will be able to recalculate the children, and so the files of interest.
We select the folders variable and propagate with Cmd + Shift + M, Cmd + Shift + P.
Then, we open the Distribution map from the Moose Menu.
We first get a visualisation without any color.
So, in the settings of the visualisation, we select the tags to display.
We select every authors and get the last visualisation.
Again, the visualisation well presents the different packages and the one with major author.
The gray square corresponds to file less modified in the system the last two years.
When managing a software system, it is important to know who is responsible for which part of the system.
This information can be useful for various reasons, such as identifying potential bottlenecks, understanding the impact of changes, and improving collaboration among team members.
One major threat could, for example, come from a developer leaving the team.
This is often referred to as the “bus factor” or “truck factor”.
But how to detect such a situation?
Research have shown that the ownership of a system can be inferred from its version control history.
To do so, one approach is to first investigate the ownership of each file.
Two nice papers, not from the Moose community, are:
Looking at the research paper, we can find a formula that can help build a first version of ownership.
It consists for each file of a git repository to compute a degree of authorship for each potential author using this formula.
Given a file f with path fp, the degree-of-authorship of a developer d whose Git user has been mapped to md is given by.
DOA depends on three factors: (i) first authorship (FA): if md originally created f, FA is 1; otherwise it is 0; (ii) number of deliveries (DL): number of changes in f made by md; and (iii) number of acceptances (AC): number of changes in f made by any developer, except md.
Looking at the source code of the Truck-Factor tool, we can see thet DL corresponds to a commit that modifies a file.
The file can be added, modified, and renamed.
AC(md,fp) is computed by the total number of modifications of a file minus the number of modifications made by the developer md.
Moose comes with the GitProject Health project to represent a Git repository and analyze its history.
So, I wanted to implement the DOA formula in Moose.
One concern is that GitProject Health is firstly made to analyze Git Social Platform and not only git repository with the truck factor in mind.
Thus, it is not optimized for this use case.
The major concern comes when loading the model that rely on several calls to the Git Platform API.
In order to avoid this, I decided to implement a companion importer named GitLocalModelImporter that can load a model from a local git repository.
To go faster, it relies on the FFI LibGit2.
This project includes a LGitRevwalk, a walker that can iterate over the commits of a git repository.
Thus, we can iterate over the commits and retrieve the files, the diffs, the commits, and the authors in a repository.
This version is still not optimized to compute truck factor, but it can be used to do lot more than just computing truck factor. It comes as a new classic importer for GitProject Health and so can be used side by side with other importers for optimization.
To implement the DOA formula, I created a simple but effective class named GLPDegreeOfAuthorshipAnalyzer.
This class is now part of the GitProject Health project set of metrics.
It reuses the exact same algorithm as the Truck-Factor tool to compute the degree of authorship for each file in a git repository.
We note several additionnal considerations to compute the DOA formula.
These considerations are mention because they enable analysis of a project without considering the history of the project, which is long to consider.
The first authorship (FA) is computed by looking at the first commit that added or modified a file to the repository.
GitProject Health does not consider renaming of files yet, so it does not consider file renaming.
Even though not considering rename can create issues, this work has been performed on an industrial project and considering renaming would not drastically change the results of the analysis.
The last step I wanted to investigate is to visualize the ownership of a system.
I did not create a vizualization dedicated to this point yet.
But let’s investigate a running example for the pharo project.
After cloning the project, I create a Moose image with the GitProject Health project installed.
Then, I load the pharo project.
glhModel := GLHModel new.
repository := GLHRepository new
cacheAt: #localImporterReference
put: '/path/to/MSR/pharo' asFileReference;
yourself.
glhModel add: repository.
localImporter := GitLocalModelImporter new.
localImporter withFiles: true.
localImporter glhModel: glhModel.
"I will import the last two years"
localImporter withCommitsSince: 2 years.
localImporter importRepository: repository.
After a possibly long execution, we get the repository loaded.
It is thus possible to investigate it using our tools.
We want to build a cartography of the application to identify the coverage of files with an author.
To do so, I decided to build a TreeMap using the Roassal framework.
b shapes @ (RSPopup text: [:f| f path, String crlf, ((filesAuthorShip at: f ifPresent: [ :dic| dic sum ] ifAbsent: [ 0 ]) printString) ]).
b build
Using this script, we built a vizualisation with different color for each package, and a red scale for know packages.
It ouput the following image
Looking at the image we see that some package have author from the 2 last year modification whereas other were not modified at all.
More than showing the Truck Factor, this vizualization highlights the code that have been updated by people that gain or conserve knowledge on the codebase.
One improvment could be to highlight the core contributors.
author := (dic associations sorted: [ :a:b| a value> b value ]) first key.
box color: (paletteContributor scale: author) ] ]
ifFalse: [
| path |
path := (box model path splitOn: '/').
box color: Color white ] ] ].
This attributes the color of the core contributors to each box.
We present below a final computed visualization that highlight one contributor who seems to became the expert of the Refactoring package.
Recently, I needed to implement a feature in FAST-Python requiring a visitor for my metamodel. People knowing me knows that I if I can automatize a task to save time and to keep the code coherant with the model, I’ll go for it. Nicolas Anquetil and Clotilde Toullec recently started to implement a visitor generator, so I tried it. Since it was a POC, I encountered some problems.
I’ll explain in this blog post how to use this visitor generator, what problems I encountered and the solutions I proposed.
Once my changes will be integrated, using the visitor generator is as simple as adding one method on the class side of your generator. For example, in FAST-Python:
The first one I encountered is that FAST-Python is depending on FAST that is depending on Famix that is depending on MooseQuery.
In order to have the FAST-Python visitor working, all dependencies needed to have a visitor themselves which was not the case. So I had to generate a FAST visitor for example. I could do it since I am part of the moosetechnology organization, but this is a hassle.
The solution I found was to generate standalone visitors. Instead of using the visitors of the sub metamodels, now we generate a visitor containing the visit of all entities of the model and the visit of the remote trait used in the model.
This can seems like a lot of useless code at first, but it has two advantages:
Visitors and now standalone
This allow to customize the visit of remote traits for the language we are currently working on. This will be important for one of the other problems I encountered.
The second problem I faced was that the generated visitor had infinit loops while I tried to use it. The reason is simple, when visiting an entity, we were visiting all its relations, but Moose is a cyclic graph since we have relations to the contained entities and relations to the containers of the entity.
Since most analysis that we are doing are “top down” (meaning that we start by the top level entities of our project and we visit their children) and we do not often need a “bottom up” visitor (for this we can just iterate on parents), I decided to exclude the visit of parent entities from the visit.
In some cases I noticed the we visited two times the same nodes. This is due to the fact that Famix model mix the usage of superclasses and traits. Here is a simple case to understant what is happening:
Here we see that a FASTPyReturnStatement inherits from FASTPyStatement. This class is using FASTTStatement. FASTPyReturnStatement also uses the trait FASTPyReturnStatement, but this trait also uses FASTTStatement.
Now when we visit FASTPyReturnStatement, we end up visiting two times FASTTStatement. Once via its superclass and once via its trait composition.
The solution I proposed is that when we visit a trait composition, we check the users of the trait and if some of them already visit the trait in their superclasses, we skip the visit of this relation.
With this, #visitFASTPyReturnStatement in the context of FAST-Python become:
visitFASTTReturnStatement: aTReturnStatement
<generated>
"We should visit FASTTStatement but all its users in this language already visit it in their superclasses so we skip the call here.".
self visitEntity: aTReturnStatement expression
In some cases, we need to skip the visit for some entities, but not all of them. For example, with conditional statements. In that case we generate a method like this:
"We do not visit all behaviorals because some classes already visit it in their superclasses in this language implementation. Visiting them here also would cause a double visit of this trait."
With those changes I’m hoping it will be easier to generate and use visitors with Famix. The big advantage is that if we do not need to touch the visitor by hand, it will follow all the evolutions of the metamodel.
In Java, we can define behavior that is executed exclusively at the initialization of an instance. For now, our metamodel represented these behaviors as methods. This evolution represents them as Initializers.
We consider as initializers the following elements:
Constructors: they are called when creating a new instance. When a constructor is called, if no explicit call is defined, it implicitly calls the default no-argument constructor, that calls the no-argument constructor in the superclass. We do not represent implicit constructors and these invocations.
Initialization blocks: blocks that are executed when a new instance is created. They are copied by the Java compiler into each constructor and avoid code duplication. We do not represent this implicit invocation.
In Famix: the <Initializer> method: we create a method to hold all attribute definitions in a type.
The main motivation for this change is to adapt the metamodel to the needs of building call graphs.
Call graphs must be able to create the implicit invocations described above and to distinguish between the 3 types of initializers.
Another motivation is to differentiate between initializers and actual methods.
In analyses, we often need to focus on methods and initializers can add noise when treated as actual methods, especially the <Initializer> method.
We introduce FamixJavaInitializer, a subclass of FamixJavaMethod.
An Initiliazer has 2 properties:
#isInitializationBlock: boolean, false by default.
#isConstructor: boolean, derived. In java, a constructor is an initializer with the same name as its parent type, with no declared type (or void as declared type).
We do not merge all initializers as we did before, but we still merge similar initializers, that will always be called together.
In a Java model, a type (TWithMethods) can define a maximum of 4 initializers (besides constructors):
An instance initialization block, that is the merge of all instance initialization blocks.
An instance-side <Initializer> method, similar to the one we created before.
A static initialization block, merge of all static initialization blocks (isClassSideis true).
A static <Initializer>method (isClassSideis true) for static attributes definition.
When inspecting a Java model, initializers can now be found under Initializers and Model initializers.
In order to be able to work with Moose there is a prerequisite we cannot avoid: we need a model to analyze. This can be archieved in 2 principal ways:
Importing an existing JSON/MSE file containing a model
Importing a model via a Moose importer such as the Pharo importer or Python importer
While doing this, we create a lot of entities and set a lot of relations. But this can take some time. I found out that this time was even bigger than I anticipated while profiling a JSON import.
Here is the result of the profiling of a JSON of 330MB on a Macbook pro M1 from 2023:
Form this profiling we can see that we spend 351sec for this import. We can find more information in this report:
On this screenshot we can see some noise due to the fact that the profiler was not adapted to the new event listening loop of Pharo. But in the leaves we can also see that most of the time is spent in FMSlotMultivaluedLink>>#indexOf:startingAt:ifAbsent:.
This is used by a mecanism of all instance variables that are FMMany because those we do not want duplicated elements. Thus, we check if the collection contains the element before adding it.
But during the import of a JSON file, we should have no duplicates making this check useless. This also explains why we spend so much time in this method: we always are in the worst case scenario: there is no element matching.
In order to optimize the creation of a model when we know we will not create any duplicates, we can disable the check.
For this, we can use a dynamic variable declaring that we should check for duplicated elements by default, but allowing to disable the check during the execution of some code.
Now let’s try to import the same JSON file with the optiwization enabled:
We can see that the import time went from 351sec to 113sec!
We can also notice that we do not have one bottleneck in our parsing. This means that it will be harder to optimize more this task (even if some people still have some ideas on how to do that).
This optimization has been made for the import of JSON but it can be used in other contexts.
For example, in the Moose Python importer, the implementation is sure to never produce a duplicate. Thus, we could use the same trick this way:
FamixPythonImporter >> import
FMShouldCheckForDuplicatedEntitiesInMultivalueLinks value: false during: [ super import ]
