Skip to content

Clotilde Toullec

3 posts by Clotilde Toullec

Parametrics next generation

How do we represent the relation between a generic entity, its type parameters and the entities that concretize it? The Famix metamodel has evolved over the years to improve the way we represent these relations. The last increment is described in a previous blogpost. We present here a new implementation that eases the management of parametric entities in Moose.

The major change between this previous version and the new implementation presented in this post is this: We do not represent the parameterized entities anymore.

What’s wrong with the previous parametrics implementation?

Section titled “What’s wrong with the previous parametrics implementation?”

Difference between parametric and non-parametric entities

Section titled “Difference between parametric and non-parametric entities”

The major issue with the previous implementation was the difference between parametric and non-parametric entities in practice, particularly when trying to trace the inheritance tree. Here is a concrete example: getting the superclass of the superclass of a class.

  • For a non-parametric class, the sequence is straightforward: ask the inheritance for the superclass, repeat.

Getting super inheritances - Non-parametric entities.

  • For a parametric class (see the little code snippet below), there was an additional step, navigating through the concretization:
import java.util.ArrayList; "public class ArrayList<E> { /* ... */ }"
public MySpecializedList extends ArrayList<String> {}

Getting super inheritances - Parametric entities.

This has caused many headaches to developers who wanted to browse a hierarchy: how do we keep track of the full hierarchy when it includes parametric classes? How to manage both situations without knowing if the classes will be parametric or not? The same problem occurred to browse the implementations of parametric interfaces and the invocations of generic methods.

The previous implementation naming choices were a little complex to grasp and did not match the standard vocabulary, especially in Java:

  • A type parameter was named a ParameterType
  • A type argument was named a ConcreteParameterType

Each time there was a concretization, a parametric entity was created. This created duplicates of virtually the same entity: one for the generic entity and one for each parameterized entity. Let’s see an example:

public MyClass implements List<Float> {
public List<Integer> getANumber() {
List<Number> listA;
List<Integer> listB;
}
}

For the interface List<E>, we had 6 parametric interfaces:

  • One was the generic one: #isGeneric >>> true
  • 3 were the parameterized interfaces implemented by ArrayList<E>, its superclass AbstractList<E> and MyClass. They were different because the concrete types were different: E from ArrayList<E>, E from AbstractList<E>and Float.
  • 2 were declared types: List<Number> and List<Integer>.

When deciding of a new implementation, our main goal was to create a situation in which the dependencies would work in the same way for all entities, parametric or not. That’s where we introduce parametric associations. These associations only differ from standard associations by one property: they trigger a concretization.

Here is the new Famix metamodel traits that represent concretizations:

Class diagram for Parametric Associations

There is a direct relation between a parametric entity and its type parameters. A concretization is the association between a type parameter and the type argument that replaces it. A parametric association triggers one or several concretizations, according to the number of type parameters the parametric entity has. Example: a parametric association that targets Map<K,V> will trigger 2 concretizations.

The parametric entity is the target of the parametric association. It is always generic. As announced, we do not represent parameterized entities anymore. Coming back to the entities’ duplication example above, we now represent only 1 parametric interface for List<E>and it is the target of the 5 parametric associations.

This metamodel evolution is the occasion of another major change: the replacement of the direct relation between a typed entity and its type. This new association is called Entity typing.

Class diagram for Entity Typing

The choice to replace the existing relation by a reified association is made to represent the dependency in coherence with the rest of the metamodel.

With this new association, we can now add parametric entity typings.

In a case like this:

public ArrayList<String> myAttribute;

we have an “entity typing” association between myAttribute and ArrayList. This association is parametric: it triggers the concretization of E in ArrayList<E> by String.

Type parameters can be bounded:

public class MyParametricClass<T extends Number> {}

In the previous implementation, the bounds of type parameters were implemented as inheritances: in the example above, Number would be the superclass of T. Since this change, bounds were introduced for wildcards. We have now the occasion to also apply them to type parameters. In the new implementation, Number is the upper bound of T.

This diagram sums up the new parametrics implementation in Famix traits and Java metamodel. Please note that this is not the full Java metamodel but only a relevant part.

Class diagram for all changes

Should Concretization really be an association?

Section titled “Should Concretization really be an association?”

The representation of parametric entities is a challenge that will most likely continue as Famix evolves. The next question will probably be this one: should Concretization really be an association? An association is the reification of a dependency. Yet, there is no dependency between a type argument and the type parameter it replaces. Each can exist without the other. The dependency is in fact between the source of the parametric association and the type parameter.

With one of our previous examples:

public MySpecializedList extends ArrayList<String> {}

MySpecializedList has a superclass (ArrayList<E>) and also depends on String, as a type argument. However, String does not depend on E neither E on String.

The next iteration of the representation of parametric entities will probably cover this issue. Stay tuned!

How to build a toolbar with Spec

Let’s build a simple presenter showing a counter. To manage this counter, we will build a toolbar which buttons increase, decrease or reset the value of this counter.

"Counter Presenter"

Our presenter is implemented in CounterPresenter, a subclass of SpPresenter. It will define 3 methods to manage the counter value: #increaseCounter, #decreaseCounter and #resetCounter. We will not consider the building of the presenter itself but we will focus on the toolbar.

Spec provides an API to build toolbars and add dedicated buttons in it. We will use it in the presenter initialization method: #initializePresenters, overriden from SpPresenter, to instantiate a toolbar:

toolbar := self newToolbar

Then, we manually build toolbar buttons and add them into the toolbar:

toolbar
add: (SpToolbarButtonPresenter new
label: 'Increase';
icon: (self iconNamed: #smallAdd);
action: [ self increaseCounter ]);
add: (SpToolbarButtonPresenter new
label: 'Decrease';
icon: (self iconNamed: #remotesManagerRemoveRemote);
action: [ self decreaseCounter ]);
add: (SpToolbarButtonPresenter new
label: 'Reset';
icon: (self iconNamed: #smallUpdate);
action: [ self resetCounter ]).

We also need to add the toolbar to the presenter layout. Since Pharo 10, we can do this instance side:

intializeLayout
self layout: SpBoxLayout newTopToBottom
add: toolbar height: self class toolbarHeight;
"... other widgets ...";
yourself

This version works perfectly. However, the definition of the buttons, their actions, labels and icons are only defined locally and are not easy to reuse. We will extract this behavior to Commands and use them to build our toolbar.

We use Commander, the implementation of the Command pattern in Pharo. Let’s create 3 subclasses of CmCommand, one for each of our buttons. These classses define an instance variable #context. In this case, it will be our presenter.

Each class should define the #execute method according to its expected behavior:

execute
self context increaseCounter

Our commands should also implement the class methods #defaultTitle, and #defaultIconName.

defaultTitle
^ 'Increase'
defaultIconName
^ #smallAdd

To be used in Spec, our commands are decorated as SpCommands via the method #asSpecCommand. We override this method to correctly use the icon name, as follows:

asSpecCommand
^ super asSpecCommand
iconName: self class defaultIconName;
yourself

SpCommand provides an API to be converted as buttons in spec. A first way to do it is to directly add them in the toolbar. Here is a second version of the #initializePresenters method:

toolbar
add: (IncreaseCounterCommand forSpecContext: self) buildPresenter;
add: (DecreaseCounterCommand forSpecContext: self) buildPresenter;
add: (ResetCounterCommand forSpecContext: self) buildPresenter.

Here, we give the presenter as context for the commands before building each button.

At this step, we have a functional toolbar using built with commands. We can still improve this by using commands groups

Spec presenters can define commands groups to be used in toolbars and menus via the class method #buildCommandsGroupWith:forRoot:. We implement it in our presenter, on class side:

buildCommandsGroupWith: aPresenter forRoot: aRootCommandGroup
aRootCommandGroup
register: (IncreaseCounterCommand forSpecContext: aPresenter);
register: (DecreaseCounterCommand forSpecContext: aPresenter);
register: (ResetCounterCommand forSpecContext: aPresenter)

To get this command group, SpPresenter implements #rootCommandsGroup. This method collects the commands defined in #buildCommandsGroupWith:forRoot: and set the current presenter as their context. We call it in the #initializePresenters method.

toolbar fillWith: self rootCommandsGroup.

⚠️ Be careful, #fillWith: will remove all items already present in the toolbar. In this code snippet, aButton will not be in the toolbar:

toolbar
add: aButton; "Will be removed by next line"
fillWith: aCommandGroup.

Instead of defining a toolbar as a subpresenter, it is a good practice to define the toolbar in the window directly. We remove the toolbar instance variable and all the related code in #initializePresenters and #initializeLayout. We then override #initializeWindow: from SpPresenter.

initializeWindow: aMiWindowPresenter
super initializeWindow: aMiWindowPresenter.
aMiWindowPresenter toolbar:
(SpToolbarPresenter new fillWith: self rootCommandsGroup)

Building toolbars in Spec can be done manually. However, by using commands, we separate responsibilities and we can re-use, extend and test these commands. The commands our presenter builds can be used not only in toolbars, but also in menus in a similar manner.

Analysis of Sindarin API: a modular Moose use case

How is our API used? This is the question the Pharo debugging team came to me with. They needed to know if it is used as they intended to or if there are some improvements to do. To answer that question, we used some of Moose new browsers.

The Pharo debugging team works on Sindarin, an API that eases the expression and automation of debugging tasks. Sindarin is made of two layers:

  • The “technical” API is the minimal API needed to interact with a debugging kernel (the Pharo kernel here).
  • The “scripting” API is here to ease the writing of debugging scripts without having to manipulate low-level concepts.

"Sindarin API"

Now, that’s the theory. In reality, the debugging team realized that these two layers are not really respected: users bypass the scripting API to use technical methods and the debugging kernel methods directly. We used Moose to analyze Sindarin API usage, to help understand if, and how, it should be improved.

Moose has recently been upgraded and now offers more modularity. Users can analyze their model through new specialized browsers that allow users to navigate and visualize their model entities. Entities are propagated from one browser to the others, dynamically updating their contents.

Helping the debugging team was an opportunity to test some of these browsers under operating conditions.

Analysis of the interactions of the users scripts with Sindarin and with the debugging kernel

Section titled “Analysis of the interactions of the users scripts with Sindarin and with the debugging kernel”

To analyze Sindarin, we built a model constituted of 3 key elements:

  • Users debugging scripts: script methods gathered in 4 Classes (green in the figure above)
  • Sindarin: methods from the SindarinDebugger class (both APIs, yellow and blue in the figure)
  • Pharo debugging kernel (pink in the figure)

Our goal was to understand how the users’ scripts use Sindarin and if they interact directly with the debugging kernel.

We imported the model directly from Pharo, using the Model Root browser. This browser is the entry point to modular Moose. It shows the list of installed models and allows to import new ones from MSE files or from Pharo.

"Model Root Browser"

Once the model was imported, it was automatically propagated in modular Moose, to other browsers.

Querying the model to select key elements from the model

Section titled “Querying the model to select key elements from the model”

To explore the debugging scripts, Sindarin and the debugger kernel - and the interactions between them - we used the Query browser.

The purpose of this browser is to query a model or a group of entities. It can filter entities according to their type or the value of their properties. It can also use MooseQuery to navigate associations (inheritances, invocations, accesses, and references) and scopes (containment).

"Queriest Browser"

The left pane of the browser is a visualization of the queries that were made. On the image above, no queries were made yet. The left pane only shows the model on which queries will apply. The right pane of the browser shows the result of the selected query and the code used to build it.

We obtained the 3 elements relevant to our analysis (users scripts, Sindarin, and the debugger kernel) by combining simple queries:

  • A Type query (Classes) gave us all the classes in the model, then
  • a property query (name beginsWith: 'SindarinScript') gave us the 4 scripts classes.

The two images below show how this last query was constructed:

"Query Scripts - Step 1"

"Query Scripts - Step 2"

Queries to get Sindarin API, i.e. methods of the class SindarinDebugger:
Section titled “Queries to get Sindarin API, i.e. methods of the class SindarinDebugger:”
  • A type query (Classes), gave us all classes in the model, then
  • a property query (name = 'SindarinDebugger’), gave us the Sindarin API class.
  • finally, a scope query (scope down to methods) gave us all Sindarin methods, i.e. the API.
Queries to get methods from the debugging kernel:
Section titled “Queries to get methods from the debugging kernel:”

The image below shows the sequence of queries created to obtain methods from the Context class and the Debugger-Model package:

  • Two Type queries (Classes and Packages, green on right), gave us all classes and packages in the model, then
  • two property queries (name = 'Context' and name = 'Debbuger-Model', blue), gave us the Context class and the Debugger-Model package, then
  • we made the union (pink) of the class and package, and finally
  • a scope query (scope down to methods, turquoise, bottom right query) gave us the methods they contain.

"Kernel methods"

Saving queries combinations as custom queries

Section titled “Saving queries combinations as custom queries”

We saved these combinations of simple queries as custom queries, as shown in the image below:

  • SindarinScript classes are the 4 script classes.
  • SindarinDebugger methods are the Sindarin API, i.e. methods from SindarinDebugger class
  • Debugger Kernel methods are methods from the debugging kernel, that should not be called directly from the scripts.

"Saved queries"

How do scripts interact with Sindarin and the debugging kernel?

Section titled “How do scripts interact with Sindarin and the debugging kernel?”

We added a navigation query to obtain methods that were called in the script. We then narrowed this result by selecting methods from Sindarin or from the debugging kernel.

To find which methods are called in the users scripts, we queried outgoing invocations from the scripts classes. We obtained a group of 331 candidate methods whose signatures match messages sent in the scripts:

"Script outgoing"

The next step was to compare this group of candidates with the methods from Sindarin scripting API. We established with the debugging team that scripting methods should have "scripting" as comment. This convention allowed us to get them with a property query (blue query in the figure below).

We queried for the intersection between the scripting API and the group of candidates invoked from the users scripts (salmon query at the bottom left in the figure below). This gave us 26 methods that are called from the scripts, out of 45 in the Sindarin scripting API:

These 26 mehtods, result of the last query performed, are listed in the top right pane of the Query browser.

"API called in scripts"

Sindarin methods that are not used in the scripts

Section titled “Sindarin methods that are not used in the scripts”

It was important for the debugging team to know the other 19 scripting API methods that are not called from the scripts. We obtained them by querying the difference between Sindarin methods and the group of candidates (purple query at the bottom left in the figure below):

"API not called in scripts"

Again the 19 resulting methods are listed in the “Result” pane.

The debugging team now knows which methods from their scripting API have been useful for users and, most importantly, which methods have not.

Debugging kernel methods called from the scripts

Section titled “Debugging kernel methods called from the scripts”

We also compared the group of candidates invoked from the users scripts with the methods of the debugging kernel and obtained 15 methods. These methods are used directly in the users scripts without using Sindarin:

"Kernel in scripts"

Identifying the scripts that use the debugging kernel:

Section titled “Identifying the scripts that use the debugging kernel:”

Let’s get back to the fact that outgoing invocations gave us a group of candidates methods. When several methods share the same signature, we cannot know for certain which method is called.

For example: Sindarin defines #arguments, which is also defined in Context.

Sindarin defines:

arguments
^ self context arguments

that should be used as follows:

sindarin := SindarinDebugger new.
"some steps"
(sindarin arguments at: 1) doSomething.

We wanted to detect cases where the users did not use Sindarin, like in the following:

sindarin := SindarinDebugger new.
"some steps"
(sindarin context arguments at: 1) doSomething.

This is a simple example or wrong practice. In more complex cases, interacting directly with the kernel would force the user to write verbose scripts and increase the risk of bugs. Why not benefit from a tested API when you can ?

To detect these cases, we used a convention of the users scripts: each time a call to Sindarin API is made, the receiver is a variable named sindarin. This means that invocations which receiver is not sindarin are probably cases where the user bypassed Sindarin API, calling the debugging kernel directly.

We inspected the 15 debugging kernel methods that may be used directly in the scripts and selected the ones that are sent from the scripts to a receiver other than sindarin. This was done in Pharo (not the Query browser), using MooseQuery:

self select: [ :method |
method incomingInvocations anySatisfy: [ :invocation |
(invocation sender parentType name beginsWith: 'SindarinScript')
and: [ invocation receiver isNil
or: [ invocation receiver name ~= 'sindarin' ] ] ] ]

We narrowed it down to 11 methods of the debugging kernel that are called from the scripts:

"Kernel called in scripts"

We then collected the scripts where these methods are called:

self flatCollectAsSet: [ :method |
method incomingInvocations
select: [ :invocation |
(invocation sender parentType name beginsWith: 'SindarinScript')
and: [ invocation receiver isNil
or: [ invocation receiver name ~= 'sindarin' ] ] ]
thenCollect: #sender ]

We obtain 15 user script methods that need to be investigated manually. Moose cannot determine which object will receive a message when the script is executed. The debugging team will analyze in the scripts the statements where the probable kernel methods are invoked. They will run the debugging scripts to identify at runtime which objects are the receivers.

Exploring these 15 scripts will help the debugging team to understand whether:

  • Users did not use Sindarin correctly: they interacted directly with the debugging Kernel, even though Sindarin methods are available,
  • Sindarin API is incomplete: users had no choice but to interact with the debugging kernel because Sindarin does not provide the necessary methods,
  • There are other reasons why the API is not used correctly?

"Scripts calling kernel"

With this experiment, we were able to provide valuable information to the debugging team. In return, we got suggestions of improvements to be implemented in modular Moose tools.

To know how many times a method is called, we need the number of invocations of this method. That is querying the size of a query result. The query browser does not allow it yet, and it should.

To get the 15 users scripts that need to be explored, we had to inspect a group of methods and run a selection manually (Pharo code). This should be possible directly in the queries browser: users should be able to build a query that executes a query script.

This experiment will help Pharo debugging team to improve Sindarin. They will be able to make it more complete and documented, so users continue to develop debugging scripts.

In the meantime, we will improve modular Moose to widen the queries possibilities. Then, we will challenge these new features: we will use them to analyze the upgraded Sindarin API to document its evolution.