The arrival to my new job in Sevilla has coincided with the preparations and launch of a new product. We at Icinetic, are releasing Radarc 3.0. Radarc is a very easy to use code generator highly integrated with Visual Studio and targeting .NET technologies.

Radarc has the ability produce multiple architectures using the same base models and keeping in-sync generated artifacts when model element changes. Architectures and DSLs for defining the models are packaged in so called “Formulas”.

Currently, the following formulas are available for download and it is free for non-commercial usage:

• ASP.NET Web Forms + Entity Framework
• ASP.NET MVC 3.0 + Entity Framework
• ASP.NET MVC 3.0 + Entity Framework + Azure Storage & deployment
• Windows Phone 7

Radarc 3.0 is available with three licensing models and its free for non-commercial usage.

Other technologies are available on demand, such as:

• .NET 4.0 Domain Driven Design N-Layered Architecture
• NHibernate & more to come…

The Hello World sample is a nice starting point to show the syntax and capabilities of every new language. This test is also useful for code generators and Domain Specific Languages (DSLs) also as a proof of concept.

Following this honorable tradition, I have created a video showing the capabilities of Essential: the tool I am working on for doing agile Model Driven Development.

In this 10 minutes video you will get a general idea of the DSL the language provides to create:

• metamodels
• models
• templates
• and control transformations

In order to see the details, jump to Vimeo, activate the High Definition mode (HD) and set full screen (sorry embebed version is not good enough).

Essential IDE – Hello World sample from Pedro J. Molina on Vimeo.

More info about it and 12 usage scenarios in the last Code Generation 2010 presentation about Tailored Code Generators.

Share your impressions!

On April 27, my colleague Nicolas Cornaglia and I will be presenting a talk with some live demos in Valencia, Spain representing our company Capgemini, in the scope of the event Semana Informatica 2010.

The title of the talk will be: Productivity through frameworks and MDD.

The session will be delivered in Spanish. Full agenda (PDF version) and session details.

Abstract:

Business applications for Enterprise Software usually follows a fixed set of standards (global or in-house) to help in keeping the maintenance cost as lower as possible (reducing TCO). In this context, homogeneity and regulation compliance is frequently a must.

The main issue in our presentation will be to show how an approach based in a good framework, modeling tools and code generation techniques can be the right tools to achieve a high degree of standarization, quality, productivity and flexibility to evolve the Enterprise Architecture. Such flexibility is key to provide a better Time to Market when a business process change or a technical requirement suddenly emerges.

When creating a DSL (Domain Specific Language) one of the most important choices is to decide about what items in your domain are going to be considered variable, changeable and which ones are going to be considered fixed, carved in stone.

The former need to be specified in your DSL, in your design or may be coded. The latter are considered immutable and will remain static for all your derived applications for ages.

Considering that everything is static it is obviously useless. On the contrary, considering every aspect variable drives to another no-end getting nothing tangible again as a result. Therefore, in the middle we will have to search for the virtue.

The main issue here is to study a domain and ponder between variable parts and fixed parts. It is not a trivial thing to do from the very beginning. Experience in DSL construction and specially, experience in the domain helps to train your smell, but there are not clear rules for it, nevertheless.

It is not only about knowing your requirements. It is about trying to predict how your requirements will change across time and what types of requirements have more likelihood and tendency to change.

Adding variability

A variable part could be, for example, the background color of your application. If so, you need to add syntax and semantics to your DSL to capture such property. Let’s say you can express somewhere in your specification:  { background-color = peach; }

We can select the peach color for app1, and may be ivory for app2.

However, nothing is for free and this freedom comes with the followings possible drawbacks:

• You need to increase the size of your language (DSL), editors, model checkers, compilers and code generation or interpreters.
• Users have to provide a value for such property unless you have also provided a sensible default value in case of missing information.
• Homogeneity across applications vanishes with respect to background-color. Now it’s a user choice (the one in control of the modeling tool).
• Specs are more complex.

Adding commonality

On the other hand, if you consider the background of your application should be always the same because you are following, for example, a user interface style guide then, the background color is a fixed issue. Its value is provided by design by a style guide, by an architect, or design choice and the user modeling has no control over it.

In this scenario, the DSL is smaller. No need to specify the background color, it is implicit, it is no included in the model/specification.

With this kind of choice, we are betting for standardization. A shared library, a runtime framework or an interpreter will take care of supplying the right color in the right moment.

• Users can not change the background color, specs are smaller.
• Standardization is improved across applications.
• User has no control on the feature.

But, what is the right choice?

It depends. There is no right choice with the information given till the moment. To answer the question we need to consider if the background color is a fundamental feature in our domain and it is needed to be different from application to application or may be, on the contrary, the color should be used in an homogeneous way following a predefined style guide.

Again, the domain imposes the rules to follow. Studding the domain and its variability is crucial to create a consistent DSLs focused in gathering the key features of the domain in a model: the important and variable ones. The important and fixed ones must be also identified but they shouldn’t be included into the model, but into the framework or the runtime.

Standards, policy assurance, compliance

Everything related to standard procedures, compliance and in-house stile guidelines are first-class candidates for standardization. If done in that way, your developers will not have to remember all that weird standard and compliance rules when developing a specific artifact.

A code generator will provide the right value for them. It will do it silently, without errors neither oversights. All the boring code dedicated to plumbing applications like: naming guidelines, service publication, serialization, persistence, adapters, proxies, skeletons, stubs, DAO code are driven by strict standards and best practices and are natural candidates for strong automation by code generators.

Moreover, if the regulation or the standard changes, the change will have impact in the following assets:

• a single change to a framework will be enough
• or a change to a code generator and then forcing a regeneration process and redeploy.

In both cases, it is cheaper that manually reviewing a set of in-production applications.

For example, think about replacing your data-layer access code from a DAO pattern and SQL to an ORM based approach like Hibernate.

Business Know-How

The core of the business Know-How is the important and the variable parts we are interested in to be collected in a specification. Such features need to be modeled, and if possible, abstracted from the technology that will implement it.

If we do it in this way, the model can survive the current technology.

Why we could be interested in do it in such a way?

Just because technology evolves like fashion. Today everyone likes red T-shirts, tomorrow blue jeans will be sublime! Basic, Cobol, C, Java, C#, Ruby… what is the next language to use in 5 years time?

Use your best bet, whatever platform better fulfills your requirements, but I it could be nice to see the business process surviving the technology.   We don’t know in which direction, but technology will evolve, and will change for sure.

Maintaining a language or a DSL

When a DSL or a language needs a review you will be probably considering adding new features to the language.

Each new feature will increase the variability and increase the complexity of the language. Before deciding to add a new modeling feature make a cost/benefits analysis and double check that the valued added by the improvement is greater than the cost of implementing it.

I like to follow the golden rule proposed by Gordon S. Novak about automatic programming:

“Automatic Programming is defined as the synthesis of a program from a specification. If automatic programming is to be useful, the specification must be smaller and easier to write than the program would be if written in a conventional programming language.”

Conclusion

Whenever is possible:

• Business Know-How should be captured by models, specs, DSLs.
• Technical Know-How should be captured by code generators, model interpreters, best practices and patterns.

So, at the end of the day I like the following pair of quotes to sum up about what to include in a model:

• The Spanish writer Baltasar Gracián in the XVII century said “Lo bueno si breve, dos veces bueno.” (a literal translation from Spanish could be: “Good things if brief, twice good.”)
• On the other side, Albert Einstein (XX century) counterpoints “Things should be as simple as possible, but not simpler.”

The Programme for Code Generation 2010 has been published.

This year Mark has invited me to give an introductory session to Model Driven Software Development (MDSD) oriented to begginers.

Also, I will discuss in a second session about creating tailored code generators.

See you in Cambridge in June!

I want to share with you a set of videos to show what I understand when talking about Forward Engineering applied to MDD.

First of all, a legal disclaimer: my apologies for the quality of the videos and for my rusty English: I am starting to play with video editing tools and recording software so I expect to improve my recording and editing skills on the way. Anyway, I found (I hope) they have enough quality to explain the main ideas. So seeing it is totally up to you! You have been warned! 

Also note: see the videos in High Quality mode (HQ) in Youtube. Otherwise, details of the samples probably will not be visible.

In a previous post I introduced a sample of the code that can be generated from a very basic conceptual model. I have created three videos to show you the main steps involved.

1. Modeling (Video 1/3). The first video uses a minimalistic class model created inside Visual Studio 2005 with Microsoft DSL Tools. The sample creates a basic blog structure in less than 5 minutes. Note that in the specification there are no technological choices (neither the types are bind to a concrete language representation).
2. Code Generation (Video 2/3). A quick step: Selecting a code generation (selecting a target architecture), fixing the design choices offered by the code generation and pressing the red button: Generate! A full .NET Solution is generated in less than 5 seconds ready to compile.
3. A quick code review of the generated code (Video 3/3). Finally, I am sure you have curiosity to take a look to the output code, don’t you? This third video shows a walkthrough to show:
   • DB Scripts (table creation, foreign keys, drop scripts)
   • Database creation
   • Logic layer: POCOs (Plain Old CLR objects), NHibernate mappings and a Business Service Layer with fully functional CRUD operations.

So this is it. It is a proof of concept of how fast and direct MDD tools can be starting from a minimalistic model.

When talking about using or buying modeling & code generation products my advice is:

• Don’t use models just for documentation. They will be outdated soon or later. On the contrary, a living (generating) model is always in sync with its target application.
• Don’t resign yourself to just using code generation of skeletons. As you just have seen the current technology allows you to generate much more.
• Don’t be content if anyone try to sell you a model too close or tied in any way to a given target language. Today we have just generated C#, but tomorrow may be we prefer Ruby? Python?
• Don’t resign yourself to use a tool married with a specific database. You know, technology changes faster that we usually expect.

In the economic crisis scenario, some voices argues about the death of SOA (more, more and more) just even before few organizations has already started to adopt it.

However, the SOA approach has valuable principles for Enterprise Software organization for the long term maintenance, integration and minimizing the TCO.

Johan den Haan has a nice post about it: SOA is dead; long live Model-Driven SOA.

As a supporter of MDD, I agree with Cesar and Johan, MD SOA could be a way to deliver the good principles behing SOA.

SOA is always a tough topic to explain for first time visitors. To help to introduce it I will link a superb presentation about it: Meet mike… (credits for Eduard Hildebrandt).

MDD with Oslo

On the other hand, as Cesar pointed, Microsoft Oslo project has/had (after the rebranding) potential to be a good platform for domain modelling with textual DSL (MGrammar) and visual DSL (Quadrant).

As commented in the previous posts, it’s a pity that for the moment modeling efforts at Microsoft has been focused too much into one particular tree “the SQL Server scope” and loose the full forest vision.

Since 1998 I’ve been testing almost lots of them and forming an opinion about it at the same time the technology evolved. Been at CG2009 and attending the session of Kathleen Dollard on Template Specialization brought to my mind all the times I wasn’t satisfied with my current template engine and changed to try for a new one.

Doing commercial code generators, I’ve been tested lost of techniques: direct string concatenation, XSLT, direct code generation (or what Kathleen calls brute force code gen), ASP, JSP based approaches, developing custom template engines (two of them), taking a look to Code Smith, T4 and others, using NVelocity and finally arriving till StringTemplate (for the moment).

The essence of a Code Generation

Code generation is most of the time producing output source code as text files.
In the text-based generation process, the model is used as metadata by a the control (a transformation program or control rules) to fulfill a text template (basically a text form with named holes).

Models, transformation algorithms and templates are core assets to be reused and recombined in different ways across the MDD discipline. So, selecting a good technology to mix all them’up in the proper way is essential for success.

When writing templates for code generation (M2T) or (M2M model to model) you need to have at least three languages in mind at the same time: the input language (or model), the transformation language by itself and the output language.

Usually in and out models/langs are complex enough so it would be interesting to have the easiest of use transformation tools we found and at the same time powerful enough to make the job.

Pros and cons for each template engine approach can only be stated respect to some requirements. Different requirements imply different selection criteria. So let’s start expliciting my requirements for a text template engine.

My requirements

1. Direct recognition of the final output. Easy to read and understand. A template represent a piece of output of the target language.
2. Easy to maintain. Templates are created to be changed. And when needed, the change should be done in a quick and agile way.
3. Reusable. Templates should be reusable in different contexts.
4. Minimalistic. Due to requirement (1) & (2) a template language must resemble the final output with the minimal necessary distracting extra lexicon not belonging to the output language.
5. Composable. A template must for sure be callable from other templates.
6. Separation of Concerns. Separate the control (the transformation) from the template. MVC Pattern.
7. Idempotent. A template execution must be repeatable whenever is needed. Reapplying the template should generate always the same output, no matter how many times you call it. Therefore, it should to contain lateral effects: not more than the line timestamp of when the generation process took place if needed.

Now, we can review the main pros & cons of the technologies involved respect to these requirements:

Direct string concatenation (aka Brute Force approach)

This is one of the first techniques we all tried for first time. It’s easy and direct. You start choosing your favorite language like C# and start doing things like this one:

Output code:

        select Id, Name, Surname
        from Customer
        where Country LIKE @vCountry

Generator:

        string sql;
        string sqlRes;
        sql += " select Id, Name, Surname";
        sql += " from " + tableName;
        sql += " where Country LIKE @vCountry";
        sqlRes = String.Format(sql);

If you are a bit more sophisticated (or geek enough!!) and concerned about the performance, you will change string concatenation for a buffer based approach like this one:

        System.Text.StringBuilder sql = new System.Text.StringBuilder();
        sql.Append(" select Id, Name, Surname");
        sql.AppendFormat(" from {0}", tableName);
        sql.Append(" where Country LIKE @vCountry");
        string sqlRes = String.Format(sql.ToString());

But at the end of the day you will still not fulfilling the majority of the requirements. It’s not easy to maintain such template: mainly because we are mixing the transformation language (C#) with the output one (SQL in this case).

As an extreme case of this to illustrate how much the thing can get worse, I remember myself concatenating strings in 1999 inside C++ to generate ColdFusion code, this CF output code was, at the same time, generating Javascript on the fly, and this one, finally generating some kind of Dynamic HTML for a web application. It’s weird, isn’t it?

Character escaping like \” for adding a quote in the right place or thinking twice if a semicolon is needed or not was a totally nightmare in addition of trying to have in mind the syntax of 4 languages at the same time.

XSLT

Xml Stylesheets Transformations is a W3C standard proposed for translating XML to different text outputs like XML again, HTML, text, code, etc.

The idea originally was quite good, XML needs to be transformed quite frequently to adapt to special needs. And making a XML transformation based on XML languages makes sense because you were using apparently the same XML base language.

However, if you ever used XSLT you will find that is poorly user-friendly, it has a very bad legibility and, therefore, it ends in an unmaintainability hell. A week later than you coded the template, you will start to have problems in remember what these templates were trying to solve.

A sample:

    
        
            select Id, Name, Surname
            from 
            where Country LIKE @vCountry
        
    

So, if you are in a proffessional context, you need easy maintainability and XSLT shouldn’t be your first choice (nor also the second unless you can generate the XSLT itseft).

It is also a paradox to see how XML was designed to be user readable, however XSLT it isn’t once you start to create non trivial samples.

XSLT was not bad for doing simple formatting on the XML. For example, generating HTML from XML with XSLT is ok as far as you don’t try to change the structure of the input tree too much (the output must be following the same tree structure or be a subtree of it).

Any way, mixing the XML syntax of XSLT and the output like HTML makes the whole much more cryptic.

CodeDom

CodeDom is a powerful (but not nice) API allowing anyone inside a .NET program to generate .NET code in any .NET language. The price to pay is that CodeDom flights at a very, very low level altitude respect to compilers and the underground MSIL instructions.

Code is quite complex even for trivial constructions. On the other hands provides language independence, allowing you to generate C#, VB.NET or any other .NET language as soon a CodeDom serializer is implemented.

Visual Studio uses CodeDom intensively to do things such as generating code for controls whenever you drag & drop a control on the designer surfaces. And it does it in the language you are working with.

If you don’t need to generate code in such context, avoid the usage of CodeDom.

A hardcore version of CodeDom is to use the Emit() functions to create tangible MSIL (like bytecode in Java) on the fly. But this is more a hacking technique or a final compiler than a real need for our usual needs.

ASP, JSP syntax-like (Code Smith and others)

ASP based languages mixes XML tags with code inserted in between.

Having the opportunity to add code inside the template, gives you the “bad” chance to mix presentation (the template) and the control (the code you apply to drive the transformation).

It’s interesting to see how the Web developer of the Java community moved apart from this kind of practices and started to use the MVC pattern (Model View Controller) providing an strict separation as they did with the Struts framework. Ruby guys did the same with Ruby on Rails. And finally ASP.NET is doing the same with the MVC framework for .NET.

So, if web developers notice that having templates and control code mixed at the same file it not desirable for maintainability and reuse, why us, the code generators designer and developers, are not seeing it in general?

NVelocity

NVelocity is the .NET port of the Apache Velocity Project. I used it for two years because it has a good features for splitting the templates in manageable chunks and call to subtemplates whenever it is needed. The control was starting to be limited to keep the control as simple as possible: I liked the approach and used it extensibly, but still having too much open control.

Xpand

On the Java side of the world, Eclipse guys have another approach called Xpand.

OpenArchitectuWare and other projects use it extensively and has well supported on Eclipse with color syntax and code completion features.

To be honest I don’t tested it yet, but I promise to do it soon. Therefore, I will not comment more on it before trying it.

T4

With the apparition of Microsoft DSL Tools, the T4 template engine came into scene.
T4 generally runs inside Visual Studio (can be executed from the command line also) and each *.tt file can be generating one dependent code file.

The first thing I don’t like is the one to one mapping. Usually I need to apply my code generation in a one to many files operation to process collections of them. It’s true that there are workarounds allowing you to generate multiples files from a T4 template. But it is more a hack that a built-in feature of the engine.

As a derivate of the ASP syntax, control and template are mixed. Templates are compiled before executed. This should be great for the responsiveness of code generation time. However, generation time usually takes longer inside Visual Studio that applying other technologies based in interpretation: I do not know why, may be the bottleneck is in an external dependency in the generation process, such us loading the input models.

On the pros side, the templates have intelisense support and syntax coloring inside Visual Studio to make it easy the editing experience.

Of course T4 was designed to be a convenient way of extension for adding code generation to a Visual Studio project. However, it also encompasses some design choices not necessarily optimal for other usages. This is mainly my case: I’am not saying T4 is a useless. Just it is simply is not well suited for my needs as stated in my previous requirements.

StringTemplate

Finally, I’ve been using the great Terence Parr’s ANTLR parser for a long time. And it was only a question of time to try his companion template engine: String Template.

I like Terence Parr’s StringTemplate approach because this enforces a strict Separation of Concerns when doing code generation: separating the template (a text form with holes and nothing else), from the model, and from the transformation itself (control code to apply the template).

For years I’ve been looking for something so well aligned with my ideas, so finally I felt that I found it.

Terence has a nice article to read to understand the point of view: Enforcing Strict Model View separation in Template Engines.

A StringTemplate sample:

group sql-gen;

genSelect(tbl)::=<<
select $tbl.Fields:genColumn(); sepatator=", "$
from table $tbl.Name$
>>

genColumn()::=<<
[$it.Name$]
>>

In StringTemplate each template can receive parameters. A template can call a subtemplate, and express nice things to take care of the formatting like separators or automatic indentation. Iterations are expressed in the following way $collectionExpression:templateToApply()$. But nothing more: there is not more content for control. $it$ stands for the model item been iterated (implicit this reference if you want).

The template is reusable. It will work whenever you pass an object containing a property called Name and having a collection of objects convertible to string called Fields. These are all the requisites the template is imposing to the input model.

During the CG2009 conference and also during CG2008 we always discuss with our Microsoft colleagues why we are using DSL Tools but skipping the usage of T4 templates? I promised myself to write a blog-post like this one to try to explain my reasons.

To sum up

From an MDD point of view, text engine templates constitute a well defined domain. Having a precise DSL language to deal with the domain makes perfect sense in this context. Therefore, the language must be designed to express what is expected to do in the best way without introducing any further unneeded fanfare.

And you? What do you think about it? What is your experience in the usage of template engines?

The central idea of my speech was devoted to achieve scalability in MDD tools & methods for industrial application of the MDD technology.

I was so glad to see that Markus Völter and Steven Kelly also just 30 minutes before my talk, were addressing the same key issues in their inspiring keynote: scalability, partitioning, modularization of DSL. This fact reinforce me in the idea that we are really growing as a community (CG) with common problems and aligned in the solutions improving the tools & techniques more and more every year.

Scaling MDD for production: enabling SoC at model time

There was also time for an small but complete live demo of a small tool socDriven running and demonstrating the concepts explained. The tool was implementing a compiler and merger for a small textual DSL. The DSL was parsed using the OSLO M_Grammar to create the AST. A merged engine in a second phase glued the partial specifications producing as output:

• the same specs merged in a unique file,
• the same spec ready to be open with a graphical editor with DSL Tools,
• a full code generation solution on VS2008 containing:
  • scripts for SQL Server
  • a logic layers with POCOs & full NHibernate mappings
  • a default CRUD service definition,
  • a default Web Service definition,
  • and a Windows Form UI ready to query and execute CRUD operations.

After the generation of the solution, it can be opened with VS, create the DB launching the scripts provided and pressing just F5 for building and running it all! == (a.k.a the utopia of 0 custom code) };-)

Please, feel free to add your comments!