Dissecting an AppNow Specimen

Artisan Spanish Knife
AppNow is a minimalist service that converts simple models in a cloud deployed back-end.
The simplicity of this approach encourages developers to focus on what to build (business needs) instead of how (the technical skills to build it) and where to deploy it (devOps).

In this article, we are going to focus and delve into the technical choices made for the backend: architecture, languages & tools, organization and code practices used.

In particular, this can be a useful read for a developer wanting to introduce her/himself into MEAN development. On the other hand, the text is full of low-level technical details and is not suitable for those unfamiliar with programming.
You have been been warned.

Good! If you have read this far and still want to read ahead despite the disclaimer above, then you deserve what you are looking for! Let’s go for it.

Sample Model, Code & Application

To serve as guiding sample throughout this article, an exemplar of an App generated by AppNow has been derived from this reference model:

class User
{
	string FirstName;
	string LastName;
	string? Address;
	string City;
	string? County;
	string State;
	string Zip;
	string Phone;
	int Age;
}
class Country
{
	string Name;
	string Code;
	string? Region;
}
class OlympicMedal
{
	string Athlete;
	int Age;
	string Country;
	int Year;
	string Sport;
	int Gold;
	int Silver;
	int Bronze;
	int Total;
}
class Weather
{
	string Location;
	int Year;
	int Month;
	int Day;
	time Time;
	decimal? Temperature;
	int? RelHum;
	decimal? Presure;
	int? Hmdx;
}
class Exoplanet
{
	string Name;
	decimal? Msini;
	decimal? SemiMajorAxis;
	decimal? OrbitalPeriod;
	decimal? OrbitalExcentricity;
	decimal? Om;
	decimal? Velocity;
	string? OrbitRef;
	string? FirstRef;
}

The model has been imported into AppNow from a sample spreadsheet containing the following data sets with sample data:

  • a simulated set of users and addresses,
  • a list of countries,
  • Olympic medals data by year, sport and athlete name,
  • sample weather data, and
  • a list of exoplanets.

The model has been then converted into a running application via AppNow. The app can be explored live here: (use the password sampleApp to log in and test it).

Full source code is also in this github repo: Feel free to clone it, add issues, or suggest improvements.

Technology Selection

Javascript communities have been growing fast in the latest months. There are a good set of tools and libraries mature enough to develop with JS both on the server and the client side. Therefore, not surprisingly, AppNow has choosen as its first targets NodeJS for the backend and AngularJS for the client side.

On the persistence layer, MongoDB has been chosen because it provides good scalability, easy installation and maintenance compared with heavier RDBMSs which require more complex installation, configuration and sometimes even, dedicated hardware.

All these choices keep us inside what is commonly called MEAN architectures: MongoDB, Express (a framework for services in NodeJS), AngularJS, and NodeJS.
Architecture
The MEAN architecture is decoupled naturally in three layers: Persistence, Backend and Frontend. Following this layer separation, we have in place for each role: the MongoDB database, the backend, or server side, and the frontend running on the user’s browser.

Let’s review each of them in more detail.

Persistence

As anticipated, the persistence layer is implemented by MongoDB. MongoDB is a light NoSLQ, document database very well suited for storing data in JSON format. In fact, all the communication and APIs exposed to the programmers follow the JSON conventions and look & feel, and internal representation used is BSON (Binary JSON).

When needed, MongoDB offers options for scalability to increase the number of servers and data organization (like sharding).

The support provided by cloud provider for MongoDB is very good. For local environments, as MongoDB is open source, it is very easy to deploy both for Linux, Mac or Windows machines.

The associated data schema for the database is injected by the application as needed: usually on the first call which includes indexing and fine tuning. This is a very frequent approach for provisioning new instances of NoSQL databases.

Backend

On our backend’s domains, NodeJS is without a doubt, the king. NodeJS has quickly and rightfully become known for been fast, light, portable and having an effervescent community of developers creating open-source libraries for almost everything you could ever need or imagine.
Such libraries are organized in packages and distributed to developers using the npm tool (Node Package Manager). Take a look to the index of packages created by the community at npmjs.org

NodeJS provides a good set of build-in modules (or packages) but the extensibility added by third-parties is overwhelming.
In particular AppNow backends employs them. The most representatives are: Express, Mongoose, & Baucis.

Express
Express is the leading library to work with HTTP connections and expose services from NodeJS. It provides a very good middleware to plug-in interceptors to include authentication, authorization, error handling, content-negotiation, format-encoding and more.

Mongoose
Mongoose is an ORM for MongoDB. Yes, it is! I will say it again: Mongoose is an ORM for MongoDB.

There are many people, with a traditional RDBMs database background, moving recently to NoSQL databases and trying to escape of the complexities of using ORMs. This fact is very shocking beacuse they don not expect to see an ORM at the interface for a NoSQL database.

Although it is totally possible to use MongoDB directly, and there are some usage scenarios where an ORM can be a costly overhead for the app performance, the fact is that, in 95% of the cases, the ORM simplifies a lot data access code and saves you writing tons of error-prone boiler-plate code.

Moreover, having attended MongoDB conferences, the experts recommend that when worried about performance, you need to carefully plan and design your MongoDB schema. Having a no-schema on MongoDB or mixed document types in the same collection is definitely not good practice and something to avoid for performance.

Therefore, Mongoose takes our model data as input data and creates the schema for MongoDB as needed. All persistence operations like create, update, delete and many rich forms of data query are handled by Mongoose.

Baucis
Baucis is another hidden gem in the npm registry. Created by @wprl, Baucis is a REST/JSON framework over NodeJS using Mongoose for persistence.

Giving a set of Mongoose models, Baucis creates on the fly REST/JSON services for if and expose them using Express. And more, a convenient Swagger meta-data for service discovery is also added. What more can we ask for? Extensibility? Yes: it was designed with this requirement in mind also.

I am serious, if you are a backend NodeJS developer, take a look to Baucis. It can save you a lot of time for many frequent backend tasks.

Once the backend has been reviewed, it is time to jump to the browser side and review the frontend.

Lightweight Frontend

The code provided is intended to be used as a backend. Strictly speaking, To accomplish this goal, no furthers layers are needed. Not even, a User Interface. However, AppNow provides an administrative User Interface to help administrators to operate and configure the backend.

AngularJS was the selected tool for this mission. AngularJS is a superb framework for modern JS frontend development. Many ideas like the good databinding from WPF and Silverlight has influenced the declarative syntax of AngularJS.

The UI Bootstrap library was also used here to embrace full HTML5 and CSS3 and at the same time to provide an adaptable and responsive design in desktops, tablets and smartphones, as well as been able to gradually adapt to different form-factors and screen resolutions.

Angular code follows the MV* pattern and organizes its code in modules and inside these ones, views, controllers, models mediating between them (with plain JS data-structures models), services, directives and filters.

It is out of the scope of this article to teach AngularJS, but code in the repo contains many common problems solved and provides a rational organization for the client code.

Organization

The project contains two code layers as previously commented: frontend and backend.
The backend is locatet under the /app folder with the server.js as the entrypoint.

The frontend, on the other hand, is located under the /public folder. All the content behind this folder is served by the webserver to the outside.

Delving into the next level, the folders are organized as follows:

  • /public/app The code for the AngularJS application
  • /public/app/controllers Code for controllers
  • /public/app/directives Directives
  • /public/app/dist 3rd party libraries to include
  • /public/app/filters Filters
  • /public/app/services Application services
  • /public/css Cascade Style-sheets employed
  • /public/images Graphic resources
  • /public/views HTML views
  • index.html Front-end entry-point/li>

A /test folder finally is placed to contain mocha test (still to grow in numbers).

Prerequirements

To run the code locally, you will only need to install two dependences:

Both of them are very portable, and very well supported. There is a version available for Linux, Unix, Mac and Windows.

Installation and launch

Use:

  • npm install to ensure all npm dependences are downloaded and installed.
  • npm start to launch the application listening at port 5000.

Conventions employed

Using the MEAN set of tools, Javascript community conventions are the path to follow. NodeJS, Express and AngularJS expose their own idioms and conventions and we also try to stick to on the generated code for each software artifact.

Other features

Having an application ready to run on the web requires taking into account many little details. Some of them can be solved directly by sensible default values coming from savvy frameworks. In other cases, some configuration or direct action is required. In this section we will cover some of the most significant of them.

Basic Auth

Basic Auth is implemented to check that the correct API Key is provided by the client on authentication. The lines 293 to 318 in /app/sever.js show the three functions implementing it. It is a middleware hook into the Express framework to double check for credentials on every request under /api routes. The rest is considered as public resources.

//Auth ----------
app.all('*', function(req, res, next) {
	if (req.url.substr(0,5) != '/api/'){
		return next();
	}
	return isLoggedIn(req, res, next);
});

function isLoggedIn(req, res, next) {
	var incomingKey = getHeaderOrParam(req, 'apikey');
	if (incomingKey == auth.security.apiKey){
		return next();
	}
	incomingKey = getHeaderOrParam(req, 'api_key');
	if (incomingKey == auth.security.apiKey){
		return next();
	}	
	res.status(401).send('Unathorized.');
}

function getHeaderOrParam(req, key){
	var cookie = req.cookies[key];
	if (cookie != undefined) return cookie;
	var header = req.headers[key];
	if (header != undefined) return header;
	return req.query[key];
}

CORS

CORS stands for Cross Origin Resource Sharing and it is a protocol to allow third party websites to access API resources in a third-party domain.

Enabling CORS in Express can be done in 4 lines with the following middleware (lines 282-289 of /app/sever.js):

//CORS enabled for allowing 3rd party web-apps to consume Swagger metadata and backend. 
//Disable it commenting this block if you don not need it. ----------
app.all('*', function(req, res, next) {
	res.header("Access-Control-Allow-Origin", "*");  //Change * to your host domain
	res.header("Access-Control-Allow-Headers", "X-Requested-With, Content-Type");
	res.header("Access-Control-Allow-Methods", "OPTIONS,GET,POST,PUT,DELETE");
    next();
});

UI Authentication

The administrative User Interface needs a server side endpoint for dealing with login and logout operations. It checks for the validity of the credentials and creates a cookie for the browser session.
See lines (210-225 at /app/server.js)

//Simple Portal ApiKey based auth ------
app.post('/weblogin', function(req, res) {
	if (req.body.apikey == auth.security.apiKey) {
		res.cookie('apikey', req.body.apikey);
		res.status(200).send('{"id": 0, "user": {"id":0, "role": "admin"} }');
	}
	else {
		res.cookie('apikey', null);
		res.status(401).send('Unauthorized.');
	}
});

app.post('/webLogout', function(req, res) {
  res.clearCookie('apikey')
  res.status(200).send('OK');
});

Exposing resources

Finally, exposing resources to the webserver is done with Express again.
The following lines (320-324 /app/server.js:

app.use('/api', baucis());
app.use('/', express.static(__dirname + '/../public'));

var port = Number(process.env.PORT || 5000);
app.listen(port);

do the following:

  • publish the REST API under the /api route
  • publish the front-end files under the root route /
  • select the configured port for starting the application (using 5000 as default)
  • and launch the web-server to listen for connection in such port.

Import & Export Data

Import and export from CSV and XLSX are also built-in features tht allow us to import and export data. The library xlsx was used here to help with XLSX parsing.

Search & Pagination

On one side, Baucis exposes a rich set of query, sort, pagination options with direct access to the MongoDB query language. On the other side, AngularJS allows to plugin search and pagination with little effort providing controls like pagination and pager on UI Bootstrap. So, adding it requires very little code on the front-end and no code at all on the backend.

So far, so good. A lot of features were described here and many more are still to come.

Future work & improvements

Some improvements to the generated code already on our backlog include:

  • Adding bower on the client-side (it is like npm, but or the client-side library and dependencies management).
  • Adding unit-tests: many more on the server side and client side to serve as regression test.
  • Adding grunt task to compact and minimize the frontend javascript on production sites.

As discussed previously, more improvements can be added to other levels not considered here: the domain model, target language & frameworks and target deployment.

Your feedback is welcomed

This is the current state of the output at the time of writing this article: September 2014. It will evolve for sure, as living software to accommodate business (first) & tech (later) changes.

If you want to change, improve or suggest any change to any part, feel free do it. We would love to hear what you have to say. Some choices for that:

Thanks for reading till the end. I hope you found it useful.

Post a comment.