paul vidal - pragmatic big data nerd

Tag Archives

2 Articles

Data As A Microservice: the future of data architecture

by paul 0 Comments
Data As A Microservice: the future of data architecture

Let me preface this article with an understatement: sometimes, enterprise architecture can be complicated. Large companies run thousands of applications, multiplied by dozens of environments replicated for testing, user testing, sandboxing, accumulated over years of acquisitions, re-architecturing (yes, it is a word I made up), and experiments all with the purpose of driving business forward. Like any complex system, human beings have been trying to make sense out of it by conceptualizing models and architectures aimed at simplifying the system thus making it more efficient, robust, scalable, secure, and spiritually vertuous (OK, maybe not the last part, although can a piece of software be inherently virtuous? A question for another day). With all this in mind, I would like to take some time to reflect on one of these concepts: Micro-Services and how this concept can apply in the realm of data management.

Microservices VS Enterprise Service Bus

First introduced in 2011 during workshop of software architects held near Venice in May 2011, Microservice Architecture is defined by James Lewis as follows:

The term “Microservice Architecture” has sprung up over the last few years to describe a particular way of designing software applications as suites of independently deployable services. While there is no precise definition of this architectural style, there are certain common characteristics around organization around business capability, automated deployment, intelligence in the endpoints, and decentralized control of languages and data.

Microservice architecture is a subset of a Service Oriented Architecture (SOA), aiming at distributing microcomponents to deploy applications as opposed to a centralized application integraiton layer, often called Enterprise Integration Layer (EAI) or Enterprise Service Bus (ESB). Leaving aside the obvious angry developer argument stating that all of this is marketing jargon and rebranding of the same products, it’s interesting to take note of a fundamental trend I covered before in this blog: enterprise are looking to implement agile enviroment, which extremely granular elements in order to ensure business reactivity. The dream of the all-integrated all-consolidated entreprise layer is fading.

Data As A Microservice

In a very similar manner, the idea of single source of truth containing all the enterprise data is coming to an end. And, unlike some data lakes proponents would like to make you believe, it is not because of the pitfalls of traditional technologies that can’t handle large volume of data or distribute it efficiently. Building a single centralized source of data is an utopia. Instead, companies are now shifting their focus towards platforms enabling rapid agnostic data integration, agile data schema modification, and complete distribution. These platforms can then be used in a microservice architecture, making them Data As A Microservice platforms. I’ll admit, I may have made that term up too because it sounds cool, but it is very important to note for you data vendor, data scientist or data consumer (CIOs and CTOs organization). The future of data microservice-like agility, not monolithic unification.

References:

http://martinfowler.com/articles/microservices.html
http://stackoverflow.com/questions/25501098/difference-between-microservices-architecture-and-soa
https://www.voxxed.com/blog/2015/01/good-microservices-architectures-death-enterprise-service-bus-part-one/
https://en.wikipedia.org/wiki/Microservices

On the simplicity of experts systems

by paul 0 Comments
On the simplicity of experts systems

Today I want to share with you an unfinished thought. Perhaps because I have been working non stop all week, perhaps because people keep complaining about apple correct strategy of fleshing out their devices’ design (remember how everyone complained when they removed the CD player?), or perhaps because I’m not sure I ever will reach a satisfying definite answer on the matter. At any rate, here is the question: what’s the optimal ratio simplicity/features of experts software systems?

The state of the art

If you’ve ever been in contact with someone who has to use a computer system beyond browsing the web or desperately trying to string 1000 words together the night the essay is due, i.e. had to work with any software at all from point of sale retail, to graphic design passing by cloud CRMs, you will notice a common thread: that piece of software is awful. These complaints usually boil down to one of these three categories: the software is either buggy, too complicated to use or missing functionalities. Leaving aside the unstable aspect inherent to the current process of software development, the other categories seem to be two ends of the same spectrum. And this is what keeps me up at night. Probably for 30 seconds, then it’s my kids crying.

Developing software in a customer is king world

Having worked for years for B2B software providers, where sales cycles are long, requirements are complex and clients must be satisfied, I can tell you that most of these pieces of software require training in order to be used to their full potential (which in part drives the sales through maintenance and support costs). In this world, roadmaps always add functionalities, new modules and new ways to customize a product. Ultimately, this renders the sales, delivery and support extremely complex (and why I have a job).

A new era of software delivery

However, in the recent years we have seen a new development with Software As A Service. Companies are limited to the SaaS current version’s functionalities. Because of the effectiveness of the cloud sales model and the centralized maintenance of these pieces of software it is much easier for software publishers to prioritize, consolidate and even drop features according to what clients are actually using. With SaaS, customization is often out of the question. Yet, more and more companies are moving to software as a service and have clear initiatives to simplify their current eco-system.

My dilemma

First, since when did dilemma lose its ‘n’? I digress. In all seriousness, and as I argued before I do believe that simplicity is key to software success. Thus, when do you decide to add functionalities to your software to respond to market demand? How do you ensure that this addition is not only relevant but it also does not throw away your user experience, brand and sales process? Ultimately, I think that finding this balance is really more art than science. And this is always going to bug my extremely rational mind.