paul vidal - pragmatic big data nerd

Tag Archives

3 Articles

The Big Data market in 2017

by paul 0 Comments
The Big Data market in 2017
Buildings always look badass from the ground and with a black and white filter

Accepting reality is not trivial. As we sit in our echo-chambers, particularly exacerbated by our social networks, preference algorithms and suggested searches, our cognitive biases betray our picture of the world around us. Add that to the fact that everyone else than us is an idiot with whom we should not engage in a conversation (or if you prefer, call this mild social anxiety), and pretty soon you can convince yourself of anything. With this in mind, I decided to take the time today to expose my understanding of the reality of the big data market in 2017, for large entreprises. While it is inherently biased, arguably like any piece of writing, I did try to do my research reading a lot of white papers recently (of which you can find some reference below), but mostly, this is my domain of expertise which means I’m confronted to it every day. Of course, this categorization is subject to discussion and constructive criticism that I always welcome.

Out, or very little relevancy

  • Data Lakes: The fascination for unlimited data distribution has passed. Enterprises struggle to find a use to their data lakes and the layers written on top of them to make them useful seem too much effort for little reward.
  • Pure data analytics: The terms Data Analytics or Business Intelligence encapsulate a vast number of concepts that will always be useful one way or another in our data driven world. What is nowadays losing momentum is solutions making analytics the end goal (analyzing trends, population sub group preferences, etc.). BI is a very small portion of what Big Data offers and if the end goal of a solution is to give you trend analytics, it is too reductive.
  • SOA, ESBs, Convergent applications: This is been dead for a while but worth mentioning. The idea of a single convergent enterprise solution to encapsulate all data and functionalities is practically not feasible (too much market change, too much cost, too much complexity, too little agility).

Extremely relevant for 2017

  • Agile data platforms: At the opposite end of ESBs and massive consolidation into one data system is the micro-services architecture. The architecture enables extremely rapid and agile deployment of applications to respond to an ever changing market where end customers have more choices than ever and thus are very hard to retain. The bottleneck of micro-services architectures is often data. Being able to consolidate, cleanse and expose rapidly data to anywhere is a complicated proposition but some platforms can do it. If a platform is able to integrate from multiple sources, consolidate and expose data rapidly, then it enables today’s hottest use cases: digital transformation, agile test data management, micro-services implementation, and more.
  • Cloud enablement: More than ever, and despite previous reticense vis-a-vis security (just like older generations are still reluctant to use their credit card numbers on the web), the movement to cloud applications and platforms. Enabling the cloud, that is not only exposing/migrating data to cloud applications but also ensuring security, compliance and control over the data exposed is therefore a very important market trend.
  • Data Personalization: we live in a world where everyone expects their experience to be catered to them. Having to repeat your identity while being tossed from department to department on a help desk line is one of the most infuriating experience (after the complete loss of human rights and dignity one experiences in an airport). Seriously though, enabling the understanding of the individual is crucial, whether that individual is a person, a product or a machine in IoT use cases.

Not quite there yet

  • Predictive Individual Analytics: We already see some implementations of this in ad personalization or preference settings, but being able to predict what an entity (a person, a machine, a car, a product) will do, what it wants and needs is going to open the door to systems that give answer instead of respond to questions. It requires the problem of data personalization to be solved beforehand though.
  • Smart Data Discovery: Once agile data platforms are in place, the use-case of automated data mining will explode. Too many systems with too little experts on the systems will give birth to solution that’ll enable the enterprise to recover a fair percentage of the relevant data without human intervention.
  • Expert AI systems: Finally, and most exciting of all are expert AI systems. These are software that will replace the way that data is currently fed to our everyday software (CRM, machine monitoring, marketing analytics, etc.). The use cases are still not clear in my head but I know that finding the point where human intervention is the most costly (where it requires pattern recognition), and replacing it with automated AI will be a game changer.

Some references

  • Is the Cloud Secure? (Gartner): link
  • Marketing data management (Ascend2): link
  • Seizing the Digital Advantage in Banking and Financial Services (Cognizant): link
  • The Big data workbook (Informatica): link
  • Agile Test Data Management: The New Must-Have (Forrester): link

3 essential features of the perfect software

3 essential features of the perfect software
Hi Mark. John left the company 3 months ago. Can you help us find a bug in his code? It is somewhere in this file.

I have recently been thinking quite a bit about what make a piece of software successful. Very early in my career, I got to temper my idealistic view of computer science and the world in general, quite well captured by the saying “you don’t have to be the best to be the first”. As I progress in my professional journey, I have been trying to identify key aspects that make a software go head and shoulders above their competitors or see a exponential growth in a niche of the market not exploited at the time. While it is most likely impossible to find the 3 magic words you have to pronounce to make the perfect software appear, I am still going to pretend I did this, for web traffic purposes, because I have no integrity. All bad humor aside, and for the sake of readability, I did try to funnel my thinking into 3 major aspects which, combined, are a successful piece of software. Quick aside: the purpose of this piece is not to dive into what technical aspect of a piece of software is valuable, but rather to present the software features to which the market responds positively. With that out of the way, let me present you the current state of my cogitation: the perfect software is:

screen-shot-2016-10-07-at-8-50-30-am

SIMPLE

Simplicity is essential for the end-user to open their eyes. While the algorithms, architecture and other under-the-hood building blocks can and will most likely be complex, the idea here is to present something that is simple to understand. Simplicity can be driven by multiple factors. It could be for instance the front end of your application. This is why UI/UX is such a sought-after skill, and while it is predominant in the B2C industry it is severely underused in the B2B world. It could also be driven by the product packaging. If building a platform with many potential uses, packaging them into specific solutions recognized by the industry is a fair way to achieve simplicity. It could also be targeted: focusing on solving the problems of one vertical for instance.

NON-INTRUSIVE

This characteristic is epitomized by the success of cloud computing. Software As A Service particularly is the perfect example of non-intrusiveness being a successful business model: end users do not want to have to install and maintain software. It is an obvious cost reduction feat for big enterprises, it is just as much a reality for consumers: no one wants to have to install a software on their computer, and the ones we do install are the ones we hate the most (he who had no complaint about Microsoft Word cast the first stone). Even more interesting, web software like appointment booking, ticket sales and so on are considered websites and not pieces of software, but I digress. That being said, and as I argued as early as last week, SaaS isn’t the only model and non-instrusiveness can be characterized by other traits. Backward compatibly or maintenance of current set of existing skills and application is a great way to ensure a non intrusive model. One of the reason why I think disruption is nonsense, see previous rant.

ACTIONABLE

Being simple and non-intrusive are essential qualities, but your software must actually do something in order to be valuable. The important question here is: what’s in it for your user? What is the value? And I don’t think that a value proposition such as “we are doing it better than the others” is enough, nor is “imagine what you could do with that”. You need to be able to be able to show tangible results right off the bat, drive your customer through a story of what they will be able to do now that they weren’t able to do before. This feature is in my opinion one of the hardest and most often forgotten feature for a software to possess, especially put in relation with the other two. Indeed, innovation while maintaining non-intrusiveness could be seen as an oxymoron. In reality, the perfect unique innovative actionable value that of which no one thought before does not exist. That being said, many tech companies today start by building technical prowesses instead of focusing on creating value. My recommendation is to think about the value first, then focus on making the solution simple and non-intrusive, which is ironically why this article is written in the opposite order.

Conclusion

Can a piece of software truly possess all these qualities fully? Probably not, but at I think it is at least an ideal to strive for. As mentioned awkwardly at the beginning of this article, this is also a very preliminary assessment of my thought process. I do believe that if a piece of software possess a good balance of theses 3 features, it is set for success. More importantly, I think that these features should drive the development of new softwares. I know that I have a few ideas about what to develop, and I’m going to make sure to keep that in mind.

The importance of specialization in software sales

by paul 1 Comment
The importance of specialization in software sales
“Bust of Adam Smith” by Patric Parc, 1845. (Wikipedia)

After spending some time reflecting on whether or not Data Scientist was a useful role within any organization churning a big amount of data, I stumbled upon this post on LinkedIn: There is Only One Type of Software Engineer.

In short, this post calls for a de-specialization of the role of engineers in order to avoid siloed professionals refusing to take responsibility of a task if it does not exactly match their job description.

While I agree with some of this argument, especially in big organizations where unfortunately the lack of ownership of a task and fear of risk taking can be quite flagrant (which I will try to tackle in a future post), I think that small organizations are in serious lack of specialization, the effect of which are particularly visible in the sales process.

Establishing the premise: specialization scarcity versus tangible gains.

Quick disclaimer: as for every post I write, I am not trying to establish and write the ultimate truth but rather I’m engaging into a conversation, so if I’ll be happy to see my premise challenged. It also applies mostly to my domain of expertise: large software sales for big organization.

In this context, here is what I observed and gathered from the market. The era of all-in-one platforms is over. I debated this in my first article speaking about consolidation, but it is still true. And while some giant companies manage and should attack different segment of the market, they still have clear marketing messages and product names for these different segments.

The problem is for companies that have an extremely innovative product, that could tackle a lot of use-cases. I know we have been struggling with this in my company, although it is getting fixed now, but I also observed this statement for many other companies I read or directly got in contact with. Without clear messaging of what narrow use-case your platform solves, sales struggle to happen.

On the flip side, I hear the opposite statement as soon as the product or marketing message get focused. It becomes the most major growth factor and even drives people to your product instead of having to chase opportunities.

Rationale: why and when I think specialization is working.

“It is the great multiplication of the productions of all the different arts, in consequence of the division of labour, which occasions, in a well-governed society, that universal opulence which extends itself to the lowest ranks of the people” -Adam Smith

Look, the concept is not new. The argument for specialization is at the core of our modern society, and many philosophers, economists or other sales guru addressed it before me. The goal of this post, and this blog in general is not to debate whether or not capitalism is the most suitable model for our society but rather to give down-to-earth testimonies based on factual experiences.

With this in mind, here is ultimately why I think specialization enable sales: people hate complication. Despite what my inner nerd would like to think, everyone suffers from decision fatigue. We want to have one solution for one problem. This is why you use what’s app to text your Facebook friends instead of using Messenger in most cases. And I think it is particularly relevant for my generation that is driven by immediate selfish satisfaction (yes, I include myself in this) and want a quick response to a problem they have.

Take away: what you and I should reflect upon.

First and foremost, you need to make sure that your sales and marketing message is clear. You should be able to say what your product does, what it solves, what’s the market and who are your competitors. Then you need to be able to specialize your message even more, and drill down to what the person in front of you is looking for. When you’re playing tic-tac-toe against someone, you’re not thinking about every move that could have happened prior to the current move. You’re focusing on what move will give you the best chance to win now considering the current state of the game. It’s the same thing with sales: you’re not trying to sell your product to a range of hypothetical buyers, you’re trying to sell it to a specific person to solve a specific problem. Personalization is the ultimate specialization, thus the ultimate growth factor.

Now comes the hard question: what should I focus on? What is my product’s area of specialization? This is an extremely complicated question, because while people want reality to be simple, it isn’t. One current tendency established by Eric Ries in the Lean Startup is to use customer feedback and adapt your product to their needs: be data driven. While I adhere to this approach, especially when put against visionary decision making from leaders (which often equates to magical thinking), I think it needs be adjusted to account for lack of specialization. Yes, your product/company can pivot in any direction but it needs to settle. I haven’t found the formula to determine when to settle and what is the best specialization, nor do I think anyone has. But the thrill of uncertainty is what drives me everyday.