paul vidal - pragmatic big data nerd

Monthly Archives

4 Articles

Becoming intimate with Big Data

by paul 0 Comments
Becoming intimate with Big Data
Come on guys, we are all made of blue glass inside.

About a year ago, I had a chance to have a discussion with one of the smartest person I’ve ever met, currently a board member of our company. This man has not only built his fortune out of nothing by being able to identify trends in the market and position his companies accordingly, he is also a genuine human being that forces admiration. But I digress. During this conversation, he mentioned that one of the things that helped him succeed was his capacity to understand the intrinsic values that define a generation. As an example, he mentioned that his generation, during the 90s was all about financial success. The following generation, the 2000s kids was all about fame (big brother anyone?). Then he told me that he was yet to figure out what my generation was all about. Since then I have been able to understand what makes my generation tick. After about a year of poking around, I think that I found the answer: my generation is the selfish generation. We are all selfish and think about our individuality. Look around, it’s selfies, freedom above all, my Facebook or my privacy, my right for an opinion, my right for an outlet to express my idea. I’m including myself in this of course, I am writing a blog after all. What’s interesting about this realization is to understand the consequences it has on the market, and specifically in a domain in which I have at least a bit of expertise: Big Data.

Big Data is driven by the individual

In a recent report from Forrester (link), companies were asked “Which use cases are driving the demand for continuous global data availability at your organization?”. The most common use case representing 52% of the answers received was 360-degree view of the business, product. This means that more than half of the big data drivers are coming from the consolidation of data to represent an individual unit of business. Make no mistake, in many cases, the product is you. What drives big data is the intimate knowledge of the individual. This makes perfect sense if you agree with the premise of my first paragraph: big data, and the market in general wants to cater to the selfish generation, and therefore is implementing solutions to know each individual personally.

This report is only one of numerous examples corroborating what I’m trying to explain here. We see machine learning algorithms and data scientists arguing about what algorithm is the best to target individual with the right add. IoT is tracking and personalizing every aspects of our lives. Anecdotally, I even witnessed the re-naming of a data analytics team in a large company to “Your Data”.

What does this mean for your Big Data implementation

First you need to consider that in order to be able to keep a relevant edge on your competition, you must be able to have access to a solution to individualize your data collection. I have expressed this opinion quite a bit, but I believe that ultimately individualization of data is a use case that requires its own solution. There is no magic end to end consolidation platform that will do everything. You need to consider a big data individualization platform, as opposed to a big data generic platform that you then try to morph in order to cater to your individualization needs. Once implemented, this data individualization platform can be leveraged to implement further features like real-time provisioning, data virtualization, personalized analytics or real customer centric support, but your platform must be intimate with your unit of business first.

Essential resources on Machine Learning

by paul 0 Comments
Essential resources on Machine Learning
"Maybe you should be spending some time learning instead of relying on machines" - Some hipster

I’ve always been fascinated by Artificial Intelligence in science fiction. I’m so lucky to live in an era that is seeing the birth of a new kind of Artificial Intelligence, enabled by Big Data, advancement in super computers and Machine Learning. I’m even working in a field that gets to implement that kind of technologies, which continues to excite and fascinate me. Machine learning is today moving out of the realm of pure research to real-world applicability. But like any new cutting-edge technology, we need to beware of products untruthfully using the word Machine Learning in their marketing message or Machine Learning being the cure for all diseases. Therefore, I think it’s important that we spend some time understanding what Machine Learning is, as well as what it does and can do in the industry. Since I’m not an expert on Machine Learning (… yet), I spent some time gathering resources to enhance your Human Learning about Machine Learning. Happy reading!

Introductions

  • First things first, wikipedia: link
  • An excellent visual introduction on Machine Learning from R2D3: link
  • An early draft of a Machine Learning book from Stanford University: link
  • Introduction to Machine Learning from Cambridge University: link
  • Technical courses

  • In-depth videos on Machine Learning, from Data School: link
  • What is Machine Learning, from Data Camp: link
  • Introduction to Machine Learning, from Udacity: link
  • Machine Learning, from Coursera: link
  • Machine Learning in the market

  • Gartner 2015 Hype Cycle: Big Data is Out, Machine Learning is in: link
  • Gartner 2016 top 10 trends: link
  • Machine Learning, What it is & why it matters, from SAS: link
  • Marketplace for Machine Learning Algorithm, Algorithmia: link
  • The future of Machine Learning, from David Karger on Quora: link
  • The importance of specialization in software sales

    by paul 1 Comment
    The importance of specialization in software sales
    “Bust of Adam Smith” by Patric Parc, 1845. (Wikipedia)

    After spending some time reflecting on whether or not Data Scientist was a useful role within any organization churning a big amount of data, I stumbled upon this post on LinkedIn: There is Only One Type of Software Engineer.

    In short, this post calls for a de-specialization of the role of engineers in order to avoid siloed professionals refusing to take responsibility of a task if it does not exactly match their job description.

    While I agree with some of this argument, especially in big organizations where unfortunately the lack of ownership of a task and fear of risk taking can be quite flagrant (which I will try to tackle in a future post), I think that small organizations are in serious lack of specialization, the effect of which are particularly visible in the sales process.

    Establishing the premise: specialization scarcity versus tangible gains.

    Quick disclaimer: as for every post I write, I am not trying to establish and write the ultimate truth but rather I’m engaging into a conversation, so if I’ll be happy to see my premise challenged. It also applies mostly to my domain of expertise: large software sales for big organization.

    In this context, here is what I observed and gathered from the market. The era of all-in-one platforms is over. I debated this in my first article speaking about consolidation, but it is still true. And while some giant companies manage and should attack different segment of the market, they still have clear marketing messages and product names for these different segments.

    The problem is for companies that have an extremely innovative product, that could tackle a lot of use-cases. I know we have been struggling with this in my company, although it is getting fixed now, but I also observed this statement for many other companies I read or directly got in contact with. Without clear messaging of what narrow use-case your platform solves, sales struggle to happen.

    On the flip side, I hear the opposite statement as soon as the product or marketing message get focused. It becomes the most major growth factor and even drives people to your product instead of having to chase opportunities.

    Rationale: why and when I think specialization is working.

    “It is the great multiplication of the productions of all the different arts, in consequence of the division of labour, which occasions, in a well-governed society, that universal opulence which extends itself to the lowest ranks of the people” -Adam Smith

    Look, the concept is not new. The argument for specialization is at the core of our modern society, and many philosophers, economists or other sales guru addressed it before me. The goal of this post, and this blog in general is not to debate whether or not capitalism is the most suitable model for our society but rather to give down-to-earth testimonies based on factual experiences.

    With this in mind, here is ultimately why I think specialization enable sales: people hate complication. Despite what my inner nerd would like to think, everyone suffers from decision fatigue. We want to have one solution for one problem. This is why you use what’s app to text your Facebook friends instead of using Messenger in most cases. And I think it is particularly relevant for my generation that is driven by immediate selfish satisfaction (yes, I include myself in this) and want a quick response to a problem they have.

    Take away: what you and I should reflect upon.

    First and foremost, you need to make sure that your sales and marketing message is clear. You should be able to say what your product does, what it solves, what’s the market and who are your competitors. Then you need to be able to specialize your message even more, and drill down to what the person in front of you is looking for. When you’re playing tic-tac-toe against someone, you’re not thinking about every move that could have happened prior to the current move. You’re focusing on what move will give you the best chance to win now considering the current state of the game. It’s the same thing with sales: you’re not trying to sell your product to a range of hypothetical buyers, you’re trying to sell it to a specific person to solve a specific problem. Personalization is the ultimate specialization, thus the ultimate growth factor.

    Now comes the hard question: what should I focus on? What is my product’s area of specialization? This is an extremely complicated question, because while people want reality to be simple, it isn’t. One current tendency established by Eric Ries in the Lean Startup is to use customer feedback and adapt your product to their needs: be data driven. While I adhere to this approach, especially when put against visionary decision making from leaders (which often equates to magical thinking), I think it needs be adjusted to account for lack of specialization. Yes, your product/company can pivot in any direction but it needs to settle. I haven’t found the formula to determine when to settle and what is the best specialization, nor do I think anyone has. But the thrill of uncertainty is what drives me everyday.

    10 things you need to tell your inner 20 year-old to stop doing

    by paul 0 Comments
    10 things you need to tell your inner 20 year-old to stop doing

    This week I’m trying something a little different. I realize that I have less experience in career advice than in the data world, and that this post might veer into self-indulging therapy, but I do think it has value nonetheless. The elements in the list I am about to present are things I genuinely wish I knew when I started my career and that I still fight against. Hopefully, some of you can relate to this and who knows, it may give you some perspective on your current actions. With the context set, let’s dive into this: 10 things you (and I) need to tell our inner 20 year-old to stop doing.

    1. Being a cynical jerk

    “If we believe in nothing, if nothing has any meaning and if we can affirm no values whatsoever, then everything is possible and nothing has any importance.” -Albert Camus

    Granted this is more existentialism than cynicism but it does illustrates my point. Spend your time contemplating the supposedly uselessness of your work, complaining about your routine and trying to establish solidarity around this vision of your job is idiotic. There will be displeasing aspects in your work, just as it is the case of any endeavor to which you devote more than 30 minutes. The question that you need to ask yourself is how bad the situation actually is, does it serve a purpose in your career plan, and if you need to change something. Complaining for complaining is counter productive, despite what every fiber of my French body wants me to believe, so don’t do it.

    2. Not wanting to pick up the phone

    This is why texting and apps like grubhub are so popular, and why voicemail is dead. We all hate calling someone, especially for a difficult conversation. Don’t misinterpret this with organizing meetings for the sake of organizing meetings, this is a completely different issue. What I’m talking about here is calling someone before or instead of sending an email/chat/text. Electronic communication is essential, but it does not replace phone calls. If you can’t find the right words to put in an email, try calling first.

    3. Not having a career plan

    What do you want to be in 2, 3, 5 years? Is what you’re doing know actively helping towards that goal? If you can’t answer that, you’re doing it wrong. I’m not saying that plans don’t change or that it is easy to know which direction to take (queue Camus’ quote from the first point), but it is the key to not waste time. It’s a difficult thing to face your own state of despair, but it will make you a more focused individual. Identify where you want to be. Give yourself the tools to get there.

    4. Underestimating the value of relationships

    However much you value relationships in your work environment, I’m fairly certain you can multiply that value by 2 or 3. At least that was the case for me until extremely recently. Take the time to speak with people. Pick up the phone, talk to them on social media, the world we live in make building relationships easier than before. Use it to your advantage. Don’t think you will get where you want to be alone.

    5. Thinking that other people are idiots

    I used to say, following the word of wisdom of many elders, that the worst thing in life is to work with idiots. I realize now that this was misguided. Not that I literally thought that a significant portion of my co-workers were dumber than I was (although if that’s what you actually think, you may have to climb down off your narcissist pedestal and join us common folks), more that they were many areas in which they weren’t as skilled as I am (or so I thought). The misguided part of this thought is thinking that everyone has the same objectives and interests as you. They don’t. And the fact that they don’t understand one thing you do, or even if we take this to an extreme that their cognitive capabilities are less proficient than yours, is irrelevant. When faced with someone who does not seem to understand something, or does something you think he shouldn’t have done, use it as an opportunity to learn about them, not to judge them; which brings us to our next point.

    6. Failing to understand others’ ultimate motive

    This is the next level of my previous point. I can attest from personal experience that I did (and still do in some occasions) misunderstand completely a person and get genuinely surprised by some of their action, which more often than not end up costing me a great deal. So try to do this: instead of dismissing people for the behavior and actions in which they engage, use these as data points aggregating towards building a portrait of them; this will save you a few surprises.

    7. Hesitating to ask questions …

    I was once working at a customer’s site to install one of our products and we needed a few boring environment access to be setup. Assigned to help me out was a freshly hired guy. As usual in that kind of situations, nothing was easy, and we were basically stuck until someone approved something. So I ask that guy: “who’s the person in charge of this approval?”; once he gave me his name I said: “good, let’s ask him when we can expect the approval process to be finished”. Of course, this made the freshly hired quite uncomfortable and he did not want to do it. In the end, we did contact the guy, and he was happy that we called (see point #2) because he had a few questions about what we were trying to do; I even think that ultimately it was quite beneficial for the new guy’s visibility in the organization. So if you have a legitimate question to ask to someone, regardless of their position, you should ask it. You have very little to risk unless it is misguided, which is why you should think before asking.

    8. … especially to yourself.

    This is fairly simple. Think before you ask something. Don’t be afraid, but use everything you know about your audience before asking a question or replying to an email. I know I’m guilty of angsty responses and I’m working on it. The key here is that if everyone else is applying the principles laid out in point #6, then they are building a portrait of you from multiple data points. Don’t give them negative data points.

    9. Not looking for ways to go the extra mile

    “Ask yourself ‘Can I give more?’. The answer is usually: Yes.” -Paul Tergat, Kenyan professional marathoner

    I admit this might sound a little cheesy, but it is worth saying. I believe that you can always find more relevant things to do to better your knowledge, your business relationships and your recognition from other people. While this extra work might sometimes feel overwhelming, you can always prioritize it. Don’t keep this out of sight. Do more.

    10. Compromising your values

    … unless you have terrible values. More seriously, this is very good advice that I gave to myself after my first job. In my first job, I tried to become someone I wasn’t for the sake of fitting into a certain culture. I believe this was one of the biggest mistake I have ever made. Excelling at a job and pushing your career forward is hard enough, you don’t want to have to battle fundamental internal conflicts. I use the word fundamental knowingly. Like I touched on during my first point, you will always have doubts about what you’re doing, rightfully so. You spend a very high percentage of your time working, I don’t believe it is sustainable long term if you compromise your values, the same way you can’t be in a long term love relationship if you lie to one of the parties (you or your partner). Make sure your work environment does not make you become something you’re not.