Hadoop, Fridges & Insight Cakes
Category : Blogs
So, you like cake, right? Think of your favourite one.Maybe it’s a devil’s food cake with thick chocolate icing on the top, or perhaps a Victoria Sponge, strawberry jam oozing out. Maybe you’re a bit weird and it’s a carrot cake or something.Whatever it is, just hold that image of tasty, tasty cake in your mind, because I’m about to construct a long and torturous cake-based analytics analogy, and the best way to get through it is to focus mainly on the cake (or scroll back up to the top of this page and look at the picture).Now then, I’ve established that the goal is cake, but what if that cake was made of actionable insights that you want to feed into the hungry maws of your business end users (stay with me, it’s only going to get worse from here)?Well you’re going to want to gather the ingredients together, and for this metaphor the ingredients are going to be data (you saw that coming, right?). Now you might keep your ingredients in separate cupboards, and some of them might be right at the back and a bit inaccessible or something, so it might be a pain to get your flour, eggs and sugar all in the same place, which is how I segue into talking about Hadoop (No, really).Hadoop is pretty much the hottest topic in data circles right now, hailed left right and centre as the future of data storage and big data analysis. It’s fast, it’s extremely scalable, it’s relatively cheap and easy to setup, but while all these things are great and have terrific implications for both mass data storage and the provision of web services, there’s an elephant in the room when what you want to do is actually interrogate and interpret your data – it won’t bake you the cake.To explain what I’m talking about, let’s return to my imaginary kitchen. If you’ve got your ingredients scattered all over the place, the eggs in the SQL database cupboard, the flour in the analytics platform cupboard, that’s fine, you can still bake a cake, but it’s going to take you longer, right? In this situation Hadoop works like a fridge.All your ingredients go in the fridge, and they’ll stay there until you need them, fresh and available and from just one place. There are other data warehousing options that would also work like this (Teradata, Oracle, various other SQL platforms) but because Hadoop is one of those massive American fridges, with double doors and an ice dispenser on the front, it doesn’t care how many cakes you want to make, or even which kind of cake you’re making, it’ll take all the ingredients you’ve got.“Sure,” it’ll say, “you’re making Chocolate Cake today? Well I’ve got your chocolate here, and by the way, it might go really well with these cherries I just had lying around. Don’t worry, I’ll keep the walnuts until next time you want carrot cake.”As an aside; yes, I did just compare Hadoop to a metaphorical fridge and then anthropomorphise it so it could talk to you about cake. No, I’m not on drugs.So your new shiny, talking, infinitely capacious Hadoop fridge has all your cake ingredients in one place, and you can hear those business users are clamouring for their cake. Time to get Hadoop to churn out some fabulous data insights with a few cherries on the top, right? Except it can’t do that, because it’s a fridge (obviously). On its own, all you’re going to get out is the same ingredients you put in. There’s two things still needed for the perfect cake – something to actually cook it in, and someone to do the cooking.Surprise, surprise, I’m going to tell you that the key part of making the cake is the analyst cooking it; it isn’t going to matter where the ingredients came from if you don’t know how to mix them together in the right way. It isn’t going to matter if you cook the cake in the microwave (Tableau), the Aga (Adobe Data Workbench) or the sous-vide machine with attached dehydrator (R), if you’ve no idea how long it needs to bake for.I guess what I’m trying to say in a very rambling kind of way, is this:Whichever tools you have, you’re going to need a good cook to make a delicious cake that’ll have your business stakeholders coming back for seconds. That’s why, regardless of your business’s level of technology, you’re always going to need good analysts who can turn the basic ingredients into a mouth-watering treat (which is lucky, because otherwise I’d be out of a job). All you have to do is make sure you’ve got the right people and the right ingredients, and you can have your cake and eat it.Now, where are the chocolate sprinkles?