Project Management and Big Data – as a project

Post written by Jason Z., Project Manager at Ideaca. Read more about project management on his blog: Unnatural Leadership.

As part of this month’s Ideaca blogging network challenge, we were tasked with discussing our thoughts on Big Data.

This is going to be a 2 part post:

• The first part will cover how you, as a project manager, should approach a project that carries the mantle of “Big Data.”
• The second part will cover how you, as someone in a Project/Program Management Office, can use Big Data without getting snookered by the hype.

Part 1 – So you’ve been asked to “implement Big Data”… what now?

Defining Your Terms
I am going to assume that you – like me – tend to be baffled by the marketing speak until you can speak with someone intelligently about a topic. In the case of Big Data, I have heard a few definitions. The one that seems to stick the most for me is the one from Wikipedia:

• Data sets that are too big for traditional database management systems to handle
• Data sets that comprise information from multiple sources to try to infer correlation

Sounds easy enough, right?
Where it starts to get complicated (thanks Wade!) is when you try to integrate “unstructured and semi-structured data with our 'traditional' structured data.”

You will never “implement Big Data”
When it comes to Big Data, you do not implement it. You may be implementing a technology to support the analysis, but you will never actually implement this “thing.” A project of this sort relies on understanding the user requirements, selecting the right technology, and taking an exploratory approach when developing reporting capabilities.

Understanding the User Requirements
In the case of a new process and technology, such as this, your user requirements may be fairly light. "We want to correlate information from disparate sources to identify predictive trends” or “I don’t know – but I really want some cool looking reports” may be common lines that you hear. Like all projects, the user requirements are your definition of success. Because “Big Data” is still a technology in the exploratory stage, though, expecting detailed requirements may be the wrong sorts of requirements. The ones that you should be really focused on are the data sources and ensuring that the information being presented is right.

To wit, if I were to ask you to present the information on the average CEO compensation for the top 50 companies in North America, how would you start? How would you define the Top 50?  By Market Capitalization? By Environmental Performance? By Stock Price? By Revenue? What about getting access to private company information? All of the sudden, a fairly simple question about the average CEO compensation gets a little more complex.

The same will be true of your Big Data project. Start by understanding that to present the information your users want, you will either have to ask a whole lot of detailed questions, or provide a platform to enable them to answer their own questions.

Understanding the available technology
As Project Managers, we know that when we are asked to Implement something, it’s never that simple. Understanding what the technology can and cannot do is critical to ensuring that your project can meet the user’s definition of success.

One might want to satisfy the guiding principles of a company’s Enterprise Architecture. A quick scan of the landscape will reveal that tools like SAP HANA, Oracle’s Exadata, and Amazon’s AWS can all fulfill the technology requirements quite nicely and potentially support a company’s Enterprise Architecture. However, since this is a new application of technology, fulfillment of requirements needs to trump Enterprise Architecture.

Take an Exploratory and Iterative Approach to reporting
Some organizations will judge success of your project by its ability to deliver a load of reports. If this sounds like your organization, be realistic as to what can be delivered. Deliver a robust and reliable dataset, some transactional reports, and one report that really helps demonstrate the art of the possible.

Smarter organizations will judge the success of your project by its ability to deliver analytic capabilities to the user base. The robust and reliable dataset is still mandatory, but the ability for users to generate their own reports will satisfy all of the “what about …?” requirements that would blow your project budget and schedule out of the water.

In the end… it’s the people that matter
If we believe all of the marketing hype, Big Data will help us explore all the myriad of ways our world is constructed. But from the perspective of a Big Data as a project, an empowered user base will produce much more value than some canned reports.

Have you been asked to “implement big data”?
If so, what did your project look like? Let me know in the comments down below. Stay tuned for another post on making the most of Big Data in a PMO.


Special thanks to Wade Walker and Chris Sorensen for keeping me honest with this post.

Just Imagine...

Post written by Chris S., BI Consultant at Ideaca. Read more about BI on his blog: The Outspoken Data Guy.

For quite some time I have been imagining what the possibilities of Big Data might be. I am certainly no expert in the area but being the data guy that I am, I often wonder what might be able to be done with data that may be being collected at any point in time. Face it, we are so connected now that our every move generates some form of data and often multiple pieces of it.

For example, if a marketer wanted to know everything about Chris Sorensen in a given day, chances are that most of that data is logged somewhere. What time I leave my house is available via my cell phone, my driving directions and speed are also available there as well. When I sit on the train I surf the web, send emails and organize my task list, all of these actions generate recorded data. What time I log into work, how often I am active on my computer and what I am do all day long is logged. Where I shop, what I buy (if I have a rewards card) is all tracked. My Facebook views, tweets all contain things that could be used to build a personality model of myself and my habits.

It is not really that big of stretch to think that this data could be used in one gigantic model to predict my next move and perhaps even entice me to make a different one. Maybe instead of stopping at Home Depot to get my painting supplies, an app could suggest the best place for me to go based on what I am doing. Sound like a stretch? Not really…Think about the labor that gold miners went through just to get a few stones. Now gigantic machinery does the same thing. The same thing is happening with Big Data where machines are able to gather information from a variety of sources and store large volumes of it in order to form predictive models. We are only at the tip of the iceberg but just imagine what the possibilities might be

The importance of a shared vision

Post written by Jason Z., Project Manager at Ideaca. Read more about project management on his blog: Unnatural Leadership.

In a post from my series “Advice for Junior PMs," I touched on the concept of saying what you mean when working with your project team. The same concept should be applied when communicating outside of your project team.

There’s a fairly common graphic that gets passed around IT departments, and it’s somewhat self-deprecating. It shows that project teams tend to not understand what the customer needs – which is endemic of lacking a shared vision.

This graphic makes me cringe every time I see it.

As we all know, a project is a temporary group activity designed to produce a unique product, service or result. However, more often than not, project teams take an “I know best” view of the world when designing solutions for their customer.

A strong project manager will not only sit with their customer to understand what is required, but will bring the whole project team along to understand as well. We all have our own perceptions and filters, and as a result may play broken telephone.

At this point, you may be asking if a shared vision is different from the project scope statement. It is, in that the shared vision is what the customer will see as the product, service, or result of the project, whereas the project scope is everything that will be delivered (including training, documentation, organizational change management).

To create a shared vision of what the project will produce (be it a unique product, service, or result):

1. Bring everyone to the table to ensure open communication

1. Define what is to be produced in simple language – do not say “we are going to produce a tree swing,” and leave it there, say “we are going to produce a tree swing, which is comprised of a tire hanging from a sturdy branch of a large oak tree by a piece of polyester rope.”

1. Involve the customer in design meetings. Subject Matter Experts (SMEs) should definitely lead, but should be eliciting feedback so that the customer’s requirements are re-confirmed by the team.

1. Revisit the shared vision often. Ask your customer at difference acceptance testing points if what is being developed meets the shared vision.

Most importantly, communicate the shared vision often. Use it as the first line in your status reports, use it as part of your elevator speech, and when people ask you what you are working on, relay your project’s shared vision.

What are your tips for creating a shared vision? What have you seen work well? Do you have any stories of spectacular failures? Share your tips and stories below!

Standards = Starting Point

 Post written by Wade W., BI Consultant at Ideaca. Read more about BI on his blog: Pragmatic Business Intelligence.  

In a data migration project, standards are synonymous with quality.

Every developer has a different philosophy of what works. Many say that it is easier to develop with “what I know," which sounds a lot like “quick and dirty."

The definition of, or existence of Standards of Development and Naming Conventions provide guidelines within which developers should be expected to work. Without these, your environment quickly becomes rife with development packages, interfaces and jobs with different naming conventions, different approaches and widely varying levels of development quality.

I think it is common that, lacking a mentor or some kind of guidance, developers new to data migration start the same way – monster jobs, lack of flexibility, lack of clarity…and lack of documentation. Result: effectively, unmaintainable, throw-away jobs.

The good news is, as discussed, there is a remedy: Take the time to define standards, or work with a supplier who uses a proven methodology based on established standards and quality-centric processes…ideally processes that can be templated and reused.

Re-usability of processes (i.e. “templates") should be your objective. Ensure that in your environment, your team lead is responsible to establish a set of skeleton templates (say 5-10?) that 95% of all your data migration mappings can be based on. “Skeleton” templates means that they are pre-populated with the parameters (that’s “placeholders” for the project-specific values) – these skeleton templates contain no table structure information – just as much development that can be reused in all cases.

Once you have this in place, you can quantifiably calculate substantial cost savings just from having these templates in place… from every project.

Really.

Sliced or Shaved? Avoiding spreading your BI team too thin

 Post written by Chris S., BI Consultant at Ideaca. Read more about BI on his blog: The Outspoken Data Guy.

As a consultant with a background in Agile, I often get questions about how Agile can be used to solve certain problems that people are having with their Business Intelligence Programs.

I recently sat with a client to listen to some of the issues that they are currently having with their BI program. One of the biggest issues that this client is facing is what I would classify as a simple supply and demand problem. Basically their team of around 8 people cannot keep up with the demands of developing and sustaining their BI/DW environment in what is a large organization. The main question for me was could Agile help solve this problem. In my experience, Agile cannot solve the problem directly but it can be used to highlight the root cause.

This is a very common problem that BI programs face. It is the fact that teams are often small relative to the size of an organization and are also too small to manage the tasks that they need to perform to grow and maintain a BI portfolio. And in certain circumstances it is compounded by the fact that teams are often staffed with the wrong skills sets needed to grow and manage a BI offering.

So how can Agile help?

With proper tracking and monitoring of what the team does on a daily basis, teams can begin to gather data on what types of work the team is doing on a daily basis. What we often find is that at a certain point new development will stop coming from small teams charged with both the development and sustainment of a program as they cannot keep up with both. The ironic thing is that most BI managers have no real data to back this up. So taking advantage of some of the rigor around agile in terms of tracking what is done on a daily basis and how slowly new work burns down, one can begin to understand and report better on how time is spent and in fact how little time is available to delivering new functionality.