Lisa Morgan's Official Site

Click-worthy Content Development

Tag: Data science

Big Data: The Interdisciplinary Vortex

As seen in  InformationWeek.

vortexGetting the most from data requires information sharing across departmental boundaries. Even though information silos remain common, CIOs and business leaders in many organizations are cooperating to enable cross-functional data sharing to improve business process efficiencies, lower costs, reduce risks, and identify new opportunities.

Interdepartmental data sharing can take a company only so far, however, as evidenced by the number of companies using (or planning to use) external data. To get to the next level, some organizations are embracing interdisciplinary approaches to big data.

Why Interdisciplinary Problem-Solving May Be Overlooked

Breaking down departmental barriers isn’t easy. There are the technical challenges of accessing, cleansing, blending, and securing data, as well as very real cultural habits that are difficult to change.

Today’s businesses are placing greater emphasis on data scientists, business analysts, and data-savvy staff members. Some of them also employ or retain mathematicians and statisticians, although they may not have considered tapping other forms of expertise that could help enable different and perhaps more accurate forms of data analysis and new innovations.

“Thinking of big data as one new research area is a misunderstanding of the entire impact that big data will have,” said Dr. Wolfgang Kliemann, associate VP for research at Iowa State University. “You can’t help but be interdisciplinary because big data is affecting all kinds of things including agriculture, engineering, and business.”

Although interdisciplinary collaboration is mature in many scientific and academic circles, applying non-traditional talent to big data analysis is a stretch for most businesses.

But there are exceptions. For example, Ranker, a platform for lists and crowdsourced rankings, employs a chief data scientist who is also a moral psychologist.

“I think psychology is particularly useful because the interesting data today is generated by people’s opinions and behaviors,” said Ravi Iyer, chief data scientist at Ranker. “When you’re trying to look at the error that’s associated with any method of data connection, it usually has something to do with a cognitive bias.”

Ranker has been working with a UC Irvine professor in the cognitive sciences department who studies the wisdom of crowds.

“We measure things in different ways and understand the psychological biases each method of data creates. Diversity of opinion is the secret to both our algorithms and the philosophy behind the algorithms,” said Iyer. “Most of the problems you’re trying to solve involve people. You can’t just think of it as data, you have to understand the problem area you’re trying to solve.”

Why Interdisciplinary Problem-Solving Will Become More Common

Despite the availability of new research methods, online communities, and social media streams, products still fail and big-name companies continue to make high-profile mistakes. They have more data available than ever before, but there may be a problem with the data, the analysis, or both. Alternatively, the outcome may fall short of what is possible.

“A large retail chain is interested in figuring out how to optimize supply management, so they collect the data from sales, run it through a big program, and say, ‘this is what we need.’ This approach leads to improvements for many companies,” said Kliemann. “The question is, if you use this specific program and approach, what is your risk of not having the things you need at a given moment? The way we do business analytics these days, that question cannot be answered.”

One mistake is failing to understand the error structure of the data. With such information, it’s possible to identify missing pieces of data, what the possible courses of action are, and the risk associated with a particular strategy.

“You need new ideas under research, ideas of data models, [to] understand data errors and how they propagate through models,” said Kliemann. “If you don’t understand the error structure of your data, you make predictions that are totally worthless.”

Already, organizations are adapting their approaches to accommodate the growing volume, velocity, and variety of data. In the energy sector, cheap sensors, cheap data storage, and fast networks are enabling new data models that would have been impossible just a few years ago.

“Now we can ask ourselves questions such as if we have variability in wind, solar, and other alternative energies, how does it affect the stability of a power system? [We can also ask] how we can best continue building alternative energies that make the system better instead of jeopardizing it,” said Kleinman.

Many universities are developing interdisciplinary programs focused on big data to spur innovation and educate students entering the workforce about how big data can affect their chosen field. As the students enter the workforce, they will influence the direction and culture of the companies for which they work. Meanwhile, progressive companies are teaming up with universities with the goal of applying interdisciplinary approaches to real-world big data challenges.

In addition, the National Science Foundation (NSF) is trying to accelerate innovation through Big Data Regional Innovation Hubs. The initiative encourages federal agencies, private industry, academia, state and local governments, nonprofits, and foundations to develop and participate in big data research and innovation projects across the country. Iowa State University is one of about a dozen universities in the Midwestern region working on a proposal.

In short, interdisciplinary big data problem-solving will likely become more common in industry as organizations struggle to understand the expanding universe of data. Although interdisciplinary problem-solving is alive and well in academia and in many scientific research circles, most businesses are still trying to master interdepartmental collaboration when it comes to big data.

Six Characteristics of Data-Driven Rock Stars

As seen in InformationWeek

Rock starData is being used in and across more functional aspects of today’s organizations. Wringing the most business value out of the data requires a mix of roles that may include data scientists, business analysts, data analysts, IT, and line-of-business titles. As a result, more resumes and job descriptions include data-related skills.

A recent survey by technology career site Dice revealed that nine of the top 10 highest-paying IT jobs require big data skills. On the Dice site, searches and job postings including big data skills have increased 39% year-over-year, according to Dice president Shravan Goli. Some of the top-compensated skills include big data, data scientist, data architect, Hadoop, HBase, MapReduce, and Pig — and the pay range for those skills ranges from more than $116,000 to more than $127,000, according to data Dice provided to InformationWeek.

However, the gratuitous use of such terms can cloud the main issue, which is whether the candidate and the company can turn that data into specific, favorable outcomes — whether that’s increasing the ROI of a pay-per-click advertising campaign or building a more accurate recommendation engine.

If data skills are becoming necessary for more roles in an organization, it follows that not all data-driven rock stars are data scientists. Although data scientists are considered the black belts, it is possible for other roles to distinguish themselves based on their superior understanding and application of data. Regardless of a person’s title or position in an organization, there are some traits common to data-driven rock stars that have more to do with attitudes and behaviors than technologies, tools, and methods. Click through for six of them.  [Note to readers:  This appeared as a slideshow.]

They Understand Data

Of course data-driven rock stars are expected to have a keener understanding of data than their peers, but what exactly does that mean? Whether a data scientist or a business professional, the person should know where the data came from, the quality of it, the reliability of it, and what methods can be used to analyze it, appropriate to the person’s role in the company.

How they use numbers is also telling. Rather than presenting a single number to “prove” that a certain course of action is the right one, a data-driven rock star is more likely to compare the risks and benefits of alternative courses of action so business leaders can make more accurate decisions.

“‘Forty-two’ is not a good answer,” said Wolfgang Kliemann, associate VP for research at Iowa State University. “‘Forty-two, under the following conditions and with a probability of 1.2% chance that something else may happen,’ is a better answer.”

They’re Curious

Data-driven rock stars are genuinely curious about what data indicates and does not indicate. Their curiosity inspires them to explore data, whether toggling between data visualizations, drilling down into data, correlating different pieces of data, or experimenting with an alternative algorithm. The curiosity may be inspired by data itself, a particular problem, or problem-solving methods that have been used in a similar or different context.

Data scientists are expected to be curious because their job involves scientific exploration. Highly competitive organizations hire them to help uncover opportunities, risks, behaviors, and other things that were previously unknown. Meanwhile, some of those companies are encouraging “out of the box” thinking from business leaders and employees to fuel innovation, which increasingly includes experimenting with data. Some businesses even offer incentives for data-related innovation.

They Actively Collaborate with Others

The data value chain has a lot of pieces. No one person understands everything there is to know about data structure, data management, analytical methods, statistical analysis, business considerations, and other factors such as privacy and security. Although data-driven rock stars tend to know more about such issues than their peers, they don’t operate in isolation because others possess knowledge they need. For example, data scientists need to be able to talk to business leaders and business leaders have to know something about data. Similarly, a data architect or data analyst may not have the ability to manipulate, explore, understand, and dig through large data sets, but a data scientist could dig through and discover patterns and then bring in statistical and programming knowledge to create forward-looking products and services, according to Dice president Shravan Goli.

They Try to Avoid Confirmation Bias

Data can be used to prove anything, especially a person’s opinion. Data-driven rock stars are aware of confirmation bias, so they are more likely to try to avoid it. While the term itself may not be familiar, they know it is not a best practice to disregard or omit evidence simply because it differs from their opinions.

“People like to think that the perspective they bring is the only perspective or the best perspective. I’m probably not immune to that myself,” said Ravi Ivey, chief data scientist at Ranker, a platform for lists and crowdsourced rankings. “They have their algorithms and don’t appreciate experiments or the difference between exploratory and confirmatory research. I don’t think they respect the traditional scientific method as such.”

The Data Science Association’s Data Science Code of Professional Conduct has a rule dedicated specifically to evidence, data quality, and evidence quality. Several of its subsections are relevant to confirmation bias. Among them are failing to “disclose any and all data science results or engage in cherry-picking” and failing to “disclose failed experiments or disconfirming evidence known to the data scientist to be directly adverse to the position of the client.”

They Update Their Skill Sets

Technology, tools, techniques, and available data are always evolving. The data-driven rock star is motivated to continually expand his or her knowledge base through learning, which may involve attending executive education programs, training programs, online courses, boot camps, or meetups, depending on the person’s role in the company.

“I encourage companies to think about growing their workforce because there aren’t enough people graduating with data science degrees,” said Dice president Shravan Goli. “You have to create a pathway for people who are smart, data-driven, and have the ability to analyze patterns so they have to add a couple more skills.”

Job descriptions and resumes increasingly include more narrowly defined skills because it is critical to understand which specific types of big data and analytical skills a candidate possesses. A data-driven rock star understands the technologies, tools, and methods of her craft as well as when and how to apply them.

They’re Concerned About Business Impact

With so much data available and so many ways of analyzing it, it’s easy to get caught up in the technical issues or the tasks at hand while losing site of the goal: using data in a way that positively impacts the business. A data-driven rock star understands that.

Making a business impact requires three things, according to IDC adjunct research adviser Fred McGee: having a critical mass of data available in a timely manner, using analytics to glean insights, and applying those insights in a manner that advances business objectives.

A data-driven rock star understands the general business objectives as well as the specific objective to which analytical insights are being applied. Nevertheless, some companies are still falling short of their goals. Three-quarters of data analytics leaders from major companies recently told McKinsey & Company that, despite using advanced analytics, their companies had improved revenue and costs by less than 1%.

Six Ways to Master the Data-Driven Enterprise

As seen in InformationWeek.

StatisticsBig data is changing the way companies and industries operate. Although virtually all businesses acknowledge the trend, not all of them are equally prepared to meet the challenge. The companies in the best position to compete have transformed themselves into “data-driven” organizations.

Data-driven organizations routinely use data to inform strategy and decision-making. Although other businesses share the same goal, many of them are still struggling to build the necessary technological capabilities, or otherwise their culture is interfering with their ability to use data, or both.

Becoming a data-driven organization isn’t easy, however. In fact, it’s very difficult. While all organizations have a glut of data, their abilities to collect it, cleanse it, integrate it, manage it, access it, secure it, govern it, and analyze it vary significantly from company to company. Even though each of these factors helps ensure that data can be used with higher levels of confidence, it’s difficult for a business to realize the value of its data if its corporate culture lags behind its technological capabilities.

Data-driven organizations have extended the use of data across everyday business functions, from the C-suite to the front lines. Rather than hoping that executives, managers, and employees will use business intelligence (BI) and other analytical tools, companies that are serious about the use of data are training employees, making the systems easier to use, making it mandatory to use the systems, and monitoring the use of the systems. Because their ability to compete effectively depends on their ability to leverage data, such data-driven organizations make a point of aligning their values, goals, and strategies with their ability to execute.

On the following pages we reveal the six traits common to data-driven organizations that make them stand out from their competitors.

Forward Thinkers

Data-driven enterprises consider where they are, where they want to go, and how they want to get there. To ensure progress, they establish KPIs to monitor the success of business operations, departments, projects, employees, and initiatives. Quite often, these organizations have also established one or more cross-functional committees of decision-makers who collectively ensure that business goals, company practices, and technology implementations are in sync.

“The companies that have integrated data into their business strategies see it as a means of growing their businesses. They use it to differentiate themselves by providing customers with better service, quicker turnaround, and other things that the competition can’t meet,” said Ken Gilbert, director of business analytics at the University of Tennessee’s Office of Research and Economic Development, in an interview with InformationWeek. “They’re focused on the long-term and big-picture objectives, rather than tactical objectives.”

Uncovering Opportunities

Enterprises have been embracing BI and big data analytics with the goal of making better decisions faster. While that goal remains important to data-driven enterprises, they also are trying to uncover risks and opportunities that may not have been discoverable previously, either because they didn’t know what questions to ask or because previously used technology lacked the capability.

According to Gartner research VP Frank Buytendijk, fewer than half of big data projects focus on direct decision-making. Other objectives include marketing and sales growth, operational and financial performance improvement, risk and compliance management, new product and service innovation, and direct or indirect data monetization.

Hypothesis Trumps Assumption

People have been querying databases for decades to get answers to known questions. The shortcoming of that approach is assuming that the question asked is the optimal question to ask.

Data-driven businesses aim to continuously improve the quality of the questions they ask. Some of them also try to discover, through machine learning or other means, what questions they should be asking that they have not yet asked.

The desire to explore data is also reflected in the high demand for interactive self-service capabilities that enable users to adjust their thinking and their approaches in an iterative fashion.

Pervasive Analytics

Data analytics has completely transformed the way marketing departments operate. More departments than ever are using BI and other forms of analytics to improve business process efficiencies, reduce costs, improve operational performance, and increase customer satisfaction. A person’s role in the company influences how the data is used.

Big data and analytics are now on the agendas of boards of directors, which means that executives not only have to accept and support the use of the technologies, they also have to use them — meaning they have to lead by example. Aberdeen’s 2014 Business Analytics survey indicated that data-driven organizations are 63% more likely than the average organization to have “strong” or “highly pervasive” adoption of advanced analytical capabilities among corporate management.

Failure Is Acceptable

Some companies encourage employees to experiment because they want to fuel innovation. With experimentation comes some level of failure, which progressive companies are willing to accept within a given range.

Encouraging exploration and accepting the risk of failure that accompanies it can be difficult cultural adjustments, since failure is generally considered the opposite of success. Many organizations have made significant investments in big data, analytics, and BI solutions. Yet, some hesitate to encourage data experimentation among those who are not data scientists or business analysts. This is often because, historically, the company’s culture has encouraged conformity rather than original thinking. Such a mindset not only discourages innovation, it fails to acknowledge that the failure to take risks may be more dangerous than risking failure.

Data Scientists And Machine Learning

Data-driven companies often hire data scientists and use machine learning so they can continuously improve their ability to compete. Microsoft, IBM, Accenture, Google, and Amazon ranked first through fifth, respectively, in a recent list of 7,500 companies hiring data scientists. Google, Netflix, Amazon, Pandora, and PayPal are a few examples of companies using machine learning with the goal of developing deeper, longer-lasting, and more profitable relationships with their customers than previously possible.

Tech Buying: 6 Reasons Why IT Still Matters

ErrorOriginally published in InformationWeek, and available as a slideshow here.

Making major tech purchases, especially big data analytics and business intelligence tools, without consulting IT may cause major problems. Here’s why.

Although shadow IT is not new, the percentage of business tech purchases made outside IT is significant and growing. When Bain & Company conducted in-depth interviews with 67 marketing, customer service, and supply chain professionals in February 2014, it found that nearly one-third of technology purchasing power had moved to executives outside of IT. Similarly, member-based advisory firm CEB has estimated that non-IT departments control 30% of enterprise IT spend. By 2020, Gartner estimates, 90% of tech spending will occur outside IT.

There are many justifications for leaving IT in the dark about departmental tech purchases. For one thing, departmental technology budgets seem to point to departmental decision making. Meanwhile, cloud-based solutions, including analytics services, have become more popular with business users because they are easy to set up. In addition, their relatively low subscription rates or pay-per-use models may be more attractive from a budgetary standpoint than their traditional on-premises counterparts, which require significant upfront investments and IT consideration. Since the cost and onboarding barriers to cloud service adoption are generally lower than for on-premises products, IT’s involvement may seem to be unnecessary.

Besides, IT is busy. Enterprise environments are increasingly complex, and IT budgets are not growing proportionally, so the IT department is resource-constrained. Rather than waiting for IT — or complicating decision-making by getting others involved — non-IT tech buyers anxious to deploy a solution may be tempted to act first and answer questions later.

However, making tech purchase without IT’s involvement may result in unforeseen problems. On the following pages, we reveal six risks associated with making business tech purchases without involving IT.

1. Tech Purchases Affect Everybody
Tech purchases made without IT’s involvement may affect IT and the IT ecosystem in ways that someone outside IT couldn’t anticipate. You might be introducing technical risk factors or tapping IT resources IT will have to troubleshoot after the fact. To minimize the potential of unforeseen risks, IT can perform an in-depth assessment of your department’s requirements, the technology options, their trade-offs, and the potential ripple effect that your tech purchase might have across the organization. This kind of risk/benefit analysis is important. Even if it seems like a barrier for your department to get what it wants, it’s better for the entire organization in the long run.
Also, you may need help connecting to data sources, integrating data sources, and ensuring the quality of data, all of which require specific expertise. IT can help you understand the scope of an implementation in greater detail than you might readily see.

2. Sensitive Information May Be Compromised
Information security policies need to be defined, monitored, and enforced. While it’s common for businesses to have security policies in place, education about those policies, and the enforcement of those policies, sometimes fall short. Without appropriate precautions, security leaks can happen innocently, or you could be opening the door to intentional bad actors.
Cloud-based services can expose organizations to risks that users haven’t considered, especially when the service’s terms of use are not understood. Asurvey of 4,140 business and IT managers, conducted in July 2012 by The Ponemon Institute and sponsored by Thales e-Security, revealed that 63% of respondents did not know what cloud providers are doing to protect their sensitive or confidential data.

3. Faulty Data = Erroneous Conclusions
There is no shortage of data to analyze. However, inadequate data quality and access to only a subset of information can negatively impact the accuracy of analytics and, ultimately, decision making.
In an interview with InformationWeek, Jim Sterne, founder of the eMetrics Summit and the Digital Analytics Association, warned that the relative reliability of sources needs to be considered since CRM system data, onsite user behavior data, and social media sentiment analysis data are not equally trustworthy.
“If I’m looking at a dashboard as a senior executive and I know where the data came from and how it was cleansed and blended, I’m looking at the numbers as if they have equal weight,” he said. “It’s like opening up a spice cabinet and assuming each spice is as spicy as any other. I will make bad decisions because I don’t know how the information was derived.”

4. Not Getting What You Bought
Similar products often sound alike, but their actual capabilities can vary greatly. IT can help identify important differences.
While it may be tempting to purchase a product based on its exhaustive feature set or its latest enhancements, feature-based buying often proves to be a mistake because it omits or minimizes strategic thinking. To reduce the risk of buyer’s remorse, consulting with IT can help you assess your current and future requirements and help you choose a solution that aligns with your needs.

5. Scope Creep
Business users typically want immediate benefits from big data, analytics packages, and BI systems. But, if the project has a lot of technological complexity — and particularly if it involves tech dependencies that are outside the control of your department — it’s often best to implement in phases. Approaching large initiatives as one big project may prove to be more complicated, time-consuming, and costly than anticipated.
IT can help you break a large, difficult-to-manage project into several smaller projects, each of which has its own timeline and goals. That way, you can set realistic end-user and C-suite expectations and effectively control risks. Phasing large projects can also provide you with the flexibility you need to adjust your implementation as business requires.

6. Missing Out On Prior Experience
IT professionals and outsourced IT resources often have prior experience with BI and analytics implementations that are specific or relevant to your department. Some of them have implemented solutions in other companies, departments, or industries and have gained valuable insight from those experiences. When armed with such knowledge, they can help you understand potential opportunities, challenges, and pitfalls you may not have considered which can affect planning, implementation, and the choice of solutions.