Lisa Morgan's Official Site

Strategic Insights and Clickworthy Content Development

Category: Data Science (Page 1 of 2)

Why AI is So Brilliant and So Stupid

September 18, 2018 /
AI

For all of the promise that artificial intelligence represents, a successful AI initiative still requires all of the right pieces to come together.

AI capabilities are advancing rapidly, but the results are mixed. While chatbots and digital assistants are improving generally, the results can be laughable, perplexing and perhaps even unsettling.

Google’s recent demonstration of Duplex, its natural language technology that completes tasks over the phone, is noteworthy. Whether you love it or hate it, two things are true: It doesn’t sound like your grandfather’s AI; the use case matters.

One of the striking characteristics of the demo, assuming it actually was a demo and not a fake, as some publications have suggested, is the use of filler language in the digital assistant’s speech such as “um” and uh” that make it sound human. Even more impressive, (again, assuming the demo is real), is the fact that Duplex reasons adeptly on-the-fly despite the ambiguous, if not confusing, responses provided by a restaurant hostess on the other end of the line.

Of course, the use case is narrow. In the demo, Duplex is simply making a hair appointment and attempting to make a restaurant reservation. In the May 8 Google Duplex blog introducing the technology, Yaniv Leviathan, principal engineer and Yossi Matias, VP of Engineering explain: “One of the key research insights was to constrain Duplex to closed domains, which are narrow enough to explore extensively. Duplex can only carry out natural conversations after being deeply trained in such domains. It cannot carry out general conversations.”

A common misconception is that there’s a general AI that works for everything. Just point it at raw data and magic happens.

“You can’t plug in an AI tool and it works [because it requires] so much manual tweaking and training. It’s very far away from being plug-and-play in terms of the human side of things,” said Jeremy Warren, CTO of Vivint Smart Home and former CTO of the U.S. Department of Justice. “The success of these systems is driven by dark arts, expertise and fundamentally on data, and these things do not travel well.”

Data availability and quality matter

AI needs training data to learn and improve. Warren said that if someone has mediocre models, processing performance, and machine learning experts, but the data is amazing, the end solution will be very good. Conversely, if they have the world’s best models, processing performance, and machine learning experts but poor data, the result will not be good.

“It’s all in the data, that’s the number one thing to understand, and the feedback loops on truth,” said Warren. “You need to know in a real way what’s working and not working to do this well.”

Daniel Morris, director of product management at real estate company Keller Williams agrees. He and his team have created Kelle, a virtual assistant designed for Keller Williams’ real estate agents that’s available as iPhone and Android apps. Like Alexa, Kelle has been built as a platform so skills can be added to it. For example, Kelle can check calendars and help facilitate referrals between agents.

“We’re using technology embedded in the devices, but we have to do modifications and manipulations to get things right,” said Morris. “Context and meaning are super important.”

One challenge Morris and his team run into as they add new skills and capabilities is handling longtail queries, such as for lead management, lead nurturing, real estate listings, and Keller Williams’ training events. Agents can also ask Kelle for the definitions of terms that are used in the real estate industry or terms that have specific meaning at Keller Williams.

Expectations are or are not managed well

Part of the problem with technology commercialization, including the commercialization of AI, is the age-old problem of over-promising and under-delivering. Vendors solving different types of problems claim that AI is radically improving everything from drug discovery to fraud prevention, which it can, but the implementations and their results can vary considerably, even among vendors focused on the same problem.

“A lot of the people who are really doing this well have access and control over a lot of first-party data,” said Skipper Seabold, director of decision sciences at decision science advisory firm Civis Analytics. “The second thing to note is it’s a really hard problem. What you need to do to deliver a successful AI product is to deliver a system, because you’re delivering software at the end. You need a cross-functional team that’s a mix of researchers and product people.”

Data scientists are often criticized for doing work that’s too academic in nature. Researchers are paid to test the validity of ideas. However, commercial forms of AI ultimately need to deliver value that either feeds bottom line directly, in terms of revenue, cost savings and ultimately profitability or indirectly, such as through data collection, usage and, potentially, the sale of that information to third parties. Either way, it’s important to set end user expectations appropriately.

“You can’t just train AI on raw data and it just works, that’s where things go wrong,” said Seabold. “In a lot of these projects you see the ability for human interaction. They give you an example of how it can work and say there’s more work to be done, including more field testing.”

Decision-making capabilities vary

Data quality affects AI decision-making. If the data is dirty, the results may be spurious. If it’s biased, that bias will likely be emphasized.

“Sometimes you get bad decisions because there are no ethics,” said Seabold. “Also, the decisions a machine makes may not be the same as a human would make. You may get biased outcomes or outcomes you can’t explain.”

Clearly, it’s important to understand what the cause of the bias is and correct for it.

Understanding machine rendered decisions can be difficult, if not impossible, when a black box is involved. Also, human brains and mechanical brains operate differently. An example of that was the Facebook AI Research Lab chatbots that created their own language, which the human researchers were not able to understand. Not surprisingly, the experiment was shut down.

“This idea of general AI is what captures people’s imaginations, but it’s not what’s going on,” said Seabold. “What’s going on in the industry is solving an engineering problem using calculus and algebra.”

Humans are also necessary. For example, when Vivint Smart Homes wants to train a doorbell camera to recognize humans or a person wearing a delivery uniform, it hires people to review video footage and assign labels to what they see. “Data labelling is sometimes an intensely manual effort, but if you don’t do it right, then whatever problems you have in your training data will show up in your algorithms,” said Vivint’s Warren.

Bottom line

AI outcomes vary greatly based on a number of factors which include their scope, the data upon which they’re built, the techniques used, the expertise of the practitioners and whether expectations of the AI implementation are set appropriately. While progress is coming fast and furiously, the progress does not always transfer well from one use case to another or from one company to another because all things, including the availability and cleanliness of data, are not equal.

Ethical Tech: Myth or Reality?

July 4, 2018 /

New technologies continue to shape society, albeit at an accelerating rate. Decades ago, societal change lagged behind tech innovation by many years, a decade or more. Now, change is occurring much faster as evidenced by the impact of disrupters including Uber and Airbnb.

Central to much of the change is the data being collected, stored and analyzed for various reasons, not all of which are transparent. As the pace of technology innovation and tech-driven societal change accelerate, businesses are wise to think harder about the longer-term impacts of what they’re doing, both good and bad.

Why contemplate ethics?

Technology in all its forms is just a tool that can be used for good or evil. While businesses do not tend to think in those terms, there is some acknowledgement of what is “right” and “wrong.” Doing the right thing tends to be reflected in corporate responsibility programs designed to benefit people, animals, and the environment. Doing the wrong thing often involves irresponsible or inadvertent actions that are harmful to people, whether it’s invading their privacy or exposing their personal data.

While corporate responsibility programs in their current form are “good” on some level, ethics on a societal scale tends to be missing.

In the tech industry, for example, innovators are constantly doing things because they’re possible without considering whether they’re ethical. A blatant recent example is the human-sheep hybrid. Closer to home in high tech are fears about AI gone awry.

Why ethics is a difficult concept

The definition of ethics is simple. According to Merriam Webster, it is “the discipline dealing with what is good and bad and with moral duty and obligation.”

In practical application, particularly in relation to technology, “good” and “bad” coexist. Airbnb is just one example. On one hand, homeowners are able to take advantage of another income stream. However, hotels and motels now face new competition and the residents living next to or near Airbnb properties often face negative quality-of-life impacts.

According to Gartner research, organizations at the beginning stages of a digital strategy rank ethics a Number 7 priority. Organizations establishing a digital strategy rank it Number 5 and organizations that are executing a digital strategy rank it Number 3.

“The [CIOs] who tend to be more enlightened are the ones in regulated environments, such as financial services and public sector, where trust is important,” said Frank Buytendijk, a Gartner research vice president and Gartner fellow.

Today’s organizations tend to approach ethics from a risk avoidance perspective; specifically, for regulatory compliance purposes and to avoid the consequences of operating an unethical business. On the positive side, some view ethics as a competitive differentiator or better yet, the right thing to do.

“Unfortunately, it’s regulatory compliance pressure and risk because of all the scandals you see with AI, big data [and] social media, but hey, I’ll take it,” said Buytendijk. “With big data there was a discussion about privacy but too little, too late. We’re hopeful with robotics and the emergence of AI, as there is active discussion about the ethical use of those technologies, not onlyt by academics, but by the engineers themselves.”

IEEE ethics group emerges

In 2016, the IEEE launched the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. Its goal is to ensure that those involved in the design and development of autonomous and intelligent systems are educated, trained, and empowered to prioritize ethical considerations so that technologies are advanced for the benefit of humanity.

From a business perspective, the idea is to align corporate values with the values of customers.

“Ethics is the new green,” said Raja Chatila, Executive Committee Member of the IEEE Global Initiative on Ethics of Autonomous and Intelligent Systems. “People value their health so they value products that do not endanger their health. People want to buy technology that respects the values they cherish.”

However, the overarching goal is to serve society in a positive way, not just individuals. Examples of that tend to include education, health, employment and safety.

“As an industry, we could do a better job of being responsible for the technology we’re developing,” said Chatila.

At the present time, 13 different committees involved in the initiative are contemplating ethics from different technological perspectives, including personal data and individual access control, ethical research and design, autonomous weapons, classical ethics in AI, and mixed reality. In December 2017, the group released “Ethically Aligned Design volume 2,” a 266-page document available for public comment. It includes the participation of all 13 committees.

In addition, the initiative has proposed 11 IEEE standards, all of which have been accepted. The standards address transparency, data privacy, algorithmic bias, and more. Approximately 250 individuals are now participating in the initiative.

Society must demand ethics for its own good

Groups within society tend to react to technology innovation differently due to generational differences, cultural differences, and other factors. Generally speaking, early adopters tend to be more interested in a new technology’s capabilities than its potential negative effects. Conversely, laggards are more risk averse. Nevertheless, people in general tend to use services, apps, and websites without bothering to read the associated privacy policies. Society is not protecting itself, in other words. Instead, one individual at a time is acquiescing to the collection, storage and use of data about them without understanding to what they are acquiescing.

“I think the practical aspect comes down to transparency and honesty,” said Bill Franks, chief analytics officer at the International Institute for Analytics(IIA). “However, individuals should be aware of what companies are doing with their data when they sign up, because a lot of the analytics –- both the data and analysis –- could be harmful to you if they got into the wrong hands and were misused.”

Right now, the societal impacts of technology tend to be recognized after the fact, rather than contemplated from the beginning. Arguably, not all impacts are necessarily foreseeable, but with the pace of technology innovation constantly accelerating, the innovators themselves need to put more thought into the positive and negative consequences of bringing their technology to market.

Meanwhile, individuals have a responsibility to themselves to become more informed than they are today.

“Until the public actually sees the need for ethics, and demands it, I just don’t know that it would ever necessarily go mainstream,” said Franks. “Why would you put a lot of time and money into following policies that add overhead to manage and maintain when your customers don’t seem to care? That’s the dilemma.”

Businesses, individuals, and groups need to put more thought into the ethics of technology for their own good and for the good of all. More disruptions are coming in the form of machine intelligence, automation, and digital transformation which will impact society somehow. “How” is the question.

The Trouble with Data About Data

June 28, 2018 /

Two people looking at the same analytical result can come to different conclusions. The same goes for the collection of data and its presentation. A couple of experiences underscore how the data about data — even from authoritative sources — may not be as accurate as the people working on the project or the audience believe. You guessed it: Bias can turn a well-meaning, “objective” exercise into a subjective one. In my experience, the most nefarious thing about bias is the lack of awareness or acknowledgement of it.

The Trouble with Research

I can’t speak for all types of research, but I’m very familiar with what happens in the high-tech industry. Some of it involves considerable primary and secondary research, and some of it involves one or the other.

Let’s say we’re doing research about analytics. The scope of our research will include a massive survey of a target audience (because higher numbers seem to indicate statistical significance). The target respondents will be a subset of subscribers to a mailing list or individuals chosen from multiple databases based on pre-defined criteria. Our errors here most likely will include sampling bias (a non-random sample) and selection bias (aka cherry-picking).

The survey respondents will receive a set of questions that someone has to define and structure. That someone may have a personal agenda (confirmation bias), may be privy to an employer’s agenda (funding bias), and/or may choose a subset of the original questions (potentially selection bias).

The survey will be supplemented with interviews of analytics professionals who represent the audience we survey, demographically speaking. However, they will have certain unique attributes — a high profile or they work for a high-profile company (selection bias). We likely won’t be able to use all of what a person says so we’ll omit some stuff — selection bias and confirmation bias combined.

We’ll also do some secondary research that bolsters our position — selection bias and confirmation bias, again.

Then, we’ll combine the results of the survey, the interviews, and the secondary research. Not all of it will be usable because it’s too voluminous, irrelevant, or contradicts our position. Rather than stating any of that as part of the research, we’ll just omit those pieces — selection bias and confirmation bias again. We can also structure the data visualizations in the report so they underscore our points (and misrepresent the data).

Bias is not something that happens to other people. It happens to everyone because it is natural, whether consciously or unconsciously. Rather than dismiss it, it’s prudent to acknowledge the tendency and attempt to identify what types of bias may be involved, why, and rectify them, if possible.

I recently worked on a project for which I did some interviews. Before I began, someone in power said, “This point is [this] and I doubt anyone will say different.” Really? I couldn’t believe my ears. Personally, I find assumptions to be a bad thing because unlike hypotheses, there’s no room for disproof or differing opinions.

Meanwhile, I received a research report. One takeaway was that vendors are failing to deliver “what end customers want most.” The accompanying infographic shows, on average, that 15.5% of end customers want what 59% of vendors don’t provide. The information raised more questions than it answered on several levels, at least for me, and I know I won’t get access to the raw data.

My overarching point is that bias is rampant and burying our heads in the sand only makes matters worse. Ethically speaking, I think as an industry, we need to do more.

 

Why Businesses Must Start Thinking About Voice Interfaces, Now

May 7, 2018 /

Voice interfaces are going to have an immense impact on human-to-machine interaction, eventually replacing keyboards, mice and touch. For one thing, voice interfaces can be much more efficient than computers, laptops, tablets and smartphones. More importantly, they provide an opportunity to develop closer relationships with customers based on a deeper understanding of those customers.

Despite the popularity of Alexa among consumers, one might assume that voice interfaces are aspirational at best for businesses, although a recent Capgemini conversational commerce study tells a different story. The findings indicate that 40% of the 5,000 consumers interviewed would use a voice assistant instead of a mobile app or website. In three years, the active users expect 18% of their total expenditures will take place via a voice assistant, which is a six-fold increase from today. The study also concluded that voice assistants can improve Net Promoter Scores by 19%. Interestingly, this was the first such study by Capgemini.

“Businesses really need to come to grips with voice channels because they will change the customer experience in ways that we haven’t seen since the rise of ecommerce,” Mark Taylor, chief experience officer, DCX Practice, Capgemini. “I think it’s going to [have a bigger impact] than ecommerce because it’s broader. We call it ‘conversational commerce,’ but it’s really voice-activated transactions.”

Voice interfaces need to mimic humans

The obvious problem with voice interfaces is their limited understanding of human speech, which isn’t an easy problem to solve. Their accuracy depends on understanding of the words spoken in context, including the emotions of the speaker.

“We’re reacting in a human way to very robotic experience and as that experience evolves, it will only increase our openness and willingness to experience that kind of interaction,” said Taylor. “Businesses have recognized that they’re going to need a branded presence in voice channels, so some businesses have done a ton of work to learn what that will be.”

For example, brands including Campbell’s Soup, Sephora and Taco Bell are trying to understand how consumers want to interact with them, what kind of tone they have as a brand and what to do with the data they’re collecting.

“Brands have spent billions of dollars over the years representing how they look to their audience,” said Taylor. “Now they’re going to have to represent how they sound. What is the voice of your brand? Is it a female or male voice, a young voice or an older voice? Does it have a humorous or dynamic style? There are lots of great questions that will need to be addressed.”

Don’t approach voice like web or mobile

Web and mobile experiences guide users down a path that is meant to translate human thought into something meaningful, but the experience is artificial. In web and mobile experiences, it’s common to search using keywords or step through a pre-programmed hierarchy. Brands win and lose market share based on the customer experience they provide. The same will be true for voice, but the difference is that voice will enable deeper customer relationships.

Interestingly, in the digital world, voice has lost its appeal. Businesses are replacing expensive call centers with bots. Meanwhile, younger generations are using smartphones for everything but traditional voice phone calls. Voice interfaces will change all of that that, albeit not in the way older generations might expect. In Europe, for example, millennials prefer to use a voice assistant in stores rather than talking to a person, Taylor said.

“What’s interesting here is the new types of use cases [because you can] interact with customers where they are,” said Ken Dodelin, VP of Conversational AI Products at Capital One.

Instead of surfing the web or navigating through a website, users can simply ask a question or issue a command.

“[Amazon’s] dash button was the early version a friction-free thing where someone can extend their finger and press a button to go from thought to action,” said Dodelin. “Alexa is a natural evolution of that.”

In banking, there is considerable friction between wanting money or credit and getting it. Capital One has enabled financial account access via voice on Alexa and Cortana platforms. It is also combining visual and voice access on Echo Show. The reasoning for the latter is because humans communicate information faster by speaking and consume information faster visually.

“[I]t usually boils down to what’s the problem you’re solving and how do you take friction out of things,” said Dodelin. “When I think about what it means for a business, it’s more about how can we [get] good customer and business outcomes from these new experiences.”

When Capital One first started with voice interfaces, customers would ask about the balance on their credit cards, but when they asked about the balance due, the system couldn’t handle it.

“Dialogue management is really important,” said Dodelin. “The other piece is who or what is speaking?”

Brand image is reflected in the characteristics of the voice interface. Capital One didn’t have character development experts, so it hired one from Pixar that now leads the conversational AI design work.

“Natural language processing technology has progressed so much that we can expect it to become an increasingly common channel for customer experience,” said Dodelin. “If they’re not doing it directly through a company’s proprietary voice interface, they’re doing it by proxy through Alexa, Google Home or Siri and soon through our automobiles.”

The move to voice interfaces is going to be a challenge for some brands and an opportunity for others. Now is the time for companies to experiment and if they’re successful, leap ahead of their competitors and perhaps even set a new standard for creating customer experiences.

Clearly, more works needs to be done on natural language processing, but already, some consumers have been tempted to thank Alexa, despite its early-stage capabilities, said Capgemini’s Taylor.

In short, voice interfaces are here and evolving rapidly. What will your brand do?

Why Surveys Should Be Structured Differently

April 4, 2018 /

keyboard-417093_1280If you’re anything like me, you’re often asked to participate in surveys.  Some of them are short and simple.  Others are very long, very complicated, or both.

You may also design and implement surveys from time to time like I do.   If you want some insight into the effectiveness of your survey designs and their outcomes, pay attention to the responses you get.

Notice the Drop-off Points

Complicated surveys that take 15 or 20 minutes to complete tend to reflect drop off points at which the respondents decided that the time investment required wasn’t worth whatever incentive was offered.  After all, not everyone actually cares about survey results or a  1-in-1,000 chance of winning the latest iPad, for example.  If there’s no incentive whatsoever, long and complicated surveys may  be even less successful, even if you’re pinging your own  database.

A magazine publisher recently ran such a survey, and boy, was it hairy.  It started out like similar surveys, asking questions about the respondent’s title, affiliation, company revenue and size.  It also asked about purchasing habits – who approves, who specifies, who recommends, etc. for different kinds of technologies.  Then, what the respondent’s content preferences are for learning about tech (several drop-down menus), using tech (several drop-down menus), purchasing tech (several drop-down menus), and I can’t remember what else.  At that point, one was about 6% done with the survey.  So much for “10 – 15 minutes.”  It took about 10 or 15 minutes just to wade through the first single-digit percent of it.  One would really want a slim chance of winning the incentive to complete that survey.

In short, the quest to learn everything about everything in one very long and complex survey may end in more knowledge about who took the survey than how how people feel about important issues.

On the flip side are very simple surveys that take a minute or two to answer.  Those types of surveys tend to focus on whether a customer is satisfied or dissatisfied with customer service, rather than delving into the details of opinions about several complicated matters.

Survey design is really important.  Complex fishing expeditions can and often do reflect a lack of focus on the survey designer’s part.

Complex Surveys May Skew Results

Overly complicated surveys may also yield spurious results.  For example, let’s say 500 people agree to take a survey we just launched that happens to be very long and very complex.  Not all of the respondents will get past the who-are-you questions because those too are complicated.  Then, as the survey goes on, more people drop, then more.

The result is that  X% of of the survey responses at the end of the survey are not the same as X% earlier in the survey.  What I mean by that is 500 people started, maybe 400 get past the qualification portion, and the numbers continue to fall as yet more complicated questions arise but  the “progress bar” shows little forward movement.  By the end of the survey, far less than 500 have participated, maybe 200  or 100.

Of course, no one outside the survey team knows this, including the people in the company who are presented with the survey results.  They only know that 500 people participated in the survey and X% said this or that.

However, had all 500 people answered all the questions, the results of some of the questions would likely look slightly or considerably different, which may be very important.

Let’s say 150 people completed our  survey and the last question asked whether they planned to purchase an iPhone 7 within the next three months.  40% of them or 60 respondents said yes.  If all 500 survey respondents answered that same question, I can almost guarantee you the answer would not be 40% .  It might be close to 40% or it might not be even close to 40%.

So, if you genuinely care about divining some sort of “truth” from surveys, you need to be mindful about how to define and structure the survey and that the data you see may not be telling you the entire story, or even an accurate story.

The point about accuracy is very important and one that people without some kind of statistical background likely haven’t even considered because they’re viewing all aggregate numbers as having equal weight and equal accuracy.

I, for one, think that survey “best practices” are going to evolve in the coming years with the help of data science.  While the average business person knows little about data science now, in the future it will likely seem cavalier not to consider the quality of the data you’re getting and what you can do to improve the quality of that data.  Your credibility and perhaps your job may depend on it.

In the meantime, try not to shift the burden of thinking entirely to your survey audience because it won’t do either of you much good.  Think about what you want to achieve, structure your questions in a way that gives you insight into your audience and their motivations (avoid leading questions!), and be mindful that not all aggregate answers are equally accurate or representative, even within the same survey.

Your Data Is Biased. Here’s Why.

January 11, 2018 /

Bias is everywhere, including in your data. A little skew here and there may be fine if the ramifications are minimal, but bias can negatively affect your company and its customers if left unchecked, so you should make an effort to understand how, where and why it happens.

“Many [business leaders] trust the technical experts but I would argue that they’re ultimately responsible if one of these models has unexpected results or causes harm to people’s lives in some way,” said Steve Mills, a principal and director of machine intelligence at technology and management consulting firm Booz Allen Hamilton.

In the financial industry, for example, biased data may cause results that offend the Equal Credit Opportunity Act (fair lending). That law, enacted in 1974, prohibits credit discrimination based on race, color, religion, national origin, sex, marital status, age or source of income. While lenders will take steps not to include such data in a loan decision, it may be possible to infer race in some cases using a zip code, for example.

“The best example of [bias in data] is the 2008 crash in which the models were trained on a dataset,” said Shervin Khodabandeh, a partner and managing director of Boston Computing Group (BCG) Los Angeles, a management consulting company. “Everything looked good, but the datasets changed and the models were not able to pick that up, [so] the model collapsed and the financial system collapsed.”

What Causes Bias in Data

A considerable amount of data has been generated by humans, whether it’s the diagnosis of a patient’s condition or the facts associated with an automobile accident.  Quite often, individual biases are evident in the data, so when such data is used for machine learning training purposes, the machine intelligence reflects that bias.  A prime example of that was Microsoft’s infamous AI bot, Tay, which in less than 24 hours adopted the biases of certain Twitter members. The results were a string of shocking, offensive and racist posts.

“There’s a famous case in Broward County, Florida, that showed racial bias,” said Mills. “What appears to have happened is there was historically racial bias in sentencing so when you base a model on that data, bias flows into the model. At times, bias can be extremely hard to detect and it may take as much work as building the original model to tease out whether that bias exists or not.”

What Needs to Happen

Business leaders need to be aware of bias and the unintended consequences biased data may cause.  In the longer-term view, data-related bias is a governance issue that needs to be addressed with the appropriate checks and balances which include awareness, mitigation and a game plan should matters go awry.

“You need a formal process in place, especially when you’re impacting people’s lives,” said Booz Allen Hamilton’s Mills. “If there’s no formal process in place, it’s a really bad situation. Too many times we’ve seen these cases where issues are pointed out, and rather than the original people who did the work stepping up and saying, ‘I see what you’re seeing, let’s talk about this,’ they get very defensive and defend their approach so I think we need to have a much more open dialog on this.”

As a matter of policy, business leaders need to consider which decisions they’re comfortable allowing algorithms to make, the safeguards which ensure the algorithms remain accurate over time, and model transparency, meaning that the reasoning behind an automated decision or recommendation can be explained.  That’s not always possible, but still, business leaders should endeavor to understand the reasoning behind decisions and recommendations.

“The tough part is not knowing where the biases are there and not taking the initiative to do adequate testing to find out if something is wrong,” said Kevin Petrasic, a partner at law firm White & Case.  “If you have a situation where certain results are being kicked out by a program, it’s incumbent on the folks monitoring the programs to do periodic testing to make sure there’s appropriate alignment so there’s not fair lending issues or other issues that could be problematic because of key datasets or the training or the structure of the program.”

Data scientists know how to compensate for bias, but they often have trouble explaining what they did and why they did it, or the output of a model in simple terms. To bridge that gap, BCG’s Khodabandeh uses two models: one that’s used to make decisions and a simpler model that explains the basics in a way that clients can understand.

Drexel University’s online MS in Data Science will set you on the path to success in one of today’s fastest growing fields. Learn how to examine and manipulate data to solve problems by creating machine learning algorithms and emerge from the program work-place ready.

Brought to you by Drexel University

BCG also uses two models to identify and mitigate bias.  One is the original model, the other is used to test extreme scenarios.

“We have models with an opposite hypothesis in mind which forces the model to go to extremes,” said Khodabandeh. “We also force models to go to extremes. That didn’t happen in the 2008 collapse. They did not test extreme scenarios. If they had tested extreme scenarios, there would have been indicators coming in in 2007 and 2008 that would allow the model to realize it needs to adjust itself.”

A smart assumption is that bias is present in data, regardless.  What the bias is, where it stems from, what can be done about it and what the potential outcomes of it may be are all things to ponder.

Conclusion

All organizations have biased data.  The questions are whether the bias can be identified, what effect that bias may have, and what the organization is going to do about it.

To minimize the negative effects of bias, business leaders should make a point of understanding the various types and how they can impact data, analysis and decisions. They should also ensure there’s a formal process in place for identifying and dealing with bias, which is likely best executed as a formal part of data governance.

Finally, the risks associated with data bias vary greatly, depending on the circumstances. While it’s prudent to ponder all the positive things machine learning and AI can do for an organization, business leaders are wise to understand the weaknesses also, one of which is data bias.

3 Cool AI Projects

July 18, 2017 /

AI is all around us, quietly working in the background or interacting with us via a number of different devices. Various industries are using AI for specific reasons such as ensuring that flights arrive on time or irrigating fields better and more economically.

Over time, our interactions with AI are becoming more sophisticated. In fact, in the not-too-distant future we’ll have personal digital assistants that know more about us than we know about ourselves.

For now, there are countless AI projects popping up in commercial, industrial and academic settings. Following are a few examples of projects with an extra cool factor.

Get Credit. Now.

Who among us hasn’t sat in a car dealership, waiting for the finance person to run a credit check and provide us with financing options? We’ve also stood in lines at stores, filling out credit applications, much to the dismay of those standing behind us in line. Experian DataLabs is working to change all that.

Experian created Experian DataLabs to experiment with the help of clients and partners. Located in San Diego, London, and Sao Paulo, Experian DataLabs employ scientists and software engineers, 70% of whom are Ph.Ds. Most of these professionals have backgrounds in machine learning.

“We’re going into the mobile market where we’re pulling together data, mobile, and some analytics work,” said Eric Haller, EVP of Experian’s Global DataLabs. “It’s cutting-edge machine learning which will allow for instant credit on your phone instead of applying for credit at the cash register.”

That goes for getting credit at car dealerships, too. Simply text a code to the car manufacturer and get the credit you need using your smartphone. Experian DataLabs is also combining the idea with Google Home, so you can shop for a car, and when you find one you like, you can ask Google Home for instant credit.

There’s no commercial product available yet, but a pilot will begin this summer.

AI About AI

Vicarious is attempting achieve human-level intelligence in vision, language, and motor control. It is taking advantage of neuroscience to reduce the amount of input machine learning requires to achieve a desired result. At the moment, Vicarious is focusing on mainstream deep learning and computer vision.

It’s concept is compelling to many investors. So far, the company has received $70 million from corporations, venture capitalists and affluent private investors including Ashton Kutcher, Jeff Bezos, and Elon Musk.

On its website, Vicarious wisely points out the downsides of model optimization ad infinitum that results in only incremental improvements. So, instead of trying to beat a state-of-the-art algorithm, Vicarious is to trying to identify and characterize the source of errors.

Draft Better Basketball Players

The Toronto Raptors is working with IBM Watson to identify what skills the team needs and which prospective players can best fill the gap. It is also pre-screening each potential recruits’ personality traits and character.

During the recruiting process, Watson helps select the best players and it also suggests ideal trade scenarios. While prospecting, scouts enter data into a platform to record their observations. The information is later used by Watson to evaluate players.

And, a Lesson in All of This

Vicarious is using unsupervised machine learning. The Toronto Raptors are using supervised learning, but perhaps not exclusively. If you don’t know the difference between the two yet, it’s important to know. Unsupervised learning looks for patterns. Supervised learning is presented with classifications such as these are the characteristics of “good” traits and these are the characteristics of “bad” traits.

Supervised and unsupervised learning are not mutually exclusive since unsupervised learning needs to start somewhere. However, supervised learning is more comfortable to humans with egos and biases because we are used to giving machines a set of rules (programming). It takes a strong ego, curiosity or both to accept that some of the most intriguing findings can come from unsupervised learning because it is not constrained by human biases. For example, we may define the world in terms of red, yellow and blue. Unsupervised learning could point out crimson, vermillion, banana, canary, cobalt, lapis and more.

Why Recommendation Engines Still Aren’t Accurate

March 28, 2017 /

Recommendation engines are deeply embedded in American culture. Anyone who shops online, subscribes to a streaming media service, conducts on online search, or uses social media is encouraged to do something — buy this, click on that, listen to this song, watch that movie. Sometimes the recommendations are accurate. Sometimes they’re not.

For example, Google’s search engine thinks I’m male. Netflix thinks I might enjoy comedies aimed at college-age men. Amazon’s recommendations can be strange if not laughable.

“On some level, these algorithms are amazing, but the types of errors that can be made are foolish,” said Patrick Wolfe, professor of statistics and honorary professor of computer cience at University College London (UCL) and Chair of IEEE Signal Processing Society’s Big Data Special Interest Group. “It probably wouldn’t take a lot of data to teach Google you’re not male.”

Credit: Pixabay

Credit: Pixabay

Context is Everything

Google and Facebook continue to serve up ads for the fringed boots I bought at Macy’s two weeks ago. They’re also recommending the infant and toddler car seats I bought as a gift in the same time frame. Clearly, the recommendations lack appropriate context.

“One of the reasons I think recommendation engines aren’t as accurate as they could be is that much of machine learning is about making predictions — predicting the weather or whether a document is about politics or not,” said Thorsten Joachims, a professor in the Department of Computer Science and in the Department of Information Science at Cornell University. “A recommendation engine has to be smart about the actions it takes.”

Pandora serves up more music that I like than dislike, although my genre preferences, Native American flute music and smooth jazz, represent dramatically smaller universes of choices than classical music or rock and roll. Pandora doesn’t tell me anything about local musicians or the local music scene. Granted, that would be an extremely difficult undertaking given the number of cities that consider their local music scene an integral part of their culture — Chicago, Las Vegas, and Ithaca, New York, for example.

Ithaca College Associate Professor Doug Turnbull and Cornell’s Joachims are tackling that very problem. With the help of their students, they developed MegsRadio, a music app that recommends local musicians and local music events. The app is specific to Ithaca’s local music scene, although others could benefit from the work if they chose to do something similar in another city, Turnbull said.

I tried the app and created a smooth jazz station. The playlist included some of favorite smooth jazz artists and some very talented artists (presumably from Ithaca) with whom I was unfamiliar.

Solving the Problem Pragmatically

Turnbull and Joachims are using counterfactual machine learning to compensate for the fact that they lack a user base large enough to train the algorithms. Counterfactual learning considers the recommendations an algorithm made, users’ historical choices, the accuracy of the recommendations made by the algorithm, and what would happen if a different algorithm were applied in the same circumstance.

“Counterfactual learning is saying that if we don’t have new users but we do have what the old algorithm predicted and the probability of the thing it predicted and given what the user historically picked in the past, would this new algorithm be better?” said Turnbull. “It’s like an alternate parallel universe.”

By comparison, popular search engines, shopping sites, and streaming media companies use A/B testing on a small percentage of users to determine whether one algorithm performs better than another. If the new one performs better than the existing one, the new new algorithm is eventually rolled out to all users. Counterfactual learning enables offline research. Users can be brought in later for validation purposes.

“It’s really making these systems much more like an agent that thinks about what the effects of its actions will be so that it can learn from past mistakes,” said Cornell’s Joachims.

What’s Your Take on Recommendation Engines?

As more analytics become prescriptive, recommendation engines will find their way into more use cases. Are recommendation engines helping or hindering you? What improvements would you like to see?

Common Biases That Skew Analytics

January 28, 2017 /

How do you know if you can trust analytical outcomes? Do you know where the data came from? Is the quality appropriate for the use case? Was the right data used? Have you considered the potential sources and effects of bias?

All of these issues matter, and one of the most insidious of them is bias because the source and effects of the bias aren’t always obvious. Sadly, there are more types of bias than I can cover in this blog, but following are a few common ones.

Selection bias

Vendor research studies are a good example of selection bias because several types of bias may be involved.

Think about it: Whom do they survey? Their customers. What are the questions? The questions are crafted and selected based on their ability to prove a point. If the survey reveals a data point or trend that does not advance the company agenda, that data point or trend will likely be removed.

Data can similarly be cherry-picked for an analysis. Different algorithms and different models can be applied to data, so selection bias can happen there. Finally, when the results are presented to business leaders, some information may be supplemented or withheld, depending on the objective.

This type of bias, when intentional, is commonly used to persuade or deceive. Not surprisingly, it can also undermine trust. What’s less obvious is that selection bias sometimes occurs unintentionally.

Confirmation bias

A sound analysis starts with a hypothesis, but never mind that. I want the data to prove I’m right.

Let’s say I’m convinced that bots are going to replace doctors in the next 10 years. I’ve gathered lots of research that demonstrates the inefficiencies of doctors and the healthcare system. I have testimonials from several futurists and technology leaders. Not enough? Fine. I’ll torture as much data as necessary until I can prove my point.

As you can see, selection bias and confirmation bias go hand-in-hand.

Outliers

Outliers are values that deviate significantly from the norm. When they’re included in an analysis, the analysis tends to be skewed.

People who don’t understand statistics are probably more likely to include outliers in their analysis because they don’t understand their effect. For example, to get an average value, just add up all the values and divide by the sum of the individuals being analyzed (whether that’s people, products sold, or whatever). And voila! End of story. Except it isn’t…

What if 9 people spent $100 at your store in a year, and the10th spent $10,000? You could say that your average customer spend per year is $1,090. According to simple math, the calculation is correct. However, it would likely be unwise to use that number for financial forecasting purposes.

Outliers aren’t “bad” per se, since they are critical for such use cases as cybersecurity and fraud prevention, for example. You just have to be careful about the effect outliers may have on your analysis. If you blindly remove outliers from a dataset without understanding them, you may miss an important indicator or the beginning of an important trend such as an equipment failure or a disease outbreak.

Simpson’s Paradox

Simpson’s Paradox drives another important point home: validate your analysis. When Simpson’s Paradox occurs, trends at one level of aggregation may reverse themselves at different levels of aggregation. Stated another way, datasets may tell one story, but when you combine them, they may tell the opposite story.

A famous example is a lawsuit that was filed against the University of California at Berkeley. At the aggregate level, one could “prove” more men were accepted than women. The reverse proved true in some cases at the departmental level.

Misunderstood? Try Data Storytelling

December 29, 2016 /

Data visualizations help explain complex data, although individuals can and do come to different conclusions nevertheless. It’s a significant problem for data scientists and data analysts, especially when they’re trying to explain something important to business people.

Part of the problem is one’s ability to communicate. Another problem is expecting too much from data visualizations — specifically, the clear communication of an analytical result.

Data storytelling can help, because it goes beyond data visualizations. It also helps individuals think a bit harder about what the data is saying and why.

Are data visualizations dead?

Clearly not. They remain an extremely important part of turning data into insights, but they do have their limitations. The first limitation is that data visualizations don’t always explain the details of what the data is saying and why. Another limitation, as I mentioned earlier, is the possibility of diverse interpretations and therefore diverse conclusions, which, in a business context, can lead to some rather heated and unpleasant debates.

A simple form of data storytelling is adding text to data visualizations to promote a common understanding. Like PowerPoint, however, it’s entirely possible to add so much text or so many bullets to a data visualization that the outcome is even more confusing than it was without the “improvement.”

The same observation goes for infographics. Bright colors, geometric shapes, and “bleeds” (the absence of a border) do little to aid communication when used ineffectively. It’s important to avoid clutter if you want others to understand an important point quickly.

One complaint I hear about using data visualizations alone is that they lack context. Data storytelling helps provide that context.

How to tell a good data story

Humans tend to be storytellers naturally, whether they’re explaining how a car accident happened or why they weren’t home at 7:00, again. However, when it comes to telling data stories, it’s easy to forget what an effective story entails.

An effective story has a beginning, a middle, and an end like a book or a movie. A data story should have those elements, but beware of telling linear stories that are passively consumed. Interactive stories tend to be more effective in this day and age because people have become accustomed to interacting with data at work and at home. In addition, work styles have become more collaborative over time. Allowing audience members to do some of their own exploration enables them ask more informed, if not challenging, questions. In addition, unlike storytelling generally, data story endings tend not to be definite (e.g., “… triumphant at last, they rode off into the sunset”) but rather possibilities.

Data stories are also vulnerable to the same kinds of flaws that detract from blogs, articles, presentations, and books: typos. Make sure to proof your work. Otherwise, you may lose credibility. Also avoid jargon, not only because it’s considered a bad practice, but because it may confuse at least part of the audience, which brings me to another important point: consider the audience,

Data scientists often are criticized for failing to understand their audiences — namely, business people. It’s fine to talk about linear regressions and sample variances among people who understand what they are, how they work, and why they’re important. A business person’s concern is business impact, not particular forms of statistics.

While you’re at it, be careful about language use generally. The word, “product” can mean different things to different people who work at the same company. Bottom line, it’s all about making it easier for other people to understand what you intend to communicate.

« Older posts

© 2024 Lisa Morgan's Official Site

Theme by Anders Noren — Up ↑