Thursday, February 28, 2013

Early Indications February 2013: Big Caveats Regarding Big Data

Review essay

Michael Mauboussin, The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing (Boston: Harvard Business Review Press, 2012)

Nate Silver, The Signal and the Noise: Why So Many Predictions Fail — but Some Don't (New York: Penguin, 2012)

Nassim Nicholas Taleb, Antifragile: Things That Gain from Disorder (New York: Random House, 2012)

Preface
To set context, here is a sampling of what IT vendors are saying about Big Data:

SAP:
Get the tools and technology you need to harness big data from any source – structured or unstructured – for a serious competitive advantage. Our big data solutions can help you capture, analyze, report, predict, and visualize mammoth volumes of data instantly – so you can make the best possible business decision, every time.

IBM:
Big data is more than simply a matter of size; it is an opportunity to find insights in new and emerging types of data and content, to make your business more agile, and to answer questions that were previously considered beyond your reach.

SAS:
The hopeful vision for big data is that organizations will be able to harness relevant data and use it to make the best decisions.
Technologies today not only support the collection and storage of large amounts of data, they provide the ability to understand and take advantage of its full value, which helps organizations run more efficiently and profitably.

Oracle:
For decades, companies have been making business decisions based on transactional data stored in relational databases. Beyond that critical data, however, is a potential treasure trove of less structured data: weblogs, social media, email, sensors, and photographs that can be mined for useful information.
Oracle offers the broadest and most integrated portfolio of products to help you acquire and organize these diverse data sources and analyze them alongside your existing data to find new insights and capitalize on hidden relationships.
--------
In sum, the vision of the Big Data movement appears to be to as follows:

to measure and capture, in greater detail and quantity, things that have happened in order to analyze the data, find insights/answer hard questions/capitalize on hidden relationships, and act more effectively in the future ("make better decisions.")

It all sounds reasonable, except the foundational logic has yet to be tested. Fortunately, we have some very smart people from diverse backgrounds who can help in that quest. It turns out that if these three gentlemen are correct, the very premises of Big Data need to be tempered, not with better computer science, but a better comprehension both of how people think, act, and decide and of how much luck and randomness still shape our world.

The three books all overlap to a degree, often in their appreciation for the behavioral economics of Daniel Kahneman, and each author brings serious credentials to the table:

-Mauboussin teaches at Columbia in addition to working at Legg Mason; he wrote an early and influential report on the financial implications of power laws back in the late 1990s.

-Silver gained fame on election night 2012 after correctly calling 50 out of 50 state results in the presidential race, after going 49 for 50 in 2008. His first data-centric venture was in baseball statistics.

-Taleb's previous books, The Black Swan and Fooled by Randomness, provided prescient color commentary to the financial crisis of 2008. Stylistically, existentially, and intellectually, he swims upstream but has repeatedly been proven right.

Three macro-level insights emerged from these books.

A) Luck remains a critically important factor in success, so prediction, even when successful (that is, skillful), may not generate much advantage

Mauboussin looks at the relationship of luck and skill in a variety of domains. The book owes many debts to Moneyball, but across more sports and extending convincingly into business. Results in the NBA, for example, are decided by skill to a much higher degree than in the NHL: in 2-1 or 1-0 games on ice, the slightest deflection or fluke play can determine a game. When the Spurs beat the Suns 104-98, however, random chance events are fewer (how many deflected shots actually go through the hoop?) and their impact is minimal.

When he turns to business and investing, Mauboussin makes similarly compelling points. For our purposes, the central insight relevant to Big Data concerns what might be called the water level: as the skill level rises in a population, differences between competitors shrink. Thus luck becomes more of a factor: "if stocks are priced efficiently in the market, luck will determine whether an investor correctly anticipates the next price move up or down. When everyone in business, sports, and investing copies the best practices of others, luck plays a greater role in how they all do." (p. 56)

This insight would seem to apply to the algorithmic arms races in baseball talent scouting, investing, and consumer data mining. In situations where no actor can accumulate a commanding lead (as Google has and Facebook might, however), whether in computing horsepower, algorithmic quality, or data to be analyzed, the skill premium dissipates. Luck, by this theory, will play a greater role than skill in a more heterogeneous environment.

Mauboussin concludes the book with 10 suggestions to improve the "art of good guesswork":

1 Understand where you are on the luck-skill continuum

2 Assess sample size, significance, and swans

3 Always consider a null hypothesis

4 Think carefully about feedback and rewards [many financial advisors get paid when clients trade, not when clients prosper, for example: what's the feedback loop there?]

5 Make use of counterfactuals

6 Develop aids to guide and improve your skill [checklists are a case in point]

7 Have a plan for strategic interactions [such as asymmetric warfare or disruptive innovations]

8 Make reversion to the mean work for you

9 Develop useful statistics

10 Know your limitations

This tenth maxim serves as a convenient segue to Silver's book.  From its title -- signals and noise are fundamental to information theory -- to its examples (which include economics, earthquakes, and climate change), the book would appear to be enthusiastic about using numbers to predict the future, to realize the promise of Big Data. But as Silver writes very early in the book, his focus is less on data and more on the people who use it:

"Big Data _will_ produce progress -- eventually. How quickly it does, and whether we regress in the meantime, will depend on us. . . .
Our biological instincts are not always very well adapted to the information-rich modern world. Unless we work _actively_ to become aware of the biases we introduce, the returns to additional information may be minimal -- or diminishing." (pp. 12-13)

Thus, the second macro-level idea concerns consciously testing ideas, assumptions, and admitting uncertainty.

B) Bayesian statistics, in particular its insistence on carefully articulated prior probabilities, forces human analysts to attach values to the context for their predictions rather than let them float ahistorically, otherwise known as "letting the data speak for itself."

This illusion of statistical sufficiency known sometimes as "frequentism" dates to the early 20th century, and the school of thought persists today. As Silver summarizes, "it emphasizes the objective purity of the experiment -- every hypothesis could be tested to a perfect conclusion if only enough data were collected. However, to achieve that purity, it denies the need for Bayesian priors or any other sort of messy real-world context." (p. 255)

Echoing his opening assertions in the conclusion, Silver plausibly argues that "distinguishing the signal from the noise requires both scientific knowledge and self-knowledge: the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference." (p. 453)

Possibly because Silver's book ranges more widely than does Mauboussin, it felt more engaging. Written as it was before his successful handicapping of the Obama re-election, The Signal and the Noise is itself something of a prior: a self-aware assessment of Silver’s own methods and their probabilistic limits. The book forces erstwhile predictors to examine their methods, their objectives, and ultimately themselves -- not at all what the two-dimensional stat-geek stereotype would suggest.

In contrast to Mauboussin, Silver offers but two admonitions in his conclusion:

Know Where You're Coming From

and

Think Probabilistically.

In contrast to closed-end events -- when will be the first snowfall, who will win the championship, how many widgets will Samsung sell -- open-end events are the terrain of Nassim Nicholas Taleb: Black Swans, as they have come to be called. As Silver notes, nobody can remotely predict earthquakes or most natural phenomena, with the exception of weather. Nor can political revolts (in either London or Cairo, for example), equity or currency fluctuations, or other large-scale man-made phenomena be forecast at all reliably. Rather than predicting, Taleb advocates an entirely different approach.

C) Because of the nature of a highly complex and connected world, "Black Swan" events can generate very large, unforeseen effects, very quickly. A prudent strategy for living in such a world is to seek shelter to a substantial (but not complete) degree, while finding exposure to the upside of unforeseeable events with small bets in as many big-multiplier arenas as possible, often via optionality. Taleb calls this a "dumbbell" strategy for its bimodal distribution: for example, very large positions in cash or other low-risk and low-reward instruments, with focused but small investment in high-risk/very high-reward (and thus probably exotic) positions. Note that the middle is avoided entirely: Taleb's antipathy for bell curve distributions, especially where misapplied, is vehement.

The title of Taleb's book hints at how unaccustomed we are to thinking this way. Everyone knows that a wine glass is fragile: physical volatility is usually fatal. Note that fragility scales non-linearly: a fall from 32 inches onto the hardwood floor is far more than four times as damaging as an 8-inch drop, which is likely survivable. Many people say that "robust" is the antithesis of fragile, but Taleb disputes this position: what, instead, are the opposite of fragile phenomena, the things that actively IMPROVE in the presence of volatility? He looked in dozens of languages: none had a word to connote this property, which is, nonetheless, quite real. Taleb's contrarian-ness is of a high order indeed.

It turns out that the natural world, biology in particular, abounds in situations where volatility improves matters. Young children learning language, muscles after exercise, and immune defenses all qualify. In the human order, Taleb praises the Swiss city-state (canton), noting that many people can pick Switzerland as the most stable regime on earth and yet nobody outside the country knows who the president is: decentralized authority keeps the scale of both problems and solutions closer to human-friendly and risk-limited. Swiss disorder occurs in domains the exact opposite of "too big to fail," itself a curse in this system of thinking because increasing scale implies massive risk. Man-made "stabilization" often leads to instability, whether in financial systems, forest fires (preventing healthy little ones means a later, inevitable inferno), or corporate planning. When small, routine failures are prevented through naive bureaucratic intervention, stressors magnify until the impact is multiplied to the scale of the entire system (as witness the mortgage banking mess, rogue traders at Societe Generale and JPMorgan, and the flash crash). And on the basis of what empirical evidence is "equilibrium" the economist's ideal?

Thus rather than fail to predict the mechanism of [by-definition] unpredictable disaster, we can see the quite foreseeable effects of 100-year-old subway tunnels in New York (whether the stressor is a riot, a terrorist, or a hurricane is irrelevant), or slow responses to climate change, or overly long supply chains for food. In short, Taleb proves that prediction is systematically broken for both psychological -- yes, Kahneman gets his props here too -- and systematic/organizational reasons. The 425-page excursion into many nooks and crannies of the Western intellectual tradition (Seneca plays a featured role, for example) is itself unpredictable: Taleb does not so much explicate his argument as embody it, with frequent personal examinations that prove he literally has skin in the game. His conclusion is much more straightforward that its telling:

"Everything gains or loses from volatility. Fragility is what loses from volatility and uncertainty." (p. 421)

Rather than seek certainty in data* or in anything else, Taleb seeks to find situations, investments, and modes of living that are not only resistant to volatility but _thrive_ in its inevitable presence. The notion of antifragility thus stands as the most robust challenge to the uncritical application of data, algorithms, and prediction more generally -- especially outside realms (such as weather) where we can actually document a certain degree of success. As for lavish investments in police and fire departments for cities, in R&D at the corporate level, and in universities in any particular society, Taleb contends that we really know little about correlation vs. causation. This fundamental lack of evidence suggests that for data to improve our world, there are more intermediate steps between the computer and a better future than the apparent consensus would suggest.


*"There is a nasty phenomenon called 'Big Data' in which researchers bring cherry-picking to an industrial level. Modernity provides too many variables (but to little data per variable), and the spurious relationships grow much, much faster than real information, as noise is convex and information is concave." (p. 418) More simply, "the more data you get, the less you know what's going on." (p, 128)

Thursday, January 31, 2013

Early Indications January 2013: A Plethora of Predictions

Longtime readers know that I have in some years written prediction newsletters, then followed up 11 months later to grade the list. This year, we'll look at several prediction lists from other people to see what commonalities, divergences, agendas, and insights might emerge. And then I'll offer a modest prediction list of my own.

Assembling predictions was extremely easy, so I settled on a random sample of six lists, ranging in size from five items to about a dozen. (Why so many people feel the need to predict is probably worth puzzling over.) Herewith are the lists:

1) Time

Augmented reality will go mainstream
Google’s Chromebooks will get more consumer attention, gaining traction in 2013
Hybrids and convertibles will see high interest from IT departments
7-in. tablets will dominate tablet sales
Apple will create a hybrid tablet-laptop
Samsung will invest in its own future [in the form of a mobile OS]
Microsoft will get into smart-phone hardware
Apple will make a large investment in its supply chain
Google will go fully vertical with Motorola
RIM will make a modest rebound but will eventually be acquired

2) Deloitte
I quote:

Disruptors are opportunities that can create sustainable positive disruption in IT capabilities, business operations, and sometimes even business models.

    CIO as the Postdigital Catalyst
    Mobile Only (and beyond)
    Social Reengineering by Design
    Design as a Discipline
    IPv6 (and this time we mean it)

Enablers are technologies in which many CIOs have already invested time and effort, but which warrant another look because of new developments or opportunities.

    Finding the Face of Your Data
    Gamification Goes to Work
    Reinventing the ERP Engine
    No Such Thing as Hacker-proof
    The Business of IT

3) Shel Israel in Forbes

Apple's edge will dull
The year of wearables
They'll know who we are [mobile + social + big data + sensors + location = predictive context]
Buzzwords will die
Data contracts and margins will erode [for telcos]

4) The Telegraph
Surprises are guaranteed in:

Mobile
Microsoft
TV
Security
Facebook monetization
Wearables

5) ZDNet

3D printing
Augmented reality
[Apple] TV
Anonymous/Wikileaks revival
Big data gets bigger
BYOD soars at work
Tech giants go BRIC
Rim and Nokia declare themselves
Apple faces testing times
Internet of things
T-Mobile+MetroPCS, Sprint+Clearwire
Cloud computing growth meets legal tangles
Yahoo bets it all on mobile?

6) Gartner

Mobile devices battle for market share as mobile > PC
Mobile applications & HTML 5 [fight it out]   
Personal Cloud means services > devices
Internet of Things [esp. embedded sensors, image recognition, and NFC]
Hybrid IT and Cloud Computing
Strategic Big Data
Actionable Analytics
Mainstream In-Memory Computing
Integrated Ecosystems   
Enterprise App Stores

Commonalities

The PC is the platform of the past -- except possibly as a tablet hybrid. Note how every list assumes the primacy of the mobile device. Even so, the PC industry remains big, if highly contested and low-margin.

The TV looks like the next battlefield.

Even when it's not mentioned, cloud computing is assumed in the need for powerful mobile capabilities. A trend is truly powerful when people stop talking about it and just use it: NFC fails on this score; maps/location are huge by this metric.

Several companies face make-or-break (or at least win-or-give up the lead) moments in their corporate history: Apple is no longer "cool" or singularly cool. Microsoft must prove it can win off the PC desktop. Nokia and RIM may both be gone by year end. Google has many irons in the fire but neither Chromebooks nor Android nor Glass nor maps nor YouTube has proven to make serious money. Yahoo can't have many more lives (or CEOs) left to play out if it doesn't crack the mobile code.

Google's wearable/augmented reality play successfully drove very strong PR -- just as search used to do with journalists 15 years ago.

Divergences

Other than differences in focus -- consumer gadgets vs CIOs -- there wasn't a lot of overt disagreement among the lists.

ZDNet is the only list that mentions 3D printing, which, while not the second coming of Watt's steam engine, is going to make a difference in the long run. Possibly much like GPS or WiFi, there won't be a breakthrough moment but rather one day we will simply notice that it's extremely widespread and wonder why there wasn't much fuss.

I was surprised there wasn't more emphasis on security. Deloitte's tagline, "no such thing as hacker-proof," gets at the conundrum of BYOD, mobile access, heterogeneous computing and networking environments, effectively infinite horsepower for bad guys to crack passwords, substantial investments by nation-states in cyber-disruption (or worse), etc.

Agendas

Just a word here on what might drive these lists. ZDNet presented theirs as a slideshow, which does wonders for page view counts even when the photos did nothing to illustrate mangy of the abstract concepts: how many more data center server racks must we see to illustrate Big Data? Shel Israel wants to sell books, and his list reflects his current research themes: fair enough. As for Deloitte, the firm clearly needs to sell consulting services to CIOs, but the list was both obtuse (what is a "post digital catalyst"?) and partially wrong: I would wager serious money that enterprise gamification will only be marginally more prevalent a year from now, if that.

Insights

The business landscape is extremely volatile. "Built to last" is getting harder to find, particularly in information industries. Leaders in one quarter can lose their advantage very quickly, and stock market responses can be irrational: Amazon missed its numbers pretty badly and its stock prize rose while the fall in Apple's share price is hard to square with objective performance. In that context of volatility, speed, and unpredictability, here are my seven predictions for 2013:

1) Value chain realignment
Whether at Amazon, Apple, Comcast, Dell, Google/Motorola, HP, Microsoft, or Samsung, the mobile-first world is forcing hardware, software, services, content, and carriage to align in new ways. Each of these players is experimenting with extensions and reconfigurations of its original core value proposition. We will see more.

2) Cyber insecurity
The latest James Bond move had strands of Stuxnet, Wikileaks, Anonymous, and the Internet of Things effectively woven into a plausible tapestry of cyber-threat. We will see new unpleasant and potentially dangerous effects of global interconnectedness and reliance on digital records, personas, and entertainment.

3) CIO speciation
For all the talk about IT as a tool for competitive advantage over the past 20 years, for every Amazon there are dozens of companies that use computers to schedule production, ship orders, and pay bills. Rather than insist -- annually -- that the latter will grow to resemble the former, I think it's realistic to see that there are several kinds of enterprise computing scenarios. Just as tigers can't change their stripes, most keep-the-lights-on IT shops will never become sources of strategic advantage.

4) Computing in three dimensions
The Internet of Things, GPS, sensors, robotics, and mass mobility mean that computing power is being applied to physical space as much as to numbers and symbols on a screen. Augmented reality, self-driving (and software-augmented) cars, robots (including drones), and other developments point to a future in which computing leaves the beige box.

5) People and data are two sides of the same coin
Nate Silver's excellent book The Signal and the Noise (more on that next month) raises a fundamental point: numbers do not speak for themselves. People assemble, parse, and interpret data in many ways, few of which are currently very self-aware. Numeracy is one big challenge, especially given U.S. high school math teaching, but so is a more nuanced view of the relationship between subjectivity and objectivity, the observer and the observed.

6) Mobile implies social, social implies mobile 
Investors and tech strategists underestimate Facebook at their peril. The connection of people to people in a mobile world is changing profoundly, not all for the better I don't think, but Facebook's massive presence at the nexus of computing, communication, and social signaling gives the company terrific leverage. This stature could change quickly -- Apple's relative loss of cool was inevitable, with significant effects, yet still sudden -- but until proven otherwise, Facebook remains a force to be reckoned with. When Mark Zuckerberg said yesterday that Facebook is NOT building a phone, he merely highlighted the complex set of issues and opportunities at the juncture of people, their tools, and their emotional needs.

7) It's a big world after all
Globalization in the context of computing and communications is playing out in fascinating ways. Japan, Inc is largely absent from the leadership: Panasonic, Sharp, and Sony are losing prodigious amounts of money. Africa is teaching the world about mobility as it applies to banking, social networks, and ad hoc resiliency. The Arab Spring story continues to be written, too often in blood as well as in texts and tweets. Finally, apart from Samsung, name a non-US company of strategic significance: every other tech firm in India, China, and Europe is jostling for position in markets heading toward commoditization. Expect to see even clearer separation of the giants (Apple, Google, Samsung, maybe Microsoft) from the rest of the corporate pack even as local habits and innovations propagate.

As in the past I will self-grade these predictions after the year plays out.

Saturday, December 01, 2012

Early Indications November 2012: How will TV be disrupted?

No sector has been more transformed by the internet than audience aggregators. These content companies, through various means, make money from assembling viewers, readers, and listeners for artifacts that in the past 20 years have increasingly become digital rather than analog. Whether it is Kindle with books, iTunes for music, Google news vs. newspapers, or Netflix for movies, Internet-based content has disrupted whole sectors, eliminating such brands as Borders, Newsweek, Tower Records, Blockbuster, and Virgin Megastores.

The pattern seems to be based on relative simultaneity: Columbia sold more than 100 million copies of Michael Jackson's Thriller LP/CD, but that audience doesn't all listen at the same time. The New York Times company moves many copies of its flagship newspaper, and while readers will pick it up to read their selected articles during the day it's distributed, again, the audience is asynchronous. Harry Potter books are even more distributed in time than printed news. Television, by contrast, assembles mass audiences at a particular time: more than 100 million US viewers saw the 2011 Super Bowl, the vast majority of them in real time. But: every content bundling model, from the record album to financial advising/stock purchasing to the newspaper, has come under attack in the Internet era. (University degrees might be next, a topic for another time.)

A number of signals are suggesting that TV's time of reckoning is coming. Tablet sales are soaring: 20% of US adults now own the devices only two years after launch, and the intimacy of the tablet form changes viewing habits: social interaction, for example, is facilitated in ways that traditional TV does not allow. Time shifting, whether through Hulu, BitTorrent, or DVR, is becoming predominant among those in their 20s. Games, online shopping, social networking, and other new tasks threaten to shrink the staggering five hours per day the average American spends watching TV, if Nielsen's numbers are to be believed. Bottom line: as the Wall Street Journal put it on November 30,

"Television viewership is declining across the board. Although CBS remains the most-watched network in prime time, its average overall audience of 11.5 million in that time period is down 10% in the fall season so far, compared with the same time the year before, according to Nielsen. Its audience among 18-to-49-year-olds, the demographic most prized by advertisers, has tumbled 20%."

Two questions thus emerge: what will Internet video look like in terms of markets, business models, and financial attractiveness, and how will today's incumbents respond?

I use the term "Internet video" deliberately: just as all sparrows are birds but not all birds are sparrows, television is just one of many categories of Internet video. Without the constraints of 30-minute multiples, we are seeing a proliferation of new content forms. TED talks, for example, have been viewed about a billion times, and many of those are presumably in front of room-size audiences. YouTube served eight million simultaneous streams of Felix Baumgartner's Red Bull space jump, something no earthly TV network could have done, given the global composition of the audience. Jerry Seinfeld's Web series, "Comedians in Cars Getting Coffee," focuses on exactly that scenario, but episodes are only as long as they are: 7 minutes, 13 minutes, whatever. TV ads, particularly global ones pulled out of context to fascinating effect, have long been prime YouTube material.

Going forward, every aspect of the video experience may be contested. Apple TV, whatever it turns out to be, should improve on the remote by substituting a tablet/smartphone for the dumb infrared devices we all currently use. As for the viewing device, a big smartphone is only slightly smaller than a small tablet. As the fates of Sharp, Panasonic, and Sony illustrate, global consumers aren't as infatuated with 3D television and other innovations as they are with these smaller devices: Panasonic lost $9 billion -- $9 billion -- in its last quarter. Transport, meanwhile, is "obviously" over cable, but as mobile bandwidth gets faster and faster (is 4G as fast as we can go? There's no reason to believe so), that wire infrastructure may no longer be a monopoly. Finally, content creation used to be the private province of a small number of studios, but barriers to entry have dropped so literally anyone can produce viewable material, which then of course costs precisely zero to distribute.

In the current model for TV, big audiences equate to big ad revenues and big salaries: last season Ashton Kutcher earned $24 million for his role on "Two and a Half Men." Apart from sports, however, "big" is smaller than it used to be. Because of channel proliferation, audience fragmentation, and other factors, the top-rated non-sports shows currently running -- "60 Minutes" and "NCIS" -- earned only an 8.0 and 10.5 rating respectively (depending on week), meaning 8-11% of households are watching. Compare today's picture to that of past decades, when "I Love Lucy" pulled numbers in the 60s; the farewell episode of M*A*S*H in 1983 also drew 60.2% of households.

On one hand, the fragmentation of cable that spawns such critical successes as "Mad Men," "The Sopranos," "Breaking Bad," and "Homeland" is extending into web video. That is, quality work with niche audiences might be better addressed via YouTube or other distribution models, particularly because interactive advertising can be measured so much more precisely than broadcast or even cable numbers. At the same time, the networks' cost structure has already been reset, beginning about 15 years ago: reality and other non-scripted programming proved to be a convenient way around the Hollywood writers' strike in 2007 and also kept talent costs to a minimum. Even today, such shows as American Idol, X Factor, Survivor, and Dancing with the Stars score well while being relatively cheap to produce.

The energy drink/adventure sports sector has discovered this fact of online behavior and exploited it aggressively. Ken Block's driving stunts, the aforementioned Red Bull oeuvre (including non-ad programming such as "Jackson's Hole"), and Monster (with more than 34 million views) offer programming that would be ill-suited for even niche cable channels but perfectly situated for viral distribution, repeated viewing, and powerful branding opportunities.

Because of the sheer volume of material -- 72 hours of YouTube video are uploaded every MINUTE -- viewers confront a classic long-tail scenario: every niche of interest, taste, culture, and video quality is addressed, including some that haven't been invented yet. Cable television solves this problem with brand identity: viewers who tune in to HGTV, Food Network, or Fox Soccer Channel have a pretty good idea of what they'll see. While YouTube, TED, and other outlets have adopted some variation of the "channel" philosophy, it doesn't scale. Neither does search work very well: if you didn't know what Gangnam Style was, how on earth could you phrase the search to find that video?  Social networking is, to date, the best option for finding this stuff, raising the prospect of more formal varieties of trusted filters who watch thousands of hours of bad video to snare the treasures.

To give just one example of such valuable artifacts, there are a multitude of black-and-white clips of musical, stand-up comedy, and other performances from the 1950s and '60s. Seeing Thelonius Monk on Swedish television can be wonderful, surreal, and dazzling all at the same time. Again, television, even of the cable variety, is a poor distribution mechanism for such work, but the planet's artistic inheritance is far richer than it was just a decade ago: it's one thing to dust off a Wes Montgomery LP and play it for the budding musician and something else entirely for her to see the genius himself, often explaining the music between takes. There are doubtless thousands of other examples (university lectures are one - reading Richard Feynman and seeing him are two very different experiences).

How will the incumbents respond to these and the many other changes ahead? Hulu is trying to impose cable economics on web video, and it might be able to do so for a few years. As cable subscription prices rise relentlessly, however, cord-cutting will increase, I predict. The fact that cable operators control the majority of high-speed home connections complicates this switchover somewhat, but as mentioned above, wireless broadband and tablets could lead to a different economic model in the future. At what point could AT&T/Verizon supplant Comcast?

Sports, particularly football, appear to be immune to time-shifting, so much so that multi-billion dollar bets are being made on this assumption. ESPN paid $1.1 billion in 2010 for 18 games of Monday Night Football; the rate nearly doubles, to $1.9 billion, only three years later. The Big Ten cable network (half owned by Fox) generated $7 million per school in 2009; Maryland joined the conference last month in part because the school was reportedly promised $43 million (along with all other members) in 2017.

Big Ten commissioner Jim Delany is reported to be a student of demographics, and the athletic conference's core states in the Rust Belt are indeed losing population to the American West and South. But Delany is, I believe, falling victim to linear projections of cable access fees paid by households that, in the main, are not seeing big jumps in income. As the NFL, college football, the Olympics, the World Cup, and other sports bodies continue to rely on heavier funding from their television partners, this technological sea change threatens to undermine that rising tide of non-advertising revenue. When might the golden goose stop laying so many eggs? Delany's math is reminiscent of that of the financial industry in 2005-6, when it was believed that housing prices could never go down.

What might happen next? Pay-as-you-go for cable channels may become a reality at some point, the global audiences for web video could become a factor in ways broadcast cannot reach, and new advertising technologies could alter the landscape. Might brand loyalists get more, better ads, and subsidized cable subscriptions, for example? Will overlays, quizzes, games, and even biological sensors augment the 30-second spot?

Who should we be watching for signs of business model change? Clearly the current rights holders are not standing still, and ventures such as Hulu will evolve. Given Apple's past relationships with Hollywood content companies, iTunes could turn into a new kind of cable TV experience.  Startups such as GetGlue, Showyou, Vimeo, and Yahoo's IntoNow provide a variety of social layers for video production, distribution, and consumption. Amazon has lots of digital assets, a hardware platform in the Kindle Fire, and a history of surprise moves.

Whatever its future, broadcast TV had a good run, and will of course deliver value in certain circumstances in the future. But the monopoly that television has had over video audience aggregation for the past 70 years is being broken. As a result, the possibilities for the future are both exciting (for content producers and consumers) and potentially expensive for incumbents. In any case, creativity should flourish as it has in every other technology revolution that empowered artists, whether printing, paint chemistry, or photography.

Thursday, November 01, 2012

Early Indications October 2012: The State of Mobile Payment (U.S. market)

With so many powerful players getting involved, some more standing on the sidelines, and a few wild-card startups, the smartphone-as-wallet market is truly compelling business theater. Far more is unknown than certain right now, so rather than speculate, I'll lay out some of the knowns then concentrate on the relevant questions that must be sorted out.

What we know

Multiple projections suggest that consumers will use smartphones as wallets more and more frequently in the next five years. In addition, Japan and Kenya, among other markets, provide useful precedents for mobile payment behavior. Even so, U.S. adoption is likely to be unique for a variety of reasons.

Smartphones are being adopted extremely rapidly, and they possess some important characteristics that make them well suited to be used in commercial transactions:

     *They can be configured with extra-secure memory and other hardware features to increase confidence of consumers whose money and/or information might be lost.

     *They have cameras that can serve as scanners of bar codes and other physical hyperlinks.

     *The cameras and displays can also facilitate biometric identification in the form of face recognition, either algorithmic or by humans.

     *They can be configured with a radio devoted to secure payment at close range.

     *They are pretty reliably carried on one's person and thus can replicate some roles of a wallet.

Who has entered the arena

Competition in mobile payment is already intense. As the list of active players illustrates, however, different entrants may have differing strategic endgames. To say that the U.S. market does not need smartphone-based wallets because of the mature credit-card reader infrastructure may be true, but does not negate the potential impact of couponing, loyalty cards, location-sensitive promotions, and ad placement: smartphones in the U.S. market may never get the functionality of a Safaricom M-pesa account in Kenya, but plenty of other interesting scenarios are nonetheless possible.

Mobile carriers: Sprint is the only carrier to support Google Wallet. AT&T, T-Mobile, and Verizon comprise the ISIS consortium. Isis is running trials.

Credit cards: Visa has PayWave while MasterCard launched PayPass. For its part, American Express has Serve, which is pre-paid rather than a credit service like the other two. Discover has announced an alliance with PayPal (see below).

Merchants: Announced in August, the Merchant Customer Exchange is led by Best Buy, CVS, Shell, Target, and Wal-Mart among others.

Google: Google Wallet currently runs only on a few Sprint phones. A sticker with an embedded chip can bring smart wallet features to Android phones that do not meet carrier standards for NFC (near field communications) in the hardware layer, but these do not appear to be widely available and/or adopted.

Square: The mobile payment startup founded by Jack Dorsey of Twitter fame (he works 8 hours a day at each company) does not require NFC chips and interfaces neatly with the company's merchant solutions which attempt to disrupt the Verisign-dominated credit card terminal market.

PayPal + Discover card: PayPal announced the credit card partnership in August and expects to process $10 billion in mobile phone transactions this year. This appears to be the first mover with competitive advantage, in the early going.

It is difficult to believe that Amazon and Apple (which currently supports non-payment wallet functionality) will stay out of this market.

The questions:

1) Hardware vs software

While the Google Wallet and Isis solutions rely on a hardware component -- the NFC chip in the phone and accompanying reader at point of sale -- PayPal and Square employ more of a software play. By way of comparison, in Japan, the prepaid cards used to pay train fares use contactless hardware, while in Kenya the popular M-pesa service runs on SMS.

In the U.S., wireless carriers control access to the secure memory area that is required for Google Wallet, and that access has not been widely agreed to thus far. Alternatively, cloud-based solutions move consumer information off of the handset, obviating the need for carrier-controlled access to the so-called secure element. In addition, they do not require new point-of-sale infrastructure at the merchant, as in the case of NFC. In the long run, the Isis solution appears to be the most likely NFC-based scenario, if any take hold, while Google recently announced it is moving to a cloud approach possibly in parallel; the fate of the hardware-resident Google Wallet is unclear. Apple has yet to include NFC chips on iPhones.

2) Payment vs promotion

Even though the rubric is to speak of mobile payment, because the US has such a broad credit card reader network, the promotional aspects of mobile wallets will be a critical factor for user uptake. This opens the possibility of an arbitrage app, to sort through a consumer's competing offers, point systems, privacy compromises, and time constraints. Do I buy gas at Shell on MCX or at Sunoco with Isis today? What about next week's offers? How close am I to Gold standing in a given reward program? Who do I trust to maintain multiple account information? Apple has Passbook, an iOS app for ticketing and loyalty but not (as of yet) payment. Google Wallet does many of the same things.

3) Multi-home vs single-home

These terms derive from the economic literature on platforms. Many "multihoming" people carry both Mastercard and Visa. Some electronic game titles are available for both Xbox and Playstation. At the same time, very few people maintain both Facebook and MySpace identities, or checkbooks from consumer accounts at two different banks. Right now Visa in particular works with many other digital wallets. The homing question will be interesting to follow: will consumers maintain simultaneous Starbucks, Visa, Isis, and MCX accounts, their high degree of overlap notwithstanding? Will reward systems interoperate, or will they be more like frequent flier miles, which are expensive to convert across carriers? In short, will there be a few huge winners or many players of varying scales?

4) Sweet spot (transaction amount)

In the U.S., credit cards are fast and easy, but not widely used for purchases under $10 or so. Retail foodservice establishments, meanwhile, process millions of transactions a day under $25, and the experience of making change is slow, error-ridden, and inconvenient. Given the power of the U.S. penny lobby, we still use the copper coins even though it likely costs more to spend the extra time for a McDonalds clerk to make exact change for 47 cents rather than 45 or 50. Thus a wave-and-go payment solution for fast food and other outlets (where customer throughput is a key profitability metric) makes a lot of sense. Burger King does not really "average" 500 (or whatever) customers per hour, but instead deals with large spikes in demand at lunch time, when big events let out, and so on. Decreasing transaction time will pay off quickly for these merchants, and if they do not need to invest in new reader infrastructure, so much the better. Thus if mobile payment can fill in a niche at the low end of the market rather than compete with plastic cards, it could expand rather than only cannibalize the existing market.

5) Pre-pay/credit/mobile bill

What will be the preferred source of funds in the digital wallet(s)? Visa and Mastercard can easily extend their credit model to new payment modalities. Pre-paid plans work well for mobile carriers and do not require credit checks and other risks. Wireless carriers already have credit ratings, home addresses, and locality information for their customers. Each of three models could catch on for some population (mass transit passes for school kids in cities, meal plans for college campuses, gift cards for retail shoppers, and the list goes on); it may not be a winner-take-all scenario.

6) Global vs national

A credit card can currently be used nearly anywhere. If NFC or other hardware solutions prove popular in a given geography, how widely will any particular technology take hold? Is a variety of local technologies the long-term scenario? Most likely not.

7) Killer app(s)

To unseat the incumbents (cash money and plastic money), mobile money will need to be secure, easy, widely accepted, and deliver additional benefits. In Kenya, one killer app was processing expatriate payments from abroad without a trip to Western Union. In Japan, train fares at rush hour are processed much faster with mobile money than with other forms, thus easing workflow. In the US, mass transit is not widely used, but coupons and loyalty programs are. Somewhat weirdly, many demo videos for various smart wallet solutions all focus on one merchant scenario: coffee shops. This segment doesn't feel sufficiently substantial to stand as the cornerstone of a new shopping paradigm, however.

The corporate heft of the various competitors suggests a sumo-like contest to win consumer affection. What's the payoff? Consumers love discounts, and smartphones can geoposition those offers to current place and time. On the platform owner's end of the transaction, meanwhile, the prospect of terabytes of consumer data piling up from offers at various price levels or under different conditions (25% off versus "buy three get 1 free") with the location tracking aspect presents a "big data" scenario of staggering proportions. It's clear to see the prize -- consumer data at unprecedented scale -- but the value proposition to wallet-carriers remains unclear.

8) Wild cards

What happens when some segment of the infrastructure (power, wireless data, terminals) goes down?

Who pays to retrofit millions of vending machines, arcade games, and other devices that currently take cash and coins? What about merchant terminals?

What happens when data is compromised?

What do I do when my phone/wallet is lost or stolen?

Who bears what risk?

What happens when my battery dies?

What happens with competing standards or entities? Can merchants "jam" competing offers? If Shell sees I'm gassing up at Citgo, do I get coupons? Cut off? Slower service next time I pull into Shell?

What if public backlash at RFID extends to NFC, an RFID variant?

Will anyone understand the privacy agreements they sign?

Sunday, September 30, 2012

Early Indications September 2012: Party platform comparison

*****
It should be stated explicitly that all views are my own and do not represent official policy or any other attribute of my employers. Politics is always delicate territory and I have tried both to critique and respect both parties in my analysis.
*****
Given the stubbornly high un- and under-employment numbers, the rapidly diverging fates of the very rich and everyone else, and the increasing speed of technological change, it appears that the U.S. economy is in the midst of some fundamental transitions. Consider the evidence:

*Apple, with a trillion-dollar valuation in range, employs fewer than 50,000 U.S. workers, not counting retail store personnel. Facebook, said by Wall Street to be worth $100 billion at one point, employs only about  3,000. Not only are their overall numbers small, these highly-prized jobs create very few "multiplier" jobs for lower- and middle-class workers the way the automobile did. Where will large numbers of new jobs come from? In most cases the old ones are not coming back.

*Municipalities in multiple states are declaring bankruptcy, and unlike the 1994 Orange County precedent, these are likely to be followed by more. It is also likely that a U.S. state will have to declare bankruptcy, or something similar. Between direct hiring of employees as well as support of homebuilding (through the mortgage interest deduction and deep involvement in mortgage markets) and public works, government-fueled employment has helped counter a long-running loss of manufacturing jobs. As the wave of government hiring from the last 65 years bequeaths us its total costs -- in extremely generous health care and retirement benefits -- how can we afford the past hiring at the federal, state, and local levels, much less the governments of the future?

*A college education historically contributes to higher earning histories, but the price of that education is also growing at an unsustainable rate. According to the College Board, the average cost of a 4-year public college education has more than quadrupled since 1981, after counting the effects of inflation. Student loans are available to cover this ever-increasing sum (thus creating a moral hazard for the universities?), with the result that student loan debt has crossed a trillion dollars. Insofar as student loans are not able to be discharged by bankruptcy, and given that requiring high credit scores along with a large down payment is now standard practice in the mortgage industry, many of the students who leave college with these loans will not be buying houses any time soon. How can we educate students in tomorrow's skills without saddling them with excessive debt?

Given these interconnected forces, the role of elected officials in encouraging job creation is limited: governments are shedding police officers, fire fighters, and teachers. Much as every president loves a construction boom, most housing markets are still digesting excess capacity built in the 2003-2006 period even as they are more boomer home-sellers than there are Gen Y couples to buy real estate. Even though the benefit of more education used to be axiomatic, honest discussions of student aptitude and projected debt should be a factor in college choice in more and more cases.

Thus I believe that the nation is at a crossroads, a historical moment when new approaches are necessary. What can voters look forward to? Rather than try to decode the speeches of the presidential candidates, I examined the two parties' platform documents. Sadly, there is little indication that either party is approaching today's unique challenges with an adequate arsenal of ideas.

The Democratic platform is long on punchlines: "We see an America that out-educates, out-builds, and out-innovates the rest of the world," it repeats at several junctures. The next American economy needs to be built "from the middle out," not from the top down. The proposals in this vein have the overall whiff of a free lunch: middle-class tax cuts, refinancing options for mortgages under water, "accessible, affordable, high quality health care," and a promise that "we will find a solution to protect Social Security for future generations." How will this last goal be accomplished? Later retirement, higher taxes, and/or lower benefits are not mentioned as the only ways out. The document is short on the necessity of sacrifice.

As for education, the Democratic platform focuses more on teachers than on students. While "Democrats believe that getting an education is the surest path to the middle class," the cost of bad teachers for tax payers and students alike is never addressed. Instead, in a nod to union resistance to removing underperformers, "We also believe in carefully crafted evaluation systems that give struggling teachers a chance to succeed and protect due process if another teacher has to be put in the classroom." For every "struggling" teacher, however, hundreds of students are denied an adequate education, but they are not mentioned.

This commitment to public-sector unions appears in another passage, one that implies that high-paying government jobs have no discernible price. Compared to the labor-management strife that fueled the big industrial unions, teachers' and police officers' economic interests are opposed not to bosses but to taxpayers: the teachers' neighbors. Nevertheless, "we will fight for collective bargaining rights for police officers, nurses, firefighters, emergency medical technicians, teachers, and other public sector workers -- jobs that are a proven path to the middle class for millions of Americans." But who pays that bill? In France, the government (followed by the post office) is the largest employer and the top marginal tax rate is 75%. Such an end-game feels unsustainable in a country as large as the US.

In the transportation domain, it's hard to see anything encouraging. In addition to the "all-of-the-above energy policy" and encouragements for investment in clean energy, the "Democratic Party supports a broad-based strategy to further strengthen an American renaissance in manufacturing," including "incentives to build advanced vehicles in the United States." The particulars (what's an "advanced vehicle"? The Chevy Volt from Government Motors?) are nowhere to be found.

In sum, much of the Democratic platform document is given over not to specifics, as we have seen, but to fear-mongering about what a Romney presidency might entail. For what is ostensibly a policy document, the Democratic platform could have done far better.

Which brings us to the other ticket. Richer in specifics, the Republican platform is so dominated by trickle-down ideology that job creation and economic recovery at the scale necessary seem extremely unlikely: the document looks into an idealized rear-view mirror on an America that never existed in the first place -- then apparently wants to bring it back, never mind the actual forces at work both then and now. Tax cuts, reductions in regulation (particularly the environmental variety), and hot-button social issues (immigration, firearms, Judeo-Christian religion, gay unions, the gold standard) dominate. Just as with the Democratic document, the Republican platform spends a lot of time name-calling, rather than constructively moving forward. Part of this behavior is the nature of our times, I realize, but I persist in believing the majority of the American people are sick of partisan bickering.

The second sentence of the Republican document begins with a bold assertion: "Prosperity is the product of self-discipline, work, savings, and investment by individual Americans." The converse is clearly implied: lack of prosperity is the product of weak moral fiber, laziness, careless spending, and short-sightedness by individual Americans. The rich are rich because they existentially deserve their wealth, and taxation is thus a form of theft on behalf of those who are not prosperous, presumably because of the aforementioned factors.

The problem is that many who are wealthy inherited their fortune (what Warren Buffet calls the "lucky sperm club"), while still others had the good fortune to be in the right place at the right time. And luck works both ways. Most restaurants fail, notwithstanding chefs' 80-hour weeks. People pick economically unfortunate college majors or slow-track entry-level jobs. Factories close. Jobs get outsourced. Illness happens. Across geographic and ethnic lines, family units vary in size, strength, and respect for education; unfortunately, children can't pick their parents. Can anyone say these individuals are less morally worthy than the fortunate few wealthy? This resistance to acknowledge the role of luck -- alongside intelligence, prudence, and effort, I hasten to add -- in people's lives contributes to the overall tenor of the Republican economic plan.

Furthermore, the document is counterfactual at several points. Despite the sluggish economic performance of the decade during which the Bush tax cuts were in force, the Republican platform nonetheless asserts that "lowering taxes promotes substantial economic growth" and later bemoans the current federal budget rules' refusal "to recognize the beneficial budgetary impact of lower tax rates." Similarly, the party of creationism and of politicized redactions to climate research in the Bush White House states that "We must restore scientific integrity to our public research institutions."

In addition to denying the relevant past, the party document gives the ability of the market system to create positive outcomes undue credit: infrastructure, paid for with public tax dollars, should be privatized even though the track record for such projects, such as various toll roads, is mixed. Likewise, mortgages, student loans, and Social Security should be subject to the free market of financial services firms. Health care, too, can be fixed with more free market dynamics. Even schools -- choice via vouchers -- can be repaired with market mechanisms. Following the money may be useful here: who stands to gain under each scenario? Will the nation be stronger overall? Consider that in health insurance, rational behavior for the insurers, whose CEOs earn handsome salaries, is to deny as much care as possible. Somehow, maximizing shareholder return does not seem to be the overall point of this particular exercise.

Despite serious issues with their overall vision, give the Republicans credit for specificity. The platform contains some economic ideas that definitely should be tested further:

-workforce retraining programs should be modernized

-the tax code should be simplified

-immigrant innovators should be welcomed rather than turned away

-the Telecom Act of 1996 is "woefully out of date"

-the 130-year old Civil Service system should be reformed

-tort reform should deliver measurable benefits to both health care costs and physician service in rural areas, which are often underserved

-new systems of learning and new models for life-long skills refreshment should be a priority.

If the Democratic drift is toward free lunches without sacrifice, the Republican platform venerates wealth but says little about not-wealth: the sick, the aged, those left behind by economic change. Similarly, while Democrats were criticized for not mentioning God in their platform, from a policy standpoint the Republican refusal to acknowledge climate change is more worrisome.

Okay, the reader may reasonably ask, so it's easy to poke holes. What would you offer that's better? Fair question. Here are five big proposals that could be adequate to the times we face and, more important, recognize that the status quo is inadequate: we, and especially our children, deserve better. Looking back in 20 years, will we honestly be able to say we did the best we could? Just as important, the rest of the world is looking for leadership amidst tremendous potential and instability. Will they find it here?

1) Invest in our nation and in our future: initiate national service for 18-19 year olds. Make it compulsory and universal. Mormons back from their mission and Israelis who serve in the defense forces both are more skilled and more mature than 20-somethings without the experience. From the nation's perspective, national service should increase the skills base, reduce welfare and similar payments, and create roads, parks, libraries, literacy programs, broadband networks, and other projects of lasting value. Giving potential college students a taste of the work world should either a) make them more motivated students or b) convince them they're better suited for trade craft or other non-college work. Letting teens earn some of their future tuition could help address student loan debt too.

2) Stop paying for so much waste in health care delivery. a) If we spent less on insurance processing, and yes less on tort-driven behaviors as per the Republican platform, quality of care should go up while costs drop. I'm not pretending to know what combination of single-payer, health savings accounts, and/or lifestyle change would be required, but addressing the problem is imperative. b) A vastly disproportional share of health care expense is incurred at the end of life. We as a culture need to stop trying to "beat" death and start acknowledging the life that was lived and comforting the individuals in transition. Physicians need to feel, and be allowed to feel, human and neither like gods nor life-extension technicians in such moments.

3) Increase funding for DARPA. Given stealth aviation technology, GPS, and of course the Internet, this agency has paid for itself many times over. Their current emphasis on robots is going to be similarly game-changing, I predict.

4) Fix the trademark and patent system. Intellectual property law is a disaster in that those without rights can profitably stall progress while those with certain rights are protected for far too long.

5) Attack the obesity epidemic with the same vigor of putting a man on the moon. Diet, exercise, mental health care, workplace arrangements, incentives and disincentives -- the problem can and should be fought on many fronts. Given the Olympic success of this great nation, it is clear that we can devise physical and related programs so that no adult and especially no child should lose quality of life or overload the healthcare system with preventable weight gain. While such measures could be read as intrusions into individual liberty, the precedent of smoking is relevant, I believe.

Saturday, July 28, 2012

Early Indications July 2012: The Media Beast

Those of you who have read this newsletter for a while know that it
has, nominally anyway, a technology focus. While the current number
may not feel like it delivers a tech message, that is in fact where
the argument is heading.

The recent events at Penn State raise a broad range of questions, from
"how can humans be so evil?" to "what were those men thinking?" to
"who was the client for Louis Freeh's report?" to "what was the NCAA's
agenda in the sanctions?"  Many of these questions remain extremely
loaded with emotional and legal baggage, and I will tread lightly
around them, speaking only as a private citizen and not in any
University capacity.

My purpose is somewhat more macroscopic. Think back to some sports
heroes of the 1990s. All of these individuals have, to some extent,
been disgraced and fallen from their pedestals.

In alphabetical order,

Barry Bonds
Michael Irvin
Allen Iverson
Marion Jones
Bob Knight
Mark McGwire
Joe Paterno
Rick Pitino
Tiger Woods

Lance Armstrong could well be next.

Each of these individuals boasts a Hall of Fame resume. Each at some
point transcended his or her sport, becoming an icon in some larger
arena. But each at some point, having been made larger than life, fell
hard from the extra-human heights to which he or she had ascended.
Without mass media, I will argue, these individuals could not have
soared to such cultural visibility, but many people's cognitive and
emotional apparatus may not be well suited for life at such a scale,
setting the stage for subsequent disappointment.

Accordingly, I'd like to explore the role of technology in this
process of creating and then questioning heroes, using the Penn State
experience as an example close at hand. My proximity to the story is
recent: I moved here in late 2005 from Boston. The Paterno statue had
already been erected four years earlier, but I immediately noticed
that State College felt different. My grad work was done at Michigan,
home of the winningest college football program in history, both by
wins and winning percentage. My undergrad years were spent at Duke,
though Coach K mania can hardly be said to have hit in that era: while
his first three years' record was 38 wins and 47 losses, home
basketball games at Cameron had already became one of the great
experiences in all of sports. So I came from academic institutions
where sports Mattered.

While geography is certainly a factor (State College, for those who
haven't visited, is said to be equally inaccessible from everywhere),
part of the State College difference is architectural. The football
stadium, which is enormous, has a presence beyond mere mass: Michigan
Stadium, the "Big House," seated more people when I was there in the
1980s, but it's a giant hole in the ground. Driving by it in Ann
Arbor, there was little in the pre-skybox days to announce its
vastness, so walking inside was a wonderful experience to grasp the
scale from the top row of seats. (At graduation, we got to walk out
onto the field level through the tunnel the players used, though there
was no M Go Blue banner to leap up and touch.) Beaver Stadium, by
contrast, is all above ground. Given that it was added to in sections
as Coach Paterno's success generated interest, it looks like an ad-hoc
spaceship, with spindly underpinnings and bolted-on exterior ramps and
walkways. (Here's a view that shows that tendency) Contrast the
Rose Bowl, or even the Horseshoe at the state university to the west,
and unlike Beaver Stadium, they were designed from their origins to be
monumental.

The last of the renovations was completed in 2001, when seating
capacity, which had doubled in stages from 46,000 in 1960 to 95,000 in
1991, was increased to 107,000. The move seemed justified as the 1990s
were heady days for the program. Penn State joined the Big 10
conference in 1993, and immediately delivered strong results,
including three top-10 national rankings in seven years, top-20
rankings every year, and an undefeated season in 1994. The statistical
rankings become relevant in a moment.

1993   10-2

1994   12-0  (#70 in total defense; #1 in scoring offense (48
points/game); #1 total offense)

1995   9-3  (#50 in total defense)

1996   11-2 (#41 in total defense)

1997   9-3   (#86 in total defense)

1998   9-3   (#12 in total defense)

1999   10-3  (#27 in total defense)

2001, when the current stadium was completed, was also the year when
the statue of Paterno was installed, and a year when success was hard
to come by. Before the wins were vacated by the NCAA, here was the
story:

2000   5-7   (#55 in total defense)

2001   5-6   (#98 in total defense)

2002   9-4   (#27 in total defense)

2003   3-9   (#49 in total defense)

2004   4-7   (#10 in total defense)

So consider the context for the 2001 decision to cover up the Sandusky
abuse, which is what is alleged to have happened:

-The football team was in the midst of a run that would get most major
college coaches fired: only one bowl game (a loss) in five years.

-The Paterno years had been remarkably consistent until that time. The
program was the 6th most winning program of the 1970s, the 1980s, and
the 1990s. Only Nebraska could join Penn State in the top 6 all three
decades: Oklahoma did it twice, while Alabama, Florida, Florida State,
Miami, and Michigan did it once apiece. LSU, Ohio State, Texas, and
USC did not make the top 6 in any of the three decades.

-The losing streak coincided with the retirement of Sandusky, a
respected defensive coordinator, though the statistically best
defenses did not always translate to winning seasons. Penn State's
reputation as "Linebacker U" derives from Sandusky (and continues
under Ron Vanderlinden, who has roughly a half-dozen linebackers/pass
rushers currently in the NFL).

-Penn State basketball has rarely made Big 10 hoops royalty very
nervous. Since 1983, the program's conference record is .317. Thus
football must generate sufficient revenue to support many non-revenue
sports, several of which -- wrestling, and men's and women's
volleyball -- win national championships.

Looking back, the turn of the millennium seems to have seen a changing
of the guard. Paterno turned 75 at the end of the 2001 season. The
game, and his university, had changed dramatically over his career.
To take the latter first, Penn State's average SAT scores went up a
relatively modest 67 points between 1966 and 2006 (LSU's dropped 110;
Virginia's increased 141). I cannot find a list of federal research
expenditures by university from 1970, but Penn State has risen to rank
in the top 15 in total research dollars received. Like the University
of Washington, the Penn State undergrad ranking is not selective (tied
with UW at 184), but both schools are top-tier research universities,
standing in the top 20 institutions in PhDs granted and other key
measures. Like Duke, Arizona State, and other institutions, Penn
State's academic reputation has risen considerably over the years
during which Paterno was coaching. He claimed, plausibly, that he had
raised $1 billion for the university, money that had helped academics
gain in stature.

How did football change in the Paterno era? In 1966, his first year,
ABC was the only network to broadcast regular-season college football
games. With a Supreme Court decision in 1984 ruling that the NCAA
television plan violated anti-trust law (the University of Oklahoma
and the University of Georgia were the lead plaintiffs), the number of
televised games spiked from 89 to roughly 200 in just one year. Rights
to college football became truly big business: the Longhorn Network
joint venture at the University of Texas is potentially a $1 billion
enterprise over 20 years, and the deal contributed to the near-demise
of the Big XII conference of which Texas is a member.

ESPN, which began broadcasting in 1979 when only about 20% of US
households had cable, seized on college football after the Supreme
Court decision with a 48-game package in 1984. As the 1958 NFL
championship game proved, television and football are a potent
combination, and ESPN's rise is intertwined with that of college
football. College GameDay began broadcasting in 1987, but became a
major part of campus life in 1993 when it went to its current live
remote packaging. (ESPN was originally an investment of Getty Oil and
. . . Nabisco, which later sold its stake to Hearst, which still holds
it: Disney only acquired 80% of ESPN in the Cap Cities/ABC acquisition
in 1996.)

Other aspects of the game changed over Paterno's career. Modern
athletic departments can be $100 million operations, with commensurate
losses when times are bad. The athletic directors prepared for
business at that scale are rare; in Paterno's era, many football
coaches (including Joe himself) were ADs either during or after their
coaching career. University of Florida president Bernie Machen has
said the ESPN money means that athletic departments become
semi-autonomous entities outside university control. Thursday night
football games, for example, often result in class cancellations and
other academic accommodations to the networks. Television exposure, in
turn, generates student applications for admission: Duke's increase in
selectivity coincides closely with the Coach K era.

Coaches can become millionaires. While it is said that John Wooden,
probably the greatest college coach in American history, never made
more than $35,000 in salary, today the head football coach at Wyoming
earns $1,000,000 in a complex package, not counting a car and cell
phone. The head man at University of South Florida pulls in $2 million
a year. Life at the top of the ranks is a $5 million/year proposition.
That's a lot of money for a coach to lose if the program goes south.

So cable sports television was one major change in sports life. What
does that have to do with the list of disgraced figures with which we
began? The pressure of 24-hour media takes a toll. For all the sports
figures in the limelight who stumbled, there are entertainers who
responded even more tragically: Kurt Cobain, Whitney Houston, Elvis
Presley, Michael Jackson. In sports, the pressure to win, and then win
AGAIN, can become enormous. The urge to cheat is one temptation (as at
the University of Connecticut under Jim Calhoun, currently banned from
the NCAA basketball tournament), as is steroid and related drug use.
But the human cost of life in the mass media spotlight is complex:
even Duke's Coach K, he of the military discipline, solid family, and
ideal circumstances, had to reset his life in 1995 when his wife
staged a full-scale intervention amidst his physical and emotional
burnout.

The pressures are both subtle and obvious. Coach Charlie Weis at New
England and later Notre Dame and Kansas has a serious weight issue for
which he had gastric bypass surgery; Rex Ryan of the New York Jets had
similar surgery to help lose more than 100 pounds off a high of about
350. Urban Meyer at Florida found himself reacting to the stress
differently, 35 years Paterno's junior and even more recently
successful and therefore powerful (two national championships in his
first four years in Gainesville). Meyer reported feeling physical
symptoms of stress and exhaustion, and quit then unquit coaching. The
age of the Internet, however, created an alternate news channel where
rumors spread that Meyer had followed the Bobby Petrino/Rick
Pitino/Tiger Woods road of temptation, with a UF student. That's
another way the game has changed: 24-hour/52-week news coverage, talk
radio where spleneticism equals ratings, and message boards where
fandom has no fact check.

Technology will always shape sport. As I write, NBC is refusing to air
live coverage of the Olympic games so Twitter is providing a rough
facsimile of the pomp and pageantry that I will later refuse to watch
in prime time. Fan interest in statistics is growing annually, to the
point where the NBA just signed a deal to use SAP's powerful HANA
analytics tool to power a basketball stats website. Cellphone cameras
can document any moment, as Michael Phelps discovered the hard way
after the 2008 games as he relaxed with some weed. The interplay
between athletic drama, success, power, and personal identity has
grown more complex with television and later the web, mobility, and
social media.

When Joe Paterno rebuffed his president's "suggestion" to retire after
the losses mounted in 2004, he proved he effectively reported to no
one but himself.  Paterno helped build Penn State to what it is today,
and televised football both helped create the Paterno mystique and the
events that dismantled it: Paterno's weak replies in 2008 to an ESPN reporter asking about his players' formidable arrest record, vividly contrasting with the "Grand Experiment" rhetoric he helped propagate, are some of the defining moments of the last decade of his long
career.  Paterno famously eschewed cell phones, e-mail, and text messages. At the same time, he championed video replay of officiating calls, and vigorously supported
the Penn State libraries, home to a multitude of electronic databases
and other learning tools. Once again, technology was a two-sided coin.

The power of broadcast media to aggregate enormous audiences, and the
simultaneous need of the technology for drama sufficient to the stage,
creates the possibility of truly super-human figures. When mere humans
feel small compared to their projected selves, bad things can often
happen. Other times, when people feel their media image has changed
their off-the-field life so rules don't apply, we can get the sexual
assault ugliness of Kobe Bryant, Ben Rothlisberger, and others.
Either way, aligning human existence with life in the athletic
spotlight (and cellphone lens, and Twitter stream, and Facebook
profile) has never been more challenging.

Saturday, June 30, 2012

Early Indications May 2012: The limits of consumerism

In 1968, a young reporter named Joe McGinnis covered the presidential
campaign of eventual winner Richard Nixon. Nixon was determined to
overcome the liabilities that were thought to have contributed to his
loss in 1960 to John F. Kennedy. At the Kennedy-Nixon debates, for
example, Nixon's skin tone, choice of suit color, and physical
awkwardness contrasted with his verbal command of the material: radio
listeners voted Nixon the debate winner, but Kennedy was viewed as the
winner by the TV audience. In 1968, a television producer named Roger
Ailes took charge of managing the Nixon campaign to avoid any such
missteps. McGinness's chronicle of the campaign, The Selling of the
President
, put McGinniss on the map as a political observer, and Ailes
has continued to shape American political debate as well.

Fast-forward to 2012. The selling techniques for a television
candidate are blunderbusses in the age of Internet marketing. The term
of art is "microtargeting," and both the Obama and Romney campaigns
are expending considerable sums of money and expertise on the analysis
of sub-segments of the electorate. At base, political campaigns
overlap consumer products branding. In fact, to see how political
intensiveness and likelihood to vote map against soft drink
preferences, television viewing, and automobile ownership, I urge you
to take a look at the charts published in The Atlantic or the New York Times.

One widely reported microtargeting effort occurred in 2004 when the
Bush campaign neutralized a segment of the traditionally Democratic
coalition: socially conservative black voters responded particularly
strongly to anti-gay marriage messages. In the runup to the 2008
campaign, and to some degree in response to Bush's 2004 success,
Hilary Clinton's pollster and chief campaign strategist Mark Penn
published a book on microtrends that illustrated the blurred line
between his job at a public relations firm and his work on the
campaign. (In 2009, the conflict of interest resurfaced as firms
affiliated with Penn won nearly $6 million in stimulus spending.)

Very few people understand how web advertising works. Rather than try
to explain it here, I can include a few references:

- The Atlantic, 2/29/12

- Wall Street Journal

- Penn professor Joseph Turow's new book, The Daily You

Such data-rich customer -- er, voter -- profiling means that political
ads will be triggered by seemingly apolitical searches or site visits,
and persist across one's web travels. Both the Obama and Romney
campaigns mine Facebook data from individual voters, who repeatedly
say in surveys that they are uncomfortable with behavioral tracking,
to the limited extent they understand it. Microsoft's decision to
limit tracking in the new Internet Explorer browser is not well
received by advertisers.

As any good marketing executive knows, integrated marketing can amount
to more than the sum of its parts. Microtargeted online ads, e-mail
campaigns, negative TV ads, positive radio spots, statesman-like
thought pieces, friendly voices on the Sunday morning pundit shows,
billboards, direct mail -- campaign marketing has become a complex
beast indeed. And for all the sophisticated microtargeting that is
possible, the power of the brand - Hope, in 2008 for example -
dictates many of the other possibilities. As usual, we are witnessing
the fights to define the brands in the post-primary, pre-election
doldrums. Will Mitt Romney be equated to a wealthy, out-of-touch
business executive or a can-do manager? Especially in light of weak
job numbers, does President Obama get to boast of his foreign policy
record or defend the limited impact on employment of hundreds of
billions of borrowed stimulus dollars?

I'm very sensitive about romanticizing the past. The good old days are
seldom as good as selective memory can portray them. Nevertheless, I
am concerned that developing political campaigns as just another
consumer preference has definite limits: it ignores the necessity for
sacrifice.  David Brooks made this point nicely in a New York Times
piece on May 17: in contrast to the limits on popular sentiment
engineered by the Founding Fathers (Madison most notably), "Leaders
today do not believe their job is to restrain popular will. Their job
is to flatter and satisfy it. A gigantic polling apparatus has
developed to help leaders anticipate and respond to popular whims.
Democratic politicians adopt the mind-set of marketing executives.
Give the customer what he wants. The customer is always right."

But the customer/voter's self-interest, even when it might be
enlightened, will not solve hard problems, of which there is no
shortage:

-Public pension funds continue to budget for 8% investment returns
even though the reality is far less rosy than that. Thus the shortfall
in public pensions will be far larger than predicted. (WSJ story here.)
Even aggressive, expensive investments such as hedge funds are not
making up the slack. According to author Simon Lack, cited in the Economist , "Indeed, since 1998, the effective return to hedge-fund  clients has only been 2.1% a year, half the return they could have
achieved by investing in boring old Treasury bills." Social security is only the
tip of an underfunded iceberg among elders that we can see from afar
yet pretty much ignore.

-Student loans are being taken by naive families, to finance
educations with low or uncertain returns, at an unsustainable rate.
Total student loan debt is roughly $1 trillion, and rose 8% this year.
At the same time, college graduates are the majority of the
unemployed, and the institutions must change their form, incentives,
and content to address a new world. So far that is not happening on a
wide basis.

-The cost of the wars in Afghanistan and Iraq can never be completely
calculated, extending as it does probably six decades out in veterans'
health care costs. Amputations and concussion ramifications will cost
billions to address. More urgent still is the need for mental health treatment adequate to the need: far more veterans of those wars have
committed suicide after mustering out than were killed by combat
injuries.

The limits of politics as consumer preference are being played out
everywhere from France, to Italy, to Mexico, to Greece, to the U.S. As
the planet encounters physical and other limits to infinite growth on
multiple axes -- the nexus of aging, health care, and pensions, along
with carbon, petroleum, animal protein, and others -- the question is
no longer only or primarily "what do I want?"

At base, the fallacy of the consumer hypothesis is deeper than
politics. As research from scholars including Barry Schwartz, Daniel
Gilbert, Shawn Achor, and many others is showing, material stuff,
money, and success do not make people happy. (Excesses of the same
ruin lives, whether of lottery winners, Elvis Presley, Kurt Cobain,
Michael Jackson, Tiger Woods, Whitney Houston, Bobby Petrino, and many
others.) Meaning, purpose, community -- these old-fashioned notions
run counter to the consumerism mythology. When will politics be forced
and/or ready to embrace civic values that acknowledge shared sacrifice
rather than endlessly finance subsidies with a tax on the future?

The answer may lie on the web, with hints coming from the likes of
Kickstarter, Kiva.org, and Ushahidi, rather than from the promised
algorithmic paradise of the Big Data political analysts that, probably
earnestly, try to bring to American elections the same mechanics that
have made Wall Street what it is today.

Early Indications June 2012 Review essay: Daniel Kahneman, Thinking Fast and Slow

In his masterful reframing of the history of science, The Structure of Scientific Revolutions, Thomas Kuhn introduced the notion of a paradigm shift. These shifts begin when what he called "normal science," the use of an accepted theory of the era in a given field, was confronted by a sufficiently large number of anomalies the theory couldn't explain. If the sun revolves around the earth, for example, then how can earth's night, day, and position relative to other non-solar heavenly bodies  be explained? The process of theory-building followed by anomaly accretion helps explain many pivotal moments in the history of science, which does not, Kuhn found, progress by steady addition to existing theory.

Daniel Kahneman won a Nobel prize in economics in 2002, despite not being an economist, for work done with his late collaborator Amos Tversky on numerous departures of observed human behavior from the tenets of neoclassical economics: briefly, that economic agents act independently on the basis of full and relevant information, that all actors seek always to maximize their utility, and that people have rational preferences among identifiable outcomes. Kahneman's new book, Thinking Fast and Slow, furthers this mission, but also does much more.

Kahneman begins the book by recounting a series of stories about the extraordinarily rich collaboration he enjoyed with Tversky, who died in 1996. The output of their professional and personal partnership continues to redefine the world's understandings of human choice, possibility, and self-deception.  As a personal tribute, this section is warm and affecting, but I believe Kahneman is building another layer of argument as well, to which we will return presently.

Humans pose (and face) a series of three paradoxes, embodied in the architecture of the book. The first of these, the contrast between quick intuitive judgments and considered calculation, is fascinating to observe and ponder: humans display a persistent habit of using cognitive shortcuts known as heuristics when confronted by new questions. A troublesome feature of the interplay between these two systems of thought is substituting an answerable, easier nearby question for a difficult one. "Is political candidate A qualified to govern a complex democracy?" is a hard question. On the other hand, "Is candidate A well dressed and carefully spoken?" is readily answered, so we frequently act on the false understanding that we have answered the tough, original question. In the search for easier questions, humans will conflate loudness with accuracy, or the first number at hand with a realistic estimate.

The second paradox exists between economic man and humans, who can be counted on to value fairness, for example, rather than reliably maximize utility at every opportunity. What Richard Thaler of the University of Chicago refers to as "econs" (the theoretical creatures) are rational and selfish, with persistent tastes that do not change, in contrary to "humans," who of course have limited information, change their mind, and are often generous. Experiment after experiment has shown patterns of preference and behavior that violate the neoclassical assumptions that continue to dominate economic orthodoxy, and the many examples in the book are fascinating, if unsettling: in Kuhnian fashion, Kahneman and his colleagues have built a mountain of anomalies.

Finally, Kahneman differentiates between the remembering self and the experiencing self. This set of insights draws in part on experiments involving colonoscopies and other unpleasant procedures as well as vacations and similar peak experiences.  Tellingly, if asked whether they would take a dream vacation under the condition that they would take an amnesiac drug and be denied all photos afterward, most people refused the vacation. In many instances, it appears we live for the future memory of the moment rather than for the moment itself. As elsewhere in the book, the implications of one single insight are momentous.

Within the extreme richness of Thinking Fast and Slow, the nuances and surprises embedded in these three discussions are far too numerous to summarize. Several insights struck me as illustrating the broad implications of this book. Certainly investing and business decisions need be examined in a new light, and policy choices should be informed as well. But unlike most examples of Kuhn's paradigm shifts, I believe Kahneman seeks also to improve individual human existence as well. Three examples follow:

-People are extremely deft at telling stories to explain the past: the Allies were confident of the ability to win World War II; the rich deserve their wealth as reward for hard work, moral worth, or other factors; the U.S. had a manifest destiny to subdue the American west and the various nations with prior claims thereto. Life is lived forward, however, and human foresight is demonstrably poor. Nevertheless, we as a species persist in overconfidence in our predictions in multiple domains.

-In contrast to hindsight narrative or overconfident foresight, an underappreciated proportion of existence derives from luck. A corollary of this fact is that reward structures at the top (CEOs) and bottom (the unemployed and disabled) of an income pyramid should probably be rethought.

-The mere fact of focusing on something artificially raises its profile: "Nothing is as important as we think it is as we are thinking about it."

Psychology is commonly defined as the study of the mind, singular. What Kahneman implies with Thinking Fast and Slow is perhaps the most profound legacy of his friendship with his longtime collaborator: by calling on people "around water coolers" to hold each other accountable for watching themselves think and for spotting each other's potential self-deception, Kahneman turns psychology -- and perhaps thinking itself -- into a communal enterprise. Bigger than prospect theory, heuristics, and the other theoretical breakthroughs of this one man's brilliant career, the potential of a renewed collective discourse could be the farthest reaching paradigm shift of all generated by what Kahneman and Tversky thought about together, at whatever speed. 

Wednesday, May 02, 2012

April 2012 Early Indications: The Robotic Moment

Everywhere I look, it seems, all I see are robots. Not literally, but while the Apple narrative consumes lots of headline space, there's an incredible surge in innovation in the field of "unboxed computing." If we take the analogy of computers as brains VERY loosely, then the body metaphor is being built out: hydraulics, motors, and actuators as muscles, with sensors as eyes and ears. A robot can be thought of as a computer moving in three dimensions and interacting with the world through more than a keyboard and mouse.

Many of these developments have been in the works for a long time; Boston Dynamics (about which more in a moment) was founded 20 years ago. At the same time, the pace of progress can be dizzying: the DARPA autonomous vehicle grand challenge in 2004 saw only minimal, partial success, while in 2005 5 vehicles completed the course and now only 6-plus years later, Google's self-driving cars are pushing 200,000 miles of testing on public roads. I think these individual stories coalesce into something very big indeed:

-Self-driving cars got a lot of attention as Google turned up the publicity engine last fall. Nevada even has a distinctive license plate for such vehicles, and the Department of Defense has committed to using autonomous vehicles on land, under the sea, and in the air. In fact, the critical role of drone aircraft in the Obama administration's foreign policy -- without much public debate or legal clarity -- may be one of the signature facts of this presidency.

-Amazon first bought Zappos, one of Kiva Robotics' major customers, in 2009, then earlier this year, purchased the MIT spinout itself for $775 million. Jeff Bezos has been called many things, but "slow" and "clueless" aren't usually on the list.

-Medical robotics is a huge area all by itself. Stock-watchers know about Intuitive Surgical's otherworldly share price given the robust market for its operating room systems, but mind-operated prosthetics, for example, will also likely become increasingly common, particularly given how many military amputees are young and otherwise strong.

-One promising development is the advancement of the Robot Operating System, an open-source effort. The University of Washington has shipped seven identical Raven surgical robots across the country to serve as a common research platform. And at Willow Garage in Palo Alto, the Personal Robot 2 uses the same ROS to build robots as teaching platforms for universities and other research entities around the world to build on another shared platform. The PR2 is truly amazing: scroll to the bottom of the page and watch it play billiards.

-Robots for personal care are advancing rapidly. Just as they do with Roomba vacuum cleaners, people can get deeply engaged with an inanimate device. Sherry Turkle of MIT talks with valid concern about how "we expect more from technology and less from each other" in her recent work and robots serve as fascinating screens for human projection.

-Another strand of work is aimed at teaching robots more about those human cues. Heather Knight is an MIT Media Lab alum now affiliated with Carnegie Mellon, and her "social roboticism" includes a humanoid robot that tells jokes and otherwise interacts with its audience.

 -Robots are not just an "it," they can be plural. The GRASP lab at the University of Pennsylvania has done impressive work with swarms of flying bots that can maintain formation, compensating for the loss of a squadron member by resetting formations on the fly, as it were.

-A common theme is many of these efforts is the future of warfare. It's one thing for an iRobot PackBot to safely disarm explosives, but what happen when bots become suicide bombers? Boston Dynamics has created some incredible machines, largely in the DARPA orbit, including an 18-mph 4-legged cheetah-like creature that may haunt your dreams. As always in the complexity of war, offensive and defensive capabilities could easily become indistinguishable in a given circumstance. Do drone pilots on American soil, or packbots, enjoy Geneva Convention or other status? What happens when a robot is deployed as an instrument of torture?

One implication of this collection of activities is the solidification of DARPA as potentially the company's pre-eminent innovation engine. After the era of Bell Labs, Xerox PARC, and other corporate R&D centers, only IBM labs remains as a US industry-based research investment at scale. DARPA, meanwhile, had the Internet explode in public impact about 25 years after launching the ARPAnet in 1969, then saw GPS migrate into invisible ubiquity in everyday life somewhat faster. DARPA's impact through robotics will make a powerful trifecta.