Early Indications May 2013 First-world problems: Too much choice

Several smart people have written about choice as a paralyzing force in western consumer economies. A recent experience reminded me of a great consulting parable that has stuck with me for nearly 20 years, and those combined to raise some thoughts about some of the costs we bear for excessive choice.

1) Many of you know Jim Gilmore's work; along with Joe Pine, he wrote The Experience Economy. A long time ago, I worked with him at my first consulting-firm job. He told a great story that I can't find on line, and given that this is years later, I'm no doubt getting some of it wrong. If you read this and I got something wrong, sorry Jim, but the insight is still a good one, and I trust I'm true to the spirit of the tale:

Several businesspeople sit down in a hotel bar in New Orleans and a waitress approaches the table. "Hello everyone - may I bring you anything to drink?" One guy speaks up. "Yes, I'd like a draft beer." The waitress had heard this before, and responded semi-automatically. "I'm sorry, but we don't have any beer on tap. We have a great list of bottled beers though," and rattled off a long list of macro- and micro-brews. "Which of those may I get you?"

There's a reason consultants get a reputation, but knowing the guys in question, I doubt the following exchange was done in a snarky fashion. "I asked for a draft beer, and you don't have draft beers," the man confirmed. "No, we don't, but we have bottled beers," and the waitress recited the list of bottled brews again.

At the end of said recitation, the patron said gently, "You asked if you could bring us something to drink. I asked for a draft beer. Right across the street, I see the Hyatt bar has Bud on tap. Is it possible to bring a beer over from there?" Never having heard this before, the waitress had to go ask her manager, but a few minutes later, the table got its draft beer, ensuring the anonymous server immortality in consulting lore, not to mention a generous tip for being such a good sport.

2) I was thinking of this story this weekend as I was painting my kitchen. Armed with a fat Benjamin Moore color-swatch book, the lady and I held up tiny paint chip after tiny paint chip, settling on a promising candidate. Not wanting to buy multiple quarts of to-be-discarded colors, I bought a gallon of the I-hope-it-will-look-good chip color. After bringing it home to try on the wall, though, the paint was . . . wrong. We decided to try to lighten it to to the next higher swatch on the card of six colors. At the store, I was surprised to discover that I would need FOUR gallons of white to dilute my gallon one shade, but the paint guy (at Sherwin Williams, who used the Moore colors with no problem) was able to add green tint to make the color less rosy in just the gallon I'd already bought.

Back at home, the color was true to a neighboring chip -- the clerk did a great job, at no charge -- but still not right. After some on-line grazing of kitchen color advice, we found the Moore book had a tiny subsection called America's Colors and in that limited palette of maybe 25 colors, there was a color that looked good in pictures, looked good on a swatch, and, I can report, looks good on the walls (not to mention my hands, calves, and hair).

3) In both of these vignettes, the point is the same, and echoes the provocative TED talk by Barry Schwartz on the tyranny of too much choice (http://www.ted.com/talks/barry_schwartz_on_the_paradox_of_choice.html). People don't want infinite choice (not just because of the mental-health implications Schwartz outlines). No, people do not want infinite choice: they want what they want.

What does this idea have to do with technology and business? Two main ideas come to mind. First, extensive choice adds to the customer service burden: ordering a beer from a list of 75 is harder, takes more time, and requires more guidance than ordering from a list of, say, 10 brews. Restaurant servers get asked all the time, "what's the Basement Brewery Old Wheaties taste like?" and skilled waiters and waitresses have a variety of useful answers at the ready: a) compare it to something less obscure, b) offer a personal testimonial or diplomatic warning, or c) offer to bring you a sample. Each of these makes his or her job harder than it might otherwise be, but the burden of service is higher in a high-choice scenario, whether in wedding dresses, paints, restaurants, or car shopping: before Ford rationalized the optioning process, the 2008 F-150 pickup came in more than a billion possible combinations.

Recommendation engines can help cut through the noise. Amazon's recommendations are so good they sometimes hit  too close to home; eBay's are pretty generic and/or too obviously linked to my last visit, which may have been a one-off (e.g., 2005 Toyota Camry door handle). Netflix and iTunes are working hard on this set of technologies, but for domains outside of media, it's hard to build a sufficiently robust profile. No great recommendation engines jump to mind for financial services, consumer electronics, or other expensive, highly complex purchases.

Another alternative to requiring more and more skilled service is to emphasize design. Apple has basically three mobile devices, with varying amounts of memory/storage, usually two colors, and a few connectivity options. Android, meanwhile, offers more than 4,000 different devices. While one may be EXACTLY what I need, cutting through the noise to find it is non-trivial at point of purchase. In the supply chain, meanwhile, keeping parts, contractors, documentation, and manufacturing expertise all current (never mind optimized) for so many devices to be sold in so many geographies is a significant challenge. Furthermore, the network effects that result from a shared platform are constrained somewhat because of Android sub-speciation. An app built for a touch screen display of size x won't render quite the same way on a size y display, and keyboard-driven devices don't enjoy full reciprocity with the glass keyboard on the touchscreen models. I often do an experiment in my classes: swap Android devices with a stranger. Now try to open a familiar app, send a text, or call someone. Very few Android users feel comfortable on kindred but non-identical devices.

Apple, meanwhile, has made iPods, iPhones, and iPads look and feel like siblings; the learning curve, while cumulatively considerable (think about being handed an iPad in 1999), is continuously compatible. The exception is the current Apple TV remote: unintuitive, slow, too easily lost, not predictably responsive. I mention this because Microsoft's next Xbox, revealed last week, was demoed with a convincing, lag-free voice interface. All the Google Glass jokes notwithstanding, voice control will have its place -- but in the living room, most likely, rather than on the subway, in restrooms, or at the grocery.

Voice control, as anyone who has used IVR trees can attest, is non-trivial to get right, but does have the potential to render the negative implications of device proliferation less onerous. In the meantime, especially when dealing with the digital Swiss army knives of our era, consumers will continue to face a bewildering array of choices until Samsung and/or Google get a handle on simplification. At the macro level, choices are often easy for vendors to generate, particular insofar as they are increasingly defined by software (adding another menu item is "free" to the developer), but consumer frustration would suggest that the insights of the Don Normans, the Barry Schwartzes, the Bruce Schneiers, and the Jered Spools of the world are falling on infertile ground. In plenty of situations, less is truly more.

Early Indications June 2013: What gets measured, gets . . .

It has become a business truism that “what gets measured, gets managed” after the great Peter Drucker allegedly wrote it. (There is no citation, however, and it may be that original credit goes to Lord Kelvin, who stated that “If you cannot measure it, you cannot improve it.”) In the “big data” era, it has become an article of faith that the more measurements we can gather and presumably analyze, the more we can optimize behavior that drives medical outcomes, social welfare, and corporate profitability. While I believe that we will see some extremely positive validations of this hypothesis, there are also enough cautionary tales that suggest some skepticism is warranted before accepting the promises of the big data evangelists as articles of faith.

Five unrelated examples combine to suggest an alternative mantra:

what gets measured, gets gamed.

That is, the scorecard gets attention _at the expense of_ the nominal task that was being measured in the first place.

  

Example 1: A former student reported that forecasting tools in a consumer products company were generating remarkably consistent projections, regardless of seasonality, competitors’ new product launches, or other visible alterations to the landscape. After some investigation, it was determined that a specific forecast curve had become popular (whether with procurement, finance, marketing, or plant managers was not made clear). To generate the “acceptable” forecast month after month, analysts took to [essentially] defeating the forecast algorithms by adjusting past actual quantities: to get the future curve they wanted, employees rewrote history.

Sales forecasting is gamed by definition, given the way commissions, market uncertainty, and expectation management affect the process. Numerous attempts have been made to induce “best-guess” estimations by the sales people, but even those companies that deployed prediction markets reported mixed results.

Example 2: For a time there was a breed of financial planner who was paid not on the basis of his or her clients’ rate of return, but by commissions generated by equities trades. Not surprisingly, clients did not get advice based on the long-term growth of their portfolio, but on the hottest stock of the moment. Moving clients in and out of different equities based on magazine cover stories proved to be good business for the planners, and only incidentally and accidentally profitable for the clients.

Example 3: A former colleague of mine recently analyzed the marketing activities of a large technology company. Even though the company sells B-to-B with a direct sales force, an executive dashboard someplace measures website clicks. The word came down through marketing that each product group had to “win the dashboard,” in this case, piling up web clicks through heavy ad placement even though this behavior could in no way be tied to revenue, customer satisfaction, or even lead generation.

Example 4 comes from closer to home. Course evaluations have become the focus of many universities’ professional assessments of non-research faculty, trying to ensure that students feel the instructor did his or her job. At Penn State the forms are called not “course evaluations” but SRTEs: Student Ratings of Teacher Effectiveness, though I doubt I am alone in believing the E stands for Entertainment. In my last teaching job, now 20 years ago, I was known to game the evaluation process, bringing cookies for the class on the last day before passing out paper evaluation forms. In our modern age, however, the assessment has gone online, so students are able to fill out the forms at their convenience and administrators can get scores reported in days rather than the months it took to code paper instruments.

At Penn State, the move toward paperless assessment has coincided with a startling drop in the completion rate. Like some other schools, we have an institute for the advancement of teaching skills. Upon seeing the drop in SRTE completion, our center undertook a project to try to improve compliance with the assessment. Note that these efforts do nothing to improve pedagogy or understand why compliance is dropping; the focus on the course assessment process is completely unrelated to helping students learn. Once again, the tail is wagging the dog.

Example 5: Information technology has become the backbone of most modern organizations. Grading the performance of the IS group, however, is extremely difficult. In many IS shops, measuring system uptime is readily quantifiable and usually scores in the high 90s. (For reference, 99.5% is a great score on a test but in this context it means the system was down for almost two full days a year.) What is much more difficult to measure, yet more important to business performance, is whether the right applications were running in the first place, how much inefficiency in the data center was required to get the gaudy uptime number, or how good the data was that the system delivered. Information quality is one of those metrics that is incredibly hard (and sometimes embarrassing) to measure, hard to improve, and hard to justify in terms of conventional ROI. Yet while it is, more often than not, truly critical for business performance, information quality was not in years past a component of a CIO’s performance plan. I’m told the situation is changing, although measuring application portfolio management – how well IS gets the right tools into production and the old ones retired – remains a challenge.

The five examples, along with many others from your own experience, suggest two important lessons. First, more data will by definition – thank you Claude Shannon – contain more noise. As Nassim Taleb notes in his critique of uncritical big-data love, more data simply means more cherry-picking (and not, Nate Silver would add, better hypothesis generation).

Second, in the domain of human management, incentive structures remain hard to get right, so there will be more and more temptations to let “numbers speak for themselves.” Such attitudes can emphasize the most readily measured phenomena, often of activity rather than outcomes – web clicks are easier to count than conversions; sales calls are easier to generate than revenues; incoming SAT scores are easier to average than student loan debt or job placement rates of the graduating class.

One would hope that getting the assessments right, even though it usually means counting something that doesn’t look as good, should dictate performance assessment. Given so much evidence from the worlds of medicine, commerce, sports, the military (remember Robert McNamara's "kill ratios"?), and academia to the contrary, however, it would appear that these games will forever be with us.

April 2013 Early Indications: What are humans good at?

"So we will be producing about 1026 to 1029 cps of nonbiological computation per year in the early 2030s. This is roughly equal to our estimate for the capacity of all living biological human intelligence ... This state of computation in the early 2030s will not represent the Singularity, however, because it does not yet correspond to a profound expansion of our intelligence. By the mid-2040s, however, that one thousand dollars' worth of computation will be equal to 1026 cps, so the intelligence created per year (at a total cost of about $1012) will be about one billion times more powerful than all human intelligence today. That will indeed represent a profound change, and it is for that reason that I set the date for the Singularity—representing a profound and disruptive transformation in human capability—as 2045."

-Ray Kurzweil, The Singularity is Near (New York: Penguin Group, 2005), pp. 135–136.

In the manner of all true technology revolutions, this one has crept up on us. I speak of the vast array of ways that computing is augmenting human life. Rather than think of people and computers, or people and robots, I believe it makes more sense to think of a continuum, with a naked newborn on one end -- pure human, zero augmentation -- and 2001's HAL or an Asimov fictional robot on the other: pure cyborg, with ample human characteristics. Everywhere in between these two poles, we can see combinations of human traits and computo-mechanical assistance. For now, humans call the shots in most scenarios (but not all; see below) and our devices can assist in any of thousands of ways.

I've never really bought Kurzweil's Singularity hypothesis: that machine capability, whether in the solo or collective mode, will eclipse human capability with profound consequences. The simplistic equation of CPU capacity with "living biological human intelligence" has never been argued with any serious evidence. The fact that Kurzweil has recently become a senior exec at Google, meanwhile, raises some pretty interesting questions.

At the same time, there's a tendency to "privilege" (sorry - it's an academic phrasing) humanity. Every robot that is said to be convincing, for example, serves as evidence that people are somehow Special, if less so every year. At the same time, people impart human characteristics to machines, naming cars and Roombas, for example, but not mobile phones, as far as I can tell. There's a whole lot to be researched and written about how humans anthropomorphize non-humans (animals and now machines/devices), but that's out of scope for the moment.

I think these two viewpoints -- the Singularity school and human exceptionalism -- both carry a substantial sum of unacknowledged baggage.  Kurzweil et al adopt a simplistic understanding of humanity as simply the sum of our circuits. For those who worry about machines overtaking humans, meanwhile, this is not news. Humans are far from the strongest creatures or even strongest mammals, and nobody I know seems to feel the lesser for it. In the realm of mental capacity, computers have resoundingly beat our cerebral gladiators at both chess and trivia. In the realm of the everyday, pocket calculators from 40 years ago outperform everybody at mathematical figuring. Engines, hydraulics, and now computers are clearly better than humans at some tasks.

Here's one example. Inspired by the Economist's cover story of April 20, I conclude that machines are in fact better than humans at driving cars under many circumstances. Consider that calculating the speed of an oncoming car is guesswork for a human: people misjudge their window of opportunity (to make a left turn across oncoming traffic for example) literally hundreds of times every day. LIDAR plus processing power makes that calculation trivial for a computer-car. Knowing the limits of the car's handling, done by feel, is beyond the experience of 99% of drivers, most likely: especially with automatic transmissions, traction control (a computer assist), automatic pitch/yaw compensation, and other safety features, it's very difficult to get a car sideways, deliberately, to figure out how and when to react. For every "driving enthusiast" who squawks in a car magazine about "having all the decisions taken away from us by Big Brother," there will be many people who would LOVE to be chauffeured to their destination, especially when it's the Interstate grind crawling to work at 7:00 rather than the stick-shift byways of rural Virginia on a springtime Sunday morning. Even apart from the Google car, computers are doing more and more driving every year.

Preceded with the caveat that I am not not can I aspire to be a cognitive scientist, there is a question embedded here: what are humans good at, what are computers good at, and how will the person/machine partnership change shape over the coming years? There's got to be research done on this, but I couldn't find a clear, definitive list in plain English, so this fool will rush in, etc.

Let's start with machines: machines can count, multiply, and divide way faster than any person. Time and distance, easily quantified, are readily calculable. Short- and long-term memory (storage) can be pretty much permanent, especially in a well-engineered system, not to mention effectively infinite at Google scale. If-this/then-that logic, in long long chains, is a computer specialty. IBM's Watson, after winning Jeopardy with a really big rules engine, is now being used for medical diagnosis, where it should do well at both individual scenarios and public-health bigger pictures. Matching numbers, data patterns, and text strings is straightforward.

What about people? People can feel empathy. People can create art. People can see visual/logical nuances better than machines: a 5-year-old can know that a spoon is "like" a fork but not like a pencil but a computer must be taught that in explicit terms. Similarly, machine filters that "know it when they see it," in the Potter Stewart sense of hard-core material, have been spectacularly unreliable. People can read body language better than computers can. People can integrate new experience. People can infer better than machines. People can invent: recipe creation software can't duplicate even middling chefs, for example. Computers can be taught to recognize puns and, more recently, "that's what she said" double-entendres; only humans can create good ones.

Anthony Damasio's brilliant book Descartes' Error should be required reading for the Kurzweilians. Rather than accept the Cartesian split of mind from body - embodied in the epigram "I think therefore I am" -- Damasio insists, with evidence, that it is emotion, the blurry juncture of mind and body, that enabled human survival and continues to define the species. All the talk about calculations equaling and surpassing human intelligence ignores this basic reality. Until computers can laugh, cry, sing, and otherwise integrate mind and body, they cannot "surpass" what makes people people.

Here's a nice summary, from INC magazine of all places, in 2002:
*********
    Yet is thinking outside the box all it takes to be innovative? Are reasoning and imagination -- the twin faculties that most of us associate with innovation -- enough for Ray Kurzweil to know which of the formulas that he's dreamed up based on past technological trends will lead to the best mathematical models for predicting future trends?
    No, says Antonio Damasio, head of the neurology department at the University of Iowa College of Medicine. The innovator has to be able to feel outside the box, too -- that is, to make value judgments about the images and ideas that he or she has produced in such abundance. "Invention," as the French mathematician Henri Poincaré said, "is discernment, choice." And choice, notes Damasio, is based on human emotion -- sensations that originate in the brain but loop down into the body and back up again. "What you're really doing in the process of creating is choosing one thing over another, not necessarily because it is factually more positive but because it attracts you more," says Damasio. "Emotion is literally the alarm that permits the detection."
    Kurzweil, for his part, calls that alarm "intuitive judgment." But he disagrees that it -- or reasoning or imagination, for that matter -- is exclusively human. He sees a day in the not-too-distant future when we will merge mechanical processes with biological ones in order to amplify what our brains alone do today. "Ultimately, we'll be able to develop machines that are based on the principles of operation of the  human brain and that have the complexity of human intelligence," he says. "As we get to the 2030s and 2040s, the nonbiological component of our civilization's thinking power will dominate."
**********

As I suggested earlier, the human/machine distinction is far from binary. Even assuming a continuum, however, perhaps the most important category of tasks has been little discussed: computer systems which possess emergent properties that cannot be understood by humans. Wall Street is in this category, given algorithmic trades occurring in the millionths of a second, in a system where the interactions of proprietary codes are occasionally catastrophic yet beyond human comprehension (both in real time and after the fact) not to mention regulation. When algorithmic trades in synthetic instruments inadvertently wipe out underlying assets, who's left holding the bag? Sometimes it's the algorithm's "owner": Knight Capital basically had to sell itself to a rescue party last summer because its bad code (apparently test scripts found their way into the live NYSE and nobody noticed) lost $440 million in less than an hour; the program was buying, in automatic mode, $2.6 million of equities _per second._ Just because people could write algorithms and code doesn't mean they can foresee all potential interactions of said code -- assuming it was written as designed in the first place. (For more see here and here)

I'm not going to get nostalgic, or apocalyptic, or utopian here. Humanity has always built tools, and the tools always have unintended consequences. Those consequences have been substantial before: the rise of cities, extension of human life spans, atomic bombs, Tang. This time around, however, when the unintended consequence cuts so close to our identity it probably means that some self-awareness -- something computers can't do -- is probably in order. On that front, I'm not entirely hopeful: during the recent Boston bomb drama, when people were shown at their worst and finest, news feeds were dominated by updates on a Kardashian divorce development. I don't know if we're "amusing ourselves to death," as Neil Postman put it a long time ago, but maybe there's the chance that some people will dumb themselves down to computers rather than the machines catching up.

Early Indications March 2013: Digital Heirlooms

I've been thinking a lot lately about the invisible consequences of our smartphone/mobile/digital world. Somewhere down the road, the dematerialization of cultural artifacts will be viewed, I believe, as a major shift. Looking back from today, books are our oldest mass cultural form, then between 1880 and 2000 we got music, movies, then television into widely available portable formats. Eventually, and rapidly, all of these became digital, and fungible: 15 years ago the radio couldn't play back voicemail nor could a VCR host video games.

The business competition between Amazon, which won the first leg of the e-book/e-reader race, Netflix (ditto for movies), and Apple (music) is for extremely high stakes, but not our concern today. As the barrier to cultural creation drops, artifacts get easier to make. Compare the process of creating photographs in 1913, 1963, and today. Humanity has never made -- or shared -- so many images, but how will these increasingly ephemeral artifacts get passed down? Finding one or two photos of my grandfather when he was a young boy was lucky and important; in 100 years, what will my grandkids have to show for their infancy, adolescence, and young adulthood?

Google's recent decision to drop the Reader product is instructive here. At what point do changing cloud computing business models endanger and/or support preservation? Is there any conceivable way Facebook can keep adding billions and billions of photo uploads in perpetuity? Given that some kind of limits will be reached, where do our cloud-identities go when businesses fail? As more and more variations emerge, what will be the fate of digital personae after we die? We may well confront a paradox: we make more images than ever before, yet in the future, we could have less of a visual inheritance.

A whole other branch of issues revolves around platform compatibility. Some of my written masterpieces from the 1990s are stuck on 3 1/2" floppies for which I no longer own a working drive. That's a hardware question. What about software compatibility? For how long will Adobe support the PDF standard? In the absence of such support, and the possibility that a given standard will not be open-sourced to a community that can maintain it, we will see further stranding of digital assets.

In such a world, what lasts? I was pondering this question when considering graduation gifts. An Apple device, no matter how sleek and easy to hold, will be obsolete in five years, maybe before. Music is hard to give: for how long can we assume most every household will be able to play a CD? The last two computers I bought, not to mention every tablet, lack the capability. One day I will wake up and realize, yet again, that there is another format of information I can't access, joining the floppies, VHS, Jazz, and Zip media boxed up, worthless, in the basement.

Books have played a huge role in my life. Leaving grad school, the moving company found that our books on the van outweighed the car that was also on the truck. Many books tell a story, independent of the printed page. Bookplates were a classy accoutrement of prior generations; inscriptions can still be precious.  But the fact remains that, apart from university press books, most paper rots, some startlingly quickly. Books weigh a lot and occupy substantial space. The stereotype of a book-lined academic household is giving way to cloud-ish realities: it's quicker to consult Google to hunt down a footnote than to drive to the campus library or plow through the boxes in the garage, given that my book collection currently surpasses my available wall space for shelving it. Much as I hate to admit it, books are losing their appeal for me as gifts, especially "special" ones. The good news is that books' operating system is now stable, and is likely to remain so.

To return to the question, what lasts in a digital world? Paper is a mixed blessing, but Moleskine has made a very profitable global luxury brand out of blank books (if you are a fan of the Italian-made gems, check out this fascinating article related to the company's upcoming IPO). Pens continue to satisfy; alongside the European classics, several Kickstarter businesses growing out of the cult following that has emerged around the 0.3 mm Pilot Hi-Tec C are fascinating to track. I don't watch people in their 20s and 30s closely enough to know whether pens are being replaced in the preparation of grocery lists, birthday cards, or journals, but sense they are not. (From "Dear Diary" to "Dear Evernote"?) Relating pens to a broader category, tools can be truly lasting gifts, the antithesis of digital ephemera. Specifically, bladed tools seem to hold some deep appeal: knives, kitchen or otherwise, and chisels/planes strike me as heirlooms more than, say, striking tools ("here son, a titanium framing hammer as your graduation present"), mechanics' tools, or even saws. In the grooming arena, shaving razors, and those lovely badger brushes, seem to continue the theme. Scissors, whether run with or standing still, don't hold the same appeal, but I don't write as a quilter or scrapbooker, for whom such tools might indeed be long-lived, essential, and personal.

Ah yes, say some women friends, you're so much of a guy, always missing the point: jewelry has struck a nerve for millennia. Gold, precious stones, and other articles of adornment appeal deeply to many women from many cultures. To this I say: true, but "little jewelry" is an oxymoron in my experience. Finding something well-made, lasting, and appealing for the same price as a Swiss Army knife or decent "graduation" pen has been difficult for me. There's also the strong sentimentality: giving jewelry to the babysitter graduating from high school feels a little too personal. Tools have a safety zone that rings do not. In both cases, however, the appeal relates to hands: things that people before us touched, treasured, and took care of mean so much more than something shiny and new -- unless we can imagine the new present enduring across generations.

The essential role of blades in our species' survival speaks to some deep parts of the psyche located, I suspect, far removed from the dopamine pumps so capably triggered by multitasking, texting, tweeting, and online grazing. To the question of "what lasts in a digital age?" the answer, I submit, is simple: tools that fit the hand of the user. Or gold.

Early Indications February 2013: Big Caveats Regarding Big Data

Review essay

Michael Mauboussin, The Success Equation: Untangling Skill and Luck in Business, Sports, and Investing (Boston: Harvard Business Review Press, 2012)

Nate Silver, The Signal and the Noise: Why So Many Predictions Fail — but Some Don't (New York: Penguin, 2012)

Nassim Nicholas Taleb, Antifragile: Things That Gain from Disorder (New York: Random House, 2012)

Preface
To set context, here is a sampling of what IT vendors are saying about Big Data:

SAP:
Get the tools and technology you need to harness big data from any source – structured or unstructured – for a serious competitive advantage. Our big data solutions can help you capture, analyze, report, predict, and visualize mammoth volumes of data instantly – so you can make the best possible business decision, every time.

IBM:
Big data is more than simply a matter of size; it is an opportunity to find insights in new and emerging types of data and content, to make your business more agile, and to answer questions that were previously considered beyond your reach.

SAS:
The hopeful vision for big data is that organizations will be able to harness relevant data and use it to make the best decisions.
Technologies today not only support the collection and storage of large amounts of data, they provide the ability to understand and take advantage of its full value, which helps organizations run more efficiently and profitably.

Oracle:
For decades, companies have been making business decisions based on transactional data stored in relational databases. Beyond that critical data, however, is a potential treasure trove of less structured data: weblogs, social media, email, sensors, and photographs that can be mined for useful information.
Oracle offers the broadest and most integrated portfolio of products to help you acquire and organize these diverse data sources and analyze them alongside your existing data to find new insights and capitalize on hidden relationships.
--------
In sum, the vision of the Big Data movement appears to be to as follows:

to measure and capture, in greater detail and quantity, things that have happened in order to analyze the data, find insights/answer hard questions/capitalize on hidden relationships, and act more effectively in the future ("make better decisions.")

It all sounds reasonable, except the foundational logic has yet to be tested. Fortunately, we have some very smart people from diverse backgrounds who can help in that quest. It turns out that if these three gentlemen are correct, the very premises of Big Data need to be tempered, not with better computer science, but a better comprehension both of how people think, act, and decide and of how much luck and randomness still shape our world.

The three books all overlap to a degree, often in their appreciation for the behavioral economics of Daniel Kahneman, and each author brings serious credentials to the table:

-Mauboussin teaches at Columbia in addition to working at Legg Mason; he wrote an early and influential report on the financial implications of power laws back in the late 1990s.

-Silver gained fame on election night 2012 after correctly calling 50 out of 50 state results in the presidential race, after going 49 for 50 in 2008. His first data-centric venture was in baseball statistics.

-Taleb's previous books, The Black Swan and Fooled by Randomness, provided prescient color commentary to the financial crisis of 2008. Stylistically, existentially, and intellectually, he swims upstream but has repeatedly been proven right.

Three macro-level insights emerged from these books.

A) Luck remains a critically important factor in success, so prediction, even when successful (that is, skillful), may not generate much advantage

Mauboussin looks at the relationship of luck and skill in a variety of domains. The book owes many debts to Moneyball, but across more sports and extending convincingly into business. Results in the NBA, for example, are decided by skill to a much higher degree than in the NHL: in 2-1 or 1-0 games on ice, the slightest deflection or fluke play can determine a game. When the Spurs beat the Suns 104-98, however, random chance events are fewer (how many deflected shots actually go through the hoop?) and their impact is minimal.

When he turns to business and investing, Mauboussin makes similarly compelling points. For our purposes, the central insight relevant to Big Data concerns what might be called the water level: as the skill level rises in a population, differences between competitors shrink. Thus luck becomes more of a factor: "if stocks are priced efficiently in the market, luck will determine whether an investor correctly anticipates the next price move up or down. When everyone in business, sports, and investing copies the best practices of others, luck plays a greater role in how they all do." (p. 56)

This insight would seem to apply to the algorithmic arms races in baseball talent scouting, investing, and consumer data mining. In situations where no actor can accumulate a commanding lead (as Google has and Facebook might, however), whether in computing horsepower, algorithmic quality, or data to be analyzed, the skill premium dissipates. Luck, by this theory, will play a greater role than skill in a more heterogeneous environment.

Mauboussin concludes the book with 10 suggestions to improve the "art of good guesswork":

1 Understand where you are on the luck-skill continuum

2 Assess sample size, significance, and swans

3 Always consider a null hypothesis

4 Think carefully about feedback and rewards [many financial advisors get paid when clients trade, not when clients prosper, for example: what's the feedback loop there?]

5 Make use of counterfactuals

6 Develop aids to guide and improve your skill [checklists are a case in point]

7 Have a plan for strategic interactions [such as asymmetric warfare or disruptive innovations]

8 Make reversion to the mean work for you

9 Develop useful statistics

10 Know your limitations

This tenth maxim serves as a convenient segue to Silver's book.  From its title -- signals and noise are fundamental to information theory -- to its examples (which include economics, earthquakes, and climate change), the book would appear to be enthusiastic about using numbers to predict the future, to realize the promise of Big Data. But as Silver writes very early in the book, his focus is less on data and more on the people who use it:

"Big Data _will_ produce progress -- eventually. How quickly it does, and whether we regress in the meantime, will depend on us. . . .
Our biological instincts are not always very well adapted to the information-rich modern world. Unless we work _actively_ to become aware of the biases we introduce, the returns to additional information may be minimal -- or diminishing." (pp. 12-13)

Thus, the second macro-level idea concerns consciously testing ideas, assumptions, and admitting uncertainty.

B) Bayesian statistics, in particular its insistence on carefully articulated prior probabilities, forces human analysts to attach values to the context for their predictions rather than let them float ahistorically, otherwise known as "letting the data speak for itself."

This illusion of statistical sufficiency known sometimes as "frequentism" dates to the early 20th century, and the school of thought persists today. As Silver summarizes, "it emphasizes the objective purity of the experiment -- every hypothesis could be tested to a perfect conclusion if only enough data were collected. However, to achieve that purity, it denies the need for Bayesian priors or any other sort of messy real-world context." (p. 255)

Echoing his opening assertions in the conclusion, Silver plausibly argues that "distinguishing the signal from the noise requires both scientific knowledge and self-knowledge: the serenity to accept the things we cannot predict, the courage to predict the things we can, and the wisdom to know the difference." (p. 453)

Possibly because Silver's book ranges more widely than does Mauboussin, it felt more engaging. Written as it was before his successful handicapping of the Obama re-election, The Signal and the Noise is itself something of a prior: a self-aware assessment of Silver’s own methods and their probabilistic limits. The book forces erstwhile predictors to examine their methods, their objectives, and ultimately themselves -- not at all what the two-dimensional stat-geek stereotype would suggest.

In contrast to Mauboussin, Silver offers but two admonitions in his conclusion:

Know Where You're Coming From

and

Think Probabilistically.

In contrast to closed-end events -- when will be the first snowfall, who will win the championship, how many widgets will Samsung sell -- open-end events are the terrain of Nassim Nicholas Taleb: Black Swans, as they have come to be called. As Silver notes, nobody can remotely predict earthquakes or most natural phenomena, with the exception of weather. Nor can political revolts (in either London or Cairo, for example), equity or currency fluctuations, or other large-scale man-made phenomena be forecast at all reliably. Rather than predicting, Taleb advocates an entirely different approach.

C) Because of the nature of a highly complex and connected world, "Black Swan" events can generate very large, unforeseen effects, very quickly. A prudent strategy for living in such a world is to seek shelter to a substantial (but not complete) degree, while finding exposure to the upside of unforeseeable events with small bets in as many big-multiplier arenas as possible, often via optionality. Taleb calls this a "dumbbell" strategy for its bimodal distribution: for example, very large positions in cash or other low-risk and low-reward instruments, with focused but small investment in high-risk/very high-reward (and thus probably exotic) positions. Note that the middle is avoided entirely: Taleb's antipathy for bell curve distributions, especially where misapplied, is vehement.

The title of Taleb's book hints at how unaccustomed we are to thinking this way. Everyone knows that a wine glass is fragile: physical volatility is usually fatal. Note that fragility scales non-linearly: a fall from 32 inches onto the hardwood floor is far more than four times as damaging as an 8-inch drop, which is likely survivable. Many people say that "robust" is the antithesis of fragile, but Taleb disputes this position: what, instead, are the opposite of fragile phenomena, the things that actively IMPROVE in the presence of volatility? He looked in dozens of languages: none had a word to connote this property, which is, nonetheless, quite real. Taleb's contrarian-ness is of a high order indeed.

It turns out that the natural world, biology in particular, abounds in situations where volatility improves matters. Young children learning language, muscles after exercise, and immune defenses all qualify. In the human order, Taleb praises the Swiss city-state (canton), noting that many people can pick Switzerland as the most stable regime on earth and yet nobody outside the country knows who the president is: decentralized authority keeps the scale of both problems and solutions closer to human-friendly and risk-limited. Swiss disorder occurs in domains the exact opposite of "too big to fail," itself a curse in this system of thinking because increasing scale implies massive risk. Man-made "stabilization" often leads to instability, whether in financial systems, forest fires (preventing healthy little ones means a later, inevitable inferno), or corporate planning. When small, routine failures are prevented through naive bureaucratic intervention, stressors magnify until the impact is multiplied to the scale of the entire system (as witness the mortgage banking mess, rogue traders at Societe Generale and JPMorgan, and the flash crash). And on the basis of what empirical evidence is "equilibrium" the economist's ideal?

Thus rather than fail to predict the mechanism of [by-definition] unpredictable disaster, we can see the quite foreseeable effects of 100-year-old subway tunnels in New York (whether the stressor is a riot, a terrorist, or a hurricane is irrelevant), or slow responses to climate change, or overly long supply chains for food. In short, Taleb proves that prediction is systematically broken for both psychological -- yes, Kahneman gets his props here too -- and systematic/organizational reasons. The 425-page excursion into many nooks and crannies of the Western intellectual tradition (Seneca plays a featured role, for example) is itself unpredictable: Taleb does not so much explicate his argument as embody it, with frequent personal examinations that prove he literally has skin in the game. His conclusion is much more straightforward that its telling:

"Everything gains or loses from volatility. Fragility is what loses from volatility and uncertainty." (p. 421)

Rather than seek certainty in data* or in anything else, Taleb seeks to find situations, investments, and modes of living that are not only resistant to volatility but _thrive_ in its inevitable presence. The notion of antifragility thus stands as the most robust challenge to the uncritical application of data, algorithms, and prediction more generally -- especially outside realms (such as weather) where we can actually document a certain degree of success. As for lavish investments in police and fire departments for cities, in R&D at the corporate level, and in universities in any particular society, Taleb contends that we really know little about correlation vs. causation. This fundamental lack of evidence suggests that for data to improve our world, there are more intermediate steps between the computer and a better future than the apparent consensus would suggest.


*"There is a nasty phenomenon called 'Big Data' in which researchers bring cherry-picking to an industrial level. Modernity provides too many variables (but to little data per variable), and the spurious relationships grow much, much faster than real information, as noise is convex and information is concave." (p. 418) More simply, "the more data you get, the less you know what's going on." (p, 128)

Early Indications January 2013: A Plethora of Predictions

Longtime readers know that I have in some years written prediction newsletters, then followed up 11 months later to grade the list. This year, we'll look at several prediction lists from other people to see what commonalities, divergences, agendas, and insights might emerge. And then I'll offer a modest prediction list of my own.

Assembling predictions was extremely easy, so I settled on a random sample of six lists, ranging in size from five items to about a dozen. (Why so many people feel the need to predict is probably worth puzzling over.) Herewith are the lists:

1) Time

Augmented reality will go mainstream
Google’s Chromebooks will get more consumer attention, gaining traction in 2013
Hybrids and convertibles will see high interest from IT departments
7-in. tablets will dominate tablet sales
Apple will create a hybrid tablet-laptop
Samsung will invest in its own future [in the form of a mobile OS]
Microsoft will get into smart-phone hardware
Apple will make a large investment in its supply chain
Google will go fully vertical with Motorola
RIM will make a modest rebound but will eventually be acquired

2) Deloitte
I quote:

Disruptors are opportunities that can create sustainable positive disruption in IT capabilities, business operations, and sometimes even business models.

    CIO as the Postdigital Catalyst
    Mobile Only (and beyond)
    Social Reengineering by Design
    Design as a Discipline
    IPv6 (and this time we mean it)

Enablers are technologies in which many CIOs have already invested time and effort, but which warrant another look because of new developments or opportunities.

    Finding the Face of Your Data
    Gamification Goes to Work
    Reinventing the ERP Engine
    No Such Thing as Hacker-proof
    The Business of IT

3) Shel Israel in Forbes

Apple's edge will dull
The year of wearables
They'll know who we are [mobile + social + big data + sensors + location = predictive context]
Buzzwords will die
Data contracts and margins will erode [for telcos]

4) The Telegraph
Surprises are guaranteed in:

Mobile
Microsoft
TV
Security
Facebook monetization
Wearables

5) ZDNet

3D printing
Augmented reality
[Apple] TV
Anonymous/Wikileaks revival
Big data gets bigger
BYOD soars at work
Tech giants go BRIC
Rim and Nokia declare themselves
Apple faces testing times
Internet of things
T-Mobile+MetroPCS, Sprint+Clearwire
Cloud computing growth meets legal tangles
Yahoo bets it all on mobile?

6) Gartner

Mobile devices battle for market share as mobile > PC
Mobile applications & HTML 5 [fight it out]   
Personal Cloud means services > devices
Internet of Things [esp. embedded sensors, image recognition, and NFC]
Hybrid IT and Cloud Computing
Strategic Big Data
Actionable Analytics
Mainstream In-Memory Computing
Integrated Ecosystems   
Enterprise App Stores

Commonalities

The PC is the platform of the past -- except possibly as a tablet hybrid. Note how every list assumes the primacy of the mobile device. Even so, the PC industry remains big, if highly contested and low-margin.

The TV looks like the next battlefield.

Even when it's not mentioned, cloud computing is assumed in the need for powerful mobile capabilities. A trend is truly powerful when people stop talking about it and just use it: NFC fails on this score; maps/location are huge by this metric.

Several companies face make-or-break (or at least win-or-give up the lead) moments in their corporate history: Apple is no longer "cool" or singularly cool. Microsoft must prove it can win off the PC desktop. Nokia and RIM may both be gone by year end. Google has many irons in the fire but neither Chromebooks nor Android nor Glass nor maps nor YouTube has proven to make serious money. Yahoo can't have many more lives (or CEOs) left to play out if it doesn't crack the mobile code.

Google's wearable/augmented reality play successfully drove very strong PR -- just as search used to do with journalists 15 years ago.

Divergences

Other than differences in focus -- consumer gadgets vs CIOs -- there wasn't a lot of overt disagreement among the lists.

ZDNet is the only list that mentions 3D printing, which, while not the second coming of Watt's steam engine, is going to make a difference in the long run. Possibly much like GPS or WiFi, there won't be a breakthrough moment but rather one day we will simply notice that it's extremely widespread and wonder why there wasn't much fuss.

I was surprised there wasn't more emphasis on security. Deloitte's tagline, "no such thing as hacker-proof," gets at the conundrum of BYOD, mobile access, heterogeneous computing and networking environments, effectively infinite horsepower for bad guys to crack passwords, substantial investments by nation-states in cyber-disruption (or worse), etc.

Agendas

Just a word here on what might drive these lists. ZDNet presented theirs as a slideshow, which does wonders for page view counts even when the photos did nothing to illustrate mangy of the abstract concepts: how many more data center server racks must we see to illustrate Big Data? Shel Israel wants to sell books, and his list reflects his current research themes: fair enough. As for Deloitte, the firm clearly needs to sell consulting services to CIOs, but the list was both obtuse (what is a "post digital catalyst"?) and partially wrong: I would wager serious money that enterprise gamification will only be marginally more prevalent a year from now, if that.

Insights

The business landscape is extremely volatile. "Built to last" is getting harder to find, particularly in information industries. Leaders in one quarter can lose their advantage very quickly, and stock market responses can be irrational: Amazon missed its numbers pretty badly and its stock prize rose while the fall in Apple's share price is hard to square with objective performance. In that context of volatility, speed, and unpredictability, here are my seven predictions for 2013:

1) Value chain realignment
Whether at Amazon, Apple, Comcast, Dell, Google/Motorola, HP, Microsoft, or Samsung, the mobile-first world is forcing hardware, software, services, content, and carriage to align in new ways. Each of these players is experimenting with extensions and reconfigurations of its original core value proposition. We will see more.

2) Cyber insecurity
The latest James Bond move had strands of Stuxnet, Wikileaks, Anonymous, and the Internet of Things effectively woven into a plausible tapestry of cyber-threat. We will see new unpleasant and potentially dangerous effects of global interconnectedness and reliance on digital records, personas, and entertainment.

3) CIO speciation
For all the talk about IT as a tool for competitive advantage over the past 20 years, for every Amazon there are dozens of companies that use computers to schedule production, ship orders, and pay bills. Rather than insist -- annually -- that the latter will grow to resemble the former, I think it's realistic to see that there are several kinds of enterprise computing scenarios. Just as tigers can't change their stripes, most keep-the-lights-on IT shops will never become sources of strategic advantage.

4) Computing in three dimensions
The Internet of Things, GPS, sensors, robotics, and mass mobility mean that computing power is being applied to physical space as much as to numbers and symbols on a screen. Augmented reality, self-driving (and software-augmented) cars, robots (including drones), and other developments point to a future in which computing leaves the beige box.

5) People and data are two sides of the same coin
Nate Silver's excellent book The Signal and the Noise (more on that next month) raises a fundamental point: numbers do not speak for themselves. People assemble, parse, and interpret data in many ways, few of which are currently very self-aware. Numeracy is one big challenge, especially given U.S. high school math teaching, but so is a more nuanced view of the relationship between subjectivity and objectivity, the observer and the observed.

6) Mobile implies social, social implies mobile 
Investors and tech strategists underestimate Facebook at their peril. The connection of people to people in a mobile world is changing profoundly, not all for the better I don't think, but Facebook's massive presence at the nexus of computing, communication, and social signaling gives the company terrific leverage. This stature could change quickly -- Apple's relative loss of cool was inevitable, with significant effects, yet still sudden -- but until proven otherwise, Facebook remains a force to be reckoned with. When Mark Zuckerberg said yesterday that Facebook is NOT building a phone, he merely highlighted the complex set of issues and opportunities at the juncture of people, their tools, and their emotional needs.

7) It's a big world after all
Globalization in the context of computing and communications is playing out in fascinating ways. Japan, Inc is largely absent from the leadership: Panasonic, Sharp, and Sony are losing prodigious amounts of money. Africa is teaching the world about mobility as it applies to banking, social networks, and ad hoc resiliency. The Arab Spring story continues to be written, too often in blood as well as in texts and tweets. Finally, apart from Samsung, name a non-US company of strategic significance: every other tech firm in India, China, and Europe is jostling for position in markets heading toward commoditization. Expect to see even clearer separation of the giants (Apple, Google, Samsung, maybe Microsoft) from the rest of the corporate pack even as local habits and innovations propagate.

As in the past I will self-grade these predictions after the year plays out.

Early Indications November 2012: How will TV be disrupted?

No sector has been more transformed by the internet than audience aggregators. These content companies, through various means, make money from assembling viewers, readers, and listeners for artifacts that in the past 20 years have increasingly become digital rather than analog. Whether it is Kindle with books, iTunes for music, Google news vs. newspapers, or Netflix for movies, Internet-based content has disrupted whole sectors, eliminating such brands as Borders, Newsweek, Tower Records, Blockbuster, and Virgin Megastores.

The pattern seems to be based on relative simultaneity: Columbia sold more than 100 million copies of Michael Jackson's Thriller LP/CD, but that audience doesn't all listen at the same time. The New York Times company moves many copies of its flagship newspaper, and while readers will pick it up to read their selected articles during the day it's distributed, again, the audience is asynchronous. Harry Potter books are even more distributed in time than printed news. Television, by contrast, assembles mass audiences at a particular time: more than 100 million US viewers saw the 2011 Super Bowl, the vast majority of them in real time. But: every content bundling model, from the record album to financial advising/stock purchasing to the newspaper, has come under attack in the Internet era. (University degrees might be next, a topic for another time.)

A number of signals are suggesting that TV's time of reckoning is coming. Tablet sales are soaring: 20% of US adults now own the devices only two years after launch, and the intimacy of the tablet form changes viewing habits: social interaction, for example, is facilitated in ways that traditional TV does not allow. Time shifting, whether through Hulu, BitTorrent, or DVR, is becoming predominant among those in their 20s. Games, online shopping, social networking, and other new tasks threaten to shrink the staggering five hours per day the average American spends watching TV, if Nielsen's numbers are to be believed. Bottom line: as the Wall Street Journal put it on November 30,

"Television viewership is declining across the board. Although CBS remains the most-watched network in prime time, its average overall audience of 11.5 million in that time period is down 10% in the fall season so far, compared with the same time the year before, according to Nielsen. Its audience among 18-to-49-year-olds, the demographic most prized by advertisers, has tumbled 20%."

Two questions thus emerge: what will Internet video look like in terms of markets, business models, and financial attractiveness, and how will today's incumbents respond?

I use the term "Internet video" deliberately: just as all sparrows are birds but not all birds are sparrows, television is just one of many categories of Internet video. Without the constraints of 30-minute multiples, we are seeing a proliferation of new content forms. TED talks, for example, have been viewed about a billion times, and many of those are presumably in front of room-size audiences. YouTube served eight million simultaneous streams of Felix Baumgartner's Red Bull space jump, something no earthly TV network could have done, given the global composition of the audience. Jerry Seinfeld's Web series, "Comedians in Cars Getting Coffee," focuses on exactly that scenario, but episodes are only as long as they are: 7 minutes, 13 minutes, whatever. TV ads, particularly global ones pulled out of context to fascinating effect, have long been prime YouTube material.

Going forward, every aspect of the video experience may be contested. Apple TV, whatever it turns out to be, should improve on the remote by substituting a tablet/smartphone for the dumb infrared devices we all currently use. As for the viewing device, a big smartphone is only slightly smaller than a small tablet. As the fates of Sharp, Panasonic, and Sony illustrate, global consumers aren't as infatuated with 3D television and other innovations as they are with these smaller devices: Panasonic lost $9 billion -- $9 billion -- in its last quarter. Transport, meanwhile, is "obviously" over cable, but as mobile bandwidth gets faster and faster (is 4G as fast as we can go? There's no reason to believe so), that wire infrastructure may no longer be a monopoly. Finally, content creation used to be the private province of a small number of studios, but barriers to entry have dropped so literally anyone can produce viewable material, which then of course costs precisely zero to distribute.

In the current model for TV, big audiences equate to big ad revenues and big salaries: last season Ashton Kutcher earned $24 million for his role on "Two and a Half Men." Apart from sports, however, "big" is smaller than it used to be. Because of channel proliferation, audience fragmentation, and other factors, the top-rated non-sports shows currently running -- "60 Minutes" and "NCIS" -- earned only an 8.0 and 10.5 rating respectively (depending on week), meaning 8-11% of households are watching. Compare today's picture to that of past decades, when "I Love Lucy" pulled numbers in the 60s; the farewell episode of M*A*S*H in 1983 also drew 60.2% of households.

On one hand, the fragmentation of cable that spawns such critical successes as "Mad Men," "The Sopranos," "Breaking Bad," and "Homeland" is extending into web video. That is, quality work with niche audiences might be better addressed via YouTube or other distribution models, particularly because interactive advertising can be measured so much more precisely than broadcast or even cable numbers. At the same time, the networks' cost structure has already been reset, beginning about 15 years ago: reality and other non-scripted programming proved to be a convenient way around the Hollywood writers' strike in 2007 and also kept talent costs to a minimum. Even today, such shows as American Idol, X Factor, Survivor, and Dancing with the Stars score well while being relatively cheap to produce.

The energy drink/adventure sports sector has discovered this fact of online behavior and exploited it aggressively. Ken Block's driving stunts, the aforementioned Red Bull oeuvre (including non-ad programming such as "Jackson's Hole"), and Monster (with more than 34 million views) offer programming that would be ill-suited for even niche cable channels but perfectly situated for viral distribution, repeated viewing, and powerful branding opportunities.

Because of the sheer volume of material -- 72 hours of YouTube video are uploaded every MINUTE -- viewers confront a classic long-tail scenario: every niche of interest, taste, culture, and video quality is addressed, including some that haven't been invented yet. Cable television solves this problem with brand identity: viewers who tune in to HGTV, Food Network, or Fox Soccer Channel have a pretty good idea of what they'll see. While YouTube, TED, and other outlets have adopted some variation of the "channel" philosophy, it doesn't scale. Neither does search work very well: if you didn't know what Gangnam Style was, how on earth could you phrase the search to find that video?  Social networking is, to date, the best option for finding this stuff, raising the prospect of more formal varieties of trusted filters who watch thousands of hours of bad video to snare the treasures.

To give just one example of such valuable artifacts, there are a multitude of black-and-white clips of musical, stand-up comedy, and other performances from the 1950s and '60s. Seeing Thelonius Monk on Swedish television can be wonderful, surreal, and dazzling all at the same time. Again, television, even of the cable variety, is a poor distribution mechanism for such work, but the planet's artistic inheritance is far richer than it was just a decade ago: it's one thing to dust off a Wes Montgomery LP and play it for the budding musician and something else entirely for her to see the genius himself, often explaining the music between takes. There are doubtless thousands of other examples (university lectures are one - reading Richard Feynman and seeing him are two very different experiences).

How will the incumbents respond to these and the many other changes ahead? Hulu is trying to impose cable economics on web video, and it might be able to do so for a few years. As cable subscription prices rise relentlessly, however, cord-cutting will increase, I predict. The fact that cable operators control the majority of high-speed home connections complicates this switchover somewhat, but as mentioned above, wireless broadband and tablets could lead to a different economic model in the future. At what point could AT&T/Verizon supplant Comcast?

Sports, particularly football, appear to be immune to time-shifting, so much so that multi-billion dollar bets are being made on this assumption. ESPN paid $1.1 billion in 2010 for 18 games of Monday Night Football; the rate nearly doubles, to $1.9 billion, only three years later. The Big Ten cable network (half owned by Fox) generated $7 million per school in 2009; Maryland joined the conference last month in part because the school was reportedly promised $43 million (along with all other members) in 2017.

Big Ten commissioner Jim Delany is reported to be a student of demographics, and the athletic conference's core states in the Rust Belt are indeed losing population to the American West and South. But Delany is, I believe, falling victim to linear projections of cable access fees paid by households that, in the main, are not seeing big jumps in income. As the NFL, college football, the Olympics, the World Cup, and other sports bodies continue to rely on heavier funding from their television partners, this technological sea change threatens to undermine that rising tide of non-advertising revenue. When might the golden goose stop laying so many eggs? Delany's math is reminiscent of that of the financial industry in 2005-6, when it was believed that housing prices could never go down.

What might happen next? Pay-as-you-go for cable channels may become a reality at some point, the global audiences for web video could become a factor in ways broadcast cannot reach, and new advertising technologies could alter the landscape. Might brand loyalists get more, better ads, and subsidized cable subscriptions, for example? Will overlays, quizzes, games, and even biological sensors augment the 30-second spot?

Who should we be watching for signs of business model change? Clearly the current rights holders are not standing still, and ventures such as Hulu will evolve. Given Apple's past relationships with Hollywood content companies, iTunes could turn into a new kind of cable TV experience.  Startups such as GetGlue, Showyou, Vimeo, and Yahoo's IntoNow provide a variety of social layers for video production, distribution, and consumption. Amazon has lots of digital assets, a hardware platform in the Kindle Fire, and a history of surprise moves.

Whatever its future, broadcast TV had a good run, and will of course deliver value in certain circumstances in the future. But the monopoly that television has had over video audience aggregation for the past 70 years is being broken. As a result, the possibilities for the future are both exciting (for content producers and consumers) and potentially expensive for incumbents. In any case, creativity should flourish as it has in every other technology revolution that empowered artists, whether printing, paint chemistry, or photography.

Early Indications October 2012: The State of Mobile Payment (U.S. market)

With so many powerful players getting involved, some more standing on the sidelines, and a few wild-card startups, the smartphone-as-wallet market is truly compelling business theater. Far more is unknown than certain right now, so rather than speculate, I'll lay out some of the knowns then concentrate on the relevant questions that must be sorted out.

What we know

Multiple projections suggest that consumers will use smartphones as wallets more and more frequently in the next five years. In addition, Japan and Kenya, among other markets, provide useful precedents for mobile payment behavior. Even so, U.S. adoption is likely to be unique for a variety of reasons.

Smartphones are being adopted extremely rapidly, and they possess some important characteristics that make them well suited to be used in commercial transactions:

     *They can be configured with extra-secure memory and other hardware features to increase confidence of consumers whose money and/or information might be lost.

     *They have cameras that can serve as scanners of bar codes and other physical hyperlinks.

     *The cameras and displays can also facilitate biometric identification in the form of face recognition, either algorithmic or by humans.

     *They can be configured with a radio devoted to secure payment at close range.

     *They are pretty reliably carried on one's person and thus can replicate some roles of a wallet.

Who has entered the arena

Competition in mobile payment is already intense. As the list of active players illustrates, however, different entrants may have differing strategic endgames. To say that the U.S. market does not need smartphone-based wallets because of the mature credit-card reader infrastructure may be true, but does not negate the potential impact of couponing, loyalty cards, location-sensitive promotions, and ad placement: smartphones in the U.S. market may never get the functionality of a Safaricom M-pesa account in Kenya, but plenty of other interesting scenarios are nonetheless possible.

Mobile carriers: Sprint is the only carrier to support Google Wallet. AT&T, T-Mobile, and Verizon comprise the ISIS consortium. Isis is running trials.

Credit cards: Visa has PayWave while MasterCard launched PayPass. For its part, American Express has Serve, which is pre-paid rather than a credit service like the other two. Discover has announced an alliance with PayPal (see below).

Merchants: Announced in August, the Merchant Customer Exchange is led by Best Buy, CVS, Shell, Target, and Wal-Mart among others.

Google: Google Wallet currently runs only on a few Sprint phones. A sticker with an embedded chip can bring smart wallet features to Android phones that do not meet carrier standards for NFC (near field communications) in the hardware layer, but these do not appear to be widely available and/or adopted.

Square: The mobile payment startup founded by Jack Dorsey of Twitter fame (he works 8 hours a day at each company) does not require NFC chips and interfaces neatly with the company's merchant solutions which attempt to disrupt the Verisign-dominated credit card terminal market.

PayPal + Discover card: PayPal announced the credit card partnership in August and expects to process $10 billion in mobile phone transactions this year. This appears to be the first mover with competitive advantage, in the early going.

It is difficult to believe that Amazon and Apple (which currently supports non-payment wallet functionality) will stay out of this market.

The questions:

1) Hardware vs software

While the Google Wallet and Isis solutions rely on a hardware component -- the NFC chip in the phone and accompanying reader at point of sale -- PayPal and Square employ more of a software play. By way of comparison, in Japan, the prepaid cards used to pay train fares use contactless hardware, while in Kenya the popular M-pesa service runs on SMS.

In the U.S., wireless carriers control access to the secure memory area that is required for Google Wallet, and that access has not been widely agreed to thus far. Alternatively, cloud-based solutions move consumer information off of the handset, obviating the need for carrier-controlled access to the so-called secure element. In addition, they do not require new point-of-sale infrastructure at the merchant, as in the case of NFC. In the long run, the Isis solution appears to be the most likely NFC-based scenario, if any take hold, while Google recently announced it is moving to a cloud approach possibly in parallel; the fate of the hardware-resident Google Wallet is unclear. Apple has yet to include NFC chips on iPhones.

2) Payment vs promotion

Even though the rubric is to speak of mobile payment, because the US has such a broad credit card reader network, the promotional aspects of mobile wallets will be a critical factor for user uptake. This opens the possibility of an arbitrage app, to sort through a consumer's competing offers, point systems, privacy compromises, and time constraints. Do I buy gas at Shell on MCX or at Sunoco with Isis today? What about next week's offers? How close am I to Gold standing in a given reward program? Who do I trust to maintain multiple account information? Apple has Passbook, an iOS app for ticketing and loyalty but not (as of yet) payment. Google Wallet does many of the same things.

3) Multi-home vs single-home

These terms derive from the economic literature on platforms. Many "multihoming" people carry both Mastercard and Visa. Some electronic game titles are available for both Xbox and Playstation. At the same time, very few people maintain both Facebook and MySpace identities, or checkbooks from consumer accounts at two different banks. Right now Visa in particular works with many other digital wallets. The homing question will be interesting to follow: will consumers maintain simultaneous Starbucks, Visa, Isis, and MCX accounts, their high degree of overlap notwithstanding? Will reward systems interoperate, or will they be more like frequent flier miles, which are expensive to convert across carriers? In short, will there be a few huge winners or many players of varying scales?

4) Sweet spot (transaction amount)

In the U.S., credit cards are fast and easy, but not widely used for purchases under $10 or so. Retail foodservice establishments, meanwhile, process millions of transactions a day under $25, and the experience of making change is slow, error-ridden, and inconvenient. Given the power of the U.S. penny lobby, we still use the copper coins even though it likely costs more to spend the extra time for a McDonalds clerk to make exact change for 47 cents rather than 45 or 50. Thus a wave-and-go payment solution for fast food and other outlets (where customer throughput is a key profitability metric) makes a lot of sense. Burger King does not really "average" 500 (or whatever) customers per hour, but instead deals with large spikes in demand at lunch time, when big events let out, and so on. Decreasing transaction time will pay off quickly for these merchants, and if they do not need to invest in new reader infrastructure, so much the better. Thus if mobile payment can fill in a niche at the low end of the market rather than compete with plastic cards, it could expand rather than only cannibalize the existing market.

5) Pre-pay/credit/mobile bill

What will be the preferred source of funds in the digital wallet(s)? Visa and Mastercard can easily extend their credit model to new payment modalities. Pre-paid plans work well for mobile carriers and do not require credit checks and other risks. Wireless carriers already have credit ratings, home addresses, and locality information for their customers. Each of three models could catch on for some population (mass transit passes for school kids in cities, meal plans for college campuses, gift cards for retail shoppers, and the list goes on); it may not be a winner-take-all scenario.

6) Global vs national

A credit card can currently be used nearly anywhere. If NFC or other hardware solutions prove popular in a given geography, how widely will any particular technology take hold? Is a variety of local technologies the long-term scenario? Most likely not.

7) Killer app(s)

To unseat the incumbents (cash money and plastic money), mobile money will need to be secure, easy, widely accepted, and deliver additional benefits. In Kenya, one killer app was processing expatriate payments from abroad without a trip to Western Union. In Japan, train fares at rush hour are processed much faster with mobile money than with other forms, thus easing workflow. In the US, mass transit is not widely used, but coupons and loyalty programs are. Somewhat weirdly, many demo videos for various smart wallet solutions all focus on one merchant scenario: coffee shops. This segment doesn't feel sufficiently substantial to stand as the cornerstone of a new shopping paradigm, however.

The corporate heft of the various competitors suggests a sumo-like contest to win consumer affection. What's the payoff? Consumers love discounts, and smartphones can geoposition those offers to current place and time. On the platform owner's end of the transaction, meanwhile, the prospect of terabytes of consumer data piling up from offers at various price levels or under different conditions (25% off versus "buy three get 1 free") with the location tracking aspect presents a "big data" scenario of staggering proportions. It's clear to see the prize -- consumer data at unprecedented scale -- but the value proposition to wallet-carriers remains unclear.

8) Wild cards

What happens when some segment of the infrastructure (power, wireless data, terminals) goes down?

Who pays to retrofit millions of vending machines, arcade games, and other devices that currently take cash and coins? What about merchant terminals?

What happens when data is compromised?

What do I do when my phone/wallet is lost or stolen?

Who bears what risk?

What happens when my battery dies?

What happens with competing standards or entities? Can merchants "jam" competing offers? If Shell sees I'm gassing up at Citgo, do I get coupons? Cut off? Slower service next time I pull into Shell?

What if public backlash at RFID extends to NFC, an RFID variant?

Will anyone understand the privacy agreements they sign?

Early Indications September 2012: Party platform comparison

*****
It should be stated explicitly that all views are my own and do not represent official policy or any other attribute of my employers. Politics is always delicate territory and I have tried both to critique and respect both parties in my analysis.
*****
Given the stubbornly high un- and under-employment numbers, the rapidly diverging fates of the very rich and everyone else, and the increasing speed of technological change, it appears that the U.S. economy is in the midst of some fundamental transitions. Consider the evidence:

*Apple, with a trillion-dollar valuation in range, employs fewer than 50,000 U.S. workers, not counting retail store personnel. Facebook, said by Wall Street to be worth $100 billion at one point, employs only about  3,000. Not only are their overall numbers small, these highly-prized jobs create very few "multiplier" jobs for lower- and middle-class workers the way the automobile did. Where will large numbers of new jobs come from? In most cases the old ones are not coming back.

*Municipalities in multiple states are declaring bankruptcy, and unlike the 1994 Orange County precedent, these are likely to be followed by more. It is also likely that a U.S. state will have to declare bankruptcy, or something similar. Between direct hiring of employees as well as support of homebuilding (through the mortgage interest deduction and deep involvement in mortgage markets) and public works, government-fueled employment has helped counter a long-running loss of manufacturing jobs. As the wave of government hiring from the last 65 years bequeaths us its total costs -- in extremely generous health care and retirement benefits -- how can we afford the past hiring at the federal, state, and local levels, much less the governments of the future?

*A college education historically contributes to higher earning histories, but the price of that education is also growing at an unsustainable rate. According to the College Board, the average cost of a 4-year public college education has more than quadrupled since 1981, after counting the effects of inflation. Student loans are available to cover this ever-increasing sum (thus creating a moral hazard for the universities?), with the result that student loan debt has crossed a trillion dollars. Insofar as student loans are not able to be discharged by bankruptcy, and given that requiring high credit scores along with a large down payment is now standard practice in the mortgage industry, many of the students who leave college with these loans will not be buying houses any time soon. How can we educate students in tomorrow's skills without saddling them with excessive debt?

Given these interconnected forces, the role of elected officials in encouraging job creation is limited: governments are shedding police officers, fire fighters, and teachers. Much as every president loves a construction boom, most housing markets are still digesting excess capacity built in the 2003-2006 period even as they are more boomer home-sellers than there are Gen Y couples to buy real estate. Even though the benefit of more education used to be axiomatic, honest discussions of student aptitude and projected debt should be a factor in college choice in more and more cases.

Thus I believe that the nation is at a crossroads, a historical moment when new approaches are necessary. What can voters look forward to? Rather than try to decode the speeches of the presidential candidates, I examined the two parties' platform documents. Sadly, there is little indication that either party is approaching today's unique challenges with an adequate arsenal of ideas.

The Democratic platform is long on punchlines: "We see an America that out-educates, out-builds, and out-innovates the rest of the world," it repeats at several junctures. The next American economy needs to be built "from the middle out," not from the top down. The proposals in this vein have the overall whiff of a free lunch: middle-class tax cuts, refinancing options for mortgages under water, "accessible, affordable, high quality health care," and a promise that "we will find a solution to protect Social Security for future generations." How will this last goal be accomplished? Later retirement, higher taxes, and/or lower benefits are not mentioned as the only ways out. The document is short on the necessity of sacrifice.

As for education, the Democratic platform focuses more on teachers than on students. While "Democrats believe that getting an education is the surest path to the middle class," the cost of bad teachers for tax payers and students alike is never addressed. Instead, in a nod to union resistance to removing underperformers, "We also believe in carefully crafted evaluation systems that give struggling teachers a chance to succeed and protect due process if another teacher has to be put in the classroom." For every "struggling" teacher, however, hundreds of students are denied an adequate education, but they are not mentioned.

This commitment to public-sector unions appears in another passage, one that implies that high-paying government jobs have no discernible price. Compared to the labor-management strife that fueled the big industrial unions, teachers' and police officers' economic interests are opposed not to bosses but to taxpayers: the teachers' neighbors. Nevertheless, "we will fight for collective bargaining rights for police officers, nurses, firefighters, emergency medical technicians, teachers, and other public sector workers -- jobs that are a proven path to the middle class for millions of Americans." But who pays that bill? In France, the government (followed by the post office) is the largest employer and the top marginal tax rate is 75%. Such an end-game feels unsustainable in a country as large as the US.

In the transportation domain, it's hard to see anything encouraging. In addition to the "all-of-the-above energy policy" and encouragements for investment in clean energy, the "Democratic Party supports a broad-based strategy to further strengthen an American renaissance in manufacturing," including "incentives to build advanced vehicles in the United States." The particulars (what's an "advanced vehicle"? The Chevy Volt from Government Motors?) are nowhere to be found.

In sum, much of the Democratic platform document is given over not to specifics, as we have seen, but to fear-mongering about what a Romney presidency might entail. For what is ostensibly a policy document, the Democratic platform could have done far better.

Which brings us to the other ticket. Richer in specifics, the Republican platform is so dominated by trickle-down ideology that job creation and economic recovery at the scale necessary seem extremely unlikely: the document looks into an idealized rear-view mirror on an America that never existed in the first place -- then apparently wants to bring it back, never mind the actual forces at work both then and now. Tax cuts, reductions in regulation (particularly the environmental variety), and hot-button social issues (immigration, firearms, Judeo-Christian religion, gay unions, the gold standard) dominate. Just as with the Democratic document, the Republican platform spends a lot of time name-calling, rather than constructively moving forward. Part of this behavior is the nature of our times, I realize, but I persist in believing the majority of the American people are sick of partisan bickering.

The second sentence of the Republican document begins with a bold assertion: "Prosperity is the product of self-discipline, work, savings, and investment by individual Americans." The converse is clearly implied: lack of prosperity is the product of weak moral fiber, laziness, careless spending, and short-sightedness by individual Americans. The rich are rich because they existentially deserve their wealth, and taxation is thus a form of theft on behalf of those who are not prosperous, presumably because of the aforementioned factors.

The problem is that many who are wealthy inherited their fortune (what Warren Buffet calls the "lucky sperm club"), while still others had the good fortune to be in the right place at the right time. And luck works both ways. Most restaurants fail, notwithstanding chefs' 80-hour weeks. People pick economically unfortunate college majors or slow-track entry-level jobs. Factories close. Jobs get outsourced. Illness happens. Across geographic and ethnic lines, family units vary in size, strength, and respect for education; unfortunately, children can't pick their parents. Can anyone say these individuals are less morally worthy than the fortunate few wealthy? This resistance to acknowledge the role of luck -- alongside intelligence, prudence, and effort, I hasten to add -- in people's lives contributes to the overall tenor of the Republican economic plan.

Furthermore, the document is counterfactual at several points. Despite the sluggish economic performance of the decade during which the Bush tax cuts were in force, the Republican platform nonetheless asserts that "lowering taxes promotes substantial economic growth" and later bemoans the current federal budget rules' refusal "to recognize the beneficial budgetary impact of lower tax rates." Similarly, the party of creationism and of politicized redactions to climate research in the Bush White House states that "We must restore scientific integrity to our public research institutions."

In addition to denying the relevant past, the party document gives the ability of the market system to create positive outcomes undue credit: infrastructure, paid for with public tax dollars, should be privatized even though the track record for such projects, such as various toll roads, is mixed. Likewise, mortgages, student loans, and Social Security should be subject to the free market of financial services firms. Health care, too, can be fixed with more free market dynamics. Even schools -- choice via vouchers -- can be repaired with market mechanisms. Following the money may be useful here: who stands to gain under each scenario? Will the nation be stronger overall? Consider that in health insurance, rational behavior for the insurers, whose CEOs earn handsome salaries, is to deny as much care as possible. Somehow, maximizing shareholder return does not seem to be the overall point of this particular exercise.

Despite serious issues with their overall vision, give the Republicans credit for specificity. The platform contains some economic ideas that definitely should be tested further:

-workforce retraining programs should be modernized

-the tax code should be simplified

-immigrant innovators should be welcomed rather than turned away

-the Telecom Act of 1996 is "woefully out of date"

-the 130-year old Civil Service system should be reformed

-tort reform should deliver measurable benefits to both health care costs and physician service in rural areas, which are often underserved

-new systems of learning and new models for life-long skills refreshment should be a priority.

If the Democratic drift is toward free lunches without sacrifice, the Republican platform venerates wealth but says little about not-wealth: the sick, the aged, those left behind by economic change. Similarly, while Democrats were criticized for not mentioning God in their platform, from a policy standpoint the Republican refusal to acknowledge climate change is more worrisome.

Okay, the reader may reasonably ask, so it's easy to poke holes. What would you offer that's better? Fair question. Here are five big proposals that could be adequate to the times we face and, more important, recognize that the status quo is inadequate: we, and especially our children, deserve better. Looking back in 20 years, will we honestly be able to say we did the best we could? Just as important, the rest of the world is looking for leadership amidst tremendous potential and instability. Will they find it here?

1) Invest in our nation and in our future: initiate national service for 18-19 year olds. Make it compulsory and universal. Mormons back from their mission and Israelis who serve in the defense forces both are more skilled and more mature than 20-somethings without the experience. From the nation's perspective, national service should increase the skills base, reduce welfare and similar payments, and create roads, parks, libraries, literacy programs, broadband networks, and other projects of lasting value. Giving potential college students a taste of the work world should either a) make them more motivated students or b) convince them they're better suited for trade craft or other non-college work. Letting teens earn some of their future tuition could help address student loan debt too.

2) Stop paying for so much waste in health care delivery. a) If we spent less on insurance processing, and yes less on tort-driven behaviors as per the Republican platform, quality of care should go up while costs drop. I'm not pretending to know what combination of single-payer, health savings accounts, and/or lifestyle change would be required, but addressing the problem is imperative. b) A vastly disproportional share of health care expense is incurred at the end of life. We as a culture need to stop trying to "beat" death and start acknowledging the life that was lived and comforting the individuals in transition. Physicians need to feel, and be allowed to feel, human and neither like gods nor life-extension technicians in such moments.

3) Increase funding for DARPA. Given stealth aviation technology, GPS, and of course the Internet, this agency has paid for itself many times over. Their current emphasis on robots is going to be similarly game-changing, I predict.

4) Fix the trademark and patent system. Intellectual property law is a disaster in that those without rights can profitably stall progress while those with certain rights are protected for far too long.

5) Attack the obesity epidemic with the same vigor of putting a man on the moon. Diet, exercise, mental health care, workplace arrangements, incentives and disincentives -- the problem can and should be fought on many fronts. Given the Olympic success of this great nation, it is clear that we can devise physical and related programs so that no adult and especially no child should lose quality of life or overload the healthcare system with preventable weight gain. While such measures could be read as intrusions into individual liberty, the precedent of smoking is relevant, I believe.

Early Indications July 2012: The Media Beast

Those of you who have read this newsletter for a while know that it
has, nominally anyway, a technology focus. While the current number
may not feel like it delivers a tech message, that is in fact where
the argument is heading.

The recent events at Penn State raise a broad range of questions, from
"how can humans be so evil?" to "what were those men thinking?" to
"who was the client for Louis Freeh's report?" to "what was the NCAA's
agenda in the sanctions?"  Many of these questions remain extremely
loaded with emotional and legal baggage, and I will tread lightly
around them, speaking only as a private citizen and not in any
University capacity.

My purpose is somewhat more macroscopic. Think back to some sports
heroes of the 1990s. All of these individuals have, to some extent,
been disgraced and fallen from their pedestals.

In alphabetical order,

Barry Bonds
Michael Irvin
Allen Iverson
Marion Jones
Bob Knight
Mark McGwire
Joe Paterno
Rick Pitino
Tiger Woods

Lance Armstrong could well be next.

Each of these individuals boasts a Hall of Fame resume. Each at some
point transcended his or her sport, becoming an icon in some larger
arena. But each at some point, having been made larger than life, fell
hard from the extra-human heights to which he or she had ascended.
Without mass media, I will argue, these individuals could not have
soared to such cultural visibility, but many people's cognitive and
emotional apparatus may not be well suited for life at such a scale,
setting the stage for subsequent disappointment.

Accordingly, I'd like to explore the role of technology in this
process of creating and then questioning heroes, using the Penn State
experience as an example close at hand. My proximity to the story is
recent: I moved here in late 2005 from Boston. The Paterno statue had
already been erected four years earlier, but I immediately noticed
that State College felt different. My grad work was done at Michigan,
home of the winningest college football program in history, both by
wins and winning percentage. My undergrad years were spent at Duke,
though Coach K mania can hardly be said to have hit in that era: while
his first three years' record was 38 wins and 47 losses, home
basketball games at Cameron had already became one of the great
experiences in all of sports. So I came from academic institutions
where sports Mattered.

While geography is certainly a factor (State College, for those who
haven't visited, is said to be equally inaccessible from everywhere),
part of the State College difference is architectural. The football
stadium, which is enormous, has a presence beyond mere mass: Michigan
Stadium, the "Big House," seated more people when I was there in the
1980s, but it's a giant hole in the ground. Driving by it in Ann
Arbor, there was little in the pre-skybox days to announce its
vastness, so walking inside was a wonderful experience to grasp the
scale from the top row of seats. (At graduation, we got to walk out
onto the field level through the tunnel the players used, though there
was no M Go Blue banner to leap up and touch.) Beaver Stadium, by
contrast, is all above ground. Given that it was added to in sections
as Coach Paterno's success generated interest, it looks like an ad-hoc
spaceship, with spindly underpinnings and bolted-on exterior ramps and
walkways. (Here's a view that shows that tendency) Contrast the
Rose Bowl, or even the Horseshoe at the state university to the west,
and unlike Beaver Stadium, they were designed from their origins to be
monumental.

The last of the renovations was completed in 2001, when seating
capacity, which had doubled in stages from 46,000 in 1960 to 95,000 in
1991, was increased to 107,000. The move seemed justified as the 1990s
were heady days for the program. Penn State joined the Big 10
conference in 1993, and immediately delivered strong results,
including three top-10 national rankings in seven years, top-20
rankings every year, and an undefeated season in 1994. The statistical
rankings become relevant in a moment.

1993   10-2

1994   12-0  (#70 in total defense; #1 in scoring offense (48
points/game); #1 total offense)

1995   9-3  (#50 in total defense)

1996   11-2 (#41 in total defense)

1997   9-3   (#86 in total defense)

1998   9-3   (#12 in total defense)

1999   10-3  (#27 in total defense)

2001, when the current stadium was completed, was also the year when
the statue of Paterno was installed, and a year when success was hard
to come by. Before the wins were vacated by the NCAA, here was the
story:

2000   5-7   (#55 in total defense)

2001   5-6   (#98 in total defense)

2002   9-4   (#27 in total defense)

2003   3-9   (#49 in total defense)

2004   4-7   (#10 in total defense)

So consider the context for the 2001 decision to cover up the Sandusky
abuse, which is what is alleged to have happened:

-The football team was in the midst of a run that would get most major
college coaches fired: only one bowl game (a loss) in five years.

-The Paterno years had been remarkably consistent until that time. The
program was the 6th most winning program of the 1970s, the 1980s, and
the 1990s. Only Nebraska could join Penn State in the top 6 all three
decades: Oklahoma did it twice, while Alabama, Florida, Florida State,
Miami, and Michigan did it once apiece. LSU, Ohio State, Texas, and
USC did not make the top 6 in any of the three decades.

-The losing streak coincided with the retirement of Sandusky, a
respected defensive coordinator, though the statistically best
defenses did not always translate to winning seasons. Penn State's
reputation as "Linebacker U" derives from Sandusky (and continues
under Ron Vanderlinden, who has roughly a half-dozen linebackers/pass
rushers currently in the NFL).

-Penn State basketball has rarely made Big 10 hoops royalty very
nervous. Since 1983, the program's conference record is .317. Thus
football must generate sufficient revenue to support many non-revenue
sports, several of which -- wrestling, and men's and women's
volleyball -- win national championships.

Looking back, the turn of the millennium seems to have seen a changing
of the guard. Paterno turned 75 at the end of the 2001 season. The
game, and his university, had changed dramatically over his career.
To take the latter first, Penn State's average SAT scores went up a
relatively modest 67 points between 1966 and 2006 (LSU's dropped 110;
Virginia's increased 141). I cannot find a list of federal research
expenditures by university from 1970, but Penn State has risen to rank
in the top 15 in total research dollars received. Like the University
of Washington, the Penn State undergrad ranking is not selective (tied
with UW at 184), but both schools are top-tier research universities,
standing in the top 20 institutions in PhDs granted and other key
measures. Like Duke, Arizona State, and other institutions, Penn
State's academic reputation has risen considerably over the years
during which Paterno was coaching. He claimed, plausibly, that he had
raised $1 billion for the university, money that had helped academics
gain in stature.

How did football change in the Paterno era? In 1966, his first year,
ABC was the only network to broadcast regular-season college football
games. With a Supreme Court decision in 1984 ruling that the NCAA
television plan violated anti-trust law (the University of Oklahoma
and the University of Georgia were the lead plaintiffs), the number of
televised games spiked from 89 to roughly 200 in just one year. Rights
to college football became truly big business: the Longhorn Network
joint venture at the University of Texas is potentially a $1 billion
enterprise over 20 years, and the deal contributed to the near-demise
of the Big XII conference of which Texas is a member.

ESPN, which began broadcasting in 1979 when only about 20% of US
households had cable, seized on college football after the Supreme
Court decision with a 48-game package in 1984. As the 1958 NFL
championship game proved, television and football are a potent
combination, and ESPN's rise is intertwined with that of college
football. College GameDay began broadcasting in 1987, but became a
major part of campus life in 1993 when it went to its current live
remote packaging. (ESPN was originally an investment of Getty Oil and
. . . Nabisco, which later sold its stake to Hearst, which still holds
it: Disney only acquired 80% of ESPN in the Cap Cities/ABC acquisition
in 1996.)

Other aspects of the game changed over Paterno's career. Modern
athletic departments can be $100 million operations, with commensurate
losses when times are bad. The athletic directors prepared for
business at that scale are rare; in Paterno's era, many football
coaches (including Joe himself) were ADs either during or after their
coaching career. University of Florida president Bernie Machen has
said the ESPN money means that athletic departments become
semi-autonomous entities outside university control. Thursday night
football games, for example, often result in class cancellations and
other academic accommodations to the networks. Television exposure, in
turn, generates student applications for admission: Duke's increase in
selectivity coincides closely with the Coach K era.

Coaches can become millionaires. While it is said that John Wooden,
probably the greatest college coach in American history, never made
more than $35,000 in salary, today the head football coach at Wyoming
earns $1,000,000 in a complex package, not counting a car and cell
phone. The head man at University of South Florida pulls in $2 million
a year. Life at the top of the ranks is a $5 million/year proposition.
That's a lot of money for a coach to lose if the program goes south.

So cable sports television was one major change in sports life. What
does that have to do with the list of disgraced figures with which we
began? The pressure of 24-hour media takes a toll. For all the sports
figures in the limelight who stumbled, there are entertainers who
responded even more tragically: Kurt Cobain, Whitney Houston, Elvis
Presley, Michael Jackson. In sports, the pressure to win, and then win
AGAIN, can become enormous. The urge to cheat is one temptation (as at
the University of Connecticut under Jim Calhoun, currently banned from
the NCAA basketball tournament), as is steroid and related drug use.
But the human cost of life in the mass media spotlight is complex:
even Duke's Coach K, he of the military discipline, solid family, and
ideal circumstances, had to reset his life in 1995 when his wife
staged a full-scale intervention amidst his physical and emotional
burnout.

The pressures are both subtle and obvious. Coach Charlie Weis at New
England and later Notre Dame and Kansas has a serious weight issue for
which he had gastric bypass surgery; Rex Ryan of the New York Jets had
similar surgery to help lose more than 100 pounds off a high of about
350. Urban Meyer at Florida found himself reacting to the stress
differently, 35 years Paterno's junior and even more recently
successful and therefore powerful (two national championships in his
first four years in Gainesville). Meyer reported feeling physical
symptoms of stress and exhaustion, and quit then unquit coaching. The
age of the Internet, however, created an alternate news channel where
rumors spread that Meyer had followed the Bobby Petrino/Rick
Pitino/Tiger Woods road of temptation, with a UF student. That's
another way the game has changed: 24-hour/52-week news coverage, talk
radio where spleneticism equals ratings, and message boards where
fandom has no fact check.

Technology will always shape sport. As I write, NBC is refusing to air
live coverage of the Olympic games so Twitter is providing a rough
facsimile of the pomp and pageantry that I will later refuse to watch
in prime time. Fan interest in statistics is growing annually, to the
point where the NBA just signed a deal to use SAP's powerful HANA
analytics tool to power a basketball stats website. Cellphone cameras
can document any moment, as Michael Phelps discovered the hard way
after the 2008 games as he relaxed with some weed. The interplay
between athletic drama, success, power, and personal identity has
grown more complex with television and later the web, mobility, and
social media.

When Joe Paterno rebuffed his president's "suggestion" to retire after
the losses mounted in 2004, he proved he effectively reported to no
one but himself.  Paterno helped build Penn State to what it is today,
and televised football both helped create the Paterno mystique and the
events that dismantled it: Paterno's weak replies in 2008 to an ESPN reporter asking about his players' formidable arrest record, vividly contrasting with the "Grand Experiment" rhetoric he helped propagate, are some of the defining moments of the last decade of his long
career.  Paterno famously eschewed cell phones, e-mail, and text messages. At the same time, he championed video replay of officiating calls, and vigorously supported
the Penn State libraries, home to a multitude of electronic databases
and other learning tools. Once again, technology was a two-sided coin.

The power of broadcast media to aggregate enormous audiences, and the
simultaneous need of the technology for drama sufficient to the stage,
creates the possibility of truly super-human figures. When mere humans
feel small compared to their projected selves, bad things can often
happen. Other times, when people feel their media image has changed
their off-the-field life so rules don't apply, we can get the sexual
assault ugliness of Kobe Bryant, Ben Rothlisberger, and others.
Either way, aligning human existence with life in the athletic
spotlight (and cellphone lens, and Twitter stream, and Facebook
profile) has never been more challenging.