The Dirty Little Secrets of Search

[Sabre]

The Spec

PRETEND for a moment that you are Google’s search engine.

Someone types the word “dresses” and hits enter. What will be the very first result?

There are, of course, a lot of possibilities. Macy’s comes to mind. Maybe a specialty chain, like J. Crew or the Gap. Perhaps a Wikipedia entry on the history of hemlines.

O.K., how about the word “bedding”? Bed Bath & Beyond seems a candidate. Or Wal-Mart, or perhaps the bedding section of Amazon.com.

“Area rugs”? Crate & Barrel is a possibility. Home Depot, too, and Sears, Pier 1 or any of those Web sites with “area rug” in the name, like arearugs.com.

You could imagine a dozen contenders for each of these searches. But in the last several months, one name turned up, with uncanny regularity, in the No. 1 spot for each and every term:

J. C. Penney.

The company bested millions of sites — and not just in searches for dresses, bedding and area rugs. For months, it was consistently at or near the top in searches for “skinny jeans,” “home decor,” “comforter sets,” “furniture” and dozens of other words and phrases, from the blandly generic (“tablecloths”) to the strangely specific (“grommet top curtains”).

This striking performance lasted for months, most crucially through the holiday season, when there is a huge spike in online shopping. J. C. Penney even beat out the sites of manufacturers in searches for the products of those manufacturers. Type in “Samsonite carry on luggage,” for instance, and Penney for months was first on the list, ahead of Samsonite.com.

With more than 1,100 stores and $17.8 billion in total revenue in 2010, Penney is certainly a major player in American retailing. But Google’s stated goal is to sift through every corner of the Internet and find the most important, relevant Web sites.

Does the collective wisdom of the Web really say that Penney has the most essential site when it comes to dresses? And bedding? And area rugs? And dozens of other words and phrases?

The New York Times asked an expert in online search, Doug Pierce of Blue Fountain Media in New York, to study this question, as well as Penney’s astoundingly strong search-term performance in recent months. What he found suggests that the digital age’s most mundane act, the Google search, often represents layer upon layer of intrigue. And the intrigue starts in the sprawling, subterranean world of “black hat” optimization, the dark art of raising the profile of a Web site with methods that Google considers tantamount to cheating.

Despite the cowboy outlaw connotations, black-hat services are not illegal, but trafficking in them risks the wrath of Google. The company draws a pretty thick line between techniques it considers deceptive and “white hat” approaches, which are offered by hundreds of consulting firms and are legitimate ways to increase a site’s visibility. Penney’s results were derived from methods on the wrong side of that line, says Mr. Pierce. He described the optimization as the most ambitious attempt to game Google’s search results that he has ever seen.

“Actually, it’s the most ambitious attempt I’ve ever heard of,” he said. “This whole thing just blew me away. Especially for such a major brand. You’d think they would have people around them that would know better.”

TO understand the strategy that kept J. C. Penney in the pole position for so many searches, you need to know how Web sites rise to the top of Google’s results. We’re talking, to be clear, about the “organic” results — in other words, the ones that are not paid advertisements. In deriving organic results, Google’s algorithm takes into account dozens of criteria, many of which the company will not discuss.

But it has described one crucial factor in detail: links from one site to another.

Pretty interesting... but not against the law...

[complacent]

nor was google's response...

i think they're on page 78 now.


but you're right, not against the law. can open, worms everywhere.

[Sabre]

haha, agreed... I should have put that in here too:

J. C. Penney, it seems, will not suffer the same fate. But starting Wednesday, it was the subject of what Google calls “corrective action.”

Last week, The Times sent Google the evidence it had collected about the links to JCPenney.com. Google promptly set up an interview with Matt Cutts, the head of the Webspam team at Google, and a man whose every speech, blog post and Twitter update is parsed like papal encyclicals by players in the search engine world.

“I can confirm that this violates our guidelines,” said Mr. Cutts during an hourlong interview on Wednesday, after looking at a list of paid links to JCPenney.com.

He said Google had detected previous guidelines violations related to JCPenney.com on three occasions, most recently last November. Each time, steps were taken that reduced Penney’s search results — Mr. Cutts avoids the word “punished” — but Google did not later “circle back” to the company to see if it was still breaking the rules, he said.

...

“Am I happy this happened?” he later asked. “Absolutely not. Is Google going to take strong corrective action? We absolutely will.”

And the company did. On Wednesday evening, Google began what it calls a “manual action” against Penney, essentially demotions specifically aimed at the company.

At 7 p.m. Eastern time on Wednesday, J. C. Penney was still the No. 1 result for “Samsonite carry on luggage.”

Two hours later, it was at No. 71.

At 7 p.m. on Wednesday, Penney was No. 1 in searches for “living room furniture.”

By 9 p.m., it had sunk to No. 68.

In other words, one moment Penney was the most visible online destination for living room furniture in the country.

The next it was essentially buried.

PENNEY reacted to this instant reversal of fortune by, among other things, firing its search engine consulting firm, SearchDex. Executives there did not return e-mail or phone calls.

Penney also issued a statement: “We are disappointed that Google has reduced our rankings due to this matter,” Ms. Brossart wrote, “but we will continue to work actively to retain our high natural search position.”

In their (JC Penny's) response, they knew it was wrong and took the right action.