Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

New Metasearch Engine Leaves Google, Yahoo Crawling

27.03.2009
One day in the not-too-distant future, you’ll be able to type a query into an online search engine and have it deliver not Web pages that may contain an answer, but just the answer itself, says Weiyi Meng, a professor of computer science at Binghamton University, State University of New York.

For instance, imagine typing in “Who starred in the film Casablanca?” The search engine would respond with “Humphrey Bogart and Ingrid Bergman.”

Not impressed?

Try asking a more nuanced question, such as “What do Americans think of universal health care?” A search engine will create a report indicating trends in opinion based on what has been posted to the Web.

Search engines may eventually be used to conduct polling and even help sort fact from fiction, said Meng, who is helping to make such possibilities a reality, both through his research and as president of a company called Webscalers.

The way Meng sees it, big search engines such as Google and Yahoo are fundamentally flawed. The Web has two parts: the surface Web and the deep Web. The surface Web is made up of perhaps 60 billion pages. The deep Web, at some 900 billion pages, is about 15 times larger.

Google, which relies on a “crawler” to examine pages and catalog them for future searches, can search about 20 billion pages. Web crawlers follow links to reach pages and often miss content that isn’t linked to any other page or is in some way “hidden.”

Meng, along with researchers at the University of Illinois at Chicago and the University of Louisiana at Lafayette, has helped pioneer large-scale metasearch-engine technology that harnesses the power of small search engines to come up with results that are more accurate and more complete.

“Most of the pages on the deep Web aren’t directly ‘crawlable.’ We want to connect to small search engines and reach the deep Web,” he said. “That’s the idea. Many people have the misconception that Google can search everything, and if it’s not there it doesn’t exist. But we should be able to retrieve many times more than what Google can search.”

Not only can a metasearch engine probe deeper, it can also offer the latest information.

“In principle,” Meng said, “small guys are much better able to maintain the freshness of their data. Google has a program to ‘crawl’ all over the world. Depending on when the crawler has last visited your server, there’s a delay of days or weeks before a new page will show up in that search. We can get fresher results.”

The concept is not new. In fact, the first metasearch engine was built in 1994.
“The big difference between our technology and the ones pursued by other people is that most of the other technologies do the metasearching on top of a small number of general-purpose search engines, such as Yahoo, Google or MSN,” Meng explained. “We have a completely different perspective. We want to build large-scale metasearch engines on top of many small search engines.”

The Web has millions of search engines at businesses, universities, newspapers and other organizations. Since 1997, and with continued funding from the National Science Foundation, Meng and his collaborators have found ways to run queries across multiple search engines and sort through the results.

Webscalers is based in the Start-Up Suite at Binghamton University’s Innovative Technologies Complex, which is home to several young companies that have their roots in faculty inventions.

“If the Web keeps on growing, a company like Google may run out of resources to crawl all of those pages,” said Vijay V. Raghavan, vice president of Webscalers and a faculty member at the University of Louisiana at Lafayette. “We won’t have that problem. We will scale much better.”

Webscalers’ technology could be useful for large organizations with many divisions. For example, Webscalers has developed a prototype that would allow a search of all 64 campuses in the State University of New York system as well as SUNY’s central administration.

“People can use it to find collaborators,” Meng said. “It could also help prospective students find programs they’re interested in.”

The technology could be adapted to large companies or even the government, Meng said.

Challenges for large-scale metasearch engines include determining which search engines are the best for a given query, automating the interaction with search engines as well as organizing the search results.

Meng hopes to build a grand metasearch engine one day that would integrate all of the 1 million small search engines into a single system. “There are still a lot of significant challenges in creating a system of such magnitude,” he said, “but I am optimistic that such a metasearch engine can be built.”

Try out the concept online
Webscalers has already launched several metasearch products:
• The first is a news metasearch engine called AllinOneNews. Available at www.allinonenews.com, it connects to 1,800 news sources in 200 countries. That’s the largest metasearch engine in the world.

• Webscalers also offers MySearchView, a system that allows any user to create his or her own metasearch engine just by checking off a few options at www.mysearchview.com

Gail Glover | Newswise Science News
Further information:
http://www.binghamton.edu
http://research.binghamton.edu/

More articles from Information Technology:

nachricht Smart Computers
18.08.2017 | Albert-Ludwigs-Universität Freiburg im Breisgau

nachricht AI implications: Engineer's model lays groundwork for machine-learning device
18.08.2017 | Washington University in St. Louis

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Fizzy soda water could be key to clean manufacture of flat wonder material: Graphene

Whether you call it effervescent, fizzy, or sparkling, carbonated water is making a comeback as a beverage. Aside from quenching thirst, researchers at the University of Illinois at Urbana-Champaign have discovered a new use for these "bubbly" concoctions that will have major impact on the manufacturer of the world's thinnest, flattest, and one most useful materials -- graphene.

As graphene's popularity grows as an advanced "wonder" material, the speed and quality at which it can be manufactured will be paramount. With that in mind,...

Im Focus: Exotic quantum states made from light: Physicists create optical “wells” for a super-photon

Physicists at the University of Bonn have managed to create optical hollows and more complex patterns into which the light of a Bose-Einstein condensate flows. The creation of such highly low-loss structures for light is a prerequisite for complex light circuits, such as for quantum information processing for a new generation of computers. The researchers are now presenting their results in the journal Nature Photonics.

Light particles (photons) occur as tiny, indivisible portions. Many thousands of these light portions can be merged to form a single super-photon if they are...

Im Focus: Circular RNA linked to brain function

For the first time, scientists have shown that circular RNA is linked to brain function. When a RNA molecule called Cdr1as was deleted from the genome of mice, the animals had problems filtering out unnecessary information – like patients suffering from neuropsychiatric disorders.

While hundreds of circular RNAs (circRNAs) are abundant in mammalian brains, one big question has remained unanswered: What are they actually good for? In the...

Im Focus: RAVAN CubeSat measures Earth's outgoing energy

An experimental small satellite has successfully collected and delivered data on a key measurement for predicting changes in Earth's climate.

The Radiometer Assessment using Vertically Aligned Nanotubes (RAVAN) CubeSat was launched into low-Earth orbit on Nov. 11, 2016, in order to test new...

Im Focus: Scientists shine new light on the “other high temperature superconductor”

A study led by scientists of the Max Planck Institute for the Structure and Dynamics of Matter (MPSD) at the Center for Free-Electron Laser Science in Hamburg presents evidence of the coexistence of superconductivity and “charge-density-waves” in compounds of the poorly-studied family of bismuthates. This observation opens up new perspectives for a deeper understanding of the phenomenon of high-temperature superconductivity, a topic which is at the core of condensed matter research since more than 30 years. The paper by Nicoletti et al has been published in the PNAS.

Since the beginning of the 20th century, superconductivity had been observed in some metals at temperatures only a few degrees above the absolute zero (minus...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

Call for Papers – ICNFT 2018, 5th International Conference on New Forming Technology

16.08.2017 | Event News

Sustainability is the business model of tomorrow

04.08.2017 | Event News

Clash of Realities 2017: Registration now open. International Conference at TH Köln

26.07.2017 | Event News

 
Latest News

A Map of the Cell’s Power Station

18.08.2017 | Life Sciences

Engineering team images tiny quasicrystals as they form

18.08.2017 | Physics and Astronomy

Researchers printed graphene-like materials with inkjet

18.08.2017 | Materials Sciences

VideoLinks
B2B-VideoLinks
More VideoLinks >>>