Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Pinning Down the Fleeting Internet: Web Crawler Archives Historical Data for Easy Searching

19.11.2008
University of Washington researchers are grabbing hold of the fleeting Web and storing historical Web sites that users can easily search using an intuitive application called Zoetrope.

The Internet contains vast amounts of information, much of it unorganized. But what you see online at any given moment is just a snapshot of the Web as a whole – many pages change rapidly or disappear completely, and the old data gets lost forever.

“Your browser is really just a window into the Web as it exists today,” said Eytan Adar, University of Washington computer science and engineering doctoral student. “When you search for something online you’re only getting today’s results.”

Now, Adar and his colleagues at UW and Adobe Systems Inc. are grabbing hold of the fleeting Web and storing historical sites that users can easily search using an intuitive application called Zoetrope.

“There are so many ways of finding and manipulating and visualizing data on what we call ‘the today Web’ that it’s kind of amazing that there’s no way to do anything similar to the ephemeral Web,” said Dan Weld, a UW computer science and engineering professor who also worked on the application. One service, the Internet Archive, has been capturing old versions of Web sites for years, but the records for the stored sites are inconsistent, Weld said. More importantly, there’s no easy way to search the archive.

With Zoetrope, anyone will be able to use easy keyword searches to find archived Web information or look for patterns over time. The research was presented Oct. 22 by Mira Dontcheva, the system’s co-creator and a recently graduated UW computer science and engineering doctoral student now at Adobe Systems Inc., at the ACM Symposium on User Interface Software and Technology in Monterey, Calif.

There are a variety of ways people might want to search the historical Internet. For example, to find a history of traffic patterns in the Seattle area, you’d have to sort through lengthy PDF files from the state Department of Transportation, Adar said. With Zoetrope, you could easily view past versions of any traffic Web site, and getting more specific, search for drive-times on Interstate 90 at 6 p.m. on rainy Fridays. Zoetrope can also capture and help analyze information that might otherwise not be available anywhere.

Sports fanatics could use the program to check historical rankings of their favorite teams or players, information that currently may not be easy to find. The application can do more than just simple keyword searches, Adar said. It also can be used to analyze historical data or link information from different sites. For example, Adar wondered whether air pollution conditions could affect the performance of Olympic athletes, so he used Zoetrope to find daily records of pollution levels in Beijing and the number of world records broken in the 2008 Olympics on each day, and looked to see whether fewer records were broken on days with high pollution levels.

“Zoetrope is aimed at the casual researcher,” Weld said. “It’s really for anyone who has a question.”

Zoetrope could eventually be built in to any other Web browser, Adar said. If you just want to browse the past versions of a given site, you drag a slider backwards to see older and older versions. Alternatively, you can draw a box around just one part of the site, if you’re interested in, say, the lead story on CNN.com but don’t care about the rest of the page. These boxes can be filtered by keyword searches or date, so you could look only for lead stories featuring Hollywood actors or stories that ran on Fridays.

Users can view historical data by moving the slider, but more sophisticated analyses are available as well. If you’re looking at something numerical, such as gas prices over time, the program can draw graphs for you. Or you can pull out images from specific times, such as traffic pictures, and compare them all side by side. These kinds of visualizations can be further organized in a timeline or by clustering – Zoetrope can make an image comparing traffic patterns on sunny days versus cloudy days, for example.

Right now, Zoetrope saves a new version of approximately 1,000 different sites every hour, Adar said. It’s been running for four months, so records go no further than that, but Adar hopes to eventually incorporate information from the Internet Archive’s nearly 14 years of records into the program.

He wants to figure out how to scale the program up from 1,000 Web pages to all pages in existence, and has run studies to figure how often each page would need to be captured. For example, a traffic site or stock-watching page would need versions saved much more often than every hour, but there are many unchanging pages that could be archived less frequently. Eventually, Zoetrope could automatically figure out how often to capture a page based on how frequently it changes, Adar said.

“This is really a new way to think about storing information on the Web,” he said.

The researchers hope to release Zoetrope free, and say it may be available as early as next summer.

The National Science Foundation, the Achievement Rewards for College Scientists Foundation and the Washington Research Foundation provided funding for Zoetrope. James Fogarty, UW computer science and engineering assistant professor, also worked on the application.

For more information, contact Eytan Adar at eadar@cs.washington.edu or (650) 799-8823, or Weld at weld@cs.washington.edu or (206) 543-9196.

Rachel Tompa | Newswise Science News
Further information:
http://www.washington.edu
http://uwnews.org/article.asp?articleID=45255

More articles from Information Technology:

nachricht World first: 'Storing lightning inside thunder'
18.09.2017 | University of Sydney

nachricht New software turns mobile-phone accessory into breathing monitor
14.09.2017 | The Optical Society

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Ultrafast snapshots of relaxing electrons in solids

Using ultrafast flashes of laser and x-ray radiation, scientists at the Max Planck Institute of Quantum Optics (Garching, Germany) took snapshots of the briefest electron motion inside a solid material to date. The electron motion lasted only 750 billionths of the billionth of a second before it fainted, setting a new record of human capability to capture ultrafast processes inside solids!

When x-rays shine onto solid materials or large molecules, an electron is pushed away from its original place near the nucleus of the atom, leaving a hole...

Im Focus: Quantum Sensors Decipher Magnetic Ordering in a New Semiconducting Material

For the first time, physicists have successfully imaged spiral magnetic ordering in a multiferroic material. These materials are considered highly promising candidates for future data storage media. The researchers were able to prove their findings using unique quantum sensors that were developed at Basel University and that can analyze electromagnetic fields on the nanometer scale. The results – obtained by scientists from the University of Basel’s Department of Physics, the Swiss Nanoscience Institute, the University of Montpellier and several laboratories from University Paris-Saclay – were recently published in the journal Nature.

Multiferroics are materials that simultaneously react to electric and magnetic fields. These two properties are rarely found together, and their combined...

Im Focus: Fast, convenient & standardized: New lab innovation for automated tissue engineering & drug

MBM ScienceBridge GmbH successfully negotiated a license agreement between University Medical Center Göttingen (UMG) and the biotech company Tissue Systems Holding GmbH about commercial use of a multi-well tissue plate for automated and reliable tissue engineering & drug testing.

MBM ScienceBridge GmbH successfully negotiated a license agreement between University Medical Center Göttingen (UMG) and the biotech company Tissue Systems...

Im Focus: Silencing bacteria

HZI researchers pave the way for new agents that render hospital pathogens mute

Pathogenic bacteria are becoming resistant to common antibiotics to an ever increasing degree. One of the most difficult germs is Pseudomonas aeruginosa, a...

Im Focus: Artificial Enzymes for Hydrogen Conversion

Scientists from the MPI for Chemical Energy Conversion report in the first issue of the new journal JOULE.

Cell Press has just released the first issue of Joule, a new journal dedicated to sustainable energy research. In this issue James Birrell, Olaf Rüdiger,...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

“Lasers in Composites Symposium” in Aachen – from Science to Application

19.09.2017 | Event News

I-ESA 2018 – Call for Papers

12.09.2017 | Event News

EMBO at Basel Life, a new conference on current and emerging life science research

06.09.2017 | Event News

 
Latest News

“Lasers in Composites Symposium” in Aachen – from Science to Application

19.09.2017 | Event News

New quantum phenomena in graphene superlattices

19.09.2017 | Physics and Astronomy

A simple additive to improve film quality

19.09.2017 | Power and Electrical Engineering

VideoLinks
B2B-VideoLinks
More VideoLinks >>>