Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Pinning Down the Fleeting Internet: Web Crawler Archives Historical Data for Easy Searching

19.11.2008
University of Washington researchers are grabbing hold of the fleeting Web and storing historical Web sites that users can easily search using an intuitive application called Zoetrope.

The Internet contains vast amounts of information, much of it unorganized. But what you see online at any given moment is just a snapshot of the Web as a whole – many pages change rapidly or disappear completely, and the old data gets lost forever.

“Your browser is really just a window into the Web as it exists today,” said Eytan Adar, University of Washington computer science and engineering doctoral student. “When you search for something online you’re only getting today’s results.”

Now, Adar and his colleagues at UW and Adobe Systems Inc. are grabbing hold of the fleeting Web and storing historical sites that users can easily search using an intuitive application called Zoetrope.

“There are so many ways of finding and manipulating and visualizing data on what we call ‘the today Web’ that it’s kind of amazing that there’s no way to do anything similar to the ephemeral Web,” said Dan Weld, a UW computer science and engineering professor who also worked on the application. One service, the Internet Archive, has been capturing old versions of Web sites for years, but the records for the stored sites are inconsistent, Weld said. More importantly, there’s no easy way to search the archive.

With Zoetrope, anyone will be able to use easy keyword searches to find archived Web information or look for patterns over time. The research was presented Oct. 22 by Mira Dontcheva, the system’s co-creator and a recently graduated UW computer science and engineering doctoral student now at Adobe Systems Inc., at the ACM Symposium on User Interface Software and Technology in Monterey, Calif.

There are a variety of ways people might want to search the historical Internet. For example, to find a history of traffic patterns in the Seattle area, you’d have to sort through lengthy PDF files from the state Department of Transportation, Adar said. With Zoetrope, you could easily view past versions of any traffic Web site, and getting more specific, search for drive-times on Interstate 90 at 6 p.m. on rainy Fridays. Zoetrope can also capture and help analyze information that might otherwise not be available anywhere.

Sports fanatics could use the program to check historical rankings of their favorite teams or players, information that currently may not be easy to find. The application can do more than just simple keyword searches, Adar said. It also can be used to analyze historical data or link information from different sites. For example, Adar wondered whether air pollution conditions could affect the performance of Olympic athletes, so he used Zoetrope to find daily records of pollution levels in Beijing and the number of world records broken in the 2008 Olympics on each day, and looked to see whether fewer records were broken on days with high pollution levels.

“Zoetrope is aimed at the casual researcher,” Weld said. “It’s really for anyone who has a question.”

Zoetrope could eventually be built in to any other Web browser, Adar said. If you just want to browse the past versions of a given site, you drag a slider backwards to see older and older versions. Alternatively, you can draw a box around just one part of the site, if you’re interested in, say, the lead story on CNN.com but don’t care about the rest of the page. These boxes can be filtered by keyword searches or date, so you could look only for lead stories featuring Hollywood actors or stories that ran on Fridays.

Users can view historical data by moving the slider, but more sophisticated analyses are available as well. If you’re looking at something numerical, such as gas prices over time, the program can draw graphs for you. Or you can pull out images from specific times, such as traffic pictures, and compare them all side by side. These kinds of visualizations can be further organized in a timeline or by clustering – Zoetrope can make an image comparing traffic patterns on sunny days versus cloudy days, for example.

Right now, Zoetrope saves a new version of approximately 1,000 different sites every hour, Adar said. It’s been running for four months, so records go no further than that, but Adar hopes to eventually incorporate information from the Internet Archive’s nearly 14 years of records into the program.

He wants to figure out how to scale the program up from 1,000 Web pages to all pages in existence, and has run studies to figure how often each page would need to be captured. For example, a traffic site or stock-watching page would need versions saved much more often than every hour, but there are many unchanging pages that could be archived less frequently. Eventually, Zoetrope could automatically figure out how often to capture a page based on how frequently it changes, Adar said.

“This is really a new way to think about storing information on the Web,” he said.

The researchers hope to release Zoetrope free, and say it may be available as early as next summer.

The National Science Foundation, the Achievement Rewards for College Scientists Foundation and the Washington Research Foundation provided funding for Zoetrope. James Fogarty, UW computer science and engineering assistant professor, also worked on the application.

For more information, contact Eytan Adar at eadar@cs.washington.edu or (650) 799-8823, or Weld at weld@cs.washington.edu or (206) 543-9196.

Rachel Tompa | Newswise Science News
Further information:
http://www.washington.edu
http://uwnews.org/article.asp?articleID=45255

More articles from Information Technology:

nachricht Drones learn to navigate autonomously by imitating cars and bicycles
23.01.2018 | Universität Zürich

nachricht Cloud technology: Dynamic certificates make cloud service providers more secure
15.01.2018 | Technische Universität München

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: Optical Nanoscope Allows Imaging of Quantum Dots

Physicists have developed a technique based on optical microscopy that can be used to create images of atoms on the nanoscale. In particular, the new method allows the imaging of quantum dots in a semiconductor chip. Together with colleagues from the University of Bochum, scientists from the University of Basel’s Department of Physics and the Swiss Nanoscience Institute reported the findings in the journal Nature Photonics.

Microscopes allow us to see structures that are otherwise invisible to the human eye. However, conventional optical microscopes cannot be used to image...

Im Focus: Artificial agent designs quantum experiments

On the way to an intelligent laboratory, physicists from Innsbruck and Vienna present an artificial agent that autonomously designs quantum experiments. In initial experiments, the system has independently (re)discovered experimental techniques that are nowadays standard in modern quantum optical laboratories. This shows how machines could play a more creative role in research in the future.

We carry smartphones in our pockets, the streets are dotted with semi-autonomous cars, but in the research laboratory experiments are still being designed by...

Im Focus: Scientists decipher key principle behind reaction of metalloenzymes

So-called pre-distorted states accelerate photochemical reactions too

What enables electrons to be transferred swiftly, for example during photosynthesis? An interdisciplinary team of researchers has worked out the details of how...

Im Focus: The first precise measurement of a single molecule's effective charge

For the first time, scientists have precisely measured the effective electrical charge of a single molecule in solution. This fundamental insight of an SNSF Professor could also pave the way for future medical diagnostics.

Electrical charge is one of the key properties that allows molecules to interact. Life itself depends on this phenomenon: many biological processes involve...

Im Focus: Paradigm shift in Paris: Encouraging an holistic view of laser machining

At the JEC World Composite Show in Paris in March 2018, the Fraunhofer Institute for Laser Technology ILT will be focusing on the latest trends and innovations in laser machining of composites. Among other things, researchers at the booth shared with the Aachen Center for Integrative Lightweight Production (AZL) will demonstrate how lasers can be used for joining, structuring, cutting and drilling composite materials.

No other industry has attracted as much public attention to composite materials as the automotive industry, which along with the aerospace industry is a driver...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

10th International Symposium: “Advanced Battery Power – Kraftwerk Batterie” Münster, 10-11 April 2018

08.01.2018 | Event News

See, understand and experience the work of the future

11.12.2017 | Event News

Innovative strategies to tackle parasitic worms

08.12.2017 | Event News

 
Latest News

Researchers reveal how microbes cope in phosphorus-deficient tropical soil

23.01.2018 | Earth Sciences

Opening the cavity floodgates

23.01.2018 | Life Sciences

Siberian scientists suggested a new method for synthesizing a promising magnetic material

23.01.2018 | Materials Sciences

VideoLinks Science & Research
Overview of more VideoLinks >>>