Forum for Science, Industry and Business

Sponsored by:     3M 
Search our Site:

 

Pinning Down the Fleeting Internet: Web Crawler Archives Historical Data for Easy Searching

19.11.2008
University of Washington researchers are grabbing hold of the fleeting Web and storing historical Web sites that users can easily search using an intuitive application called Zoetrope.

The Internet contains vast amounts of information, much of it unorganized. But what you see online at any given moment is just a snapshot of the Web as a whole – many pages change rapidly or disappear completely, and the old data gets lost forever.

“Your browser is really just a window into the Web as it exists today,” said Eytan Adar, University of Washington computer science and engineering doctoral student. “When you search for something online you’re only getting today’s results.”

Now, Adar and his colleagues at UW and Adobe Systems Inc. are grabbing hold of the fleeting Web and storing historical sites that users can easily search using an intuitive application called Zoetrope.

“There are so many ways of finding and manipulating and visualizing data on what we call ‘the today Web’ that it’s kind of amazing that there’s no way to do anything similar to the ephemeral Web,” said Dan Weld, a UW computer science and engineering professor who also worked on the application. One service, the Internet Archive, has been capturing old versions of Web sites for years, but the records for the stored sites are inconsistent, Weld said. More importantly, there’s no easy way to search the archive.

With Zoetrope, anyone will be able to use easy keyword searches to find archived Web information or look for patterns over time. The research was presented Oct. 22 by Mira Dontcheva, the system’s co-creator and a recently graduated UW computer science and engineering doctoral student now at Adobe Systems Inc., at the ACM Symposium on User Interface Software and Technology in Monterey, Calif.

There are a variety of ways people might want to search the historical Internet. For example, to find a history of traffic patterns in the Seattle area, you’d have to sort through lengthy PDF files from the state Department of Transportation, Adar said. With Zoetrope, you could easily view past versions of any traffic Web site, and getting more specific, search for drive-times on Interstate 90 at 6 p.m. on rainy Fridays. Zoetrope can also capture and help analyze information that might otherwise not be available anywhere.

Sports fanatics could use the program to check historical rankings of their favorite teams or players, information that currently may not be easy to find. The application can do more than just simple keyword searches, Adar said. It also can be used to analyze historical data or link information from different sites. For example, Adar wondered whether air pollution conditions could affect the performance of Olympic athletes, so he used Zoetrope to find daily records of pollution levels in Beijing and the number of world records broken in the 2008 Olympics on each day, and looked to see whether fewer records were broken on days with high pollution levels.

“Zoetrope is aimed at the casual researcher,” Weld said. “It’s really for anyone who has a question.”

Zoetrope could eventually be built in to any other Web browser, Adar said. If you just want to browse the past versions of a given site, you drag a slider backwards to see older and older versions. Alternatively, you can draw a box around just one part of the site, if you’re interested in, say, the lead story on CNN.com but don’t care about the rest of the page. These boxes can be filtered by keyword searches or date, so you could look only for lead stories featuring Hollywood actors or stories that ran on Fridays.

Users can view historical data by moving the slider, but more sophisticated analyses are available as well. If you’re looking at something numerical, such as gas prices over time, the program can draw graphs for you. Or you can pull out images from specific times, such as traffic pictures, and compare them all side by side. These kinds of visualizations can be further organized in a timeline or by clustering – Zoetrope can make an image comparing traffic patterns on sunny days versus cloudy days, for example.

Right now, Zoetrope saves a new version of approximately 1,000 different sites every hour, Adar said. It’s been running for four months, so records go no further than that, but Adar hopes to eventually incorporate information from the Internet Archive’s nearly 14 years of records into the program.

He wants to figure out how to scale the program up from 1,000 Web pages to all pages in existence, and has run studies to figure how often each page would need to be captured. For example, a traffic site or stock-watching page would need versions saved much more often than every hour, but there are many unchanging pages that could be archived less frequently. Eventually, Zoetrope could automatically figure out how often to capture a page based on how frequently it changes, Adar said.

“This is really a new way to think about storing information on the Web,” he said.

The researchers hope to release Zoetrope free, and say it may be available as early as next summer.

The National Science Foundation, the Achievement Rewards for College Scientists Foundation and the Washington Research Foundation provided funding for Zoetrope. James Fogarty, UW computer science and engineering assistant professor, also worked on the application.

For more information, contact Eytan Adar at eadar@cs.washington.edu or (650) 799-8823, or Weld at weld@cs.washington.edu or (206) 543-9196.

Rachel Tompa | Newswise Science News
Further information:
http://www.washington.edu
http://uwnews.org/article.asp?articleID=45255

More articles from Information Technology:

nachricht NASA CubeSat to test miniaturized weather satellite technology
10.11.2017 | NASA/Goddard Space Flight Center

nachricht New approach uses light instead of robots to assemble electronic components
08.11.2017 | The Optical Society

All articles from Information Technology >>>

The most recent press releases about innovation >>>

Die letzten 5 Focus-News des innovations-reports im Überblick:

Im Focus: A “cosmic snake” reveals the structure of remote galaxies

The formation of stars in distant galaxies is still largely unexplored. For the first time, astron-omers at the University of Geneva have now been able to closely observe a star system six billion light-years away. In doing so, they are confirming earlier simulations made by the University of Zurich. One special effect is made possible by the multiple reflections of images that run through the cosmos like a snake.

Today, astronomers have a pretty accurate idea of how stars were formed in the recent cosmic past. But do these laws also apply to older galaxies? For around a...

Im Focus: Visual intelligence is not the same as IQ

Just because someone is smart and well-motivated doesn't mean he or she can learn the visual skills needed to excel at tasks like matching fingerprints, interpreting medical X-rays, keeping track of aircraft on radar displays or forensic face matching.

That is the implication of a new study which shows for the first time that there is a broad range of differences in people's visual ability and that these...

Im Focus: Novel Nano-CT device creates high-resolution 3D-X-rays of tiny velvet worm legs

Computer Tomography (CT) is a standard procedure in hospitals, but so far, the technology has not been suitable for imaging extremely small objects. In PNAS, a team from the Technical University of Munich (TUM) describes a Nano-CT device that creates three-dimensional x-ray images at resolutions up to 100 nanometers. The first test application: Together with colleagues from the University of Kassel and Helmholtz-Zentrum Geesthacht the researchers analyzed the locomotory system of a velvet worm.

During a CT analysis, the object under investigation is x-rayed and a detector measures the respective amount of radiation absorbed from various angles....

Im Focus: Researchers Develop Data Bus for Quantum Computer

The quantum world is fragile; error correction codes are needed to protect the information stored in a quantum object from the deteriorating effects of noise. Quantum physicists in Innsbruck have developed a protocol to pass quantum information between differently encoded building blocks of a future quantum computer, such as processors and memories. Scientists may use this protocol in the future to build a data bus for quantum computers. The researchers have published their work in the journal Nature Communications.

Future quantum computers will be able to solve problems where conventional computers fail today. We are still far away from any large-scale implementation,...

Im Focus: Wrinkles give heat a jolt in pillared graphene

Rice University researchers test 3-D carbon nanostructures' thermal transport abilities

Pillared graphene would transfer heat better if the theoretical material had a few asymmetric junctions that caused wrinkles, according to Rice University...

All Focus news of the innovation-report >>>

Anzeige

Anzeige

Event News

Ecology Across Borders: International conference brings together 1,500 ecologists

15.11.2017 | Event News

Road into laboratory: Users discuss biaxial fatigue-testing for car and truck wheel

15.11.2017 | Event News

#Berlin5GWeek: The right network for Industry 4.0

30.10.2017 | Event News

 
Latest News

NASA detects solar flare pulses at Sun and Earth

17.11.2017 | Physics and Astronomy

NIST scientists discover how to switch liver cancer cell growth from 2-D to 3-D structures

17.11.2017 | Health and Medicine

The importance of biodiversity in forests could increase due to climate change

17.11.2017 | Studies and Analyses

VideoLinks
B2B-VideoLinks
More VideoLinks >>>