org.jscience.net
Class MediaCrawler

java.lang.Object
  java.lang.Thread
      org.jscience.net.MediaCrawler

All Implemented Interfaces:

java.lang.Runnable, CrawlerSetting

public class MediaCrawler
extends java.lang.Thread
implements CrawlerSetting

MediaCrawler is a single thread that searches the web for files that are of a given type.

Since:

10/29/2002

See Also:

Spider

Nested Class Summary

static interface

MediaCrawler.Handler used to handle the media files found during the search of the MediaCrawler

Nested classes/interfaces inherited from class java.lang.Thread

java.lang.Thread.State, java.lang.Thread.UncaughtExceptionHandler

Field Summary

Fields inherited from class java.lang.Thread

MAX_PRIORITY, MIN_PRIORITY, NORM_PRIORITY

Constructor Summary

MediaCrawler(java.net.URL rootURL, int depth, java.lang.String mediaExtension, boolean currentSiteOnly, MediaCrawler.Handler handler, java.lang.String[] pattern)

MediaCrawler(java.net.URL rootURL, int depth, java.lang.String mediaExtension, boolean currentSiteOnly, java.lang.String[] pattern)

Method Summary

void	addHandler(MediaCrawler.Handler handler)
boolean	followLinks(java.net.URL url, java.net.URL referer, int depth, java.util.List<java.net.URL> resultURLList, java.util.List<java.net.URL> closedURLList, java.util.List<Spider.URLWrapper> searchURLWrapperList) followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level.
URLCache[]	getFilesFound()
boolean	matchesCriteria(java.net.URL url, java.net.URL referer, int depth, java.util.List<java.net.URL> resultURLList, java.util.List<java.net.URL> closedURLList) This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do.
void	run()

Methods inherited from class java.lang.Thread

activeCount, checkAccess, countStackFrames, currentThread, destroy, dumpStack, enumerate, getAllStackTraces, getContextClassLoader, getDefaultUncaughtExceptionHandler, getId, getName, getPriority, getStackTrace, getState, getThreadGroup, getUncaughtExceptionHandler, holdsLock, interrupt, interrupted, isAlive, isDaemon, isInterrupted, join, join, join, resume, setContextClassLoader, setDaemon, setDefaultUncaughtExceptionHandler, setName, setPriority, setUncaughtExceptionHandler, sleep, sleep, start, stop, stop, suspend, toString, yield

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Constructor Detail

MediaCrawler

public MediaCrawler(java.net.URL rootURL,
                    int depth,
                    java.lang.String mediaExtension,
                    boolean currentSiteOnly,
                    java.lang.String[] pattern)

MediaCrawler

public MediaCrawler(java.net.URL rootURL,
                    int depth,
                    java.lang.String mediaExtension,
                    boolean currentSiteOnly,
                    MediaCrawler.Handler handler,
                    java.lang.String[] pattern)

Method Detail

addHandler

public void addHandler(MediaCrawler.Handler handler)

run

public void run()

Specified by:

run in interface java.lang.Runnable

Overrides:

run in class java.lang.Thread

getFilesFound

public URLCache[] getFilesFound()

followLinks

public boolean followLinks(java.net.URL url,
                           java.net.URL referer,
                           int depth,
                           java.util.List<java.net.URL> resultURLList,
                           java.util.List<java.net.URL> closedURLList,
                           java.util.List<Spider.URLWrapper> searchURLWrapperList)

Description copied from interface: CrawlerSetting

followLinks() determines whether the given URL is to be searched for its links to be examined further in the next level. The three List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs.

Specified by:

followLinks in interface CrawlerSetting

Parameters:

url - the URL that is to be examined for its links

referer - url's referer URL

depth - distance from the original root URL where the search began

resultURLList - List of URLs that have already been found to match this CrawlerSetting's criteria

closedURLList - List of URLs that have already been found not to match the CrawlerSetting's criteria

searchURLWrapperList - List of Spider.URLWrapper objects already identified to be examined in the next level

See Also:

Spider.URLWrapper

matchesCriteria

public boolean matchesCriteria(java.net.URL url,
                               java.net.URL referer,
                               int depth,
                               java.util.List<java.net.URL> resultURLList,
                               java.util.List<java.net.URL> closedURLList)

Description copied from interface: CrawlerSetting

This method decides whether either the URL itself or its content qualifies for what this CrawlerSetting searches for; as this function is also called on every URL encountered, it is also the place for any custom parsing this CrawlerSetting wants to do. The two List objects allow the CrawlerSetting to act on potential constrains that may result from e.g. a maximum number of total nodes to be examined (or any other custom checking imaginable). Note that it is the responsibility of the calling object to ensure that this function isn't called multiple times on the same URL if that's not desired. The url may include any URL, including non-HTTP protocols (such as mailto:, ftp:) and image or media URLs

Specified by:

matchesCriteria in interface CrawlerSetting

Parameters:

url - the URL in question to satisfy the criteria

referer - url's referer URL

depth - link distance from the original root URL where the search began

resultURLList - List of URLs that have already been found to match this CrawlerSetting's criteria

closedURLList - List of URLs that have already been found not to match the CrawlerSetting's criteria