Processing Monitor

• Crawler
• Loader
• Rejected URLs

Queues

• Local
#(navigation-p2p)#::
• Global
#(/navigation-p2p)# #(navigation-p2p)#::
• Remote
#(/navigation-p2p)#
• No-Load

Crawler Steering

• Scheduler and Profile Editor
• robots.txt Monitor

• Overview
#(navigation-p2p)#::
• (1) Receipts
#(/navigation-p2p)#
• (2) Queries
#(navigation-p2p)#::
• (3) DHT Transfer
#(/navigation-p2p)#
• (4) Proxy Use
• (5) Local Crawling
#(navigation-p2p)#::
• (6) Global Crawling
#(/navigation-p2p)#
• (7) Surrogate Import