Select the Sources Tab of the left pane

Figure 8.1. The Sources Tab

and press the New button at the bottom of the same pane. A Source labelled as (new) is shown.

Figure 8.2. A new unconfigured Source

Assign a Name and a Description to the newly created Source and press the Apply button to confirm provided information.

Figure 8.3. Setting Name and Description of a new Source

Now that the Source is created you need to add almost one Seed to it.

Just after the creation of a Source the list of Seeds it is composed of is empty.

Figure 8.4. The Seed box

Clicking on the Add... the following window is shown.

Figure 8.5. The Seed window

In the left side of the window the list of available types of Seed is possible to add to the current Source.

Here it is a detailed description of the configuration issues of any available type of Seed.

Web site

For a Seed related to a Web Site accessible through HTTP protocol the only information searchbox needs are the URL of the page where the gathering process must begin from and if the site needs a secure access.

Figure 8.6. Web site Seed

Checking the Secure checkbox the URL prefix will change from http:// to https://

FTP site

For FTP Seed the only needed parameter is its URL.

Figure 8.7. FTP site Seed

Gopher site

For Gopher sites just the URL is needed

Figure 8.8. Gopher site Seed

For more information about the Gopher protocol you can see: http://en.wikipedia.org/wiki/Gopher_protocol

Usenet site

In this case both server name and the name of the newsgroup we want to gather from are needed.

Figure 8.9. Usenet site Seed

Filesystem

Documents must reside in the local filesystem or in a filesystem remotely mounted on the server where searchbox is running. The complete path must be specified.

Note

Remote folders locally mounted are guaranteed to work only for Linux OSX and Windows 2000. For Windows XP the SMB protocol must be used.

Figure 8.10. Filesystem Seed

Mailbox

The complete address of the incoming mail server and its type. POP3/S and IMAP/S are the SSL version of POP3 e IMAP servers.

Figure 8.11. Mailbox Seed

For all types of servers the messages are not removed from the original location and are not marked as read.

WebDav share

The address of remote server complete of the full path is required. Check the Secure checkbox if the server requires a secure access.

Figure 8.12. WebDav share Seed

SMB share

It needs the server which is publishing a folder using the SAMBA protocol and the full path where the gathering agent must start its job.

Figure 8.13. SMB share Seed

ODBC

An ODBC seed is made of three parts:

• The ODBC connection string. It is usually a System DSN^[10]. Note that your database might impose access restrictions that will stop searchbox to access it even if you can access the database itself from your desktop. E.g. the database might be configured to refuse queries coming from the searchbox server computer.

• The query. This is basically any sql query with the additional requirement that exactly one of the special strings --!!!PKW!!!-- or --!!!PKA!!!-- must be present. These special strings are described below.

• The keys. This is a subset of the names of the columns of the result set returned by query whose value uniquely identifies a row in the result set itself. For example if query involves a single table you can use the table primary key. You must specify the order of the fields^[11] and whether each field contains a string or a numeric value.

Figure 8.14. ODBC Seed

As stated above query must contain exactly one of --!!!PKW!!!-- or --!!!PKA!!!--. searchbox will expand these with value taken from the keys fields to access specific rows in the database. The two strings are expanded nearly in the same way but --!!!PKW!!!-- begins with an sql where keyword, while --!!!PKA!!!-- begins with and. Thus you must use the latter when you have placed a where clause in query (typically you append --!!!PKA!!!-- after your clause) while the former is useful when you don't have any explicit where clause in your query. For example:

select * from tableA --!!!PKW!!!--

but

select * from tableB, tableC
  where tableB.id = tableC.id
  --!!!PKA!!!--

If you don't specify the --!!!PKW!!!-- nor the --!!!PKA!!!-- placeholder then the controlpanel will complain.

Figure 8.15. ODBC Seed: no placeholder warning

In order to successfully crawl an odbc source adding a seed is not enough: you also have to specify a set of rules to let searchbox turn the odbc result set into documents. These rules are specified in an odbc plugin configuration. When you add an odbc seed to a source you must then enable the suitable plugin configuration on the source.

Figure 8.16. ODBC Seed: enabling the odbc plugin

If you don't searchbox complains.

Figure 8.17. ODBC Seed: no plugin warning

Other

This type of seed is reserved for all protocols that are not natively available in searchbox nor in one of its bundled plugins.

Figure 8.18. "other" Seed

Please see the documentation of the fetching plugin you have to use to properly configure this seed.

All Seeds specified will be listed in the URL box of the Info section of the Source configuration tab.

Figure 8.19. The list of active Seeds

Pressing the Apply... button or any other widget of the Control Panel interface the following warning message will pop up

Figure 8.20. No configured filters warning

In order to let any Source to gather contents from specified Seeds almost one inclusion filter for each of them must be specified. If you do not specify any the Control Panel will automatically add the default one. The default inclusion filter will include all subsections starting from the specified Seed entry.

Experienced users who need to perform a selective gathering will have to edit manually filters.

Figure 8.21. Source configured with default inclusion filters

Any gathering activity starting from a specific seed is characterized by a discrete distance from the starting point. Such distance is the number of links that the gatherer (spider for the web) had followed to reach a document. Such distance is called Depth Limit.

Figure 8.22. Depth Limit

The above picture shows a graph with nodes labelled with their Depth Level calculated from the Seed (labelled as 0).

Even if the Depth Limit concept seems correct only for tree data structures (because they have a "root" node) it is used for graphs too as in the case of the Web.

In the following table describes how the Depth Limit concept is used by each type of Seed available in searchbox:

In the Option Tab of the Source configuration section is possible to specify the Depth Limit parameter.

Figure 8.23. Depth Limit parameter

As default this option is unchecked that means "no Depth Limit".

This option is checked as default and force searchbox to respect the robots.txt protocol.

Figure 8.24. robots.txt parameter

If you want to override the default of this option please be sure that the owner of Source is informed. Ignoring the robots.txt directives can be a sufficient reason to be banned from accessing to a Web source.

Figure 8.25. Has side-by-side metadata option

In some cases it is very useful to gather explicit metadata referred to a document directly from the Source even if they cannot be embedded into document itself. For this cases searchbox can manage a proprietary data format that can be adopted in the case we can control how the source publish such information.

The basic idea is to associate to each document of a Source a set of metadata contained in another XML formatted document (resource document) in the same location of the original one and with the same name but with a special extension.

Given a document named as mydocument.pdf the associated resource document will be mydocument.pdf.sbm

If this option is checked, for each file of the source, the gathering agent will check if the corresponding resource file is present and will merge the metadata contained in it with the original document.

The resource document has the following structure:

<?xml version="1.0"?>
<metainfo>
<meta type="documentwide" sliceid=”slicename” key="name1">value1</meta>
<meta type="documentwide" sliceid=”slicename” key="name2">value2</meta>
...
</metainfo>

where:

Moreover the following optional attributes can be used in a meta tag:

searchbox lets you put in your HTML documents special tags that it can use to extract metadata to your documents. This feature is disabled by default. To enable it you must set the USE_SEARCHBOX_METAS parameter in your searchbox configuration file as detailed in the section called “Handling of <meta name="searchbox-xxxx"> tags in HTML documents”.

You specify metadata by placing suitably formatted <META> tags in your HTML document <HEAD> section.

The full format is:

where:

For example the following HTML excerpt:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
	<head>
        	<meta name="searchbox-NT-centralNorm-metakey0" content="metavalue0" />
	        <meta name="searchbox--centralNorm-metakey1" content="metavalue1" />
<!-- more stuff follows ... -->

will add two metadata: a normalized, tokenized metadata with key metakey0 and value metavalue0 and another, literal one, with key metakey1 and value metavalue1. Both metadata will be placed in the centralNorm slice.

Some web sources grant access only to authenticated users. This happens for both sources that requires username and password and sources that use cookies. The searchbox gathering agent is able to simulate the way a real user explore a web site using a standard web browser so usually such Sources can be easily gathered once configured.

This option let you chose between three different types of authentication methods for a Source.

Figure 8.26. Authentication methods

Cookie authentication

With this modality searchbox can acquire authentication cookies from a server simulating what a user will do with his/her browser (filling a form and clicking the "submit" button). All what searchbox needs to log into a web site is to perform a POST action in the right forms so that a configuration with all variable names and its values is needed.

Let's imagine that the authentication is performed through the page http://somesite.com/login.php: the login name would be a variable called username and the password a variable called password. In this case searchbox will provide a POST action of these variables at another URL (i.e. http://somesite.com/authcheck.php) and will store all cookies that will be provided. An problem that can occur is that page http://somesite.com/authcheck.php is a redirect to another page. In this case we must refer the POST action to the destination page.

Some sites generates cookies in the login page and in other pages between the home page and the login page itself. In this case searchbox must be configured to manage this situation. Such pages must be specified into Cookie Pre URLs fields.

Clicking on the Details... button the following window appears

Figure 8.27. Cookie authentication details

• The URL field must be filled with the URL of the page where a normal user authenticate itself.

• The Action parameter specifies if data are passed with GET or POST command and the maximum age the cookie can have before a new login is required.

• Into Username and Password fields the name of the parameter for username and password must be specified (i.e. USERID and PASSWORD). Values of these parameters have to be specified into the configuration of the Archive associated to the Source.

• In the Other parameters section more parameter/value couple can be specified if needed by the above Action

• In the Cookie Pre URLs section the list of pages between the home page and the login page that generate cookies.

This is a specific feature of searchbox for HTML documents. into the Gathering Tab of each Source is possible to specify two regular expressions, one for the "start ignoring stuff" marker and the other for the "stop ignoring stuff" marker.

Figure 8.28. "Exclude text between" option

All the text between the specified markers will not be passed to the rendering module.

Select the Sources Tab of the left panel.

Figure 8.29. The Plugin status panel

Select the Fetching item and click on New button.

To create a new plugin there are two different way: New plugin or Inherit from existing plugin as shown in the following window:

Figure 8.30. The new plugin window

Choosing the New plugin option select the one of available types. In the following example the odbcplugin is selected.

Figure 8.31. A new odbc type plugin

The list of available type of plugin may vary depending from your searchbox Engine installation.

After clicking on the Ok button the Plugin configuration panel appear.

Figure 8.32. The plugin configuration panel

In order fully configure the plugin the following action are required:

As soon the configuration is completed the new plugin is listed in the left panel under its category (Under fetching plugings in this case).

Figure 8.33. A freshly configured plugin

A Fetching Plugin is used to add to a specific Source a new fetching protocol not natively managed by searchbox. In order to configure a Source to use such type of plugin it must be previously installed and configured from the Plugins Tab of the left panel of Control Panel (see Configuration of a Fetching Plugin).

The crawl filters are used to define the gathering agents behaviour using a list of constraints on the syntax about the URLs that must be included or excluded by the gathering action. For this purpose two types of filters exists: inclusion and exclusion filters.

In the Crawling Tab of a Source a default inclusion filter can be added checking a specific checkbox. The purpose of such default filter is to limit the action of the gathering agent all pages contained in the original domain of the seed page. For instance with the seed:

www.repubblica.it

the automatically generated inclusion filter is:

^http://www\.repubblica\.it/.*$

where:

Such filter is able to gather all reachable pages from the home page of Italian newspaper "La Repubblica".

Other examples:

^http://www\.repubblica\.it/indici/cronaca/cronaca\.htm$

The crawler starts from a subsection of "La repubblica" web site gathering pages with .htm extension only.

^http://www\.repubblica\.it/200[0-9]/[a-zA-Z]/sezioni/cronaca/.*$

The crawler gathers all the pages contained in a subsection of "La Repubblica" web site with a parametric path. In the specific case there is a division by years so the regular expression 200[0-9] has been used to consider all years between 2000 al 2009. The same thing for the other directory level that is described by one alphabetic char.

^http://www\.agronotizie\.com/(/)?sezioni/articolo\.cfm\?(C|c)odic
e=[0-9]+&codcanale=[0-9]+&codargomento=.*$

This is a more complex case that is used to gather dynamic pages. In the example the regular expression (/)? take into account that a single or a double slash can be present while the (C|c)odice expression that the parameter can be with capital letter or not.

An detailed discussion about the use of regular expressions is beyond thee scope of this manual. A good reference for this topic can be found in every manual of a programming language with regular expression management like Perl, Python, Grep, Lex, etc. Alternatively you can read the Wikipedia regular expressions page: http://en.wikipedia.org/wiki/Regular_expressions. searchbox implements the POSIX modern (extended) regular expressions variant described on that page.

Select the Sources Tab of the left pane

Figure 8.34. The Archive Tab

and press the New button at the bottom of the same pane. A pop-up window with the list of active Sources is shown.

Figure 8.35. List of active sources

To create an Archive you need to choose the Source to gather from. If the Source has been already configured is shown in this list and can be chosen.

Pressing the Ok button a new source named by default as the corresponding Source is shown.

Figure 8.36. A new unconfigured Archive

To finish the basic configuration of an Archive you have to choose the depth and the type of the document cache.

In the Cache depth section by default only the last gathered document is stored. Once a new one is gathered the older is thrown away. Selecting the All document versions option different versions of the same document are historicized and are thrown away only in accordance with the garbage collection configuration.

In the Cache type section there are three possible choices:

From the Info Tab of the Archive configuration panel it is possible to obtain some synthetic information about the current gathering process:

Figure 8.37. Gathering infos

For testing purposes or if you need to run gathering only once the Start and Stop buttons are available into the Info Tab of the Archive configuration panel.

Figure 8.38. Start/Stop Gathering buttons

If you stop a scheduled gathering activity it will start again according with its scheduling.

When the plugin chain associated to a Source changes if you need to update with the new configuration the documents already stored into the Archive you need to perform a reprocess operation. Such operation is equivalent to a Gathering apart that documents are not fetched from the original Source but from the local Archive.

Figure 8.39. The reprocessing button

Clicking the Reprocessing documents button an immediate gathering from the local archive starts.

Toi completely remove the content if an archive without deleting itself use the Reset button in the Archive Maintencance section of Info Tab of of the Archive configuration panel.

Figure 8.40. The Reset button

After clicking the above button a confirmation alert will be displayed before resetting the Archive.

In the Info Tab of the Archive configuration panel the View gathering logs... button

Figure 8.41. The View gathering logs button

shows a pop-up window that reports a detailed live log of the gathering process.

Figure 8.42. The Gathering Logs windows

Each row of the log is structured as:

Time. The date and time of the gathering action

URL. The URL of the gathered document

Status. One of the following:

Description. More details about the status generated by specific errors.

Stopping the on-line log download (the default when the window is opened) the Export... button became active.

Figure 8.43. Gathering Logs export

A Choose Export File window let you choose location where to save the full log in plain text format.

In almost every situation is strongly recommended you accurately configure gathering parameters from the Options Tab of the Archive configuration panel.

Figure 8.44. Gathering limits configuration

The first parameter is called HTTP User Agent and it is the string that the web crawler presents to the HTTP server to describe itself. This string should be descriptive enough to let web master of the remote site to recognize the owner of the spider. Someone prefers to use the same User Agent of a popular browser like Firefox or Internet Explorer in order to simulate that the spidering has been done by a real human user.

The second parameter is called Throttling and represents the minimum number of seconds that must pass between two fetching action performed for the current Archive. Good web citizens behave nicely with other sites and limit the load they put on remote servers and searchbox self-limits its page download speed to a user configured setting with a default value of 20 seconds. In the case you are gathering from a file system or from a partner site you can decrease this limit up to 0 seconds.

The three other parameters (by default they are ignored) limit:

The Gathering process is "by definition" a cycling activity. The gathering agent periodically visits the Source and put into the Archive only new documents or new versions of existing ones.

Some different options are possible from the Option Tab of the Archive configuration panel:

Figure 8.45. Gathering periodicity: hourly

The hours option configures a gathering cycle of n hours (8 hours in the above example)

Figure 8.46. Gathering periodicity: daily

The days option configures a gathering cycle of n days starting from a specific hour. In the above example the gathering starts at midnight of every day.

Figure 8.47. Gathering periodicity: weekly

The week option configures a weekly gathering cycle starting from at specific hour of a specific day of the week. In the above example the gathering starts every week on Monday at midnight.

Figure 8.48. Gathering periodicity: monthly

The month option configures a monthly gathering cycle starting from a specific hour of a specific day of the month.

From the Option Tab of the Archive configuration panel is possible to configure how searchbox throw out documents from Archives.

The mechanism is called Garbage Collecting and it is an automated process that periodically remove documents that satisfy some limits from the Archive.

Figure 8.49. The Garbage Collector configuration

Three parameters can be configured:

The Control Panel lets you browsing the content of an Archive using many parameters. In the following picture the result of a simple query is shown.

Figure 8.50. Browsing Archive content

The text box at the top of the windows accepts the user query. The syntax is the standard one of searchbox (See “The query syntax” section).

The next row of widget specify how query results must be shown:

Results: are the number of results shown after pressing Search button. To show more query results click more times on the Search button or increase this value.

Sort: the ranking algorithm used to sort the query results. Possible choices are:

Set date... Let you set a time interval where documents of query results must be contained. The time is referred to the date of gathering.

Weights... Modify the weight of each slice of the index in order to better control the ranking mechanism (See section about rendering) In the following table all available slices are described with their default value.

The rank of a document is calculated following the rule:

The Rank of a document is the absolute weight of the document multiplied for weights of those slices that contains at least a term specified in the query string.

Figure 8.51. Query Weights

Values can be changed with the following conventions:

Selecting one of the query results in the Context box a short extract of the document related to the query terms is shown.

Figure 8.52. The Context Box

From the Browsing Tab of Archive configuration is possible to show any document listed as result of a query in three possible ways:

Figure 8.53. Show document buttons

In order to access to any cached document you need to give to the viewer the authentication for searchbox (you username/password). Usually operating systems let you to store usernama/password in a keychain so that typing such information are needed the first time only.

Figure 8.54. Authentication for searchbox cache

The FFF is the XML format that searchbox internally use to store all documents extracted content and metadata. Viewing the FFF is useful for debugging purposes.

Please not the address of the page shown in the web browser (in the case of an html document)

Figure 8.55. Cached document address

It is an URL that points to the platform where searchbox Engine is installed (localhost in this case).

Three different views of the FFF are available.

Text FFF view

Figure 8.56. Text view of FFF

The full-text extracted from the FFF of the selected document. This text is distributed on different slices by the rendering process so that selecting and deselecting the appropriate checkboxes on the left side of the windows is possible to see what is contained in each slice. By default all slices are shown.

Metadata FFF view

Figure 8.57. Metadata view of FFF

Both standard and template metadata containted into the FFF representation of the selected document.

Raw FFF view

Figure 8.58. Raw view of FFF

The raw XML format of FFF internal representation of the selected document.

In the Browsing Tab of the Archive the Add... and Remove buttons are used to manually add and remove documents from the current Archive.

Figure 8.59. Add and Remove buttons

The following windows appears clicking on the Add... button

Figure 8.60. Manual add parameters

File: the complete path of the document that must be added to the Archive. It is automatically filled browsing the local file system by the ... button.

URL: the URL assigned to the document. If it is not specified the path of the file in the local file system is taken.

MIME: the MIMETYPE assigned to the document. It can be one of the following types:

Selecting the custom option the textbox under MIME filed will be ready to accept the custom string.

The Copy ID button in the Browsing Tab of Archive configuration panel let you copy into clipboard the ID of the document currently selected in the list of results.

Figure 8.61. Copy ID button