Pro-search reference manual
===========================

Free open-source file search engine
-----------------------------------

Copyright  2003-2007 Oleg Alexeenkov, <proler(at)gmail.com>

------------------------------------------------------------------------------------------------------------------------



   1. 
      1.1. 

   2. 
      2.1. 
      2.2. 
         2.2.1. perl
         2.2.2. mysql
         2.2.3. PostgreSQL
         2.2.4. apache
         2.2.5. mod_perl
         2.2.6. nmap
         2.2.7. samba
         2.2.8. rsync
         2.2.9. bzip2
         2.2.10. freebsd
         2.2.11. windows
         2.2.12. cygwin
      2.3. 
         2.3.1.   
         2.3.2.  
         2.3.3. 
         2.3.4.  
         2.3.5.  
         2.3.6.  
         2.3.7. 
         2.3.8. sql
         2.3.9. 
         2.3.10. dc
         2.3.11. web
         2.3.12. advanced

   3.  
      3.1.   
      3.2. ..
      3.3. 
      3.4.  

   4. api
   5. FAQ
   6. BUGS
   7.  

1. 
=============

   1.1. 

1.1. 
----------------

         .
---------------------------------------------------------

   *    Perl, web    mod_perl  FastCGI
   *    - windows, freebsd, linux, ...
   *   mysql  PostgreSQL  sqlite
   *         ( )
   *   ftp, windows shares, rsync, http[ ,   ?]
   *    ftp (ls-lR,    )
   *   ftp 
   *    *.m3u, files.bbs, descript.ion, ...
   *  DC ,  
   *  - DC++ (    magnet, dchub:// )
   * proxy -    ftp,file,dchub  web 
   *      (HTML, M3U, RSS, XML,  , ...)
   *     (, , flash, )
   *       (  )
   *       (online)
   *        -
   *    / 
   *  -  []
   *   
   *    [ ]
   *   
   *    
   *    
   *        (stemming)
   *  ,   

  :

   *          100  

homepage: http://pro.setun.net/search/ sf.net : http://sourceforge.net/projects/pro-search

 : http://search.setun.net/ http://proisk.ru/

 :  , ,  .    ,  
,  .

      %  .

    ,     ,      -    
8)

    WMZ419798751917  WMR783677313448

2. 
============

   2.1. 
   2.2. 
      2.2.1. perl
      2.2.2. mysql
      2.2.3. PostgreSQL
      2.2.4. apache
      2.2.5. mod_perl
      2.2.6. nmap
      2.2.7. samba
      2.2.8. rsync
      2.2.9. bzip2
      2.2.10. freebsd
      2.2.11. windows
      2.2.12. cygwin

   2.3. 
      2.3.1.   
      2.3.2.  
      2.3.3. 
      2.3.4.  
      2.3.5.  
      2.3.6.  
      2.3.7. 
      2.3.8. sql
      2.3.9. 
      2.3.10. dc
      2.3.11. web
      2.3.12. advanced

2.1. 
------------

release: http://pro.setun.net/search/download/?C=M;O=D mirror on http://sf.net:
http://sourceforge.net/project/showfiles.php?group_id=149797 svn snapshot:

   | svn co svn://svn.setun.net/search/trunk/ search

trac: http://svn.setun.net/search/trac.cgi/browser/trunk-> Download in other formats: Zip Archive  
     ./lib  ./web http://svn.setun.net/dcppp/trac.cgi/browser/trunk->
Download in other formats: Zip Archive  dcppp     ( crawler.pl)  
./web

2.2. 
---------

   2.2.1. perl
   2.2.2. mysql
   2.2.3. PostgreSQL
   2.2.4. apache
   2.2.5. mod_perl
   2.2.6. nmap
   2.2.7. samba
   2.2.8. rsync
   2.2.9. bzip2
   2.2.10. freebsd
   2.2.11. windows
   2.2.12. cygwin

     

Web

   * apache or any http server with cgi
   * perl>=5.6 (DBI; DBD::mysql)
   * mysql>=3  PostgreSQL  sqlite
   * [mod_perl]

Crawler

   * perl>=5.6 (DBI; DBD::mysql; Net::FTP; LWP; URI;)
   * mysql>=3  PostgreSQL  sqlite
   * nmap (with winpcap for win32)
   * samba (unix)
   * bzip2 (   DC++)
   * [rsync]

2.2.1. perl
-----------

!   perl - 5.8.8  .    .    
  -    our->use vars, open(x,y,z)->open(x,y.z)

       .

    perl  *nix: perl -MCPAN -e "install DBI, Bundle::DBD::mysql, Net::FTP, LWP, URI,
FCGI"  perl -MCPAN -e shell > [force] install DBI, Bundle::DBD::mysql, Net::FTP, LWP, URI, FCGI

 windows: http://www.activestate.com/Products/Download/Download.plex?id=ActivePerl  activeperl 
c:\usr\ (Run) ppm-shell.bat  ppm.bat   C:\usr\bin      :
install DBI install DBD-mysql install FCGI ActivePerl 

2.2.2. mysql
------------

http://dev.mysql.com/downloads/mysql/5.0.html http://dev.mysql.com/downloads/mysql/5.1.html freebsd: mysql 5.0 confiure
for russian language: sh configure --prefix=/usr/local/mysql --with-charset=cp1251 --with-extra-charsets=complex
--with-collation=cp1251_general_ci mysql: to set '_' as not word symbol mysql-x.y.z/myisam/ft[_]defs.h :

   |         - #define true_word_char(s,X)   (my_isalnum(s,X) || (X)=='_')
   | + #define true_word_char(s,X)   (my_isalnum(s,X))

mysql-x.y.z/# make all install my.cnf  my.ini:

   |         
   |   [mysqld]
   |   ft_min_word_len = 1
   |   ft_stopword_file=''
   |   #default-character-set = cp1251
   |   #init_connect          = "set names 'cp1251';"
   |   #TESTING# max_seeks_for_key    = 1000
   |   #TESTING# max_write_lock_count = 100
   |   #TESTING# myisam_max_sort_file_size = 15G
   |   #TESTING# myisam_sort_buffer_size = 256M

   mysql    .    512  :

   |         
   | key_buffer=256M
   | sort_buffer=8M
   | table_cache=256
   | tmp_table_size = 64M

  key_buffer    filebase.MYI    .

2.2.3. PostgreSQL
-----------------

     PostgreSQL.

  windows:      8.2   .   
.   - russia, russia  - utf8   (     
$config{'sql'}{'pass'} )    tsearch2     tsearch2 -    
$config{'sql'}{'use_fulltext'} = 0;

  freebsd: cd /usr/ports/databases/postgresql83-server/ && make install cd
/usr/ports/databases/postgresql-contrib/ && make install echo postgresql_enable="YES" > /etc/rc.conf
/usr/local/etc/rc.d/postgresql initdb /usr/local/etc/rc.d/postgresql start createdb -U pgsql prosearch psql -U pgsql -d
prosearch -f /usr/local/share/postgresql/contrib/tsearch2.sql   OS: c  PostgreSQL 
 tsearch2 8-)

2.2.4. apache
-------------

http://httpd.apache.org/download.cgi httpd.conf:

   |             
   |     AllowOverride All
   |     AddHandler cgi-script .cgi

   [  httpd.conf /usr/local/etc/apache22/Includes/search.conf]:

   |         
   | Alias /search "/usr/local/www/search/web"
   | <Directory "/usr/local/www/search/web">
   |   AllowOverride All
   |   Order allow,deny
   |   Allow from all
   |   AddHandler cgi-script .cgi
   |   DirectoryIndex index.cgi
   |   options +ExecCGI
   | </Directory>

    tools/search.conf    /usr/local/etc/apache22/Includes/search.conf
  httpd.conf

      -  htdocs    .htaccess:

   | RedirectMatch permanent ^/$ http://your_site/search/

2.2.5. mod_perl
---------------

    -  web/.htaccess.dist web/.htaccess  web/.htaccess

2.2.6. nmap
-----------

http://www.insecure.org/nmap/nmap_download.html WIN32:dont forget:

   | $config{'nmap'} = 'c:/progra....../nmapwin/nmap';

   : regedit c:\progra......\nmapwin\nmap_performance.reg

windows: http://winpcap.polito.it/install/default.htm

2.2.7. samba
------------

  
smb.conf:

   |         
   |  [global]
   |  unix charset    = koi8-r
   |  dos charset     = cp866
   |  display charset = koi8-r

2.2.8. rsync
------------

    rsync://  -   rsync 8-)

2.2.9. bzip2
------------

    DC++.  windows    .  
http://unxutils.sourceforge.net/ http://cygwin.com/

2.2.10. freebsd
---------------

 freebsd      .

     1:    (   - portsnap fetch
extract,     : portsnap fetch update), cd /usr/ports/net/prosearch && make install clean
 2:     tools/prosearch: cd tools/prosearch && make install clean

  :

             
             
             cd/usr/ports/databases/mysql50-server&&makeinstallcleanBUILD_OPTIMIZED=yesBUILD_STATIC=yesWITH_CHARSET=cp1251WITH_COLLATION=cp1251_general_ci
 
             cd/usr/ports/databases/p5-DBD-mysql&&makeinstallclean
             cd/usr/ports/www/p5-libwww&&makeinstallclean
             cd/usr/ports/security/nmap&&makeinstallclean
             cd/usr/ports/net/samba3&&makeinstallclean
             cd/usr/ports/net/rsync&&makeinstallclean
             

           

advanced: monster base (>10M files) and monster server (>1G memory) freebsd5,6 i386: /boot/loader.conf:

   |         
   | kern.dfldsiz="2G"                # Set the initial data size limit
   | kern.maxdsiz="2G"                # Set the max data size

2.2.11. windows
---------------

 windows     apache+perl+mysql+...   ,    -  
   .     -     .

http://en.wikipedia.org/wiki/Comparison_of_WAMPs

http://www.apachefriends.org/en/xampp-windows.html

http://www.denwer.ru: http://www.denwer.ru/dis/Base/+ http://www.denwer.ru/packages/perl.html

http://topserver.ru

http://www.devside.net/server/webdeveloper

http://swissknife.bougakov.com

http://pampa.phpsoft.it

    nmap

2.2.12. cygwin
--------------

       cygwin .

             
             
             win:cygwin:TESTTODOUNFINISHED
             
             DESCRIBE:http://cygwinports.dotsrc.org/
             
             ONLYWEBINTERFACE
             
             
             gotohttp://cygwin.comdownloadandrunsetup.exe,install:perl,apache,gcc,make
             downloadmysqlsource,unpack,run
             cdmysql-5.1.11-beta&&configure&&makeinstallclean
             installwindowsversionnmapwithwinpcap
             installperlmodules:run:
             perl-MCPAN-e"installDBI,Bundle::DBD::mysql,Net::FTP,LWP,URI,FCGI"
             ifBundle::DBD::mysqlreporterrorrun:
             perl-MCPAN-eshell
             cpan>forceinstallBundle::DBD::mysql
             
             #linksearch/web/toapachedir:
             #ln-s/cygdrive/c/pro/search/web/var/www/htdocs/search
             confapache:
             puttoendof/etc/apache/httpd.conf:
             Alias/search"/cygdrive/c/pro/search/web"
             <Directory"/cygdrive/c/pro/search/web">
             AllowOverrideAll
             Orderallow,deny
             Allowfromall
             AddHandlercgi-script.cgi
             DirectoryIndexindex.cgi
             options+ExecCGI
             </Directory>
             runapache:
             /usr/sbin/apachectlrestart
             confsearch:
             
             cp/cygdrive/c/pro/search/web/config.pl.dist/cygdrive/c/pro/search/web/config.pl
             mcedit/cygdrive/c/pro/search/web/config.pl
             changelineto:
             require'/cygdrive/c/pro/search/confdef.pl';
             
             cp/cygdrive/c/pro/search/config.pl.dist/cygdrive/c/pro/search/config.pl
             mcedit/cygdrive/c/pro/search/config.pl
             changelineto:
             require'/cygdrive/c/pro/search/confdef.pl';
             
             mysqlconf:
             

           

2.3. 
--------------

   2.3.1.   
   2.3.2.  
   2.3.3. 
   2.3.4.  
   2.3.5.  
   2.3.6.  
   2.3.7. 
   2.3.8. sql
   2.3.9. 
   2.3.10. dc
   2.3.11. web
   2.3.12. advanced

2.3.1.   
------------------------

 ,   :

             
             
             copy|cpconfig.pl.distconfig.pl
             copy|cpweb/.htaccess.distweb/.htaccess
             editconfig.pl,web/.htaccess
             >mysqladmincreateprosearch
             >perlcrawler.plinstall--noscanfile=range
             

           

   range    /    3  : perl crawler.pl proc=3

2.3.2.  
-----------------------

2.3.3. 
-----------

    (): 1.    : unix: /usr/local/www/search win: c:\search

2. web :     alias   httpd.conf       web
htdocs: unix:   : ln -s /usr/local/www/search/web /usr/local/apache2/htdocs/search  :
cp /usr/local/www/search/web /usr/local/apache2/htdocs/search

  : touch /usr/local/apache2/htdocs/search/count chmod 666 touch
/usr/local/apache2/htdocs/search/count

win:  copy c:\search\web c:\apache2\htdocs\search    http:///search

  : echo.>> c:\apache2\htdocs\search\count

  web  cgi-bin,        *.css, *.js, img/*.png

2.3.4.  
-------------------------

confdef.pl-    ,  .    .

   index.cgi crawler.pl  

config.pl-  ,  .       config.pl.dist 
     . confdef.pl,   - , 
  -  (gopher 8)

      config.pl.dist config.pl

config.pl.dist-       

   crawler.pl index.cgi   config.pl,   (  ,   
  do '';).

 
       .    .

    confdef.pl       config.pl

    -  -    .

2.3.5.  
------------------

    prosearch (   confdef.pl)

 
 1: mysqladmin create prosearch  2:      mysql (    PATH)
:

   | $config{'mysql'} = 'C:/mysql/bin/mysql.exe';

  :

 
perl crawler.pl install     ( crawler  web        )
 tools/grant.sql mysql > tools/grant.sql    8-)

2.3.6.  
------------------

2.3.7. 
------------

 : crawler.log

banned    $config{'file_banned'} -   ,    
:

   |           
   | ftp://10.10.110.34
   | ftp://10.0.0.252
   | file://10.0.0.252
   | 12.34.45.56

   $config{'banned'}:

   | $config{'banned'}{'ftp://1.2.3.4'} = 1;

   banned   .

         / -    
.     web 

   |           
   | $config{'action'}{'ftp34'} = {'file://10.10.110.34/public' => 'ftp://10.10.110.34/pub'};
   | $config{'lng'}{'ru'}{'ftp34'}  = 'ftp';

     .

2.3.8. sql
----------

2.3.9. 
-----------

   2.3.9.1. 
   2.3.9.2. dns
   2.3.9.3. 
   2.3.9.4. ipdig
   2.3.9.5.  
   2.3.9.6. 
   2.3.9.7. purge
   2.3.9.8. 

        ( samba(share) (file://))    config.pl
crawler.pl:

   | $processor{'prot'}{'file'}{'disabled'} = 1;

2.3.9.1. 

2.3.9.2. dns

2.3.9.3. 

      /    :  ,  range
      

   |             
   | 10.13.32.0/27
   | 10.13.110.0/24
   | ftp://192.168.50.10
   | ftp://10.10.201.201

      nmap ( 172.16.0,2,6,7.0-254)    (file://.. ,
ftp://.. , ...)     perl crawler.pl file=range --noscan      
     perl crawler.pl --noscan 10.13.32.0/27 10.13.110.0/24 ftp://192.168.50.10

2.3.9.4. ipdig

      , , ,  8) :    
(   ,  ftp, file, http): perl crawler.pl --noscan ipdig=http://search.zzz.ru/ftp_list.php

 :  10.20.1.4 ... ftp://10.20.5.7 ..... 10.20.8.100     : 10.20.1.0/24
10.20.5.0/24 10.20.8.0/24 config.pl:  

    ftp: ftp://10.20.1.4 ftp://10.20.5.7 ftp://10.20.8.100 config.pl:

   |             
   |   $config{'dig_prot_add'} = 'ftp';
   |   $config{'dig_netmask'} = '';

     ftp://* ,  ip,  -  - -   $config{'dig_find_mask'}

-   ipdigf=file.txt ,   file.txt       /

2.3.9.5.  

!       .      .   
   - cd /usr/local/www/search && perl crawler.pl params...

===    crawler.pl:

crawler.pl [  ] [  ] [  ]

-        : install , , .  
 . proc[=NUM] NUM   (  1) drop[=table] [].
purge     delhost=HOST    HOST. deldead    
 resource upload[=NUM]   NUM  var/ index_disable   
filebase index_enable,        repair[=table]mysql   
    mir    myisamchk flushmysql analyzemysql optimizemysql mystatusmysql show
processlist   mysql.???????.dump hublist[=url] dc   $config{'hublist'} mrtg  
 mrtg stat  upgrade     help  

: [ip]10.1.2.3 [ip_range](nmap style) 192.168.0.0/16 '192.88-90.*.*' [resource]ftp://10.2.3.4 \\10.3.4.5
scan=[ip|resource|resource] . [file]list.txt file=filename        
  url http://.. ftp://.. file://.. skip=[ip]   banned $config{'banned'}

: [-]|[c[onf]]-KEY=VALUE  $config{KEY} = VALUE  --log_screen=1   
--xxx__yyy=zzz,  

   | $config{'xxx'}{'yyy'} = 'zzz';

   $config{..} --force []      
--ftp_recurse_force  ftp ( ls -R)(  vsftpd) --pingonly  , 
 (  online (     [|] )) --log_dbg 
  --log_all   --fast    ,     

      -     .  perl crawler.pl drop
install file=range --noscan 10.20.0.0/16 ftp://10.30.40.50

      "-"  perl crawler.pl drop -install -debug

2.3.9.6. 

      ( 3) (    ) perl crawler.pl
proc=3  -  , ,   .

  : perl crawler.pl proc=3 &

 *nix      search.sh.dist     /usr/local/etc/rc.d/search.sh
    .

       crawler    (  ).
 :

               
                             
                 perl crawler.pl 10.10.0-20.0-255
                 perl crawler.pl 10.10.1.1
                 perl crawler.pl \\10.10.1.10 ftp://10.10.1.11
                 perl crawler.pl file://10.10.1.11
                 perl crawler.pl \\10.10.1.10\video\film
                 perl crawler.pl file://10.10.1.111/music/
               

             

2.3.9.7. purge

         --... [     proc 
  ] perl crawler.pl purge

 ,         -   perl crawler.pl optimize

2.3.9.8. 

   crawler  ./var/   ,     

perl crawler.pl upload

2.3.10. dc
----------

   crawler.pl     proc.    - perl crawler.pl --noscan
dchub://hub.com perl crawler.pl --noscan file=dchubs perl crawler.pl --noscan hublist=dchubs.xml.bz2 perl crawler.pl
--noscan hublist=http://hubs.my.lan/hubs.xml.bz2

2.3.11. web
-----------

   2.3.11.1. 
   2.3.11.2. 

2.3.11.1. 

2.3.11.2. 

  .     8)     en.    
en   :

   | $config{'lng_default_auto'} = 'ru'; # if autodetect fail - use ru

 en    -    :

   | $config{'lng_default'} = 'ru'; # disable autodetect, use ru

(    )

2.3.12. advanced
----------------

   2.3.12.1. ftp login-pass
   2.3.12.2. 
   2.3.12.3.   mrtg
   2.3.12.4.    
   2.3.12.5.    proc   
   2.3.12.6.   utf 

2.3.12.1. ftp login-pass

   ftp  -   anonymous -   ftp://login:pass@10.20.30.40 -
     c   .    

   |             
   | $config{'fine'}{'10.20.30.40'}{'anonymous_user'} = 'xxx';
   | $config{'fine'}{'10.20.30.40'}{'anonymous_pass'} = 'yyy';

   ftp://10.20.30.40

2.3.12.2. 

     -       :

   | for (0..210) { print "10.20.$_.0/24\n"; }

  , ,   alive:0 -     (       2
    -   )

      ( 1-3 ) -   : perl crawler.pl
index_disable   : perl crawler.pl index_enable    mysql  
   .

2.3.12.3.   mrtg

mrtg sample config: mrtg.conf

   |           
   | ...
   | Target[local.ips]: `cd /usr/local/www/search/ && perl crawler.pl mrtg --log_screen=0 --log_all=- --nolog_startstop --fast`
   | MaxBytes[local.ips]: 100000
   | Options[local.ips]: growright, gauge, noinfo, nopercent
   | WithPeak[local.ips]: ymw
   | Title[local.ips]: ip in local net
   | YLegend[local.ips]: ips now/total
   | LegendI[local.ips]: ips total
   | LegendO[local.ips]: ips alive
   | ShortLegend[local.ips]: ip
   | Legend1[local.ips]: - total ip's
   | Legend2[local.ips]: - alive ip's
   | Legend3[local.ips]: - total ip's max per 5 min
   | Legend4[local.ips]: - alive ip's max per 5 min
   | 
   | Target[local.res]: `cd /usr/local/www/search/ && perl crawler.pl mrtg=online --log_screen=0 --log_all=- --nolog_startstop --fast`
   | MaxBytes[local.res]: 100000000
   | Options[local.res]: growright, gauge, noinfo, nopercent
   | WithPeak[local.res]: ymw
   | Title[local.res]: online resource size
   | YLegend[local.res]: files/mbytes
   | LegendI[local.res]: files
   | LegendO[local.res]: Mbytes
   | ShortLegend[local.res]: .
   | Legend1[local.res]: - files
   | Legend2[local.res]: - mbytes
   | Legend3[local.res]: - files max per 5 min
   | Legend4[local.res]: - mbytes max per 5 min
   | ...

OR with $config{'mrtg_out'}: config.pl:

   | $config{'mrtg_out'} = '/usr/local/www/search/mrtg.out';

mrtg.cfg

   | Target[local.ips]: `cat /usr/local/www/search/mrtg.out`

for web:

   |           
   |   $config{'html_main_aft'} = '<table><tr><td>net alive:</td><td>shared:</td></tr><tr><td><a href="http://site/mrtg/local.ips.html">
   |    <img src="http://site/mrtg/local.ips-day.png"/></a></td><td><a href="http://site/mrtg/local.res.html">
   |    <img src="http://site/mrtg/local.res-day.png"/></a></td></tr></table><p/>';

2.3.12.4.    

alt-printscreen    photoshop ctrl-n enter ctrl-v zoom select ctrl-c ctrl-n enter ctrl-v
ctrl-alt-shift-s png-128 enter

2.3.12.5.    proc   

         .  $program{'upload'}{'proc'}- .
    .     -

 1:

   |             
   |  #   
   |  $program{'upload'}{'proc'} = 1 if $param->{'example'};



   |             
   |  #  
   |  if ($param->{'example'}) {
   |    $program{'upload'}{'proc'} = 1 ;
   |    $config{'log_dbg'} = 1;
   |  }

  perl crawler.pl proc --example

 2:

   |             
   |  $program{'example'}{'func'} = sub {
   |    $program{'upload'}{'proc'} = 1;
   |    $config{'log_dbg'} = 1;
   |    $program{'proc'}{'func'}->();
   |  }

  perl crawler.pl example

            8)

2.3.12.6.   utf 

               
               ...DebianLenny
               utf-8,:
               1.sambaŠ(-utf-8)
               2.$config{'cp_db'}='utf-8';
               3.config.pl(confdef)
               $config{'fine'}{'file'}{'cp_res'}='utf-8';
               $config{'cp_shell'}='utf-8';
               $config{'cp_log'}='utf-8';
               4.allow_cp_detect
                ...
               
               

             

3.  
===========================

   3.1.   
   3.2. ..
   3.3. 
   3.4.  

3.1.   
----------------------------

3.2. ..
---------------

3.3. 
---------------

3.4.  
-----------------------

4. api
======

5. FAQ
======

   :    
   :    
   :   vBadvanced (  vBulletin)   .        
   ...    html  php ....      ( iframe)? :)
   :  ,       
   :         (   )  
       .     SQL ,     
       ,    20       100.
   :      5.0
   :      !
   :     4 
   :   web-?
   :    ?
   :        ( )      1 2 3 4 5 6 7 8 9 !
     1  2 ...  
   :           -    ,     
    ...    57 43 ....
   :           48m 1d   48  1 ... 
        ...
   :     ?
   :       "firefox.1.6" ?        .
   :    ?
   :       (   ) ?  ,     -,
    ?
   :       . crawler.pl --pingonly ???
   :    crawler.pl purge        2   6 ?
   :      online- ?
   :         ( 40).     -   
   ??
   :      -?      crawler.pl proc=3 ? 
         (  windows    ).       ?
   :    294 -  .     . :  
    ,      ?  $config{'period'} (7 )??   
      ,   ?...
   : nmap    ?
   :           ?
   :   ,       -?
   :              
   :          ?
   :   DC        ?
   :  . 2.    crawler.pl   purge ?
   : 3.     crawler.pl proc=4 &          
   ?
   : 4.       crawler.pl   purge,     ? crawler.pl proc=4
   purge &
   : 5.       purge   "$config{'purge_time'}" ?
   :      
   :   $config{'scan_timeout'}  $config{'period'}?   confdef.pl   $config{'scan_timeout'} =
   $config{'period'} - (3600*20)        
   :         
   :     ,    
   :     ftp,       .   ?
   :      dc ,   ,      1 (2,
   20, 60...) Gb   .    - ?
   :   index.cgi  php
   :       : .................................

:    

:    8)         config.pl  CSS.

:    

:   ,  mysql           ~1000000 .

:   vBadvanced (  vBulletin)   .        
...    html  php ....      ( iframe)? :)

: 1.   php  index.cgi view=html ....       2.
    xml index.cgi view=xml ....    http://.../index.cgi?view=xml&...   
3. iframe 8)

:  ,       

:       (perl crawler.pl 10.10.10.10-20 10.10.20.10-20)    config.pl 
 (    1)

:         (   )  
    .     SQL ,     
    ,    20       100.

:    -          mysql -      
.     ranges  - 1.   perl crawler.pl install --sql_tranges=ranges1
--sql_tresource=resource1 perl crawler.pl install --sql_tranges=ranges2 --sql_tresource=resource2 perl crawler.pl
install --sql_tranges=ranges3 --sql_tresource=resource3 2.     diapazon1 diapazon2
diapazon3, perl crawler.pl --sql_tranges=ranges1 --sql_tresource=resource1 file=diapazon1 perl crawler.pl
--sql_tranges=ranges2 --sql_tresource=resource2 file=diapazon2 perl crawler.pl --sql_tranges=ranges3
--sql_tresource=resource3 file=diapazon3 3.      config.pl   $config{'sql_tranges'} =
'ranges1'; $config{'sql_tresource'} = 'resource1';  $config{'sql_tranges'} = 'ranges2'; $config{'sql_tresource'}
= 'resource2';  $config{'sql_tranges'} = 'ranges3'; $config{'sql_tresource'} = 'resource3';     
$config{'sql_host'} = 'db.net.ru.com'; $config{'sql_user'} = 'search'; $config{'sql_pass'} = 'cooolpaas';  
  top ..    todo 8)

:      5.0

:   8-)    crawler.pl  index.cgi     (  )    
open  open(I, '>>', $current_name_work)  open(I, '>>'. $current_name_work)   ^

:      !

: ,    - mysql       .    = 
 - .

:     4 

:   mysql  . my.cnf  my.ini: [mysqld] ft_min_word_len = 1  perl crawler.pl repair

:   web-?

:   . phpmyadmin  ;)

:    ?

:       $config{'scan_timeout'} (  20 ) ftp -    recurse 
 -    ./var/xxx.toscan   upload        dc -
      - 

:        ( )      1 2 3 4 5 6 7 8 9 ! 
 1  2 ...  

:    -      -    - ,   - 
.   ,             . 
     -  - E  [x]      &count_f=on

:           -    ,      
...    57 43 ....

: : ips -   ip   (  ranges)  -     
()  -     

:           48m 1d   48  1 ...  
    ...

:   -        (ping)  -    


:     ?

:    

:       "firefox.1.6" ?        .

:        -   mysql: ft_min_word_len = 1  perl crawler.pl repair 
    .     0.15.15    regexp 8-)

:    ?

: perl crawler.pl --force 

:       (   ) ?  ,     -,
 ?

:  .      .

:       . crawler.pl --pingonly ???

: .

:    crawler.pl purge        2   6 ?

:          mysql.  8)  .

:      online- ?

:     ?       ..

:         ( 40).     -   
??

:     C        .

:      -?      crawler.pl proc=3 ?  
     (  windows    ).       ?

:   ,     -  .       
 web. proc      -          
 8)

:    294 -  .     . :  
 ,      ?  $config{'period'} (7 )??   
   ,   ?...

:  $config{'scan_online_res'} = 1 -           .
 $config{'scan_online_res'} = 0 -   $config{'period'}.

: nmap    ?

: nmap     ip  -     -   . samba 
    -    ,     .

:           ?

: ,      ,   : $config{'web_add_mask'} = '.';

:   ,       -?

: $config{'default_param'}{'online'} = 0; $config{'force_param'}{'online'} = $config{'online_minutes'} if !defined
$param->{'online'};

:              

: $config{'table'}{$config{'sql_tfile'}}{'path'}{'fulltext'} = undef;  drop install

:          ?

: perl crawler.pl proc --once --pingonly --noscan_res --noscan_dcbot

:   DC        ?

: --dcbot_upload=0

:  . 2.    crawler.pl   purge ?

: 

: 3.     crawler.pl proc=4 &          
?

: ,  

: 4.       crawler.pl   purge,     ? crawler.pl proc=4
purge &

:  

: 5.       purge   "$config{'purge_time'}" ?

:     4   (     )

:      

: $config{'html_footer_top'} = q|    |;

:   $config{'scan_timeout'}  $config{'period'}?   confdef.pl   $config{'scan_timeout'} =
$config{'period'} - (3600*20)        

:  -    -      

:         

: $config{'allow_cp_detect'} = 0;

:     ,    

:   ie.

        .

      ie6        ie7    .

:     ftp,       .   ?

:   

   |             
   |           $config{'purge_time'} ||= 3600 * 24 * 7; # delete if not scanned last (seconds). good value is $config{'period'} * 2
   |           

  crawler.pl purge

:      dc ,   ,      1 (2,
20, 60...) Gb   .    - ?

:     DC ., config.pl.dist:

   |             
   | # $config{'fine'}{'dchub'}{'sharesize'} => '20g';      

:   index.cgi  php

: virtual("/search/index.cgi?".$_SERVER['QUERY_STRING']);

:       : .................................

:     10$  .

6. BUGS
=======

    .     .   .         
 .

   : 1.    -     2.  crawler -   
--log_all : perl crawler.pl proc --log_all  web -     &debug=on :
http://127.0.0.1/search/?q=xxx&debug=on 3.      ,     icq  , 
   2   , perl, mysql

         
         
         uname-a
         mysql--version
         perl-V
         

       

   6.1. nmap: Unable to determine what interface to route packets through to x.x.x.x
   6.2. net:ftp:  activestate perl    windows   timeout.      .
   6.3. crawler-samba: ' symol in description(m3u) file name samba dont get
   6.4. crawler-samba:  3.0.8 smbclient -d0 -N -c 'recurse;ls' "//10.10.10.10/pub"    
    ,   
   6.5. crawler-samba:      
   6.6. crawler- freebsd auto start from rc.d       smbclient  nmap.
   6.7. mysql4-utf8 codepage install: SQL: error[1071,Specified key was too long; max key length is 1000 bytes]
   6.8. perl perl -v: This is perl, v5.8.6 built for i386-linux-thread-multi uname -a : Linux leet 2.6.11-1.1369_FC4 #1
   Thu Jun 2 22:55:56 EDT 2005 i686 i686 i386 GNU/Linux   Undefined subroutine &main::full_host called at
   ./crawler.pl line 1225. Undefined subroutine &main::alarmed called at crawler.pl line 1939.
   6.9. apache error.log: [error] [client 192.168.137.41] Can't locate Time/HiRes.pm in @INC (@INC contains:
   6.10. win: activeperl: mysql.dll error after db connect on first command execution
   6.11.  
   6.12. perl 5.6.1: Can't set DBI::db=HASH(0x86cb80c)->{PrintWarn}: unrecognised attribute or invalid value at
   /usr/local/lib/perl5/site_perl/5.6.1/mach/DBI.pm line 625.
   6.13. crawler dcbot 0.17.1: bot kick:         
   ! Don't flood with GetNickList !
   6.14. 0.17.1 index.cgi: Bizarre copy of ARRAY in aassign at lib/psmisc.pm line 149
   6.15.      crawler.pl : Please create config.pl with parametrs (see config.pl.dist)
   and correct modes [No such file or directory]
   6.16. 0.17.2 dcbot:   

6.1. nmap: Unable to determine what interface to route packets through to x.x.x.x

-   x.x.x.x 8)

6.2. net:ftp:  activestate perl    windows   timeout.      .

   perl

6.3. crawler-samba: ' symol in description(m3u) file name samba dont get

none

6.4. crawler-samba:  3.0.8 smbclient -d0 -N -c 'recurse;ls' "//10.10.10.10/pub"    
 ,   

-     http://samba.org  freebsd:/usr/ports/net/samba3

6.5. crawler-samba:      

  -b1200  smbclient  :

   | $config{'smbclient'} = $config{'smbclient'} . ' -b1200 ';

6.6. crawler- freebsd auto start from rc.d       smbclient  nmap.

:

   | $config{'smbclient'} = '/usr/local/bin/' . $config{'smbclient'}; $config{'nmap'} = '/usr/local/bin/'
   |           . $config{'nmap'};

 /usr/local/bin/ -   .

6.7. mysql4-utf8 codepage install: SQL: error[1071,Specified key was too long; max key length is 1000 bytes]

   | $config{'sql_length_prot'} = 4; $config{'sql_length_name'} = 64; $config{'sql_length_path'} =
   |             128;

6.8. perl perl -v: This is perl, v5.8.6 built for i386-linux-thread-multi uname -a : Linux leet 2.6.11-1.1369_FC4 #1
Thu Jun 2 22:55:56 EDT 2005 i686 i686 i386 GNU/Linux   Undefined subroutine &main::full_host called at
./crawler.pl line 1225. Undefined subroutine &main::alarmed called at crawler.pl line 1939.

- perl  -     .  @INC :  crawler.pl  index.cgi: require 'lib/misc.pl'; 
 require '__/lib/misc.pl';

6.9. apache error.log: [error] [client 192.168.137.41] Can't locate Time/HiRes.pm in @INC (@INC contains:

  index.cgi: use Time::HiRes qw(time);   perl

6.10. win: activeperl: mysql.dll error after db connect on first command execution

      ppm  :
http://aspn.activestate.com/ASPN/docs/ActivePerl/5.8/faq/ActivePerl-faq2.html  :
http://theoryx5.uwinnipeg.ca/ppms/package.lst    DBD::mysql       
ActivePerl  .

6.11.  

web player compat: ftp http file opera,firefox img,swf all none ie all all all firefox file:// plugin:
https://addons.mozilla.org/firefox/281/ or http://kb.mozillazine.org/Security_Policies and enable links:
$config{'no_download_file_firefox'} = $config{'no_goto_file_firefox'} = 0; $processor{'prot'}{'file'}{'view'} = 1 if
$config{'client_firefox'};

6.12. perl 5.6.1: Can't set DBI::db=HASH(0x86cb80c)->{PrintWarn}: unrecognised attribute or invalid value at
/usr/local/lib/perl5/site_perl/5.6.1/mach/DBI.pm line 625.

$config{'sql_dbi_connect_params'} = { 'RaiseError' => 0, 'AutoCommit' => 1, 'PrintError'=>0, };

6.13. crawler dcbot 0.17.1: bot kick:          !
Don't flood with GetNickList !

$config{'dchub_param'}{'auto_GetNickList' } = 0;

6.14. 0.17.1 index.cgi: Bizarre copy of ARRAY in aassign at lib/psmisc.pm line 149

   

6.15.      crawler.pl : Please create config.pl with parametrs (see config.pl.dist) and
correct modes [No such file or directory]

    

6.16. 0.17.2 dcbot:   

   crawler.pl 2  $config{'no_dl'}    $config{'__no_dl'}

7.  
============================

         
         
         lan:
         http://pro.setun.net/searchsrc
         http://findexer.sourceforge.netsrc
         http://seek42.sourceforge.netsrc
         http://www.nixp.ru/shurup/codingsrc
         http://fsmbsearch.sourceforge.netsrc
         http://lostclus.linux.kiev.ua/chestnut-ftp-searchsrc
         http://ffsearch.sourceforge.netsrcinactive
         http://femfind.sourceforge.netsrcinactive
         http://wildhoney.netbynet.rusrc=http://gksus.nm.ruinactive
         http://www.strangesearch.netsrcinactive
         http://ftpsearchagent.sourceforge.netsrcinactive
         http://untzuntzlan.sourceforge.netsrcinactive
         http://turtle.ee.ncku.edu.tw/ftplocate/readme.english.htmlsrcinactive
         http://codewalkers.com/seecode/157.htmlsrcinactive
         http://mt.boston.ru/srcinactiveMocusFTPSearcher
         http://www.zaval.org/products/file-search/index.htmlsrcinactive
         http://ftp-search.web-notes.ru/ src
         http://code.google.com/p/filefind/ srchttp://www.ffind.ru=http://ftp.onet.ruoldscan
         
         http://qfs.quadra.ru
         http://search.qwerty.ru/
         http://isuas.nkosino.ru300$
         http://zebra.asearch.info/ru$
         http://ftpsearch.unilans.ru
         http://www.icn.bmstu.ru
         http://ftpsearch.ihome.net.ua
         http://wildhoney.netbynet.ru/svk/index.phphttp://locate.in.ua/
         http://www.yukh.ru/net/usearch
         http://portal.localka.ru/modules.php?op=modload&name=resources&file=index
         http://ftpsearch.deepweb.spb.ru/index.php?action=index
         http://ftps.rsspnet.ru/
         http://world.vesnet.ru/ftp/index.php
         http://www.2inet.ru/index.php?portal=filefind&st=filefind&search=
         http://homesearch.h15.ruwindowsborland
         http://tz.ints.ru/ftpsearch/
         http://ww.ints.ru/ftp/
         http://filesearch.nsu.ru/
         http://poisk.mosinter.net/
         http://www.crestyle.com.ua/poisk.html$win
         
         onesite:
         http://mirror.switch.ch:8000/ftpsearch
         http://forum.dklab.ru/perl/advises/DnsearchDenwerSearch.htmlsrc
         
         region:
         http://search.novgorod.ru/search.php?where=4src=http://sergey.weblab.ru/projects/ffindinactive
         http://www.filebox.kiev.ua
         
         inet:
         http://proisk.ru
         http://www.filewatcher.com
         http://www.mmnt.ru
         http://www.files.lt
         http://ftpsearch.rambler.ru/db/ftpsearch
         http://www.searchfile.ru
         http://findfile.net
         http://reliz.ru=http://www.e-mp3.ru=http://pubftp.com
         http://www.findmyfile.com
         http://www.fileindexer.com
         http://ftpsearch.elmundo.es[FTPsearch]
         http://search.ftphost.net/indexer/search.aspx
         http://www.filesearching.com=http://www.filesearch.ru
         http://www.freewareweb.com/ftpsearch.shtml
         http://oth2.net
         http://oth.net
         http://files.fifaleague.ru=http://ru.findfile.netmeta
         http://www.gegereka.com
         http://www.ftps.ru
         
         catalog:
         http://www.ftpsearchengines.com
         http://ligurio.blogspot.com/2007/02/blog-post.html
         
         

       

--eof--
