[Summary view] [Print] [Text view]

   1  2006/09/08
   2  ----------
   3  Google Summer of Code is finished, spent a couple of weeks away from
   4  the project to think about it and also to take a break. Working on it
   5  now I discovered bugs in the query parser (now fixed), and I also
   6  un-convoluted the querylib logic (well slighlty).
   7  
   8  Updated ZFS files to latest SVN.
   9  
  10  2006/08/21
  11  ----------
  12  Fixed index document count, and created new config variable to store
  13  the size. (Search now has 3 global vars in $CFG, date, size and complete,
  14  see indexer.php for var names). Index size is cached to provide an always
  15  current value for the index - this is to take into account the fact that
  16  deleted documents are in fact not removed from the index, but instead just
  17  marked as deleted and not returned in search results. The actual document
  18  still features in the index, and skews sizes. When the index optimiser is
  19  completed in ZFS, then these deleted documents will be pruned, thus
  20  correctly modifying the index size.
  21  
  22  Additional commenting added.
  23  
  24  Query page logic very slightly modified to clean up GET string a bit (removed
  25  'p' variable).
  26  
  27  Add/delete functions added to other document types.
  28  
  29  A few TODO fields added to source, indicating changes still to come (or at
  30  least to be considered).
  31  
  32  2006/08/16
  33  ----------
  34  Add/delete/update cron functions finished - can be called seperately
  35  or all at once via cron.php.
  36  
  37  Document date field added to index and database summary.
  38  
  39  Some index db functionality abstracted out to indexlib.php - can
  40  use IndexDBControl class to add/del documents from database, and
  41  to make sure the db table is functioning.
  42  
  43  DB sql files changed to add some extra fields.
  44  
  45  Default 'simple' query modified to search title and author, as well
  46  as contents of document, to provide better results for users.
  47  
  48  2006/08/14
  49  ----------
  50  First revision of the advanced search page completed. Functional,
  51  but needs a date search field still.
  52  
  53  2006/08/02
  54  ----------
  55  Added resource search type, and the ability to specify custom 'virtual'
  56  models to search - allowing for non-module specific information to be
  57  indexed. Specify the extra search types to use in lib.php.
  58  
  59  2006/07/28
  60  ----------
  61  Added delete logic to documents; the moodle database log is checked
  62  and any found delete events are used to remove the referenced documents
  63  from the database table and search index.
  64  
  65  Added database table name constant to lib.php, must change files using
  66  the static table name.
  67  
  68  Changed documents to use 'docid' instead of 'id' to reference the moodle
  69  instance id, since Zend Search adds it's own internal 'id' field. Noticed
  70  this whilst working on deletions.
  71  
  72  Added some additional fields to the permissions checking method, must still
  73  implement it though.
  74  
  75  2006/07/25
  76  ----------
  77  Query logic moved into the SearchQuery class in querylib.php. Should be able
  78  to include this file in any page and run a query against the index (PHP 5
  79  checks must be added to those pages then, though).
  80  
  81  Index info can be retrieved using IndexInfo class in indexlib.php.
  82  
  83  Abstracted some stuff away, to reduce rendundancy and decrease the
  84  likelihood of errors. Improved the stats.php page to include some
  85  diagnostics for adminstrators.
  86  
  87  delete.php skeleton created for removing deleted documents from the
  88  index. cron.php will contain the logic for running delete.php,
  89  update.php and eventually add.php.
  90  
  91  2006/07/11
  92  ----------
  93  (Warning: It took me 1900 seconds to index the forum, go make coffee
  94  whilst you wait.) [Moodle.org forum data]
  95  
  96  Forum search functions changed to use 'get_recordset' instead of
  97  'get_records', for speed reasons. This provides a significant improvement,
  98  but indexing is still slow - getting data from the database and Zend's
  99  tokenising _seem_ to be the prime suspects at the moment.
 100  
 101  /search/tests/ added - index.php can be used to see which modules are
 102  ready to be included in the search index, and it informs you of any
 103  errors - should be a prerequisite for indexing.
 104  
 105  Search result pagination added to query.php, will default to 20 until
 106  an admin page for the search module is written.
 107  
 108  2006/07/07
 109  ----------
 110  Search-enabling functions moved out've the mod's lib.php files and into
 111  /search/documents/mod_document.php - this requires the search module to
 112  operate without requiring modification of lib files.
 113  
 114  SearchDocument base class improved, and the way module documents extend
 115  it. A custom-data field has been added to allow modules to add any custom
 116  data they wish to be stored in the index - this field is serialised into
 117  the index as a binary field.
 118  
 119  Database field 'type' renamed to 'doctype' to match the renaming in the
 120  index, 'type' seems to be a reserved word in Lucene. Several index field
 121  names change to be more descriptive (cid -> course_id). URLs are now
 122  stored in the index, and don't have to be generated on the fly during
 123  display of query results.
 124  
 125  2006/07/05
 126  ------
 127  Started cleaning and standardising things.
 128  
 129  cvs v1.1
 130  --------
 131  This is the initial release (prototype) of Moodle's new search module -
 132  so basically watch out for sharp edges.
 133  
 134  The structure has not been finalised, but this is what is working at the
 135  moment, when I start looking at other content to index, it will most likely
 136  change. I don't recommend trying to make your own content modules indexable,
 137  at least not until the whole flow is finalised. I will be implementing the
 138  functions needed to index all of the default content modules on Moodle, so
 139  expect that around mid-August.
 140  
 141  Wiki pages were my goal for this release, they can be indexed and searched,
 142  but not updated or deleted at this stage (was waiting for ZF 0.14 actually).
 143  
 144  I need to check the PostgreSQL sql file, I don't have a PG7 install lying
 145  around to test on, so the script is untested.
 146  
 147  To index for the first time, login as an admin user and browse to /search/index.php
 148  or /search/stats.php - there will be a message and a link telling you to go index.
 149  
 150  -- Michael Champanis (mchampan)
 151     email: cynnical@gmail.com
 152     skype: mchampan
 153     Summer of Code 2006