Implementing a "Dynamite" Search in Ignition Go (and also CodeIgniter 3)

In this article, we’ll look at how easy it is to set up your own custom full text search. Often we need to search against large text documents where the database LIKE operator simply isn’t enough. It cannot handle cases where you don’t enter the exact search query, in the right order and spelled right. Databases like MySQL have full text search support, but functionality can be cumbersome and limited.

The Goal of this Tutorial

This beginner/intermediate level tutorial will walk you through getting started incorporating of a fast pure PHP search module in your web application. We will assume you've started your development using a framework called Ignition-Go which is Codeigniter-based. We will also cover using vanilla Codeigniter 3.x, since the process is very similar, although there will be some additional setup you'll need to do on your own in the base CodeIgniter. We're going to use TNTSearch, which is a fully featured full text search engine written in PHP with MIT license (demo here ).

For a nice demonstration, we will use guzzle, a http client,and create a search for the help files for phpunit, a unit testing tool. Oddly, phpunit documentation is lacking a search, so this could be handy.

What You'll Need Before We Start
We'll need to have:

1. Ignition-Go fully installed
   (or CodeIgniter 3 with composer support added)
2. the ability to run "composer" at the command line

Set up
First, we will set up the composer package for TNTsearch. Navigate to the folder where your site is installed in the command prompt. Then at the command line type

composer require teamtnt/tntsearch

If you're not using Ignition Go, you will also need to install Guzzle.

Create a controller for admin functions
For real world use, we would generate a "search" module, but for this tutorial we will just use two controllers.

Create a new controller, application/controllers/adminsearch.php. We want to create some actions that only the admin would access, so we create this controller.

 <?php defined('BASEPATH') || exit('No direct script access allowed');
 // access TNTSearch namespace
 use TeamTNT\TNTSearch\TNTSearch;
 
 class Adminsearch extends Admin_Controller
    {
        public function __construct()
        {
            parent::__construct();
        }

        //--------------------------------------------------------------------
        public function index()
        {
            // TODO: set up a screen for this or call setup method
        }

        //--------------------------------------------------------------------

    }
    /* End Adminsearch controller */

Notice that the class is named the same as the file and it extends Admin_Controller. For vanilla CodeIgniter, you will also want to create the base Admin_Controller.

A note on Composer
Make sure CodeIgniter is set to autoload composer (applications/config/config.php)
Look for this line and set it to TRUE if needed.

$config['composer_autoload'] = TRUE;

First get some text content and set up the Indexing
The way that the search engine works is by building an internal index of the text content. This step only needs to run when any of your text changes.

Let's create a method called "setup" to build/rebuild the search index.

But first, we'll make a private method that will "scrape" the phpunit.de help files from the site using Guzzle. Add this to your new controller:

private function scrapePHPUnitHelp()
{
    $client = new GuzzleHttp\Client();
    $crawler = $client->request('GET', 'https://phpunit.de/manual/current/en/index.html');
    $toc = $crawler->filter('.toc');
    file_put_contents(base_path('resources/docs/').'index.html', $toc->html());

    $crawler->filter('.toc > dt a')->each(function($node) use ($client) {
        $href = $node->attr('href');
        $this->info("Scraped: " . $href);
        $crawler = $client->request('GET', $href);
        $chapter = $crawler->filter('.col-md-8 .chapter, .col-md-8 .appendix')->html();
        file_put_contents(base_path('resources/docs/').$href, $chapter);
    });
}

Now, add our setup method:

public function setup()
{
    $this->scrapePHPUnitHelp();

    $tnt = new TNTSearch;

    $config = [
        "storage"   => storage_path(),
        "driver"    => 'filesystem',
        "location"  => base_path('resources/docs/'),
        "extension" => "html",
        "exclude"   => ['index.html']
    ];

    $tnt->loadConfig($config);
    $indexer = $tnt->createIndex('docs');
    $indexer->run();
}

Controller / Search api
Now, we create a publicly accessible controller called "Search" with public methods index (the search screen) and find (the ajax api).

<?php defined('BASEPATH') || exit('No direct script access allowed');
// access TNTSearch namespace
 use TeamTNT\TNTSearch\TNTSearch;

/**
 *
 * Search Controller
 *
 * Provides front-end functions for users to search.
 *
 */
class Search extends Front_Controller
{
    protected $tnt;

    public function find()
    {
        $this->output->set_content_type('application/json');
    
        $this->tnt = new TNTSearch;
        $this->tnt->loadConfig([
        "storage"   => storage_path(),
        "driver"    => 'filesystem'
        ]);

        $this->tnt->selectIndex("docs");
        $this->tnt->asYouType = true;

        $results = $this->tnt->search($this->input->get('query'), $this->input->get('params')['hitsPerPage']);

        return $this->processResults($results);
    }

    private function processResults($res)
    {
        $data = ['hits' => [], 'nbHits' => count($res)];

        foreach ($res as $result) {
            $file = file_get_contents($result['path']);
            $crawler = new Crawler;
            $crawler->addHtmlContent($file);
            $title = $crawler->filter('h1')->text();

            $relevant = $this->tnt->snippet($this->input->get('query'), strip_tags($file));

            $data['hits'][] = [
            'link' => base_url($result['path']),
            '_highlightResult' => [
                'h1' => [
                    'value' => $this->tnt->highlight($title, $this->input->get('query')),
                ],
                'content' => [
                    'value' => $this->tnt->highlight($relevant, $this->input->get('query')),
                ]
            ]
            ];
        }
        return json_encode($data);
    }
        

    //--------------------------------------------------------------------
    function index()
    {
        Template::render();
        // for vanilla CI substitute: $this->load->view('search/index');
    }
}
/* end ./application/controllers/Search.php */

The Search View to try it out
Note: For the view, we are assuming you have some master theme which includes bootstrap 3, otherwise you'd need to add that.

Now we create a view file: application/views/Search/index.php

<div class="container">
  <div class="row">
        <div class="col-md-6">
    		<h2>Dynamite Search</h2>
            <div id="custom-search-input">
                <div class="input-group col-md-12">
                    <input type="text" class="form-control input-lg" placeholder="Search " />
                    <span class="input-group-btn">
                        <button class="btn btn-info btn-lg" type="button">
                            <i class="glyphicon glyphicon-search"></i>
                        </button>
                    </span>
                </div>
            </div>
        </div>
  </div>
</div>
<script src="https://cdnjs.cloudflare.com/ajax/libs/typeahead.js/0.9.3/typeahead.min.js"></script>
<script>
    $(document).ready(function () {
        $('#custom-search-input input').typeahead({
            source: function (query, result) {
                $.ajax({
                    url: "/search/find",
          data: 'query=' + query,            
                    dataType: "json",
                    type: "POST",
                    success: function (data) {
            result($.map(data, function (item) {
              return item;
                        }));
                    }
                });
            }
        });
    });
</script>

Now give it a go!