Lightgrep 1.0

I've just released Lightgrep for EnCase 1.0. Lightgrep is a new multipattern regular expression search engine that I've been developing with my colleagues over the last two years, and our EnCase integration makes it the fastest way to conduct searches over digital evidence.

I'm hoping I'll be writing more blogposts now, as I allow my mind to wander a bit more.

Lightgrep for EnCase at CEIC 2012

I'll be at CEIC 2012 next week, arriving Monday and leaving mid-morning on Thursday. Geoff Black and I will be talking again about the use of statistical sampling in eDiscovery matters, based on a matter we worked on in which the Hon. Andrew J. Peck was the presiding judge. Our talk is at 4:30pm on Monday.

We are also putting the finishing touches on Lightgrep for EnCase and we'd love to show it to folks attending CEIC. Lightgrep is a new regular expression search engine we wrote, that searches evidence for many keywords very quickly. We've created an EnScript wrapper around the low-level search engine that allows you to use it instead of EnCase's own keyword search facility. It will bookmark search hits and create an Excel report with details about the search, in addition to several other features.

Lightgrep shines when you have many keywords or need more advanced grep functionality than EnCase provides. Last year we had some investigators use an early version on a case with over a million keywords, so we think it sets a new standard for scalability. Lightgrep also supports more grep features, and we try to make them work as close to Perl as possible. We don't have all of Perl's functionality just yet, but we'll keep adding more grep operators over time (and we do a lot of testing to make sure we get the same hits Perl does).

Unicode support is what we've been working on most recently, and I'm pretty excited about it. Out of the gate, we'll have support for ASCII (really, Windows code page 1252), UTF-8, UTF-16LE (two-byte Unicode on Windows), UTF-16BE, and UTF-32LE and BE. We are able to look for every character in the Unicode standard, including brand new ones in Unicode version 6.1.0, like the infamous U+1F4A9. We won't support other code pages just yet, but the code is mostly developed and we simply need our test suites to catch up.

The cool thing about Lightgrep's Unicode support is that you can use Unicode properties in patterns. You can specify \p{Digit} or \p{Letter} and look for only valid digits or letters, but in any language. You can also specify different scripts, which is amazingly useful when looking for text that's not in the Latin alphabet. For example, \p{Cyrillic}+ and \p{Arabic}+ will find all words in Cyrillic (i.e., Russian) and Arabic, respectively.

If you'd like to see Lightgrep in action at CEIC, just email me, at jon@lightboxtechnologies.com, and we'll find a good time to chat. I'll have a few dozen thumb drives with a Lightgrep trial version installer, so don't be shy.

Differential EnScript

My former colleague Jamie Levy released a new EnScript today, for doing differential analysis of hard drives. You should check it out.

Don't call virtual functions in a destructor

Sometimes I like to define the outline of an operation in a base class and then have it delegate to virtual functions in derived classes. This is the Template Method design pattern.

class Base {
  uint Counter;


  void foo() {
    ++Counter;
    _foo();
  }


  pure void _foo();
}


class Derived {
  virtual void _foo() {
    Console.WriteLine("In Derived::_foo");
  }
}



Last night I tried to gin up a sort of "stack guard," something that would let me perform an operation once I got to a certain point in a function, but which also would be executed if the function returned early. When something needs to be done at the end of a block of code, I usually define a new class that has code in its destructor. For example, a lock class could take a critical section and acquire it in its constructor and release it in its destructor, and you just need to create the object in the appropriate place to perform your thread-safe operation. I wanted the same thing, but just a bit more complex, where I could call the release function manually once I'd reached a certain point and then be assured it would not be called in the destructor.

class Committer {
  bool Committed;


  ~Committer() {
    if (!Committed) {
      commit();
    }
  }


  void commit() {
    _commit();
  }


  pure void _commit();
}


class SafeDB: Committer {
  SafeDB(Database db): Committer() {
    // open a database connection
  }


  virtual void _commit() {
    // commit any pending transactions
  }
}



Simple enough. You can quibble about whether it's really worth it to define a class hierarchy for such simple functionality, but I've been trying hard to increase the amount of abstraction I use in EnScript to reduce code bloat and time wasted debugging.

The problem is that this doesn't work and generates run-time errors about null references. Why?

Answer: SafeDB's destructor is called first, and then Committer's destructor is called. As part of executing SafeDB's destructor, it nulls out whatever class members are a part of SafeDB before calling Committer's destructor. Hence, by the time you get to Committer's destructor, you really only have a Committer object, not a SafeDB object. The virtual table is still active, however, so the call to SafeDB::_commit() happens successfully, but it finds itself confronted with no class variables. Boom!

So, this is now another little corner of the EnScript world that's been explored. Time to light out for the territory...

ThreadClass::WAITSAFE

A New Year's Resolution, if you can call it that, is to blog more in the moment, as I'm figuring stuff out. I think I just figured out how the ThreadClass::WAITSAFE option works, so here goes...

ThreadClass::WAITSAFE is an enum value under ThreadClass::WaitOptions. It can be passed to several of the different Wait() methods related to the threading classes. The gloss on it says "Exits early if thread stops running," but there aren't any examples that show it in action and it seems to be the default parameter for some of the Wait() methods, but not for all of them. Specifically, it's not the default parameter for either of the ThreadClass::WaitClass methods.

To test it, we're going to have three threads. One is just going to spin its wheels—it's the thread we'd have doing real work, so we'll call it the "doer." Another thread will launch the doer and wait for it to finish, which we'll call the "waiter." This seems kind of silly, except that in a real application, the waiter could have a pool of doer threads, and could be coordinating their work and that may justify a separate thread of its own (especially if it's happening while you have a dialog up). Finally, we'll have our main thread that creates and launches the waiter, sleeps for a bit, then tells it to cancel. The waiter in turn will cancel the doer thread, and exit once the doer has exited. Here's the code:

class Doer: ThreadClass {
  Doer(): ThreadClass() {}

  virtual void Run() {
    while (IsRunning()) {
      Console.WriteLine("Doer sleeping...");
      SystemClass::Sleep(1000);
    }
    Console.WriteLine("Doer exiting.");
  }
}

class Waiter: ThreadClass {
  Waiter(): ThreadClass() {}

  virtual void Run() {
    Doer d();
    ThreadClass::WaitClass w();
    Console.WriteLine("Starting doer...");
    d.Start();
    w.AddObject(d);
    Console.WriteLine("Waiting on doer...");
    bool waitRet = w.WaitAll();
    Console.WriteLine("Woke up waiter");
    if (!IsRunning()) {
      Console.WriteLine("Waiter was cancelled. Stopping doer...");
      d.StopRunning();
      d.Wait();
    }
    Console.WriteLine("Waiter exiting, waitRet = " + waitRet);
  }
}

class MainClass {
  void Main() {
    SystemClass::ClearConsole();
    Waiter t();
    Console.WriteLine("Starting waiter...");
    t.Start();
    Console.WriteLine("Sleeping for 10 seconds");
    SystemClass::Sleep(10000);
    Console.WriteLine("Cancelling waiter...");
    t.StopRunning();
    Console.WriteLine("Waiting for waiter to quit...");
    t.Wait();
    Console.WriteLine("Waiter returned successfully. Done.");
  }
}

This code produces the following output:

Starting waiter...
Sleeping for 10 seconds
Starting doer...
Waiting on doer...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Cancelling waiter...
Waiting for waiter to quit...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
...

The problem is that this script never terminates successfully. The Waiter class uses WaitAll() to wait for the Doer thread to finish, but the Doer thread is just hanging out, enjoying its infinite loop until someone calls StopRunning() on it... which never happens. This is where ThreadClass::WAITSAFE comes in. Here's some modified code:

class Doer: ThreadClass {
  Doer(): ThreadClass() {}

  virtual void Run() {
    while (IsRunning()) {
      Console.WriteLine("Doer sleeping...");
      SystemClass::Sleep(1000);
    }
    Console.WriteLine("Doer exiting.");
  }
}

class Waiter: ThreadClass {
  Waiter(): ThreadClass() {}

  virtual void Run() {
    Doer d();
    ThreadClass::WaitClass w();
    Console.WriteLine("Starting doer...");
    d.Start();
    w.AddObject(d);
    Console.WriteLine("Waiting on doer...");
    bool waitRet = w.WaitAll(ThreadClass::INFINITE, ThreadClass::WAITSAFE);
    Console.WriteLine("Woke up waiter");
    if (!IsRunning()) {
      Console.WriteLine("Waiter was cancelled. Stopping doer...");
      d.StopRunning();
      d.Wait();
    }
    Console.WriteLine("Waiter exiting, waitRet = " + waitRet);
  }
}

class MainClass {
  void Main() {
    SystemClass::ClearConsole();
    Waiter t();
    Console.WriteLine("Starting waiter...");
    t.Start();
    Console.WriteLine("Sleeping for 10 seconds");
    SystemClass::Sleep(10000);
    Console.WriteLine("Cancelling waiter...");
    t.StopRunning();
    Console.WriteLine("Waiting for waiter to quit...");
    t.Wait();
    Console.WriteLine("Waiter returned successfully. Done.");
  }
}

This time, we pass ThreadClass::INFINITE (the default) and ThreadClass::WAITSAFE (not the default) as parameters to the WaitAll() method in the Waiter. This is what we get for output:

Starting waiter...
Sleeping for 10 seconds
Starting doer...
Waiting on doer...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Cancelling waiter...
Waiting for waiter to quit...
Doer sleeping...
Woke up waiter
Waiter was cancelled. Stopping doer...
Doer exiting.
Waiter exiting, waitRet = 0
Waiter returned successfully. Done.

Huzzah! Calling StopRunning() in the main thread on the waiter causes the waiter to wake up from WakeAll(), now that we're using ThreadClass::WAITSAFE. Once the Waiter wakes up, it checks IsRunning() to check whether it's been given the signal to quit, via StopRunning(), and then does likewise with the Doer thread. The Doer exits gracefully, then the Waiter exits gracefully, and then the whole script exists. Quite nice.

Notice that we've output the return value from WaitAll(), and it seems to have been false in this case. The question this raises is, would it return true if Doer exited before Waiter was cancelled with StopRunning()? Here's some modified code where the Doer exits after about 5 seconds, while the main thread still waits about 10 seconds to tell the Waiter to stop.

class Doer: ThreadClass {
  Doer(): ThreadClass() {}

  virtual void Run() {
    uint i;
    while (IsRunning() && i++ < 5) {
      Console.WriteLine("Doer sleeping...");
      SystemClass::Sleep(1000);
    }
    Console.WriteLine("Doer exiting.");
  }
}

class Waiter: ThreadClass {
  Waiter(): ThreadClass() {}

  virtual void Run() {
    Doer d();
    ThreadClass::WaitClass w();
    Console.WriteLine("Starting doer...");
    d.Start();
    w.AddObject(d);
    Console.WriteLine("Waiting on doer...");
    bool waitRet = w.WaitAll(ThreadClass::INFINITE, ThreadClass::WAITSAFE);
    Console.WriteLine("Woke up waiter");
    if (!IsRunning()) {
      Console.WriteLine("Waiter was cancelled. Stopping doer...");
      d.StopRunning();
      d.Wait();
    }
    Console.WriteLine("Waiter exiting, waitRet = " + waitRet);
  }
}

class MainClass {
  void Main() {
    SystemClass::ClearConsole();
    Waiter t();
    Console.WriteLine("Starting waiter...");
    t.Start();
    Console.WriteLine("Sleeping for 10 seconds");
    SystemClass::Sleep(10000);
    Console.WriteLine("Cancelling waiter...");
    t.StopRunning();
    Console.WriteLine("Waiting for waiter to quit...");
    t.Wait();
    Console.WriteLine("Waiter returned successfully. Done.");
  }
}

And here's the output:

Starting waiter...
Sleeping for 10 seconds
Starting doer...
Waiting on doer...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer sleeping...
Doer exiting.
Woke up waiter
Waiter exiting, waitRet = 1
Cancelling waiter...
Waiting for waiter to quit...
Waiter returned successfully. Done.

So, as you can see, WaitAll() still returns when the Doer thread exits, and this time it returns true. Waiter then exits from its Run() function before the main thread has the chance to cancel it. The upshot is that we can use the return value from WaitAll() to tell whether the thread waiting has had StopRunning() called on it (return value is false) or whether simply the operations it's waiting on have completed (return value is true).

Oh frabjous day, everything works!

SANS 360 Lightning Talk

I've published a write-up on the Lightbox blog of my SANS 360 talk, "Factory Forensics." You could watch the video, of course, but then you'd have to suffer through my stumbling and general awkwardness. The turnout was great, and I was humbled by the quality of the speakers.

ExecuteClass pitfall

Of the ways that EnScript can interact with other programs, using ExecuteClass to launch programs from the command-line is probably my favorite. It's simple to use, simple to debug, and doesn't leave you with an unholy mess of code, like COM. However, I recently ran into a pitfall with ExecuteClass and hopefully sharing it here will keep others from running into it or, at least, recognizing it when they're at their wits' end.

The problem? ExecuteClass becomes angry when the target program writes a lot of data to stdout or stderr, and you wouldn't like ExecuteClass when it's angry.

ExecuteClass currently has the Output property, a String, for giving you back stdout output from the application. My somewhat shaky belief is that this is updated once the application has ended—I haven't tested this in a while, though. Additionally, notice that there's no property representing stderr.

If the target application you are running generates lots of output to stdout or to stderr, you will find that the application will hang after a while. Why? Well, in the words of the illustrious Beej, "On many systems, pipes will fill up after you write about 10K to them without reading anything out." Once the target application fills up whatever pipe EnCase has connected to stdout or stderr, it will block on the next write to the stream... and EnCase will never consume from the pipe, causing the target application to hang indefinitely.

One possible workaround to this is not to call your target application, but to call cmd.exe instead, with the argument being the command you want to run, redirecting output to a file. I haven't yet tested this, but it should work. The other, of course, is to change the target application to write output to a file itself. As for the underlying win32 calls involved, I found an MSDN article with a sample program demonstrating how child processes are created and how to read data from stdout and stderr.