DFRWS 2011

I spent this last week in New Orleans at the Digital Forensic Research Workshop. DFRWS is primarily a  research workshop for academics, to which they can submit formal papers for acceptance by peer review and then eventual publication. Folks whose papers are accepted then give presentations about their work.

Highlights

• Vassil Roussev presented on the work he's done on fuzzy hashing, comparing his sdhash tool to Jesse Kornblum's venerable ssdeep in an information retrieval-type experiment. In general, sdhash does a significantly better job of identifying similar files, with the main trade-off being that hash values are not fixed-length (I haven't played with sdhash enough to know whether performance is a significant factor, either). Fuzzy hashing is a great example of applying computer science theory to forensics. More of this, please. (pdf)
• Robert Beverly presented on work at the Naval Postgraduate School in finding artifacts of common structures used by programmers in networking code. It turns out that these ephemeral structures in RAM can often be found on disk (pagefile/hiberfil/unallocated), allowing the astute investigator to construct a partial remembrance of networks past. (pdf)
• Michael Cohen of PyFlag fame presented on GRR. GRR stands for "GRR Rapid Response", but you can probably substitute "Google" for the G if you'd like to avoid the left-recursion. GRR is an open source tool for collecting and analyzing data from remote machines on an enterprise network, created for internal investigations at Google. It is the closest thing I've seen to my old friend ESAC, and built on a far-richer software stack. It's early days, but I'll be following development of GRR very closely and I recommend that you do as well (it's much cheaper to buy Cory a beer than to buy your overly-friendly enterprise software salesperson a new Ferrari). The one aspect of GRR I'm leery of is the use of AFF's RDF-based data model, because I think RDF is a bit abstruse for most problems in forensics (or, at least, doesn't have the pay-off you'd like in return for learning RDF, a technology only semioticians could love). (pdf)
• Clay Shields presented on some work he's done using, essentially, document indexing for supporting enterprise investigations. By creating bag-of-word style feature vectors for documents on hosts, he can perform offline queries to help discover machines involved in an investigation. Indexing and the bag-of-words do have a host of problems that require careful consideration, but this would no doubt be a useful technique for many investigations. I'd love to see such functionality put into GRR. (pdf)
• Christopher King and Tim Vidas from Carnegie Mellon presented on some aspects of Solid State Disks. This is a paper everyone should read carefully once it's been posted. (pdf)
• James Okolica and Gilbert Peterson figured out how to find and extract data from the Windows clipboard in RAM images. (pdf)
• Judson Powers from ATC-NY gave lightning talks both on MacOS X Lion's new full disk "encryption" (the quotes are needed, my fellow Mac weenie brethren) and Dropbox artifacts. 
• Last but certainly not least, Ralf Brown talked about his ZipRec tool, which is able to find and recover data from corrupt DEFLATE streams (i.e., from zip files and many other compressed formats). It'll be very interesting to see how this tool develops and what other topics in forensics Ralf turns to. (pdf)

Format

Not being an academic, I find these sorts of conferences kinda' weird. Their primary purpose is to provide a forum for academic researchers to present their work. I don't have skin in that game, and there were, frankly, some pretty boring moments for me... and, in the case of the legal panel, some pretty boring moments for everyone (a panel of class action plaintiffs attorneys, replete with hair product and suits, for an audience of bits-and-bytes forensics researchers is an odd pairing; the caliber of the audience  demands the likes of Craig Ball, Ralph Losey, Jason Baron, or other folks capable of engaging with the best forensics researchers; throwing a prosecutor into the mix would be good, too).

While my ADD-addled brain would like a more-compressed, faster-paced format (can someone organize a conference consisting of forensics lightning talks? that'd be great), DFRWS is the place to be to learn about the future of forensics. Most folks there are working on substantive problems and I had some great conversations with various researchers throughout the conference. I definitely need to start planning now so I can submit a paper or two for next year and participate a bit more actively. DEFCON and Black Hat may get the press, and DFRWS probably missed out on a few practitioners in attendance because of the scheduling conflict, but I think DFRWS is the best mash up of computer science and digital forensics I've yet seen.

The Fail Whale

Finally, an expedition was organized to The Beach, a bar on Bourbon Street that has a mechanical killer whale (instead of a mechanical bull). Life's not all about hex, code, and catching bad guys; it's also about taming wild beasts.

Notes on raw FileVault access, for live system forensics

Yesterday, I stupidly/accidentally deleted a source file that I hadn't yet checked in. Unfortunately, I was unable to recover the data, but I did learn how to go about doing so, given that the file was stored in my home directory, protected by FileVault.

The first thing to know is that FileVault is essentially an encrypted DMG file. The DMG is also "sparse," which means that it looks really big to the computer (to get around being limited to a fixed-size), but only stores blocks which haven't been allocated.

This sparse DMG image essentially looks like a physical disk. It's then formatted with a partitioning scheme, and the primary partition is attached to the filesystem at your home path, e.g. /Users/jon, not unlike how a normal /home partition would be mounted under Linux.

The cool thing is that the encryption/decryption happens at the physical device layer, not at the partition layer. So, if you happen to lose a file in OS X in a FileVault-protected directory, here's what you should do:

1. Open the terminal

2. Run "hdiutil info". This will spit out information about any mounted DMGs you have open:

framework       : 283
driver          : 10.6v283
================================================
image-path      : /Users/.jon/jon.sparsebundle
image-alias     : /Users/.jon/jon.sparsebundle
shadow-path     :
icon-path       : /System/Library/PrivateFrameworks/DiskImages.framework/Resources/CDiskImage.icns
image-type      : sparse bundle disk image
system-image    : TRUE
blockcount      : 1258291200
blocksize       : 512
writeable       : TRUE
autodiskmount   : TRUE
removable       : TRUE
image-encrypted : TRUE
mounting user   : root
mounting mode   : -rwx------
process ID      : 646
/dev/disk1    Apple_partition_scheme   
/dev/disk1s1    Apple_partition_map   
/dev/disk1s2    Apple_HFS    /Users/jon

You can see here that /dev/disk1 is the "physical" device associated with the DMG, and that the second partition, /dev/disk1s2 is formatted Apple_HFS and mounted at /Users/jon. Make a note of the path of this partition. It's a valid HFS+ partition, by the way; running the sleuthkit on it works just fine.

3. cd to somewhere that's outside of the mount point, e.g., /Users.

4. Run "sudo hdiutil unmount -force ", e.g., "hdiutil unmount -force /Users/jon". This unmounts the filesystem in the DMG, but still leaves the DMG open.

5. Now you can do whatever you'd like with /dev/disk1s2, or /dev/disk1. You can dd it, grep it, run recovery tools against it, whatever. You should be warned, however, that because it's a sparse image, it will be appear much bigger than what it is. But it will be decrypted.

Nota Bene: All of this information is based on my own experimentation yesterday. Please let me know if I've made any mistakes.

2nd Fundamental Law of EnScript

The Second Fundamental Law of EnScript is more complicated than the First or the Third:

All simple objects are ref-counted, but only root NodeClass objects are ref-counted. NodeClass objects in a list or tree are not ref-counted.

If you want to keep from crashing EnCase with your scripts, you must thoroughly understand this law. Unfortunately, it's a bit tricky.

What does ref-counted mean?

Like most languages in use these days, EnScript features garbage collection. When you create new objects in EnScript, those objects take up space in RAM. Rather than making you, the programmer, tell EnScript when that memory should be released, EnScript safely decides for you. It's really less like having your trash picked up once a week and much more like maid service.

An example of garbage collection in EnScript:

class MainClass {

  void Main(CaseClass c) {

    NameListClass strings = new NameListClass();

    strings.Parse("w h a t e v e r", " ");

    NameListClass strings2 = strings;

    Console.WriteLine(strings2.Count());

  }

}

In this example, you can see that I've allocated at least one NameListClass, and have a second reference. What you don't see is me calling delete, as you would if this were C++, or free, if this were C. Folks with Java, Python, or Perl experience should be used to this.

What happens is that EnScript gets to the end of the Main() function and realizes that the NameListClass object will never be used again. It's at that time that it is destroyed and the memory is released.

Reference counting is the underlying mechanism that makes this work. The key is to understand that the strings and strings2 variables in the above code are variables that refer to a NameListClass object; they don't contain the objects themselves. If you've programmed in Java or Python, this is normal. If you're coming from a C/C++ background, references are essentially pointers that prohibit you from doing fun pointer arithmetic or things like recasting 17 to an object. In the example above, the strings reference is created and assigned the memory location of the new NameListClass object. The Parse() method is called on the object, creating several child objects. The strings2 reference is then created and assigned the memory location within the strings reference; now both references refer to the same object.

Because objects can only be manipulated through references in EnScript, EnScript can track how many references point to each object. When a new object is created, a counter variable is created along with it, and when the object's memory location is assigned to a reference, the counter is incremented. When a reference is destroyed or has another object assigned to it, it decrements the counter of the object it had been pointing to. Here's the magic part: if the counter is zero after the decrement, the reference then says, whoa, I'm the last guy out of the room so I should turn out the lights... and it destroys the object.

Java uses a more advanced form of garbage collection called "mark-and-sweep" where a background thread runs periodically and tries to figure out which objects can be destroyed. Reference counting is used in Python and can also be used in C++ via the new shared_ptr<> template in the standard library (new in the TR1 update; use Boost's shared_ptr<> if you don't have it, since they're functionally equivalent). The upside to reference-counting, though, is that it's deterministic; you know that an object will be destroyed when the last reference goes away, which allows you considerable control over memory usage.

There are a couple of advantages to references and garbage collection. The first is that you don't have to worry about managing memory. The second is that you can't hose yourself with bad pointers the way you can in C or C++. A reference is either null (dereferencing it causes a runtime error, which will kill your script but won't crash EnCase) or it refers to a valid object. That's the theory, at least...

Great, what does this have to do with "root NodeClass objects"?

In our example above, we know that strings and strings2 both refer to the same NameListClass object, and at the end of the Main() function the reference count of the object should therefore be 2. It is. What about the child objects that were created by Parse()?

class MainClass {

  void Main(CaseClass c) {

    NameListClass strings = new NameListClass();

    strings.Parse("w h a t e v e r", " ");
    Console.WriteLine(strings.GetChild(2).Name());

    NameListClass strings2 = strings;

    Console.WriteLine(strings2.Count());

  }

}

Well, as it turns out, child objects in a NodeClass are not ref-counted. They're contained by the root NodeClass object. When the root is destroyed, so, too, are the children.

Okay... why aren't all objects reference counted?

Because. The main reason is that most everything you see in EnCase is derived from NodeClass (as pointed out in the First Law, which we'll revisit in the Third Law). Reference-counting has some overhead and, like most Windows apps, EnCase is written in C++ and can manage memory manually. So, internally, EnCase doesn't use reference counting for all those child objects, since that would be inefficient when working with millions of objects. Rather than make NodeClass work completely differently for EnScript objects than it does for EnCase's internal data, the semantics are the same.

Why do I need to keep this in mind as a Fundamental Law of EnScript?

Because this can hose you. You can write a script that crashes EnCase, killing not just your script, but any work it's done and any unsaved work in the case.

Consider this script:

class MainClass {

  typedef NameListClass[] ListArray;

  void Main(CaseClass c) {
    NameListClass found; // null reference;

    NameListClass strings = new NameListClass();

    strings.Parse("w h a t e v e r", " ");

    found = strings.Find("e"); // found points to child 5
    Console.WriteLine(found.Name());
    strings.Close();           // destroy all children


    // this code causes a lot more memory to be used
    ListArray array();
    for (unsigned int i = 0; i < 1000; ++i) {
      array.add(new NameListClass());
      array[i].Parse("g a r b a g e", " ");
    }


    if (found != null) { // totally true
      found.SetName("buffer overflow"); // can crash
      Console.WriteLine(found.Name());
    }

  }

}

Now, if you run this script... it may work; it just might print out "buffer overflow" to the console without any problems. But it's not guaranteed to work.

The problem is that found is set to refer to the fifth child of strings, and then we delete all the child objects. found, though, still thinks it's pointing to the fifth object... but it's really pointing to the memory formerly occupied by that object, and now that object could be used for something else. And since we ran through some code in the meantime that created 1,000 new lists, each with child objects, chances are high that that memory is now being used for something else.

The term of art for what happens when we call SetName() on found is "undefined behavior." We can't be sure.

If you're running a recent version of EnCase, what will probably happen is that the script will terminate and EnCase will report an error, but keep on running. If you're running an older version of EnCase, it'll probably crash and burn. Newer versions of v6 ask Windows to monitor for crashes that occur in EnScript and give EnCase a chance to recover from them. This is similar to the crash protection in the gallery view.

Even with the crash protection, though, you can still never be sure that you didn't hose everything. That's why you need to be careful when manipulating references to child NodeClass objects, and you need to understand how EnScript's memory model works.

3 Laws of EnScript

The other day I stumbled across an old presentation (uploaded by Mark Morgan) that I'd created a few years ago. The presentation was my guide through a 1-3 day workshop/seminar that I led a few times for some clients, with time taken throughout for exploring example scripts and working in teams.

In the presentation, I make reference—with tongue firmly in my cheek—to "Stewart's 3 Fundamental Laws of EnScript." The three laws are:

1. Data structures should almost always be composed [sic] from NodeClass.
2. All simple objects are ref-counted, but only root NodeClass objects are ref-counted. NodeClass objects in a list or tree are not ref-counted. (link)
3. Most of the EnScript classes are auto-generated by handlers from the EnCase view. WYSIWYG. (link)

I don't really explain these three laws in the presentation. Let me rectify that situation. Today, I'll cover the first law.

First Fundamental Law of EnScript
Data structures should almost always be derived from NodeClass


What this means:

class FooClass: NodeClass { // use NodeClass as parent of FooClass

  uint AnInt;

  FooClass(NodeClass parent = null):

    NodeClass(parent),

    AnInt = 7

  {}

}

The important bits are in red. This defines our own class, FooClass, declares that it inherits from NodeClass, and defines a constructor for FooClass (you are, of course, free to define other constructors). The constructor takes a reference to another NodeClass object (parent), and calls the NodeClass constructor for the object in the constructor initializer list, passing in the parent reference.

What is NodeClass and why should our own classes inherit from it?

I'm glad you asked. NodeClass is the quintessence of EnCase, the manifestation of the central metaphor: everything in EnCase is a tree. (If you haven't noticed that before, look around; everything in EnCase is a tree.) To use some software development language from the GoF book, NodeClass represents the Composite Design Pattern.

NodeClass has a lot of methods, so let's focus on the important ones:

class NodeClass {

  NodeClass(parent = null); // Constructor

  NodeClass FirstChild();

  NodeClass LastChild();

  NodeClass Next();

  NodeClass Parent();

}

What this says is that a NodeClass object can access the first item in a child list, the last item in a child list, the next item in its own list, and its parent. A picture is helpful:

This picture shows an item in green, and how it relates to other NodeClass items in the same tree. Each NodeClass object has just enough data to form a tree, if we link them together. For example, let's consider things from the perspective of "First Child" in the picture above. We'll rename it "Item", rename the old "Item" to be "Parent", blank out the other labels, and make the old links dotted lines.

Essentially, NodeClass allows you to construct lists of lists. You can see in the second image that we can have another child list, and because of the Next and Parent links, it can be linked into the overall tree.

Almost all of the API classes in EnScript inherit from NodeClass. EntryClass, BookmarkClass, NameListClass, FileSignatureClass, etc.

So, why should my classes inherit from NodeClass?

For a few reasons:

1. foreach() and forall() loops work with NodeClass. If you have multiple objects of your class, you can form them into a list using NodeClass (by passing in the parent to the constructor, or by calling Insert() on the parent) and then iterate over the loop using foreach(). You can be extra-fancy and create trees, too, and then iterate over every item in the tree using forall().
2. Some user interface widgets only work with NodeClass. For example, ListEditClass, TreeEditClass, and TreeTableEditClass can only display objects which inherit from NodeClass.
3. Properties only work with NodeClass. EnScript supports a limited form of reflection in a manner, somewhat similar to Python or Java. Given an object, foo, you can use the typeof() keyword to ask for a SymbolClass object that describes the class of foo. You can then ask the SymbolClass for all properties of foo, and PropertyClass::GetProperty() and PropertyClass::SetProperty() allow you to get and set, respectively, the underlying property values of foo. That is, you can access the data indirectly, without knowing what the class is or calling the property methods directly. This is great for writing your own object-relational mapper (ORM) like Ruby on Rails' ActiveRecord, auto-serializing data out to xml, and implementing a limited form of duck-typing.

This obviously isn't a complete introduction to EnScript, or even NodeClass, but I hope it clarifies the First Law. I'll post about the Second Law in the next few days.

Link files run malware

Brian Krebs has news of a new vulnerability in Windows 7, where link files on a thumb drive can execute arbitrary code when encountered by Windows Explorer. The link files themselves do not need to be clicked on by the user.

AND the malware they describe seems to target SCADA systems. I used to think that stego was a non-issue and that no one in the history of computer forensics ever came across legitimate use of it... but then there were the Russians. Similarly, I used to think that all the hype about securing SCADA systems from malware was a non-issue...

Forensics and Forgery

A fascinating article in The New Yorker about Peter Paul Biro, an art dealer in Montreal who uses fingerprint analysis to authenticate disputed works of art...

...or does he? Be sure to read the whole thing.

Gratz to 'gesh

I am unsurprised to see that my old partner Yogesh provided a correct answer to the latest SANS network forensics contest. Say what you will, but between Lance, Yogesh, Geoff Black, Jamie Levy, and many others, there was once an impressive collection of talent at a particular forensics company.

BOD: Bytes On Disk

My introduction to computer forensics was of the sink-or-swim variety, a 4 day training class followed by a rough tasking of "hey, kid, here are some EnScripts, make 'em better."

It didn't take me long, though, to figure out one of the primary rules of computer forensics. It is thus:

Always refer to the bytes on disk.

I'll admit I've never had to image a disk for a real case, and I never could get interested in imaging and its arcana. I've seen enough of it to know that it's harder than it sounds, often due to Murphy's Law, and there's no substitute for experience when you encounter an old drive on an old system. But, to me, my rule above feels more fundamental.

I am all for push-button forensics. Rote tasks should be automated, and expertise should be modeled in code. The results of automation are all for naught, though, if they do not refer to the relevant bytes on disk. It's a variation of citing your sources in a paper; if you can't refer back to the bytes on disk from which your results stem, it's awfully hard to verify them.

The Sausage Factory illustrated this rule rather neatly today. There was a simple bug in the script that reads results from C4P back into EnCase. It was easy to resolve, though, because the results referred back to the relevant bytes on disk. It's to Trevor's credit that he includes this information in the results. If he didn't, the fix wouldn't be so simple.

Always, always, always refer back to the Bytes On Disk.

LegalTech thoughts

LegalTech was overwhelming (NB: I only hung out in the exhibitor halls). People said that the number of vendors exhibiting was down from last year, which itself was down from the year before, but it's hard to imagine it being any larger.

My primary impression is that there isn't a lot of differentiation among the eDiscovery software vendors. With a few exceptions, they are all the same... at least, their booths are all the same. Everyone can take data and let you cull it in some fashion or another. Most then allow you to review. Some have collection capabilities. Everyone loves the EDRM.

Since the legal market is so fragmented and since the eDiscovery process is so well-defined (if you've lived it, at least), I think it's relatively easy for small software companies to gain a foothold. They can call up local firm offices and make some wins. When they all get together in the same place, though, they're indistinguishable. If I were a CIO, I'd have a very hard time telling them apart...

...which means I'd probably just call up Autonomy. Sure, if I looked hard, I might find better software than Autonomy, but the odds are I'll have to wade through a lot of bad software. Plus, Autonomy is sucking all the air out of the market at the top end. Even if a firm has a better mousetrap, they may kill themselves by mismanaging growth or their product line, or Autonomy may just buy them. I might as well buy Autonomy, run some keywords, ship the results out to counsel, and call it a day. Autonomy is too big to fail.

Or, instead of putting up with enterprise software, I can call my friendly neighborhood service provider. Even just a couple of years ago, this would be asking for per GB bilking, at $900/GB or more. If there's one nice thing about the great recession and the move to take eDiscovery "inhouse," it's that extreme pricing pressure has now been applied to eDiscovery service vendors. Frankly, I see this being a good thing: there's enough data and litigation volume that good consultants can still be profitable, and companies won't make dramatically expensive mistakes when they hire bad consultants.

The service providers do have an easier road to travel. They know they're competing on service. They can adopt new tools and approaches. They'll never grow too fast. They can drop bad clients.

In the few short demos I saw, I was still disappointed in user interfaces. Almost everyone has a web interface, but these still involve a tremendous amount of navigation, clicking, and mouse travel to get things done. With the price premiums no longer such easy pickins', I think this gives the present advantage to service providers. Sure, it's good to have some kind of content management system with a search engine, but prepping, processing, filtering, and reviewing data is still too complex to be handled correctly in-house by inexpert staff.

Another impression I came back with is that everyone can say Early Case Assessment and no one can do it. I mean, yeah, it's great that you can give me some bar graphs and pie charts, but if that's all there is, what's the fuss about? Good consultants have been able to provide the same data in spreadsheets for a long time, and good counsel have been asking for the same for a long time, too.

If I go next year, it'll be interesting to see which software vendors also return. I'm betting some won't.

JPEGs

1 out of 3 papers on computer forensics research seems to be about advanced techniques for carving fragmented jpegs from unallocated space.

I am sure that there have been cases where that one hard-to-reassemble jpeg was the key to cracking the case and sending the bad guy to jail for life. However, I feel equally sure that in the vast majority of cases, the problem is not how to recover that one screwy jpeg, but what to do with the hundreds of thousands of jpegs recovered with a simple header/footer search.

Trevor's work on C4P has probably done more for investigators than all the work done on fragmented jpeg carving combined. Of course, research is research, and there is nothing wrong with finding better ways to recover fragmented jpegs. There is something right, though, about applying research to solve pressing problems.

Path overflow

Speaking of linkedin, I came across this article today about the evils of MAX_PATH in eDiscovery:

http://rewcomputing.dyndns.org/REWBlog.nsf/d6plinks/RWEL-7ZVLQA

This article is spot-on for its audience. Practitioners honestly struggle with this in the eDiscovery industry. It'd be funny if it weren't so pathetic. In a day, a good programmer could write a tool to copy files around correctly and deal with long paths and timestamps, and then live off shareware sales for a year.