The current version can be downloaded from:
https://github.com/simsong/tcpflow/downloads

get clone git://github.com/simsong/AFFLIBv3.git

Key features of Version 1.2 include:

• Dramatically improved performance of the AES and IP packet scanning modules. (scan_aes runs in 15% the time of the original implementation.) As a result, scan_aes and scan_net are now enabled by default.
• The stop-list and context-sensitive stop-list processing has been rewritten:
  • Feature files can now be used as context-sensitive stop lists.
  • Feature files with different sized contxt windows can be freely intermixed as stop lists.
  • The program make_context_stop_list.py is no longer used.
  • Stop-list files that are not feature files may contain literals or regular expressions.
• In practice, this means that the -s option has been removed. You can use -w with a text file that is a list of words, a list of regular expressions, or a feature file. If it is a feature file, it should just work as a context-sensitive stop list. It turns out that it was easier to write it this way than to have different switches for the different kinds of stop lists and then to throw error messages when the wrong kind of list was given to the wrong option.
• The find (“-f”) option now searches for regular expressions, not globs.
• Dramatically improved defenses against compression bombs. Now bulk_extractor detects that it is decompressing a compression bomb and goes into a “safe decompress” mode in which new compressed regions are not decompressed if they have an MD5 that matches other compressed regions that have been decompressed. A notation is written into the zip.txt feature file that a compression bomb was encountered.
• scan_net now carves both IPv4 and IPv6 packets. As in Version 1.1, the resulting packets are put into PCAP files.
• A new -G option allows the page size to be specified.
• The pre-compiled Windows binary now runs faster than the Linux binary, although this is because it is not running scan_exif.
• Wordlist deduplication is significantly faster.

PERFORMANCE STATISTICS
Disk image: /corp/drives/nps/nps-2009-ubnist1/ubnist1.gen3.E01
            /corp/drives/nps/nps-2009-ubnist1/ubnist1.gen3.E02
            Media size:         1.9 GiB (2106589184 bytes)
            MD5:                49a775d8b109a469d9dd01dc92e0db9c
Hardware:   MacBook Pro 2 Ghz Intel Core i7, 8GB 1333 Mhz DDR3
OS:         MacOS 10.7.3
Compiler:   i686-apple-darwin11-llvm-g++-4.2 (GCC) 4.2.1
            (Based on Apple Inc. build 5658)
            (LLVM build 2336.1.00) 

bulk_extractor version 1.1.3:   468.6 seconds (4.28 MBytes/sec)
bulk_extractor version 1.2.0:   350.7 seconds (5.72 MBytes/sec)
Windows 7, same platform, scan_exiv disabled:
bulk_extractor.exe 1.2.0:       207.4 seconds (9.69 MBytes/sec) 

Current list of bulk_extractor scanners:
scan_accts   - Looks for phone numbers, credit card numbers, etc
scan_base64  - decodes BASE64 text
scan_kml     - Detects KML files
scan_gps     - Detects XML from Garmin GPS devices
scan_aes     - Detects in-memory AES keys from their key schedules
scan_json    - Detects JavaScript Object Notation files
scan_exif    - Detects EXIF structures from JPEGs
scan_zip     - Detects and decompresses ZIP files and zlib streams
scan_gzip    - Detects and decompresses GZIP files and gzip stream
scan_pdf     - Extracts text from some kinds of PDF files
scan_hiber   - Detects and decompresses Windows hibernation
               file fragments
scan_winprefetch
             - Detects and extracts fields from Windows
               prefetch files and file fragments.
Current list of bulk_extractor feature files:
aes_keys.txt - AES encryption keys
alerts.txt   - Features found on alert list (redlist)
ccn.txt      - credit card numbers
ccn_track2.txt - Track 2 information
domain.txt   - All extracted domain names and IP addresses
email.txt    - extracted email addresses
ether.txt    - extracted ethernet addresses. Currently
               overcollecting due to a failure to consider
               local context.
exif.txt     - All exif fields from JPEGs; extracted as XML.
find.txt     - Hits on find command.
gps.txt      - Extracted GPS coordinates from Garmin XML and
               GPS-enabled JPEG files
ip.txt       - Extracted IP addresses from scan_net
               cksum-bad indicates checksum test failed;
               those are less likely to actually be IP
               addresses.
json.txt     - Extracted and validated JavaScript Object
               Notation fragments.
kml.txt      - Extracted KML files
report.xml   - The DFXML file that explains what happened.
rfc822.txt   - All extracted RFC822 headers
tcp.txt      - Summaries of all extracted UDP and TCP packets.
telephone.txt- Extracted phone numbers
url.txt      - Extracted URLs
  url_facebook-id - extracted Facebook IDs
  url_microsoft-live - extracted Microsoft Live IDs
  url_searches       - extracted search terms
  url_services       - extracted services from URLs
winprefetch.txt - Windows prefetch files and fragments,
                  recoded as XML for easy processing.
wordlist.txt - All the words
zip.txt      - Information about all ZIP files and zip
               components.

Feature List for 1.3:

We are considering the following features for 1.3:

• Putting a BOM at the beginning of all feature files and forcing the coding of the features to UTF-8 (The context will still be reported as ASCII with octal escaping of values outside the printable range.)
• Replacing FTS with a new implementation for searching files.
• Replacing exiv2 with our own EXIF processor.
• Automatically detecting and reporting Window shortcut files and IE history.
• Scanning for the start of bitlocker protected volumes.
• Support for checkpointing using BLCR.
• Improved restarting, so that each page is retried once but only once. (Frankly, the improved reliability in verson 1.2 made this request less important.)
• Support on distributed computing arrays.

We are also considering the following scanners (and need
help!):

• LZMA decompression
• RAR & RAR2 decompression
• BZIP2 decompression
• MSI decompression
• CAB decompression
• NTFS decompression
• VCARD detection
• PE Header Detection
• Better handling of MIME encoding
• SQLite database identification
• Processing of physical drives
• Scanning for MD5 hash codes
• Scanning for word lists
• Python bridge, so scanners can be written in python

As always, bulk_extractor can be downloaded from http://afflib.org/

• AFFLIB 3.6.15
• fiwalk 0.6.16. (This is the last version of fiwalk that will include the DFXML tools; they will be moved into a new release as fiwalk is being folded into SleuthKit.)
• bulk_extractor version 1.1.1. (Version 1.0 had two minor bugs, one having to do with the identify_filenames.py script, and one having to do with histogram generation.)


$ src/bulk_extractor -o test5 0005.aff
Hostname: t
Input file: 0005.aff
Output directory: test5
Disk Size: 130351104
Threads: 8
Phase 1.
17:18:00 Page 0 (0.00%) Completed. Done in n/a
17:18:02 Page 1 (12.50%) Completed. Done in 0:00:46
17:18:04 Page 2 (25.00%) Completed. Done in 0:00:26
17:18:06 Page 3 (37.50%) Completed. Done in 0:00:18
17:18:08 Page 4 (50.00%) Completed. Done in 0:00:12
17:18:10 Page 5 (62.50%) Completed. Done in 0:00:09
17:18:13 Page 6 (75.00%) Completed. Done in 0:00:06
17:18:13 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...

... this shouldn't take more than an hour. Exiting ...
... Please report to the bulk_extractor maintainer ...
All Threads Finished!
Phase 2. Shutting down scanners
Phase 3. Creating Histograms
0: make_histogram(,histogram) -> test5/ccn_histogram.txt
0: make_histogram(,histogram) -> test5/ccn_track2_histogram.txt
0: make_histogram(,histogram) -> test5/domain_histogram.txt
0: make_histogram(,histogram) -> test5/email_histogram.txt
0: make_histogram(([^(]+),histogram) -> test5/ether_histogram.txt
0: make_histogram(([^(]+),histogram) -> test5/ip_histogram.txt
0: make_histogram(,histogram) -> test5/tcp_histogram.txt
0: make_histogram(,histogram) -> test5/telephone_histogram.txt
0: make_histogram(,histogram) -> test5/url_histogram.txt
0: make_histogram(://([^/]+),services) -> test5/url_services.txt
0: make_histogram(://(cid-[0-9a-f])+[a-z.].live.com/),microsoft-live) -> test5/url_microsoft-live.txt
0: make_histogram(://[-_a-z0-9.]+facebook.com/.*(id=[0-9]+),facebook-id) -> test5/url_facebook-id.txt
0: make_histogram(search.*[?&/;fF][pq]=([^&/]+),searches) -> test5/url_searches.txt
# elapsed time: 3618.5 seconds


As can be seen, the program terminated itself when one of the threads required more than 60 minutes to complete activity.

The file report.xml in the output directory contains an accounting of how many times each scanner is called. Given that there are 8 pages in the file 0005.aff, each scanner should be called 8 times at the top level. Here is a section of the file:

  <scanners>
    <path>
      <name>ACCTS</name>
      <calls>8</calls>
      <seconds>19.733836</seconds>
    </path>
    <path>
      <name>BASE64</name>
      <calls>8</calls>
      <seconds>2.000809</seconds>
    </path>
    <path>
      <name>EMAIL</name>
      <calls>8</calls>
      <seconds>24.539532</seconds>
    </path>
    <path>
      <name>EXIF</name>
      <calls>8</calls>
      <seconds>28.422602</seconds>
    </path>
    <path>
      <name>FIND</name>
      <calls>8</calls>
      <seconds>10.511128</seconds>
    </path>
    <path>
      <name>GPS</name>
      <calls>8</calls>
      <seconds>26.975640</seconds>
    </path>
    <path>
      <name>GZIP</name>
      <calls>8</calls>
      <seconds>1.215070</seconds>
    </path>
    <path>
      <name>HIBER</name>
      <calls>7</calls>
      <seconds>0.712045</seconds>
    </path>
    <path>
      <name>JSON</name>
      <calls>8</calls>
      <seconds>3.598577</seconds>
    </path>
    <path>
      <name>KML</name>
      <calls>8</calls>
      <seconds>13.851020</seconds>
    </path>
    <path>
      <name>NET</name>
      <calls>8</calls>
      <seconds>116.055016</seconds>
    </path>
    <path>
      <name>PDF</name>
      <calls>7</calls>
      <seconds>317.052024</seconds>
    </path>
    <path>
      <name>WINPREFETCH</name>
      <calls>7</calls>
      <seconds>5.515840</seconds>
    </path>
    <path>
      <name>ZIP</name>
      <calls>8</calls>
      <seconds>6.532860</seconds>
    </path>
    <path>
      <name>ZIP-ACCTS</name>
      <calls>1</calls>
      <seconds>0.426945</seconds>
    </path>
    <path>
      <name>ZIP-BASE64</name>
      <calls>1</calls>
      <seconds>0.025480</seconds>
    </path>
    <path>
      <name>ZIP-EMAIL</name>
      <calls>1</calls>
      <seconds>0.463570</seconds>
    </path>
    <path>
      <name>ZIP-EXIF</name>
      <calls>1</calls>
      <seconds>0.473475</seconds>
    </path>
    <path>
      <name>ZIP-FIND</name>
      <calls>1</calls>
      <seconds>0.245742</seconds>
    </path>
    <path>
      <name>ZIP-GPS</name>
      <calls>1</calls>
      <seconds>0.394898</seconds>
    </path>
    <path>
      <name>ZIP-GZIP</name>
      <calls>1</calls>
      <seconds>0.013586</seconds>
    </path>
    <path>
      <name>ZIP-HIBER</name>
      <calls>1</calls>
      <seconds>0.013528</seconds>
    </path>
    <path>
      <name>ZIP-JSON</name>
      <calls>1</calls>
      <seconds>0.038624</seconds>
    </path>
    <path>
      <name>ZIP-KML</name>
      <calls>1</calls>
      <seconds>0.295015</seconds>
    </path>
    <path>
      <name>ZIP-NET</name>
      <calls>1</calls>
      <seconds>2.536063</seconds>
    </path>
    <path>
      <name>ZIP-PDF</name>
      <calls>1</calls>
      <seconds>0.242040</seconds>
    </path>
    <path>
      <name>ZIP-WINPREFETCH</name>
      <calls>1</calls>
      <seconds>0.110463</seconds>
    </path>
    <path>
      <name>ZIP-ZIP</name>
      <calls>1</calls>
      <seconds>0.013513</seconds>
    </path>
  </scanners>

As can be seen, the scanner PDF is only called 7 times, not 8. That’s because the 8th time it was called it never finished, and was instead interrupted by the driver program.

To verify that scan_pdf was the problem, we ran bulk_extractor on the test disk with the PDF scanner disabled:


simsong@t:~/domex/src/bulk_extractor$ src/bulk_extractor -xpdf -o test6 0005.aff
Hostname: t
Input file: 0005.aff
Output directory: test6
Disk Size: 130351104
Threads: 8
Phase 1.
18:21:56 Page 0 (0.00%) Completed. Done in n/a
18:21:58 Page 1 (12.50%) Completed. Done in 0:00:42
18:22:00 Page 2 (25.00%) Completed. Done in 0:00:24
18:22:02 Page 3 (37.50%) Completed. Done in 0:00:17
18:22:04 Page 4 (50.00%) Completed. Done in 0:00:11
18:22:05 Page 5 (62.50%) Completed. Done in 0:00:08
18:22:08 Page 6 (75.00%) Completed. Done in 0:00:05
18:22:08 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...

Time elapsed waiting for 1 thread to finish: 37 sec (wait at least 50 min)All Threads Finished!
Phase 2. Shutting down scanners
Phase 3. Creating Histograms
0: make_histogram(,histogram) -> test6/ccn_histogram.txt
0: make_histogram(,histogram) -> test6/ccn_track2_histogram.txt
0: make_histogram(,histogram) -> test6/domain_histogram.txt
0: make_histogram(,histogram) -> test6/email_histogram.txt
0: make_histogram(([^(]+),histogram) -> test6/ether_histogram.txt
0: make_histogram(([^(]+),histogram) -> test6/ip_histogram.txt
0: make_histogram(,histogram) -> test6/tcp_histogram.txt
0: make_histogram(,histogram) -> test6/telephone_histogram.txt
0: make_histogram(,histogram) -> test6/url_histogram.txt
0: make_histogram(://([^/]+),services) -> test6/url_services.txt
0: make_histogram(://(cid-[0-9a-f])+[a-z.].live.com/),microsoft-live) -> test6/url_microsoft-live.txt
0: make_histogram(://[-_a-z0-9.]+facebook.com/.*(id=[0-9]+),facebook-id) -> test6/url_facebook-id.txt
0: make_histogram(search.*[?&/;fF][pq]=([^&/]+),searches) -> test6/url_searches.txt
# elapsed time: 52.9 seconds
simsong@t:~/domex/src/bulk_extractor$ src/bulk_extractor -Epdf -o test7 0005.aff


Now the program finished in 52.9 seconds.

We re-ran bulk_extractor under the GDB debugger, this time with ONLY the PDF scanner enabled. When the was only one thread remaining, we interrupted the program:


(gdb) run -Epdf -o test7 0005.aff
Starting program: /home/simsong/domex/src/bulk_extractor/src/bulk_extractor -Epdf -o test7 0005.aff
[Thread debugging using libthread_db enabled]
Hostname: t
Input file: 0005.aff
Output directory: test7
Disk Size: 130351104
Threads: 8
[New Thread 0x7ffff4c82700 (LWP 1042)]
[New Thread 0x7ffff4431700 (LWP 1044)]
[New Thread 0x7ffff3be0700 (LWP 1045)]
[New Thread 0x7ffff338f700 (LWP 1046)]
[New Thread 0x7ffff2b3e700 (LWP 1048)]
[New Thread 0x7ffff22ed700 (LWP 1050)]
[New Thread 0x7ffff1a9c700 (LWP 1052)]
[New Thread 0x7ffff124b700 (LWP 1053)]
Phase 1.
18:31:41 Page 0 (0.00%) Completed. Done in n/a
18:31:44 Page 1 (12.50%) Completed. Done in 0:00:42
18:31:46 Page 2 (25.00%) Completed. Done in 0:00:24
18:31:48 Page 3 (37.50%) Completed. Done in 0:00:17
18:31:49 Page 4 (50.00%) Completed. Done in 0:00:11
18:31:51 Page 5 (62.50%) Completed. Done in 0:00:08
18:31:53 Page 6 (75.00%) Completed. Done in 0:00:05
18:31:53 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...
Time elapsed waiting for 1 thread to finish: 24 min 48 sec (wait at least 50 min)


At this point we interrupt and use GDB to figure out which thread is having problems:


Time elapsed waiting for 1 thread to finish: 24 min 48 sec (wait at least 50 min) C-c C-c
Program received signal SIGINT, Interrupt.
0x00007ffff5be25ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) where
#0 0x00007ffff5be25ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff5c13984 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000040d621 in do_phase1 (fn=0x7fffffffe45d "0005.aff", cp=..., p=0xa38010, fs=..., xreport=.\
.., page_number=@0x7fffffffdaa8, total_bytes=@0x7fffffffd888, timer=...) at bulk_extractor.cpp:769
#3 0x000000000040f46c in main (argc=1, argv=0x7fffffffe0d8) at bulk_extractor.cpp:1022
(gdb) thread
[Current thread is 1 (Thread 0x7ffff7fcf740 (LWP 977))]
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff4c82700 (LWP 1042))]#0 0x00007ffff5ed4bac in pthread_cond_wait@@GLIBC_2.3.\
2 () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) where
#0 0x00007ffff5ed4bac in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000041a121 in worker::run (this=0xa3ffc0) at threadpool.cpp:163
#2 0x000000000041a2f7 in worker::start_worker (arg=0xa3ffc0) at threadpool.h:60
#3 0x00007ffff5ecfd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ffff5c1b04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff4431700 (LWP 1044))]#0 scan_pdf (sp=..., rcb=...) at scan_pdf.cpp:83
(gdb) where
#0 scan_pdf (sp=..., rcb=...) at scan_pdf.cpp:83
#1 0x000000000040c3d8 in process_extract (sp=...) at bulk_extractor.cpp:576
#2 0x000000000041a254 in worker::do_work (this=0xa40e20, sbuf=0x3d41f00) at threadpool.cpp:187
#3 0x000000000041a1c6 in worker::run (this=0xa40e20) at threadpool.cpp:173
#4 0x000000000041a2f7 in worker::start_worker (arg=0xa40e20) at threadpool.h:60
#5 0x00007ffff5ecfd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007ffff5c1b04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()
(gdb)


As expected, the problem is in scan_pdf.cpp (at line 83, in fact).

At this point we were able to identify the bug in scan_pdf that was causing an infinite loop and eliminate the problem.

• Support for VLANs
• Support for IPv6 (thanks to contributions from Jan Görig).
• Regression testing (note: the IPv6 is currently not regression tested because due to implementation differences of inet_ntop on MacOS and Linux).

The new version is available for download at http://afflib.org/downloads/tcpflow-1.0.1.tar.gz

Background: With the original author’s approval, I have taken over the management of maintenance of the tcpflow open source TCP reconstructor. I brought the software up-to-date with the current release of GNU autotools, applied various patches that were floating around, and added the VLAN support. I am now trying to get the tcpflow in various Linux distributions updated.

Future Direction: I would like to rewrite parts of tcpflow in C++ so that I can take advantage of the STL map class, which is significantly more efficient than the current data structure used by tcpflow to maintain state. I also want to make a linkable tcp flow reconstruction library. I am looking for input from tcpflow users as to 1) whether rewriting in C++ is okay, and 2) what form the library should take.

Once again, you can download the new version from http://afflib.org/downloads/tcpflow-1.0.1.tar.gz