AFFLIB http://afflib.org Open Source Computer Forensics Software Mon, 30 Jan 2012 23:17:33 +0000 en hourly 1 http://wordpress.org/?v=3.3.1 tcpflow 1.1.0 released! http://afflib.org/archives/155 http://afflib.org/archives/155#comments Sun, 29 Jan 2012 04:24:50 +0000 Simson http://afflib.org/?p=155 Version 1.1.0 19 January 2012 (SVN 8118) I am pleased to announce the release of tcpflow version 1.1. Version 1.1 represents a significant rewrite of tcpflow. All users are encouraged to upgrade. Significant changes include: * Entire code base migrated to C++ ; code generally improved. tcpflow's original hash table has been replaced with a tr1::unordered_map which should offer significantly more scalability. * tcpflow now automatically expires out old connections. This finally end the program's memory-hogging problem. (You can disable this behavior with -P, which makes tcpflow run faster because it never cleans up after itself. That's fine if you are working with less than a million connections.) * Multiple connections with the same (source/destination) are now detected and stored in different files. This is significant, as the previous implementation would make a single file 1-2GB in length if you the same host/port pairs with two different flows. Additional files have the same filename and a "c0001", "c0002" appended. * Filenames may now be prefixed with either the ISO8601 time or a Unix timestamp indicating the time that the connection was first seen. * tcpflow will now save a DFXML file containing information for each flow that it reconstructs. * The following new options are now implemented: -o outdir --- now works (previously was not implemented) -X xmfile --- now reports execution results in a DFXML file. (Version 1.1 will include complete notion in the XML file of every TCP connection as a DFXML -Fc --- Every file has the 'cXXXX' postfix, rather than just the files with duplicate source/destination. -Ft --- Every file has the T prefix. -FT --- Every file has an ISO8601 time prefix, e.g. 2012-01-01T09:45:15Z -mNNNN --- Specifies the minimum number of bytes that need to be skipped in a TCP connection before a new -Lname --- use the named semaphore 'name' to prevent multiple tcpflow processes printing to standard output from overprinting each other. -P --- do not prune the tcp connection table. Other improvements include: * Support for IPv6 * Support for VLANs * The default filter which was causing problems under MacOS has been removed. tcpflow can be downloaded from: http://afflib.org/ http://afflib.org/software/tcpflow Finally, because the previous maintainer had lost control of the old tcpflow mailing list, a new one has been created at Google Groups. You can subscribe at: http://groups.google.com/group/tcpflow-users ]]> http://afflib.org/archives/155/feed 0 docx_steg is posted http://afflib.org/archives/154 http://afflib.org/archives/154#comments Thu, 26 Jan 2012 12:58:58 +0000 Simson http://afflib.org/?p=154 I received a query about the docx stegnography tool that was developed with Jim Migletz back in 2008, so it has been posted to my private website at http://simson.net/page/Docx_steg

]]> http://afflib.org/archives/154/feed 0 New Releases! http://afflib.org/archives/151 http://afflib.org/archives/151#comments Fri, 02 Dec 2011 14:42:10 +0000 Simson http://afflib.org/?p=151 Today we are making a series of minor software releases to fix minor bugs and to address some compatibility issues with various versions of Linux. This include:

  • AFFLIB 3.6.15
  • fiwalk 0.6.16. (This is the last version of fiwalk that will include the DFXML tools; they will be moved into a new release as fiwalk is being folded into SleuthKit.)
  • bulk_extractor version 1.1.1. (Version 1.0 had two minor bugs, one having to do with the identify_filenames.py script, and one having to do with histogram generation.)
]]> http://afflib.org/archives/151/feed 0 Debugging bulk_extractor Performance Problems http://afflib.org/archives/150 http://afflib.org/archives/150#comments Wed, 09 Nov 2011 14:43:45 +0000 Simson http://afflib.org/?p=150 A user reported that disk #0005.aff was taking a surprising amount of time to complete processing—more than 50 minutes. We ran bulk_extractor on the disk:


$ src/bulk_extractor -o test5 0005.aff
Hostname: t
Input file: 0005.aff
Output directory: test5
Disk Size: 130351104
Threads: 8
Phase 1.
17:18:00 Page 0 (0.00%) Completed. Done in n/a
17:18:02 Page 1 (12.50%) Completed. Done in 0:00:46
17:18:04 Page 2 (25.00%) Completed. Done in 0:00:26
17:18:06 Page 3 (37.50%) Completed. Done in 0:00:18
17:18:08 Page 4 (50.00%) Completed. Done in 0:00:12
17:18:10 Page 5 (62.50%) Completed. Done in 0:00:09
17:18:13 Page 6 (75.00%) Completed. Done in 0:00:06
17:18:13 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...

Time elapsed waiting for 1 thread to finish: 60 min 1 sec (wait at least 50 min))

... this shouldn't take more than an hour. Exiting ...
... Please report to the bulk_extractor maintainer ...
All Threads Finished!
Phase 2. Shutting down scanners
Phase 3. Creating Histograms
0: make_histogram(,histogram) -> test5/ccn_histogram.txt
0: make_histogram(,histogram) -> test5/ccn_track2_histogram.txt
0: make_histogram(,histogram) -> test5/domain_histogram.txt
0: make_histogram(,histogram) -> test5/email_histogram.txt
0: make_histogram(([^(]+),histogram) -> test5/ether_histogram.txt
0: make_histogram(([^(]+),histogram) -> test5/ip_histogram.txt
0: make_histogram(,histogram) -> test5/tcp_histogram.txt
0: make_histogram(,histogram) -> test5/telephone_histogram.txt
0: make_histogram(,histogram) -> test5/url_histogram.txt
0: make_histogram(://([^/]+),services) -> test5/url_services.txt
0: make_histogram(://(cid-[0-9a-f])+[a-z.].live.com/),microsoft-live) -> test5/url_microsoft-live.txt
0: make_histogram(://[-_a-z0-9.]+facebook.com/.*(id=[0-9]+),facebook-id) -> test5/url_facebook-id.txt
0: make_histogram(search.*[?&/;fF][pq]=([^&/]+),searches) -> test5/url_searches.txt
# elapsed time: 3618.5 seconds

As can be seen, the program terminated itself when one of the threads required more than 60 minutes to complete activity.

The file report.xml in the output directory contains an accounting of how many times each scanner is called. Given that there are 8 pages in the file 0005.aff, each scanner should be called 8 times at the top level. Here is a section of the file:

  <scanners>
    <path>
      <name>ACCTS</name>
      <calls>8</calls>
      <seconds>19.733836</seconds>
    </path>
    <path>
      <name>BASE64</name>
      <calls>8</calls>
      <seconds>2.000809</seconds>
    </path>
    <path>
      <name>EMAIL</name>
      <calls>8</calls>
      <seconds>24.539532</seconds>
    </path>
    <path>
      <name>EXIF</name>
      <calls>8</calls>
      <seconds>28.422602</seconds>
    </path>
    <path>
      <name>FIND</name>
      <calls>8</calls>
      <seconds>10.511128</seconds>
    </path>
    <path>
      <name>GPS</name>
      <calls>8</calls>
      <seconds>26.975640</seconds>
    </path>
    <path>
      <name>GZIP</name>
      <calls>8</calls>
      <seconds>1.215070</seconds>
    </path>
    <path>
      <name>HIBER</name>
      <calls>7</calls>
      <seconds>0.712045</seconds>
    </path>
    <path>
      <name>JSON</name>
      <calls>8</calls>
      <seconds>3.598577</seconds>
    </path>
    <path>
      <name>KML</name>
      <calls>8</calls>
      <seconds>13.851020</seconds>
    </path>
    <path>
      <name>NET</name>
      <calls>8</calls>
      <seconds>116.055016</seconds>
    </path>
    <path>
      <name>PDF</name>
      <calls>7</calls>
      <seconds>317.052024</seconds>
    </path>
    <path>
      <name>WINPREFETCH</name>
      <calls>7</calls>
      <seconds>5.515840</seconds>
    </path>
    <path>
      <name>ZIP</name>
      <calls>8</calls>
      <seconds>6.532860</seconds>
    </path>
    <path>
      <name>ZIP-ACCTS</name>
      <calls>1</calls>
      <seconds>0.426945</seconds>
    </path>
    <path>
      <name>ZIP-BASE64</name>
      <calls>1</calls>
      <seconds>0.025480</seconds>
    </path>
    <path>
      <name>ZIP-EMAIL</name>
      <calls>1</calls>
      <seconds>0.463570</seconds>
    </path>
    <path>
      <name>ZIP-EXIF</name>
      <calls>1</calls>
      <seconds>0.473475</seconds>
    </path>
    <path>
      <name>ZIP-FIND</name>
      <calls>1</calls>
      <seconds>0.245742</seconds>
    </path>
    <path>
      <name>ZIP-GPS</name>
      <calls>1</calls>
      <seconds>0.394898</seconds>
    </path>
    <path>
      <name>ZIP-GZIP</name>
      <calls>1</calls>
      <seconds>0.013586</seconds>
    </path>
    <path>
      <name>ZIP-HIBER</name>
      <calls>1</calls>
      <seconds>0.013528</seconds>
    </path>
    <path>
      <name>ZIP-JSON</name>
      <calls>1</calls>
      <seconds>0.038624</seconds>
    </path>
    <path>
      <name>ZIP-KML</name>
      <calls>1</calls>
      <seconds>0.295015</seconds>
    </path>
    <path>
      <name>ZIP-NET</name>
      <calls>1</calls>
      <seconds>2.536063</seconds>
    </path>
    <path>
      <name>ZIP-PDF</name>
      <calls>1</calls>
      <seconds>0.242040</seconds>
    </path>
    <path>
      <name>ZIP-WINPREFETCH</name>
      <calls>1</calls>
      <seconds>0.110463</seconds>
    </path>
    <path>
      <name>ZIP-ZIP</name>
      <calls>1</calls>
      <seconds>0.013513</seconds>
    </path>
  </scanners>

As can be seen, the scanner PDF is only called 7 times, not 8. That’s because the 8th time it was called it never finished, and was instead interrupted by the driver program.

To verify that scan_pdf was the problem, we ran bulk_extractor on the test disk with the PDF scanner disabled:


simsong@t:~/domex/src/bulk_extractor$ src/bulk_extractor -xpdf -o test6 0005.aff
Hostname: t
Input file: 0005.aff
Output directory: test6
Disk Size: 130351104
Threads: 8
Phase 1.
18:21:56 Page 0 (0.00%) Completed. Done in n/a
18:21:58 Page 1 (12.50%) Completed. Done in 0:00:42
18:22:00 Page 2 (25.00%) Completed. Done in 0:00:24
18:22:02 Page 3 (37.50%) Completed. Done in 0:00:17
18:22:04 Page 4 (50.00%) Completed. Done in 0:00:11
18:22:05 Page 5 (62.50%) Completed. Done in 0:00:08
18:22:08 Page 6 (75.00%) Completed. Done in 0:00:05
18:22:08 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...

Time elapsed waiting for 1 thread to finish: 37 sec (wait at least 50 min)All Threads Finished!
Phase 2. Shutting down scanners
Phase 3. Creating Histograms
0: make_histogram(,histogram) -> test6/ccn_histogram.txt
0: make_histogram(,histogram) -> test6/ccn_track2_histogram.txt
0: make_histogram(,histogram) -> test6/domain_histogram.txt
0: make_histogram(,histogram) -> test6/email_histogram.txt
0: make_histogram(([^(]+),histogram) -> test6/ether_histogram.txt
0: make_histogram(([^(]+),histogram) -> test6/ip_histogram.txt
0: make_histogram(,histogram) -> test6/tcp_histogram.txt
0: make_histogram(,histogram) -> test6/telephone_histogram.txt
0: make_histogram(,histogram) -> test6/url_histogram.txt
0: make_histogram(://([^/]+),services) -> test6/url_services.txt
0: make_histogram(://(cid-[0-9a-f])+[a-z.].live.com/),microsoft-live) -> test6/url_microsoft-live.txt
0: make_histogram(://[-_a-z0-9.]+facebook.com/.*(id=[0-9]+),facebook-id) -> test6/url_facebook-id.txt
0: make_histogram(search.*[?&/;fF][pq]=([^&/]+),searches) -> test6/url_searches.txt
# elapsed time: 52.9 seconds
simsong@t:~/domex/src/bulk_extractor$ src/bulk_extractor -Epdf -o test7 0005.aff

Now the program finished in 52.9 seconds.

We re-ran bulk_extractor under the GDB debugger, this time with ONLY the PDF scanner enabled. When the was only one thread remaining, we interrupted the program:


(gdb) run -Epdf -o test7 0005.aff
Starting program: /home/simsong/domex/src/bulk_extractor/src/bulk_extractor -Epdf -o test7 0005.aff
[Thread debugging using libthread_db enabled]
Hostname: t
Input file: 0005.aff
Output directory: test7
Disk Size: 130351104
Threads: 8
[New Thread 0x7ffff4c82700 (LWP 1042)]
[New Thread 0x7ffff4431700 (LWP 1044)]
[New Thread 0x7ffff3be0700 (LWP 1045)]
[New Thread 0x7ffff338f700 (LWP 1046)]
[New Thread 0x7ffff2b3e700 (LWP 1048)]
[New Thread 0x7ffff22ed700 (LWP 1050)]
[New Thread 0x7ffff1a9c700 (LWP 1052)]
[New Thread 0x7ffff124b700 (LWP 1053)]
Phase 1.
18:31:41 Page 0 (0.00%) Completed. Done in n/a
18:31:44 Page 1 (12.50%) Completed. Done in 0:00:42
18:31:46 Page 2 (25.00%) Completed. Done in 0:00:24
18:31:48 Page 3 (37.50%) Completed. Done in 0:00:17
18:31:49 Page 4 (50.00%) Completed. Done in 0:00:11
18:31:51 Page 5 (62.50%) Completed. Done in 0:00:08
18:31:53 Page 6 (75.00%) Completed. Done in 0:00:05
18:31:53 Page 7 (87.50%) Completed. Done in 0:00:02
All Data is Read; waiting for threads to finish...
Time elapsed waiting for 1 thread to finish: 24 min 48 sec (wait at least 50 min)

At this point we interrupt and use GDB to figure out which thread is having problems:


Time elapsed waiting for 1 thread to finish: 24 min 48 sec (wait at least 50 min) C-c C-c
Program received signal SIGINT, Interrupt.
0x00007ffff5be25ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) where
#0 0x00007ffff5be25ad in nanosleep () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007ffff5c13984 in usleep () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000040d621 in do_phase1 (fn=0x7fffffffe45d "0005.aff", cp=..., p=0xa38010, fs=..., xreport=.\
.., page_number=@0x7fffffffdaa8, total_bytes=@0x7fffffffd888, timer=...) at bulk_extractor.cpp:769
#3 0x000000000040f46c in main (argc=1, argv=0x7fffffffe0d8) at bulk_extractor.cpp:1022
(gdb) thread
[Current thread is 1 (Thread 0x7ffff7fcf740 (LWP 977))]
(gdb) thread 2
[Switching to thread 2 (Thread 0x7ffff4c82700 (LWP 1042))]#0 0x00007ffff5ed4bac in pthread_cond_wait@@GLIBC_2.3.\
2 () from /lib/x86_64-linux-gnu/libpthread.so.0
(gdb) where
#0 0x00007ffff5ed4bac in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/x86_64-linux-gnu/libpthread.so.0
#1 0x000000000041a121 in worker::run (this=0xa3ffc0) at threadpool.cpp:163
#2 0x000000000041a2f7 in worker::start_worker (arg=0xa3ffc0) at threadpool.h:60
#3 0x00007ffff5ecfd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#4 0x00007ffff5c1b04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x0000000000000000 in ?? ()
(gdb) thread 3
[Switching to thread 3 (Thread 0x7ffff4431700 (LWP 1044))]#0 scan_pdf (sp=..., rcb=...) at scan_pdf.cpp:83
(gdb) where
#0 scan_pdf (sp=..., rcb=...) at scan_pdf.cpp:83
#1 0x000000000040c3d8 in process_extract (sp=...) at bulk_extractor.cpp:576
#2 0x000000000041a254 in worker::do_work (this=0xa40e20, sbuf=0x3d41f00) at threadpool.cpp:187
#3 0x000000000041a1c6 in worker::run (this=0xa40e20) at threadpool.cpp:173
#4 0x000000000041a2f7 in worker::start_worker (arg=0xa40e20) at threadpool.h:60
#5 0x00007ffff5ecfd8c in start_thread () from /lib/x86_64-linux-gnu/libpthread.so.0
#6 0x00007ffff5c1b04d in clone () from /lib/x86_64-linux-gnu/libc.so.6
#7 0x0000000000000000 in ?? ()
(gdb)

As expected, the problem is in scan_pdf.cpp (at line 83, in fact).

At this point we were able to identify the bug in scan_pdf that was causing an infinite loop and eliminate the problem.

]]> http://afflib.org/archives/150/feed 0 tcpflow 1.0.1 is released! http://afflib.org/archives/145 http://afflib.org/archives/145#comments Mon, 26 Sep 2011 02:00:08 +0000 admin http://afflib.org/?p=145 I am happy to announce that tcpflow 1.0.1 is now available. Improvements in tcpflow 1.0.1 over the version widely in use today (version 0.21) include:

  • Support for VLANs
  • Support for IPv6 (thanks to contributions from Jan Görig).
  • Regression testing (note: the IPv6 is currently not regression tested because due to implementation differences of inet_ntop on MacOS and Linux).

The new version is available for download at http://afflib.org/downloads/tcpflow-1.0.1.tar.gz

Background: With the original author’s approval, I have taken over the management of maintenance of the tcpflow open source TCP reconstructor. I brought the software up-to-date with the current release of GNU autotools, applied various patches that were floating around, and added the VLAN support. I am now trying to get the tcpflow in various Linux distributions updated.

Future Direction: I would like to rewrite parts of tcpflow in C++ so that I can take advantage of the STL map class, which is significantly more efficient than the current data structure used by tcpflow to maintain state. I also want to make a linkable tcp flow reconstruction library. I am looking for input from tcpflow users as to 1) whether rewriting in C++ is okay, and 2) what form the library should take.

Once again, you can download the new version from http://afflib.org/downloads/tcpflow-1.0.1.tar.gz

 

]]> http://afflib.org/archives/145/feed 0 aimage-3.2.5 released http://afflib.org/archives/137 http://afflib.org/archives/137#comments Wed, 17 Aug 2011 23:37:32 +0000 admin http://afflib.org/?p=137 Although I previously stated aimage was withdrawn from support, I have continued to receive requests for support. As the final version of aimage did not compile with the current version of AFFLIB, I have updated aimage so that now it does.

aimage is still not supported, but version 3.2.5 has been released.

]]> http://afflib.org/archives/137/feed 2 bulk_extractor 1.0.1 is released http://afflib.org/archives/136 http://afflib.org/archives/136#comments Mon, 18 Jul 2011 04:07:42 +0000 admin http://afflib.org/?p=136 This version features minor bug fixes in the C++ program. The python system features a significantly improved identify_filenames.py for locating features within their source files. Use of this module requires that either fiwalk be installed or that a DFXML file exist.

]]> http://afflib.org/archives/136/feed 1 fiwalk 0.6.15 is released http://afflib.org/archives/135 http://afflib.org/archives/135#comments Mon, 18 Jul 2011 03:42:19 +0000 admin http://afflib.org/?p=135 This is a minor release which includes improved handling of byte_runs for compressed regions. It also includes an improved dfxml.py and fiwalk.py modules.

]]> http://afflib.org/archives/135/feed 0 Lecture Snippets Discusses bulk_extractor http://afflib.org/archives/131 http://afflib.org/archives/131#comments Sun, 03 Jul 2011 20:35:27 +0000 admin http://afflib.org/?p=131 bulk_extractor is now in Backtrack 5 and Lecture Snippets has published a five-minute video explaining its use.

]]> http://afflib.org/archives/131/feed 0 bulk_extractor 1.0.0 is released http://afflib.org/archives/129 http://afflib.org/archives/129#comments Wed, 15 Jun 2011 03:32:15 +0000 admin http://afflib.org/?p=129 I am pleased to report that bulk_extractor 1.0.0 is released for Linux, Mac and Windows.

Initial testing indicates that the Windows version runs at 50% the speed of the Mac and Linux versions on the same hardware. If you are using Windows, try to move to Mac or Linux.

The distribution includes the python tools in the python directory and the bulk_extractor papers and slides in the doc directory.

]]> http://afflib.org/archives/129/feed 0