Bug #513

Ruby 1.9.1 segfaults with threaded scanners and an active database

Added by James Lee about 2 years ago. Updated about 2 years ago.

Status:Closed Start date:11/13/2009
Priority:High Due date:
Assignee:James Lee % Done:

100%
Category:general
Target version:Metasploit 3.3.2
Resolution: Release Note:

Description

msf > use auxiliary/scanner/http/version 
msf auxiliary(version) > set RHOSTS 74.125.53.1-74.125.53.254
RHOSTS => 74.125.53.1-74.125.53.254
msf auxiliary(version) > set THREADS 10
THREADS => 10
msf auxiliary(version) > run

[*] 74.125.53.17 is running gws
[*] 74.125.53.19 is running gws
[*] 74.125.53.18 is running gws
[*] Scanned 028 of 254 hosts (011% complete)
[*] 74.125.53.32 is running gws
[*] 74.125.53.34 is running gws
[*] 74.125.53.33 is running gws
[*] 74.125.53.36 is running gws
[*] 74.125.53.35 is running gws
[*] 74.125.53.38 is running gws
[*] 74.125.53.37 is running gws
[*] 74.125.53.39 is running gws
[*] 74.125.53.40 is running gws
[*] 74.125.53.42 is running gws
[*] 74.125.53.41 is running gws
[*] 74.125.53.43 is running gws
[-] Auxiliary failed: fatal deadlock detected
[-] Call stack:
[-]   /home/egypt/svn/framework3/trunk/lib/msf/core/auxiliary/scanner.rb:114:in `join'
[-]   /home/egypt/svn/framework3/trunk/lib/msf/core/auxiliary/scanner.rb:114:in `run'
[*] Auxiliary module execution completed
msf auxiliary(version) >

And...
msf auxiliary(version) > set THREADS 100
THREADS => 100
msf auxiliary(version) > run

[*] 74.125.53.58 is running gws
... bunch of google webservers in here, snipped ...
[*] 74.125.53.136 is running gws
[BUG] Segmentation fault
ruby 1.9.1p243 (2009-07-16 revision 24175) [i686-linux]

-- control frame ----------
---------------------------
-- Ruby level backtrace information-----------------------------------------

-- C level backtrace information -------------------------------------------
0x812dd48 ruby(rb_vm_bugreport+0x48) [0x812dd48]
0x815c8a1 ruby [0x815c8a1]
0x815c918 ruby(rb_bug+0x28) [0x815c918]
0x80d95f6 ruby [0x80d95f6]
0xb7f60410 [0xb7f60410]
0x811e967 ruby(rb_funcall+0xb7) [0x811e967]
0x805d177 ruby(rb_make_exception+0x77) [0x805d177]
0x8131614 ruby [0x8131614]
0x81316af ruby [0x81316af]
0x813213d ruby [0x813213d]
0x81323a1 ruby [0x81323a1]
0xb7f3b4ff /lib/tls/i686/cmov/libpthread.so.0 [0xb7f3b4ff]
0xb7e5049e /lib/tls/i686/cmov/libc.so.6(clone+0x5e) [0xb7e5049e]

[NOTE]
You may encounter a bug of Ruby interpreter. Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted

Associated revisions

Revision bfe2cc0c
Added by HD Moore about 2 years ago

Force stale connections to be freed, fixes #513.

git-svn-id: file:///home/svn/framework3/trunk@7641 4d416f70-5f16-0410-b530-b9f4589650da

Revision ee0c1d46
Added by James Lee about 2 years ago

remove the lock that was causing segfaults on 1.9; fixes #513, see #514

git-svn-id: file:///home/svn/framework3/trunk@7703 4d416f70-5f16-0410-b530-b9f4589650da

History

#1
Updated by James Lee about 2 years ago

Does not segfault when db is inactive.
This list of commands appears to reliably reproduce the segfault.

db_destroy
db_create
use auxiliary/scanner/http/version
set RHOSTS 74.125.53.1-74.125.53.254
set THREADS 100
run

#2
Updated by HD Moore about 2 years ago

I was hoping this was a bug in SQLite3, but it triggers with Postgres-PR as well (ruby only driver):

[BUG] Segmentation fault
ruby 1.9.1p243 (2009-07-16 revision 24175) [x86_64-linux]

-- control frame ----------
---------------------------
-- Ruby level backtrace information-----------------------------------------

-- C level backtrace information -------------------------------------------
0x4e1601 ruby(rb_vm_bugreport+0x41) [0x4e1601]
0x50df48 ruby [0x50df48]
0x50e0b1 ruby(rb_bug+0xb1) [0x50e0b1]
0x48f928 ruby [0x48f928]
0x7f1b04a99190 /lib/libpthread.so.0 [0x7f1b04a99190]
0x4dbc55 ruby [0x4dbc55]
0x4dce77 ruby(rb_funcall+0x1a7) [0x4dce77]
0x4193e4 ruby(rb_make_exception+0x74) [0x4193e4]
0x4e49ba ruby [0x4e49ba]
0x4e4a60 ruby [0x4e4a60]
0x4e5632 ruby [0x4e5632]
0x4e583e ruby [0x4e583e]
0x7f1b04a90a04 /lib/libpthread.so.0 [0x7f1b04a90a04]
0x7f1b03f317bd /lib/libc.so.6(clone+0x6d) [0x7f1b03f317bd]

[NOTE]
You may encounter a bug of Ruby interpreter. Bug reports are welcome.
For details: http://www.ruby-lang.org/bugreport.html

Aborted

Looks like a Ruby 1.9.1 bug, testing with Ruby 1.8.x now

#3
Updated by HD Moore about 2 years ago

  • Subject changed from Threading issues with scanners to Ruby 1.9.1 segfaults with threaded scanners and an active database
  • Priority changed from Normal to High
  • Target version set to Metasploit 3.4.0

Confirmed that this is 1.9.1 only and seems to trigger when database support is used, could be related to event handling (or at least avoidable through some changes).

#4
Updated by HD Moore about 2 years ago

Re-enabling the mutex lock around DBObject.save didn't help

#5
Updated by HD Moore about 2 years ago

Triggers on 1.9.1-head as well:

ruby 1.9.1p334 (2009-11-13 revision 25741) [x86_64-linux]

#6
Updated by HD Moore about 2 years ago

Removing the calls to thread.join() avoids the crash, however the threads get stuck in limbo when the report* methods are called.

#7
Updated by HD Moore about 2 years ago

  • Assignee changed from James Lee to HD Moore

#8
Updated by HD Moore about 2 years ago

The root cause was the connection pool being full, but Ruby didnt handle it very well.

#9
Updated by HD Moore about 2 years ago

  • Status changed from New to Closed
  • % Done changed from 0 to 100

Applied in changeset r7641.

#10
Updated by HD Moore about 2 years ago

  • Target version changed from Metasploit 3.4.0 to Metasploit 3.3.1

#11
Updated by HD Moore about 2 years ago

  • Status changed from Closed to Assigned
  • Target version changed from Metasploit 3.3.1 to Metasploit 3.4.0

Reproduced when db_tracker was loaded

#12
Updated by HD Moore about 2 years ago

  • % Done changed from 100 to 0

#13
Updated by HD Moore about 2 years ago

  • % Done changed from 0 to 50

Only reproduces with db_tracker now at least

#14
Updated by James Lee about 2 years ago

Wrapping the save in a transaction ("ActiveRecord::Base.transaction do") doesn't help.

Removing the Lock.mutex.synchronize around the save and find methods fixes the segfaults, but reintroduces the sqlite3 BusyException problem.

#15
Updated by HD Moore about 2 years ago

  • Assignee changed from HD Moore to James Lee

Next step would be to look into alternative locking mechanisms

#16
Updated by James Lee about 2 years ago

  • Status changed from Assigned to Closed
  • % Done changed from 50 to 100

Applied in changeset r7703.

#17
Updated by HD Moore about 2 years ago

  • Target version changed from Metasploit 3.4.0 to Metasploit 3.3.2

Also available in: Atom PDF