iPhone SDK 3.0 and symbolicatecrash not getting along?
* UPDATE * I've reinstalled with Snow Leopard, clean install. Completely wiped my existing Leopard install. Same problem persists.
I've tried numbers of versions of symbolicatecrash to resolve symbols in my crash reports. From the version provided by Apple, to Alan's Quatermain's version posted on GitHub and finally from http://openradar.appspot.com/6438643.
For whatever reason, the best results I can get is for symbols on my own libraries to get resolved. Normally, this is enough data to point me in the right direction -- other times it is not. With 2.x I had no problems getting the symbols for my code + Apple provided libraries from within the stack traces in each thread.
Most likely an issue with my environment here, I'm not at all doubting the work that Apple or Alan have done. Yes I'm certain the dSYM I have stashed away is the same exact one that's generating the crash report.
Although 'Foo' is me, and getting symbols from it is wonderful, I need to see symbols from the other functions in the stack to truly understand my reports.
Note: For devices that crash running the app on iPhone OS 2.2.1, I have no problem getting all symbols. This is an iPhone OS 3.0 issue it appears.
Also, while running symbolicatecrash in verbose mode here's a few of the things that struck me as wrong:
- NO MATCH
NOT searching in Spotlight for dsym with UUID of /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
## Warning: Can't find any unstripped binary that matches version of /System/Library/Frameworks/CoreFoundation.framework/CoreFoundation
..........fetching symbol file for libobjc.A.dylib--[undef]
Searching [/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A345)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A347)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.1 (5B108)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.2 (5C1)/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2.1/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2/Symbols/usr/lib/libobjc.A.dylib /Developer/Platforms/iPhoneOS.platform/DeviceSupport/3.0 (7A341)/Symbols/usr/lib/libobjc.A.dylib]...--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A345)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0 (5A347)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.1 (5B108)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.0.2 (5C1)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2.1/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/2.2/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
--[/Developer/Platforms/iPhoneOS.platform/DeviceSupport/3.0 (7A341)/Symbols/usr/lib/libobjc.A.dylib] -- NO MATCH
NOT searching in Spotlight for dsym with UUID of /usr/lib/libobjc.A.dylib
## Warning: Can't find any unstripped binary that matches version of /usr/lib/libobjc.A.dylib
Here's an example of the crash report after running it through symbolicatecrash:
Thread 0 Crashed:
0 libSystem.B.dylib 0x31dc476c 0x31d46000 + 517996
1 libSystem.B.dylib 0x31dc4755 0x31d46000 + 517973
2 Foo 0x00053075 uncaught_exception_handler + 21
3 CoreFoundation 0x3028f65f 0x301fd000 + 599647
4 libobjc.A.dylib 0x30013693 0x3000c000 + 30355
5 libstdc++.6.dylib 0x374ccc2d 0x3748a000 + 273453
6 libstdc++.6.dylib 0x374ccc81 0x3748a000 + 273537
7 libstdc++.6.dylib 0x374ccd4d 0x3748a000 + 273741
8 libobjc.A.dylib 0x300135ff 0x3000c000 + 30207
9 CoreFoundation 0x30222f2d 0x301fd000 + 155437
10 CoreFoundation 0x30222ecb 0x301fd000 + 155339
11 Foundation 0x30521e33 0x30501000 + 134707
12 Foundation 0x30570d47 0x30501000 + 458055
13 Foo 0x0000a1db -[Bar barfoo] (Bar.m:1617)
14 Foo 0x00032f73 -[MyViewController foobar] (MyViewController.m:727)
15 Foo 0x000329b9 -[MyViewController foobar] (MyViewController.m:666)
16 Foo 0x00031fab -[MyViewController tabBar:tabSelected:] (MyViewController.m:440)
17 Foo 0x00068d41 -[TTTabBar setSelectedTabIndex:] (TTTabBar.m:160)
18 Foo 0x00068ca3 -[TTTabBar setSelectedTabView:] (TTTabBar.m:142)
19 Foo 0x000689cf -[TTTabBar tabTouchedUp:] (TTTabBar.m:83)
20 CoreFoundation 0x302552f9 0x301fd000 + 361209
21 UIKit 0x3094d101 0x308ed000 + 393473
22 UIKit 0x3094d0a1 0x308ed000 + 393377
23 UIKit 0x3094d073 0x308ed000 + 393331
24 UIKit 0x3094cdcd 0x308ed000 + 392653
25 UIKit 0x309779c1 0x308ed000 + 567745
26 UIKit 0x30977011 0x308ed000 + 565265
27 UIKit 0x309767d9 0x308ed000 + 563161
28 UIKit 0x30923613 0x308ed000 + 222739
29 UIKit 0x30923163 0x308ed000 + 221539
30 GraphicsServices 0x32045a4d 0x32041000 + 19021
31 CoreFoundation 0x30253041 0x301fd000 + 352321
32 CoreFoundation 0x30252771 0x301fd000 + 350065
33 GraphicsServices 0x32044b0f 0x32041000 + 15119
34 GraphicsServices 0x32044bbb 0x32041000 + 15291
35 UIKit 0x308f0363 0x308ed000 + 13155
36 UIKit 0x308ef121 0x308ed000 + 8481
37 Foo 0x00002097 main (main.m:13)
8 Answers
Greg's suggestions, didn't work for me but helped to point the right direction to get plcrashreporter logs (reporter version 103) to symbolize again with sdk 4.0.
BTW: this is an issue you will definitly have with reports created on newer devices (armv7) because symbolicate assumes reports version 103 are armv6.
So this is what i did ...
1)
make a copy from the newest symbolicatecrash script (sdk 4.0) for the changes you make.
Note: depending on the editor u use you might have to set the executable bit after editing.
chmod 755 symbolicatecrash
2) issue: finding symbols -- NO MATCH
this is because the default is set to armv6.
Resolution: in sub parse_images set the default architecture my $default_arch = 'armv7';
3) issue: "Use of uninitialized value $bundle ..."
Resolution: (hack: removing some restrictions that don't seem to work on 103 reports)
replace:
$app = $bundlename if (!defined $app && defined $image{plus} && length $image{plus});
by:
$app = $bundlename if (!defined $app);
4) issue: "can't understand output from otool"
Resolution: Greg's fix for sub matchesUUID.
replace:
} else { die "Can't understand the output from otool ($TEST_uuid)"; }
by:
} else { if ($arch eq "armv7") { return matchesUUID($path, $uuid, "armv6"); } else { die "Can't understand the output from otool ($TEST_uuid)"; } }
5) issue: "atos can not find symbols"
here Gregs script changes didn't work,
- it seems that
<$ph>inrindex(got lost in the code highlighting. - the "atos cannot load symbols" wasn't beeing passed into the script - so the check wouldn't work
- the
<$ph>inrindex(check reads the first line ... so one frame gets lost in case it was successfull. As a "hack" i simply faked a frame with adreass1... which is "used" during the atos check.
So this is what i changed in sub symbolize_frames:
replace:
my $cmd = "$atos -arch $arch -o '$escapedSymbol' @{[ keys %$frames ]} | ";
by:
my $cmd = "$atos -arch $arch -o '$escapedSymbol' 1 @{[ keys %$frames ]} 2>&1 | ";
replace:
open my($ph),$cmd or die $!;
by:
open my($ph),$cmd or die $!;
if (rindex(<$ph>, "atos cannot load symbols") != -1 && $arch eq "armv7") {
my $arch = "armv6"; my $cmd = "$atos -arch $arch -o '$escapedSymbol' @{[ keys %$frames ]} 2>&1 | ";
print STDERR "Running $cmd\n" if $opt{v};
open $ph,$cmd or die $!;
}
Note: with this changes the script runs fine for me - the "fixes" are probably not very elegant but i'm not fluent in perl :) ...
archglIf you are experiencing the situation where, when you run symbolicate crash only your own libraries are symbolicated, but the system libraries are not symbolicated and you have 3.0+ iPhone SDK installed most likely the issue is that the report format that you are symbolicating is an older format. One of the header fields in the crash report is "Report Version", this is the version of your crash report. The most current version (at the time of this writing iPhone OS 3.2.1) is "104". However, symbolicatecrash handles version 102 and 103 as well. Version 104 introduced new a new chip architecture (armv7) and the binary images have the ability to specify the chip architecture that is used. Version 102 and 103 of the crash report do not specify the chip architecture to use. The most current version of symbolicatecrash assume to use armv6 for version 102 and 103 of the report. This is probably wrong.
Symbolicate crash needs to be written to be more compatible with the older report formats. In general the UUID's for the binary images are really all that you need to correctly match up the system library symbols for the correct OS version as well as architecture--the architecture is probably redundant. However, in order for this to occur correctly, the symbolicate crash script needs to be modified, such that in addition to iterating through all the system library versions installed on the host machine to find the correct library, it also needs to iterate through all the chip architectures as well. This, fortunately is a pretty easy change to make.
To handle this I made the following changes:
in the getSymbolPathFor subroutine:
if ( defined($temp_path) && matchesUUID($temp_path, $uuid, $arch) ) {
$out_path = $temp_path;
@out_path_arr = {};
} else {
undef $temp_path;
print STDERR "-- NO MATCH\n" if $opt{v};
}
change to:
if ( defined($temp_path) && matchesUUID($temp_path, $uuid, $arch) ) {
$out_path = $temp_path;
@out_path_arr = {};
} else {
if ( defined($temp_path) && matchesUUID($temp_path, $uuid, 'armv7') ) {
$out_path = $temp_path;
@out_path_arr = {};
} else {
undef $temp_path;
print STDERR "-- NO MATCH\n" if $opt{v};
}
}
This will test out different chip architectures's UUID on the host system to find a match with the UUID in the binary image section in the crash report (and if the upcoming 2010 iPhone has a different chip architecture that should be added to this if-else condition).
For the next change, the UUID searching piece of code is brittle. If otool does not return something that symbolicatecrash can understand, symbolicatecrash will exit. In the case where you test the UUID of an architecture that is not recognized by the host machine, otool will return nothing, and symbolicatecrash dies. You need to make symbolicate crash a little more lenient by just ignore the otool result so that it can move on to the next architecture you are checking.
Change the matchesUUID subroutine:
if ( $test eq $uuid ) {
## See that it isn't stripped. Even fully stripped apps have one symbol, so ensure that there is more than one.
my ($nlocalsym) = $TEST_uuid =~ /nlocalsym\s+([0-9A-Fa-f]+)/;
my ($nextdefsym) = $TEST_uuid =~ /nextdefsym\s+([0-9A-Fa-f]+)/;
my $totalsym = $nextdefsym + $nlocalsym;
print STDERR "\nNumber of symbols in $path: $nextdefsym + $nlocalsym = $totalsym\n" if $opt{v};
return 1 if ( $totalsym > 1 );
print STDERR "## $path appears to be stripped, skipping.\n" if $opt{v};
} else {
print STDERR "Given UUID $uuid for '$path' is really UUID $test\n" if $opt{v};
}
} else {
die "Can't understand the output from otool ($TEST_uuid)";
}
To something more lenient like this:
if ( $test eq $uuid ) {
## See that it isn't stripped. Even fully stripped apps have one symbol, so ensure that there is more than one.
my ($nlocalsym) = $TEST_uuid =~ /nlocalsym\s+([0-9A-Fa-f]+)/;
my ($nextdefsym) = $TEST_uuid =~ /nextdefsym\s+([0-9A-Fa-f]+)/;
my $totalsym = $nextdefsym + $nlocalsym;
print STDERR "\nNumber of symbols in $path: $nextdefsym + $nlocalsym = $totalsym\n" if $opt{v};
return 1 if ( $totalsym > 1 );
print STDERR "## $path appears to be stripped, skipping.\n" if $opt{v};
} else {
print STDERR "Given UUID $uuid for '$path' is really UUID $test\n" if $opt{v};
}
} else {
#die "Can't understand the output from otool ($TEST_uuid)";
print "Can't understand the output from otool ($TEST_uuid)";
}
After I made these changes I was able to see the system objects get symbolicated correctly (as well as my own code) when I use crash report versions 102 and 103 with the current iPhone SDK (3.1.2).
Hassan Abdel-Rahman • edited Jan 15, 2010 by Hassan Abdel-RahmanI experienced the exact same problem and was able to fix it by altering the symbolicatecrash script. The problem for me was that both 'otool', 'atos' and 'size' didn't behave well with modules compiled for armv7. There are versions of these tools that work well with such modules which can be found at /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin. For otool, the symbolicatecrash script found at /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources already utilizes a mechanism to find the relevant version:
# Find otool from the latest iphoneos
my $otool = `xcrun -sdk iphoneos -find otool`;
chomp($otool);
my $atos = `xcrun -sdk iphoneos -find atos`;
chomp($atos);
However, 'size' is used directly:
if (-e '/usr/bin/size') {
open my($ph),"-|",'size','-m','-l','-x',$symbol or die $!;
Altering this piece of code to the ugly:
if (-e '/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/size') {
open my($ph),"-|",'/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/size','-m','-l','-x',$symbol or die $!;
seems to make it work.
yonilevyI know that for the 3.0 SDK they moved symbolicatecrash into a new location and I had to get this new version in order to get the symbol mappings to work again.
For 3.0 it is found at: /Developer/Platforms/iPhoneOS.platform/Developer/Library/PrivateFrameworks/DTDeviceKit.framework/Versions/A/Resources
So it might be worth making sure you have the most recent version. I am seeing proper symbol mapping for both my source and API functions.
paulthenerdI was experiencing exactly the same issue - symbols for system modules were being displayed but not from my application. Turns out it was because my .dSym files could not be found by spotlight as I had "hidden" all the build products by specifying in the preferences that all build files go in the ".xcode-build" directory. Changing this to "xcode-build" fixed the problem!
Gabrielone thing to check. if you know the dsym file that you want to symbolicate against, then right click on the dsym file and select "Get Info". In the more info section is the UUID for the dsym file. In order for symbolication to work, the UUID for your application (enclosed in angle brackets in the binary images section of the crash log), needs to match the UUID of the DSYM file.
I found a problem in my usage of PLCrashReporter where the UUID in the crash log was not matching the UUID of the generated dsym file, resulting in symbolication failing.
Hassan Abdel-Rahman • edited Jan 14, 2010 by Hassan Abdel-RahmanEdit symbolicatecrash. Go to about line 506, and change this:
ARM => "armv6",
to this:
ARM => "armv7",
leftspinI've had numerous runins with symbolicatecrash, the latest since upgrading to SDK 4. Despite building for both armv6 and amrv7 architectures, otool and atos choke on my app binary image. I made the following changes and things are working again. Hope this helps.
FROM (sub symbolize_frames):
open my($ph),$cmd or die $!;
TO:
open my($ph),$cmd or die $!;
if (rindex(, "atos cannot load symbols" != -1 && $arch eq "armv7")) {
$arch = "armv6";
my $cmd = "$atos -arch $arch -o '$escapedSymbol' @{[ keys %$frames ]} | ";
print STDERR "Running $cmd\n" if $opt{v};
open $ph,$cmd or die $!;
}
AND
FROM (sub matchesUUID):
} else {
die "Can't understand the output from otool ($TEST_uuid)";
}
TO:
} else {
if ($arch eq "armv7") {
return matchesUUID($path, $uuid, "armv6");
} else {
die "Can't understand the output from otool ($TEST_uuid)";
}
}
GregUser contributions licensed under CC BY-SA 3.0