Debugging Java JVM Crashes - Native Code Access

▲
3
▼

Anyone have any ideas what is causing the below JVM crash? I'm using java 6 and Windows 32 bit. I'm trying to access native (JNI) C APIs. I can access some of them, but a few cause the below crash.

C APIs Referenced in crashes below:

int wrapOpenSession1(int two, char *host, char *localhost, int udpport, int one){
    return 1;
}

int wrapOpenSession5(char *host, char *localhost){
    return 1;
}

JNI Code (generated using SWIG):

package com.test.ewapi.jni;

public class Sample {

  public static int wrapOpenSession1(int two, String host, String localhost, int udpport, int one) {
    return SampleJNI.wrapOpenSession1(two, host, localhost, udpport, one);
  }


  public static int wrapOpenSession5(String host, String localhost) {
    return SampleJNI.wrapOpenSession5(host, localhost);
  }
}

package com.test.ewapi.jni;

public class SampleJNI {
  public final static native int wrapOpenSession1(int jarg1, String jarg2, String jarg3, int jarg4, int jarg5);
  public final static native int wrapOpenSession5(String jarg1, String jarg2);
}

JVM Crash w/o Xcheck:jin

    #

# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0054fc83, pid=912, tid=3428
#
# JRE version: 6.0_30-b12
# Java VM: Java HotSpot(TM) Client VM (20.5-b03 mixed mode, sharing windows-x86 )
# Problematic frame:
# C  0x0054fc83
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x01861c00):  JavaThread "main" [_thread_in_native, id=3428, stack(0x00500000,0x00550000)]

siginfo: ExceptionCode=0xc0000005, writing address 0xac338cd8

Registers:
EAX=0x01861d28, EBX=0x338d3eb8, ECX=0x7c34218f, EDX=0x0000000c
ESP=0x0054fc30, EBP=0x0054fc78, ESI=0x338d3eb8, EDI=0x01861c00
EIP=0x0054fc83, EFLAGS=0x00210287

Top of Stack: (sp=0x0054fc30)
0x0054fc30:   01861d28 0054fc70 0054fc7c 0054fc78
0x0054fc40:   0054fc74 6d90311e 0054fc48 338d3eb8
0x0054fc50:   0054fc7c 338d48b8 00000000 338d3eb8
0x0054fc60:   00000000 0054fc78 0054fca0 01962f45
0x0054fc70:   338d4858 019682d9 338d4bb0 338d4b80
0x0054fc80:   0054fc80 338cccaa 0054fcac 338cd988
0x0054fc90:   00000000 338cccb0 0054fc78 0054fca8
0x0054fca0:   0054fcd0 01962f45 338d4bb0 338d4b80 

Instructions: (pc=0x0054fc83)
0x0054fc63:   00 78 fc 54 00 a0 fc 54 00 45 2f 96 01 58 48 8d
0x0054fc73:   33 d9 82 96 01 b0 4b 8d 33 80 4b 8d 33 80 fc 54
0x0054fc83:   00 aa cc 8c 33 ac fc 54 00 88 d9 8c 33 00 00 00
0x0054fc93:   00 b0 cc 8c 33 78 fc 54 00 a8 fc 54 00 d0 fc 54 


Register to memory mapping:

EAX=0x01861d28 is an unknown value
EBX=0x338d3eb8 is an oop
{method} 
 - klass: {other class}
ECX=0x7c34218f is an unknown value
EDX=0x0000000c is an unknown value
ESP=0x0054fc30 is pointing into the stack for thread: 0x01861c00
EBP=0x0054fc78 is pointing into the stack for thread: 0x01861c00
ESI=0x338d3eb8 is an oop
{method} 
 - klass: {other class}
EDI=0x01861c00 is a thread


Stack: [0x00500000,0x00550000],  sp=0x0054fc30,  free space=319k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
C  0x0054fc83

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.test.jni.TestJNI.wrapOpenSession5(Ljava/lang/String;Ljava/lang/String;)I+0
j  com.test.jni.Test.wrapOpenSession5(Ljava/lang/String;Ljava/lang/String;)I+2
j  com.test.jni.Example.main([Ljava/lang/String;)V+773
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x018de000 JavaThread "Low Memory Detector" daemon [_thread_new, id=1208, stack(0x00000000,0x00000000)]
  0x018cec00 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=2412, stack(0x03c80000,0x03cd0000)]
  0x018cdc00 JavaThread "Attach Listener" daemon [_thread_blocked, id=1824, stack(0x03c30000,0x03c80000)]
  0x018ccc00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1848, stack(0x03be0000,0x03c30000)]
  0x018c7000 JavaThread "Finalizer" daemon [_thread_blocked, id=3224, stack(0x03b90000,0x03be0000)]
  0x018c2400 JavaThread "Reference Handler" daemon [_thread_blocked, id=3404, stack(0x03b40000,0x03b90000)]
=>0x01861c00 JavaThread "main" [_thread_in_native, id=3428, stack(0x00500000,0x00550000)]

Other Threads:
  0x01885c00 VMThread [stack: 0x03af0000,0x03b40000] [id=2320]
  0x018f8c00 WatcherThread [stack: 0x03d20000,0x03d70000] [id=1380]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation   total 4928K, used 482K [0x238b0000, 0x23e00000, 0x28e00000)
  eden space 4416K,  10% used [0x238b0000, 0x23928a30, 0x23d00000)
  from space 512K,   0% used [0x23d00000, 0x23d00000, 0x23d80000)
  to   space 512K,   0% used [0x23d80000, 0x23d80000, 0x23e00000)
 tenured generation   total 10944K, used 0K [0x28e00000, 0x298b0000, 0x338b0000)
   the space 10944K,   0% used [0x28e00000, 0x28e00000, 0x28e00200, 0x298b0000)
 compacting perm gen  total 12288K, used 146K [0x338b0000, 0x344b0000, 0x378b0000)
   the space 12288K,   1% used [0x338b0000, 0x338d4be0, 0x338d4c00, 0x344b0000)
    ro space 10240K,  51% used [0x378b0000, 0x37ddda30, 0x37dddc00, 0x382b0000)
    rw space 12288K,  55% used [0x382b0000, 0x38949b50, 0x38949c00, 0x38eb0000)

Code Cache  [0x01960000, 0x019d0000, 0x03960000)
 total_blobs=134 nmethods=3 adapters=68 free_code_cache=33104512 largest_free_block=0

Dynamic libraries:
0x00400000 - 0x00425000     C:\Windows\system32\java.exe
0x77b80000 - 0x77cbd000     C:\Windows\SYSTEM32\ntdll.dll
0x77950000 - 0x77a24000     C:\Windows\system32\kernel32.dll
0x75d50000 - 0x75d9a000     C:\Windows\system32\KERNELBASE.dll
0x77a30000 - 0x77ad0000     C:\Windows\system32\ADVAPI32.dll
0x77ad0000 - 0x77b7c000     C:\Windows\system32\msvcrt.dll
0x76fb0000 - 0x76fc9000     C:\Windows\SYSTEM32\sechost.dll
0x77d00000 - 0x77da1000     C:\Windows\system32\RPCRT4.dll
0x7c340000 - 0x7c396000     C:\Program Files\Java\jre6\bin\msvcr71.dll
0x6d7f0000 - 0x6da9f000     C:\Program Files\Java\jre6\bin\client\jvm.dll
0x77790000 - 0x77859000     C:\Windows\system32\USER32.dll
0x76e90000 - 0x76ede000     C:\Windows\system32\GDI32.dll
0x77120000 - 0x7712a000     C:\Windows\system32\LPK.dll
0x76c30000 - 0x76ccd000     C:\Windows\system32\USP10.dll
0x743d0000 - 0x74402000     C:\Windows\system32\WINMM.dll
0x77cc0000 - 0x77cdf000     C:\Windows\system32\IMM32.DLL
0x77130000 - 0x771fc000     C:\Windows\system32\MSCTF.dll
0x75bd0000 - 0x75c1b000     C:\Windows\system32\apphelp.dll
0x6d7a0000 - 0x6d7ac000     C:\Program Files\Java\jre6\bin\verify.dll
0x6d320000 - 0x6d33f000     C:\Program Files\Java\jre6\bin\java.dll
0x6d7e0000 - 0x6d7ef000     C:\Program Files\Java\jre6\bin\zip.dll
0x67140000 - 0x67224000     C:\jni\lib\CiscoEnergyWiseJni.dll
0x6cc80000 - 0x6cee7000     c:\jni\lib\CiscoEnergyWiseApi.dll
0x10000000 - 0x10113000     c:\jni\lib\libeay32.dll
0x76ee0000 - 0x76f15000     C:\Windows\system32\WS2_32.dll
0x76c20000 - 0x76c26000     C:\Windows\system32\NSI.dll
0x68430000 - 0x684d3000     C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4926_none_508ed732bcbc0e5a\MSVCR90.dll
0x77cf0000 - 0x77cf5000     C:\Windows\system32\PSAPI.DLL

VM Arguments:
java_command: CiscoEnergyWiseJni.jar -h 2.2.2.18 -l 10.211.55.3 -d colin -s lab
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\swig\swigwin-2.0.4;C:\cygwin\jdk1.6.0_30\bin;C:\cygwin;c:\jni\lib
USERNAME=cgswtsu78
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 37 Stepping 5, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows 7 Build 7600 

CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 37 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

Memory: 4k page, physical 1048120k(542096k free), swap 2096696k(1160996k free)

vm_info: Java HotSpot(TM) Client VM (20.5-b03) for windows-x86 JRE (1.6.0_30-b12), built on Nov 10 2011 01:51:00 by "java_re" with MS VC++ 7.1 (VS2003)

time: Sun Jan 08 21:29:01 2012
elapsed time: 0 seconds

Crash Report with -Xcheck:jni:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x6d896d32, pid=3552, tid=3264
#
# JRE version: 6.0_30-b12
# Java VM: Java HotSpot(TM) Client VM (20.5-b03 mixed mode, sharing windows-x86 )
# Problematic frame:
# V  [jvm.dll+0xa6d32]
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#

---------------  T H R E A D  ---------------

Current thread (0x01921c00):  JavaThread "main" [_thread_in_native, id=3264, stack(0x00170000,0x001c0000)]

siginfo: ExceptionCode=0xc0000005, reading address 0x0000001d

Registers:
EAX=0x00000021, EBX=0x001bfc48, ECX=0x382b0998, EDX=0x00000004
ESP=0x001bfbc0, EBP=0x001bfbcc, ESI=0x01921c00, EDI=0x0000001d
EIP=0x6d896d32, EFLAGS=0x00210206

Top of Stack: (sp=0x001bfbc0)
0x001bfbc0:   338d3c98 01921c00 338d3c98 001bfbe8
0x001bfbd0:   67145e4b 01921d28 001bfc48 00000021
0x001bfbe0:   0000000b 00000001 001bfc30 01a29fb7
0x001bfbf0:   01921d28 001bfc38 0000000b 001bfc4c
0x001bfc00:   001bfc48 00000016 00000021 01921c00
0x001bfc10:   001bfc10 338d3c98 001bfc50 338d48b8
0x001bfc20:   00000000 338d3c98 00000000 001bfc40
0x001bfc30:   001bfc74 01a22f45 338d4858 01a282d9 

Instructions: (pc=0x6d896d32)
0x6d896d12:   f5 ff 8b 45 10 83 c4 0c 85 c0 75 11 8b 15 a0 f2
0x6d896d22:   a6 6d 50 53 57 ff 92 a8 02 00 00 eb 2a 8d 78 fc
0x6d896d32:   81 3f 12 48 12 48 74 0d 68 18 80 a1 6d e8 fc bd
0x6d896d42:   ff ff 83 c4 04 8b 45 08 8b 0d a0 f2 a6 6d 57 53 


Register to memory mapping:

EAX=0x00000021 is an unknown value
EBX=0x001bfc48 is pointing into the stack for thread: 0x01921c00
ECX=0x382b0998 is an oop
{instance class} 
 - klass: {other class}
EDX=0x00000004 is an unknown value
ESP=0x001bfbc0 is pointing into the stack for thread: 0x01921c00
EBP=0x001bfbcc is pointing into the stack for thread: 0x01921c00
ESI=0x01921c00 is a thread
EDI=0x0000001d is an unknown value


Stack: [0x00170000,0x001c0000],  sp=0x001bfbc0,  free space=318k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [jvm.dll+0xa6d32]
C  [EWJni.dll+0x5e4b]  Java_com_test_ewapi_jni_SampleJNI_sample_1wrapOpenSession1+0x133
j  com.test.ewapi.jni.SampleJNI.energywise_wrapOpenSession1(ILjava/lang/String;Ljava/lang/String;II)I+0
j  com.test.ewapi.jni.Sample.wrapOpenSession1(ILjava/lang/String;Ljava/lang/String;II)I+6
j  com.test.ewapi.jni.Example.main([Ljava/lang/String;)V+667
v  ~StubRoutines::call_stub
V  [jvm.dll+0xfac3b]
V  [jvm.dll+0x18c3a1]
V  [jvm.dll+0xfacbd]
V  [jvm.dll+0x95776]
V  [jvm.dll+0x9d6b8]
V  [jvm.dll+0xaf16b]
C  [java.exe+0x2155]
C  [java.exe+0x85b4]
C  [kernel32.dll+0x51114]  BaseThreadInitThunk+0x12
C  [ntdll.dll+0x5b429]  RtlInitializeExceptionChain+0x63
C  [ntdll.dll+0x5b3fc]  RtlInitializeExceptionChain+0x36

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  com.test.ewapi.jni.SampleJNI.energywise_wrapOpenSession1(ILjava/lang/String;Ljava/lang/String;II)I+0
j  com.test.ewapi.jni.Sample.wrapOpenSession1(ILjava/lang/String;Ljava/lang/String;II)I+6
j  com.test.ewapi.jni.Example.main([Ljava/lang/String;)V+667
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x0199f400 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=140, stack(0x03d10000,0x03d60000)]
  0x0198fc00 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=432, stack(0x03cc0000,0x03d10000)]
  0x0198ec00 JavaThread "Attach Listener" daemon [_thread_blocked, id=3520, stack(0x03c70000,0x03cc0000)]
  0x0198e000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=2868, stack(0x03c20000,0x03c70000)]
  0x01987000 JavaThread "Finalizer" daemon [_thread_blocked, id=2340, stack(0x03bd0000,0x03c20000)]
  0x01982400 JavaThread "Reference Handler" daemon [_thread_blocked, id=2684, stack(0x03b80000,0x03bd0000)]
=>0x01921c00 JavaThread "main" [_thread_in_native, id=3264, stack(0x00170000,0x001c0000)]

Other Threads:
  0x01946000 VMThread [stack: 0x03b30000,0x03b80000] [id=1420]
  0x019b9c00 WatcherThread [stack: 0x03d60000,0x03db0000] [id=2336]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 def new generation   total 4928K, used 482K [0x238b0000, 0x23e00000, 0x28e00000)
  eden space 4416K,  10% used [0x238b0000, 0x23928a30, 0x23d00000)
  from space 512K,   0% used [0x23d00000, 0x23d00000, 0x23d80000)
  to   space 512K,   0% used [0x23d80000, 0x23d80000, 0x23e00000)
 tenured generation   total 10944K, used 0K [0x28e00000, 0x298b0000, 0x338b0000)
   the space 10944K,   0% used [0x28e00000, 0x28e00000, 0x28e00200, 0x298b0000)
 compacting perm gen  total 12288K, used 146K [0x338b0000, 0x344b0000, 0x378b0000)
   the space 12288K,   1% used [0x338b0000, 0x338d4aa0, 0x338d4c00, 0x344b0000)
    ro space 10240K,  51% used [0x378b0000, 0x37ddda30, 0x37dddc00, 0x382b0000)
    rw space 12288K,  55% used [0x382b0000, 0x38949b50, 0x38949c00, 0x38eb0000)

Code Cache  [0x01a20000, 0x01a90000, 0x03a20000)
 total_blobs=126 nmethods=3 adapters=68 free_code_cache=33106048 largest_free_block=0

Dynamic libraries:
0x00400000 - 0x00425000     C:\Windows\system32\java.exe
0x77b80000 - 0x77cbd000     C:\Windows\SYSTEM32\ntdll.dll
0x77950000 - 0x77a24000     C:\Windows\system32\kernel32.dll
0x75d50000 - 0x75d9a000     C:\Windows\system32\KERNELBASE.dll
0x77a30000 - 0x77ad0000     C:\Windows\system32\ADVAPI32.dll
0x77ad0000 - 0x77b7c000     C:\Windows\system32\msvcrt.dll
0x76fb0000 - 0x76fc9000     C:\Windows\SYSTEM32\sechost.dll
0x77d00000 - 0x77da1000     C:\Windows\system32\RPCRT4.dll
0x7c340000 - 0x7c396000     C:\Program Files\Java\jre6\bin\msvcr71.dll
0x6d7f0000 - 0x6da9f000     C:\Program Files\Java\jre6\bin\client\jvm.dll
0x77790000 - 0x77859000     C:\Windows\system32\USER32.dll
0x76e90000 - 0x76ede000     C:\Windows\system32\GDI32.dll
0x77120000 - 0x7712a000     C:\Windows\system32\LPK.dll
0x76c30000 - 0x76ccd000     C:\Windows\system32\USP10.dll
0x743d0000 - 0x74402000     C:\Windows\system32\WINMM.dll
0x77cc0000 - 0x77cdf000     C:\Windows\system32\IMM32.DLL
0x77130000 - 0x771fc000     C:\Windows\system32\MSCTF.dll
0x75bd0000 - 0x75c1b000     C:\Windows\system32\apphelp.dll
0x6d7a0000 - 0x6d7ac000     C:\Program Files\Java\jre6\bin\verify.dll
0x6d320000 - 0x6d33f000     C:\Program Files\Java\jre6\bin\java.dll
0x6d7e0000 - 0x6d7ef000     C:\Program Files\Java\jre6\bin\zip.dll
0x67140000 - 0x67224000     C:\jni\lib\EWJni.dll
0x6cc80000 - 0x6cee7000     c:\jni\lib\EWCApi.dll
0x10000000 - 0x10113000     c:\jni\lib\libeay32.dll
0x76ee0000 - 0x76f15000     C:\Windows\system32\WS2_32.dll
0x76c20000 - 0x76c26000     C:\Windows\system32\NSI.dll
0x6c050000 - 0x6c0f3000     C:\Windows\WinSxS\x86_microsoft.vc90.crt_1fc8b3b9a1e18e3b_9.0.30729.4926_none_508ed732bcbc0e5a\MSVCR90.dll
0x77cf0000 - 0x77cf5000     C:\Windows\system32\PSAPI.DLL

VM Arguments:
jvm_args: -Xcheck:jni 
java_command: EWJni.jar -h 2.2.2.18 -l 10.211.55.3 -d coling -s lab
Launcher Type: SUN_STANDARD

Environment Variables:
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\swig\swigwin-2.0.4;C:\cygwin\jdk1.6.0_30\bin;C:\cygwin;c:\jni\lib
USERNAME=cgswtsu78
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 37 Stepping 5, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows 7 Build 7600 

CPU:total 1 (1 cores per cpu, 1 threads per core) family 6 model 37 stepping 5, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt

Memory: 4k page, physical 1048120k(610804k free), swap 2096696k(1308820k free)

vm_info: Java HotSpot(TM) Client VM (20.5-b03) for windows-x86 JRE (1.6.0_30-b12), built on Nov 10 2011 01:51:00 by "java_re" with MS VC++ 7.1 (VS2003)

time: Mon Jan 09 10:29:45 2012
elapsed time: 0 seconds

SWIG Compilation Output:

 [exec] mkdir -p obj
 [exec] swig -java -I/test/java/jdk-1.6.0_30/include/linux -I/test/jni/include -I/test/java/jdk-1.6.0_30/include  -package com.test.ewapi.jni -outdir ../src/java/com/test/ewapi/jni ewapi.i
 [exec] /test/mingw/mingw32/bin/i386-mingw32-gcc  -Wall -fpic -I/test/java/jdk-1.6.0_30/include/linux -I/test/jni/include -I/test/java/jdk-1.6.0_30/include  -O0 -g3 -fno-strict-aliasing -c ewapi_wrap.c -o obj/ewapi_wrap.o
 [exec] ewapi_wrap.c:1: warning: -fpic ignored for target (all code is position independent)
 [exec] /test/mingw/mingw32/bin/i386-mingw32-gcc  obj/ewapi_wrap.o -shared -L/test/jni/lib -lEWApi -lm -leay32 -lws2_32 -lrpcrt4 -o /test/jni/lib/SampleJni.dll
 [exec] rm ewapi_wrap.c

Error when running Java main with Xcheck:jni JVM option:

FATAL ERROR in native method: ReleaseStringUTFChars called on something not allo
cated by GetStringUTFChars at com.test.ewapi.jni.SampleJNI.wrapOpenSession1
(Native Method) at com.test.ewapi.jni.Sample.wrapOpenSession1(Unknown So
urce) at com.test.ewapi.jni.Example.main(Unknown Source)
java
debugging
java-native-interface
asked on Stack Overflow Jan 9, 2012 by c12 • edited Jan 9, 2012 by c12

2 Answers

▲
1
▼

Apparently the C code dereferences NULL somewhere:

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x0054fc83, pid=912, tid=3428

The 0xc0000005 index suggests accessing 5th index of NULL pointer:

char* c = NULL;
//...
c[5] = 4  //fail
answered on Stack Overflow Jan 9, 2012 by Tomasz Nurkiewicz
▲
1
▼

One possibility is the problem discussed here.

tl;dr:

You absolutely, positively must use -fno-strict-aliasing if you're compiling SWIG-generated code with a recent version of g++. Not doing so will result in hard-to-diagnose hard crashes which have precisely this "random, but consistent" behavior.

answered on Stack Overflow Jan 9, 2012 by Ernest Friedman-Hill • edited May 23, 2017 by Community

User contributions licensed under CC BY-SA 3.0