solaris zfs pool import crash
I do run my nas on solaris 11.3. I had a system crash a few days ago. Reason unknown. Afterwards I wasn't able to import the pool anymore. I exported the pool already so that I can boot without a new crash. The pool isn't the rpool. But I do need the data. Otherwise 20 years of children growing are gone. I checked the backup but this stopped a few months ago due to space limits. So status zpool import shows a healthy pool
root@solaris:~# zpool import
pool: mediapool1
id: 8470162457149274931
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the '-f' flag.
see: http://support.oracle.com/msg/ZFS-8000-EY
config:
mediapool1 ONLINE
raidz1-0 ONLINE
c0t5000C500A2C3F5D9d0 ONLINE
c0t5000C500A2C40ECEd0 ONLINE
c0t5000C500A2C4122Bd0 ONLINE
but as soon as I do the zpool import mediapool I run into a kernel oops.
I exchanged the whole system. Sas controller, proz, mainboard, ram, Os (Solaris 11.4), powersupply. Nothing changed.
zdb -e -dddd
root@solaris:~# zdb -e -dddd mediapool1
Dataset mediapool1 [ZPL], ID 18, cr_txg 1, 224K, 10 objects, rootbp DVA[0]=<0:903a3a5c000:4000:RZM:3> [L0 DMU objset] fletcher4 lzjb LE unique unencrypted size=800L/200P birth=10836736L/10836736P fill=10 contiguous 3-copy cksum=1416247ffb:6ec9c299cf2:1423227024d05:28fa8c22fe2105
Deadlist: 0 (0/0 comp)
mintxg 0 -> obj 21
Object lvl iblk dblk dsize lsize %full type
0 7 16K 16K 74.5K 16K 31.25 DMU dnode
dnode flags: USED_BYTES
dnode maxblkid: 0
Object lvl iblk dblk dsize lsize %full type
-1 1 16K 512 10.5K 512 100.00 ZFS user/group used
dnode flags: USED_BYTES
dnode maxblkid: 0
microzap: 512 bytes, 1 entries
0 = 0xdf20
Object lvl iblk dblk dsize lsize %full type
-2 1 16K 512 10.5K 512 100.00 ZFS user/group used
dnode flags: USED_BYTES
dnode maxblkid: 0
microzap: 512 bytes, 1 entries
0 = 0xdf20
Object lvl iblk dblk dsize lsize %full type
1 1 16K 512 10.5K 512 100.00 ZFS master node
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 512 bytes, 7 entries
ROOT = 0x4
SA_ATTRS = 0x2
casesensitivity = 0x2
VERSION = 0x6
DELETE_QUEUE = 0x3
SHARES = 0x7
normalization = 0
Object lvl iblk dblk dsize lsize %full type
2 1 16K 512 10.5K 512 100.00 SA master node
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 512 bytes, 2 entries
LAYOUTS = 0x6
REGISTRY = 0x5
Object lvl iblk dblk dsize lsize %full type
3 1 16K 512 10.5K 512 100.00 ZFS delete queue
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
4 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /
uid 0
gid 0
atime Tue Jun 20 09:39:20 2017
mtime Sat Jun 17 14:02:19 2017
ctime Sat Jun 17 14:02:19 2017
crtime Wed Jun 14 08:43:52 2017
gen 4
mode 040755
size 5
parent 4
links 5
pflags 0x40800000344
microzap: 512 bytes, 3 entries
backup = 11 (type: Directory)
fs_userhome = 10 (type: Directory)
export = 12 (type: Directory)
Object lvl iblk dblk dsize lsize %full type
5 1 16K 1.50K 10.5K 1.50K 100.00 SA attr registration
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 1536 bytes, 21 entries
ZPL_ZNODE_ACL = 0x5803000f : [88:3:15]
ZPL_RDEV = 0x800000a : [8:0:10]
ZPL_LINKS = 0x8000008 : [8:0:8]
ZPL_ATIME = 0x10000000 : [16:0:0]
ZPL_GID = 0x800000d : [8:0:13]
ZPL_PAD = 0x2000000e : [32:0:14]
ZPL_SCANSTAMP = 0x20030012 : [32:3:18]
ZPL_SIZE = 0x8000006 : [8:0:6]
ZPL_UID = 0x800000c : [8:0:12]
ZPL_CTIME = 0x10000002 : [16:0:2]
ZPL_MAC_LABEL = 0x30014 : [0:3:20]
ZPL_MTIME = 0x10000001 : [16:0:1]
ZPL_MODE = 0x8000005 : [8:0:5]
ZPL_FLAGS = 0x800000b : [8:0:11]
ZPL_PARENT = 0x8000007 : [8:0:7]
ZPL_DACL_ACES = 0x40013 : [0:4:19]
ZPL_CRTIME = 0x10000003 : [16:0:3]
ZPL_SYMLINK = 0x30011 : [0:3:17]
ZPL_GEN = 0x8000004 : [8:0:4]
ZPL_DACL_COUNT = 0x8000010 : [8:0:16]
ZPL_XATTR = 0x8000009 : [8:0:9]
Object lvl iblk dblk dsize lsize %full type
6 1 16K 16K 21.5K 32K 100.00 SA attr layouts
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 1
Fat ZAP stats:
Pointer table:
1024 elements
zt_blk: 0
zt_numblks: 0
zt_shift: 10
zt_blks_copied: 0
zt_nextblk: 0
ZAP entries: 1
Leaf blocks: 1
Total blocks: 2
zap_block_type: 0x8000000000000001
zap_magic: 0x2f52ab2ab
zap_salt: 0x32d772cd
Leafs with 2^n pointers:
9: 1 *
Blocks with n*5 entries:
0: 1 *
Blocks n/10 full:
1: 1 *
Entries with n chunks:
4: 1 *
Buckets with n entries:
0: 511 ****************************************
1: 1 *
2 = [ 5 6 4 12 13 7 11 0 1 2 3 8 16 19 ]
Object lvl iblk dblk dsize lsize %full type
7 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /.zfs/shares
uid 0
gid 0
atime Wed Jun 14 08:43:53 2017
mtime Wed Jun 14 08:43:52 2017
ctime Wed Jun 14 08:43:52 2017
crtime Wed Jun 14 08:43:52 2017
gen 4
mode 040555
size 2
parent 7
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
10 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /fs_userhome
uid 0
gid 0
atime Fri Dec 21 06:21:30 2018
mtime Fri Jun 16 13:07:27 2017
ctime Fri Jun 16 13:07:27 2017
crtime Fri Jun 16 13:07:27 2017
gen 20775
mode 040755
size 2
parent 4
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
11 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /backup
uid 0
gid 0
atime Fri Dec 21 06:21:30 2018
mtime Sat Jun 17 13:47:04 2017
ctime Sat Jun 17 13:47:04 2017
crtime Sat Jun 17 13:47:04 2017
gen 39220
mode 040755
size 2
parent 4
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
12 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /export
uid 0
gid 0
atime Tue Jan 29 06:46:27 2019
mtime Sat Jun 17 14:02:19 2017
ctime Sat Jun 17 14:02:19 2017
crtime Sat Jun 17 14:02:19 2017
gen 39413
mode 040755
size 2
parent 4
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
root@solaris:~# zdb -e -dddd mediapool1/export
Dataset mediapool1/export [ZPL], ID 879, cr_txg 37449, 202K, 8 objects, rootbp DVA[0]=<0:903a39cc000:4000:RZM:3> [L0 DMU objset] fletcher4 lzjb LE unique unencrypted size=800L/200P birth=10836735L/10836735P fill=8 contiguous 3-copy cksum=1695577b35:7cf5a5fbca5:16bf5de66c79d:2e4ab1d9943912
Deadlist: 107K (9.00K/9.00K comp)
mintxg 0 -> obj 217
mintxg 1 -> obj 218
mintxg 37455 -> obj 219
Object lvl iblk dblk dsize lsize %full type
0 7 16K 16K 74.5K 16K 25.00 DMU dnode
dnode flags: USED_BYTES
dnode maxblkid: 0
Object lvl iblk dblk dsize lsize %full type
-1 1 16K 512 10.5K 512 100.00 ZFS user/group used
dnode flags: USED_BYTES
dnode maxblkid: 0
microzap: 512 bytes, 1 entries
0 = 0x85e0
Object lvl iblk dblk dsize lsize %full type
-2 1 16K 512 10.5K 512 100.00 ZFS user/group used
dnode flags: USED_BYTES
dnode maxblkid: 0
microzap: 512 bytes, 1 entries
0 = 0x85e0
Object lvl iblk dblk dsize lsize %full type
1 1 16K 1K 10.5K 1K 100.00 ZFS master node
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 1024 bytes, 8 entries
SA_ATTRS = 0x2
DELETE_QUEUE = 0x3
SHARES = 0x7
casesensitivity = 0x2
normalization = 0
VERSION = 0x5
utf8only = 0
ROOT = 0x4
Object lvl iblk dblk dsize lsize %full type
2 1 16K 512 10.5K 512 100.00 SA master node
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 512 bytes, 2 entries
REGISTRY = 0x5
LAYOUTS = 0x6
Object lvl iblk dblk dsize lsize %full type
3 1 16K 512 10.5K 512 100.00 ZFS delete queue
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
4 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /
uid 0
gid 0
atime Tue Jan 29 06:46:27 2019
mtime Wed May 20 16:50:07 2015
ctime Wed May 20 16:50:07 2015
crtime Wed Mar 13 21:27:28 2013
gen 65199
mode 040755
size 3
parent 4
links 3
pflags 0x40800000344
microzap: 512 bytes, 1 entries
home = 8 (type: Directory)
Object lvl iblk dblk dsize lsize %full type
5 1 16K 1.50K 10.5K 1.50K 100.00 SA attr registration
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
microzap: 1536 bytes, 21 entries
ZPL_GID = 0x800000d : [8:0:13]
ZPL_DACL_ACES = 0x40013 : [0:4:19]
ZPL_CRTIME = 0x10000003 : [16:0:3]
ZPL_MAC_LABEL = 0x30014 : [0:3:20]
ZPL_ATIME = 0x10000000 : [16:0:0]
ZPL_SIZE = 0x8000006 : [8:0:6]
ZPL_LINKS = 0x8000008 : [8:0:8]
ZPL_PAD = 0x2000000e : [32:0:14]
ZPL_PARENT = 0x8000007 : [8:0:7]
ZPL_MODE = 0x8000005 : [8:0:5]
ZPL_DACL_COUNT = 0x8000010 : [8:0:16]
ZPL_SYMLINK = 0x30011 : [0:3:17]
ZPL_XATTR = 0x8000009 : [8:0:9]
ZPL_SCANSTAMP = 0x20030012 : [32:3:18]
ZPL_UID = 0x800000c : [8:0:12]
ZPL_GEN = 0x8000004 : [8:0:4]
ZPL_RDEV = 0x800000a : [8:0:10]
ZPL_FLAGS = 0x800000b : [8:0:11]
ZPL_ZNODE_ACL = 0x5803000f : [88:3:15]
ZPL_CTIME = 0x10000002 : [16:0:2]
ZPL_MTIME = 0x10000001 : [16:0:1]
Object lvl iblk dblk dsize lsize %full type
6 1 16K 16K 21.5K 32K 100.00 SA attr layouts
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 1
Fat ZAP stats:
Pointer table:
1024 elements
zt_blk: 0
zt_numblks: 0
zt_shift: 10
zt_blks_copied: 0
zt_nextblk: 0
ZAP entries: 1
Leaf blocks: 1
Total blocks: 2
zap_block_type: 0x8000000000000001
zap_magic: 0x2f52ab2ab
zap_salt: 0x751fd0cd
Leafs with 2^n pointers:
9: 1 *
Blocks with n*5 entries:
0: 1 *
Blocks n/10 full:
1: 1 *
Entries with n chunks:
4: 1 *
Buckets with n entries:
0: 511 ****************************************
1: 1 *
2 = [ 5 6 4 12 13 7 11 0 1 2 3 8 16 19 ]
Object lvl iblk dblk dsize lsize %full type
7 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /.zfs/shares
uid 0
gid 0
atime Wed Mar 4 06:30:06 2015
mtime Wed Mar 13 21:27:28 2013
ctime Wed Mar 13 21:27:28 2013
crtime Wed Mar 13 21:27:28 2013
gen 65199
mode 040555
size 2
parent 7
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
Object lvl iblk dblk dsize lsize %full type
8 1 16K 512 10.5K 512 100.00 ZFS directory
168 bonus System attributes
dnode flags: USED_BYTES USERUSED_ACCOUNTED
dnode maxblkid: 0
path /home
uid 0
gid 0
atime Tue Jan 29 06:46:27 2019
mtime Wed Mar 13 21:27:32 2013
ctime Wed Mar 13 21:27:32 2013
crtime Wed Mar 13 21:27:32 2013
gen 65206
mode 040755
size 2
parent 4
links 2
pflags 0x40800000344
microzap: 512 bytes, 0 entries
root@solaris:~#
from my understanding ZFS should have some backup data available. How to import these backup ?
stackelk0 Answers
Nobody has answered this question yet.
User contributions licensed under CC BY-SA 3.0