执行sudo /home/test/caffe-master/build/tools/caffe train --solver=/media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/solver.prototxt -weights /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel --log_dir=/media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/log --gpu=0时候出现如下错误：
I1113 13:08:56.731135 7806 layer_factory.hpp:77] Creating layer data
I1113 13:08:56.731192 7806 net.cpp:100] Creating Layer data
I1113 13:08:56.731199 7806 net.cpp:408] data -> data
I1113 13:08:56.731207 7806 net.cpp:408] data -> label
I1113 13:08:56.731215 7806 data_transformer.cpp:25] Loading mean file from: /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/BP4D_Test/test_mean.binaryproto
I1113 13:08:56.756274 7813 db_lmdb.cpp:35] Opened lmdb /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/BP4D_Test/img_test_lmdb
I1113 13:08:56.769150 7806 data_layer.cpp:41] output data size: 8,3,227,227
I1113 13:08:56.778970 7806 net.cpp:150] Setting up data
I1113 13:08:56.778998 7806 net.cpp:157] Top shape: 8 3 227 227 (1236696)
I1113 13:08:56.779005 7806 net.cpp:157] Top shape: 8 (8)
I1113 13:08:56.779008 7806 net.cpp:165] Memory required for data: 4946816
I1113 13:08:56.779022 7806 layer_factory.hpp:77] Creating layer label_data_1_split
I1113 13:08:56.779036 7806 net.cpp:100] Creating Layer label_data_1_split
I1113 13:08:56.779042 7806 net.cpp:434] label_data_1_split
I1113 13:08:56.779050 7806 net.cpp:408] label_data_1_split -> label_data_1_split_0
I1113 13:08:56.779062 7806 net.cpp:408] label_data_1_split -> label_data_1_split_1
I1113 13:08:56.779124 7806 net.cpp:150] Setting up label_data_1_split
I1113 13:08:56.779132 7806 net.cpp:157] Top shape: 8 (8)
I1113 13:08:56.779139 7806 net.cpp:157] Top shape: 8 (8)
I1113 13:08:56.779141 7806 net.cpp:165] Memory required for data: 4946880
I1113 13:08:56.779145 7806 layer_factory.hpp:77] Creating layer conv1
I1113 13:08:56.779160 7806 net.cpp:100] Creating Layer conv1
I1113 13:08:56.779163 7806 net.cpp:434] conv1
I1113 13:08:56.779170 7806 net.cpp:408] conv1 -> conv1
I1113 13:08:56.789156 7806 net.cpp:150] Setting up conv1
I1113 13:08:56.789198 7806 net.cpp:157] Top shape: 8 96 55 55 (2323200)
I1113 13:08:56.789208 7806 net.cpp:165] Memory required for data: 14239680
I1113 13:08:56.789232 7806 layer_factory.hpp:77] Creating layer relu1
I1113 13:08:56.789250 7806 net.cpp:100] Creating Layer relu1
I1113 13:08:56.789258 7806 net.cpp:434] relu1
I1113 13:08:56.789266 7806 net.cpp:395] relu1 -> conv1 (in-place)
I1113 13:08:56.789422 7806 net.cpp:150] Setting up relu1
I1113 13:08:56.789433 7806 net.cpp:157] Top shape: 8 96 55 55 (2323200)
I1113 13:08:56.789435 7806 net.cpp:165] Memory required for data: 23532480
I1113 13:08:56.789439 7806 layer_factory.hpp:77] Creating layer pool1
I1113 13:08:56.789448 7806 net.cpp:100] Creating Layer pool1
I1113 13:08:56.789451 7806 net.cpp:434] pool1
I1113 13:08:56.789458 7806 net.cpp:408] pool1 -> pool1
I1113 13:08:56.789494 7806 net.cpp:150] Setting up pool1
I1113 13:08:56.789500 7806 net.cpp:157] Top shape: 8 96 27 27 (559872)
I1113 13:08:56.789504 7806 net.cpp:165] Memory required for data: 25771968
I1113 13:08:56.789506 7806 layer_factory.hpp:77] Creating layer norm1
I1113 13:08:56.789513 7806 net.cpp:100] Creating Layer norm1
I1113 13:08:56.789516 7806 net.cpp:434] norm1
I1113 13:08:56.789522 7806 net.cpp:408] norm1 -> norm1
I1113 13:08:56.790541 7806 net.cpp:150] Setting up norm1
I1113 13:08:56.790552 7806 net.cpp:157] Top shape: 8 96 27 27 (559872)
I1113 13:08:56.790556 7806 net.cpp:165] Memory required for data: 28011456
I1113 13:08:56.790560 7806 layer_factory.hpp:77] Creating layer conv2
I1113 13:08:56.790571 7806 net.cpp:100] Creating Layer conv2
I1113 13:08:56.790575 7806 net.cpp:434] conv2
I1113 13:08:56.790582 7806 net.cpp:408] conv2 -> conv2
I1113 13:08:56.795622 7806 net.cpp:150] Setting up conv2
I1113 13:08:56.795636 7806 net.cpp:157] Top shape: 8 256 27 27 (1492992)
I1113 13:08:56.795656 7806 net.cpp:165] Memory required for data: 33983424
I1113 13:08:56.795666 7806 layer_factory.hpp:77] Creating layer relu2
I1113 13:08:56.795673 7806 net.cpp:100] Creating Layer relu2
I1113 13:08:56.795677 7806 net.cpp:434] relu2
I1113 13:08:56.795683 7806 net.cpp:395] relu2 -> conv2 (in-place)
I1113 13:08:56.795811 7806 net.cpp:150] Setting up relu2
I1113 13:08:56.795819 7806 net.cpp:157] Top shape: 8 256 27 27 (1492992)
I1113 13:08:56.795824 7806 net.cpp:165] Memory required for data: 39955392
I1113 13:08:56.795826 7806 layer_factory.hpp:77] Creating layer pool2
I1113 13:08:56.795835 7806 net.cpp:100] Creating Layer pool2
I1113 13:08:56.795837 7806 net.cpp:434] pool2
I1113 13:08:56.795843 7806 net.cpp:408] pool2 -> pool2
I1113 13:08:56.795881 7806 net.cpp:150] Setting up pool2
I1113 13:08:56.795886 7806 net.cpp:157] Top shape: 8 256 13 13 (346112)
I1113 13:08:56.795889 7806 net.cpp:165] Memory required for data: 41339840
I1113 13:08:56.795893 7806 layer_factory.hpp:77] Creating layer norm2
I1113 13:08:56.795899 7806 net.cpp:100] Creating Layer norm2
I1113 13:08:56.795902 7806 net.cpp:434] norm2
I1113 13:08:56.795907 7806 net.cpp:408] norm2 -> norm2
I1113 13:08:56.796339 7806 net.cpp:150] Setting up norm2
I1113 13:08:56.796350 7806 net.cpp:157] Top shape: 8 256 13 13 (346112)
I1113 13:08:56.796353 7806 net.cpp:165] Memory required for data: 42724288
I1113 13:08:56.796357 7806 layer_factory.hpp:77] Creating layer conv3
I1113 13:08:56.796366 7806 net.cpp:100] Creating Layer conv3
I1113 13:08:56.796370 7806 net.cpp:434] conv3
I1113 13:08:56.796376 7806 net.cpp:408] conv3 -> conv3
I1113 13:08:56.807670 7806 net.cpp:150] Setting up conv3
I1113 13:08:56.807696 7806 net.cpp:157] Top shape: 8 384 13 13 (519168)
I1113 13:08:56.807700 7806 net.cpp:165] Memory required for data: 44800960
I1113 13:08:56.807713 7806 layer_factory.hpp:77] Creating layer relu3
I1113 13:08:56.807721 7806 net.cpp:100] Creating Layer relu3
I1113 13:08:56.807726 7806 net.cpp:434] relu3
I1113 13:08:56.807734 7806 net.cpp:395] relu3 -> conv3 (in-place)
I1113 13:08:56.808130 7806 net.cpp:150] Setting up relu3
I1113 13:08:56.808140 7806 net.cpp:157] Top shape: 8 384 13 13 (519168)
I1113 13:08:56.808145 7806 net.cpp:165] Memory required for data: 46877632
I1113 13:08:56.808148 7806 layer_factory.hpp:77] Creating layer conv4
I1113 13:08:56.808158 7806 net.cpp:100] Creating Layer conv4
I1113 13:08:56.808162 7806 net.cpp:434] conv4
I1113 13:08:56.808169 7806 net.cpp:408] conv4 -> conv4
I1113 13:08:56.817498 7806 net.cpp:150] Setting up conv4
I1113 13:08:56.817559 7806 net.cpp:157] Top shape: 8 384 13 13 (519168)
I1113 13:08:56.817566 7806 net.cpp:165] Memory required for data: 48954304
I1113 13:08:56.817586 7806 layer_factory.hpp:77] Creating layer relu4
I1113 13:08:56.817607 7806 net.cpp:100] Creating Layer relu4
I1113 13:08:56.817615 7806 net.cpp:434] relu4
I1113 13:08:56.817628 7806 net.cpp:395] relu4 -> conv4 (in-place)
I1113 13:08:56.817857 7806 net.cpp:150] Setting up relu4
I1113 13:08:56.817873 7806 net.cpp:157] Top shape: 8 384 13 13 (519168)
I1113 13:08:56.817879 7806 net.cpp:165] Memory required for data: 51030976
I1113 13:08:56.817885 7806 layer_factory.hpp:77] Creating layer conv5
I1113 13:08:56.817903 7806 net.cpp:100] Creating Layer conv5
I1113 13:08:56.817909 7806 net.cpp:434] conv5
I1113 13:08:56.817919 7806 net.cpp:408] conv5 -> conv5
I1113 13:08:56.826001 7806 net.cpp:150] Setting up conv5
I1113 13:08:56.826032 7806 net.cpp:157] Top shape: 8 256 13 13 (346112)
I1113 13:08:56.826038 7806 net.cpp:165] Memory required for data: 52415424
I1113 13:08:56.826058 7806 layer_factory.hpp:77] Creating layer relu5
I1113 13:08:56.826071 7806 net.cpp:100] Creating Layer relu5
I1113 13:08:56.826078 7806 net.cpp:434] relu5
I1113 13:08:56.826092 7806 net.cpp:395] relu5 -> conv5 (in-place)
I1113 13:08:56.826283 7806 net.cpp:150] Setting up relu5
I1113 13:08:56.826297 7806 net.cpp:157] Top shape: 8 256 13 13 (346112)
I1113 13:08:56.826320 7806 net.cpp:165] Memory required for data: 53799872
I1113 13:08:56.826328 7806 layer_factory.hpp:77] Creating layer pool5
I1113 13:08:56.826339 7806 net.cpp:100] Creating Layer pool5
I1113 13:08:56.826345 7806 net.cpp:434] pool5
I1113 13:08:56.826354 7806 net.cpp:408] pool5 -> pool5
I1113 13:08:56.826413 7806 net.cpp:150] Setting up pool5
I1113 13:08:56.826422 7806 net.cpp:157] Top shape: 8 256 6 6 (73728)
I1113 13:08:56.826427 7806 net.cpp:165] Memory required for data: 54094784
I1113 13:08:56.826431 7806 layer_factory.hpp:77] Creating layer fc6
I1113 13:08:56.826443 7806 net.cpp:100] Creating Layer fc6
I1113 13:08:56.826448 7806 net.cpp:434] fc6
I1113 13:08:56.826457 7806 net.cpp:408] fc6 -> fc6
I1113 13:08:57.163763 7806 net.cpp:150] Setting up fc6
I1113 13:08:57.163799 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.163805 7806 net.cpp:165] Memory required for data: 54225856
I1113 13:08:57.163817 7806 layer_factory.hpp:77] Creating layer relu6
I1113 13:08:57.163830 7806 net.cpp:100] Creating Layer relu6
I1113 13:08:57.163836 7806 net.cpp:434] relu6
I1113 13:08:57.163846 7806 net.cpp:395] relu6 -> fc6 (in-place)
I1113 13:08:57.164361 7806 net.cpp:150] Setting up relu6
I1113 13:08:57.164371 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.164376 7806 net.cpp:165] Memory required for data: 54356928
I1113 13:08:57.164382 7806 layer_factory.hpp:77] Creating layer drop6
I1113 13:08:57.164391 7806 net.cpp:100] Creating Layer drop6
I1113 13:08:57.164398 7806 net.cpp:434] drop6
I1113 13:08:57.164407 7806 net.cpp:395] drop6 -> fc6 (in-place)
I1113 13:08:57.164439 7806 net.cpp:150] Setting up drop6
I1113 13:08:57.164446 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.164450 7806 net.cpp:165] Memory required for data: 54488000
I1113 13:08:57.164456 7806 layer_factory.hpp:77] Creating layer fc7
I1113 13:08:57.164466 7806 net.cpp:100] Creating Layer fc7
I1113 13:08:57.164471 7806 net.cpp:434] fc7
I1113 13:08:57.164479 7806 net.cpp:408] fc7 -> fc7
I1113 13:08:57.310674 7806 net.cpp:150] Setting up fc7
I1113 13:08:57.310706 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.310711 7806 net.cpp:165] Memory required for data: 54619072
I1113 13:08:57.310725 7806 layer_factory.hpp:77] Creating layer relu7
I1113 13:08:57.310736 7806 net.cpp:100] Creating Layer relu7
I1113 13:08:57.310744 7806 net.cpp:434] relu7
I1113 13:08:57.310752 7806 net.cpp:395] relu7 -> fc7 (in-place)
I1113 13:08:57.310950 7806 net.cpp:150] Setting up relu7
I1113 13:08:57.310958 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.310963 7806 net.cpp:165] Memory required for data: 54750144
I1113 13:08:57.310968 7806 layer_factory.hpp:77] Creating layer drop7
I1113 13:08:57.310978 7806 net.cpp:100] Creating Layer drop7
I1113 13:08:57.310984 7806 net.cpp:434] drop7
I1113 13:08:57.310992 7806 net.cpp:395] drop7 -> fc7 (in-place)
I1113 13:08:57.311024 7806 net.cpp:150] Setting up drop7
I1113 13:08:57.311031 7806 net.cpp:157] Top shape: 8 4096 (32768)
I1113 13:08:57.311035 7806 net.cpp:165] Memory required for data: 54881216
I1113 13:08:57.311041 7806 layer_factory.hpp:77] Creating layer fc8sjm
I1113 13:08:57.311058 7806 net.cpp:100] Creating Layer fc8sjm
I1113 13:08:57.311064 7806 net.cpp:434] fc8sjm
I1113 13:08:57.311072 7806 net.cpp:408] fc8sjm -> fc8sjm
I1113 13:08:57.311420 7806 net.cpp:150] Setting up fc8sjm
I1113 13:08:57.311429 7806 net.cpp:157] Top shape: 8 8 (64)
I1113 13:08:57.311434 7806 net.cpp:165] Memory required for data: 54881472
I1113 13:08:57.311441 7806 layer_factory.hpp:77] Creating layer fc8sjm_fc8sjm_0_split
I1113 13:08:57.311450 7806 net.cpp:100] Creating Layer fc8sjm_fc8sjm_0_split
I1113 13:08:57.311456 7806 net.cpp:434] fc8sjm_fc8sjm_0_split
I1113 13:08:57.311465 7806 net.cpp:408] fc8sjm_fc8sjm_0_split -> fc8sjm_fc8sjm_0_split_0
I1113 13:08:57.311475 7806 net.cpp:408] fc8sjm_fc8sjm_0_split -> fc8sjm_fc8sjm_0_split_1
I1113 13:08:57.311535 7806 net.cpp:150] Setting up fc8sjm_fc8sjm_0_split
I1113 13:08:57.311542 7806 net.cpp:157] Top shape: 8 8 (64)
I1113 13:08:57.311548 7806 net.cpp:157] Top shape: 8 8 (64)
I1113 13:08:57.311553 7806 net.cpp:165] Memory required for data: 54881984
I1113 13:08:57.311558 7806 layer_factory.hpp:77] Creating layer accuracy
I1113 13:08:57.311575 7806 net.cpp:100] Creating Layer accuracy
I1113 13:08:57.311579 7806 net.cpp:434] accuracy
I1113 13:08:57.311586 7806 net.cpp:434] accuracy
I1113 13:08:57.311594 7806 net.cpp:408] accuracy -> accuracy
I1113 13:08:57.311605 7806 net.cpp:150] Setting up accuracy
I1113 13:08:57.311612 7806 net.cpp:157] Top shape: (1)
I1113 13:08:57.311617 7806 net.cpp:165] Memory required for data: 54881988
I1113 13:08:57.311622 7806 layer_factory.hpp:77] Creating layer loss
I1113 13:08:57.311630 7806 net.cpp:100] Creating Layer loss
I1113 13:08:57.311635 7806 net.cpp:434] loss
I1113 13:08:57.311641 7806 net.cpp:434] loss
I1113 13:08:57.311650 7806 net.cpp:408] loss -> loss
I1113 13:08:57.311661 7806 layer_factory.hpp:77] Creating layer loss
I1113 13:08:57.312157 7806 net.cpp:150] Setting up loss
I1113 13:08:57.312168 7806 net.cpp:157] Top shape: (1)
I1113 13:08:57.312173 7806 net.cpp:160] with loss weight 1
I1113 13:08:57.312188 7806 net.cpp:165] Memory required for data: 54881992
I1113 13:08:57.312194 7806 net.cpp:226] loss needs backward computation.
I1113 13:08:57.312201 7806 net.cpp:228] accuracy does not need backward computation.
I1113 13:08:57.312208 7806 net.cpp:226] fc8sjm_fc8sjm_0_split needs backward computation.
I1113 13:08:57.312213 7806 net.cpp:226] fc8sjm needs backward computation.
I1113 13:08:57.312219 7806 net.cpp:226] drop7 needs backward computation.
I1113 13:08:57.312224 7806 net.cpp:226] relu7 needs backward computation.
I1113 13:08:57.312229 7806 net.cpp:226] fc7 needs backward computation.
I1113 13:08:57.312234 7806 net.cpp:226] drop6 needs backward computation.
I1113 13:08:57.312239 7806 net.cpp:226] relu6 needs backward computation.
I1113 13:08:57.312244 7806 net.cpp:226] fc6 needs backward computation.
I1113 13:08:57.312249 7806 net.cpp:226] pool5 needs backward computation.
I1113 13:08:57.312254 7806 net.cpp:226] relu5 needs backward computation.
I1113 13:08:57.312259 7806 net.cpp:226] conv5 needs backward computation.
I1113 13:08:57.312265 7806 net.cpp:226] relu4 needs backward computation.
I1113 13:08:57.312270 7806 net.cpp:226] conv4 needs backward computation.
I1113 13:08:57.312275 7806 net.cpp:226] relu3 needs backward computation.
I1113 13:08:57.312281 7806 net.cpp:226] conv3 needs backward computation.
I1113 13:08:57.312286 7806 net.cpp:226] norm2 needs backward computation.
I1113 13:08:57.312291 7806 net.cpp:226] pool2 needs backward computation.
I1113 13:08:57.312297 7806 net.cpp:226] relu2 needs backward computation.
I1113 13:08:57.312302 7806 net.cpp:226] conv2 needs backward computation.
I1113 13:08:57.312307 7806 net.cpp:226] norm1 needs backward computation.
I1113 13:08:57.312314 7806 net.cpp:226] pool1 needs backward computation.
I1113 13:08:57.312319 7806 net.cpp:226] relu1 needs backward computation.
I1113 13:08:57.312324 7806 net.cpp:226] conv1 needs backward computation.
I1113 13:08:57.312330 7806 net.cpp:228] label_data_1_split does not need backward computation.
I1113 13:08:57.312335 7806 net.cpp:228] data does not need backward computation.
I1113 13:08:57.312340 7806 net.cpp:270] This network produces output accuracy
I1113 13:08:57.312346 7806 net.cpp:270] This network produces output loss
I1113 13:08:57.312367 7806 net.cpp:283] Network initialization done.
I1113 13:08:57.312443 7806 solver.cpp:60] Solver scaffolding done.
I1113 13:08:57.312911 7806 caffe.cpp:155] Finetuning from /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel
I1113 13:09:06.313233 7806 upgrade_proto.cpp:43] Attempting to upgrade input file specified using deprecated transformation parameters: /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel
I1113 13:09:06.313297 7806 upgrade_proto.cpp:46] Successfully upgraded file specified using deprecated data transformation parameters.
W1113 13:09:06.313309 7806 upgrade_proto.cpp:48] Note that future Caffe releases will only support transform_param messages for transformation fields.
I1113 13:09:06.313415 7806 upgrade_proto.cpp:52] Attempting to upgrade input file specified using deprecated V1LayerParameter: /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel
I1113 13:09:34.474565 7806 upgrade_proto.cpp:60] Successfully upgraded file specified using deprecated V1LayerParameter
I1113 13:09:34.542678 7806 net.cpp:761] Ignoring source layer fc8
I1113 13:09:46.065749 7806 upgrade_proto.cpp:43] Attempting to upgrade input file specified using deprecated transformation parameters: /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel
I1113 13:09:46.065778 7806 upgrade_proto.cpp:46] Successfully upgraded file specified using deprecated data transformation parameters.
W1113 13:09:46.065780 7806 upgrade_proto.cpp:48] Note that future Caffe releases will only support transform_param messages for transformation fields.
I1113 13:09:46.065804 7806 upgrade_proto.cpp:52] Attempting to upgrade input file specified using deprecated V1LayerParameter: /media/test/810a7433-6292-4fef-8e87-09cc6cc4b9e5/myfile1/bvlc_reference_caffenet.caffemodel
I1113 13:09:50.075256 7806 upgrade_proto.cpp:60] Successfully upgraded file specified using deprecated V1LayerParameter
I1113 13:09:50.144877 7806 net.cpp:761] Ignoring source layer fc8
I1113 13:09:50.161957 7806 caffe.cpp:251] Starting Optimization
I1113 13:09:50.161983 7806 solver.cpp:279] Solving CaffeNet
I1113 13:09:50.161985 7806 solver.cpp:280] Learning Rate Policy: step
I1113 13:09:50.164475 7806 solver.cpp:337] Iteration 0, Testing net (#0)
I1113 13:09:52.561529 7806 blocking_queue.cpp:50] Data layer prefetch queue empty
I1113 13:09:55.686236 7814 blocking_queue.cpp:50] Waiting for data
F1113 13:09:57.330299 7814 base_data_layer.cpp:91] Check failed: error == cudaSuccess (4 vs. 0) unspecified launch failure
*** Check failure stack trace: ***
F1113 13:09:57.330299 7806 base_data_layer.cu:25] Check failed: error == cudaSuccess (4 vs. 0) unspecified launch failure
*** Check failure stack trace: ***
已放弃 (核心已转储)
