Created by [ Rowan Dempster] on Dec 28, 2019
https://www.intel.com/content/www/us/en/programmable/documentation/tgy1490191698959.html
guide to get the fpga working
quartus_pgm cannot find the libxerc 32https://forums.intel.com/s/question/0D50P00004IpHt2/error-opening-quartus-or-compiling-opencl-files-asthis-is-happening-with-quartus-191-any-env-variable-i-am-missingquartus-error-while-loading-shared-libraries-libxercesc32so-cannot-open-shared-object-file-no-such-file-or-directory?language=en_US
this was fixed through the above
following https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_GX_Configure_2019R1.html
got aocl working and inference seems to work
root@AUTODRIVE-BOLT:/home/test/inference_engine_samples_build/intel64/Release# ./classification_sample -i car.png -m /home/test/openvino_models/ir/FP32/classification/squeezenet/1.1/caffe/squeezenet1.1.xml -d HETERO:FPGA,CPU -ni 1000
[ INFO ] InferenceEngine:
API version ............ 1.6
Build .................. custom_releases/2019/R1.1_28dfbfdd28954c4dfd2f94403dd8dfc1f411038b
[ INFO ] Parsing input parameters
[ INFO ] Files were added: 1
[ INFO ] car.png
[ INFO ] Loading plugin
API version ............ 1.6
Build .................. heteroPlugin
Description ....... heteroPlugin
[ INFO ] Loading network files:
/home/test/openvino_models/ir/FP32/classification/squeezenet/1.1/caffe/squeezenet1.1.xml
/home/test/openvino_models/ir/FP32/classification/squeezenet/1.1/caffe/squeezenet1.1.bin
[ INFO ] Preparing input blobs
[ WARNING ] Image is resized from (787, 259) to (227, 227)
[ INFO ] Batch size is 1
[ INFO ] Preparing output blobs
[ INFO ] Loading model to the plugin
Error: Could not load board library /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so (error_msg: /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so: cannot open shared object file: No such file or directory)
Failed to dynamically load board MMD /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so
Error: Could not load board library /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so (error_msg: /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so: cannot open shared object file: No such file or directory)
Failed to dynamically load board MMD /opt/altera/aocl-pro-rte/aclrte-linux64/board/hddlf_1150_sg1/linux64/lib/libaltera_a10_1150_sg1_mmd.so
[ INFO ] Starting inference (1000 iterations)
[ INFO ] Processing output blobs
Top 10 results:
Image car.png
classid probability label
------- ----------- -----
817 0.8931729 sports car, sport car
511 0.0444685 convertible
479 0.0444685 car wheel
436 0.0060182 beach wagon, station wagon, wagon, estate car, beach waggon, station waggon, waggon
751 0.0060182 racer, race car, racing car
656 0.0022140 minivan
864 0.0008145 tow truck, tow car, wrecker
717 0.0008145 pickup, pickup truck
586 0.0008145 half track
581 0.0002996 grille, radiator grille
total inference time: 1503.0926671
Average running time of one iteration: 1.5030927 ms
Throughput: 665.2949761 FPS
[ INFO ] Execution successful
terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
what(): Error: One of the following environment variables must be defined: DLA_AOCX or CL_CONTEXT_COMPILER_MODE_INTELFPGA=3
[ INFO] [1572913657.585308102]: Param output_frame_name: world
terminate called after throwing an instance of 'InferenceEngine::details::InferenceEngineException'
what(): Graph is not supported on FPGA plugin due to existance of layer (Name: FeatureExtractor/MobilenetV1/MobilenetV1/Conv2d_0/Relu6, Type: Clamp)
in topology. Most likely you need to use heterogeneous plugin instead of FPGA plugin directly.
/opt/intel/computer_vision_sdk/deployment_tools/inference_engine/include/details/ie_exception_conversion.hpp:71
[ ERROR ] Graph is not supported on FPGA plugin due to existance of layer (Name: LeakyReLU_3838, Type: ReLU)
in topology. Most likely you need to use heterogeneous plugin instead of FPGA plugin directly.
Steps to run the FPGA:
https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_GX_Configure_2019R1.html
jtagconfig --getparam 1 JtagClock
quartus_pgm -c 1 -m JTAG -o "p;max5_150.pof@2"
quartus_pgm -c 1 -m JTAG -o "p;top.sof"
any bitstream works
ex:
aocl program acl0 /opt/intel/openvino/bitstreams/a10_devkit_bitstreams/2019R1_A10DK_FP11_SqueezeNet.aocx
https://docs.openvinotoolkit.org/2019_R1.1/_docs_install_guides_GX_Configure_2019R1.html
Running
roscore
rosbag play (some bag)
roslaunch perception perception-vehicle-exp-fpga.launch
gives result
aunch perception-vehicle-exp-fpga.launch
... logging to /home/autodrive/.ros/log/51f012e8-04e6-11ea-8808-ac1f6b056632/roslaunch-AUTODRIVE-BOLT-72293.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt
WARNING: disk usage in log directory [/home/autodrive/.ros/log] is over 1GB.
It's recommended that you use the 'rosclean' command.
started roslaunch server http://AUTODRIVE-BOLT:35715/
SUMMARY
========
PARAMETERS
* /perception/clip_cloud: True
* /perception/clip_max_height: 5.0
* /perception/clip_min_height: 0.2
* /perception/cluster_merge_threshold: 1.5
* /perception/cluster_size_max: 500
* /perception/cluster_size_min: 5
* /perception/clustering_distances: [15, 30, 45, 60]
* /perception/clustering_thresholds: [0.5, 1.1, 1.6, 2...
* /perception/downsample_cloud: False
* /perception/downsample_leaf_size: 0.2
* /perception/frame_pub_topic: /camera/image_color
* /perception/img_test_dir: /home/watonomous/...
* /perception/keep_lane_left_distance: 5.0
* /perception/keep_lane_right_distance: 5.0
* /perception/keep_lanes: False
* /perception/lidar_name_list: ['/velodyne_points']
* /perception/lidar_output_frame: world
* /perception/lidar_scales: [2.0]
* /perception/lidar_tf_list: [0, 0, 2.06, 0, 0...
* /perception/nn_hardware: HETERO:FPGA,CPU
* /perception/nn_obstacle_model_path: /home/autodrive/i...
* /perception/nn_segmentation_model_path: /home/autodrive/i...
* /perception/nn_traffic_sign_model_path: /home/autodrive/i...
* /perception/obj_nn_hardware: CPU
* /perception/pose_estimation: False
* /perception/remove_ground: True
* /perception/remove_points_upto: 0.0
* /perception/rm_ground_max_angle: 0.1
* /perception/rm_ground_max_height: 0.2
* /perception/use_diffnormals: False
* /rosdistro: kinetic
* /rosversion: 1.12.14
NODES
/
object_detection_node (perception/object_detection_node)
obstacle_detection_node (perception/obstacle_detection_node)
rviz (rviz/rviz)
traffic_light_detection_node (perception/traffic_light_detection_node)
traffic_sign_detection_node (perception/traffic_sign_detection_node)
ROS_MASTER_URI=http://localhost:11311
process[traffic_light_detection_node-1]: started with pid [72310]
/home/autodrive/integration/devel/lib/perception/traffic_light_detection_node: /opt/altera/quartus/linux64/liblzma.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libtiff.so.5)
process[traffic_sign_detection_node-2]: started with pid [72311]
/home/autodrive/integration/devel/lib/perception/traffic_sign_detection_node: /opt/altera/quartus/linux64/liblzma.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libtiff.so.5)
process[obstacle_detection_node-3]: started with pid [72312]
/home/autodrive/integration/devel/lib/perception/obstacle_detection_node: /opt/altera/quartus/linux64/liblzma.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libtiff.so.5)
process[object_detection_node-4]: started with pid [72314]
process[rviz-5]: started with pid [72325]
/opt/ros/kinetic/lib/rviz/rviz: /opt/altera/quartus/linux64/libssl.so.1.0.0: no version information available (required by /usr/lib/x86_64-linux-gnu/libcurl.so.4)
/opt/ros/kinetic/lib/rviz/rviz: /opt/altera/quartus/linux64/libssl.so.1.0.0: no version information available (required by /usr/lib/x86_64-linux-gnu/libcurl.so.4)
/opt/ros/kinetic/lib/rviz/rviz: /opt/altera/quartus/linux64/libssl.so.1.0.0: no version information available (required by /usr/lib/x86_64-linux-gnu/libcurl.so.4)
/opt/ros/kinetic/lib/rviz/rviz: /opt/altera/quartus/linux64/libcrypto.so.1.0.0: no version information available (required by /usr/lib/x86_64-linux-gnu/libcurl.so.4)
/opt/ros/kinetic/lib/rviz/rviz: /opt/altera/quartus/linux64/liblzma.so.5: no version information available (required by /usr/lib/x86_64-linux-gnu/libtiff.so.5)
This application failed to start because it could not find or load the Qt platform plugin "xcb"
in "".
Reinstalling the application may fix this problem.
[ INFO] [1573519988.995210978]: Got param: perception/lidar_tf_list
[ INFO] [1573519988.997745412]: Got param: perception/lidar_name_list
[ INFO] [1573519989.001843973]: Got param: perception/lidar_scales
[ INFO] [1573519989.005899740]: Param output_frame_name: world
[rviz-5] process has died [pid 72325, exit code -6, cmd /opt/ros/kinetic/lib/rviz/rviz -d /home/autodrive/integration/src/perception-year-2/perception_config.rviz __name:=rviz __log:=/home/autodrive/.ros/log/51f012e8-04e6-11ea-8808-ac1f6b056632/rviz-5.log].
log file: /home/autodrive/.ros/log/51f012e8-04e6-11ea-8808-ac1f6b056632/rviz-5*.log
[ INFO] [1573519993.142630999]: Euclidean Clustering: Difference of Normals will not be used.
[ INFO] [1573519993.143533442]: downsample_cloud: 0
[ INFO] [1573519993.144492598]: remove_ground: 1
[ INFO] [1573519993.145717309]: leaf_size: 0.200000
[ INFO] [1573519993.146633842]: cluster_size_min: 5
[ INFO] [1573519993.147559619]: cluster_size_max: 500
[ INFO] [1573519993.148500777]: pose_estimation: 0
[ INFO] [1573519993.149441911]: clip_: 1
[ INFO] [1573519993.150403588]: clip_min_height: 0.200000
[ INFO] [1573519993.151369125]: clip_max_height: 5.000000
[ INFO] [1573519993.152273273]: keep_lanes: 0
[ INFO] [1573519993.153210256]: keep_lane_left_distance: 5.000000
[ INFO] [1573519993.154151393]: keep_lane_right_distance: 5.000000
[ INFO] [1573519993.157007388]: cluster_merge_threshold: 1.500000
[ INFO] [1573519993.157040367]: output_frame: world
[ INFO] [1573519993.157952331]: remove_points_upto: 0.000000
[ INFO] [1573519993.158859758]: rm_ground_max_height: 0.200000
[ INFO] [1573519993.159777004]: rm_ground_max_angle: 0.100000
_clustering_thresholds: 0.5 1.1 1.6 2.1 2.6
_clustering_distances: 15 30 45 60
[YOLOv3 model] Total Inference time: 132.92
[ INFO] [1573692465.648741476]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692465.648815280]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692465.654816654]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.519339 (368,382)-(419,470) WILL BE RENDERED!
[9] element, prob = 0.542692 (1265,357)-(1286,410) WILL BE RENDERED!
[9] element, prob = 0.73251 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.774404 (1173,330)-(1199,390) WILL BE RENDERED!
[9] element, prob = 0.777636 (660,227)-(704,307) WILL BE RENDERED!
[9] element, prob = 0.797869 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.827907 (222,266)-(250,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 141.547
[ INFO] [1573692465.815947144]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692465.816125210]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692465.821551153]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.517651 (1166,328)-(1192,388) WILL BE RENDERED!
[9] element, prob = 0.529321 (1265,357)-(1286,410) WILL BE RENDERED!
[9] element, prob = 0.541512 (369,382)-(420,470) WILL BE RENDERED!
[9] element, prob = 0.718599 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.787882 (660,227)-(704,307) WILL BE RENDERED!
[9] element, prob = 0.805389 (222,266)-(251,339) WILL BE RENDERED!
[9] element, prob = 0.811356 (604,381)-(631,430) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 139.528
[ INFO] [1573692465.979062469]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692465.979147435]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692465.985056132]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.509362 (1265,357)-(1286,410) WILL BE RENDERED!
[9] element, prob = 0.520959 (370,382)-(420,470) WILL BE RENDERED!
[9] element, prob = 0.713559 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.775284 (1173,329)-(1199,390) WILL BE RENDERED!
[9] element, prob = 0.789382 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.806354 (222,266)-(250,339) WILL BE RENDERED!
[9] element, prob = 0.814376 (604,381)-(631,430) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 135.095
[ INFO] [1573692466.141292421]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.141357265]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.147714559]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.505828 (1166,328)-(1192,389) WILL BE RENDERED!
[9] element, prob = 0.509245 (369,383)-(419,471) WILL BE RENDERED!
[9] element, prob = 0.582197 (1264,357)-(1285,410) WILL BE RENDERED!
[9] element, prob = 0.668712 (660,228)-(703,307) WILL BE RENDERED!
[9] element, prob = 0.72905 (989,408)-(1012,444) WILL BE RENDERED!
[9] element, prob = 0.810221 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.824375 (222,266)-(250,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 143.282
[ INFO] [1573692466.311568419]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.311673307]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.317944510]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.531257 (1264,357)-(1285,410) WILL BE RENDERED!
[9] element, prob = 0.537129 (369,383)-(419,471) WILL BE RENDERED!
[9] element, prob = 0.711205 (223,268)-(251,338) WILL BE RENDERED!
[9] element, prob = 0.728396 (989,408)-(1013,443) WILL BE RENDERED!
[9] element, prob = 0.754899 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.764722 (1173,329)-(1199,390) WILL BE RENDERED!
[9] element, prob = 0.802272 (604,381)-(631,430) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 138.735
[ INFO] [1573692466.476747338]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.476849503]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.483104240]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.522163 (369,382)-(419,470) WILL BE RENDERED!
[9] element, prob = 0.727146 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.773133 (1173,330)-(1199,391) WILL BE RENDERED!
[9] element, prob = 0.789589 (660,227)-(703,306) WILL BE RENDERED!
[9] element, prob = 0.80187 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.848182 (223,266)-(251,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 141.283
[ INFO] [1573692466.644343526]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.644397735]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.649761901]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.519377 (369,383)-(420,471) WILL BE RENDERED!
[9] element, prob = 0.730925 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.772681 (1173,329)-(1199,390) WILL BE RENDERED!
[9] element, prob = 0.786392 (660,228)-(704,307) WILL BE RENDERED!
[9] element, prob = 0.790988 (223,266)-(251,339) WILL BE RENDERED!
[9] element, prob = 0.813983 (604,381)-(631,430) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 140.768
[ INFO] [1573692466.811892776]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.811958544]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.817682112]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.525941 (369,382)-(419,470) WILL BE RENDERED!
[9] element, prob = 0.723062 (989,408)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.787405 (1173,329)-(1199,390) WILL BE RENDERED!
[9] element, prob = 0.796122 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.799669 (223,266)-(251,339) WILL BE RENDERED!
[9] element, prob = 0.804626 (604,381)-(631,430) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 137.96
[ INFO] [1573692466.976160838]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692466.976222209]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692466.982404231]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.516538 (1264,357)-(1285,410) WILL BE RENDERED!
[9] element, prob = 0.721421 (988,408)-(1012,444) WILL BE RENDERED!
[9] element, prob = 0.779655 (1173,330)-(1199,391) WILL BE RENDERED!
[9] element, prob = 0.787132 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.800222 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.82265 (222,265)-(251,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 134.363
[ INFO] [1573692467.135230779]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692467.135299774]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692467.142074139]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.506846 (369,382)-(420,470) WILL BE RENDERED!
[9] element, prob = 0.713898 (989,408)-(1013,443) WILL BE RENDERED!
[9] element, prob = 0.788275 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.789782 (1173,330)-(1199,391) WILL BE RENDERED!
[9] element, prob = 0.806017 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.820947 (222,265)-(250,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 134.81
[ INFO] [1573692467.297040782]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692467.297108747]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692467.303166453]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
[9] element, prob = 0.501055 (1264,357)-(1285,410) WILL BE RENDERED!
[9] element, prob = 0.535211 (369,383)-(419,470) WILL BE RENDERED!
[9] element, prob = 0.729431 (988,409)-(1012,444) WILL BE RENDERED!
[9] element, prob = 0.780234 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.781127 (1173,330)-(1199,391) WILL BE RENDERED!
[9] element, prob = 0.798357 (604,381)-(631,431) WILL BE RENDERED!
[9] element, prob = 0.848061 (222,266)-(250,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 200.237
[ INFO] [1573692467.523475888]: published a msg to 'perception/obstacle_detection.'
[ INFO] [1573692467.523549225]: published a msg to 'traffic_light_internal.'
[ INFO] [1573692467.529240893]: obstacle_detection/main.cpp: New frame arrived.
Accepted image input
^C[rviz-4] killing on exit
[traffic_sign_detection_node-2] killing on exit
[obstacle_detection_node-3] killing on exit
[traffic_light_detection_node-1] killing on exit
[9] element, prob = 0.510156 (1166,328)-(1192,389) WILL BE RENDERED!
[9] element, prob = 0.515447 (369,383)-(419,471) WILL BE RENDERED!
[9] element, prob = 0.551537 (1264,357)-(1285,410) WILL BE RENDERED!
[9] element, prob = 0.733814 (989,409)-(1013,444) WILL BE RENDERED!
[9] element, prob = 0.791666 (660,227)-(704,306) WILL BE RENDERED!
[9] element, prob = 0.808993 (604,381)-(630,430) WILL BE RENDERED!
[9] element, prob = 0.820049 (222,266)-(251,339) WILL BE RENDERED!
[YOLOv3 model] Total Inference time: 147.271
[ INFO] [1573692467.693394209]: published a msg to 'perception/obstacle_detection.'
shutting down processing monitor...
... shutting down processing monitor complete
done
[[[]{.companion-edit-button-placeholder
.edit-button-overlay data-linked-resource-container-id=”1442304”
data-linked-resource-id=”1442305” data-template-name=”companionEditIcon”
data-source-location=”embedded-attachment”}]
fpga_result.txt (text/plain)\
Document generated by Confluence on Dec 10, 2021 04:01