FabLab Kamp-Lintfort

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Step-by-step example ======

The step-by-step example goes through the whole process of setting up and using an [[https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/|NVIDIA Jetson Xavier NX]] with our custom [[https://github.com/ultralytics/yolov5|YOLOv5]] implementation.

===== - Setting up the NVIDIA Jetson Xavier NX =====

To set up the [[https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/|NVIDIA Jetson Xavier NX]], follow their respective guide to get started:

[[https://developer.nvidia.com/embedded/learn/get-started-jetson-xavier-nx-devkit|Getting Started with Jetson Xavier NX Developer Kit]]

===== - Plugging in I/O devices =====

After the [[https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/|NVIDIA Jetson Xavier NX]] has been set up properly, you should plug in the required I/O devices.

You will need the following devices:

  * Display (over HDMI or DisplayPort)
  * USB keyboard
  * CSI or USB camera (In this example: [[https://www.elpcctv.com/-p-248.html|ELP-USB8MP02G-SFV]])
  * Configured Arduino

===== - Preparing the NVIDIA Jetson Xavier NX =====

First, you may want to head over to the system settings to turn off the automatic screen saver and screen lock. It may cause problems when you build the [[https://www.docker.com/|Docker]] image, as this will take some more time.

After that, it is recommended to update all packages on the system with:

<file shell>
sudo apt update; sudo apt upgrade -y
</file>

<WRAP info>
You can quickly open a new terminal on [[https://ubuntu.com/|Ubuntu]] with the hotkey ''Ctrl+Alt+T''.
</WRAP>

===== - Checking I/O devices =====

==== - Checking camera ====

First, list all current devices on the system with:

<file shell>
la /dev
</file>

You should be able to see a ''video0'' device in the terminal output. This represents the connected camera.

The [[https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/|NVIDIA Jetson Xavier NX]] provides a tool called ''nvgstcapture-1.0'' that can test a camera's output by using pre-constructed [[https://gstreamer.freedesktop.org/|GStreamer]] pipelines.

In this example we assume that we have connected a [[https://www.elpcctv.com/-p-248.html|ELP-USB8MP02G-SFV]] camera, which is a USB camera.

To test a USB camera with the ''nvgstcapture-1.0'' tool, run the following:

<file shell>
nvgstcapture-1.0 --camsrc=0 --cap-dev-node=0
</file>

You should now be able to see the camera's output in an additional window.

<WRAP info>
Consult the repository's [[https://gitlab.com/fablabkamplintfort1/farmrobot/-/blob/master/README.md|README]] to get more information about the ''nvgstcapture-1.0'' tool.
</WRAP>

Furthermore, you should check the camera's supported formats, as those will be relevant later when configuring [[https://github.com/ultralytics/yolov5|YOLOv5]]'s source.

The ''v4l-utils'' package makes this process very simple. First, install it via ''apt'', like so:

<file shell>
sudo apt install -y v4l-utils
</file>

If the installation was successful, you can go ahead and use it:

<file shell>
v4l2-ctl -d /dev/video0 --list-formats-ext
</file>

For the camera used in this example ([[https://www.elpcctv.com/-p-248.html|ELP-USB8MP02G-SFV]]) it will produce the following output:

<file>
ioctl: VIDIOC_ENUM_FMT
        Index       : 0
        Type        : Video Capture
        Pixel Format: 'MJPG' (compressed)
        Name        : Motion-JPEG
                Size: Discrete 1600x1200
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 3264x2448
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 2048x1536
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 1280x960
                        Interval: Discrete 0.067s (15.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 329x240
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 1600x1200
                        Interval: Discrete 0.067s (15.000 fps)
 
        Index       : 1
        Type        : Video Capture
        Pixel Format: 'YUYV'
        Name        : YUYV 4:2:2
                Size: Discrete 1600x1200
                        Interval: Discrete 0.100s (10.000 fps)
                Size: Discrete 3264x2448
                        Interval: Discrete 0.067s (2.000 fps)
                Size: Discrete 2592x1944
                        Interval: Discrete 0.333s (3.000 fps)
                Size: Discrete 2048x1536
                        Interval: Discrete 0.333s (3.000 fps)
                Size: Discrete 1280x960
                        Interval: Discrete 0.100s (10.000 fps)
                Size: Discrete 1024x768
                        Interval: Discrete 0.100s (10.000 fps)
                Size: Discrete 800x600
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 640x480
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 329x240
                        Interval: Discrete 0.067s (30.000 fps)
                Size: Discrete 1600x1200
                        Interval: Discrete 0.100s (10.000 fps)
</file>

==== - Checking configured Arduino ====

List all current devices on the system with:

<file shell>
la /dev
</file>

You should be able to see a ''ttyUSB0'' device in the terminal output. This represents the connected configured Arduino.

===== - Setting up our custom implementation of YOLOv5 =====

==== - Cloning the repositories ====

First, clone the [[https://gitlab.com/fablabkamplintfort1/farmrobot|repository]] into the home (''~'') directory:

<file shell>
git clone https://gitlab.com/fablabkamplintfort1/farmrobot.git
</file>

Then, change into the repository's directory and clone the official [[https://github.com/ultralytics/yolov5|YOLOv5]] into it:

<file shell>
cd farm-robot-yolov5

git clone https://github.com/ultralytics/yolov5.git
</file>

Once this is set up, you are theoretically ready to build the [[https://www.docker.com/|Docker]] image. But you usually want to configure the implementation via the ''.toml'' configuration files first.

==== - Configuration ====

The repository comes with a ''default_config.toml'' which contains the default configuration. **However, you should not edit this file directly, since it is version controlled by the repository!** Instead, copy the file with a new name (''config.toml''), like so:

<file shell>
cp default_config.toml config.toml
</file>

The ''config.toml'' file is ignored by default, so you can edit it however you like.

In the newly created ''config.toml'' you will have to edit atleast one value, the ''source''. The ''source'' value should consist of a [[https://gstreamer.freedesktop.org/|GStreamer]] pipeline. With the [[https://www.elpcctv.com/-p-248.html|ELP-USB8MP02G-SFV]] camera, you can use the following [[https://gstreamer.freedesktop.org/|GStreamer]] pipeline:

<file shell>
v4l2src device=/dev/video0 ! image/jpeg, width=2592, height=1944, framerate=15/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv flip-method=0 ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink
</file>

<WRAP important>
It is important to replace the ''width='', ''height='' and ''framerate='' values, with the values of a supported format of the used camera!
</WRAP>

The configured ''config.toml'' should look something like this now:

<file toml>
[yolov5]
# Input source.
source = "v4l2src device=/dev/video0 ! image/jpeg, width=2592, height=1944, framerate=15/1 ! nvv4l2decoder mjpeg=1 ! nvvidconv flip-method=0 ! video/x-raw, format=BGRx ! videoconvert ! video/x-raw, format=BGR ! appsink"

...
</file>

<WRAP info>
Consult the repository's [[https://gitlab.com/fablabkamplintfort1/farmrobot/-/blob/master/README.md|README]] to get more information about [[https://gstreamer.freedesktop.org/|GStreamer]] pipelines.
</WRAP>

<WRAP important>
If you intend to run the [[https://www.nvidia.com/en-us/autonomous-machines/embedded-systems/jetson-xavier-nx/|NVIDIA Jetson Xavier NX]] without an internet connection, you have to manually download the configured ''weights'', before you build and instantiate the [[https://www.docker.com/|Docker]] container. The default [[https://github.com/ultralytics/yolov5|YOLOv5]] weights can be found on its [[https://github.com/ultralytics/yolov5/releases|releases page]].
</WRAP>

==== - Building the Docker image ====

Building the [[https://www.docker.com/|Docker]] image is easy and straightforward. Simply run:

<file shell>
sudo docker build -t farm-robot-yolov5 .
</file>

<WRAP important>
The build process of the [[https://www.docker.com/|Docker]] image could take up to an hour!
</WRAP>

==== - Instantiating the Docker container ====

When instantiating the [[https://www.docker.com/|Docker]] container you need to pay attention to mount all required devices (camera and configured Arduino)! With the devices used in this example, you can instantiate the container, like so:

<file shell>
sudo docker run -it --rm --runtime nvidia --device /dev/video0 --device /dev/ttyUSB0 farm-robot-yolov5
</file>

<WRAP info>
Consult the repository's [[https://gitlab.com/fablabkamplintfort1/farmrobot/-/blob/master/README.md|README]] to get more information about mounting devices to a [[https://www.docker.com/|Docker]] container.
</WRAP>

==== - Starting the inference ====

Once the interactive [[https://www.docker.com/|Docker]] container booted, you can run the ''__main__.py'':

<file shell>
python3 .
</file>

If everything is configured correctly, the delta arm should start to move to its home position and then to its configured initial position. A few seconds after that, [[https://github.com/ultralytics/yolov5|YOLOv5]]'s inference should start. When [[https://github.com/ultralytics/yolov5|YOLOv5]] detects an object that has been declared as a "target" (via the ''targets'' list in the ''config.toml''), the delta arm should move to the object's position.
FabLab Kamp-Lintfort

User Tools

Site Tools

Page Tools
