Mobile Robotics · Manipulation · Autonomous Systems 2026

Autonomous Mobile
Manipulator

An ongoing R&D platform integrating a 5-axis SO-101 robotic arm onto a holonomic Mecanum-wheeled base — bridging natural language commands with physical manipulation through semantic SLAM, inverse kinematics, and deep learning vision on ROS 2.

  • ROS 2
  • Python
  • RealSense D435
  • SLAM
  • YOLO
  • Raspberry Pi 4
  • Mecanum Drive
  • SO-101 Arm

Project Overview

This platform is a learning-driven R&D testbed for mastering advanced robotics end-to-end — from holonomic kinematics and sensor fusion to high-DOF arm control and language-conditioned manipulation. The ultimate goal is to process a high-level text command (e.g., "Bring me the apple") and translate it into a complete sequence of autonomous physical actions: reasoning, locating, navigating, grasping, and delivering.

The system uses a distributed computing architecture: a Raspberry Pi 4 runs the ROS 2 master, SLAM pipeline, vision inference, and NLP logic — while an Arduino handles low-level PID control of the Mecanum drive, and a dedicated serial controller manages the STS3215 servo joints of the arm.

Current development focus is on establishing reliable inter-board communication and tuning the Mecanum wheel odometry before fully integrating the SLAM and manipulation pipelines.

NLP-to-Action Pipeline

Abstract language commands are translated into physical action through a chain of reasoning, spatial lookup, path planning, and kinematic execution stages.

NLP Command "Bring me the apple" Language Input Semantic Reasoning Map intent → object class LLM / Rule Engine Semantic SLAM Locate in spatial map RealSense + Odometry Path Planning Navigate + avoid obstacles ROS 2 Nav2 IK + Grasp Plan End-effector trajectory STS3215 Servos EXECUTE Grasp object Deliver to user

Distributed System Architecture

Computation is partitioned across three controllers — the Raspberry Pi 4 handles all high-level intelligence while dedicated microcontrollers manage time-critical low-level control loops.

Raspberry Pi 4 — Central Compute ROS 2 Master · SLAM · Vision Inference · NLP Logic USB 3.0 Intel RealSense D435 RGB-D · Point Cloud Perception · SLAM · Grasp UART Serial Servo Controller STS3215 bus protocol Arm joint management SO-101 Arm (5-DOF) 5 × STS3215 Servo Joints ROS Twist Arduino / ESP32 PID · Holonomic kinematics Low-level drive control Mecanum Drive (4WD) Holonomic · Omnidirectional

Core Engineering Focus

Four advanced robotics disciplines developed and integrated on this platform.

Mobile Platform Kinematics

  • 4-wheel Mecanum control matrix
  • True holonomic motion (strafe, diagonal, rotate)
  • Wheel encoder odometry for dead-reckoning
  • PID velocity loop on Arduino

Sensor Fusion & Semantic SLAM

  • RealSense point-cloud + wheel odometry fusion
  • Geometric and semantic map layers
  • Object-class annotations in map space
  • YOLO / RT-DETR for object identification

High-DOF Arm Control

  • Inverse kinematics for SO-101 5-axis arm
  • Smooth 3D end-effector trajectory planning
  • STS3215 servo bus communication
  • Joint-space to Cartesian-space mapping

Vision & Grasp Planning

  • YOLO / RT-DETR over RGB-D feed
  • Bounding box to 3D depth projection
  • Grasp pose estimation from depth data
  • Servo-driven gripper control

Current Status

Ongoing development — in communication and kinematics tuning phase.

Mecanum holonomic drive operational — all 4 wheels independently controlled via Arduino PID loops receiving ROS 2 Twist messages.

SO-101 arm joint control verified — STS3215 servos communicating via serial bus, forward kinematics confirmed across all 5 joints.

In progress — tuning Mecanum odometry accuracy before integrating the RealSense SLAM and vision pipelines.

Planned — NLP-to-action pipeline: connecting language reasoning to SLAM object lookup and IK-based manipulation execution.