Selective Object Rearrangement in Clutter

Video

Abstract

We propose an image-based, learned method for selective tabletop object rearrangement in clutter using a parallel jaw gripper. Our method consists of three stages: graph-based object sequencing (which object to move), feature-based action selection (whether to push or grasp, and at what position and orientation) and a visual correspondence-based placement policy (where to place a grasped object).

Experiments show that this decomposition works well in challenging settings requiring the robot to begin with an initially cluttered scene, selecting only the objects that need to be rearranged while discarding others, and dealing with cases where the goal location for an object is already occupied — making it the first system to address all these concurrently in a purely image-based setting. We also achieve an ~8% improvement in task success rate over the previously best reported result that handles both translation and orientation in less restrictive (un-cluttered, non-selective) settings. We demonstrate zero-shot transfer of our system solely trained in simulation to a real robot selectively rearranging up to 15 everyday objects, many unseen during learning, on a crowded tabletop.

82.33%

Task completion with 16–20 objects, clutter, selectivity and swaps all at once

Position error 1.22 ± 0.93 cm

Hardest setting

90.0%

Task completion in IFOR's setting, with translation, rotation and swaps

IFOR: 81.8%

+8.2 pts

−55.6%

Planning steps from accessibility-graph sequencing, 10 target objects

15.61 vs. 35.13 steps with no sequencing

Ablation

15 objects

Selectively rearranged on a real robot, many unseen in training

Trained only in simulation

Zero-shot sim-to-real

Method

The robot has three primitives: push, grasp and place. Push and grasp are available at any time; place only once an object is in hand. That splits the problem in three: when the hand is empty, decide which object to manipulate next and how; when it is holding something, decide where to put it. The goal is given as a single RGB-D image.

Object sequencing

Graph-based selection

Which object to move next?

We segment the scene with UOIS-Net-3D and build an accessibility graph: one vertex per object plus the robot, with an edge wherever a collision-free straight-line end-effector path exists. Paths from the robot vertex capture which objects are blocking which.

For each object we delete its vertex and score how close the remaining graph G_t − v_i is to the goal graph G_g, using SimGNN to approximate graph edit distance:

v* = arg max_i SimGNN( G_t − v_i, G_g )

The object behind the largest difference from the goal goes first, which keeps the action count low. Non-target objects tend to be cleared early, thinning the clutter for what follows.

Action selection

Push or grasp

How to manipulate it?

A goal-conditioned POMDP solved with deep Q-learning. A DenseNet-121 encoder and fully-convolutional heads turn the RGB-D image into pixel-wise Q-values, at 16 rotations each for push and grasp: 32 maps in total.

The grasp maps are masked to the object chosen in stage 1, and the robot executes the single highest-value action. Rewards are sparse:

GRASP 1.0 if the gripper fingers stay apart, so the object is held
PUSH 0.5 if the depth image changes past a threshold

The higher grasp reward makes the robot grasp whenever it can; pushes are only for singulating objects that cannot yet be grasped.

Object placement

Correspondence-based placing

Where to put it?

Placement is template matching. ResNet features of the grasped region, cropped around the executed grasp, are cross-correlated with features of the goal image. A depth cross-correlation is subtracted, so occupied pixels score lower:

φ^place = φ^similarity − φ^depth

This keeps objects off one another and out of other objects' goal locations, which is what makes swaps possible. Orientation comes from evaluating 16 rotations. If no location matches well enough, the object is not in the goal and goes to the bin.

Results — Simulation

Task completion across settings 3 seeds × 100 episodes

A Franka Panda with a parallel-jaw gripper in PyBullet, 3–20 YCB objects per episode, with objects unseen in training added at test time. An episode is complete when every target object is within 5 cm of its goal and every other object is in the bin. Completion drops as clutter and swaps are added, but position error stays flat: the placement policy remains accurate even when the planning gets hard.

Task completion (mean %) and position error (cm) for NeRP, IFOR and our method across combinations of rotation, swap, clutter and selectivity
Method	Rotation	Swap	Clutter	Selective	Init. # obj.	Goal # obj.	Completion ↑	Pos. error (cm) ↓
NeRP	✗	✓	✗	✗	3–8	3–8	94.56 ± 0.73	1.90 ± 1.30
IFOR	✓	✓	✗	✗	1–9	1–9	81.80	2.70 ± 2.30
Ours	✓	✗	✗	✗	3–7	3–7	96.67 ± 1.67	1.29 ± 0.91
	✓	✓	✗	✗	3–7	3–7	90.00 ± 3.00+8.2 vs IFOR	2.99 ± 2.37
	✓	✗	✗	✓	3–7	1–5	97.33 ± 0.67	1.41 ± 2.70
	✓	✓	✗	✓	3–7	1–5	97.00 ± 1.00	1.81 ± 2.66
	✓	✗	✓	✓	16–20	5–10	85.67 ± 2.33	1.64 ± 0.44
	✓	✓	✓	✓	16–20	5–10	82.33 ± 2.67	1.22 ± 0.93

NeRP and IFOR numbers are as reported in their papers; neither codebase is public. NeRP handles translation only, so the direct comparison is with IFOR in its own setting (no clutter, no selectivity, swaps allowed), at a comparable rotation error (ours 13.89°, IFOR 13.70°). The last row is the setting no prior image-based system handles: high clutter, selectivity and swaps together.

Does the scene graph matter? 10 target objects

Average planning steps to finish when picking the next object with no sequencing (Q-values only), with a simple position-based scene graph, or with the accessibility graph. The accessibility graph cuts steps by 55.56% (10 objects in the scene) and 43.71% (20 objects) compared with no sequencing, and by 21.72% and 13.11% compared with the position graph.

Average planning steps per completed episode

Lower is better · ± 1 s.d.

Average planning steps by scene graph type and number of objects in the initial scene
Scene graph	10 objects	20 objects
None	35.13 ± 3.55	45.22 ± 4.70
Position	19.94 ± 4.93	29.29 ± 3.52
Accessibility	15.61 ± 3.84	25.45 ± 3.88

Choosing the feature extractor

Placement and target identification both depend on the visual features being cross-correlated. Over 100 initial/goal arrangements with 1–20 objects, the ResNet extractor we use reaches a 93.33% match success rate, a 2.04 cm position error and 98.58% target-classification accuracy. The comparison with U-Net is in the supplement.

Results — Real World

A Panda arm with a parallel-jaw gripper and one statically mounted RGB-D camera, with a bin beside the workspace for objects that do not belong. Objects are drawn from 20 everyday items, such as a peanut butter jar or a ketchup bottle. The policy is trained only in simulation and runs on the real robot with no fine-tuning.

The 15-object task from the figure above, at 16× speed. The arm clears non-target objects into the bin, then positions the cup, Spam, ketchup, Nutella and peanut butter to match the goal image. The full run, including more demonstrations, is in the video.

Robot setup. Left: the camera input, a colour image of a cluttered tabletop and the matching depth image. Right: the lab, with a statically mounted camera on a tripod giving an ego-centric view, a Franka Panda robot arm, a clear plastic bin and the workspace marked with black tape holding everyday objects. — **Robot setup.** One RGB-D camera provides both the current observation and the goal image. The arm drops non-target objects into the bin.

Limitations

Planar rotation onlyObjects are reoriented in the plane only, with no 6-DoF reorientation, and the system cannot orient rotationally symmetric objects such as oranges or apples well. Expanding the action space is the natural fix.
Cluttered goal statesThe initial scene can be crowded, but a tightly packed goal arrangement remains hard. With many objects it turns into a packing or stacking problem.
Segmentation errorsScene graphs and sequencing are built on object segmentation, so segmentation mistakes become sequencing mistakes. A better camera setup and segmentation would help.
Motion planningJoint limits can be hit when a grasp that worked for pickup does not work at the new location, and the robot can collide while carrying an object. A trajectory-aware, obstacle-avoiding planner would address this.

BibTeX

@inproceedings{tang2022selective,
  title     = {Selective Object Rearrangement in Clutter},
  author    = {Tang, Bingjie and Sukhatme, Gaurav S.},
  booktitle = {Conference on Robot Learning (CoRL)},
  year      = {2022}
}