Bin picking: how a robot with a 3D camera decides what to pick

Bin picking means picking objects from a pile or a bin where the parts lie at random, on top of each other and differently every time. A fixed path does not work here: each time, the robot has to see what is in front of it and decide what to pick and how. This requires a 3D camera, software that locates objects in space, and a robot that can be controlled externally.

Why you need a 3D image

A 2D camera sees outlines but knows nothing about height. In a pile that is not enough: you need to know which object is on top, how it is tilted and whether there is room around it for the gripper. A 3D camera or scanner provides a point cloud or a depth map - the distance is known for every point in the image.

For large scenes and fast imaging, ToF 3D cameras are a good fit; for small parts and higher accuracy, structured light scanners such as Photoneo PhoXi. More on the technologies in our article on 3D machine vision.

How bin picking works

1. Scanning. A camera above the pile captures a 3D image. Several frames can be fused to reduce noise.
2. Segmentation. The image is divided into individual objects: where one ends and the next begins. This uses geometric algorithms and artificial intelligence - see AI segmentation in manufacturing.
3. Target selection. From the objects found, the one that is safest and simplest to pick is chosen: usually the topmost one, not pinned down by others and within the robot's reach.
4. Pick point. The software calculates where and at what angle the gripper should grip the object: for long objects from the axis and length, for flat parts from the surface.
5. Collision check. Before moving, the system checks that the gripper and robot will not hit other objects, the edges of the bin or surrounding equipment.
6. Pick rules. The program has clear rules: what not to pick, what to do if an object is detected unreliably, when to stop and call the operator.
7. Cycle. The robot picks the object, places or feeds it onward, and everything repeats with a new image, because the pile has changed.

For the coordinates seen by the camera to match the robot's coordinates, the camera and robot are calibrated relative to each other. See our article on hand-eye calibration.

Our example: a log feeding cell

For a wood-processing company we are building a cell in which a KUKA KR 60-3 robot (KR C4 controller) picks a log from a pile on a ramp and feeds it butt-end first into the feed rollers of a processing machine. The machine turns the logs into stakes and poles. The project is currently being commissioned, so the figures below are test bench values.

• 3D image. A Basler blaze-101 ToF camera is mounted above the pile.
• Object detection. A PC (Python) finds the logs in the 3D image and estimates their axis, length and ends. It uses our own 3D detector and pick rules, AI segmentation of logs in photos (SAM) and a 3D height map.
• Noise. On the test bench, fusing 20 frames gave a depth noise of approx. 0.8-1.3 mm.
• Objects. Logs 2600-3300 mm long, roughly 100-130 mm in diameter.
• Gripper. Two Delta ASDA-A2 servo drives controlled over CANopen. Both drives start together within 0 ± 2 ms; clamping or unclamping takes approx. 0.22-0.25 s.
• Robot communication. KRL modules and EthernetKRL - the PC sends the pick coordinates to the robot. More on this kind of external control in our article on industrial robot programming.
• Operator. The operator panel runs on the local network, and the operator can pause operation at any time.

Safety: AI suggests, safety permits

Image processing and neural networks decide which object to pick and where to grip it, but they do not permit motion. Motion is always permitted by the safety circuit. In a well-designed bin picking system, an unclear situation does not mean "let's try and see" - it means stopping and giving the operator a clear message.

What to prepare before a bin picking project

• A few real objects, or photos of how they lie in the pile or bin.
• The objects' dimensions, weight and surface type: shiny, dark, wet, uneven.
• The required cycle time.
• Where the object goes after picking and how it must be oriented.
• What equipment is already in place: machines, PLC, safety equipment.

We start vision projects with a trial on your real parts - before any larger investment. We have our own KUKA and Yaskawa Motoman robots for testing. More under machine vision and robot integration services and in our projects. Do you have a pile of parts that is still loaded by hand? Get in touch - we will see whether a robot with a 3D camera can pick them.