Show Bounding Box While Detecting Object Using Arkit 2

Unable to get boundingBox from result for a request VNCoreMLRequest

Did you actually train an object detection model? Or a classification model?

You only get bounding boxes for the object detection model, not for a classifier.

Assuming you trained an object detector, the correct class is VNRecognizedObjectObservation, not VNClassificationObservation.

Using Google ML object detection and draw bounding boxes on image on iOS

You don't need any library for this because this is quite a simple process..
Just call this function with the x and y values that you get from api and it should be okay. and of course add the code to the tap gesture

func draxBox(x1: CGFloat, x2: CGFloat, y1: CGFloat, y2: CGFloat) {
    //1. Find the size to the bounding box:
    width = (x2 - x1) * yourImageView.frame.width
    height = (y2 - y1) * yourImageView.frame.height

    //2. Add a subview to yourImageView:
    let boundingBox = UIView()
    boundingBox.backgroundColor = .clear
    boundingBox.layer.borderWidth = 1
    boundingBox.layer.borderColor = UIColor.red.cgColor
    boundingBox.translatesAutoresizingMaskIntoConstraints = false
    yourImageView.addSubview(boundingBox)

    NSLayoutConstraint.activate([
        boundingBox.leadingAnchor.constraint(equalTo: yourImageView.leadingAnchor, constant: x1),
        boundingBox.topAnchor.constraint(equalTo: yourImageView.topAnchor, constant: y1),
        boundingBox.widthAnchor.constraint(equalToConstant: width),
        boundingBox.heightAnchor.constraint(equalToConstant: height)
    ])

    //3. Add tap action to this view:
    let tap = UITapGestureRecognizer(target: self, action: #selector(boxTapped))
    boundingBox.isUserInteractionEnabled = true
    boundingBox.addGestureRecognizer(tap)
}

@objc private fun boxTapped() {
   //actian when tapped goes here
}

ARKit: Tracking VisonCoreML detected object

To be honest, the technologies you're using here cannot do that out of the box. YOLO (and any other object detection model you swapped out for it) have no built in concept of tracking the same object in a video. They look for objects in a 2D bitmap, and return 2D bounding boxes for them. As either the camera or object moves, and you pass in the next capturedImage buffer, it will give you a new bounding box in the correct position, but it has no way of knowing whether or not it's the same instance of the object detected in a previous frame.

To make this work, you'll need to do some post processing of those Vision results to determine whether or not it's the same object, and if so, manually move the anchor/mesh to match the new position. If you're confident there should only be one object in view at any given time, then it's pretty straightforward. If there will be multiple objects, you're venturing into complex (but still achievable) territory.

You could try to incorporate Vision Tracking, which might work though would depend on the nature and behavior of the tracked object.

Also, sceneView.hitTest() is deprecated. You should probably port that over to use ARSession.raycast()

---

Related Topics