Opengl Es 2.0 Object Picking on iOS

OpenGL ES 2.0 Object Picking on iOS

Here is working prototype of color picking, tested on most old ipads and working well. This is actually some part of project called InCube Chess that one may find in app store. The main code you will see is located in a class derived from GLKViewController like this:

@interface IncubeViewController : GLKViewController

This means you have glkview in it: ((GLKView *)self.view).

Here are also some properties:

@property (strong, nonatomic) EAGLContext *context;
@property (strong, nonatomic) GLKBaseEffect *effect;

Don't forget to synthesize them in your *.m file.

@synthesize context = _context;
@synthesize effect = _effect;

The idea is that you have chess pieces on your table (or some objects in your 3d scene) and you need to find a piece in your list of pieces by tapping on a screen. That is, you need to convert your 2d screen tap coords (@point in this case) to chess piece instance.

Each piece has its unique id that I call a "seal". You can allocate the seals from from 1 up to something. Selection function returns piece seal found by tap coords. Then having the seal you can easily find your piece in pieces hash table or array in a way like this:

-(Piece *)findPieceBySeal:(GLuint)seal
{
        /* !!! Black background in off screen buffer produces 0 seals. This allows
           to quickly filter out taps that did not select anything (will be
           mentioned below) !!! */
        if (seal == 0)
                return nil;
        PieceSeal *sealKey = [[PieceSeal alloc] init:s];
        Piece *p = [sealhash objectForKey:sealKey];
        [sealKey release];
        return p;
}

"sealhash" is a NSMutableDictionary.

Now this is the main selection function. Note, that my glkview is antialised and you can't use its buffers for color picking. This mean you need to create your own off screen buffer with antialiasing disabled for picking purposes only.

- (NSUInteger)findSealByPoint:(CGPoint)point
{
        NSInteger height = ((GLKView *)self.view).drawableHeight;
        NSInteger width = ((GLKView *)self.view).drawableWidth;
        Byte pixelColor[4] = {0,};
        GLuint colorRenderbuffer;
        GLuint framebuffer;

        glGenFramebuffers(1, &framebuffer);
        glBindFramebuffer(GL_FRAMEBUFFER, framebuffer);
        glGenRenderbuffers(1, &colorRenderbuffer);
        glBindRenderbuffer(GL_RENDERBUFFER, colorRenderbuffer);

        glRenderbufferStorage(GL_RENDERBUFFER, GL_RGBA8_OES, width, height);
        glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0_OES, GL_RENDERBUFFER, colorRenderbuffer);

        GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);
        if (status != GL_FRAMEBUFFER_COMPLETE) {
                NSLog(@"Framebuffer status: %x", (int)status);
                return 0;
        }

        [self render:DM_SELECT];

        CGFloat scale = UIScreen.mainScreen.scale;
        glReadPixels(point.x * scale, (height - (point.y * scale)), 1, 1, GL_RGBA, GL_UNSIGNED_BYTE, pixelColor);

        glDeleteRenderbuffers(1, &colorRenderbuffer);
        glDeleteFramebuffers(1, &framebuffer);

        return pixelColor[0];
}

Note that function takes into account display scale (retina or new iPads).

Here is render() function used in function above. Note, that for rendering purposes it clear s the buffer with some background color and for selecting case it makes it black so that you can easily check if you tapped on any piece at all or not.

- (void) render:(DrawMode)mode
{
        if (mode == DM_RENDER)
                glClearColor(backgroundColor.r, backgroundColor.g,
                             backgroundColor.b, 1.0f);
        else
                glClearColor(0.0f, 0.0f, 0.0f, 1.0f);

        glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

        /* Draw all pieces. */
        for (int i = 0; i < [model->pieces count]; i++) {
                Piece *p = [model->pieces objectAtIndex:i];
                [self drawPiece:p mode:mode];
        }
}

Next is how we draw the piece.

- (void) drawPiece:(Piece *)p mode:(DrawMode)mode
{
        PieceType type;

        [self pushMatrix];

        GLKMatrix4 modelViewMatrix = self.effect.transform.modelviewMatrix;

        GLKMatrix4 translateMatrix = GLKMatrix4MakeTranslation(p->drawPos.X,
                                                               p->drawPos.Y,
                                                               p->drawPos.Z);
        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, translateMatrix);

        GLKMatrix4 rotateMatrix;
        GLKMatrix4 scaleMatrix;

        if (mode == DM_RENDER) {
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X,
                                                  p->scale.Y, p->scale.Z);
        } else {
                /* !!! Make the piece a bit bigger in off screen buffer for selection
                   purposes so that we always sure that we tapped it correctly by
                   finger.*/
                scaleMatrix = GLKMatrix4MakeScale(p->scale.X + 0.2,
                                                  p->scale.Y + 0.2, p->scale.Z + 0.2);
        }

        modelViewMatrix = GLKMatrix4Multiply(modelViewMatrix, scaleMatrix);

        self.effect.transform.modelviewMatrix = modelViewMatrix;

        type = p->type;

        if (mode == DM_RENDER) {
                /* !!! Use real pieces color and light on for normal drawing !!! */
                GLKVector4 color[pcLast] = {
                        [pcWhite] = whitesColor,
                        [pcBlack] = blacksColor
                };
                self.effect.constantColor = color[p->color];
                self.effect.light0.enabled = GL_TRUE;
        } else {
                /* !!! Use piece seal for color. Important to turn light off !!! */
                self.effect.light0.enabled = GL_FALSE;
                self.effect.constantColor = GLKVector4Make(p->seal / 255.0f,
                                                           0.0f, 0.0f, 0.0f);
        }

        /* Actually normal render the piece using it geometry buffers. */
        [self renderPiece:type];

        [self popMatrix];
}

This is how to use the functions shown above.

- (IBAction) tapGesture:(id)sender
{
        if ([(UITapGestureRecognizer *)sender state] == UIGestureRecognizerStateEnded) {
                CGPoint tap = [(UITapGestureRecognizer *)sender locationInView:self.view];
                Piece *p = [self findPieceBySeal:[self findSealByPoint:tap]];

                /* !!! Do something with your selected object !!! */
        }
}

This is basically it. You will have very precise picking algorithm that is much better than ray tracing or others.

Here helpers for push/pop matrix things.

- (void)pushMatrix
{
        assert(matrixSP < sizeof(matrixStack) / sizeof(GLKMatrix4));
        matrixStack[matrixSP++] = self.effect.transform.modelviewMatrix;
}

- (void)popMatrix
{
        assert(matrixSP > 0);
        self.effect.transform.modelviewMatrix = matrixStack[--matrixSP];
}

Here also glkview setup/cleanup functions that I used.

- (void)viewDidLoad
{
        [super viewDidLoad];
        self.context = [[[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2] autorelease];
        if (!self.context)
                NSLog(@"Failed to create ES context");

        GLKView *view = (GLKView *)self.view;
        view.context = self.context;
        view.drawableDepthFormat = GLKViewDrawableDepthFormat24;

        [self setupGL];
}

- (void)viewDidUnload
{    
        [super viewDidUnload];

        [self tearDownGL];

        if ([EAGLContext currentContext] == self.context)
                [EAGLContext setCurrentContext:nil];
        self.context = nil;
}

- (void)setupGL
{
        [EAGLContext setCurrentContext:self.context];

        self.effect = [[[GLKBaseEffect alloc] init] autorelease];
        if (self.effect) {
                self.effect.useConstantColor = GL_TRUE;
                self.effect.colorMaterialEnabled = GL_TRUE;
                self.effect.light0.enabled = GL_TRUE;
                self.effect.light0.diffuseColor = GLKVector4Make(1.0f, 1.0f, 1.0f, 1.0f);
        }

        /* !!! Draw antialiased geometry !!! */
        ((GLKView *)self.view).drawableMultisample = GLKViewDrawableMultisample4X;
        self.pauseOnWillResignActive = YES;
        self.resumeOnDidBecomeActive = YES;
        self.preferredFramesPerSecond = 30;

        glDisable(GL_DITHER);
        glEnable(GL_CULL_FACE);
        glEnable(GL_DEPTH_TEST);
        glLineWidth(2.0f);

        /* Load pieces geometry */
        [self loadGeometry];
}

- (void)tearDownGL
{
        drawReady = NO;
        [EAGLContext setCurrentContext:self.context];
        [self unloadGeometry];
}

Hope this helps and may be closes "the picking question" forever :)

OpenGL ES 2.0 Object Picking on iOS (Using Color Coding)

I've actually just finished implementing a colour picking function into my iPhone game, using openGL ES 2.0, using the lighthouse tutorial funny enough.

You should be drawing to the frame buffer.

If you want to read from the frame buffer, then you're correct in that you want to use glReadPixels. More information is here:

http://www.opengl.org/sdk/docs/man/xhtml/glReadPixels.xml

The only thing that's different from the lighthouse tutorial is that you also want to store the alpha values.
Here's a quick function to get the colour of a specific pixel. Feel free to improve it or change it, but it does the job.

+ (void) ProcessColourPick : (GLubyte*) out : (Float32) x : (Float32) y
{
    GLint viewport[4];
    //Get size of screen
    glGetIntegerv(GL_VIEWPORT,viewport);

    GLubyte pixel[4];
    //Read pixel from a specific point
    glReadPixels(x,viewport[3] - y,1,1,
             GL_RGBA,GL_UNSIGNED_BYTE,(void *)pixel);

    out[0] = pixel[0];
    out[1] = pixel[1];
    out[2] = pixel[2];
    out[3] = pixel[3];
}

Hope this helps.

How do I select an object in OpenGL ES?

there is a very simple way of doing that with color coding, check the tutorial here: http://www.lighthouse3d.com/opengl/picking/index.php?color1

OpenGL ES 2.0 Ray Picking, far point

We can draw line from near_point to far_point.

  GLKVector4 normalisedVector = GLKVector4Make((2 * position.x / self.view.bounds.size.width - 1),
                                                 (2 * (self.view.bounds.size.height-position.y) / self.view.bounds.size.height - 1),  
                                                   -1,
                                                   1);

    GLKMatrix4 inversedMatrix = GLKMatrix4Invert(_modelViewProjectionMatrix, nil);

    GLKVector4 near_point = GLKMatrix4MultiplyVector4(inversedMatrix, normalisedVector);

    near_point.v[3] = 1.0/near_point.v[3];
    near_point = GLKVector4Make(near_point.v[0]*near_point.v[3], near_point.v[1]*near_point.v[3], near_point.v[2]*near_point.v[3], 1);

    normalisedVector.z = 1.0;
    GLKVector4 far_point = GLKMatrix4MultiplyVector4(inversedMatrix, normalisedVector);

    far_point.v[3] = 1.0/far_point.v[3];
    far_point = GLKVector4Make(far_point.v[0]*far_point.v[3], far_point.v[1]*far_point.v[3], far_point.v[2]*far_point.v[3], 1);

iOS : OpenGL 2.0 ES moving around an object

This turned out to be an issue where baseEffect was not init before it was being used :(

OpenGL ES color picking on the iPhone

OK, so after 18 hours I've finally fixed my issue. In the render method all I had to do was prevent the presentRenderbuffer call when the render was in SELECT mode. I could kick myself right now!

if (mode == SELECT) {
    glDisable(GL_DITHER);
    glDisable(GL_LIGHTING);
    glDisable(GL_LIGHT0);
}

// Draws the cube object, face by face and adds unique color to each face
[Face1 draw];
[Face2 draw];
[Face3 draw];
[Face4 draw];
[Face5 draw];
[Face6 draw];

if (mode == SELECT) {
    glEnable(GL_DITHER);
    glEnable(GL_LIGHTING);
    glEnable(GL_LIGHT0);
}

// Wrapping presentRenderbuffer with this if statement fixed
// the problem where the unique colors would appear onscreen
if (mode == RENDER) {
    [context presentRenderbuffer:GL_RENDERBUFFER_OES];
}

I hope this may help someone else in future :o)

picking code for opengl-es on ios

You are correct, a much better way is to render the object ids to the back buffer and read back a particular pixel (or block of pixels).

(If you're doing a lot of selection, you could even use a second offscreen renderbuffer and generate the object ids every frame in a single render pass.)

But you will have to write your own view code to allocate offscreen render buffers, depth buffers, and whatnot. GLKView is a convenience class, a high level wrapper, and the Apple doco specifically says not to mess with the underlying implementation.

Setting up your own GL render buffers isn't too difficult, and there's example code all over the place. I've used the example code on the Apple dev site and from the OpenGL SuperBible.

---

Related Topics