Using Layers and Bitmask with Raycast in Unity

Using Layers and Bitmask with Raycast in Unity

Most Raycast questions I see use the Layermask incorrectly. Although it works for them by luck but they usually run into issues when they actually want to exclude a GameObject from Raycast.

This answer is made to cover all those scenarios that a person would want to use the Layer to filter GameObjects when performing raycast.

1.How do you raycast to a particular GameObject which is in a layer called "cube"?

First you use LayerMask.NameToLayer("cube") to convert the layer name to layer number. The LayerMask.NameToLayer function returns -1 if the layer does not exist. You must check this before doing any layer bitwise operation.

Raycast to a particular layer ("cube" only):

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");

//Check if layer is valid
if (cubeLayerIndex == -1)
{
    Debug.LogError("Layer Does not exist");
}
else
{
    //Calculate layermask to Raycast to. (Raycast to "cube" layer only)
    int layerMask = (1 << cubeLayerIndex);

    Vector3 fwd = transform.TransformDirection(Vector3.forward);

    //Raycast with that layer mask
    if (Physics.Raycast(transform.position, fwd, 10, layerMask))
    {

    }
}

The most important part of the example above is int layerMask = (1 << cubeLayerIndex);.

To make this answer short, I won't be checking for errors for the rest of the answer.

---

2.What if you have 10 GameObjects in the scene but you only want to raycast to just 2 GameObjects and ignore the rest? How do you do
that?

Let's say that those Object's layers are "cube" and "sphere".

Raycast to the "cube" and "sphere" layers and ignore the rest:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Raycast to "cube" && "sphere" layers only)
int layerMask = (1 << cubeLayerIndex) | (1 << sphereLayerIndex);

---

3.What if you want to raycast to all GameObjects but ignore 1.

Let's say that the GameObject to ignore is in the "cube" layer.

Raycast to all but ignore the "cube" layer:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");

//Calculate layermask to Raycast to. (Ignore "cube" layer)
int layerMask = (1 << cubeLayerIndex);
//Invert to ignore it
layerMask = ~layerMask;

---

4.What if you want to raycast to all GameObjects but ignore 2(multiple) GameObjects.

Again, the layers to ignore are the "cube" and "sphere" layers.

Raycast to all but ignore the "cube" and "sphere" layers:

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Ignore "cube" && "sphere" layers)
int layerMask = ~((1 << cubeLayerIndex) | (1 << sphereLayerIndex));

OR

//Convert Layer Name to Layer Number
int cubeLayerIndex = LayerMask.NameToLayer("cube");
int sphereLayerIndex = LayerMask.NameToLayer("sphere");

//Calculate layermask to Raycast to. (Ignore "cube" && "sphere" layers)
int layerMask = (1 << cubeLayerIndex);
layerMask |= (1 << sphereLayerIndex);
layerMask |= (1 << otherLayerToIgnore1);
layerMask |= (1 << otherLayerToIgnore2);
layerMask |= (1 << otherLayerToIgnore3);
//Invert to ignore it
layerMask = ~layerMask;

---

Finally, if you know the layer index/number, there is no need to use the LayerMask.NameToLayer function. Just insert that layer index there. For example, let's raycast to the "cube" layer which is in index #9. You could just do int layerMask = (1 << 9);.

See the Layers manual to read more about this subjct.

Raycast only on two layers with bit shift to get the bit mask

Manipulating individual bits is mainly done using the &, |, ~ and <</>> operators.

Example (with bytes):

// single value
byte a = 1; // 00000001

// shift "a" 3 bits left
byte b = a << 3; // 00001000

// combine a and b with bitwise or (|)
byte c = a | b; // 00001001

So in your case, to get bit 9 and bit 10 set, do:

int layerMask = ( 1 << 9 ) | ( 1 << 10 );

Notice that we're using | and not ||, which is logical or.

Raycast to specific layers in Unity

Lets start with, you should not be casting a ray in every frame in the update(). Don't know how the Engine may react. You should start by setting a condition to cast the ray, for example when you press a key or similar.

Then, try this instead the 1 << 8 for the mask:

int layer_mask = LayerMask.GetMask("Floor");

//Actually you can add any layer name you want, for example:
//int layer_mask = LayerMask.GetMask("Ground","Enemy","Boxes");

 //do the raycast specifying the mask
 if (Physics.Raycast (ray, out hit, distance, layer_mask))
 {

 } 

Just in case you change the order of the layer accidentally or you modify it in the future, passing the names directly, from my point of view, it's safer.

In case that is not the problem, check what is forward for the GameObject whose transform you are using to cast the ray. Maybe you are just casting the ray in a direction where there is nothing, or at least not Ground.

Checking if the Raycast hit, hits a layer from the LayerMask?

Since you want the raycast to ignore anything that isn't a floor, you can use the mask in the raycast so that it only hits floors. To do this, you want to have your LayerMask floorLayers so that only floors are checked, then in your Raycast:

Physics.Raycast(transform.position, dirDown, out hit, 10f, ~floorLayers.value)

If you do this, any hit will be a floor hit, and you don't need to do any bit math below.

If you later decide you can't use the same mask for the raycast (maybe you really need the raycast to be able to collide with numerous possible targets and then see if the hit object happens to be a floor), then read below.

---

First, you want to compare a mask value that has its bits set to true only for floors, which seems to be floorLayers.value.

If you want to compare a GameObject.layer to a LayerMask.value, you will first need to shift a bit by GameObject.layer.

Also, if you want to check for at least one match between gameObject.layer and floorMask, you want to use binary AND (&) and see if it's not 0 (!= 0 is unnecessary but it can help with readability):

... && (1 << hit.transform.gameObject.layer & floorLayers.value) != 0

This is because GameObject.layer is the index of that single layer, but FloorMask.value is a bit mask that is based on true/false values of various bits. It's meaningless to compare them directly, without making a conversion first.

Unity objects not in layermask getting hit by raycast

Because Layers use bit shifting, you have to do as follow when raycasting :

int layerMask = 1 << controllableLayer;
Physics.Raycast(firstPersonCamera.ScreenPointToRay(Input.GetTouch(0).position), out hit, layerMask)

Otherwise, declare controllableLayer as an integer and do as follow :

int controllableLayer ;

void Awake()
{
    controllableLayer = 1 << LayerMask.NameToLayer("Controllable");
}

void Update()
{
    // ...
    if( Physics.Raycast(firstPersonCamera.ScreenPointToRay(Input.GetTouch(0).position), out hit, controllableLayer) )
    {
        // ...
    }
}

Check these links :

1. https://answers.unity.com/questions/472886/use-layer-name-for-layermask-instead-of-layer-numb.html
2. https://gamedev.stackexchange.com/questions/132132/why-do-unity-layer-masks-need-to-use-bit-shifting

Unity click an object behind other using Raycast

If I understand your question right, you want to click on the yellow sphere (see image) and want the name of the white cube?

There are two possible ways to do that:

1. Ignore Raycast Layer

You could give the yellow sphere the unity standard layer "Ignore Raycast":

Then your code should work (I added the on left mouse click)

    void Update()
{
    if (Input.GetMouseButtonDown(0)) // Click on left mouse button
    {
        RaycastHit hit;
        Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);

        if (Physics.Raycast(ray, out hit))
        {
            Debug.Log(hit.transform.name);
        }
    }
}

2. Use Layer Mask

See: Using Layers and Bitmask with Raycast in Unity

If that's not what you're looking for please give further information and a screen shot of your problem.

How can I decide what objects should be detected and what should be ignored by raycast and how?

You can use the additional optional parameter layerMask of Physics.Raycast

A Layer mask that is used to selectively ignore Colliders when casting a ray.

I would recommned to use a LayerMask

public LayerMask targetLayer;

configure the wanted layers using the Unity Inspector (select every Layer you want to hit) and finally pass it to the raycast like

if(Physics.Raycast(
    cam.transform.position, 
    cam.transform.forward, 
    out whatObjectHit, 
    distanceToSee, 
    targetLayer.value))
{
    ...
}

---

In the example Unity used a bitmask directly (btw. in a very unefficient way in Update):

// Bit shift the index of the layer (8) to get a bit mask
int layerMask = 1 << 8;

// This would cast rays only against colliders in layer 8.
// But instead we want to collide against everything except layer 8. The ~ operator does this, it inverts a bitmask.
layerMask = ~layerMask;

as said I didn't like this way of hardcoding it. Additionally you now only exclude the layer 8 but hit everything else.

To rather include only exactly one specific layer remove the line

layerMask = ~layerMask;

so you only hit layer 8.

Anyway using my way from before is way more flexible and you rather select those layers you actually want to hit instead of excluding one. And using the LayerMask is way easier than having to code some bitmask operations in order to construct the layermask you want.

Problem with Unity's Pyshics.RaycastAll layer parameter

You are using the Layer mask wrong!

See Layers

The Physics.Raycast function takes a bitmask, where each bit determines if a layer will be ignored or not. If all bits in the layerMask are on, we will collide against all colliders. If the layerMask = 0, we will never find any collisions with the ray.

For further understanding of bitmasks

A bit mask basically works like the following:

Each (in this case) layer is represented by one enabled (1) or disabled (0) "bit".

Remember that one int in the binary backend is basically just 4 bytes => 32 bits => 32 available layers in Unity.

Let's say for now there are only 8 layers as an example.

So the bitmask looks like e.g.

0000 0001

Now I'll add the indices on top of it so you understand what I'm talking about

//indices 7654 3210
          0000 0001

this means the layer at index 0 is enabled, anything else is disabled. This would have the int value 1 (= 2^0)

A second example

// indices 7654 3210
           0000 1010

this would mean the layers at index 1 and 3 are enabled, the int value would be 10. **Why? Because it equals

0*2^7 + 0*2^6 + 0*2^5 + 0*2^4 + 1*2^3 + 0*2^2 + 1*2^1 + 0*2^0

= 0 + 0 + 0 + 0 + 8 + 0 + 2 + 0

= 10

In order to write it easier you would usually use bitshift operations like e.g.

var layers = 1<<3 | 1<<1;
// In bits   1000   10

which would result in the last mask example.

---

So the Default layer actually would be

int defaultLayer = 1<<0;

which does not equal 0 but rather 1!

---

In order to not have to do the Layer calculations or hardcode it via string you should rather use a LayerMask field like

public LayerMask raycastLayers;

in your component, configure it via the Inspector and later pass it in like

// This is possible because LayerMask has an implicit conversion from int to LayerMask
//                                                       |
//                                                       v
private RaycastHit[] GetRaycastHits(LayerMask hitLayers = 1<<0)
{
    return Physics.RaycastAll(partToRotate.position, partToRotate.forward, ShootRangeDistance, hitLayers.value);
}

and

var hits = GetRaycastHits(raycastLayers);

Raycast ignores all layers when trying to exclude just one

I think what you might be trying to do is following:

if (Physics.Raycast(ray1, out hit1, maxDistance, ~layer))
{
     Debug.Log("Hit: " + hit1.collider.gameObject.name);
}

The third parameter is suppose to be raycast distance. You can read more about it from documentation here: https://docs.unity3d.com/ScriptReference/Physics.Raycast.html

---

Related Topics