Jquery Validate Plugin - How to Create a Simple Custom Rule

How to use custom function in jQuery Validation Plugin

As per the documentation for the library you linked https://jqueryvalidation.org/jQuery.validator.addMethod/

Define it like this,

jQuery.validator.addMethod("validUser", function(value, element) {
  //value is the val in the textbox, element is the textbox.
  var firstChar = value.slice(0, 1);
  var lastChar = value.slice(-1);

 if (firstChar === '.' || firstChar === '_' || lastChar === '.' || lastChar === '_') 
 {
    return true;
 } else {
    return false;
 }
}, 'Please enter a valid username.');

And then use it like this;

$("#UserName").rules("add", {
   validUser: true
});

JQuery Validation - custom rule

There is no element with id [id='reasonFive']. Try to change your custom method like this:-

$.validator.addMethod("customRule",function(value,element){
   if($('#reasonThree').is(':checked')){
     return $('#other').val().trim().length>0;
  }else{
    return true;
  }
},"Please input a reason");

Demo

How to make custom jQuery-Validate Rule work?

The name method is not part of this plugin. And I don't see anywhere in your OP that you've created it using .addMethod(), so this is going to get the validation stuck before it evaluates your nameCheck rule.

firstName: {
    required: true,
    name: true,  // <- what is this? 
    nameCheck: true
},

jQuery Validation Plugin 1.11.1 rules custom logical operators (= rule1 OR rule2 OR rule3...)

Here is how to do it. You cannot use digits: true in your case since you want it to be either a number or "foo".

var myValidator = $("#myFormId").validate({

  rules: {

    myField: {

      required: false,

      maxlength: 9,

      myCustomRule: true

    }

  }

});

$.validator.addMethod("myCustomRule", function(value, element) {

  return value === "" || value === "foo" || !isNaN(value);

}, "myMessage");

<script src="https://ajax.googleapis.com/ajax/libs/jquery/2.1.1/jquery.min.js"></script>

<script src="https://cdnjs.cloudflare.com/ajax/libs/jquery-validate/1.17.0/jquery.validate.min.js"></script>

<form id="myFormId">

  <input type="text" name="myField" />

</form>

jQuery Validation Adding A Rule

The addMethod function takes two inputs, the first is the name of the rule you are adding. The second is the function itself which does not need naming.
The following code should do what you are hoping for.
The only bit to be careful of is how the date variable is set.
See this page for more examples. jQuery Validator addMethod()

jQuery.validator.addMethod("validateVin", function(vin) {
  var date = Number($("#vehicleyear").val());
  var re;
  if (date >= 1981) {
    re = new RegExp("^[A-HJ-NPR-Z\\d]{8}[\\dX][A-HJ-NPR-Z\\d]{2}\\d{6}$");
  } else if (date < 1981) {
    re = new RegExp("^[A-Z\\d]{2,17}$");
  } else {}
  return vin.match(re);
}, 'Please enter valid VIN.');