Access Denied Page With Headless Chrome on Linux While Headed Chrome Works on Windows Using Selenium Through Python

  • Home
  • Languages
  • Python
  • Access Denied Page With Headless Chrome on Linux While Headed Chrome Works on Windows Using Selenium Through Python

Access Denied page with headless Chrome on Linux while headed Chrome works on windows using Selenium through Python

As per your code trials on your Windows local machine non-headless Chrome works perfecto while on Linux server centos7 using headless Chrome you are redirected to the Access Denied page.

<html><head>\n<title>Access Denied</title>\n</head><body>\n<h1>Access Denied</h1>\n \nYou don\'t have permission to access "http://www.newark.com/" on this server.<p>\nReference #18.456cd417.1576243477.e007b9f\n\n\n</p></body></html>

Access Denied

As per the article How to bypass “Access Denied” pages when using Headless Chrome there is a little difference between Chrome when run in headless and when run in headed mode. The core network stack being the same and there being no differences in how the browser transmits requests at the packet level, points us to the content of the request only. On inspecting the requests made from headless and headed Chrome, it was observed headless Chrome is making itself known through it's User-Agent header. The header for headed Chrome was almost similar minus the Headless.

The headless Chrome User-Agent was:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3578.98 Safari/537.36

Solution

So a precise solution would be to set the user-agent as of headed Chrome. The User-Agent for Chrome v79.x being:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36

You can modify your code as follows and execute:

from selenium import webdriver

options = webdriver.ChromeOptions()
options.add_argument('--headless')
options.add_argument('--no-sandbox')
options.add_argument(f'user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36')
driver = webdriver.Chrome(executable_path="/usr/local/bin/chromedriver", chrome_options=options)
driver.get("https://www.google.com")
s = driver.page_source
print((s.encode("utf-8")))
driver.quit()

Executing on Windows 10 OS

Result of executing on windows-10 box:

  • Code Block:

    from selenium import webdriver

    options = webdriver.ChromeOptions()
    options.add_argument("start-maximized")
    options.add_argument('--headless')
    options.add_argument('--no-sandbox')
    options.add_argument(f'user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.79 Safari/537.36')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get("https://www.google.com")
    s = driver.page_source
    print((s.encode("utf-8")))
    driver.quit()

  • Console Output:

    [1214/041553.069:INFO:CONSOLE(0)] "A cookie associated with a resource at http://google.com/ was set with `SameSite=None` but without `Secure`. A future release of Chrome will only deliver cookies marked `SameSite=None` if they are also marked `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5633521622188032.", source: data:, (0)
    [1214/041553.165:INFO:CONSOLE(0)] "A cookie associated with a resource at http://google.com/ was set with `SameSite=None` but without `Secure`. A future release of Chrome will only deliver cookies marked `SameSite=None` if they are also marked `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5633521622188032.", source: https://www.google.com/ (0)
    [1214/041555.336:INFO:CONSOLE(0)] "A cookie associated with a cross-site resource at http://google.co.in/ was set without the `SameSite` attribute. A future release of Chrome will only deliver cookies with cross-site requests if they are set with `SameSite=None` and `Secure`. You can review cookies in developer tools under Application>Storage>Cookies and see more details at https://www.chromestatus.com/feature/5088147346030592 and https://www.chromestatus.com/feature/5633521622188032.", source: https://www.google.com/ (0)
    b'<html itemscope="" itemtype="http://schema.org/WebPage" lang="en-IN"><head><meta charset="UTF-8"><meta content="origin" name="referrer"><meta content="/images/branding/googleg/1x/googleg_standard_color_128dp.png" itemprop="image"><meta content="origin" name="referrer"><title>Google</title><script src="https://apis.google.com/_/scs/abc-static/_/js/k=gapi.gapi.en.7kWSr24wXFc.O/m=gapi_iframes,googleapis_client,plusone/rt=j/sv=1/d=1/ed=1/rs=AHpOoo-i9r7IbCTUQfJ0v-FPhRKRS8aihQ/cb=gapi.loaded_0" nonce="M5SQ7iANEaS7TdDvxjlnQA==" async=""></script><script nonce="M5SQ7iANEaS7TdDvxjlnQA==">(function(){window.google={kEI:\'0xr0Xe_oJNiV4-EP3NWg2As\',kEXPI:\'31\',authuser:0,kscs:\'c9c918f0_0xr0Xe_oJNiV4-EP3NWg2As\',kGL:\'IN\',kBL:\'9hVE\'};google.sn=\'webhp\';google.kHL=\'en-IN\';google.jsfs=\'Ffpdje\';})();(function(){google.lc=[];google.li=0;google.getEI=function(a){for(var b;a&&(!a.getAttribute||!(b=a.getAttribute("eid")));)a=a.parentNode;return b||google.kEI};google.getLEI=function(a){for(var b=null;a&&(!a.getAttribute||!(b=a.getAttribute("leid")));)a=a.parentNode;return b};google.https=function(){return"https:"==window.location.protocol};google.ml=function(){return null};google.time=function(){return(new Date).getTime()};google.log=function(a,b,e,c,g){if(a=google.logUrl(a,b,e,c,g)){b=new Image;var d=google.lc,f=google.li;d[f]=b;b.onerror=b.onload=b.onabort=function(){delete d[f]};google.vel&&google.vel.lu&&google.vel.lu(a);b.src=a;google.li=f+1}};google.logUrl=function(a,b,e,c,g){var d="",f=google.ls||"";e||-1!=b.search("&ei=")||(d="&ei="+google.getEI(c),-1==b.search("&lei=")&&(c=google.getLEI(c))&&(d+="&lei="+c));c="";!e&&google.cshid&&-1==b.search("&cshid=")&&"slh"!=a&&(c="&cshid="+google.cshid);a=e||"/"+(g||"gen_204")+"?atyp=i&ct="+a+"&cad="+b+d+f+"&zx="+google.time()+c;/^http:/i.test(a)&&google.https()&&(google.ml(Error("a"),!1,{src:a,glmm:1}),a="");return a};}).call(this);(function(){google.y={};google.x=function(a,b){if(a)var c=a.id;else{do c=Math.random();while(google.y[c])}google.y[c]=[a,b];return!1};google.lm=[];google.plm=function(a){google.lm.push.apply(google.lm,a)};google.lq=[];google.load=function(a,b,c){google.lq.push([[a],b,c])};google.loadAll=function(a,b){google.lq.push([a,b])};}).call(this);google.f={};(function(){document.documentElement.addEventListener("submit",function(b){var a;if(a=b.target){var c=a.getAttribute("data-submitfalse");a="1"==c||"q"==c&&!a.elements.q.value?!0:!1}else a=!1;a&&(b.preventDefault(),b.stopPropagation())},!0);}).call(this);(function(){google.hs={h:true,sie:false};})();(function(){google.c={gl:false,lhc:false};(function(){var e=window.performance;var g=function(a,b,c,d){a.addEventListener?a.removeEventListener(b,c,d||!1):a.attachEvent&&a.detachEvent("on"+b,c)},h=function(a,b,c,d){a.addEventListener?a.addEventListener(b,c,d||!1):a.attachEvent&&a.attachEvent("on"+b,c)};google.timers={};google.startTick=function(a){google.timers[a]={t:{start:google.time()},e:{},m:{}}};google.tick=function(a,b,c){google.timers[a]||google.startTick(a);c=void 0!==c?c:google.time();b instanceof Array||(b=[b]);for(var d=0,f;f=b[d++];)google.timers[a].t[f]=c};google.c.e=function(a,b,c){google.timers[a].e[b]=c};google.c.b=function(a){var b=google.timers.load.m;b[a]&&google.ml(Error("a"),!1,{m:a});b[a]=!0};google.c.u=function(a){var b=google.timers.load.m;if(b[a]){b[a]=!1;for(a in b)if(b[a])return;google.csiReport()}else google.ml(Error("b"),!1,{m:a})};google.rll=function(a,b,c){var d=function(f){c(f);g(a,"load",d);g(a,"error",d)};h(a,"load",d);b&&h(a,"error",d)};google.aft=function(a){a.setAttribute("data-iml",google.time())};google.startTick("load");var k=google.timers.load;a:{var l=k.t;if(e){var m=e.timing;if(m){var n=m.navigationStart,p=m.responseStart;if(p>n&&p<=l.start){l.start=p;k.wsrt=p-n;break a}}e.now&&(k.wsrt=Math.floor(e.now()))}}google.c.b("pr");google.c.b("xe");if(google.c.gl){var q=function(a){a&&google.aft(a.target)};h(document.documentElement,"load",q,!0);google.c.glu=function(){g(document.documentElement,"load",q,!0)}};}).call(this);})();(function(){var b=[function(){google.tick&&google.tick("load","dcl")}];google.dclc=function(a){b.length?b.push(a):a()};function c(){for(var a;a=b.shift();)a()}window.addEventListener?(document.addEventListener("DOMContentLoaded",c,!1),window.addEventListener("load",c,!1)):window.attachEvent&&window.attachEvent("onload",c);}).call(this);(function(){var b=[];google.jsc={xx:b,x:function(a){b.push(a)},mm:[],m:function(a){google.jsc.mm.length||(google.jsc.mm=a)}};}).call(this);(function(){var f=this||self,g=Date.now||function(){return+new Date};var y={};var aa=function(a,c){if(null===c)return!1;if("contains"in a&&1==c.nodeType)return a.contains(c);if("compareDocumentPosition"in a)return a==c||!!(a.compareDocumentPosition(c)&16);for(;c&&a!=c;)c=c.parentNode;return c==a};var ba=function(a,c){return function(d){d||(d=window.event);return c.call(a,d)}},A=function(a){a=a.target||a.srcElement;!a.getAttribute&&a.parentNode&&(a=a.parentNode);return a},B="undefined"!=typeof navigator&&/Macintosh/.test(navigator.userAgent),ca="undefined"!=typeof navigator&&!/Opera/.test(navigator.userAgent)&&/WebKit/.test(navigator.userAgent),da={A:1,INPUT:1,TEXTAREA:1,SELECT:1,BUTTON:1},ea=function(){this._mouseEventsPrevented=!0},F={A:13,BUTTON:0,CHECKBOX:32,COMBOBOX:13,FILE:0,GRIDCELL:13,LINK:13,LISTBOX:13,MENU:0,MENUBAR:0,MENUITEM:0,MENUITEMCHECKBOX:0,MENUITEMRADIO:0,OPTION:0,RADIO:32,RADIOGROUP:32,RESET:0,SUBMIT:0,SWITCH:32,TAB:0,TREE:13,TREEITEM:13},G={CHECKBOX:!0,FILE:!0,OPTION:!0,RADIO:!0},H={COLOR:!0,DATE:!0,DATETIME:!0,"DATETIME-LOCAL":!0,EMAIL:!0,MONTH:!0,NUMBER:!0,PASSWORD:!0,RANGE:!0,SEARCH:!0,TEL:!0,TEXT:!0,TEXTAREA:!0,TIME:!0,URL:!0,WEEK:!0},fa={A:!0,AREA:!0,BUTTON:!0,DIALOG:!0,IMG:!0,INPUT:!0,LINK:!0,MENU:!0,OPTGROUP:!0,OPTION:!0,PROGRESS:!0,SELECT:!0,TEXTAREA:!0};var I=function(){this.h=this.a=null},K=function(a,c){var d=J;d.a=a;d.h=c;return d};I.prototype.g=function(){var a=this.a;this.a&&this.a!=this.h?this.a=this.a.__owner||this.a.parentNode:this.a=null;return a};var L=function(){this.i=[];this.a=0;this.h=null;this.j=!1};L.prototype.g=function(){if(this.j)return J.g();if(this.a!=this.i.length){var a=this.i[this.a];this.a++;a!=this.h&&a&&a.__owner&&(this.j=!0,K(a.__owner,this.h));return a}return null};var J=new I,M=new L;var O=function(){this.o=[];this.a=[];this.g=[];this.j={};this.h=null;this.i=[];N(this,"_custom")},ha="undefined"!=typeof navigator&&/iPhone|iPad|iPod/.test(navigator.userAgent),P=String.prototype.trim?function(a){return a.trim()}:function(a){return a.replace(/^\\s+/,"").replace(/\\s+$/,"")},ia=/\\s*;\\s*/,ma=function(a,c){return function p(b,l){l=void 0===l?!0:l;var m=c;if("_custom"==m){m=b.detail;if(!m||!m._type)return;m=m._type}if("click"==m&&(B&&b.metaKey||!B&&b.ctrlKey||2==b.which||null==b.which&&\n4==b.button||b.shiftKey))m="clickmod";else{var k=b.which||b.keyCode;ca&&3==k&&(k=13);if(13!=k&&32!=k)k=!1;else{var e=A(b),n;(n="keydown"!=b.type||!!(!("getAttribute"in e)||(e.getAttribute("type")||e.tagName).toUpperCase()in H||"BUTTON"==e.tagName.toUpperCase()||e.type&&"FILE"==e.type.toUpperCase()||e.isContentEditable)||b.ctrlKey||b.shiftKey||b.altKey||b.metaKey||(e.getAttribute("type")||e.tagName).toUpperCase()in G&&32==k)||((n=e.tagName in da)||(n=e.getAttributeNode("tabindex"),n=null!=n&&n.specified),n=!(n&&!e.disabled));if(n)k=!1;else{n=(e.getAttribute("role")||e.type||e.tagName).toUpperCase();var q=!(n in F)&&13==k;e="INPUT"!=e.tagName.toUpperCase()||!!e.type;k=(0==F[n]%k||q)&&e}}k&&(m="clickkey")}e=b.srcElement||b.target;k=Q(m,b,e,"",null);b.path?(M.i=b.path,M.a=0,M.h=this,M.j=!1,n=M):n=K(e,this);for(;q=n.g();){var h=q;var r=void 0;var v=h;q=m;var t=v.__jsaction;if(!t){var z;t=null;"getAttribute"in v&&(t=v.getAttribute("jsaction"));if(z=t){t=y[z];if(!t){t={};for(var C=z.split(ia),ja=C?C.length:0,D=0;D<ja;D++){var x=C[D];if(x){var E=x.indexOf(":"),R=-1!=E,ka=R?P(x.substr(0,E)):"click";x=R?P(x.substr(E+1)):x;t[ka]=x}}y[z]=t}v.__jsaction=t}else t=la,v.__jsaction=t}v=t;"maybe_click"==q&&v.click?(r=q,q="click"):"clickkey"==q?q="click":"click"!=q||v.click||(q="clickonly");r={m:r?r:q,action:v[q]||"",event:null,s:!1};k=Q(r.m,r.event||b,e,r.action||"",h,k.timeStamp);if(r.s||r.action)break}k&&"touchend"==k.eventType&&(k.event._preventMouseEvents=ea);if(r&&r.action){if(e="clickkey"==m)e=A(b),e=(e.type||\ne.tagName).toUpperCase(),(e=32==(b.which||b.keyCode)&&"CHECKBOX"!=e)||(e=A(b),n=e.tagName.toUpperCase(),r=(e.getAttribute("role")||"").toUpperCase(),e="BUTTON"===n||"BUTTON"===r?!0:!(e.tagName.toUpperCase()in fa)||"A"===n||"SELECT"===n||(e.getAttribute("type")||e.tagName).toUpperCase()in G||(e.getAttribute("type")||e.tagName).toUpperCase()in H?!1:!0);e&&(b.preventDefault?b.preventDefault():b.returnValue=!1);if("mouseenter"==m||"mouseleave"==m)if(e=b.relatedTarget,!("mouseover"==b.type&&"mouseenter"==\nm||"mouseout"==b.type&&"mouseleave"==m)||e&&(e===h||aa(h,e)))k.action="",k.actionElement=null;else{m={};for(var u in b)"function"!==typeof b[u]&&"srcElement"!==u&&"target"!==u&&(m[u]=b[u]);m.type="mouseover"==b.type?"mouseenter":"mouseleave";m.target=m.srcElement=h;m.bubbles=!1;k.event=m;k.targetElement=h}}else k.action="",k.actionElement=null;h=k;a.h&&!h.event.a11ysgd&&(u=Q(h.eventType,h.event,h.targetElement,h.action,h.actionElement,h.timeStamp),"clickonly"==u.eventType&&(u.eventType="click"),a.h(u,!0));if(h.actionElement){if(a.h){!h.actionElement||"A"!=h.actionElement.tagName||"click"!=h.eventType&&"clickmod"!=h.eventType||(b.preventDefault?b.preventDefault():b.returnValue=!1);var w=a.h(h);if(w&&l){p.call(this,w,!1);return}}else{if((u=f.document)&&!u.createEvent&&u.createEventObject)try{w=u.createEventObject(b)}catch(pa){w=b}else w=b;h.event=w;a.i.push(h)}if("touchend"==h.event.type&&h.event._mouseEventsPrevented){w=h.event;for(var qa in w);g()}}}},Q=function(a,c,d,b,l,p){return{eventType:a,event:c,targetElement:d,action:b,actionElement:l,timeStamp:p||g()}},la={},na=function(a,c){return function(d){var b=a,l=c,p=!1;"mouseenter"==b?b="mouseover":"mouseleave"==b&&(b="mouseout");if(d.addEventListener){if("focus"==b||"blur"==b||"error"==b||"load"==b)p=!0;d.addEventListener(b,l,p)}else d.attachEvent&&("focus"==b?b="focusin":"blur"==b&&(b="focusout"),l=ba(d,l),d.attachEvent("on"+b,l));return{m:b,l:l,capture:p}}},N=function(a,c){if(!a.j.hasOwnProperty(c)){var d=ma(a,c),b=na(c,d);a.j[c]=d;a.o.push(b);for(d=0;d<a.a.length;++d){var l=a.a[d];l.g.push(b.call(null,l.a))}"click"==c&&N(a,"keydown")}};O.prototype.l=function(a){return this.j[a]};var V=function(a,c){var d=new oa(c),b;a:{for(b=0;b<a.a.length;b++)if(S(a.a[b],c)){b=!0;break a}b=!1}if(b)return a.g.push(d),d;T(a,d);a.a.push(d);U(a);return d},U=function(a){for(var c=a.g.concat(a.a),d=[],b=[],l=0;l<a.a.length;++l){var p=a.a[l];W(p,c)?(d.push(p),X(p)):b.push(p)}for(l=0;l<a.g.length;++l)p=a.g[l],W(p,c)?d.push(p):(b.push(p),T(a,p));a.a=b;a.g=d},T=function(a,c){var d=c.a;ha&&(d.style.cursor="pointer");for(d=0;d<a.o.length;++d)c.g.push(a.o[d].call(null,c.a))},Y=function(a,c){a.h=c;a.i&&\n(0<a.i.length&&c(a.i),a.i=null)},oa=function(a){this.a=a;this.g=[]},S=function(a,c){for(var d=a.a,b=c;d!=b&&b.parentNode;)b=b.parentNode;return d==b},W=function(a,c){for(var d=0;d<c.length;++d)if(c[d].a!=a.a&&S(c[d],a.a))return!0;return!1},X=function(a){for(var c=0;c<a.g.length;++c){var d=a.a,b=a.g[c];d.removeEventListener?d.removeEventListener(b.m,b.l,b.capture):d.detachEvent&&d.detachEvent("on"+b.m,b.l)}a.g=[]};var Z=new O;V(Z,window.document.documentElement);N(Z,"click");N(Z,"focus");N(Z,"focusin");N(Z,"blur");N(Z,"focusout");N(Z,"error");N(Z,"load");N(Z,"change");N(Z,"dblclick");N(Z,"input");N(Z,"keyup");N(Z,"keydown");N(Z,"keypress");N(Z,"mousedown");N(Z,"mouseenter");N(Z,"mouseleave");N(Z,"mouseout");N(Z,"mouseover");N(Z,"mouseup");N(Z,"paste");N(Z,"touchstart");N(Z,"touchend");N(Z,"touchcancel");N(Z,"speech");(function(a){google.jsad=function(c){Y(a,c)};google.jsaac=function(c){return V(a,c)};google.jsarc=function(c){X(c);for(var d=!1,b=0;b<a.a.length;++b)if(a.a[b]===c){a.a.splice(b,1);d=!0;break}if(!d)for(d=0;d<a.g.length;++d)if(a.g[d]===c){a.g.splice(d,1);break}U(a)}})(Z);window.gws_wizbind=function(a){return{trigger:function(c){var d=a.l(c.type);d||(N(a,c.type),d=a.l(c.type));var b=c.target||c.srcElement;d&&d.call(b.ownerDocument.documentElement,c)},bind:function(c){Y(a,c)}}}(Z);}).call(this);(function(){window.jsarwt=function(){return!1};}).call(this);var a=window.location,b=a.href.indexOf("#");if(0<=b){var c=a.href.substring(b+1);/(^|&)q=/.test(c)&&-1==c.indexOf("#")&&a.replace("/search?"+c.replace(/(^|&)fp=[^&]*/g,"")+"&cad=h")};var k="function"==typeof Object.defineProperties?Object.defineProperty:function(a,e,c){a!=Array.prototype&&a!=Object.prototype&&(a[e]=c.value)},l="undefined"!=typeof window&&window===this?this:"undefined"!=typeof global&&null!=global?global:this,m=function(a,e){if(e){for(var c=l,b=a.split("."),f=0;f<b.length-1;f++){var d=b[f];d in c||(c[d]={});c=c[d]}b=b[b.length-1];f=c[b];d=e(f);d!=f&&null!=d&&k(c,b,{configurable:!0,writable:!0,value:d})}};m("String.prototype.startsWith",function(a){return a?a:function(e,c){if(null==this)throw new TypeError("The \'this\' value for String.prototype.startsWith must not be null or undefined");if(e instanceof RegExp)throw new TypeError("First argument to String.prototype.startsWith must not be a regular expression");var b=this+"";e+="";for(var f=b.length,d=e.length,h=Math.max(0,Math.min(c|0,b.length)),g=0;g<d&&h<f;)if(b[h++]!=e[g++])return!1;return g>=d}});google.arwt=function(a){a.href=document.getElementById(a.id.substring(a.id.startsWith("vcs")?3:1)).href;return!0};(function(){function e(a){if(!a||/[?&]dsh=1(&|$)/.test(a))return null;if(/[?&]ae=1(&|$)/.test(a)){var c=/[?&]adurl=([^&]+)/.exec(a);if(!c)return null;var 
    .
    .
    .
    <textarea class="csi" name="csi" style="display:none"></textarea>
    .
    .
    .
    </script><script src="/xjs/_/js/k=xjs.s.en_GB.67AovsT7uRo.O/ck=xjs.s.8N3EJVB8puY.L.W.O/am=AAAAgEUAu-6AIP9vBQAAgB0DAAABbsEGC4QhocJYnYAIEA/d=1/exm=Fkg7bd,HcFEGb,IvlUe,MC8mtf,OF7gzc,RMhBfe,T4BAC,TJw5qb,TbaHGc,Y33vzc,cdos,csi,d,hsm,iDPoPb,jsa,mvYTse,tg8oTe,uz938c,vWNDde,ws9Tlc,yQ43ff/ed=1/dg=2/br=1/ct=zgms/rs=ACT90oGiS0w5o50czzw_k4qZG0GKuVBevw/m=WgDvvc,aa,abd,async,dvl,fEVMic,foot,lu,m,mUpTid,mu,sb_wiz,sf,sonic,spch,xz7cCd?xjs=s1" async="" gapi_processed="true"></script></body></html>'

tl; dr

You can find a relevant discussion in:

  • Way to change Google Chrome user agent in Selenium?

How to access a site via a headless driver without being denied permission

Adding in the following code snippet got the page to return for me:

user_agent = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.50 Safari/537.36'    
chrome_options.add_argument('user-agent={0}'.format(user_agent))

The site is obviously checking for headless browsers and then denying them access. Here's an article on avoiding detection: Making Chrome Headless Undetectable

To get the user agent being used by the driver you can run the following command:

driver.execute_script("return navigator.userAgent")

Chromes headless user agent is something like this:

u'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/71.0.3578.98 Safari/537.36'

HeadlessChrome is not regarded as a real user by Barnes and Noble website

You are seeing the following Access Denied error page:

Access Denied

due to presence of the keyword Headless within the user-agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/98.0.4758.102 Safari/537.36

Solution

You can override the default useragent with some other regular UserAgent as follows:

  • Code Block:

    options = Options()
    options.headless = True
    options.add_argument("start-maximized")
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)
    options.add_argument('--disable-blink-features=AutomationControlled')
    options.add_argument("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36")
    s = Service('C:\\BrowserDrivers\\chromedriver.exe')
    driver = webdriver.Chrome(service=s, options=options)
    driver.get("https://www.barnesandnoble.com/w/the-woman-they-could-not-silence-kate-moore/1138489968?ean=9781728242576")
    driver.save_screenshot("barnesandnoble.png")

  • Screenshot:

barnesandnoble


References

You can find a couple of relevant detailed discussions in:

  • How to set a custom name for the user-agent using Selenium and Python
  • How to change the displayed operating system information within the useragent using Selenium and Python?
  • How to change the Google Chrome UserAgent using the ChromeDriver installed through webdriver_manager

python Selenium --headless argument triggers safety feature of the webside

options = webdriver.ChromeOptions()
#options.headless = True
options.add_argument("--window-size=1920,1080")
options.add_argument("--headless")
options.add_argument("--disable-gpu")
options.add_argument(
    "user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36")
browser = webdriver.Chrome(options=options)

Try using user agent , as headless uses a different useragent than normal

Headless chrome cannot detect elements(selenium)

I took your code and modified a bit and executed at my end and here is the execution result:

  • Code Block:

    from selenium import webdriver

    options = webdriver.ChromeOptions()
    options.headless = True
    options.add_argument('window-size=1400,600')
    options.add_experimental_option("excludeSwitches", ["enable-automation"])
    options.add_experimental_option('useAutomationExtension', False)

    driver = webdriver.Chrome(options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
    driver.get('https://moneyforward.com/users/sign_in')
    print(driver.page_source)
    driver.save_screenshot('./save_screenshot_method.png') #Capture the screen
    driver.quit()

  • Console Output:

    <html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">Forbidden</pre></body></html>

  • Browser Snapshot:

save_screenshot_method.png

Analysis

It seems ChromeDriver driven google-chrome-headless is getting detected and being Access Denied the message Forbidden is shown.

Solution

As a solution you can adopt some strategies so ChromeDriver driven Chrome Browsing context doesn't gets detected and you can find a couple of detailed discussions in:

  • Selenium webdriver: Modifying navigator.webdriver flag to prevent selenium detection
  • Selenium “selenium.common.exceptions.NoSuchElementException” when using Chrome

Why my selenium code isn't showing any text?

Bypass Access Denied in headless mode solution use different user-agent, reference. You can use your own user agent, google "my user agent" to get it.

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'
driver_exe = 'chromedriver'
options = ChromeOptions()
options.add_argument("--headless")
options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(options=options)

driver.get("https://www.justdial.com/Bangalore/Bakeries")
x = driver.find_elements_by_class_name("store-name")

for i in x:
    print(i.text)

Using requests and beautifulsoup

from bs4 import BeautifulSoup
import requests

user_agent = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/537.36 (KHTML, like Gecko) ' \
             'Chrome/80.0.3987.132 Safari/537.36'

response = requests.get('https://www.justdial.com/Bangalore/Bakeries', headers={'user-agent': user_agent})
soup = BeautifulSoup(response.text, 'lxml')
stores = soup.select('.store-name')
for store in stores:
    print(store.text.strip())

Output:

Big Mishra Pedha
Just Bake
The Cake Factory
Queen Of Cakeland
SREENIVASA BRAHMINS BAKERY ..
Aubree Eat Play Love Chocol..
Jain Bakes
Ammas Pastries
Facebake
Holige Mane Brahmins Bakery

Related Topics

Two Versions of Python on Linux. How to Make 2.7 the Default

Force Python to Use an Older Version of Module (Than What I Have Installed Now)

Listing Available Devices in Python-Opencv

How to Filter Pandas Dataframe Using 'In' and 'Not In' Like in Sql

Re.Findall Behaves Weird

Limiting Floats to Two Decimal Points

Error "Microsoft Visual C++ 14.0 Is Required (Unable to Find Vcvarsall.Bat)"

How to Copy a Dictionary and Only Edit the Copy

How to Randomly Select an Item from a List

Download Large File in Python With Requests

Extracting Text from Ms Word Files in Python

Simulate Keystroke in Linux With Python

Store Large Data or a Service Connection Per Flask Session

Pandas Conditional Creation of a Series/Dataframe Column

Why Does This Unboundlocalerror Occur (Closure)

Error: Unable to Find Vcvarsall.Bat

How to Access Object Attribute Given String Corresponding to Name of That Attribute

Using Python'S Eval() Vs. Ast.Literal_Eval()


Leave a reply



Submit