How to save the text from a link in bash file?
Using awk
$ cp wp-config.php wp-config.bak
$ curl https://api.wordpress.org/secret-key/1.1/salt/ | awk -F"'" 'NR==FNR {a[$2]=$4;next} {$4=a[$2]}1' OFS="'" - wp-config.php > wp-config.tmp
$ mv wp-config.tmp wp-config.php
$ cat wp-config.php
define( 'AUTH_KEY', '+koF5XB%cw@d[0-ki.7, L<Jjun`r7U)e]N4T`x` -PgG|MBDjVKqn|v;F f9]?1' );
define( 'SECURE_AUTH_KEY', 'zk_kt$P|41U-|Vz:r&wfJW=b6D=_?c`=3.`v1n~K~u1-1Qp|]7&4q:8URK+7i)a+' );
define( 'LOGGED_IN_KEY', '_R.$kMfRnn6jm#hBN-C04610P.Yg.mxLiaSjH{}9%4c0>/{uiAk+}9oCpfj_<wnJ' );
define( 'NONCE_KEY', '40*|=BJN0qo=8O2~x4Rz.A~I+gsF,3UsF?6q/RK5k2;8Dd:o+6o~F&7S_y+^TL-;' );
define( 'AUTH_SALT', 'irwQFh*=:kyToG= aUF;+K%]$E-]]=^@V[l[7@u-pR;ea*Z+d$&YS]i|<-Gj[Bj^' );
define( 'SECURE_AUTH_SALT', ',/rVV-JlDv~`R8ocz*`+T^.CQF.`X3+dA?q5MZHVz(8>&H&r:A#XH?/XV>)L :8s' );
define( 'LOGGED_IN_SALT', '8zf>mA|rmvH%2C`>acJe?*O-TEJ>f0F&K82)Q4u{T1lOH#kG%IbO5>Vw1Bf9b-98' );
define( 'NONCE_SALT', 'p,6nSXBfw~4EySqIu!fp):B1ye1Hu{S{V)ef;m9X;/z}/UA/G|I:2|]Awx,K/k+,' );
SHELL save file from link to disk
!/usr/bin/sh
wget /pathwheretosave/filename websiteurl/filename
The wget command is a command line utility for downloading files from Internet
If you use wget -O /pathwheretosave/newfilename websiteurl/filename
will make you able download a file and change the name.
To execute in background you can launch the script in nohup
nohup filename.sh
Script to download a web page
Simply unzip
ping the new version of your content overtop the old may not be the best solution. What if you remove a file from your site? The local copy will still have it. Also, with a zip-based solution, you're copying EVERY file each time you make a copy, not just the files that have changed.
I recommend you use rsync
instead, to synchronize your site content.
If you set your local documentroot to something like /var/www/mysite/
, an alternative script might then look something like this:
#!/usr/bin/env bash
logtag="`basename $0`[$$]"
logger -t "$logtag" "start"
# Build an array of options for rsync
#
declare -a ropts
ropts=("-a")
ropts+=(--no-perms --no-owner --no-group)
ropts+=(--omit-dir-times)
ropts+=("--exclude ._*")
ropts+=("--exclude .DS_Store")
# Determine previous version
#
if [ -L /var/www/mysite ]; then
linkdest="$(stat -c"%N" /var/www/mysite)"
linkdest="${linkdest##*\`}"
ropts+=("--link-dest '${linkdest%'}'")
fi
now="$(date '+%Y%m%d-%H:%M:%S')"
# Only refresh our copy if flag.txt exists
#
statuscode=$(curl --silent --output /dev/stderr --write-out "%{http_code}" http://www.example.com/flag.txt")
if [ ! "$statuscode" = 200 ]; then
logger -t "$logtag" "no update required"
exit 0
fi
if ! rsync "${ropts[@]}" user@remoteserver:/var/www/mysite/ /var/www/"$now"; then
logger -t "$logtag" "rsync failed ($now)"
exit 1
fi
# Everything is fine, so update the symbolic link and remove the flag.
#
ln -sfn /var/www/mysite "$now"
ssh user@remoteserver rm -f /var/www/flag.txt
logger -t "$logtag" "done"
This script uses a few external tools that you may need to install if they're not already on your system:
- rsync, which you've already read about,
- curl, which could be replaced with wget .. but I prefer curl
- logger, which is probably installed in your system along with syslog or rsyslog, or may be part of the "unix-util" package depending on your Linux distro.
rsync
provides a lot of useful functionality. In particular:
- it tries to copy only what has changed, so that you don't waste bandwidth on files that are the same,
- the
--link-dest
option lets you refer to previous directories to create "links" to files that have not changed, so that you can have multiple copies of your directory with only single copies of unchanged files.
In order to make this go, both the rsync
part and the ssh
part, you will need to set up SSH keys that allow you to connect without requiring a password. That's not hard, but if you don't know about it already, it's the topic of a different question .. or a simple search with your favourite search engine.
You can run this from a crontab every 5 minutes:
*/5 * * * * /path/to/thisscript
If you want to run it more frequently, note that the "traffic" you will be using for every check that does not involve an update is an HTTP GET of the flag.txt file.
wget/curl large file from google drive
WARNING: This functionality is deprecated. See warning below in comments.
Have a look at this question: Direct download from Google Drive using Google Drive API
Basically you have to create a public directory and access your files by relative reference with something like
wget https://googledrive.com/host/LARGEPUBLICFOLDERID/index4phlat.tar.gz
Alternatively, you can use this script: https://github.com/circulosmeos/gdown.pl
Ubuntu: Using curl to download an image
curl
without any options will perform a GET request. It will simply return the data from the URI specified. Not retrieve the file itself to your local machine.
When you do,
$ curl https://www.python.org/static/apple-touch-icon-144x144-precomposed.png
You will receive binary data:
|�>�$! <R�HP@T*�Pm�Z��jU֖��ZP+UAUQ@�
��{X\� K���>0c�yF[i�}4�!�V̧�H_�)nO#�;I��vg^_ ��-Hm$$N0.
���%Y[�L�U3�_^9��P�T�0'u8�l�4 ...
In order to save this, you can use:
$ curl https://www.python.org/static/apple-touch-icon-144x144-precomposed.png > image.png
to store that raw image data inside of a file.
An easier way though, is just to use wget
.
$ wget https://www.python.org/static/apple-touch-icon-144x144-precomposed.png
$ ls
.
..
apple-touch-icon-144x144-precomposed.png
Related Topics
What Is the Fastest Way to Find All the File with the Same Inode
How to Dump Part of Binary File
How to Remove Jenkins Completely from Linux
Git Merge Branch of Another Remote
Why Using Dirname in Find Command Gives Dots for Each Match
Gpg: Sorry, No Terminal at All Requested - Can't Get Input
Write-Through Ram Disk, or Massive Caching of File System
Differencebetween Nptl and Posix Threads
Error: "Grep: Argument List Too Long"
Changing Contents of a File Through Shell Script
Securing a Linux Webserver for Public Access
What Does the Mkdir -P Mean in a Script File