TCP Socket communication between processes on Heroku worker dyno
Have you tried Fifo?
http://www.gnu.org/software/libc/manual/html_node/FIFO-Special-Files.html#FIFO-Special-Files
Communicating between two processes on heroku (what port to use)
Processes are isolated and cannot communicate directly with each other.
http://www.12factor.net/processes
There are, however, a few other ways. One is to use a backing service such as Redis, or Postgres to act as an intermediary - another is to use FIFO to communicate.
http://en.wikipedia.org/wiki/FIFO
It is a good thing that your processes are isolated and share-nothing, but you do need to architecture your application slightly differently to accommodate this.
can a Heroku worker job with external socket connection run in parallel?
Yes - Heroku workers can connect to the outside world - however, there is no built in provision for handling the sort of problems that you mention - you'd need to do that bit yourself.
Just look at the workers as a variety of separate EC2 instances.
Deploy only worker dyno to heroku (for firebase-queue)
The first important part, as stated by Yoni, is to tell Heroku that you only need a background worker and not a web worker:
worker: node <path_to_your_worker>
The second important part is: Heroku will launch a web dyno by default. This causes the application to crash if your application does not bind to the port on which web traffic is received. To disable the web dyno and prevent the crash, run the following commands from the commandline in your directory:
$ heroku ps:scale web=0 worker=1
$ heroku ps:restart
This should fix the problem!
What's the difference between a Dyno, a Worker and a Process on Heroku?
A worker is just a name given to a Dyno. You usually have a web
Dyno, but you can create others in your Procfile
and give them arbitrary names.
Many processes can be spawned by one Dyno. For example, the Unicorn web server does this.
You might be able to use Rufus Scheduler for what you need to do, but I think it would be more sensible to use a separate worker Dyno.
How to use a worker dyno to read from a local path?
Different dynos have completely separate filesystems: you'll have to copy the file somewhere persistent, such as S3.
How to communicate Web and Worker dynos with Node.js on Heroku?
As the high-level article on background jobs and queuing suggests, your web dynos will need to communicate with your worker dynos via an intermediate mechanism (often a queue).
To accomplish what it sounds like you're hoping to do follow this general approach:
- Web request is received by the web dyno
- Web dyno adds a job to the queue
- Worker dyno receives job off the queue
- Worker dyno executes job, writing incremental progress to a shared component
- Browser-side polling requests status of job from the web dyno
- Web dyno queries shared component for progress of background job and sends state back to browser
- Worker dyno completes execution of the job and marks it as complete in shared component
- Browser-side polling requests status of job from the web dyno
- Web dyno queries shared component for progress of background job and sends completed state back to browser
As far as actual implementation goes I'm not too familiar with the best libraries in Node.js, but the components that glue this process together are available on Heroku as add-ons.
Queue: AMQP is a well-supported queue protocol and the CloudAMQP add-on can serve as the message queue between your web and worker dynos.
Shared state: You can use one of the Postgres add-ons to share the state of an job being processed or something more performant such as Memcache or Redis.
So, to summarize, you must use an intermediate add-on component to communicate between dynos on Heroku. While this approach involves a little more engineering, the result is a properly-decoupled and scalable architecture.
Related Topics
Why Does .All? Return True on an Empty Array
Ruby Converting String Encoding from Iso-8859-1 to Utf-8 Not Working
Unpermitted Parameters for Dynamic Forms in Rails 4
Integrating @Font-Face Files into Rails Asset Pipeline
How Might I Simulate a Private Browsing Experience in Watir? (Selenium)
When Do You Need a Require in a Rails Gemfile
Rails 3, Displaying Jpg Images from the Local File System Above Rails_Root
Cast Between String and Classname
Controller Can Not Detect Ajax Requests
Ruby 2.7 Says Uri.Escape Is Obsolete, What Replaces It
Activerecord Connection Warning. (Database Connections Will Not Be Closed Automatically)
How to Find If Range Is Contained in an Array of Ranges
Couldn't Find User with Id=Sign_Out
Ruby on Rails - Link_To Button/Css
Optional Parens in Ruby for Method with Uppercase Start Letter
How to Rescue from a Require "Gem_Name" When the Gem Is Not Installed
Erroneous "Insecure World Writable Dir Foo in Path" When Running Ruby Script