Stream Ftp Download to Output

Stream FTP download to output

Found a solution!

Create a socket pair (anonymous pipe?). Use the non-blocking ftp_nb_fget function to write to one end of the pipe, and echo the other end of the pipe.

Tested to be fast (easily 10MB/s on a 100Mbps connection) so there's not much I/O overhead.

Be sure to clear any output buffers. Frameworks commonly buffer your output.

public function echo_contents() {
    /* FTP writes to [0].  Data passed through from [1]. */
    $sockets = stream_socket_pair(STREAM_PF_UNIX, STREAM_SOCK_STREAM, STREAM_IPPROTO_IP);

    if($sockets === FALSE) {
        throw new Exception('Unable to create socket pair');
    }

    stream_set_write_buffer($sockets[0], 0);
    stream_set_timeout($sockets[1], 0);

    try {
        // $this->ftp is an FtpConnection
        $get = $this->ftp->get_non_blocking($this->path, $sockets[0]);

        while(!$get->is_finished()) {
            $contents = stream_get_contents($sockets[1]);

            if($contents !== false) {
                echo $contents;
                flush();
            }

            $get->resume();
        }

        $contents = stream_get_contents($sockets[1]);

        if($contents !== false) {
            echo $contents;
            flush();
        }
    } catch(Exception $e) {
        fclose($sockets[0]);    // wtb finally
        fclose($sockets[1]);

        throw $e;
    }

    fclose($sockets[0]);
    fclose($sockets[1]);
}

// class FtpConnection
public function get_non_blocking($path, $stream) {
    // $this->ftp is the FTP resource returned by ftp_connect
    return new FtpNonBlockingRequest($this->ftp, $path, $stream);
}

/* TODO Error handling. */
class FtpNonBlockingRequest {
    protected $ftp = NULL;
    protected $status = NULL;

    public function __construct($ftp, $path, $stream) {
        $this->ftp = $ftp;

        $this->status = ftp_nb_fget($this->ftp, $stream, $path, FTP_BINARY);
    }

    public function is_finished() {
        return $this->status !== FTP_MOREDATA;
    }

    public function resume() {
        if($this->is_finished()) {
            throw BadMethodCallException('Cannot continue download; already finished');
        }

        $this->status = ftp_nb_continue($this->ftp);
    }
}

Send partial of FTP stream to php://output

Instead of using PHP's FTP extension (eg. ftp_get), it is possible to open a stream using PHP's built-in FTP wrapper.

The following code would stream parts of an FTP-file to php://output:

$start = 300; // First byte to stream
$stop = 499; // Last byte to stream
$url = "ftp://username:password@server.com/path/file.mp3";

$ctx = stream_context_create(array('ftp' => array('resume_pos' => $start)));
$fin = fopen($url, 'r', false, $ctx);
$fout = fopen('php://output', 'w');

stream_copy_to_stream($fin, $fout, $stop-$start+1);

fclose($fin);
fclose($fout);

While stream_copy_to_stream has an $offset parameter, using it resulted in an error because the stream was not seekable. Using the context option resume_pos worked fine however.

what decides ftp download and stream?

If using FTP, streaming is implemented client side using the REST command (for Start Position), as explained at How does a FTP server resume a download? and (in more detail) at http://cr.yp.to/ftp/retr.html .

Your server therefore needs to allow the REST verb (most do by default). Throttling (flow control) is also managed client side.

Long story:

This mechanism is similar to the strategy used by HTTP too. Streaming, however, is a wide subject. and there are other approaches to streaming. Some protocols provide extra verbs to signal other events like changes of bandwidth/resolution to account for unstable connections (like videoconference / desktop share protocols). Some are more suitable for live broadcasting and others for buffered/stored video.

Nowadays, most Streaming Players like YouTube are web based and built on top of the HTTP protocol. Streaming is achieved using the HTTP RANGE Header and by dividing the media in chunks that can be retrieved separately, as explained in this magnific video: https://www.youtube.com/watch?v=OqQk7kLuaK4 .

Download from FTP and serve result using streams

You can create a FileResult from a stream. By default ASP.NET Core buffers a response before sending it to the browser. This can be a problem with big files. To avoid this you need to disable buffering.

Try using :

Response.BufferOutput = false;
return File(responseStream);

How to stream a file I get from an FTP without storing it locally?

Your problem consists of two parts:

1. Reading FTP file as a stream (see an example with fread(): "PHP: How do I read a .txt file from FTP server into a variable?")

2. Streaming a Response in Symfony2

can I download file from ftp without save it my directory first

You could use ByteArrayOutputStream instead of BufferedOutputStream.

ByteArrayOutputStream outputStream1 = new ByteArrayOutputStream();
boolean success = ftpClient.retrieveFile(remoteFile1, outputStream1);
String fileContent = outputStream1.toString("UTF-8");

New file is getting created during FTP download

The retrieveFile() method always writes a local file, whether or not the remote file exists. Instead, you can use retrieveFileStream() and check the reply code.

A handy list of FTP reply codes is available from Wikipedia. If 550 is received, it means the file does not exist.

Finally, you need to use completePendingCommand() to complete the transaction and a FileOutputStream to write the file.

InputStream inputStream = ftpClient.retrieveFileStream(remoteFile1);
int returnCode = ftpClient.getReplyCode();
if (inputStream == null || returnCode == 550) {
    System.out.println("Remote file does not exist");
} else {
    ftpClient.completePendingCommand();
    byte[] buffer = new byte[inputStream.available()];
    inputStream.read(buffer);
    OutputStream outputStream = new FileOutputStream(downloadFile1);
    outputStream.write(buffer);
    outputStream.close();
}

Download file via PHP script from FTP server to browser with Content-Length header without storing the file on the web server

Just remove the output buffering (ob_start() and the others).

Use just this:

ftp_get($conn_id, "php://output", $file, FTP_BINARY);

---

Though if you want to add Content-Length header, you have to query file size first using ftp_size:

$conn_id = ftp_connect("ftp.example.com");
ftp_login($conn_id, "username", "password");
ftp_pasv($conn_id, true);

$file_path = "remote/path/file.zip";
$size = ftp_size($conn_id, $file_path);

header("Content-Type: application/octet-stream");
header("Content-Disposition: attachment; filename=" . basename($file_path));
header("Content-Length: $size"); 

ftp_get($conn_id, "php://output", $file_path, FTP_BINARY);

(add error handling)

---

For more broad background, see:

List and download clicked file from FTP

---

Related Topics