Capture Generated Dynamic Content at Server Side

Capture generated dynamic content at server side

If the request is idempotent (such as GET requests are), then just use java.net.URL to get an InputStream of the JSP output. E.g.

InputStream input = new URL("http://localhost/context/page.jsp").openStream();

If the request is not idempotent (such as POST requests are), then you need to create a Filter which wraps the ServletResponse with a custom implementation of the PrintWriter with the five write() methods been overridden wherein you copy output into some buffer/builder which you store in the session or a temporary folder at local disk file system so that it can be accessed afterwards in the subsequent requests. E.g.

package mypackage;

import java.io.IOException;
import java.io.OutputStreamWriter;
import java.io.PrintWriter;
import java.io.Writer;

import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import javax.servlet.http.HttpServletResponseWrapper;

public class CopyResponseFilter implements Filter {

    public void init(FilterConfig config) throws ServletException {
        // NOOP.
    }

    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) 
        throws IOException, ServletException
    {
        // Set character encoding for better world domination.
        response.setCharacterEncoding("UTF-8");

        // Create copy writer.
        HttpServletResponse httpResponse = (HttpServletResponse) response;
        CopyWriter copyWriter = new CopyWriter(new OutputStreamWriter(
            httpResponse.getOutputStream(), httpResponse.getCharacterEncoding()));

        // Filter request with response which is wrapped with new writer.
        chain.doFilter(request, wrapResponse(httpResponse, copyWriter));

        // Store the copy writer afterwards in session so that it's available in next request.
        HttpServletRequest httpRequest = (HttpServletRequest) request;
        httpRequest.getSession().setAttribute("copyWriter", copyWriter);
    }

    public void destroy() {
        // NOOP.
    }

    private static HttpServletResponse wrapResponse
        (final HttpServletResponse response, final PrintWriter writer)
    {
        return new HttpServletResponseWrapper(response) {
            public PrintWriter getWriter() throws IOException {
                return writer;
            }
        };
    }

}

class CopyWriter extends PrintWriter {

    StringBuilder copy = new StringBuilder();

    public CopyWriter(Writer out) {
        super(out);
    }

    public void write(int c) {
        copy.append((char) c); // It is actually a char, not an int.
        super.write(c);
        super.flush();
    }

    public void write(char[] chars) {
        copy.append(chars);
        super.write(chars);
        super.flush();
    }

    public void write(char[] chars, int offset, int length) {
        copy.append(chars, offset, length);
        super.write(chars, offset, length);
        super.flush();
    }

    public void write(String string) {
        copy.append(string);
        super.write(string);
        super.flush();
    }

    public void write(String string, int offset, int length) {
        copy.append(string, offset, length);
        super.write(string, offset, length);
        super.flush();
    }

    public String getCopy() {
        return copy.toString();
    }

}

You can access the final output in any servlet of the subsequent request (note that you cannot access it in any servlet of the current request, because it's already too late to do something with it) by just accessing the CopyWriter in the session:

CopyWriter copyWriter = (CopyWriter) request.getSession().getAttribute("copyWriter");
String outputOfPreviousRequest = copyWriter.getCopy();

Note that you should map this filter on an url-pattern covering the JSP pages of interest and thus not on /* or so, otherwise it would run on static files (css, js, images, etc) which are included in the same JSP as well.

Also note that multiple requests inside the same session would override each other, it's up to you to distinguish between those requests by using a proper url-pattern or another way of storing it in session, e.g. in flavor of a Map<URL, CopyWriter> or so.

Hope this helps.

Dynamically-created content for download without writing a file on server-side in Vaadin Flow web app

Caveat: I am no expert on this matter. My example code presented here seems to be functioning properly. I cobbled this solution together by studying the limited documentation and reading many other posts on the web. Mine may not be the best solution.

For more information, see the Dynamic Content page of the Vaadin manual.

We have three major pieces in your Question:

Widget on Vaadin web app page to offer the user a download.
Dynamic content creator
Default name of file being created on user’s machine

I have a solution to the first two, but not the third.

Download widget

As mentioned in the Question, we do use the Anchor widget (see Javadoc).

We define a member variable on our layout.

private Anchor anchor;

We instantiate by passing a StreamResource object. This class is defined in Vaadin. Its job here is to wrap a class of our making that will produce an implementation extending the Java class InputStream.

An input stream provides data one octet at a time by returning from its read method an int whose value is numeric number of the intended octet, 0-255. When reaching the end of the data, a negative one is returned by read.

In our code, we have implemented a makeStreamOfContent method to act as the InputStream factory.

private InputStream makeInputStreamOfContent ( )
{
    return GenerativeInputStream.make( 4 );
}

When instantiating our StreamResource, we pass a method reference that refers to that makeInputStreamOfContent method. We are getting a bit abstract here, as no input stream nor any data is yet being generated. We are just setting the stage; the action occurs later.

The first argument passed to new StreamResource is the default name of the file to be created on the user’s client-side machine. In this example, we are using the unimaginative name of report.text.

anchor = 
    new Anchor( 
        new StreamResource( "report.text" , this :: makeInputStreamOfContent ) , 
        "Download generated content" 
    )
;

Next, we set an attribute of download on the HTML5 anchor element. This attribute indicates to the browser that we intend to have the target downloaded when the user clicks the link.

anchor.getElement().setAttribute( "download" , true );

You can display an icon by wrapping the anchor widget inside a Button.

downloadButton = new Button( new Icon( VaadinIcon.DOWNLOAD_ALT ) );
anchor.add( downloadButton );

If using an icon like this, you should drop the text label from the Anchor widget. Instead, place any desired text in the Button. So we would pass empty string ("") to new Anchor, and pass the label text as a first argument to new Button.

anchor = 
    new Anchor( 
        new StreamResource( "report.text" , this :: makeInputStreamOfContent ) , 
        "" 
    )
;
anchor.getElement().setAttribute( "download" , true );
downloadButton = 
    new Button( 
        "Download generated content" , 
        new Icon( VaadinIcon.DOWNLOAD_ALT ) 
    )
;
anchor.add( downloadButton );

Dynamic content creator

We need to implement a InputStream subclass, to give to our download widget.

The InputStream abstract class provides implementations of all but one of its methods. We need implement only the read method to satisfy the needs of our project.

Here is one possible such implementation. When you instantiate a GenerativeInputStream object, pass the number of rows you want to generate. Data is generated one row at a time, then fed octet-by-octet to the client. When done with that row, another row is generated. So we conserve memory by working only with one row at a time.

The octets fed to the client are the octets making up the UTF-8 text of our row. Each character of intended text may consist of one or more octets. If you do not understand this, read the entertaining and informative post The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!) by Joel Spolsky.

package work.basil.example;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.Charset;
import java.time.Instant;
import java.util.Objects;
import java.util.Optional;
import java.util.UUID;
import java.util.function.IntSupplier;

// Generates random data on-the-fly, to simulate generating a report in a business app.
//
// The data is delivered to the calling program as an `InputStream`. Data is generated
// one line (row) at a time. After a line is exhausted (has been delivered octet by octet
// to the client web browser), the next line is generated. This approach conserves memory
// without materializing the entire data set into RAM all at once.
//
// By Basil Bourque. Use at your own risk.
// © 2020 Basil Bourque. This source code may be used by others agreeing to the terms of the ISC License.
// https://en.wikipedia.org/wiki/ISC_license
public class GenerativeInputStream extends InputStream
{
    private int rowsLimit, nthRow;
    InputStream rowInputStream;
    private IntSupplier supplier;
    static private String DELIMITER = "\t";
    static private String END_OF_LINE = "\n";
    static private int END_OF_DATA = - 1;

    // --------|  Constructors  | -------------------
    private GenerativeInputStream ( int countRows )
    {
        this.rowsLimit = countRows;
        this.nthRow = 0;
        supplier = ( ) -> this.provideNextInt();
    }

    // --------|  Static Factory  | -------------------
    static public GenerativeInputStream make ( int countRows )
    {
        var gis = new GenerativeInputStream( countRows );
        gis.rowInputStream = gis.nextRowInputStream().orElseThrow();
        return gis;
    }

    private int provideNextInt ( )
    {
        int result = END_OF_DATA;

        if ( Objects.isNull( this.rowInputStream ) )
        {
            result = END_OF_DATA; // Should not reach this point, as we checked for null in the factory method and would have thrown an exception there.
        } else  // Else the row input stream is *not*  null, so read next octet.
        {
            try
            {
                result = rowInputStream.read();
                // If that row has exhausted all its octets, move on to the next row.
                if ( result == END_OF_DATA )
                {
                    Optional < InputStream > optionalInputStream = this.nextRowInputStream();
                    if ( optionalInputStream.isEmpty() ) // Receiving an empty optional for the input stream of a row means we have exhausted all the rows.
                    {
                        result = END_OF_DATA; // Signal that we are done providing data.
                    } else
                    {
                        rowInputStream = optionalInputStream.get();
                        result = rowInputStream.read();
                    }
                }
            }
            catch ( IOException e )
            {
                e.printStackTrace();
            }
        }

        return result;
    }

    private Optional < InputStream > nextRowInputStream ( )
    {
        Optional < String > row = this.nextRow();
        // If we have no more rows, signal the end of data feed with an empty optional.
        if ( row.isEmpty() )
        {
            return Optional.empty();
        } else
        {
            InputStream inputStream = new ByteArrayInputStream( row.get().getBytes( Charset.forName( "UTF-8" ) ) );
            return Optional.of( inputStream );
        }
    }

    private Optional < String > nextRow ( )
    {
        if ( nthRow <= rowsLimit ) // If we have another row to give, give it.
        {
            nthRow++;
            String rowString = UUID.randomUUID() + DELIMITER + Instant.now().toString() + END_OF_LINE;
            return Optional.of( rowString );
        } else // Else we have exhausted the rows. So return empty Optional as a signal.
        {
            return Optional.empty();
        }
    }

    // --------|  `InputStream`  | -------------------
    @Override
    public int read ( ) throws IOException
    {
        return this.provideNextInt();
    }
}

Default file name

I cannot find a way to accomplish the last part, defaulting the name of the file to include the moment when content was generated.

I even posted a Question on Stack Overflow on this point: Download with file name defaulting to date-time of user event in Vaadin Flow app

The problem is that the URL behind the link widget is created once, when the page was loaded and that Anchor widget was instantiated. After that, while the user is reading the page, time passes. When the user eventually clicks the link to initiate the download, the current moment is later than the moment recorded in the URL.

There seems to be no simple way to update that URL to the current moment of the user's click event or download event.

Tips

By the way, for real work I would not be building the exported rows with my own code. I would instead be using a library such as Apache Commons CSV to write the Tab-delimited or Comma-separated values (CSV) content.

Resources

Forum: Vaadin 10 Let user download a file
Forum: Image from byte array
Manual: Dynamic Content

SEO for dynamically generated content in SPA (single page application) using node.js + express.js on server side

Use prerender.io as first middleware of your pipe:

app.use(require('prerender-node').set('prerenderToken', 'YOUR_TOKEN')); app.get('*', (req, res) => res.render('index.html'));

How to grab dynamic content on website and save it?

Since Gmail doesn't provide any API to get this information, it sounds like you want to do some web scraping.

Web scraping (also called Web
harvesting or Web data extraction) is
a computer software technique of
extracting information from websites

There are numerous ways of doing this, as mentioned in the wikipedia article linked before:

Human copy-and-paste: Sometimes even
the best Web-scraping technology can
not replace human’s manual examination
and copy-and-paste, and sometimes this
may be the only workable solution when
the websites for scraping explicitly
setup barriers to prevent machine
automation.

Text grepping and regular expression
matching: A simple yet powerful
approach to extract information from
Web pages can be based on the UNIX
grep command or regular expression
matching facilities of programming
languages (for instance Perl or
Python).

HTTP programming: Static and dynamic
Web pages can be retrieved by posting
HTTP requests to the remote Web server
using socket programming.

DOM parsing: By embedding a
full-fledged Web browser, such as the
Internet Explorer or the Mozilla Web
browser control, programs can retrieve
the dynamic contents generated by
client side scripts. These Web browser
controls also parse Web pages into a
DOM tree, based on which programs can
retrieve parts of the Web pages.

HTML parsers: Some semi-structured
data query languages, such as the XML
query language (XQL) and the
hyper-text query language (HTQL), can
be used to parse HTML pages and to
retrieve and transform Web content.

Web-scraping software: There are many
Web-scraping software available that
can be used to customize Web-scraping
solutions. These software may provide
a Web recording interface that removes
the necessity to manually write
Web-scraping codes, or some scripting
functions that can be used to extract
and transform Web content, and
database interfaces that can store the
scraped data in local databases.

Semantic annotation recognizing: The
Web pages may embrace metadata or
semantic markups/annotations which can
be made use of to locate specific data
snippets. If the annotations are
embedded in the pages, as Microformat
does, this technique can be viewed as
a special case of DOM parsing. In
another case, the annotations,
organized into a semantic layer2,
are stored and managed separated to
the Web pages, so the Web scrapers can
retrieve data schema and instructions
from this layer before scraping the
pages.

And before I continue, please keep in mind the legal implications of all this. I don't know if it's compliant with gmail's terms and I would recommend checking them before moving forward. You might also end up being blacklisted or encounter other issues like this.

All that being said, I'd say that in your case you need some kind of spider and DOM parser to log into gmail and find the data you want. The choice of this tool will depend on your technology stack.

As a ruby dev, I like using Mechanize and nokogiri. Using PHP you could take a look at solutions like Sphider.

Capturing results from a dynamic page

I am using autoit. It is a free scriptig language.
This is the right technology for this problem.

How to dynamically generate and capture user input from modalDialog within a module?

Figured it out. I needed to include the server-side ns function in the dynamic generation of the input fields. ie:

for(i in seq_len(nrow(fieldmapping_table))) {
        fieldmapping_table[i,"DB.Field.Mapping"] <- as.character(selectInput(
          inputId = ns(glue::glue("fieldmap_select_{fieldmapping_table$File.Column[i]}")),
          label = NULL,
          choices = db_cols
        ))
      }

notice the ns() call around the inputId.

How to create dynamic assets in Meteor

The public/ folder intended use is specifically for static assets. Its content is served by the node http server.

If you want to dynamically generate assets on the server, you can rely on iron:router server side routes.

Here is a simple example :

lib/router.js

Router.route("/dynamic-asset/:filename",function(){
  var filename = this.params.filename;
  this.response.setHeader("Content-Disposition",
    "attachment; filename=" + filename);
  this.response.end("Hello World !");
},{
  name: "dynamic-asset",
  where: "server"
});

In server-side route controllers, you get access to this.response which is a standard node HTTP response instance to respond to the client with the correct server generated content. You can query your Mongo collections using the eventual parameters in the URL for example.

client/views/download/download.html

<template name="download">
  {{#linkTo route="dynamic-asset" target="_blank" download=""}}
    Download {{filename}}
  {{/linkTo}}
</template>

client/views/parent/parent.html

<template name="parent">
  {{> download filename="new.txt"}}
</template>

The linkTo block helper must be called in a context where the route parameters are accessible as template helpers. It will generate an anchor tag having an href set to Router.path(route, dataContext). It means that if our server-side route URL is /dynamic-asset/:filename, having a data context where filename is accessible and set to "new.txt" will generate this URL : /dynamic-asset/new.txt.

In this example we set the current data context of the download template to {filename: "new.txt"} thanks to the template invocation syntax.

Note that target="_blank" is necessary to avoid being redirected to the dynamic asset URL inside the current tab, and the download HTML attribute must be set to avoid considering the link as something the browser should open inside a new tab. The download attribute value is irrelevant as it's value will be overriden server-side.

Capture Generated Dynamic Content at Server Side