ISF stands for "Interactive Shader Format", and is a file format that describes a GLSL fragment shader, as well as how to execute and interact with it. The goal of this file format is to provide a simple and minimal interface for image filters and generative video sources that allows them to be interacted with and reused in a generic and modular fashion. ISF is nothing more than a [slightly modified] GLSL fragment shader with a JSON blob at the beginning that describes how to interact with the shader (how many inputs/uniform variables it has, what their names are, what kind of inputs/variables they are, that sort of thing). ISF isn't some crazy new groundbreaking technology- it's just a simple and useful combination of two things that have been around for a while to make a minimal- but highly effective- filter format.
Functionally, you are limited only to what you can accomplish in a GLSL program (vertex + fragment shader, but the frag shader is where most of the action happens in most of the ISF files we've seen so far).
ISF's defining characteristic is the JSON dict at the beginning of your shader which describes it. This JSON dict contains the information describing how to interact with your shader- all the inputs, how many rendering passes there are and how to render them, etc. As a result of this, ISF is a very loose, open-ended file format: it can easily be extended by recognizing more keys in the future if we think of new things that might be nice to have.
When you load an ISF file, the shader code in your file is modified in memory during loading- some variables are declared, and some code is find-and-replaced. If you haven't explicitly created a vertex shader, one will automatically be generated. Each time you render a frame, the INPUTS variable values are passed to the shader, and then the shader is rendered (more than once if desired, potentially into image buffers accessible from different rendering passes and across different frames).
INPUTS you define. You shouldn't declare uniform variables you want ISF to create UI items for- if you do this, your shader won't work (and the error displayed in the ISF Editor app will indicate that you've redeclared a variable). "uniform" samplers are also declared for any persistent or temporary buffers (samplers are automatically declared of the appropriate type- the shader will be recompiled behind the scenes if a texture type for any of the textures in use changes).PASSINDEX is automatically declared, and set to 0 on the first rendering pass. Subsequent passes (defined by the dicts in your PASSES array) increment this int.RENDERSIZE is automatically declared, and is set to the rendering size (in pixels) of the current rendering pass.isf_FragNormCoord is automatically declared. This is a convenience variable, and repesents the normalized coordinates of the current fragment ([0,0] is the bottom-left, [1,1] is the top-right).TIME is automatically declared, and is set to the current rendering time (in seconds) of the shader. This variable is updated once per rendered frame- if a frame requires multiple rendering passes, the variable is only updated once for all the passes.TIMEDELTA is automatically declared, and is set to the time (in seconds) that have elapsed since the last frame was rendered. This value will be 0.0 when rendering the first frame.DATE is automatically declared, and is used to pass the date and time to the shader. The first element of the vector is the year, the second element is the month, the third element is the day, and the fourth element is the time (in seconds) within the day.FRAMEINDEX is automatically declared, and is used to pass the index of the frame being rendered to the shader- this value is 0 when the first frame is rendered, and is incremented after each frame has finished rendering.
<vec4> pixelColor = IMG_PIXEL(<image> imageName, <vec2> pixelCoord);
<vec4> pixelColor = IMG_NORM_PIXEL(<image> imageName, <vec2> normalizedPixelCoord);
<vec2> imageSize = IMG_SIZE(<image> imageName);
IMG_PIXEL() and IMG_NORM_PIXEL() fetch the color of a pixel in an image using either pixel-based coords or normalized coords, respectively, and should be used instead of texture2D() or texture2DRect(). In both functions, "imageName" refers to the variable name of the image you want to work with. This image can come from a variety of sources- an "image"-type input, the target of a previous render pass or a persistent buffer, an imported image, etc: all of these sources of images have a name value associated with them, and this name is what you pass to these functions. For several examples, please check out this .zip of extremely simple shaders that demonstrate the features of ISF, one at a time:
ISF Test/Tutorial filtersIMG_SIZE() returns a two-element vector describing the size of the image in pixels.The ISF file format defines the ability to execute a shader multiple times in the process of rendering a frame for output- each time the shader's executed (each pass), the uniform int variable PASSINDEX is incremented. Details on how to accomplish this are described below in the spec, but the basic process involves adding an array of dicts to the PASSES key in your top-level JSON dict. Each dict in the PASSES array describes a different rendering pass- the ISF host will automatically create buffers to render into, and those buffers (and therefore the results of those rendering passes) can be accessed like any other buffer/input image/imported image (you can render to a texture in one pass, and then read that texture back in and render something else in another pass). The dicts in PASSES recognize a number of different keys to specify different properties of the rendering passes- more details are in the spec below.
ISF files can define persistent buffers. These buffers are images (GL textures) that stay with the ISF file for as long as it exists. This is useful if you want to "build up" an image over time- you can repeatedly query and update the contents of persistent buffers by rendering into them- or if you want to perform calculations across the entire image, storing the results somewhere for later evaluation. Details on exactly how to do this are in the spec (below).
First of all, there are super-simple examples that cover all of this- check out the various "Test__.fs" sample filters located here: ISF Test/Tutorial filters ...you will probably learn more, faster, from the examples than you'll get by reading this document: each example describes a single aspect of the ISF file format, and they're extremely handy for testing, reference, or as a tutorial (the ISF file format is very small).
ISFVSN key, this string will describe the version of the ISF specification this shader was written for. This key should be considered mandatory- if it's missing, the assumption is that the shader was written for version 1.0 of the ISF spec (which didn't specify this key). The string is expected to contain one or more integers separated by dots (eg: '2', or '2.1', or '2.1.1').VSN key, this string will describe the version of this ISF file. This key is completely optional, and its use is up to the host or editor- the goal is to provide a simple path for tracking changes in ISF files. Like the ISFVSN key, this string is expected to contain one or more integers separated by dots.DESCRIPTION key, this string will be displayed as a description associated with this filter in the host app. the use of this key is optional.CATEGORIES key in your top-level dict should store an array of strings. The strings are the category names you want the filter to appear in (assuming the host app displays categories).INPUTS key of your top-level dict should store an array of dictionaries (each dictionary describes a different input- the inputs should appear in the host app in the order they're listed in this array). For each input dictionary:
NAME must be a string, and it must not contain any whitespaces. This is the name of the input, and will also be the variable name of the input in your shader.TYPE must be a string. This string describes the type of the input, and must be one of the following values: "event", "bool", "long", "float", "point2D", "color", "image", "audio", or "audioFFT".MAX key (more on this later in the discussion of MAX).DEFAULT, MIN, MAX, and IDENTITY may be used to further describe value attributes of the input. Note that "image"-type inputs don't have any of these, and that "color"-type inputs use an array of floats to describe min/max/default colors. Everywhere else values are stored as native JSON values where possible (float as float, bool as bool, etc).MAX key- but in this context, MAX specifies the number of samples that the shader wants to receive. This key is optional- if MAX is not defined then the shader will receive audio data with the number of samples that were provided natively. For example, if the MAX of an "audio"-type input is defined as 1, the resulting 1-pixel-wide image is going to accurately convey the "total volume" of the audio wave; if you want a 4-column FFT graph, specify a MAX of 4 on an "audioFFT"-type input, etc.LABEL must be a string. This key is optional- the NAME of an input is the variable name, and as such it can't contain any spaces/etc. The LABEL key provides host sofware with the opportunity to display a more human-readable name. This string is purely for display purposes and isn't used for processing at all.VALUES key stores an array of integer values. This array may have repeats, and the values correspond to the labels. When you choose an item from the pop-up menu, the corresponding value from this array is sent to your shader.LABELS key stores an array of strings. This array may have repeats, and the strings/labels correspond to the array of values.PASSES key should store an array of dictionaries. Each dictionary describes a different rendering pass. This key is optional: you don't need to include it, and if it's not present your effect will be assumed to be single-pass.
TARGET string in the pass dict describes the name of the buffer this pass renders to. The ISF host will automatically create a temporary buffer using this name, and you can read the pixels from this temporary buffer back in your shader in a subsequent rendering pass using this name. By default, these temporary buffers are deleted (or returned to a pool) after the ISF file has finished rendering a frame of output- they do not persist from one frame to another. No particular requirements are made for the default texture format- it's assumed that the host will use a common texture format for images of reasonable visual quality.PERSISTENT key, it indicates that the target buffer will be persistent- that it will be saved across frames, and stay with your effect until its deletion. If you ask the filter to render a frame at a different resolution, persistent buffers are resized to accommodate. Persistent buffers are useful for passing data from one frame to the next- for an image accumulator, or motion blur, for example. This key is optional- if it isn't present (or contains a 0 or false value), the target buffer isn't persistent.FLOAT key, it indicates that the target buffer created by the host will have 32bit float per channel precision. Float buffers are proportionally slower to work with, but if you need precision- for image accumulators or visual persistence projects, for example- then you should use this key. Float-precision buffers can also be used to store variables or values between passes or between frames- each pixel can store four 32-bit floats, so you can render a low-res pass to a float buffer to store values, and then read them back in subsequent rendering passes. This key is optional- if it isn't present (or contains a 0 or false value), the target buffer will be of normal precision.WIDTH or HEIGHT (these keys are optional), that value is expected to be a string with an equation describing the width/height of the buffer. This equation may reference variables: the width and height of the image requested from this filter are passed to the equation as $WIDTH and $HEIGHT, and the value of any other inputs declared in INPUTS can also be passed to this equation (for example, the value from the float input "blurAmount" would be represented in an equation as "$blurAmount"). This equation is evaluated once per frame, when you initially pass the filter a frame (it's not evaluated multiple times if the ISF file describes multiple rendering passes to produce a sigle frame). For more information (constants, built-in functions, etc) on math expression evaluations, please see the documentation for the excellent DDMathParser by Dave DeLong, which is what we're presently using.IMPORTED key describes buffers that will be created for image files that you want ISF to automatically import. This key is optional: you don't need to include it, and if it's not present your ISF file just won't import any external images. The item stored at this key should be a dictionary.
IMPORTED dictionary describes a single image file to import. The key for each item in the IMPORTED dictionary is the name of the buffer as it will be used in your ISF file, and the value for each item in the IMPORTED dictionary is another dictionary describing the file to be imported.PATH key, and the object stored at that key must be a string. This string should describe the path to the image file, relative to the ISF file being evaluated. For example, a file named "asdf.jpg" in the same folder as the ISF file would have the PATH "asdf.jpg", or "./asdf.jpg" (both describe the same location). If the jpg were located in your ISF file's parent directory, its PATH would be "../asdf.jpg", etc.texture2D() or texture2DRect() with IMG_NORM_PIXEL() or IMG_PIXEL(), respectively. Images in ISF- inputs, persistent buffers, etc- can be accessed by either IMG_NORM_PIXEL() or IMG_PIXEL(), depending on whether you want to use normalized or non-normalized coordinates to access the colors of the image. If your shader isn't using these- if it's using texture2D() or texture2DRect()- it won't compile if the host application tries to send it a different type of texture.RENDERSIZE which is passed the dimensions of the image being rendered.TIME is declared, and passed the duration (in seconds) which the shader's been runing when the shader's rendered.gl_FragCoord.xy contains the coordinates of the fragment being evaluated. isf_FragNormCoord.xy contains the normalized coordinates of the fragment being evaluated. isf_vertShaderInit();.IMPORTED key. The imported images are accessed via the usual IMG_PIXEL() or IMG_NORM_PIXEL() methods. Details on how to do this are listed below, and examples are included.The first version of the ISF spec did some confusing and silly things that the second version improves on. If you want to write your own ISF host, and you want that host to support "old" ISF files, here's a link to the original ISF spec.
...and here's a list of the specific changes that were made from ISFVSN 1 to ISFVSN 2:
PERSISTENT_BUFFERS object in the top-level dict has been removed- it was redundant and confusing. Anything describing a property of the buffer a pass renders to is in the appropriate pass dictionary (PERSISTENT, FLOAT, WIDTH, HEIGHT, etc).vv_FragNormCoord has been renamed isf_FragNormCoord, and the function vv_vertShaderInit() has been renamed isf_vertShaderInit(). ISF is open-source, the use of "vv" for terms in the spec is inappropriate.INPUT types "audio" and "audioFFT" didn't exist in the first version of the ISF spec.IMG_SIZE function didn't exist in the first version of the ISF spec.TIMEDELTA, DATE, and FRAMEINDEX uniforms didn't exist in the first version of the ISF spec.VSN and ISFVSN didn't exist.