author: Alexander G. Balakhnin aka Fizick
version: 1.7
download: http://avisynth.org.ru
category: Misc Plugins
requirements: YV12 or YUY2 Colorspace
license: closed source
DePan plugin has tools (functions) for estimation of global motion (pan) in frames, and for full or partial global motion compensation.
The DePan tools can be used for:
The DePan plugin replaced my experimental GenMotion C-plugin (which uses motion data from VirtualDub Deshaker plugin log file).
The DePan works in one pass, but consist of server part (function) and one or more clients parts (functions or its instances). The server function estimates frames motion data and gives it to client functions on inquiry. Current version of DePan uses a special service clip as the motion data container.
DePanEstimate (server) - estimates frames global motion data and write it to special service clip
DePan (client) - make full or partial global motion compensation
DePanInterleave (client) - generates long motion compensated interleaved clip
DePanStabilize (client) - stabilizes motion
DePanScenes (client) - scene change indication
This function uses phase-shift method (by fast Fourier transform) for global motion estimation. It uses some central region of every frame (or field) as FFT window to find the inter-frames correlation and to calculate the most appropriate values of vertical and horizontal shifts, which fit current frame to previous one. The some relative correlation parameter is used as trust measure and for scene change detection. In zoom mode, plugin uses left and right sub-windows to estimate both displacements and zoom. Output is special service clip with coded motion data in frames, and optional log file.
DePanEstimate ( clip, int range, float trust, int winx, int winy, int dxmax, int dymax, float zoommax, bool improve, float stab, float pixaspect, bool info, string log, bool debug, bool show, bool fftw)
It generates the clip with motion compensated frames, using motion data previously calculated by DePanEstimate.
DePan (clip, clip data, float offset, int subpixel, float pixaspect, bool matchfields, int mirror, int blur, bool info, string inputlog)
Note: The offset parameter of DePan is extended version of delta parameter of GenMotion.
It generates long interleaved clipwith series of group of previous frames motion compensated (within some range), original frame, and motion compensated next frames (within range), and same groups for every following frames. In fact, it combines DePan function and Interleavefunction (AviSynth internal) for easy following temporal denoising, with following SelectEvery(prev+next+1, prev) function for selecting only cleaned source frames.
DePanInterleave (clip, clip data, int prev, int next, int subpixel, float pixaspect, bool matchfields, int mirror, int blur, bool info, string inputlog)
This function make some motion stabilization (deshake) by smoothing of global motion. Inertial filtering method is used (probably similar to Digistudio VirtualDub plugin).
DePanStabilize (clip, clip data, float cutoff, float damping, float initzoom, bool addzoom, int prev, int next, int mirror, int blur, int dxmax, int dymax, float zoommax, float rotmax, int subpixel, float pixaspect, int fitlast, float tzoom, bool info, string inputlog)
Generate clip with pixel values =255 for defined plane at scenechange and pixel values =0 at rest frames,
using motion data previously calculated by DePanEstimate.
May be used by AverageLuma function for conditional processing.
DePanScenes ( clip, string inputlog, int plane)
1. Works only in YV12 and YUY2 color formats.
2. Uses only pan and zoom motion (no rotation), but it gives advance in speed and stability. Estimation in zoom mode is not very precise.
3. The source clip must be same length as motion data clip.
4. Directly works only with progressive clips. For interlaced sources, you must use AviSynth following function SeparateFieldsand followed Weave
(after motion compensation and denoising), with AssumeTTFand AssumeBFF(both may be needed for odd fields offset). Plugin estimates and calculates motion from one field to neighbor (by time) field (from same or neighbor frame). For preserving fields order (dominance) and best denoising, set parameter MatchFields=true.
5. Mirror mode is unique but slightly strange :-). The blur is some workaround to hide sharp mirrored details.
6. Not very fast, not assembler optimized.
7. Tested with Avisynth 2.5.3 and 2.55.
8. The plugin function DePanEstimate uses free FFT2D code by Takuya Ooura (http://momonga.t.u-tokyo.ac.jp/~ooura/index.html)
Since version 1.0, DePan can also use more fast FFTW library version 3 (http://www.fftw.org)
as Windows binary DLL (compiled with gcc under MinGW by Alessio Massaro),
which support for threads and have AMD K7 (3dNow!) support in addition to SSE/SSE2.
It may be downloaded from ftp://ftp.fftw.org/pub/fftw/fftw3win32mingw.zip
For fftw using, you must put FFTW3.DLL file from this package to some directory in path (for example, C:\WINNT).
9. For best results, you may temporary add Info parameter, analyze info and tune some parameters (Trust, dxmax etc).
10. You may use not strictly same clips for motion estimation and compensation, for example try add some brightness-contrast adjusting, pre-filtering, masking, cropping to input clip used for motion estimation only (and use different processing for output compensated-stabilized results).
1. Load original (input) clip (I),
2. Make clip (F) with full forward motion compensation,
3. Make clip (B) with full backward motion compensation,
4. Make a interleave clip, with compensated frames before and after every original frame;
We will get a long clip (with triple length), with every 3 successive frames corresponded to same time.
5. Apply some temporal filter that uses pixel differences between previous, current and next frames, for example Fluxsmooth filter.
6. Select every third (original non-compensated but cleaned) frame to output.
The cleaned clip will not have a lot of artifacts, produced by global motion with denoising, and the denoising will be more strong in most areas (camera's pan will be compensated.)
Notes: with DePanInterleave, stages 2,3,4 combined to single. Moreover, the range may be large than 1.
AviSource("input.avi")
LoadPlugin("depan.dll")
LoadPlugin("fluxsmooth.dll")
i = ConvertToYV12()
mdata = DePanEstimate(i)
DePanInterleave(i, data=mdata)
FluxSmooth()
SelectEvery(3, 1)For best results, you may temporary add Info parameter, analyze info and tune some parameters (Trust, dxmax etc)
AviSource("input.avi")
LoadPlugin("depan.dll")
LoadPlugin("fluxsmooth.dll")
AssumeTFF()
SeparateFields()
i = ConvertToYV12()
mdata = DePanEstimate(i, range=1, trust=5.5, log="depan.log")
DePanInterleave(i,data=mdata, prev=1, next=1, matchfields=true)
FluxSmooth()
SelectEvery(3, 1)
Weave()Please, make a tests to add filters to the list!
For proposed denoising method with using of the DePan (previously with GenMotion), such temporal filter must compare pixel with previous and next frame, and make some smoothing if difference between previous and next frame is small. These filters also may make additional internal (small) local motion compensation (as Dust filter, which may get some speed increasing due to well global motion compensation).
DePan may be used as a tool for framerate converting and similar tasks.
For example, to change framerate with factor=1.5, from 16.6 fps progressive (old 8 mm film) to 25 fps, use script.
AviSource("kino.avi")
LoadPlugin("depan.dll")
i = ConvertToYV12()
data = DePanEstimate(i, range=1, trust=5)
f1_3 = DePan(i, data, offset=1./3)
b1_3 = DePan(i, data, offset=-1./3)
Interleave(f1_3, i, b1_3)
SelectEvery(6, 0, 1, 2)It may by written as a function:
function fps2to3(clip) {
# change FPS from 2 to 3 (or 16.66 to 25, or 20 to 30 and so on), i.e. with factor=3/2
# uses global motion compensation
# input must be YV12 or YUY2 progressive (or separated fields probably ?)
data = DePanEstimate(clip)
f1_3 = DePan(clip, data, offset=1./3)
b1_3 = DePan(clip, data, offset=-1./3)
Interleave(f1_3, clip, b1_3)
SelectEvery(6, 0, 1, 2)
}
AviSource("e:\video.avi")
LoadPlugin("depan.dll")
ConvertToYV12()
fps2to3() Here is a possible function for framerate converting (progressive) with factor=5/3, for example from 15 fps to 25 fps : function fps3to5(clip) {
# change FPS from 3 to 5 (or 15 to 25, or 18 to 30 and so on), i.e. with factor=5/3
# uses global motion compensation
# input must be YV12 or YUY2 progressive (or separated fields probably ?)
data = DePanEstimate(clip)
t3_5 = DePan(clip, data, offset=-2./5)
t6_5 = DePan(clip, data, offset=1./5).trim(2,0)
t9_5 = DePan(clip, data, offset=-1./5).trim(1,0)
t12_5 = DePan(clip, data, offset=2./5).trim(3,0)
Interleave(clip, t3_5, t6_5, t9_5, t12_5)
SelectEvery(15,0,1,2,3,4)
}
AviSource("e:\video.avi")
LoadPlugin("depan.dll")
ConvertToYV12()
fps3to5()Notes. There is more simple and general alternative method: try ChangeFPSwith following DePanStabilize.
DePan may be used as a tool for smoothing of global motion. Inertial filtering method is used in current version.
Simple sample script for progressive clip:
AviSource("input.avi")
LoadPlugin("depan.dll")
i = ConvertToYV12()
mdata = DePanEstimate(i)
DePanStabilize(i, data=mdata)We may add and tune parameters cutoff, dxmax, edges filling method etc, corresponded to your clip and you.
DepanEstimate function may write optional log file with motion data, in Deshaker - compatible format. Moreover, Depan function may read such log files (in this mode it works as GenMotion, without DepanEstimate, data clip is ignored, and source clip may be used as dummy data clip). Deshaker log may be loaded in Depan and vice versa. Depan can compensate zoom and rotation too. Therefore you may load similar AVS script files in VirtualDub, and run second pass of Deshaker for anvanced image stabilization (and coding) of filtered clip. Of course, before you must run first pass of DePanEstimate to make Depan.log file, which must be selected in Deshaker. Instead of that, you may add DePanStabilize(i,data) function to script and run all in one pass !
During Pass 1 the Deshaker plugin tries to find the panning, rotation and zoom that, when applied to the current image, makes it look like the previous image (almost). The values on each line in the file are (from left to right): frame number (or fields number), x- and y-panning (in pixels), rotation (in degrees) and zoom factor. You can edit the log file manually (but use fixed line format). You can delete lines that got completely wrong (and that you don't care to try to fix in a better way). Gaps in the frames numbers are treated as zero-panning, zero rotation and no scaling. If a frame exists more than once in the log file, the last line is used.
Note: For interlaced source, info is for every field (A - first, B - second by time)
Depan use framebuffer of special clip for storing of motion data. When client (Depan) requests the motion data for frame nfrom this clip, server (DepanEstimate) creates frame and writes such data (from start of framebuffer): one header record, and several frame motion data records, from n-rangeto n+range(nframes = 2*range+1 for non-edge frames). Definition of motion data parameters is same as in Deshaker log.
In all versions from 0.6, I use this structures:
#define DEPANSIGNATURE "depan06"
typedef struct depanheaderstruct { // structure of header depandata in framebuffer
char signature[8]; // signature for check
int reserved; // for future using
int nframes; // number of records with frames motion data in current framebuffer
} depanheader;
typedef struct depandatastruct { // structure of every frame motion data record in framebuffer
int frame; // frame number
float dx; // x shift (in pixels) for this frame
float dy; // y shift (in pixels, corresponded to pixel aspect = 1)
float zoom; // zoom
float rot; // rotation (in degrees), (now =0 - no rotation estimated data in current version)
} depandata;Note 1. Depan uses dx=0.0 as mark of scenechange.
Note 2. DepanEstimate output is cropped if not in show or info mode.
Recently I added to local motion estimation MVToolsplugin by Manao (to version 0.9.8.2) a new function MVDepanfor global motion estimation. It is based on local block motion vectors analysis, similar to first pass of DeShaker plugin. MVDepanfunction can be used instead of DepanEstimatefunction. It can estimate pan, zoom and rotation, but at present it is still experimental beta version. Search for newest MVToolsand its documentation at Manao's site . Discussion is at AviSynth doom9 forum.
Some discussion about GenMotion and DePan plugins may be found in AviSynth forum at
http://forum.doom9.org/forumdisplay.php?s=&forumid=33
in particular in thread http://forum.doom9.org/showthread.php?s=&threadid=66686
Thanks to Gunnar Thalin for detailed info about Deshaker log file format and very useful discussions.
Thanks to Takuya Ooura for free and fast FFT2D code.
Thanks to scharfis_brain and many others for useful discussions and bug reports.
This program is freeware, but WITHOUT ANY WARRANTY.
$Date: 2006/09/17 17:41:38 $