By Rainer Wittmann gorw@gmx.de
The most recent version of this document is always available at www.RemoveGrain.de.tf
The binary and the source code are subject to the GNU General Public License. In addition we deny anyone the right to derive a Photoshop plugin from the source code of RemoveGrain. This restriction was made in view of the way Adobe treats third party filter developers, which is absolutely unacceptable. Last change Juli 7, 2005.
The binary package contains four versions of RemoveGrain, two small dynamically linked versions RemoveGrain.dll, RemoveGrainSSE2.dll, RemoveGrainSSE3.dll and the big staticly linked RemoveGrainS.dll. The first one only requires integer SSE (Athlon and Pentium 3 design), the second requires a SSE2 capable cpu (Pentium 4 or Athlon 64) and the third is for Prescott P4s only. If none of these dlls work because of a missing dll, one has either to copy the missing dll to the window's system directory or one has to use RemoveGrainS.dll, which only requires SSE. Please put only one of the three plugins into the Avisynth plugin directory. RemoveGrain supports all color spaces. YV12 is directly supported and YUY2, RGB24, RGB32 are supported in planar form (see the section Color Spaces). Beginning with version 0.6 the auxiliary filter Clense has been added to RemoveGrain.dll and the plugin Repair.dll has been added to the binary package. There are again four versions of this plugin. Though Repair.dll is derived from the same source code, it serves for quite different purposes. This plugin contains the filters Repair and TemporalRepair. All three new filters are described below. Please post questions, problems, comments etc. to the support forum rather than sending me email. In particular, I would appreciate reports about artifacts and compression gain of the various modes. These may also help others to select the right mode.
RemoveGrain has started as a purely spatial denoiser (modes 1-9, 17, 18, 21, 22) and over time it has evolved into a general plugin which manipulates a pixel in terms of its eight direct neighbours (and of course the value of the pixel itself). It now includes modes for blurring (11, 12, 19, 20), modes for nondestructive sharpening (10, 25), smart bob modes (13-16) and last but not least the modes 23, 24 for removing small edge halos, which may be generated by sharpening filters or by lossy DCT compression. Then from the same source code - with some modifications, of course - the Repair plugin for correcting other filters was derived. It is primarily used for removing artifacts of the very aggressive and simple temporal cleaner Clense (also included in RemoveGrain), but it can also be used for deringing (very different from RemoveGrain mode 23,24) and taming sharpeners. Together with Clense, the filter RestoreMotionBlocks from the RemoveDirt plugin it is used to built the RemoveDirt script function - my most powerful cleaner/denoiser. Finally, in the same way as Repair was derived from RemoveGrain, TemporalRepair(mode=0) was derived from Clense and is included in the Repair plugin. TemporalRepair(mode=1,2) are much more sophisticated and slower variants of this filters, which can also be used to converting simple spatial deinterlacers like RemoveGrain(mode=12) into motion adaptive deinterlacers, which fully preserve static detail. Thus deinterlacing is another application area of the RemoveGrain/Repair plugin.
RemoveGrain modes 1-10,17-25 are for truely progressive input only, the bob modes 13-16 are for interlaced input only and modes 11,12 can be used both for progressive and interlaced input. Of course, all modes can be used for interlaced input, if the two fields are separated before applying RemoveGrain with the builtin filter SeparateFields and then merged together again, but the quality is clearly inferior, because the adjacent pixels have twice the vertical distance.
The frame width of the input clip must be > 36 (SSE version) or > 68 (SSE2/SSE3 version). The pixels at the margin are never changed, because it would require different algorithms. The SSE3 version can only be compiled with the Intel Compiler and there are compatibility problems, which I haven't tracked down yet. As far as I know this causes only problems if you use Avisource (mpeg2source works well). If you combine Avisource with the SSE3 version of RemoveGrain, VirtualDubMod crashes, when you exit it. For encoders the problem may be more severe. Thus if you experience crashes with the SSE3 version of RemoveGrain, try the SSE2 version. If the problem disappears, then it is again an Intel compiler compatibility problem and should be posted to the support forum. On the other hand, if you have an SSE3 capable cpu you should use the SSE3 version, because it is significantly faster than the SSE2 which is only slightly faster than the standard SSE version.
In modes 1-10,17,18 the value of the center pixel is replaced by the value of one of its eight neighbours. In order to discuss the various modes in full detail the reader must always keep the following picture in mind:
n(1) | n(2) | n(3) |
n(4) | c | n(5) |
n(6) | n(7) | n(8) |
The pixel "c" in the middle is called the center pixel and n(1),n(2),n(3),n(4),n(5),n(6),n(7),n(8) are the eight neighbour pixels. All the modes of RemoveGrain modify the center pixel depending only on the eight neighbours and the center pixel itself. Mode 1 replaces the c by max(min(c, MAX), MIN), where MAX=max(n(1),n(2),n(3),n(4),n(5),n(6),n(7),n(8)) and MIN=min(n(1),n(2),n(3),n(4),n(5),n(6),n(7),n(8)). Thus RemoveGrain(mode=1) is identical with trbarry's Undot (even the speed of the SSE version is nearly identical to that of Undot, but the SSE2 and especially the SSE3 versions are significantly faster). Mode 2-4 are more aggressive refinements of mode 1. To this end, let i(1),i(2),i(3),i(4),i(5),i(6),i(7),i(8) be a reordering of 1,2,3,4,5,6,7,8 such that n(i(1)) <= n(i(2)) <= n(i(3)) <= n(i(4)) <= n(i(5)) <= n(i(6)) <= n(i(7)) <= n(i(8)). In particular, MIN=n(i(1)) and MAX=n(i(8)). Then mode=k (with k ranging from 1 to 4) replaces the center pixel c by max(min(c,n(9-k)), n(k)). Mode 4 can also be interpreted in terms of the median, i.e. mode 4 replaces c by the median of the nine values c,n(1),n(2),n(3),n(4),n(5),n(6),n(7),n(8). As the mode increases (between 1 and 4 only) RemoveGrain becomes more and more aggressive. Mode 1-4 are so called neighbour clipping modes, i.e. a pair of neighbours n1,n2 is selected and c is replaced by max(min(c,max(n1,n2)), min(n1,n2)). In the case of mod k (with k between 1 and 4) n1=n(i(9-k)) and n2=n(i(k)) are selected. In the subsequent discussion, the selected pair n1,n2 is called the clipping pair. Mode 1-4 are not line sensitve, i.e. the clipping pair n1,n2 doesn't need to be a line pair. There are four line pairs: the horizontal pair n(4),n(5), the vertical pair n(2),n(7) and the two diagonal pairs n(1),n(8) and n(3),n(6). The modes 5-9,18 are line sensitive cliiping modes, i..e. the clipping pair is always a line pair. Mode 9 is the simplest mode of this kind. Here the line pair n1,n2 is chosen such that |n1 - n2| is minimal. In mode 5 the line pair n1,n2 is chosen such that |c - max(min(c,max(n1,n2)), min(n1,n2))| is minimal, i.e. such that the change is minimal. The modes 6,7,8 are intermediate modes between mode 5 and mode 9. Mode 6 is closer to mode 5 while mode 8 is closer to mode 9 and mode 7 is right in the middle between mode 5 and mode 9. Fizick has extended mode 5-9 to the temporal domain in his plugin DeGrainMedian. Because there are now 13 instead of 4 line pairs, DeGrainMedian is necessarily slower. In mode 18 the line pair n1,n2 is chosen as the clipping pair if max(|c - n1|, |c - n2|) is minimal. Because of its nice properties discussed further below mode 17 is the most important mode. In mode 17 the clipping pair m1,m2 is chosen such that m1 = max min(n1,n2) and m2=min max(n1,n2), where the maximum and the minimum are taken over all line pairs n1,n2. Note that in mode 17 the clipping pair m1,m2 is not necessarily a line pair. Mode 21,22 are the last denoising modes. These are not neighbour clipping modes. Instead clipping is done with respect to averages of neighbours. In mode 22 clipping is done with respect to MinA = min (n1 + n2 + 1)/2, MaxA = max (n1 + n2 + 1)/2, where the minimum and the maximum is taken over all line pairs (n1,n2). This mode is the fasted of all modes, even faster than mode 1. Mode 21 is very similar to mode 22, but with MinA = min (n1 + n2 )/2. Mode 21 is significantly slower than mode 22 but unlike mode 22 it has no upside bias.
Modes 11, 12, 19 and 20 are blur modes. Mode 11 replaces the center pixel c by (4*c + 2*n(2) + 2*n(4) + 2*n(5) + 2*n(7) + n(1) + n(3) + n(6) + n(8) + 8) / 16. It is similar to blur(1), which implements bluring with a (1/4, 1/2, 1/4) kernel both in the horizontal and the vertical direction. However, it is significantly more precise. Mode 12 differs from mode 11 by at most 1 but is significantly faster than mode 11. Mode 12 is still more precise than blur(1). Mode 19 replaces c by the average of its 8 neighbours. Finally mode 20 replaces c by the uniform average (c + n(1) + n(2) + n(3) + n(4) + n(5) + n(6) + n(7) + n(8) + 4) / 9.
< P> In version 0.7 the two bob modes 13 and 14 have been added. Mode 13 leaves the bottom field unchanged and interpolates the top field. Mode 14 leaves the top field unchanged and interpolates the bottom field. The interpolation in mode 13, 14 is similar to Trbarry's weird bob, which is part of Trbarry's Tomsmocomp. However, instead of optimising over 5 line pairs we optimise only over 3 line pairs. More precisely, c is replaced either by (n(2) + n(7) + 1)/2 or (n(1) + n(8) + 1)/2 or (n(3) + n(6) + 1)/2, depending on which of the three distances |n(2) - n(7)|, |n(1) - n(8)|, |n(3) - n(6)| is minimal. In Trbarry's weird bob there are two more possible interpolation values, which makes weird bob more edge sensitve, but also the likelihood of smart edge artifacts is higher and of course it is slower. Taking only 3 line pairs seems to be the best compromise to us. Together with the filter TemporalRepair, described below, it may be used for builting motion adaptive deinterlacers. Beginning with version 0.8 we have added modes 15, 16. They have the same edge sensitivity as modes 13,14 but the quality is somewhat higher and the speed somewhat slower.In version 1.0 the modes 23, 24 for removing small edge were added. They are primarily useful for digicam videos, which are loaded with these compression unfriendly artifacts caused by sharpening and DCT. They would also be quite useful for digicam photos. Let us look at a typical example of 6 consectuve pixels on a line with the following values: 102, 100, 120, 30, 50, 51. Thus there is an edge between the third and the fourth pixel. It is very typical for edge halos, that the pixels at the edge are more extreme, i.e. instead of 100, the third pixel has the value 120 and instead of 50 the fourth pixel has the value 30. Thus on one side we have an overshot on the upside, while on the other side we have an overshot on the downside and both overshots have usually similar size. To describe modes 23 and 24 we fix a line pair n1,n2 for a while. We may assume that n2 <= n1. Then in mode 23 the upside overshot of the line pair (n1,n2) is defined by u(n1,n2) = max(min(c-n1, n1 - n2), 0) and the downside overshot is defined by d(n1,n2) = max(min(n2 - c, n1 - n2), 0) . Note that we never can have u(n1,n2) > 0 and d(n1,n2) > 0 simultaneously for a line pair (n1,n2). Normally - the artifact free case - we even have u(n1,n2) = d(n1,n2) = 0. To get acquainted with these definitions, let us look at two examples. if n1=100, c=120, n2=30, then u(n1,n2)=max(min(120-100,100-30),0) = 20, d(n1,n2)=max(min(30-120,100-30),0)=0. If n1=120,c=30,n2=50, then u(n1,n2)= Finally the total upside overshot of the center pixel c is defined by U(c) = max u(n1,n2) and the total downside overshot is defined by D(c) = max d(n1,n2). In both case the maximum is taken over all the four line pairs through the pixel c. In bizarre cases, we may very have U(c) > 0 and D(c) > 0 simultaneously, but usually only of the two total overshot numbers is significantly above 0. Finally in mode 23, the center pixel c is replaced by c - U(c) + D(c). Mode 24 only differs from mode 23 by a more sophisticated definition of u(n1,n2) and d(n1,n2). More precisely, the follwing definitions are used in mode 24: u(n1,n2) = max(min(c-n1, 2*n1 - n2 -c), 0), d(n1,n2) = max(min(n2 - c, n1 + c - 2*n2), 0). These modified definitions make mode 24 considerably more conservative and slightly slower than mode 23. Mode 24 should be preferred.
Finally mode 25 is the minimal sharpening mode. In this mode the neighbours n1,n2 are selected such that n1 <= c <= n2 and such that there exists no further neighbour n with n1 < n < n2. If c is larger than all neighbours, then n2 doesn't exist. In this case, we simply replace n2 by 255. Similarily, if c is smaller than all neighbours, then n1 doesn't exist and n1 is replaced by the value 0. Finally c is sharpened as described in the section Sharpening between two points.
All modes, which replace c by a neighbour, preserve and even slightly improve sharp edges, but in many modes some pixels may be shaved off the edge. In particular, it is important property whether a RemoveGrain mode preserves the corners of edges. If the center c is close to both points of a line pair (n1,n2), in particular, n1 and n2 are both close to each other as well, then we say that a mode preserves thin lines if the modified pixel is again close to n1,n2. This is the most important property, when we measure how well a mode preserves details. Note that preserving thin lines, means only that the interior points of a thin line are preserved, line ends my be shaved off. We say that a mode preserves line ends, if the line ends are not shaved off. Preserving thin lines usually implies that also most points of a thin curve are preserved as well, at least if the curve is sufficiently smooth. We say that a mode preserves thin curves if it preserves all points of the curve except possibly the two ends of the curve. Finally it is important, how well a mode improves compression. We have now named all the properties, which are quantified in the following table:
mode | sharp edges | corners | thin lines | line ends | thin curves | compression |
1 | 10 | 10 | 10 | 10 | 10 | 1 |
2 | 10 | 10 | 10 | 4 | 10 | 2 |
3 | 10 | 9 | 3 | 1 | 3 | 5 |
4 | 10 | 3 | 1 | 0 | 1 | 8 |
5 | 10 | 10 | 10 | 9 | 9 | 2 |
6 | 10 | 6 | 9 | 4 | 3 | 3 |
7 | 10 | 6 | 9 | 4 | 3 | 3 |
8 | 10 | 5 | 8 | 3 | 3 | 4 |
9 | 10 | 3 | 4 | 1 | 2 | 5 |
10 | 10 | 10 | 10 | 10 | 10 | 1 |
11 | 1 | 1 | 2 | 1 | 2 | 9 |
12 | 1 | 1 | 2 | 1 | 2 | 9 |
17 | 10 | 2 | 8 | 2 | 4 | 7 |
18 | 10 | 6 | 9 | 6 | 5 | 2 |
19 | 1 | 1 | 1 | 0 | 1 | 10 |
20 | 1 | 1 | 1 | 0 | 1 | 8 |
21 | 6 | 2 | 8 | 4 | 4 | 5 |
22 | 6 | 2 | 8 | 4 | 4 | 5 |
23 | 6 | 5 | 6 | 4 | 6 | 4 |
24 | 7 | 6 | 7 | 5 | 7 | 3 |
25 | 10 | 10 | 10 | 10 | 10 | -1 |
The entries range from 0 to 10. The higher the number the higher the quality of this property. Since mode 25 even worsens compression slightly the compression entry of mode 25 has the value -1. The numbers itself are primarily based theoretical and sometimes also on discussions in various forums. The compression numbers are underpinned by extensive one pass XviD compression tests with the fixed quantiser 5 with sources either from digital TV or from a digicam.
1. As already mentioned, RemoveGrain cannot change border pixels. To get rid of these, cropping should be done after RemoveGrain. On the other hand, for RemoveDirt cropping should be done before such that DCT blocks and RemoveDirt blocks are the same. Thus, if RemoveGrain is combined with RemoveDirt cropping should be done at best between RemoveGrain and Removedirt.
2. Although RemoveGrain always makes a lot of unaligned memory access aligning frames nevertheless has a positive performance impact. Thus crop with align=true, if you need to crop before RemoveGrain.
3. If mode=0 (resp. modeU=0, resp. modeV=0), then the Y plane (resp. the U plane, resp. the V plane) is simply copied from input to output. If even mode < 0 (resp. modeU < 0, resp. modeV < 0) then for the respective planes nothing is done at all (not even a copy operation). This may be used for faster processing of black&white clips. For instance RemoveGrain(mode=8, modeU= -1).RemoveDirt(grey= true) is the fasted way to process a black&white clip by the RemoveGrain/RemoveDirt combo. Because modeV inherits the value -1 from modeU, RemoveGrain doesn't process the chroma planes and RemoveDirt subsequently sets the chroma planes uniformly to 128. If you use modeU=-1, you must add a chroma later through the builtin filters MergeChroma or GreyScale.
4. RemoveGrain is fairly fast. It makes perfect sense to iterate it for better denoising. For instance, RemoveGrain(mode=2). RemoveGrain(mode=2) is still a fairly defensive denoiser, which doesn't destroy thin lines.
5. If grain is too crude, RemoveGrain can only partially remove it or cannot remove it at all. Because in such a case the source is quite poor anyway, it is a good idea to shrink. Then also the grain gets smaller and RemoveGrain may be able to erase it. In general, downsizing should be done before RemoveGrain while upsizing should be done after RemoveGrain.
6. As described above, some modes, especially mode 3,4,9, have a thin line problem. Now YUY2 and especially YV12 have a buitlin thin line problem for the chroma, because several pixels share one chroma value. This problem is amplified if RemoveGrain is applied to the chroma. It should be kept in mind, that RemoveGrain has roughly four times the strength, if it is applied to the YV12 chroma rather than the luma. This is the main reason for edge softness caused by the more conservative modes 18,6,7,8,17. Thus to avoid compounding the thin line problem, it is a good idea to use modes without a thin line problem for the chroma even if mode 3,4,9 is used for the luma. For instance RemoveGrain(mode=4, modeU=2) uses the high compression mode 4 for the luma and the low artifact mode 2 for the chroma. On the other hand the chroma is often particularily noisy (especially if the clip originates from a video tape), which speaks against applying a more conservative mode for the chroma.
Clense uses the same simple cleaning technique as RemoveDirt, but without any protection against artifacts. Thus, if p[i,j],c[i,j],s[i,j] denotes the luma of the pixel with coordinates i,j on the previous, the current and the subsequent frame, then c[i,j] is replaced by min(max(c[i,j], min(p[i,j],s[i,j])), max(p[i,j],s[i,j]))), i.e. temporal minmax clipping. Clense is a very brutal cleaner and cannot be used without Repair or some other artifact remover. Nevertheless if dirt or grain overlaps with similar dirt or grain on only one adjacent frame, this overlap cannot be cleaned. This remark is particularily important for grain (even invisable grain may have a very significant negative impact on compression). Thus it enhances compression quite a bit to remove this left over. The left over of Clense is usually significantly smaller than the original grain, whence Clense + RemoveGrain is considerably more effective and is able to remove much cruder grain than RemoveGrain alone, but of course there is a severe artifact problem.
Clense accepts two arguments. The first is the input clip and the second is the boolean variable grey (grey=false is the default). If grey=true, then there is no chroma processing at all, i.e. the chroma planes contain random values and the greyscale filter has to be applied at some later stage. The grey variable is ignored for clips with color spaces other than YV12.
Beginning with version 0.9 there are also the filters BackwardClense and ForwardClense. They are primarily for clensing at sharp scene changes in the RemoveDirt script function. ForwardClense is for the first frame of a new scene and BackwardClense is for the last frame of a scene. The usage is the same as Clense.
Beginning with version 0.6 we have added the Repair plugin. It is generated from the same source code as RemoveGrain, but serves a very different purpose. Instead of removing grain, Repair should remove artifacts introduced by previous filters. It does so by comparing the video before these filters were applied with the video after application of these filters. Thus Repair requires two clips as input. The first clip is the filtered and the second the unfiltered clip. Otherwise Repair uses the same variables as RemoveGrain, but modes 11-17, which do not make sense for Repair are disabled. Thus there are currently 10 Repair modes.
Repair works very similar as RemoveGrain, but unlike RemoveGrain, where the center pixel and the neighbour pixels are both from the same frame of the clip, in Repair the center pixel (i.e. the pixel to be changed) is taken from the first clip (i.e. the filtered clip) and the neighbour pixels are taken from the second clip (i.e. the unfiltered original clip). For instance Repair(filtered, original, mode= 1) clips the luma of a pixel of the clip "filtered", by the luma minimum and the luma maximum of the neighbour pixels of the pixel in the clip "original". Thus Repair limits the amount of change from clip "original" to the clip "filtered". This is clearly some kind of artifact removal. The higher the mode (from 1 to 4 only) the stronger the artifact removal. With mode=4, Repair(filtered, original, mode=4) is fairly close to RemoveGrain(original, mode= 4, limit= 0). As described above the center pixel of the original clip would be ignored. This doesn't make sense. We have therefore made changes to include also the center pixel of the original clip for all Repair modes. For instance, Repair(filtered, original, mode=1) clips the center pixel of the filtered clip by the minimum of all the 8 neighbours and the center pixel of the original clip on one side and by the maximum of all the 8 neighbours and the center pixel of the original clip on the other side. In particular, Repair(input, input, mode=1) doesn't change at all the clip input. Without inlcluding the center pixel of the original clip, Repair(input, input, mode= 1) would have been identical to RemoveGrain(input, mode= 1) or Undot(input). In general, we have RemoveGrain(input, mode=n) = Repair(input, input, n+1) for n=0,1,2,3.
In version 0.9 we have added new Repair modes, which are derived from RemoveGrain modes in a different way. To this end, let "f" be the luma value of the center pixel of the filtered clip (which is to be repaired) and "o" be the luma value of the same pixel in the original. Then as in the corresponding RemoveGrain mode, two neighbour pixels with luma values, say, "n1", "n2" are selected and "f" is clipped at min(n1,n2,o) and max(n1,n2,o). Of course, the chroma is treated the same way. Such a Repair mode can be derived from any RemoveGrain mode except modes 11-16. We have done this for mode 1-6, 17, 18 only. 11-18 are the corresponding repair modes. Thus Repair modes 11-16 correspond to the RemoveGrain modes 1-6, while Repair modes 17,18 correspond to the RemoveGrain Modes 17,18. For mode 1 both methods for deriving Repair modes coincide, whence Repair(mode=1) and Repair(mode=11) are identical. (Repair mode 16 is of particular importance. It restores thin lines and there is good chance (but no guarantee) that even the end points are restored. It is currently the primary Repair mode for the RemoveDirt script.
Let us now discuss some examples for Repair. The first is a preliminary version of RemoveDust:
function RemoveDust(clip input, int _mode) { repmode = 16 clensed = Clense(input) rep=Repair(clensed, input, mode=repmode) return RemoveGrain(rep, mode=_mode) } |
Firstly Clense brutally removes all temporal dirt but also introduces many artifacts. Repair then removes these artifacts but also restores all bigger pieces of dirt. While most of the motion artifacts from Clense are removed by Repair, motion gets blurred quite a bit, but static areas are not touched. Finally RemoveGrain (I recommend mode=17) removes the left over from Clense. Blurring motion has a much more significant impact on compression than blurring static detail. On the other hand, the human eye can not really concentrate on motion detail and may even get "nervous" from motion details. Thus in my view RemoveDust is much better than the usual "blur all a little" approach of most denoisers. If you want a more a agressive Removedust, you may choose repmode= 2,5,1 and/or apply RemoveGrain before Repair. Repair(mode= 9) is very interesting. It restores all thin lines destroyed by clense. Nevertheless its compression is remarkably good (the output size is usailly less than 5%-20% above that with Repair(mode= 2) or Repair(mode= 5) . I recommend to use Repair only with modes 2, 5-9,16,18.< /P> The second example is a high quality deringing filter:
function RGDering(clip input, int _mode) { repmode = 1 average = RemoveGrain(input, mode=_mode) return Repair(input, average, mode=repmode) } |
function ModerateSharpen(clip input, float amount) { repmode = 16 sharpened = Sharpen(input, amount) return Repair(sharpened, input, mode=repmode) } |
Unfortunately the name LimitedSharpen is already used. Thus we had to call the above script function ModerateSharpen. Sharpen(1.0) looks truely ugly and is a nightmare for any codec. We obtained the following compression results (5000 frames, high quality source, 696x448, xvid 1.0.1 with quantiser 5):
19.260.837 Bytes without any filtering 85.830.020 Bytes Sharpen(1.0) 41.743.528 Bytes ModerateSharpen(1.0), repmode=1 34.255.894 Bytes ModerateSharpen(1.0), repmode=2 |
Also ModerateSharpen(1.0) especially with repmode=9 looks a lot better than Sharpen(1.0). Mode 10 is not suitable for taming sharpeners.
In the same way as Repair is derived from RemoveGrain, TemporalRepair is derived from Clense. While Repair is a spatial filter most suitable for removing artifacts of temporal filters like Clense, TemporalRepair is a temporal filter, primarily useful for restoring static (non moving) details of spatial filters like RemoveGrain. Especially RemoveGrain(mode= 4) looses its "washed out" look, if combined with TemporalRepair:
function RemoveTemporalGrain(clip input, int _mode) { rg = RemoveGrain(input, mode=_mode) return TemporalRepair(rg, input) } |
mode=4 is definitely recommended for RemoveTemporalGrain. Of course compression of RemoveTemporalGrain(4) output is worse than that of RemoveGrain(mode= 4) but it is well above the more defensive RemoveGrain modes and fully preserves static detail. For RemoveDust the situation is even better:
function RemoveDust(clip input, int _mode) { repmode = 2 clensed = Clense(input) rep=Repair(clensed, input, mode=repmode) rg = RemoveGrain(rep, mode=_mode) return TemporalRepair(rg, rep) } |
Note that instead of the original clip "input", now the intermediate clip "rep" is taken as the second clip. Compared with the previous version of RemoveDust in the current (not yet the final) version RemoveGrain has just been replaced by RemoveTemporalGrain and softness is reduced substantially if mode=4. Static detail even remains untouched and compression doesn't suffer too much. Though the edges of objects are even sharpened, if these objects are not to tiny, RemoveDust looks somewhat soft and tiny moving objects may be whiped out, a good part of the softness is only psychological, because virtually all the natural flicker in a film, which doesn't contain any information, but to which the human eye is quite used to, is removed. If such flicker is reintroduced by a sharpener after decoding by a player (with ffdshow for example), then RemoveDust clips should look quite crisp again.
TemporalRepair can also be used to build a reasonably fast, high compression, threshold free, no motion no change deinterlacer:
function RGDeinterlace(clip input) { rg = RemoveGrain(input, mode=12) return TemporalRepair(rg, input) } |
While RemoveGrain(input, mode=12) deinterlaces every pixel, TemporalRepair restores the static areas. This is rather typical for TemporalRepair, if a filter messes up the static parts of a video these can be restored nicely - without any threshold artifacts - by TemporalRepair. RGDeinterlace is a blur deinterlacer, which behaves similar to AlignFields(mode=2) but without any flicker, because there are no thresholds. If on one frame the "motion value" of an area is slightly above the threshold and on the next it is slightly below, then this area is deinterlaced on one frame and unchanged on the other. This usually shows up as flicker, which is typical for threshold based motion adaptive deinterlacers. It has quite negative effects on compression. Unfortunately, in this form RGDeinterlace performs quite badly in my ticker tape test, because TemporalRepair restores to much. Though the artifacts in the ticker tape test are only visable with 400% magnification in VirtualDub, they indicate that the deinterlacer is not compression optimal, i.e. compression is wasted without gaining image quality. To improve the performance in this test I have added the integer variable smooth to TemporalRepair in version 0.7. If smooth=0, the default value, then TemporalRepair performs as before. If smooth= 1 (currently only the values 0 and 1 are allowed), then TemporalRepair is no more purely temporal, because it takes into account also the temporal fluctuation of the 8 neighboring pixels. Consequently, TemporalRepair "repairs" less with smooth=1 than with smooth=0, but still the resulting deinterlacer cannot pass the ticker tape test satisfactorily (though it is better). Thus we have introduced smooth=2,3 in version 0.9 for this purpose. Mode 2 restores less than mode 0,1,3 and is therefore the safest mode for deinterlacing and gives fairly good compression results. Mode 3 is also suitable for deinterlacing and should be chosen, if one aims to preserve a maximum amount of detail. In the ticker tape test mode 2 performs better than mode 3 and a lot better than mode 1, but it is still not artifact free. However, it competes well with most other motion adaptive deinterlacers (usually threshold based) in this test. One has to remove the remaining artifacts by another application of RemoveGrain. In this way one gets the following deinterlacer:
function RGDeinterlace(clip input) { rg = RemoveGrain(input, mode=12) rg2 = TemporalRepair(rg, input, smooth=2) return RemoveGrain(rg2, mode=2) } |
However, this deinterlacer obviously no more adheres to the no motion no change principle. Another idea to improve the deinterlacer is to iterate TemporalRepair. In this way we get the following deinterlacer:
function RGDeinterlace(clip input) { rg = RemoveGrain(input, mode=12) rg2 = TemporalRepair(rg, input, smooth=2) return TemporalRepair(rg, rg2, smooth=2) } |
This version of RGDeinterlace is again no motion no change and performs better in the ticker tape test. However, with smooth > 0 TemporalRepair is substantially slower than with smooth=0 and iteration doesn't make sense with smooth=0. It also makes sense to use TemporalRepair(smooth=1) within the RemoveDust script.
The above deinterlacers are all blur deinterlacers. To obtain sharp deinterlacers one has to apply RemoveGrain with the new mode 14 first. For example
function RGSDeinterlace(clip input) { rg = RemoveGrain(input, mode=16).RemoveGrain(mode=12) return TemporalRepair(rg, input, smooth=2) } |
Clense and TemporalRepair both have the optional grey variable. If grey= true, then Clense and TemporalRepair leave the chroma completely unprocessed. It is not even copied. Thus one has to use the builtin filter Greyscale at some later stage to clean out the random values from the chroma. This option is only valid for YV12 clips and is ignored for other color spaces. The same effect can be achieved for the filters RemoveGrain and Repair if one sets modeU= -1. Here is the RemoveDust filter as I use it now:
function RemoveDust(clip input, int "repmode", int "_smooth", bool "_grey") { default(repmode, 16) default(_smooth, true) default(_grey, false) clmode = 4 clensed = Clense(input, grey=_grey) rep=Repair(clensed, input, mode=repmode, modeU=_grey ? -1 : repmode ) rg = RemoveGrain(rep, mode=clmode, modeU=_grey ? -1 : clmode) return TemporalRepair(rg, rep, grey=_grey, smooth=_smooth) } |
function RGDeinterlace(clip input, bool _grey) { rg = RemoveGrain(input, mode=12, modeU=_grey ? -1 : 12) rg2 = TemporalRepair(rg, input, smooth=2, grey=_grey) return RemoveGrain(rg2, mode=2, modeU=_grey ? -1 : 2) } |
function RGBob(clip input, bool _grey) { top = RemoveGrain(input, mode=16, modeU=_grey ? -1 : 16) bottom = RemoveGrain(input, mode=15, modeU=_grey ? -1 : 15) return Interleave(top, bottom) } |
Until version 0.7 the above filters did also support the color spaces YUY2, RGB24 and RGB32. It did so by converting these interleaved formats internally into a planar ones, then the planar intemediate formats were processed as YV12 and finally the intermediate planar formats were converted back into the original interleaved formats. If ,as in some of the above scripts, many instances of the above filters are used, these conversion routines resulted in enormous overhead. Also the conversion routines were not optimised. Beginning with verison 0.8 we have dropped support for the interleaved color spaces in favor of their planar analogs. For Avisynth planar YUY2, RGB24 or RGB32 frames look just like the ordinary interleaved frames of these color spaces. However, the organization of the data on these frames is very different. The not yet released plugin SSETools (preliminary versions are included in the RemoveGrain package) contains the basic filters Interleaved2Planar and Planar2Interleaved. For YV12 these filters do absolutely nothing. For the other color spaces, Interleaved2Planar converts the interleaved frames into frames with planar data organisation and Planar2Interleaved reverses this conversion. Thus Interleaved2Planar().Planar2Interleaved and Planar2Interleaved.Interleaved2Planar(). leaves any frame unchanged. Thus Planar2Interleaved and Interleaved2Planar are lossless filters. For YUY2 the filters Interleaved2Planar() and Planar2Interleaved are highly optimised and shouldn't cost much more than a simple bitblt. Now, in order to prohibit unexperienced users from applying the above filters to interleaved color spaces, we have added the boolean variable planar to RemoveGrain, Repair and TemporalRepair. If planar=false (the default value), then these filters reject YUY2, RGB24 and RGB32 frames. Only if planar=true, then these filters accept YUY2, RGB24 and RGB32 input, but always assume that the data are organised in a planar way. If you use planar=true with ordinary interleaved input, you get garbage. Clense and MCClense like most other purely temporal filters work the same way with planar and interleaved input. Thus these filters have no planar variable and work both with planar and interleaved input. TemporalRepair(smooth=0) is also purely temporal and thus accepts YUY2, RGB24 and RGB32 input even with planar=false. On the other hand,TemporalRepair(smooth=1) has also spatial aspects and therefore rejects any YUY2, RGB24 and RGB32 input with planar=false. In the not too distant future there should be a web site www.PlanarYUY2.de.tf , which will list filters and there compatibility with planar YUY2, RGB24, RGB32.
As an example for the application of Planar2Interleaved and Interleaved2Planar we give a version of RemoveDust, which works for YV12, and all interleaved color spaces (but not planar YUY2, RGB24, RGB32):
function RemoveDust(clip input, bool grey, int "repmode", int "_smooth") { default(repmode, 16) default(_smooth, true) clmode = 4 input = Interleaved2Planar(input) clensed = Clense(input, grey=_grey) rep=Repair(clensed, input, mode=repmode, modeU=_grey ? -1 : repmode ) rg = RemoveGrain(rep, mode=clmode, modeU=_grey ? -1 : clmode) return TemporalRepair(rg, rep, grey=_grey, smooth=_smooth).Planar2Interleaved() } |