Average Regression Surface for Dependent Data

Transkript

1 Joural of Multivariate Aalysis 75, 242 (2000) doi:0.006jmva , available olie at o Average Regressio Surface for Depedet Data Zogwu Cai Uiversity of North Carolia zcaiucc.edu ad Jiaqig Fa Uiversity of Califoria jfakedall.stat.ucla.edu Received February 9, 999; published olie August 3, 2000 We study the estimatio of the additive compoets i additive regressio models, based o the weighted sample average of regressio surface, for statioary :-mixig processes. Explicit expressio of this method makes possible a fast computatio ad allows a asymptotic aalysis. The estimatio procedure is especially useful for additive modelig. I this paper, it is show that the average surface estimator shares the same optimality as the ideal estimator ad has the same ability to estimate the additive compoet as the ideal case where other compoets are kow. Formulas for the asymptotic bias ad ormality of the estimator are established. A small simulatio study is carried out to illustrate the performace of the estimatio ad a real example is also used to demostrate our methodology Academic Press AMS 99 subject classificatios: 62G07, 62H0, 60F05, 60G0. Key words ad phrases: additive models, :-mixig, asymptotic bias, asymptotic ormality, local liear estimate, kerel estimates. I recet years o-liear time series aalysis has gaied attetio because of limitatios of liear time series models i describig atural pheomea such as asymmetric limit cycles, time irreversibility, amplitude-depedet frequecy, ad chaos. See Tog (990), Grager ad Tera svirta (993), Tjo% stheim (994), ad Ha rdle et al. (997) for detailed expositios ad reviews of the curret literature. Moder computatioal power ad developmets i oparametric regressio ad curve estimatio are also major cotributors to the recet surge of iterest. There are may oliear forms to be explored. Of importace is the additive modelig. The iterest i Partially supported by NSF Grat DMS ad NSA Grat X Copyright 2000 by Academic Press All rights of reproductio i ay form reserved. 2

2 AVERAGE REGRESSION SURFACE 3 oliear additive time series ad regressio models has bee icreasig i ecoometrics as well as i related fields. See more refereces later. Let [X i, Y i ] i=& be joitly statioary processes with X i=(x T i, XT 2i )T ad X i ad X 2i takig values i R r ad R q, respectively, where the dimesios r0, q0, d=r+q, ad T deotes the traspose of a matrix or vector. Assume E Y < ad defie the multivariate regressio fuctio m(x, x 2 )=E(Y X =x, X 2 =x 2 ), (.) where (X, X 2, Y) has the same distributio as (X i, X 2i, Y i ). It is wellkow that the regressio fuctio m( }, } ) plays a predomiat role i data aalysis, i particular, i forecastig i the time series cotext. The additive regressio model is a useful statistical tool for high-dimesioal data aalysis. I this paper, we focus o the additive model m(x, x 2 )=++ f (x )+ f 2 (x 2 ), (.2) where + is a costat. I the above additive model, we assume that the variables X ad X 2 are cotiuous. For idetifiability, we assume without loss of geerality that E[ f (X )]=E[ f 2 (X 2 )]=0. This geeral setup was cosidered by Fa et al. (998; heceforth FHM) i a idepedet ad idetically distributed (iid) settig. The eed for oliear time series modelig ad forecastig (see Tog, 990; Che ad Tsay, 993a, 993b; Cai et al. 998; Cai ad Masry, 999) motivates us to cosider the above model for depedet data. Model (.2) is wide eough to iclude may useful statistical models i time series. Some of these are as follows. Cosider X t = f(x t&,..., X t& j )+g(x t&,..., X t& j2 ) & t, (.3) where f(}) ad g( } ) are Lebesgue measurable fuctios, ad [& i ] are a sequece of iid radom variables with mea zero ad variace oe. It is easy to show that the coditioal mea ad variace are respectively give by ad E(X t X t&,..., X t& j0 )= f(x t&,..., X t& j ) (.4) Var(X t X t&,..., X t& j0 )=g 2 (X t&,..., X t& j2 ), (.5) where j 0 =max( j, j 2 ). The classes defied by (.3) iclude may of more familiar oliear parametric models commoly ecoutered i ecoometrics (see, e.g., Tjo% stheim ad Auestad, 994, hereafter referred to as TA): the threshold model ad its various modificatios, the autoregressive coditioal

3 4 CAI AND FAN heteroscedastic (ARCH) model as defied by Egle (982), the expoetial autoregressive model itroduced i Tog (990), ad the multivariate adaptive splies (MARS) models i Lewis ad Steves (99). The oparametric kerel-type estimatio of the coditioal mea (see (.4)) ad the coditioal variace (see (.5)) was studied i detail by TA (994) usig the projectio method. The appeal of imposig the additive structure o f(}) or g(}) is i avoidig the so-called ``curse of dimesioality.'' I particular, the followig additive model is icluded i our settig via takig Y t =X t ad X t = (X t&,..., X t& j ) T : j X t =++ : j= f j (X t& j )+g(x t&,..., X t& j2 ) & t. The model is a useful extesio of the classical autoregressive model. Our approach will eable oe to costruct a explicit estimator of [f j (})] which possesses certai optimality criterio ad hece to predict the future value of the series. The geeral setup also icludes partial (semi-parametric) autoregressive models such as j X t =++ : j= j 2 f j (X t& j )+ : j= ; j X t& j & j+ g(x t&,..., X t& j3 ) & t. (.6) The models have flexibility of modelig some of compoets oparametrically (reducig possible modelig bias) ad other compoets liearly (reducig effective umber of parameters). We refer to the paper by Gao ad Liag (995) for the theoretical work (asymptotic ormality) to model (.6). There are may applicatios of model (.6) i various fields. For applicatios i ecoometrics, we refer to the book by Grager ad Tera svirta (993). A illustrative example of semi-parametric models is give by Egle et al. (986), who modeled electricity sales usig a umber of predictor variables. A similar situatio arose i Shumway et al. (988) i a study of mortality as a fuctio of weather ad pollutio variables i the Los Ageles regio. Also, see Tjo% stheim (994) for a brief discussio. Recetly, Li ad Pourahmad (998) used model (.6) to explore the Caadia lyx data, bechmark status i the literature of time series. Model (.2) icludes the additive model i the oparametric regressio with idepedet data q m(x,..., x q )=++ : j= f j (x j ). (.7) A thorough discussio of this model ca be foud i Buja et al. (989) ad Hastie ad Tibshirai (990) for the iid settig ad i Che ad Tsay (993b) for time series situatios. The additive compoets [f j (})] ca be

4 AVERAGE REGRESSION SURFACE 5 estimated with the oe-dimesioal oparametric rate; see, e.g., Stoe (985, 994) for details. I most papers, for the estimatio of additive compoets, algorithms have bee proposed, based o the iterative backfittig procedures such as the ACE algorithm ad the BRUTO algorithm. But their asymptotic properties are ot well uderstood due to the implicit defiitio of the estimator. For some recet theoretical developmets, see Opsomer ad Ruppert (997) ad Wad (999). Also, computatio ca be itesive i the high-dimesioal case ad the issues of algorithmic covergece ca arise. To atteuate the difficulty of the iterative procedures, i Auestad ad Tjo% stheim (990), TA (994), Lito ad Nielse (995), ad FHM (998), a direct method has bee proposed based o ``average regressio surface.'' The procedure was referred to as the ``projectio method'' by Auestad ad Tjo% stheim (990) ad TA (994) ad as the ``margial itegratio method'' by Lito ad Nielse (995) ad Lito (997). As poited out by FHM (998), the direct method has some advatages: it does ot use iteratios, it allows fast computatio to be implemeted, ad it allows detailed asymptotic aalysis. Efficiet estimatio of additive compoets was creatively studied i Lito (997) ad FHM (998) by usig two idepedet approaches. Lito (997) used the oe-step backfittig approach, ad FHM (998) employed the weight fuctio procedure. Masry ad Tjo% stheim (997) ad Cai ad Masry (999) exteded the applicability of the average surface idea to the additive oliear ARX time series. A useful modificatio of the average surface idea is give 3 i a upublished work by N. W. Hegarter. 2 The aim of this paper is to estimate the low-dimesioal additive compoet f ( } ) i (.2). Aalogously, f 2 ( } ) ca be estimated i the same fashio. The basic idea for estimatig f ( } ) is to first estimate directly the highdimesioal regressio surface m(x, x 2 ) ad the average the regressio surface over variables X 2 to stabilize the variace. The regressio surface is estimated by usig local polyomial fittig, which has bee studied extesively by, for example, Tsybakov (986), Fa (993), Ruppert ad Wad (994), Fa ad Gijbels (996), Masry (996), ad Masry ad Fa (997). It is well kow that the local liear method has advatages over the NadarayaWatso regressio estimator. I particular, it reduces bias of the NadarayaWatso estimator ad copes well with the edge effect. For more details, see Fa ad Gijbels (996). We show that the average regressio surface approach has the followig advatages. With a appropriate choice of the weight fuctio, additive compoets ca be efficietly estimated: A additive compoet ca be estimated with the same asymptotic bias ad variace as if the other compoets were kow. A applicatio of local 2 N. W. Hegarter, Rate optimal estimatio of additive regressio via the itegratio method i the presece of may covariate, submitted for publicatio.

5 6 CAI AND FAN liear fit reduces bias. The results reveal isightfully ew pheomea of additive modelig. The paper is orgaized as follows. I the ext sectio, we itroduce our estimatio procedure. I Sectio 3 the mai results of the paper, asymptotic bias ad ormality, are formulated, but their proofs are deferred to Sectio 6, based o some lemmas, which are proved i the Appedix. A applicatio to the additive model is discussed i Sectio 4. I Sectio 5, a small simulatio study is carried out to illustrate the estimatio ad the methodology is also applied to a real example. Fially, the assumptios used throughout the paper are gathered together i Sectio 6 for easy referece, followed by some brief commets. 2. AVERAGE OF REGRESSION SURFACE We first itroduce some otatio. Deote by X i =(X, i,..., X r, i ) T, X 2i = (X r+, i..., X d, i ) T, X i =(X T i, XT 2i )T =(X, i,..., X d, i ) T, ad m(x)=e(y X=x). Let W(}):R q R be a kow weight fuctio with E[W(X 2 )]=. Observe that, uder model (.2), where E[m(x, X 2 ) W(X 2 )]=++ f (x )+E[ f 2 (X 2 ) W(X 2 )] =+ 0 + f (x )#f *(x ), (2.) + 0 =++E[ f 2 (X 2 ) W(X 2 )]. (2.2) Thus, f ( } ) ca be costructed, withi a costat shift, via averagig the regressio surface with respect to variable X 2. This i tur suggests a direct estimatio procedure: Estimate the regressio fuctio m( } ) first ad the average out the estimated regressio surface with respect to the variable X 2. The costat factor is ot related to the fial estimator, sice f (}), i practice, is cetered to have mea zero for idetifiability purpose. This kid of averagig idea was studied by TA (994) uder time series models, ad by Lito ad Nielse (995) for the iid settig, ad was further exteded by FHM (998). The weight fuctio W( } ) is itroduced here to optimize the estimatio procedure ad to reduce the adverse impact of the estimated regressio surface at sparse regios o the estimate of the additive compoet. Also, it plays a role i the asymptotic derivatios. See Remark (below) ad later for details. Cosider the local liear approximatio of f (u ) at a fixed poit x, f (u )ra(x )+b T (x )(u &x ),

6 AVERAGE REGRESSION SURFACE 7 where u lies i a eighborhood of x. Also, the local costat approximatio of f 2 (u 2 ) at a fixed poit x 2 is applied: f 2 (u 2 )rc(x 2 ) for u 2 rx 2. Thus, we ca approximate m(u, u 2 ) locally by a liear term i a eighborhood of (x, x 2 ), m(u, u 2 )r#+; T (u &x ) (2.3) for some # ad ;, depedig o x ad x 2. The reaso for itroducig the local costat approximatio to the ``uisace fuctio'' f 2 ( } ) is to reduce the umber of local parameters so that the method ca be more easily implemeted i practice. Higher order approximatio ca also be employed for the fuctio f 2 ( } ) at expeses of itroducig more local parameters ad theoretical results cotiue to hold. Let K(}) ad L( } ) be the kerel fuctios ad let h >0 ad h 2 >0 be the badwidths. Give the observatios [X i, Y i ], cosider the multivariate weighted least squares : [Y i &#&; T (X i &x )] 2 K h (X i &x ) L h2 (X 2i &x 2 ), (2.4) where K h (})=K(}h )h r ad L h 2 (})=L( }h 2 )h q 2. Miimizig (2.4) with respect to # ad ; gives the estimates of # ad ;, respectively. Let #^ (x) ad ; (x) be the solutio to (2.4). Thus, our local liear estimator of m(}) is m^ (x)=#^ (x). The explicit expressio of m^ (x) is give i (6.) (below). By computig the weighted sample average of m^ ( } ), the followig average regressio surface estimator is proposed i FHM (998): f *(x )= : m^ (x, X 2i ) W(X 2i ), (2.5) f (x )=f *(x )&f * ad f *= : f *(X i ). (2.6) This is a fuctioal of m^ (x) ad possesses good samplig properties. The averagig reduces sigificatly the variace, but ot bias, of the resultig estimate f ( } ). This eables us to obtai a optimal estimate of f ( } ) via adjustig the badwidths. Note that whe the local costat fit is employed (i.e. ;=0) i (2.3), the resultig estimate #^ is the multivariate kerel regressio estimator, which was discussed i TA (994) ad Masry ad Tjo% stheim (997). For details, see the relatios (5) ad (6) i TA (994).

7 8 CAI AND FAN 3. MAIN RESULTS Although our iterest i additive modelig is motivated by the oliear time series aalysis, we itroduce our methods i a more geeral settig (:-mixig) which icludes time series modelig as a special case. Our theoretical results are derived uder :-mixig assumptio. Before we state our mai result, we itroduce the mixig coefficiet. Let F b be the _-algebra of evets geerated by [(X a i, Y i ); a jb]. The statioary process [X i, Y i ] i=& is called strogly mixig (:-mixig), if, :(k)=sup[ P(AB)&P(A) P(B) : A # F 0 &, B # F k ] a 0 as k. :(k) is called the strog mixig coefficiet i the literature. Amog various mixig coditios used i the literature, :-mixig is reasoably weak ad has may practical applicatios. May stochastic processes ad time series are kow to be :-mixig. Gorodetskii (977) ad Withers (98) obtaied various coditios for liear process to be :-mixig. Uder certai weak assumptios autoregressive ad more geerally biliear time series models are strogly mixig with mixig coefficiets decayig expoetially fast. Auestad ad Tjo% stheim (990) provided illumiatig discussios o the role of :-mixig (icludig geometric ergodicity) for model idetificatio i oliear time series aalysis. Uder some mild coditios, Masry ad Tjo% stheim (995, 997) showed that both ARCH process ad additive autoregressive process with exogeous variables, which are particularly popular i ecoometrics ad fiace, are statioary ad :-mixig. For easy referece, we itroduce the followig otatio. Let p(x, x 2 )be the joit desity of (X, X 2 ), ad p (x ) ad p 2 (x 2 ) be the margial desities of X ad X 2, respectively. Let &K& l = K(u) l du ad + 2 (K)= uut K(u) du. All limits will be take as ; this will ot be metioed explicitly i the body of the paper. The, uder Assumptios ()(9) stated i Sectio 6, we have the followig theorem which geeralizes oe of the mai results i FHM (998) to the depedet case. Theorem. Uder Assumptios ()(9) stated i Sectio 6, if the badwidths are chose such that h 0, h 2 0 i such a way that h r+4 =O(), h l 2 h 2 0, ad hr hq 2log, (3.)

8 AVERAGE REGRESSION SURFACE 9 the, (h r )2 [ f *(x )&f *(x )& 2 h2 tr[f" (x ) + 2 (K)]] w d N(0, v(x )), (3.2) where f " (x ) is the secod order partial derivative of f (x ), with ad v(x )=&K& 2 p (x ) E[ 2 (X) _ 2 (X) X =x ] (3.3) (x, x 2 )= p 2(x 2 ) W(x 2 ), (3.4) p(x, x 2 ) _ 2 (x)=var(y X=x). (3.5) I practice, a simple ad quick way to estimate the asymptotic variace i (3.3) is proposed as follows. Sice all curve estimates at a fixed poit are averages t= w, ty t of the observatios Y t, their variace ca be estimated by t= w2, t e2 t, where e t=y t &+^ & f (X it )&f 2(X 2t ) are the residuals. Here we implicitly use the fact that the local correlatio is egligible as show i Theorem. Remark. If we cosider weight fuctio W( } ) that miimizes v(x ), the optimal weight fuctio is where The optimal miimal variace is W(X 2 )=c & p(x, X 2 ) p (x ) _ 2 (x, X 2 ) p 2 (X 2 ), (3.6) c= p (x ) 2 E[_ &2 (X) X =x ]. v 0 (x )#mi v(x )= &K&2 W p (x ) [E[_&2 (X) X =x ]] &, (3.7) ad mi W v(x )=&K& 2 _ 2 p (x )if_ 2 (x)=_ 2. For details, see (3.4)(3.6) i FHM (998). Note that the optimal weight (3.6) depeds o ukow fuctios. FHM (998) proposed a method to choose the optimal weight fuctio based o the data. The idea is as follows: divide the sample ito a relatively small subsample ad a relatively large secod subsample; estimate the desig desities cosistetly by the first subsample; ad estimate the regressio fuctio usig the other subsample. This shows that the optimal

9 20 CAI AND FAN variace ca be achieved, at least theoretically. I the ideal situatio where f 2 ( } ) is kow, oe ca estimate f ( } ), by directly regressig Y& f 2 (X 2 )o X ad such a ideal estimator is optimal i a asymptotic miimax sese (see Fa, 993). Surprisigly, the average surface estimator (2.5) has the same asymptotic bias ad variace as the ideal estimator whe _ 2 (x) isa costat, eve though the former does ot rely o the kowledge of f 2 (}). See FHM (998) for the further commets. Theorem idicates that the asymptotic bias of f *(x ) is 2 h2_ tr[f" (x ) + 2 (K)] ad the asymptotic optimal variace is v 0 (x ). The optimal badwidth for estimatig f *(x ) ca be defied to be the oe miimizig the squared bias plus variace. Therefore, the optimal badwidth is give by h, opt = _ (r+4) rv 0 (x ) tr[f" (x ) + 2 (K)]& (r+4). (3.8) Recetly, Fa ad Gijbels (995) ad Ruppert et al. (995) developed data-drive badwidth selectio schemes based o asymptotic formulas for the optimal badwidths, which are less variable ad more effective tha the covetioal data-drive badwidth selectors such as the cross-validatio badwidth rule. Similar algorithms ca be developed for the estimatio of additive models based o (3.8); this is, however, beyod the scope of this paper. 4. AN APPLICATION TO ADDITIVE MODEL As a applicatio of model (.2), we ow cosider the additive model q m(x)=e(y X=x)=++ : k= g k (x k ), (4.) where [g k (})] are uivariate fuctios satisfyig the idetifiability coditio E[g k (X k )]=0, k=,..., q, + is a ukow parameter, ad X=(X,..., X q ) T is a cotiuous radom vector havig a joit desity p( } ). Our goal is to estimate each additive compoet g k ( } ) usig the average surface method. As i (2.), let g k *( } ) be the average of regressio fuctio g k *(x k )#E[m(X k ) W k (X &k )]= g k (x k )++ k, (4.2)

10 AVERAGE REGRESSION SURFACE 2 where + k =++ j{k E[ g j (X j ) W k (X &k )], X k =(X,..., X k&, x k, X k+,..., X q ) T, X &k =(X,..., X k&, X k+,..., X q ) T havig the desity p &k ( } ) ad W k (}):R q& R is the weight fuctio such that E[W k (X &k )]=. For the give observatios [X i, Y i ], the average surface estimator g^ k*( } ) is defied as i (2.6) but ow usig the badwidths h =h k ad h 2 =h 2k, g^ k*(x k )= : m^ (X k i ) W k(x &k i ), (4.3) where X k i =(X, i,..., X k&, i, x k, X k+, i,..., X q, i ) T, m^ ( } ) is the local liear estimator, ad X &k i =(X, i,..., X k&, i, X k+, i,..., X q, i ) T. Theorem 2. Suppose that the coditios of Theorem hold for each compoet k. The we have the joit asymptotic ormality \- h [ g^ *(x )&g *(x )& 2 h2 g" (x ) + 2 (K)] b +w d N(0, 7), (4.4) - h q [ g^ * q (x q )&g q *(x q )& 2 h2 g" q q(x q ) + 2 (K)] where 7=&K& 2 diag[_ 2 (x ),..., _ 2 q(x q )] with _ 2 k (x k)=p k (x k ) E 2 (X) p &k (X &k ) 2 W 2 k(x &k ) p 2 (X) } X k=x k&. (4.5) Remark 2. If the ideal weight fuctio i (3.6) applies to each additive compoet, the weight fuctio W k ( } ) should become W k (X &k )= p(xk ) p k (x k ) _ 2 (X k ) p &k (X )_ p(xk ) p k (x k ) dx &k &k _ 2 (X k ) & &, (4.6) ad the ideal variace is &K& 2 [E[_ &2 (X) X k =x k ]p k (x k )], which becomes &K& 2 _ 2 p k (x k ) whe _ 2 (x)=_ SIMULATION AND AN APPLICATION 5.. Simulated Example We begi the illustratio with a simulated example of model (.2). I the simulatio, the iovatioal series [= t ] are iid N(0, ). I this example, we

11 22 CAI AND FAN cosider a AR process with a sie fuctio ad a liear compoet. Figure a shows 300 observatios geerated from the model Y t = f (Y t&2 )+ f 2 (Y t&3 )+= t with f (x)=.5 si(0.5?x) ad f 2 (x)=0.75x (5.) ad Figs. b ad c give the scatterplots of Y t versus Y t&2 ad Y t agaist Y t&3, respectively. The estimated curves of f ( } ) ad f 2 ( } ) based o local liear fittig (dotted lie) ad NadarayaWatso method (dashed lie), coupled with the true curves (solid lie), are displayed i Figs. d ad e. FIG.. (a) Time series plot of simulated data from model (5.). (b) Scatterplot of Y t versus Y t&2 with true curve f ( } ). (c) Scatterplot of Y t versus Y t&3 with true curve f 2 (}). (d) ad (e) Solid lie true curve; dotted lie, local liear estimator; dashed lie, Nadaraya Watso estimator. (f) ad (g) Local liear estimate with its 950 cofidece itervals.

12 AVERAGE REGRESSION SURFACE 23 Whe we computed f ( } ), we used the badwidths h =0.75 ad h 2 =0.4 for the local liear method ad the NadarayaWatso approach as well, ad h =0.4 ad h 2 =0.75 were used for computig f 2( } ). Dashed lies i Figs. f ad g have bee added to idicate the poitwise variace of the curve estimates based o local liear fittig. These lies are differet from the curve estimates by.96 times the (estimated) poitwise stadard deviatio of the curve estimates, which give the approximate 95 0 cofidece iterval (without bias correctio). There is a strog evidece that local liear fittig performs well ad outperforms the NadarayaWatso method i terms of bias. It is ot surprisig because the NadarayaWatso uses the local costat approximatio to the regressio fuctio so that it could ot fit the peaks well (see Fig. d) Real Example Fially, we illustrate our methodology with the Caadia lyx data (o a atural logarithmic scale) for the years The time series plot is preseted i Fig. 2a ad the scatterplots of Y t versus Y t& ad Y t agaist Y t&2 are give i Figs. 2b ad 2c, respectively. There is a vast literature to explore this bechmark data set. See Tog (990) ad Li ad Pourahmad (998) for the detailed compariso of modelig methods. Accordig to Li ad Pourahmad (998), amog several models cosidered by them, the followig partial additive autoregressive model is oe of the best models Y t =:+;Y t& + f 2 (Y t&2 )+= t. (5.2) To uderstad better about the oliear structure of this data set, we cosider the additive model Y t = f (Y t& )+ f 2 (Y t&2 )+= t (5.3) by usig the techiques described above. Figures 2d ad 2e depict the estimated curves of f (}) ad f 2 ( } ) based o both local liear (solid lie) with the approximate 95 0 poitwise cofidece iterval (without bias correctio, dashed lies) ad NadarayaWatso (dotted lie). For both methods, we used the same badwidths h =0.9 ad h 2 =0.6 for the first compoet ad h =0.6 ad h 2 =0.9 for the secod compoet. By a compariso of Figs. 2d ad 2e with the other methods such as the semiparametric approach i Li ad Pourahmad (998, p. 99), we coclude that the local liear performs much better tha the NadarayaWatso. Also, this is further evidece that the NadarayaWatso fittig has larger bias ad serious boudary effects. Furthermore, we support the use of the semiparametric additive model (5.2) to aalyze the Caadia lyx data.

13 24 CAI AND FAN FIG. 2. (a) Time series plot of the Caadia lyx data. (b) Scatterplot of Y t versus Y t&. (c) Scatterplot of Y t versus Y t&2. (d) ad (e) Solid lie, local liear estimator; dashed lies, local liear estimator plusmius twice estimated stadard errors; dotted lie, Nadaraya Watso estimator. 6. CONDITIONS AND DERIVATIONS Before we embark o the proofs of theorems, let us collect the coditios to be used throughout the paper. Coditios. () The weight fuctio W( } ) has a bouded support D ad is uiformly cotiuous. The fuctio f 2 ( } ) is bouded o the support D. (2) The kerel fuctios K( } ) ad L( } ) are symmetric ad have bouded supports. Furthermore, L( } ) is a order l kerel.

14 AVERAGE REGRESSION SURFACE 25 (3) The fuctio f ( } ) has a bouded secod derivative i a eighborhood of x ad f 2 ( } ) has a bouded l th order derivative. Furthermore, for u i a eighborhood of x ad u 2 # D, the desity p(u, u 2 ) has bouded partial derivatives up to order 2 with respect to u ad up to order l with respect to u 2, ad it also satisfies if u # N(x ) x 2 # D where N(x ) is a eigborhood of x. p(u, x 2 )>0, (4) The fuctios _ 2 (u) ad b(u)=e( Y&m(u) 2+$ X=u) are cotiuous at the poit u =x, ad is bouded for all x for some $>0. E(b(X) (X) 2+$ X =x )< (5) The joit coditioal desity f (X, X i ) (Y, Y i ) of (X, X i ) give (Y, Y i ) satisfies, for all i> ad all values of argumets ivolved, for some positive costat M. f (X, X i ) (Y, Y i )(u, v y, y 2 )M< (6) The processes [X i, Y i ] are strogly mixig with ua [:(i)] $(2+$) < for some a>$(2+$), where $ is give i Assumptio (4). (7) Assume that there is a sequece of positive itegers satisfyig v ad v =o(- h r ) such that (hr )2 :(v ) 0. (8) The coditioal distributio of G( y u) of Y give X=u is cotiuous at the poit u =x. (9) h r h2q 2 log2 ad h 4 log hq 2 0. Remark 3. Cosider the popular choice for the badwidth h r =d&%* (d>0, 0<%*<); oe ca show that a sufficiet coditio for Assumptio (7) is :()=O( &\ ) with \>(+%*)(&%*). I particular, if %*=5, the \>32. Also, a sufficiet coditio for Assumptio (6) is :()= O( &\$ ) with \$>2+2$. Therefore, if %*=5, ad $=2, the a sufficiet coditio for Assumptios (6) ad (7) is :()=O( &\" ) with \">3. For details, see Masry ad Fa (997). I Assumptio (5), the joit desity is meat to be the distict radom variables i the set (X, X i, Y, Y i ). Note that this assumptio is also used i Masry ad Tjo% stheim (995, 997) for kerel type estimatio.

15 26 CAI AND FAN Note that by the omiated covergece theorem, it ca be easily show from Assumptio (8) that for ay J>0, the fuctios m J (u)=e(yi( Y J) X=u) ad _ 2 J (u)=var(yi( Y J) X=u) are cotiuous at the poit u = x. Also, for each L >0, _~ 2 J (u)= Var(YI( Y >J) X=u) is cotiuous at the poit u =x. Remark 4. Coditio (9) is imposed to simplify the proof of Theorem. I the proof, we approximate the matrix S & (see (6.)) by a determiistic sequece. If we used a higher-order stochastic expasio of S &, Coditio (9) could be weakeed. Note that if the local polyomial of order d is used to approximate the fuctio f 2 ( } ), the the result of Theorem cotiues to hold with relaxed Coditio (9) ad without coditios o the derivatives of p(x, x 2 ). I other words, these coditios are ot essetial to our estimatio procedure. Before we give the proofs of theorems, we first preset the explicit expressio for m^ (x). Let X* _(r+) =X*(x ) be the matrix with the i th row (, (X i &x ) T ) ad W* _ =W*(x) be the diagoal weight matrix with the i th diagoal elemet W i *(x)=k h (X i &x ) L h2 (X 2i &x 2 ), to the leastsquares problem (2.4). The, the simple algebra shows that m^ (x) ca be expressed as m^ (x)=e T S&(x) X*T (x ) W*(x) Y= : K (X i &x) Y i, (6.) where e T =(, 0,..., 0), S (x)=x* T (x ) W*(x) X*(x ), Y=(Y,..., Y ) T, ad K (t&x)=k (t &x, t 2 &x 2 ) =e T S & (x) \ t &x + K h (t &x ) L h2 (t 2 &x 2 ). (6.2) It follows from the least-squares theory that, for all x, : K (X i &x)= ad : K (X i &x)(x i &x )=0. (6.3) Proof of Theorem. Let X i =(x T, X T 2i) T. The, by (2.) ad Assumptios () ad (6), ad applyig the cetral limit theorem for statioary :-mixig

16 AVERAGE REGRESSION SURFACE 27 sequeces (see, for example, Theorem i Ibragimov ad Liik, 97, p. 346), we have Thus, f *(x )&f *(x )= : : m(x i ) W(X 2i )= f *(x )+O p ( &2 ). (6.4) [m^ (X i )&m(x i )] W(X 2i )+O p ( &2 ). (6.5) Let = i =Y i &E(Y i / i )=Y i &m(/ i ) ad s i (x)=m(/ i )&m(x)&f $ (x ) T _ (X i &x ). The, it follows from (6.)(6.3) that m^ (x)&m(x)= : K (/ i &x)[= i +s i (x)] =e T S& (x) \ }}} X &x }}} +e T S&(x) \ }}} X &x }}} X &x + W*(x) = X &x + W*(x) s(x), (6.6) where ==(= i ) _ ad s(x)=(s i (x)) _.LetH=diag[, h &,..., h & ] be a (r+)_(r+) diagoal matrix ad a =(h r h q 2log ) &2. The, owig to the uiform weak covergece of the kerel desity estimator (cf. Thm. i Masry, 996) ad by Assumptios (2)(6), we have S *(x)# HS (x) H= : W i X *(x)\ i &x h +2 coverges to S(x)=p(x)( (K)) i probability uiformly i x, where A 2 =AA T for a matrix or vector A, ad HS (x) H= : W i X *(x)\ i &x h =E{W *(x)\ X &x h = \ p(x) h + 2 (K) p (, 0) (x) +2 +2=+O p (a ) h p (, 0) (x) T + 2 (K) p(x) + 2 (K) + +O p(c ),

17 28 CAI AND FAN uiformly i x, where c =h 2 +hl 2 +a ad p (, 0) ( } ) deotes the vector of partial derivatives of p( } ) with respect to x. Now ote that \ p(x) h + 2 (K) p (, 0) (x) Therefore, = \p(x) 0 +O p (h 2 ). h p (, 0) (x) T & + 2 (K) p(x) + 2 (K) + & 0 + p(x) + 2 (K)+ h 0 p(x)\ + 2 (K) p (, 0) (x) p (, 0) (x) T + 2 (K) 0 + e T S& (x) H & = p & (x)(, h p (, 0) (x) T + 2 (K))+O p (c ). (6.7) Likewise, usig the same argumet as above, ad by Assumptios (2)(6), oe has H }}} \ X &x }}} = : W i *(x) s i (x)\ X &x + W*(x) s(x) X i &x +=O p (c ) (6.8) h uiformly i x. Substitute (6.7) ad (6.8) ito (6.6) to obtai e T S& (x) \ X &x }}} }}} =p & (x) { : W i *(x) s i (x) = + p & (x) p (, 0) (x) T + 2 (K) { : X &x + W*(x) s(x) W i *(x) s i (x)(x i &x ) =+O p(c 2 ) = 2 h2 tr[f" (x ) + 2 (K)]+ p & (x) B (x)+o p (h 2 )+O p(c 2 ), (6.9) where B (x)= : W i *(x)[ f 2 (X 2i )&f 2 (x 2 )][+p (, 0) (x) T + 2 (K)(X i &x )]. Also, ote that S* & (x)=s & (x)[i+(s(x)&s*(x)) S* & (x)].

18 AVERAGE REGRESSION SURFACE 29 The, S* & (x) coverges to S & (x) i probability uiformly i x. It the (x)=s & (x)(+o p ()), where o p () is uiform i x, ad (x) H & =HS & (x)(+o p ()). Therefore, follows that S* & S & e T S& (x) H & = p & (x) e T (+o p()). (6.0) Substitutig (6.0) ito (6.6), oe obtais e T S& (x) \ X &x }}} }}} X &x + W*(x) ==(+o p()) p & (x) T (x), (6.) where T (x)= : W i *(x) = i. Substitutig (6.9) ad (6.) ito (6.6), after some algebra, we obtai m^ (x)&m(x)= 2 h2 tr[f" (x ) + 2 (K)] +p & (x)[t (x)+b (x)]+o p (h 2 )+O p(c 2 ). (6.2) Thus, by (6.5), (6.2) ad the strog law of large umbers (see, for example, Cai ad Roussas, 992), we have f *(x )&f *(x )& 2 h2 tr[f" (x ) + 2 (K)] where with A(x)=W(x 2 )p(x), T *(x )= : A simple algebra leads to =T *(x )+B *(x )+o p (h 2 )+O p(c 2 +&2 ), (6.3) T (X i ) A(X i ) ad B *(x )= : T *(x )= : + : K h (X i &x ) (X i ) = i K h (X i &x )[ (X i )&(X i )]= i B (X i ) A(X i ). (6.4) #G (x )+G *(x ), (6.5)

19 30 CAI AND FAN where ( } ) is defied i (3.4) ad (x, x 2 )= : Substitutig (6.5) ito (6.3), oe has f *(x )&f *(x )& 2 h2 tr[f" (x ) + 2 (K)] L h2 (X 2i &x 2 ) A(X i ). (6.6) =G (x )+G *(x )+B *(x )+o p (h 2 )+O p(c 2 +&2 ). (6.7) I order to complete the proof, we eed the followig two lemmas but their proofs are relegated to the Appedix sice they are quite ivolved. To this ed, let ad = *=(X i i ) = i = [Y i&m(x i )] p 2 (X 2i ) W(X 2i ), (6.8) p(x, X 2i ) The, by (6.5) ad statioarity, `i=`i(x )=K h (X i &x ) = i *. (6.9) G (x )= : `i ad Var(G (x ))=Var(`)+2 : i=2\ &i + Cov(`, `i). Lemma. Uder the assumptios of Theorem, we have h r Var(G (x ))v(x ) ad h r : Cov(`, `i) 0, where v(x ) is defied i (3.3). i=2 Lemma 2. Uder the assumptios of Theorem, we have, G *(x )=o p ((h r )&2 ), ad B *(x )=o p ((h r )&2 ). It follows from (6.7), Lemma 2 ad the coditios o the badwidths (see (3.) ad Assumptio (9)) that f *(x )&f *(x )& 2 h2 tr[f" (x ) + 2 (K)]=G (x )+o p ((h r )&2 ). (6.20)

20 AVERAGE REGRESSION SURFACE 3 We remark that the third term i the left had side of (6.20) ca be viewed as the ``asymptotic bias'' of f *(x ), ad the ``asymptotic variace'' of f *(x ) is v(x ) defied i (3.3). We ow tur to show (3.2). This is equivalet to demostratig the asymptotic ormality of G (x ) i (6.20). I discussig the covergece i (3.2), we use the familiar techique of ``big blocksmall block'' procedure. More precisely, partitio the set [,..., ] ito 2k + subsets with large block of size u ad small block of size v, where k=k =w(u +v )x. Now we first cosider the choices of the block sizes. Assumptio (7) implies that there is a sequece of positive costats # such that # v =o(- h r ) ad # (h r ) 2 :(v ) 0. (6.2) Defie the large block size u by u =w(h r ) 2 # x ad the small block size v. The, it ca easily be show from (6.2) that, as, v u 0, u 0, u (h r )&2 0, ad (u ) :(v ) 0. (6.22) Igore the depedece o x, ad for j=,..., k, set r j *=( j&)(u +v ), ad Write r*+u j! j = : i=r j *+ r* j+ `i, ' j = : i=r j *+u + `i, ad! k+ = : i=r* k+ + `i. k G (x )= : j= k! j + : j= It will be show that, as, ' j +! k+ =G, +G,2 +G,3. (6.23) h r [E[G,2] 2 +E[G,3 ] 2 ] 0, (6.24) h r k : E(! 2 ) v(x j ), (6.25) j= k } E[exp(itG,)]& ` E[exp(it! j )] 0, (6.26) } j=

21 32 CAI AND FAN ad h r : k j= E I _!2 j \! ) j >= v(x h +& 0 (6.27) r for every =>0. (6.24) implies that G,2 ad G,3 are asymptotically egligible i probability; (6.26) shows that the summads [! j ] i G, are asymptotically idepedet; ad (6.25) ad (6.27) are the stadard LidebergFeller coditios for asymptotic ormality of G, for the idepedet setup. Let us first establish (6.24). Observe that k E(G,2 ) 2 = : j= Var(' j )+2 : l< jk Cov(' l, ' j )#I +I 2. (6.28) It follows from statioarity ad Lemma that I =k Var(' )=k Var \ v : `i+ =k v h &r [v(x )+o()]. (6.29) Next cosider the secod term I 2 i the right had side of (6.28). Sice r j *&r l *u for all j>, we therefore have I 2 2 : l< jk &u 2 : j = v : j = v : j 2 = : Cov(`j, `j2 ). j 2 = j +u By statioarity ad Lemma, oe obtais I 2 2 : i=u + Hece by (6.28)(6.30), we have Cov(`rl *+u + j, `rj *+u + j 2 ) Cov(`, `i) =o(h &r ). (6.30) h r E[G,2] 2 =O(k v & )+o()=o(). (6.3) It follows from statioarity, (6.22) ad Lemma that Var(G,3 )=Var \ &k (u +v ) : `i+ =O((&k (u +v )) h &r )=o(h&r ). (6.32)

22 AVERAGE REGRESSION SURFACE 33 Combiig (6.28), (6.3), ad (6.32), we establish (6.24). As for (6.25), by statioarity, (6.22) ad Lemma, it is easily see that h r k : E(! 2 j )= hr k E(! 2 )= k u } hr Var \ u u : j= `i+ v(x ). I order to establish (6.26), we make use of Lemma. i Volkoskii ad Rozaov (959) (see also Ibragimov ad Liik, 97, p. 338) to obtai k } E[exp(itG,)]& ` E[exp(it! j )] }6(u ) :(v ) j= tedig to zero by (6.22). Fially, we will establish (6.27). To this ed, we ow employ a trucatio techique as follows. Let b J ( y)=yi( y J), where J is a fixed positive umber, ad `J =`J (x, i, i )=[b J (Y i )&m J (X i )] K h (X i &x ) (X i ), the, G (x )=G J (x J )+G (x ), where G J (x )= & : J `J, i ad G (x )= & : [`i&`j, i]. By usig the same argumets as those employed i the proof of Lemma, oe has, as, v, J (x )=h r Var(GJ ) v J(x )=&K& 2 p (x ) E[_ 2 J (X) 2 (X) X =x ]. The boudedess of K( } ) ad ( } ) implies that `J, i Bhr for some B>0. This i tur implies that - h r max jk! J j Bq - h r 0, by (6.22). Therefore, [! J <= - v j J(x ) h r ] is a empty set whe is large sufficietly. Hece, it follows that (6.27) holds true for! J. Cosequetly, j we have established the followig asymptotic ormality as. Observe that, for ay t # R, E exp(it - h r G )&exp(&v(x ) t 2 2) - h r GJ (x ) w d N(0, v J (x )) (6.33) E exp(it - h r (GJ +G J ))&exp(&v J(x ) t 2 2) + exp(&v J (x ) t 2 2)&exp(&v(x ) t 2 2) E exp(it - h r GJ )&exp(&v J(x ) t 2 2) +E exp(it - bh r G J )& + exp(&v J (x ) t 2 2)&exp(&v(x ) t 2 2).

23 34 CAI AND FAN As, the first term goes to 0 by (6.33) for each J>0 ad the third term also goes to 0 as J by the domiated covergece theorem. I order to complete the proof, it suffices to show that the secod term goes to 0 as ad the J. To this ed, usig the fact that e ix & 2 x for all x # R ad the CauchySchwartz iequality, we have E exp(it - h r G J )& 2 - hr Var(G J ). J Note that G has the same structure as G, except that Y i is replaced by Y i I( Y i >J). The, usig the same argumets as those used i the proof of Lemma, we obtai, as, h r Var(G J ) &K&2 p (x ) E[_~ 2 J (X) 2 (X) X =x ]. It ca be easily show that the right had side goes to 0 as J. This completes the proof of the theorem. K Proof of Theorem 2. By (6.20), the direct estimator of each compoet g^ k*(x k ) has the stochastic represetatio where g^ k*(x k )&g k *(x k )=T k, + 2 h2 k g" k(x k ) + 2 (K)+o p ((h k ) &2 ), (6.34) T k, = : K hk (X k, i &x k ) k (X k i ) = i ad k (X k )= W k(x &k ) p &k (X &k ). p(x k ) I order to show (4.4), it suffices to show from (6.34) that \- h T, b +N(0, 7), (6.35) - h q T q, It suffices to show from Theorem that the asymptotic covariace betwee T k, ad T l, should be zero for k{l. I other words, we will show that - 2 h k h l Cov(T k,, T l, ) 0. (6.36)

24 AVERAGE REGRESSION SURFACE 35 To this effect, by statioarity, we have Cov(T k,, T l, ) = E[K h k (X k &x k ) K hl (X l &x l ) k (X k ) l (X l ) = 2 ] + 2 : i {i 2 E[K hk (X k, i &x k ) K hl (X l, i2 &x l ) k (X k i ) l (X l i 2 ) = i = i2 ] #F +F 2. (6.37) It is easily see by Thm. i Su (984) that F =O( & ). (6.38) Employig the same argumets as those used i the proof of (A.5) (below), ad by statioarity, we have F 2 : [ E[K hk (X k, &x k ) K hl (X l, i &x l ) k (X k ) l(x l ) = i = i ] i=2 +E[K hk (X k, i &x k ) K hl (X l, &x l ) k (X k i ) l(x l ) = = i ] ]=O( & ). This, i cojuctio with (6.37) ad (6.38), cocludes that (6.36) holds true. Therefore, this completes the proof of the theorem. K APPENDIX: PROOFS OF LEMMAS Proof of Lemma. It is easily see by statioarity that Var(G (x ))=Var(`)+2 : i=2\ &i + Cov(`, `i)#j +J 2. (A.) For J, sice E(`i)=0, we have h r J =h r E(`2 )= K 2 (u ) _ 2 (x +h u, u 2 ) _ 2 (x, u 2 ) p(x +h u, u 2 ) du du 2.

25 36 CAI AND FAN By Thm. i Su (984) ad Assumptios (2)(4), we the have h r J &K& 2 _2 (x, u 2 ) 2 (x, u 2 ) p(x, u 2 ) du 2 = &K& 2 p (x ) E[_ 2 (X) 2 (X) X =x ]=v(x ). (A.2) It remais to show that h r J 2 0. To this ed, choose a sequece of positive itegers satisfyig? =O(h &$ra(2+$) ), where a is give i Assumptio (6). The? =o(h &r ) ad h&$r(2+$) : : $(2+$) (i) 0. i? (A.3) We decompose the sum ito three terms due to the possible overlap betwee X ad X i, : i=2 d& Cov(`, `i) = : i=2? + : i=d + : i=? + #J 2 +J 22 +J 23, (A.4) For J 2, there is a overlap betwee the compoets of X ad X i but ot i J 22 or J 23. Let us cosider J 22 first. By Assumptios ()(5), ad the CauchySchwartz iequality, we have Cov(`, `i) = E(``i) M &x K h r \u h + (x, u 2 ) h r Hece, _[E(Y 2 )+ m(u)+m(v) E Y + m(u) m(v) ] du du 2 dv dv 2 &x K \v h + (x, v 2 ) cost. W(u 2) W(v 2 ) [E(Y 2 )+ f 2(u 2 )+ f 2 (v 2 ) _[E Y + f (x ) ]+2 f (x ) E Y +f 2 (x )+ f (x ) _ f 2 (u 2 )+ f 2 (v 2 ) + f 2 (u 2 ) f 2 (v 2 ) ] p 2 (u 2 ) p 2 (v 2 ) du 2 dv 2 <. (A.5)? h r J 22 cost. : i=d h r =O(? h r )=o() (A.6)

26 AVERAGE REGRESSION SURFACE 37 by the choice of?. Next, work with J 2. To this ed, let r~ be the umber of the commo elemets i (X, X i ). Employig the exactly same argumets as those used i the proof of (6.0) i Masry ad Tjo% stheim (997) ad (A.5), oe ca show that d& h r J 2 cost. : i=2 h r&r~ =O(h )=o(). (A.7) For J 23, we apply Davydov's iequality (see, e.g., Hall ad Heyde, 980, Corollary A.2) to obtai, It is easily see that h (+$) r E ` 2+$ =: Cov(`, `i) 8: $(2+$) (i&)(e ` 2+$ ) 2(2+$). u 3 K(u ) 2+$ b(x +h u, u 2 ) (x, u 2 ) 2+$ p(x +h u, u 2 ) du du 2 &K& 2+$ p (x ) E(b(X) (X) 2+$ X =x )< as. Therefore, Thus, E ` 2+$ cost. h &(+$) r. Cov(`, `i) cost. : $(2+$) (i&) h &2(+$) r(2+$). (A.8) Thus, h r J 23 cost. h &$r(2+$) : i? : $(2+$) (i) 0, (A.9) as by (A.3). Thus, by (A.4), (A.6), (A.7) ad (A.9), we have h r : Cov(`, `i) 0. Cosequetly, i=2 h r J 2 0 (A.0) Combiig the above expressio with (A.) ad (A.2) completes the proof of the lemma. K

27 38 CAI AND FAN Proof of Lemma 2. By (6.5) ad (6.6), G*(x )= : K h (X i &x )[ *(X i )&(X i )]= i + : K h (X i &x )[ (X i )& *(X i )]= i #G*, (x )+G*,2 (x ), (A.) where ( } ) is defied i (6.6) ad *(x)=e[ (x)]= R q L(u) A(x, x 2 +h 2 u) p 2 (u) du. (A.2) Clearly, as, *(x) (x). (A.3) Let { i ={, i (x )=K h (X i &x )[ *(X i )&(X i )]= i. The, h r Var(G*,(x ))=h r Var({ )+2h r : i=2\ &i + Cov({, { i )#F 3 +F 4. (A.4) A simple algebra gives F 3 =h r E[K 2 h (X &x )[ *(X )&(X )] 2 _ 2 (X )] = K 2 (u )[ *(x, u 2 )&(x, u 2 )] 2 _p(x +h u, u 2 ) _ 2 (x +h u, u 2 ) du du 2 =o() (A.5) by (A.3). Similar to (A.4), we decompose the sum ito three terms due to the possible overlap betwee X ad X i, : i=2 d& Cov({, { i ) = : i=2? + : i=d + : i=? + #F 4 +F 42 +F 43, (A.6) For F 4, there is a overlap betwee the compoets of X ad X i but ot i F 42 or F 43. For F 4, by the CauchySchwartz's iequality ad (A.5), we have h r F 4=o(). (A.7)

28 AVERAGE REGRESSION SURFACE 39 Followig the same lies as those employed i the proof of (A.6) ad (A.0), we have h r F 42=o() ad h r F 43=o(). This, i cojuctio with (A.4)(A.7), implies that h r Var(G*,(x ))=o(). (A.8) Next we show that G*,2 (x ) is egligible. To this ed, let F ( } ) deote the empirical distributio of [X 2i ], ad let F( } ) be the distributio of X 2. By (6.6) ad (A.2), we obtai (x)& *(x)= R q L h2 (u&x 2 ) A(x, u) d[f (u)&f(u)]. (A.9) Let L ( } ) be the Fourier trasform of L( } ). Substitute L(u)=(2?) &q e &iv } u L (v) dv R q ito (A.9) to obtai (x)& *(x)=(2?) &q R q L (v) e iv } x 2h 2 dv _ R q h q 2 e &iv } uh 2 A(x, u) d[f (u)&f(u)] = L (v) e iv } x 2h 2 I 2 (x, v) dv, (A.20) R q where I 2 (x, v)=(2?) &q R q h q 2 e iv } uh 2 A(x, u) d[f (u)&f(u)]. Substitutig (A.20) ito G*,2 (x ) of (A.), we have G*,2 (x )= I (x, v) I 2 (x, v) L (v) dv, R q (A.2) where I (x, v)= : e iv } X 2ih 2 K h (X i &x ) = i.

29 40 CAI AND FAN I (x, v) ca be aalyzed by followig the same lies as those employed i the proof of Lemma to obtai sup v # R q E[ I 2 (x, v) ]=O((h r )& ). (A.22) As i (6.6) of Masry ad Tjo% stheim (997), sup v # R q E[ I 2 2 (x, v) ]=O((h 2q 2 )& ). (A.23) By the CauchySchwartz iequality, (A.2)(A.23) ad Assumptios (2) ad (9), E G*,2 (x ) sup [E[ I 2 (x, v) ]E[ I 2 (x 2, v) ]] 2 L (v) dv v # R q R q =O(( 2 h r h2q 2 )&2 )=o((h r )&2 ). (A.24) This completes the proof of the first part of lemma. Fially, as i FHM (998), by calculatio of the first two momets i the maer of the proofs of Lemma ad the first part of this lemma, oe ca show that B *(x )=o p ((h r ) &2 ). This cocludes the lemma. K ACKNOWLEDGMENT The authors thak the Editor ad the two referees for their isightful commets, which led to a substatial improvemet of the paper. REFERENCES B. Auestad ad D. Tjo% stheim, Idetificatio of oliear time series: First order characterizatio ad order determiatio, Biometrika 77 (990), A. Buja, T. J. Hastie, ad R. J. Tibshirai, Liear smoothers ad additive models (with discussio), A. Statist. 7 (989), Z. Cai, J. Fa, ad Q. Yao, Fuctioal-coefficiet regressio models for oliear time series, J. Amer. Statist. Assoc. (998), to appear. Z. Cai ad E. Masry, Noparametric estimatio of additive oliear ARX time series: Local liear fittig ad projectio, Eco. Theory (999), to appear. Z. Cai ad G. G. Roussas, Uiform strog estimatio uder :-mixig, with rates, Statist. Probab. Lett. 5 (992), R. Che ad R. Tsay, Fuctioal coefficiet autoregressive models, J. Amer. Statist. Assoc. 88 (993a),

30 AVERAGE REGRESSION SURFACE 4 R. Che ad R. Tsay, Noliear additive ARX models, J. Amer. Statist. Assoc. 88 (993b), R. F. Egle, Autoregressive coditioal heteroscedasticity with estimates of the variace of U.K. iflatio, Ecoometrica 50 (982), R. F. Egle, C. W. J. Grager, J. Rice, ad A. Weiss, Semiparametric estimates of the relatio betwee weather ad electricity sales, J. Amer. Statist. Assoc. 8 (986), J. Fa, Local liear regressio smoothers ad their miimax, A. Statist. 2 (993), J. Fa ad I. Gijbels, Data drive badwidth selectio i local polyomials fittig: Variable badwidth spatial adaptatio, J. Roy. Statist. Soc. Ser. B 57 (995), J. Fa ad I. Gijbels, ``Local Polyomial Modelig ad Its Applicatios,'' Chapma 6 Hall, Lodo, 996. J. Fa, W. Ha rdle, ad E. Mamme, Direct estimatio of low dimesioal compoets i additive models, A. Statist. 26 (998), J. Gao ad H. Liag, Asymptotic ormality of pseudo-ls estimator for partly liear autoregressive models, Statist. Probab. Lett. 23 (995), V. V. Gorodetskii, O the strog mixig property for liear sequeces, Theory Probab. Appl. 22 (977), 443. C. W. J. Grager ad T. Tera svirta, ``Modelig Noliear Ecoomic Relatioships,'' Oxford Uiv. Press, Oxford, 993. P. Hall ad C. C. Heyde, ``Martigale Limit Theory ad Its Applicatios,'' Academic Press, New York, 980. W. Ha rdle, H. Lu tkepohl, ad R. Che, A review of oparametric time series aalysis, Iterat. Statist. Rev. 65 (997), T. J. Hastie ad R. J. Tibshirai, ``Geeralized Additive Models,'' Chapma 6 Hall, Lodo, 990. I. A. Ibragimov ad Yu. V. Liik, ``Idepedet ad Statioary Sequeces of Radom Variables,'' WaltersNoordhoff, Groige, The Netherlads, 97. P. A. W. Lewis ad J. Steves, Noliear modelig of time series usig multivariate adaptive splies (MARS), J. Amer. Statist. Assoc. 86 (99), T. C. Li ad M. Pourahmad, Noparametric ad o-liear models ad data miig i time series: A case-study o the Caadia lyx data, Appl. Statist. 47 (998), O. B. Lito, Efficiet estimatio of additive oparametric regressio models, Biometrika 84 (997), O. B. Lito ad J. P. Nielse, A kerel method of estimatig structured oparametric regressio based o margial itegratio, Biometrika 82 (995), E. Masry, Multivariate local polyomial regressio estimatio for time series: Uiform strog cosistecy ad rates, J. Time Ser. Aal. 7 (996), E. Masry ad J. Fa, Local polyomial estimatio of regressio fuctios for mixig processes, Scad. J. Statist. 24 (997), E. Masry ad D. Tjo% stheim, Noparametric estimatio ad idetificatio of oliear ARCH time series: Strog covergece ad asymptotic ormality, Eco. Theory (995), E. Masry ad D. Tjo% stheim, Additive oliear ARX time series ad projectio estimates, Eco. Theory 3 (997), J. D. Opsomer ad D. Ruppert, Fittig a bivariate additive model by local polyomial regressio, A. Statist. 25 (997), 862. D. Ruppert, S. J. Sheather, ad M. P. Wad, A effective badwidth selectio for local least squares regressio, J. Amer. Statist. Assoc. 90 (995), D. Ruppert ad M. P. Wad, Multivariate weighted least squares estimatio, A. Statist. 22 (994),

31 42 CAI AND FAN R. H. Shumway, A. S. Azari, ad Y. Pawita, Modelig mortality fluctuatios i Los Ageles as fuctios of pollutio ad weather effects, Evirometric Rev. 45 (988), C. J. Stoe, Additive regressio ad the oparametric models, A. Statist. 3 (985), C. J. Stoe, The use of polyomial splies ad their tesor products i multivariate fuctio (with discussio), A. Statist. 22 (994), 884. Z. Su, Asymptotic ubiased ad strog cosistecy for desity fuctio estimator, Acta Math. Siica 27 (984), D. Tjo% stheim, No-liear time series: A selective review, Scad. J. Statist. 2 (994), D. Tjo% stheim ad B. Auestad, Noparametric idetificatio of oliear time series: Projectios, J. Amer. Statist. Assoc. 89 (994), H. Tog, ``Noliear Time Series: A Dyamical System Approach,'' Oxford Uiv. Press, Oxford, 990. A. B. Tsybakov, Robust recostructio of fuctios by the local approximatios method, Prob. Iform. Trasmissio 22 (986), V. A. Volkoskii ad Yu. A. Rozaov, Some limit theorems for radom fuctios, I, Theory Probab. Appl. 4 (959), M. P. Wad, A cetral limit theorem for local polyomial backfittig estimators, J. Multivariate Aal. 70 (999), C. S. Withers, Coditios for liear processes to be strog mixig, Z. Wahrsch. Geb. 57 (98),