Most infinitely wide NTK and NNGP kernels are based on the ReLU activation. In we propose a method of computing neural kernels with *general* activations. For homogeneous activations, we approximate the kernel matrices by linear-time sketching algorithms.

We open-source NNGP and NTK for new activations within the Neural Tangents dev and sketching algorithm at Joint work with Amir Zandieh @ARomanNovak @hoonkp @Locchiu @aminkarbasi.

Still, .computing full NTK matrices is a big pain, e.g., 5-layer Convolution NTK requires 151 GPU hours. We accelerate the NTK approximation by sketching techniques and provide a tight point-wise error bound. Our approximation takes only 1.5 GPU hours (x106 speedup) 🫢.

We study another simple approach for the dual kernel by Gauss-Hermite quadrature:

In addition, we propose how to automatically compute the dual kernel of the derivative without the activation, which is useful to characterize the NTK with an unknown activation (e.g., normalized Gaussian) or whose dual kernel of the derivative is unavailable (e.g., GeLU, ELU).

Our derivations are based on (1) an explicit expression of dual kernel by Hermite polynomials and (2) the fact that Hermite polynomials can play a role of random features of monomial kernels. They allow inputs from the entire *R^d* space.

We first characterize a kernel function of a single-layer neural network (a.k.a. duel kernel) of various activations. This is a key block for the NNGP and NTK of deeper architectures.

RT @dohmatobelvis: Good news: Our paper on Scalable learning and MAP inference in nonsymmetric Determinantal Point Processes .

RT @dohmatobelvis: Happy to share our recent preprint with .@mikegartrell, Insu Han, V.-E. Brunel, J. Gillenwater,.on Scalable learning and….