In our work we study the structure of polynomials of degree three and four that have high bias or high Gowers norm, over arbitrary prime fields. In particular we obtain the following results. We give a canonical representation for degree three or four polynomials that have a significant bias (i.e. they are not equidistributed). This result generalizes the corresponding results from the theory of quadratic forms. This significantly improves previous results for such polynomials.For the case of degree four polynomials with high Gowers norm we show that (a subspace of constant co-dimension of) F^n can be partitioned to subspaces of dimension \Omega(n) such that on each of the subspaces the polynomial is equal to some degree three polynomial. It was previously shown that a quartic polynomial with a high Gowers norm is not necessarily correlated with any cubic polynomial. Our result shows that a slightly weaker statement does hold. The proof is based on finding a structure in the space of partial derivatives of the underlying polynomial. Joint work with Amir Shpilka.